--- Log opened Wed Nov 27 00:00:07 2013
--- Day changed Wed Nov 27 2013
01:19 < warren> http://www.pcworld.com/article/2067400/link-between-satoshi-bitcoin-account-and-the-silk-road-dissolves.html
01:19 < warren> jgarzik was quoted.
02:32 < phantomcircuit> 2013-11-27 07:24:42 ProcessMessages(ping, 0 bytes) : Exception 'CDataStream::read() : end of data' caught, normally caused by a message being shorter than its stated length
02:32 < phantomcircuit> 2013-11-27 07:24:42 ProcessMessage(ping, 0 bytes) FAILED
02:32 < phantomcircuit> what the dicks
02:32 < warren> phantomcircuit: there's an entire thread on this
02:33 < warren> phantomcircuit: I found one way to do that by accident too
02:35 < phantomcircuit> well anyways
02:35 < phantomcircuit> this servers connection slots are 100% used
02:35 < phantomcircuit> im going to restart with maxconnections=512
02:35 < warren> phantomcircuit: huh?  legit peers?
02:36 < warren> phantomcircuit: wait
02:36 < warren> phantomcircuit: what version of the client?
02:37 < phantomcircuit> master/HEAD
02:37 < phantomcircuit> appear to be legit peers
02:37 < warren> phantomcircuit: https://github.com/litecoin-project/bitcoinomg/commits/0.8.5-OMG5   Use this bitcoin branch.  Among other things it has some of the useful debug.log print stuff from master that tells you more information about peers in real-time.
02:37 < warren> oh
02:37 < warren> nm
02:52 < midnightmagic> Nice quote, Shamir: In his email, Shamir said Bitcoin enthusiasts do not like analyses that do "not fully support their beliefs." He also took a swipe at the media.
02:52 < midnightmagic> What an utter, utter jerk.
02:54 < phantomcircuit> lol seriously
02:54 < phantomcircuit> what a retard
03:26 < gmaxwell> Yes, we do not like analyses which do not support our belief in objective reality.
03:48 < BlueMatt> midnightmagic: coming from the guy who originally did an analysis of the "chain html"?
03:49 < BlueMatt> to be fair, no one likes analyses which do not support their own belief in reality...some are willing to accept them, others are fox news
04:07 < sipa> i'm sure the bitcoin community is full of people that don't analyses that don't support their beliefs
04:08 < sipa> still doesn't mean you should go make claims that are trivially falsifiable with a google search
04:08 < gmaxwell> hah, yes indeed its true in a very empty sense. :) I've lamented the groupthink downvotes I get on reddit. :P
04:09 < warren> BlueMatt: to be fair, it's easy to believe whoever pays your paycheck
04:13 < gmaxwell> even when you think you're trying not to. :(
04:14 < warren> Just create an environment where a 24 hour news network and all your friends agree with you.
04:18 < BlueMatt> gmaxwell: human nature. it sucks.
04:18 < BlueMatt> warren: to be fair, i never said all the people there actually agree, just that they say things...
04:19 < warren> BlueMatt: I know, I was joking.
04:48 < Emcy> still complaining about that paper?
05:08 < _ingsoc> I didn't know about this place.
05:09 < mappum> one mention and 5 people join
05:09 < _ingsoc> xD
05:17 < mappum> gmaxwell: you were saying it's an issue that my PoW is outsourceable, i'm not sure that's actually a problem
05:17 < mappum> as long as the miner can serve the data when requested it's not broken
05:18 < mappum> and what is bad about cloud mining?
05:18 < gmaxwell> mappum: because it means there will only just be one copy of the data in the world at some big pool and thats it.
05:18 < warren> gmaxwell: I was really hoping more people wouldn't join here.
05:19 < petertodd> warren: we should fidelity-bond admission
05:19 < warren> or at least require showing a wizard diploma
05:20 < mappum> Hogwarts class of '11 here
05:20 < gmaxwell> warren: yea, well. It's people flooding #bitcoin-dev with talk about Proof-of-foo functions.
05:21 < gmaxwell> it's material for this channel, though indeed I like it quite in here too. :)
05:21 < petertodd> warren: proof-of-wizard
05:21 < warren> I read that as proof-of-poo
05:21 < warren> and that would have been better
05:21 < gmaxwell> mappum: in any case if you don't think thats a problem then ... maybe it's not.
05:21 < gmaxwell> though if your goal is to achieve good distribution of your data, then I'm afraid you may fail.
05:22 < swulf--> I have a solution I'll try to put up tonight that helps with distribution of data and separates miner/storage requirements
05:23 < swulf--> I have a strategy (I hope) that will incentivize as many people as possible to try and complete to claim the storage for data
05:23 < swulf--> compete*
05:23 < mappum> well part of this is a DHT, and if the work hash included the miner's DHT ID hash, i think it might fix it
05:25 < swulf--> I think a DHT of sorts is a requirement for any service like this
05:25 < gmaxwell> -EWANKDETECTED
05:25 < swulf--> gmaxwell: Yes, my apologies. But I am thoroghly excited about it;)
05:26 < mappum> me too :)
05:26 < warren> If you mention DHT your wizard license is automatically revoked.
05:26 < gmaxwell> nah not you.. sorry, like, I've developed a pratice of reflexively ignoring anyone who says DHT. It's usually invoked by people who encounter a problem they don't understand and it really means "magical distributed thingy". Like early physicists invoking god when they encountered something they couldn't explain.
05:27 < Emcy> dht is a great technology thoighh
05:27 < swulf--> I specifically mention using a kademlia network, where hashes of pubkeys are used as node-ids in the network and used to locate and store data.
05:27 < mappum> so you're saying they are magic... that means i'm a wizard?
05:27 < petertodd> mappum: around here wizards understand how their spells work
05:28 < mappum> i understand DHTs well enough
05:28 < petertodd> mappum: then you would know they don't hold up well to attack
05:28 < swulf--> petertodd: why?
05:28 < swulf--> distributed networks are all prone to attacks
05:29 < petertodd> swulf--: yes, which is the beauty of bitcoin systems where you ask very, very, little of the network
05:29 < mappum> i would think it holds up to attacks a lot more than anything that isn't distributed
05:29 < swulf--> petertodd: agreed. but I think in a paid-for kademlia net, you can sign messages saying you have a right to retrieve data.. should alleviate some DoS.
05:30 < mappum> so you mean attackers using a lot of resources? the thing i was talking about that would use that would be fine against that since it requires fees
05:30 < petertodd> mappum: you can play all kinds of games with DHT's, for instance manipulating hashes to cause biases in the key distribution
05:30 < petertodd> mappum: or sybil attacking part of the keyspace and then deleting the data
05:32 < petertodd> mappum: doesn't mean you can't fix this stuff, but in our experience people promoting DHT's don't realize the issues
05:32 < adam3us> kind of surprised at shamir - the guy is a crypto genius, to get suckered into co-authoring such a paper with unsupported claims.  about the best they could've said is the 'data doesnt disprove' until the guy stepped forward and provided more data that did disprove!
05:32 < mappum> good point, i'll have to think about that
05:33 < adam3us> dht's usuall have extremely poor even non-hostile user characteristics, dht in a byzantine threat environment with real money on the line
05:33 < petertodd> adam3us: "suckered" depends on how much money he got...
05:35 < adam3us> yeah generally i heard he's a nice guy - i mean his profile is like rivest, but he doesnt even charge a fraction of what he could for review work.  he's done a ton of cutting edge crypto stuff, in many areas of it. secret sharing, fiat shamir transform, differential cryptanalysis the publication list is huge and usually cutting eduge
05:35 < gmaxwell> yea, dht's basically appear to be unworkable in an adversarial enviroment.
05:36 < petertodd> adam3us: he could be simply naive, or not as sharp as he used to be
05:36 < mappum> don't they hold up well in BitTorrent?
05:36 < gmaxwell> petertodd: you are just young enough that the fact that intellect declines with age doesn't scare you shitless yet.
05:37 < gmaxwell> mappum: not at all, in fact. go try fetching the bitcoin blockchain torrent with no trackers.
05:37 < petertodd> gmaxwell: I've got enough older friends to be scared shitless in an exestential way already...
05:39 < mappum> interesting
05:42 < gmaxwell> mappum: bittorrent dht is mostly fail, it works .. kinda.. for very large swarms that can also do peer exchange, but mostly it just ends up helping people find other trackers. For small swarms it'll often spin finding nothing even when its not getting attacked.
05:42 < gmaxwell> It made sense in the bittorrent model because it was just enough more to make it so that you couldn't kill one (or a couple) original trackers and take out a swarm.
05:43 < midnightmagic> lol
05:43 < midnightmagic> you're never getting away from dhts are you
05:43 < gmaxwell> http://bitcoinstats.com/irc/bitcoin-dev/logs/2012/04/16#l1334585717
05:44 < mappum> sorry, i didn't know it was such a hot button D:
05:44 < midnightmagic> haha
05:45 < gmaxwell> meh, it's not a hot button, it's just .. common. Well, not in #wizards.
05:45 < gmaxwell> But there was a period of time when we couldn't go days without someone joing #bitcoin-dev and responding to the very first thing they heard with USE A DHT.
05:45 < midnightmagic> mappum: the endless, endless stream of users who come in to #bitcoin and insist we adopt dht rather than dns/irc for initial peer discovery is really astounding. it's jsut a running joke is all. no worries man.
05:46 < petertodd> mappum: pro-tip: suggest fidelity bonds instead, like a fidelity-bonded DHT
05:46 < gmaxwell> or instead of *, you name a technical challenge we've had in the bitcoin ecosystem and someone has suggested a DHT to solve it.
05:46 < gmaxwell> Signature validation slow?  Use a DHT.   etc.
05:47 < mappum> well i'm glad, i hadn't thought about the vulnerabilities. i'll have to think about making mine sybil-proof and manipulated-hash-proof
05:48 < gmaxwell> in your case, I don't see how a dht ID helps you. the pool would just store all the data for all the dht IDs. and could just produce work for any of them (assuming there was even an incentive in the system to not just have one ID)
05:48 < mappum> right, i realized that's not the solution, i'm just too tired to do the thinking right now
05:48 < gmaxwell> The nearest thing I've seen to a strong DHT system is cjdns.
05:48 < gmaxwell> and it uses the 'dht' only for routing.
05:49 < gmaxwell> maybe freenet, though freenet is ... uh.. really lossy.
05:49 < petertodd> gmaxwell: yeah, though being lossy is part of how they handle spam
05:49 < gmaxwell> and freenet opennet is not secure, while freenet darknet and cjdns are rather similar in many respects.
05:50 < gmaxwell> right, freenet works, but mostly because it doesn't promise very much. :P
05:50 < petertodd> gmaxwell: though freenet opennet is not secure in the same sense that tor isn't all that secure either...
05:50 < petertodd> gmaxwell: underpromise and... deliver
05:51 < gmaxwell> petertodd: I mean, opennet has some trivial sybil vulnerabilties. Tor doesn't but only because of the centeralized directory authorities.
05:51 < gmaxwell> darknet freenet loses the sybil risk for the same reason cjdns does. the users are expected to not select sybil peers.
05:52 < petertodd> gmaxwell: right, although I'm not sure the directory authorities actually help that much - they can't know if someone is logging
05:52 < petertodd> gmaxwell: they are only able to keep the system safe from sybils attempting to make Tor not function
05:57 < petertodd> gmaxwell: oh, speaking of, i2p has hashcash on their todo list: http://www.i2p2.de/todo
05:58 < gmaxwell> hashcash, in java, tm.
05:59 < petertodd> gmaxwell: heh, Bitcoin sacrifice is the only sane way to do it
06:10 < adam3us> btw sdl (sergio damien lerner) claims to have an efficient unpublished anonymity solution https://bitcointalk.org/index.php?topic=305791.msg3733685#msg3733685 which he has not published for year for "ethical reasons"??
06:11 < petertodd> SDL is weird...
06:13 < adam3us> petertodd: my response "I'd sure publish it immediately if I had figured it out and feel I did a good thing for society." and "Personally I think gambling has far more ethical worries than users being able to transact privately with something approaching the analogous already existing levels of privacy in other systems.  For some people gambling becomes a near ruining addiction."
06:13 < petertodd> adam3us: you're doing well with these responses lately you know
06:14 < gmaxwell> adam3us: s/some/many/
06:14 < gmaxwell> it's shocking.
06:14 < adam3us> petertodd: (his phd thesis is about fair poker) and i think he looked at anonymity because he wanted to reduce scope for gaming collusion where you can cheat
06:14 < gmaxwell> in any case, IIRC the appecoin thing was he basically proposes you make the entire txout set a reencryption mix and every miner reencrypts it every block or something.
06:15 < petertodd> adam3us: was eye opening a few months ago when I mentioned that satoshidice wanted to hire me for some analysis to my boss, and he thought doing so was totally unethical based on it being gambling - he wouldn't have raised an eyebrow if I'd told him DPR wanted to hire me
06:15 < gmaxwell> (which is a protocol you'd expect from a guy who did research on mental poker)
06:15 < adam3us> gmaxwell: hmm that doesnt sound so good for end2end privacy, its trust me privacy with the current random block winner?
06:16 < gmaxwell> (e.g. the same way that you shuffle in some poker schemes)
06:16 < adam3us> gmaxwell: y'know maybe i vaguely read that in something he wrote now you mention it
06:16 < gmaxwell> adam3us: well sort of, they shuffle the _whole_ utxo set, so even though each block winner knows his mix, the set of all block winners is presumably strong.
06:16 < gmaxwell> unless mining has become 100% centeralized.
06:17 < gmaxwell> (or unless people are bribing miners for permutation lists)
06:17 < petertodd> gmaxwell: needs to be some way to make releasing the permutation list risky, like if you could somehow use that info to steal the block reward
06:17 < gmaxwell> but of course, reshuffling the whole utxo every block (or even every N blocks) is completely unrealistic.
06:18 < gmaxwell> and the cut and choose proofs required to show that the shuffle was fair wouldn't be small. (well perhaps, I did post some optimizations which might help, at the cost of making them expensive to verify)
09:04 < adam3us> gmaxwell: yes i think having shuffling miners do a provable encrypted shuffle of utxo (or a subset of it) is interesting, i meant  its not as secure as blinding like zercoin which can be unconditionally secure anonymity (and doesnt rely on trust of a random, though growing over time, collection of miners)
17:07 < phantomcircuit> gmaxwell, *cough* https://github.com/mycelium-com/wallet/issues/9
17:10 < gmaxwell> phantomcircuit: thanks, bleh. https://github.com/mycelium-com/wallet/issues/9#issuecomment-29424301
17:10 < gmaxwell> I don't understand why this particular BIP got a firestorm of attention recently.
17:11 < gmaxwell> phantomcircuit: on that subject, your commentary on https://bitcointalk.org/index.php?topic=258678.0 would be helpful.
17:11 < gmaxwell> jrmithdobbs: as would yours.
17:14 < phantomcircuit> gmaxwell, im not sure my understanding of ECDSA is strong enough to usefully comment on it but i'll give it a read anyways
17:14 < gmaxwell> phantomcircuit: it's all symetric crypto.
19:42 < midnightmagic> btw, the public domain assertion in that hd wallets-with-optional-encryption is a potential law-bomb.
20:06 < sipa> ?
20:15 < midnightmagic> sipa: there are many places where assigning something to the public domain isn't possible, and doesn't serve as a disclaimer of rights. apparently. it has to be something more, like "this work can be used for any purpose, by anybody, forever. also at your own risk blah blah. also we grant you royalty-free use of any of our applicable patents blah blah we promise not to patent-troll you later blah blah."
20:15 < midnightmagic> it's why OSI rejected the copyright commons 0-license
20:19 < gwillen> midnightmagic: er, the whole point of the commons-0 license is to have that wording in it
20:19 < gwillen> where you put it in the public domain if you can, and if not you grant all rights to everybody forever etc. etc.
20:20 < midnightmagic> gwillen: The lack of patent language killed it. http://opensource.org/faq#public-domain
20:20 < gwillen> midnightmagic: I'm reading the faq right now, it appears that the opposite is true
20:20 < gwillen> midnightmagic: what killed is is that there _was_ patent language
20:20 < gwillen> that specifically said patent rights are _retained
20:21 < gwillen> and apparently OSI thought that was worse than licenses that don't mention patents at all
20:22 < midnightmagic> Right.
20:22 < midnightmagic> "We retain the right to sue you into oblivion whenever we want."
20:22 < gwillen> *shrugs*
20:22 < gwillen> it seems like a minor thing to me
20:23 < gwillen> since it's very likely that patent rights are in fact retained when using an actual public domain dedication, where possible
20:23 < gwillen> or a simple open source license
20:23 < gwillen> e.g. when using the MIT license which I think has no mention of patents at all
20:28 < phantomcircuit> http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/
20:29 < phantomcircuit> warren, gmaxwell ^
20:30 < warren> yeah
20:32 < phantomcircuit> if that's really leveldbs mmap strategy
20:32 < phantomcircuit> that is retarded
20:34 < cfields> phantomcircuit: agreed. It seems very inefficient and dangerous to me.
20:34 < phantomcircuit> tbh most everything about the implementation of leveldb seems insane to me
20:34 < phantomcircuit> such as journal entries not having sequence numbers
20:34 < gwillen> it does seem odd to me that munmap doesn't flush
20:34 < gwillen> that's really weird behavior
20:35 < phantomcircuit> gwillen, no no, he's saying that not only does it not flush
20:35 < phantomcircuit> but it DROPS the dirty pages
20:35 < gwillen> right
20:35 < gwillen> yeah, sorry, I mean, not flushing but keeping things pending would be okay
20:35 < gwillen> but dropping them is really odd
20:35 < phantomcircuit> yeah that sounds strongly like a bug in os x
20:35 < phantomcircuit> which is what i've assumed all along
20:36 < gwillen> well, evidently posix doesn't require it to do anything useful
20:36 < phantomcircuit> apple gaming the fuck out of benchmarks basically
20:36 < cfields> yea, the spec doesn't say it needs to flush. in fact, one of the docs explicitly says you need to msync() for that
20:36 < cfields> s/spec/bsd docs/
20:36 < phantomcircuit> right
20:36 < phantomcircuit> if im reading this right, he's saying that not only do they not flush, they also drop the dirty pages entirely
20:37 < phantomcircuit> like they have a separate page cache for mmap than from normal file io or something bizarre
20:40 < phantomcircuit> my guess is that there is some small pool of memory in the mmap subsystem which is used to buffer changes to the page cache that is dropped when munmap is called
20:41 < cfields> imo that's not the case...
20:42 < cfields> my theory was that there's a quick write, then a quick read. The read comes from an fd on osx rather than mmapping. So the last write may not be on-disk yet, since it's still showing the zeroed region
20:43 < phantomcircuit> hmm maybe
20:47 < warren> "His victory speech"
20:47 < warren> heh
20:50 < midnightmagic> cfields: You're talking about this from msync(2)?: Filesystem operations on a file that is mapped for shared modifications are unpredictable except after an msync(). ?
20:50 < cfields> yea
20:50 < phantomcircuit> actually thinking about it iirc you can only get like 50k write() syscalls/second
20:51 < phantomcircuit> so i can see how that would be a limiting factor
20:51 < phantomcircuit> except leveldb is single threaded so an inmemory buffer with a timer should work well enough
20:51 < phantomcircuit> er i mean no it's not
20:52 < phantomcircuit> but a thread with the end of the journal that gets flushed when it's a certain size or after it's 500ms old or something
20:52 < midnightmagic> phantomcircuit: the better alternative to to use scatter/gather and iovec structs
20:52 < cfields> hmm, wait a sec
20:53 < midnightmagic> then individual syscall overhead is reduced (ideally) and the structs can sometimes be utilized by underlying storage subsystems to speed up their own writes.
20:53 < phantomcircuit> midnightmagic, sure but that's not going to be obviously platform independent :)
20:53 < gmaxwell> gwillen: cc-*-4.0 has extended the patent badness into all creative commons licenses now, fwiw.
20:53 < midnightmagic> dirty internal page caches when flushed to disk in multiples shouldn't be done with plain write()s
20:53 < phantomcircuit> (which is the fundamental issue here)
20:54 < gmaxwell> gwillen: thank universities for that one, mostly.
20:54 < midnightmagic> phantomcircuit: huh?  why isn't it portable?
20:54 < midnightmagic> we only care about bsd/linux/windows/osx right?
20:54 < gwillen> gmaxwell: all the 4.0 ones appear to say is 'Patent and trademark rights are not licensed under this Public License.'
20:54 < cfields> https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/env_posix.cc#L357
20:54 < gwillen> gmaxwell: which is a weaker statement even than cc0
20:55 < gwillen> gmaxwell: and which doesn't seem like much of a statement at all
20:55 < cfields> a call to Sync() flushes the fd to disk, before msync has been called
20:55 < phantomcircuit> cfields, so basically that's backwards
20:55 < cfields> and on osx, we've forced that flush to be a really hard flush, too :)
20:56 < gmaxwell> gwillen: yes, thats the narrowest crafting that they could get through. It's still fatally bad.
20:56 < gwillen> gmaxwell: I'm really not sure I agree.
20:56 < phantomcircuit> midnightmagic, oh i see you mean the libc ones
20:56 < phantomcircuit> hmm
20:56 < gwillen> gmaxwell: There's no reason to believe that not having it would cause patent or trademark rights to be licensed.
20:56 < cfields> phantomcircuit: hmm, it sure looks that way to me
20:56 < phantomcircuit> yeah i guess that should be very portable
20:56 < gwillen> gmaxwell: It should be more or less a no-op.
20:57 < gmaxwell> gwillen: the legal minds (e.g. Eben Moglen) believe that the BSD license contains an _implied_ patent license because it permits you to use/copy the work which would otherwise require a patent license.
20:57 < gmaxwell> gwillen: there is no ambiguity, if you recieve a work under cc-by-sa-4.0 there is no implied patent license.
20:57 < gwillen> gmaxwell: Nobody's going to rely on an implied patent license
20:57 < gwillen> not on purpose, anyway
20:57 < gmaxwell> gwillen: millions of people depend on an implied patent license.
20:57 < phantomcircuit> tbh leveldb is kind of a mess
20:57 < midnightmagic> phantomcircuit: The structs can be passed to the lower-level, it takes a single syscall (usually) and there's a huge win. I've been advocating for scatter writes using iovecs internally here for like a decade but nobody listens to me and I'm too stubborn to write it for them.
20:57 < phantomcircuit> it seems like it would be easier to write a BitcoinKVDB
20:58 < gwillen> gmaxwell: also, CC is rarely used for code
20:58 < gmaxwell> gwillen: quallcom and and apple are basically filing a new patent against llvm/claim per week each. Your only ability to use clang at all is an implied patent license.
20:58 < gwillen> gmaxwell: patent licenses are not likely to be important on documentation or literature
20:58 < phantomcircuit> inb4NIHs
20:58 < midnightmagic> what the heck is NIHs ?
20:58 < gmaxwell> gwillen: indeed, which is a saving grace, though a narrow one. They are not too infrequently used for scientific publications.
20:59 < gmaxwell> gwillen: and yea, the implied patent license sucks. I look forward to seeing you try to convince all the patent carrying packages you use that are bsd licensed to adopt the apache license.
20:59 < gmaxwell> s/llvm/claim/llvm/clang/
21:00 < gwillen> gmaxwell: I dunno, I kind of handwave the whole issue with "all software ever written violates multiple patents anyway"
21:00 < gwillen> "therefore whether you get sued is not about what you write, but about who you piss off"
21:01 < gwillen> I hope the courts wipe out the whole software patent sector and we get to stop worrying about it
21:01 < gmaxwell> gwillen: true, though the magnitude of it is worse when its patents are owned by the same people writing the software (little ambiguity that the patents apply), and they're known vexatious litigants.
21:02 < gwillen> me nods
21:02  * gwillen nods*
21:02 < cfields> midnightmagic: which set of man pages did you pull that msync(2) from?
21:02 < gmaxwell> gwillen: it's all squishy in any case, — which is why an implied license is helpful,  the ambiguity makes litigation less likely, etc.
21:03 < gmaxwell> gwillen: so it's unfortunate to lose the ability to argue that maybe there is one.
21:03  * gwillen nods
21:04 < phantomcircuit> midnightmagic, not invented here syndrom
21:04 < phantomcircuit> e
21:19 < midnightmagic> cfields: NetBSD
21:20 < midnightmagic> cfields: Anytime there's a question like that, just assume NetBSD.
21:20 < cfields> heh
21:35 < gmaxwell> cfields: wrt mmap and leveldb...
21:35 < gmaxwell> cfields: I was pretty sure that 32 bit builds of leveldb do not use mmap.
21:35 < cfields> gmaxwell: for reading
21:35 < cfields> for writing they do
21:36 < gmaxwell> ah, interesting! okay.
21:36 < cfields> that's why i was bugging gavin about whether he could repro on his 64bit builds or not
21:37 < warren> cfields: we have three reports of no avoiding corruption with the mem barrier thing with people who had corruption on every run prior, weird coincidence?
21:37 < warren> s/no//
21:38 < cfields> gmaxwell: please check me, though. Anyone who's known me for a while knows that I typically go through ~3 rounds of sure-thing fixes before finding the real one :)
21:38 < gmaxwell> cfields: no, you're right I see that too now.
21:39 < cfields> warren: i still think the mem-barrier patch should go in upstream. But that was an indirection to us at best.
21:39 < cfields> warren: if it really fixed something for someone, i'd be really curious to know specifics
21:40 < warren> it's hard enough to get these people to respond
21:41 < gmaxwell> the mem barrier change is clearly right but uh. so it would be good to know why its fixing things if it indeed is.
21:42 < gmaxwell> Perhaps we should buy one of the machines that reproduces this so easily? (I dunno why we didn't do this before the bounties)
21:43 < phantomcircuit> gmaxwell, that was my suggestion last week
21:44 < phantomcircuit> but really who is going to want to sell us their laptop
21:44 < warren> do we know why many users can't reproduce the problem at all?
21:44 < phantomcircuit> warren, the race is a very close one probably
21:44 < phantomcircuit> it might be easier to write a stress test actually
21:45 < cfields> gmaxwell: i have one
21:46 < cfields> gmaxwell: rather, i borrowed one from a friend. and i was nice enough to upgrade her to 10.9 :p
21:46 < cfields> though, i was still never able to reproduce it
21:47 < gmaxwell> yea, what I was saying is that there are people who claim it always or at least frequently fails for them. Theirs is the computer you want. :)
21:47 < cfields> not without adding some sleeps in code, anyway
21:47 < gmaxwell> well the sleeps is a good idea in any case.
21:47 < warren> it happens on every run of bitcoin-qt for coblee
21:48 < cfields> warren: that's got to be a different issue i'd think
21:48 < warren> for him the mem barrier patch made bitcoin-qt usable
21:49 < cfields> warren: the thing about that change, is that it affects lots of leveldb in tiny tiny ways
21:49 < gmaxwell> cfields: we should probably do some testing of level db where we fill the source code with if(atoi(getenv("diehere"))==linenumber)exit(1);  and then run in a loop syncing the testnet chain and picking random numbers to die on then restarting and making sure it continues.
21:51 < cfields> heh
21:52 < cfields> gmaxwell: are you after a test for this one in particular? or general leveldb badness?
21:52 < gmaxwell> well, I always suspect more badness once some is found. :)
22:24 < cfields> heh
22:25 < cfields> gmaxwell: leveldb has a pretty extensive test-suite. I'm really not sure we could catch anything that they miss
22:26 < cfields> in your example above, looping their corruption test as long as syncing testnet would probably give the same result
22:26 < gmaxwell> cfields: well, you found bugs by inserting sleeps and killing it .. :)
22:27 < cfields> heh
22:27 < cfields> gmaxwell: actually, that does raise an interesting point
22:28 < cfields> a bash script to continuously send STOP/CONT to bitcoind could be interesting
22:30 < cfields> i don't know enough about the underlyings of those to know how much (if at all) that could simulate outside-world interference. loss of net connections, closed files, etc
22:33 < gmaxwell> mostly I want to detect cases where a sudden power off would leave the system in an unrecoverable state.
22:41 < cfields> i'm unfamiliar with what is allowed to finish after a SIGKILL. How close does that come?
22:41 < gmaxwell> a real test would be to create a special log structured block device that allows you to mount the log at any write along its history can continue from there.
22:42 < gmaxwell> sigkill is probably close enough to be interesting but just randomly sending sigkills are not because it doesn't get good coverage.
22:42 < gmaxwell> (and thats a test I've already done)
22:42 < cfields> gmaxwell: in any case, if it's important enough to you, i have dozens of small arm dev boards here that i'd be happy to setup for automation
22:42 < gmaxwell> e.g. 99% of the time you kill it doing nothing.
22:43 < cfields> though iirc you mentioned you have them at your disposal as well recently
22:43 < gmaxwell> yea, I have a couple pandaboards. that I mostly use for continious integration testing for code stuff, (e.g. arm simd)
22:44 < cfields> oh, right, i forgot to wtf you on that
22:44 < cfields> you build natively on those things?!
22:46 < gmaxwell> cfields: sure, on codec stuff the compile time is insubstantial compared to the actual tests.
22:46 < gmaxwell> and if something breaks self hosting is much easier to work with then using a remote debugger.
22:46 < cfields> interesting
22:47 < gmaxwell> I wouldn't want to work with bitcoin on them in that way... just because bitcoin takes a long time to compile (though, as you noted I did indeed compile bitcoin on one of them the other day)
22:47 < cfields> i guess i've gotten so used to remote debugging that the idea of debugging on embedded would never enter my mind
22:48 < cfields> though i suppose that really doesn't make much sense, considering their speeds these days
22:48 < cfields> "back in my day..." and all that :)
22:52 < gmaxwell> yea indeed, well I was there too at one point.. but when I started dealing with dual core 1ghz embedded devices...
22:53 < cfields> one of my first embedded projects was porting xbmc (and its ~50 dependencies) to a 400mhz mips SOC
22:54 < cfields> unfortunately, i think that mentality has stuck with me
23:07 < warren> cfields: did you submit the memory barrier thing anywhere?
23:07 < warren> cfields: given that it isn't wrong and it seems to have done something, perhaps more eyes ...
23:10 < cfields> warren: heh, it must be torture inside your head :)
23:12 < cfields> doing now
23:27 < cfields> warren: https://code.google.com/p/leveldb/issues/detail?id=218
23:34 < warren> cfields: thank you
--- Log closed Thu Nov 28 00:00:00 2013