| 01:46:03 | c0rw1n: | c0rw1n is now known as c0rw|zZz |
| 01:56:22 | jae: | jae is now known as Guest65908 |
| 02:29:10 | bramc: | It's been quiet in this channel lately |
| 02:30:27 | gmaxwell: | bitcoin network blew up. |
| 02:35:05 | bramc: | I don't know if that's sarcasm |
| 02:37:37 | gmaxwell: | bramc: oh you're not aware, you'll love it. |
| 02:37:46 | gmaxwell: | bramc: 6 block invalid fork about 24 hours ago. |
| 02:38:10 | CodeShark: | fun times :) |
| 02:38:15 | CodeShark: | so is everything back to normal again? |
| 02:38:18 | gmaxwell: | bramc: turns out ~50% of the hashpower was willing to mine blocks without validating anything. |
| 02:38:36 | CodeShark: | and it will only continue to get worse as blocks get bigger |
| 02:38:37 | gmaxwell: | CodeShark: normalish, there are still some invalid blocks here and there, but so far no new long invalid forks. |
| 02:39:00 | CodeShark: | good |
| 02:39:06 | bramc: | Oh god. Did the fork eventually get invalidated? |
| 02:39:08 | Luke-Jr: | not sure there is anything to prevent long invalid forks though.. |
| 02:39:24 | Luke-Jr: | bramc: it was always invalid, but it eventually got abandoned too yes |
| 02:39:30 | bramc: | I mean, orphaned? And what was invalid about it? |
| 02:39:31 | gmaxwell: | bramc: So BIP66 took effect, so all blocks were required to be version 3. Part of the <5% of the non-upgraded miners found a block, still tagged as v2, thus invalid. And then ~50% of the hashpower continued that invalid fork, because they weren't validating anything at all. |
| 02:39:45 | Luke-Jr: | at least F2Pool announced they will continue to skip validation though |
| 02:40:20 | bramc: | Are they validating with homegrown regexes or something? |
| 02:40:29 | gmaxwell: | bramc: yes, it was overtaken but it took a long time (over an hour?) and manual intervention by one of the parties that was extending the invalid fork. It could have been much larger but we caught it happening instantly. |
| 02:40:29 | Luke-Jr: | bramc: they're not validating at all |
| 02:40:33 | CodeShark: | I don't even think that, bramc |
| 02:40:44 | CodeShark: | perhaps just validating difficulty |
| 02:40:49 | Luke-Jr: | bramc: they're sitting there connected to other pools' stratum ports, and when they see a new prevblock header, they mine empty blocks on it |
| 02:40:58 | gmaxwell: | CodeShark: Petertodd said he didn't even think they were validating difficulty. |
| 02:41:01 | Luke-Jr: | CodeShark: probably not even that |
| 02:41:13 | CodeShark: | gmaxwell: seriously? then there might be something we can do ;) |
| 02:41:22 | bramc: | Luke-Jr, empty blocks, meaning they aren't accepting any transactions? |
| 02:41:25 | gmaxwell: | CodeShark: sure yes, but they'll start doing that. |
| 02:41:25 | Luke-Jr: | CodeShark: they're only doing this with other pools |
| 02:41:32 | Luke-Jr: | bramc: they won't, but the v2 miners will |
| 02:42:08 | Luke-Jr: | so something like 90% of blocks on the forked chain would have txns |
| 02:42:51 | gmaxwell: | Luke-Jr: would have no txns. |
| 02:42:53 | bramc: | What percent of miners had voted for bip66? |
| 02:43:04 | gmaxwell: | bramc: 95% over a 1001 block window. |
| 02:43:14 | gmaxwell: | (950 out of 1001 to be precise) |
| 02:43:16 | Luke-Jr: | gmaxwell: err, right |
| 02:43:31 | bramc: | I... I can't... I don't know what to say. |
| 02:43:37 | gmaxwell: | bramc: the non-verifying miners were supporting BIP66 as well, but not enforcing anything at all, including BIP66. |
| 02:44:07 | CodeShark: | so we have two serious related problems (at least): 1) the network is now tacitly trustful, 2) miners are liars when voting for rule chances |
| 02:44:09 | CodeShark: | *changes |
| 02:44:10 | gmaxwell: | bramc: I mean, this is a predicted outcome from higher system load. I'm surprised at how much of the hashrate was doing it, petertodd says he isn't however. |
| 02:44:22 | bramc: | gmaxwell, Define 'supporting' bip66? As in, not supporting bip66 non-compliant transactions? Because not accepting any transactions is a good way to do that. |
| 02:45:15 | gmaxwell: | bramc: they were signaling the version which is formally defined to do several things, including (once 950/1001 is met) you must reject blocks without the old version. |
| 02:45:16 | bramc: | CodeShark, A large miner should be able to lazily validate blocks and only have a short window where they're mining an untrusted block |
| 02:45:24 | bramc: | er, an invalid block I mean |
| 02:45:36 | CodeShark: | bramc: that requires actual engineering, as opposed to just hacking away at code |
| 02:45:41 | gmaxwell: | bramc: they previously did that, but they turned off that functionality when it caused them to fail to mine selfishly. |
| 02:46:27 | bramc: | gmaxwell, What do you mean fail to mine selfishly? They could leave on invalidating of bad blocks but still prefer their own if there was a tie |
| 02:47:01 | CodeShark: | oh right - wangchun said they had tried that but were mining atop blocks that were not theirs |
| 02:47:01 | gmaxwell: | bramc: F2pool reports they used to switch back to bitcoind; but what happened (last year?) was that ghash produced an orphan block, and they SPV-mined on top of it and produced a child, but were still tied with the main chain, then they switched to the template issued by bitcoind which was on the other fork that had rejected the ghash block; and then they managed to mine a block orphaning themsel |
| 02:47:08 | gmaxwell: | ves. |
| 02:48:09 | CodeShark: | of course, that could be fixed as well |
| 02:48:19 | gmaxwell: | They opened a bug request to bitcoin core asking us to enable forcefully mining on an older chain if its your own, due to this. And I yelled at them for the SPV mining. (As an aside, the policy to selfishly mine on your own block even ifs later is really horiffic for convergence, consider if you have >50% hashpower then this policy means you'll automatically exclude everyone elses blocks.) |
| 02:48:33 | bramc: | Good lord, I just assumed that all mining pools were doing this basic shit right |
| 02:48:41 | Luke-Jr: | lol |
| 02:48:46 | CodeShark: | hah |
| 02:49:15 | gmaxwell: | (yelled at them for SPV mining, and then lit a fire under matt to produce the relay network client so they'd have little/no reason to do dumb crap like that) |
| 02:49:22 | bramc: | gmaxwell, What do you mean by spv mining? Was a major pool not running their own full node? |
| 02:49:32 | gmaxwell: | I has ASSUMED that after the relay network client they would have stopped that junk. :( |
| 02:50:06 | gmaxwell: | bramc: SPV mining is what we're calling this practice of getting a block header from some (random?) third party and mining on top of it without regard to its validity. |
| 02:50:07 | Luke-Jr: | it's really headers-mining, not SPV |
| 02:51:14 | CodeShark: | we're so screwed |
| 02:51:50 | gmaxwell: | well it could easily be made into actual SPV with more software engineering but no other cost. |
| 02:53:24 | bramc: | Isn't that sort of like the president deciding to press the big red button based on what the morning newspaper dropped off by the local delivery kid said? |
| 02:53:34 | gmaxwell: | CodeShark: hah. I'm not that alarmed. |
| 02:54:17 | gmaxwell: | CodeShark: after all a couple days ago you were saying that SPV is broken. |
| 02:54:33 | CodeShark: | I haven't changed my mind on that ;) |
| 02:54:38 | gmaxwell: | CodeShark: the only major effect (say if almost all hashpower was doing this) is that SPV would be completely insecure. |
| 02:54:56 | midnightmagic: | it also means that bitcoin hashrate isn't effectively what its current hashrate is. |
| 02:55:05 | gmaxwell: | (well actually I guess it only takes a majority of hashpower doing this to make SPV completely insecure, technically) |
| 02:55:45 | gmaxwell: | midnightmagic: it wasn't the moment merged mining was invented in any case. :) |
| 02:55:54 | bramc: | Reducing the block time is sounding like a really, really bad idea. |
| 02:56:16 | gmaxwell: | bramc: well duh. |
| 02:56:29 | Luke-Jr: | was anyone taking that seriously? <.< |
| 02:56:35 | gmaxwell: | Luke-Jr: people on reddit. |
| 02:56:39 | CodeShark: | reducing block time, increasing block size, and eliminating latency-producing checks |
| 02:56:39 | bramc: | Luke-Jr, For altcoins they are |
| 02:56:44 | CodeShark: | sounds like a great formula for success :) |
| 02:56:47 | midnightmagic: | gmaxwell: huh? |
| 02:56:48 | bramc: | Ethereum's is, what, 20 seconds? |
| 02:57:13 | Luke-Jr: | bramc: altcoins don't count, they don't know what they're doing regardless :p |
| 02:57:13 | CodeShark: | 10 seconds, I think - but they use a GHOST variant |
| 02:57:37 | CodeShark: | GHOST at least alleviates much of the stale block crap |
| 02:58:21 | gmaxwell: | CodeShark: the ghost, as presented in the first paper at least, stuff creates other severe issues (e.g. creating enormous selfish miner advantages)... also ways that small hashrate miners can perpetuate large hashrate splits. |
| 02:59:04 | gmaxwell: | CodeShark: it also trades off goodput for latency in a given amount of available bandwidth. |
| 02:59:07 | bramc: | In lighter news, my talk at bitcoin-dev went fairly well. One of the spacecoin guys was there. He explained that they have ugly proofs of work because the much simpler ones have some (highly nontrivial) time/space tradeoffs |
| 02:59:11 | CodeShark: | yeah, I'm sure there are a number of errors in the original paper - but I think the scheme could be made workable |
| 02:59:24 | midnightmagic: | oh, backwards the other way. :-P |
| 02:59:30 | CodeShark: | or not errors, but oversights |
| 02:59:48 | bramc: | So I really, really need to read through that paper. |
| 02:59:52 | gmaxwell: | CodeShark: at least that general approach can improve the collapse rate; but it's not clear how much it matters. |
| 03:00:23 | bramc: | Given the utter impossibility of getting transactions to clear in under a minute, it all seems like a fool's errand. |
| 03:01:23 | gmaxwell: | yes, the diameter of the earth is an issue; obviously we need to move to mars. |
| 03:01:27 | gmaxwell: | faster transactions! |
| 03:03:22 | bramc: | gmaxwell, Mars is ridiculous. Putting all the continents back into Gondwonaland would be completely sufficient. |
| 03:04:55 | CodeShark: | you mean pangaea |
| 03:05:32 | bramc: | Hmm, you're right, Pangea |
| 03:06:04 | gmaxwell: | bramc: additional LOL is that the reddit big-blocks-now cabal is calling this evidence that there is no need for a block size limit: https://www.reddit.com/r/Bitcoin/comments/3c579i/yesterdays_fork_suggests_we_dont_need_a_blocksize/ (one might wonder why when I previously pointed out that miners would do this sort of thing in response to bigger blocks they argued that miners would not; rather t |
| 03:06:10 | gmaxwell: | han arguing that it would somehow be a blocksize control ...) |
| 03:07:22 | bramc: | gmaxwell, Yeah, umm, the evidence that miners are willing to just not mine transactions to increase their returns is kind of strong, given that they're already doing it. |
| 03:08:14 | gmaxwell: | bramc: the not mining transactions is incidental; they could go ahead and also mine transactions without verifying; it would just take a lot more software development. |
| 03:09:06 | CodeShark: | we need to either force all endusers to verify...or force all miners to verify...or both |
| 03:09:14 | CodeShark: | :) |
| 03:09:38 | bramc: | Right, the risk of rejecting a valid block is greater than the risk of accepting an invalid one, so miners will probably respond to huge blocks by turning off validation |
| 03:09:51 | bramc: | So then even the large miners won't be running full nodes. Full nodes will be run by...? |
| 03:10:03 | gmaxwell: | CodeShark: petertodd has been advocating the first on and off to varrious degress for a long time. But its ugly, I mean, really, even though SPV is weak it ought to be fine for vending machines! :) |
| 03:10:41 | gmaxwell: | bramc: right, thats a problem there.. if full nodes involve costs that miners-- who have huge incomes depending on the network-- don't like. |
| 03:10:46 | gmaxwell: | bramc: ohhh! also you'll like this. |
| 03:10:51 | bramc: | As a long term strategy having wallets validated is a perfectly reasonable idea. It'll take a while though. |
| 03:11:10 | gmaxwell: | bramc: almost all the block explorers displayed the invalid fork. |
| 03:11:41 | bramc: | gmaxwell, Eugh |
| 03:11:41 | gmaxwell: | Unfortunately I was too busy dealing with it to go around and measure services, but I'd expect a significant fraction of businesses were on the invalid fork. |
| 03:11:43 | midnightmagic: | bramc: full nodes will be run by "people who will have full control of the network" |
| 03:12:24 | amiller: | this story is really interesting, i wonder if there would have been any easier way to check whether a service or a miner is doing correct validation |
| 03:13:11 | gmaxwell: | amiller: well we could have discovered they were wrong faster if someone had been polling their public pool templates. |
| 03:13:19 | gmaxwell: | We were able to detect in realtime that they were on the wrong chain that way. |
| 03:13:31 | gmaxwell: | But it only would have reduced the response time by a few minutes in this cas.e |
| 03:13:57 | bramc: | gmaxwell, Also useful for telling if they're being smart about trying to de-orphan themselves in the event of a tie |
| 03:14:44 | amiller: | someone had to make an expensive error no matter what to even get that sample i think.... it's not like anyone would want to emit an invalid block every month so we can measure all the block templates |
| 03:14:47 | gmaxwell: | amiller: half jokingly before-- in the context of talking about a world where most nodes were dependant on fraud proofs, the concern was raised that the fraud proof code would never run so it would naturally bitrot. I suggested that blocks be required to commit to two candidate blocks, one of which was required to be invalid.. so fraudproofs would always be required to kill the wrong block. |
| 03:15:34 | bramc: | amiller, That's true in the current environment. If the bulk of mining power is running off of spv someone could do some fucked up shit though. |
| 03:16:11 | amiller: | bramc, yeah im just wondering about how this could be monitored earlier |
| 03:16:26 | amiller: | im really surprised blockchain.info showed the wrong chain |
| 03:16:41 | bramc: | amiller, Mostly by checking what mining pools are mining |
| 03:16:56 | gmaxwell: | at FC15 Aviv Zohar corrected someone (one of his students?) on the point that non-verifying miners amplify attacks; apparently it had been a discussion in their group before; he'll be amused to hear about this. |
| 03:16:57 | midnightmagic: | so in other words, someone querying these services and verifying tip with what their own full-validating node says, and reporting differences, ideally with some knowledge of work-valid forks/extinct branches/orphans, and signed with a key so the information can be snapshotted by someone else and the snapshots verified. |
| 03:17:09 | bramc: | Yeah what is blockchain.info running? I'd hope that they of all people would be running bitcoind |
| 03:17:27 | gmaxwell: | amiller: I would have bet $1000 that they'd be wrong. Interesting expectation gap. |
| 03:17:35 | amiller: | so if someone makes another invalid block, we'll get to see which services (like blockchain.info) have turned on full-validation rather than spv validation since then |
| 03:18:13 | bramc: | amiller, It will probably keep happening intermittently for a while, there were 5% who didn't sign on |
| 03:18:18 | gmaxwell: | amiller: when the network forked, I didn't have a pre 0.10 node handy, but I loaded bc.i to get data. (if you look in the bitcoin-dev log you can see me checking that _someone_ was using something other than bc.i as the prototypical crud accepting node) |
| 03:18:29 | midnightmagic: | b.i may be using an older bitcoind with custom patches, and is just too lazy to update because of how extensive the work was and how much churn there is in bitcoin-core |
| 03:18:34 | bramc: | Even at 1% legacy that's still one per day |
| 03:18:42 | gmaxwell: | amiller: see also this expirence: http://people.xiph.org/~greg/21mbtc.png |
| 03:19:26 | CodeShark: | I believe bc.i originally started by hacking an SQL backend into bitcoind |
| 03:19:37 | gmaxwell: | it's important to not single out bc.i, many other things were wrong too. |
| 03:19:50 | CodeShark: | merges to the bc.i codebase are probably far from trivial now |
| 03:20:17 | midnightmagic: | considering how much wallet money is stored in bc.i, I see no problem with singling them out. other services may have gotten it wrong, but it is *more important* that bc.i get it *right* |
| 03:20:22 | bramc: | * bramc longs to be having a conversation about the proofs of work in the spacecoin paper |
| 03:21:10 | CodeShark: | more the reason the consensus code needs to be isolated :) |
| 03:21:32 | CodeShark: | but that's a fairly recent development - I doubt bc.i is up to date on that |
| 03:22:10 | mr_burdell: | as a block explorer, I think it makes sense for bc.i to display all blocks, including invalid ones (although they should be able to mark them as invalid immediately) |
| 03:22:21 | bramc: | It's so much more fun imagining the glorious future than worrying about fending off the demons of stupidity in the here and now |
| 03:22:32 | mr_burdell: | the whole idea of a block explorer should be to show as much info as possible |
| 03:22:55 | CodeShark: | the future will also have demons of stupidity - but at least we don't have to deal with them yet :p |
| 03:23:26 | gmaxwell: | mr_burdell: 'mark as invalid' is not good, because they'll be mishandled by the 99% of users who don't understand or expect invalidity; putting them in a seperate list of invalid blocks would be fine. |
| 03:23:38 | gmaxwell: | mr_burdell: of course it wasn't just the explorer doing this, it was the wallets and apis too. |
| 03:23:50 | jcorgan: | also, block explorers, like everyone else, only have a local view of the network |
| 03:24:03 | jcorgan: | they may not see all transactions nor blocks |
| 03:24:25 | mr_burdell: | i had set up a bunch of nodes with different versions earlier this week to detect an event like this, but I accidentally used 0.9.5 as my earliest version, so my alarms didn't go off |
| 03:24:54 | gmaxwell: | I used to run every major prior release, but the disk space usage got to me, alas. |
| 03:25:06 | gmaxwell: | so I only run last release and master now. |
| 03:25:08 | midnightmagic: | jcorgan: again, except bc.i which has sensors mass-connecting to the world, so they *should see* more than nearly everyone else. |
| 03:25:47 | gmaxwell: | midnightmagic: they also have crazy spam filtering that often makes them see less. |
| 03:26:12 | midnightmagic: | that i didn't know. |
| 03:26:38 | mr_burdell: | they purge their mempool too so their users that send tx without fees don't get their funds stuck |
| 03:26:57 | mr_burdell: | it used to be after 4 days, but I think they do it faster now |
| 03:41:12 | jae: | jae is now known as Guest88497 |
| 04:17:55 | Luke-Jr: | gmaxwell: disk space .. so when can I sparse-pad the block files for btrfs dedupe? :P |
| 04:18:06 | Luke-Jr: | (yes, it turns out it does need that) |
| 04:47:26 | jae: | jae is now known as Guest44439 |