--- Log opened Thu Apr 11 00:00:07 2013
00:12 < warren> My surgery is a few days before the conference, so sadly I'm not going.
03:11 < warren> gmaxwell: I just had a scary thought.  Could p2pool's purported "bad luck" be attributable to a slightly higher chance of orphans because many of those nodes take longer to upload the new block than an ordinary high-bandwidth pool server?
03:14 < petertodd> Absolutely
03:14 < petertodd> >1MB blocks will without a doubt kill p2pool for that exact reason
03:15 < petertodd> Similarly p2pool has the inherent problem that it has no way to get participants to include transactions in the shares they solve.
03:15 < warren> well, that's a benefit, if you believe in decentralization
03:15 < warren> I'm more concerned about the orphan risk
03:16 < warren> When a mining client like cgminer finds a block, what exactly does it upload to the pool server?  (I never looked yet)
03:17 < petertodd> "Mining" clients, IE hashing clients, don't need to know what transactions are in a block and just upload headers.
03:17 < warren> ok, so that's fast and tiny
03:17 < warren> petertodd: even with GBT?
03:17 < petertodd> Yup, regardless of what size blocks are.
03:17 < petertodd> GBT is only from pool to client.
03:17 < petertodd> Oh, sorry, that's a bad way to describe it...
03:18 < warren> sure, but the client can change the tx set that it chooses to hash, so I assume it has to upload a lot more to the pool server
03:18 < warren> upon finding a block
03:18 < petertodd> Yes, the client with a true GBT setup needs to tell the pool what TX's they used, whcih is why GBT in that use scenario won't be allowed by pools.
03:19 < warren> Ah, so Luke-Jr's argument that "Eligius is decentralized mining" is not very accurate.
03:20 < petertodd> Yup, it's at best accurate with small blocks, and doesn't scale.
03:20 < warren> So it's fine for Litecoin, which has no tx's!
03:20 < warren> =)
03:21 < petertodd> GBT doesn't solve the competition problem either, that is, the expense of starting a new pool because the old one is dishonest. In addition, pool ops can divert hashing power to other pools, while witholding block solutions, as a way to attack those pools.
03:21 < warren> So ... p2pool can only truly compete with "normal" pools if the nodes are run with high bandwidth.
03:21 < petertodd> Exactly
03:22 < petertodd> Even worse, there is a free-rider problem where naturally people with low bandwidth can connect to p2pool and screw it up for everyone.
03:22 < warren> p2pool currently makes no attempt of optimizing peer selection, among other problems.
03:22 < petertodd> If it did though it'd run into the same problems bitcoin would by optimizing peer selection.
03:22 < warren> forrestv made an excellent proof of concept, but it never went beyond that.
03:23 < petertodd> Yeah, I'm not convicned the proof of concept can be turned into something truly robust though due to inherent issues with Bitcoin. Issues that will be made truly insolvable with large blocks.
03:24 < warren> Perhaps if the nodes were encouraged to be hosted on high-bandwidth, and peer selection scoring measured peer quality in various ways.
03:25 < warren> This will matter if we want multi-ASIC owners to decentralize (not on the big pools).
03:25 < petertodd> Impossible not to game those things though - you can't prove to someone else that a third party posessed bandwidth.
03:26 < warren> You can score things like "who sent me the new share first" and "who responds with incredible lag"
03:26 < petertodd> Yes, but only locally. As I say, you can't prove to someone else that, other than by saying "I have a lot of hashing power and say so", which means your attacker just starts off with more hashing power, and votes for themselves.
03:27 < warren> It isn't really a vote, and the scoring is used primarily to figure out which of your peers is worst and to kick them out eventually.
03:28 < warren> By these measures a high-bandwidth node without hashers could be scored high.
03:29 < petertodd> Ah, yeah, locally that can work, on the other hand, it means you can attack P2Pool by running some high-bandwidth P2Pool nodes and doing stupid crap like splitting the network.
03:30 < petertodd> Hmm.... actually I may be wrong about that, if P2Pool merges splits together by including the work on both sides, which it should.
03:30 < warren> you'd need a large number, and you would need the actual hashers to never be connected directly to each other.  Big hashers explicitly connect to each other by IP.
03:30 < petertodd> Getting large numbers of IP's is really easy for attackers.
03:30 < warren> still, hashers will link directly to each other
03:31 < warren> forrestv is considering parallel chain merging as means to get rid of the current annoying 10 second work intervals
03:31 < warren> it's all talk now though, there are some design issues
03:31 < petertodd> Oh good, so he's not doing that right now, but recognizes it.
03:32 < petertodd> P2Pool also needs multi-levels eventually, to keep varience down. IE a p2pool that mines p2pool shares collaboratively.
03:32 < warren> forrestv's priorities are clearly elsewhere.  He's wholly unprepared for ASIC's and seems uninterested in working on someone else's Avalon.  Folks are waiting for him to get his own Avalon.
03:33 < warren> People thought of that. How do you prevent dust from getting too small?
03:33 < petertodd> he's a young kid in a hard university program, so I can't blame him.
03:33 < petertodd> The sub-p2pool shares don't have to communicate with each other, though yes, I'm sure there will be plenty of tricky design issues.
03:34 < petertodd> sub-p2pool chains I mean
03:34 < warren> right.  the hard part there is just the dust gets too small.
03:34 < warren> p2pool LTC payouts are *already* too small for LTC's super high fees.  Lots of complaints from CPU miners.  (haha)
03:35 < petertodd> On, you mean the dust payments, that's what off-chain transactions are for.
03:35 < warren> off-chain transactions would be entirely outside of p2pool's design goals.  If the code is implemented right (it currently isn't), you don't really have to trust the other nodes.
03:36 < petertodd> IE at some point there is someone paying sub-p2pool miners with off-chain tx's, possibly fidelity-bonded banking where you can take humans out of the equation in terms of trust.
03:36 < petertodd> Implemented properly fidelity-bonded banking relies on incentives, not trust. You are trusting the person actually holding the funds to be economically rational, because any fraud has huge costs to you.
03:36 < petertodd> But that's a long way off.
03:36 < warren> what is "fidelity-bond" in a nutshell?
03:37 < warren> the sub-p2pool would be centralized and easy to take down with a DoS attack?
03:37 < petertodd> Long story short, it's a way of proving you threw away value.
03:39 < petertodd> Doesn't have to be. You can have people's bigger nodes make promises to submit winning shares to the main p2pool sharechain, and make those promises dynamically. The actual messages can be passed around by p2pool itself, so you don't need to have any idea who is making the payout or what their ip addr is.
03:40 < warren> p2pool sends the new block to the other nodes *and* to bitcoind.  That's promoted as a benefit as other nodes can propagate the block faster.  gmaxwell explained this to me before but I don't remember the detail of exactly how much it needs to upload between nodes ... it could be far less than a full block because the block contents were already sent earlier.
03:40 < warren> So I suppose this could counter-balance the bitcoind upload being slower.
03:41 < petertodd> The same methods can be used with bitcoin itself, so p2pool stays at a disadvantage.
03:41 < warren> can?  but would it?
03:42 < petertodd> Hard to say, worse case is the blocksize gets lifted without any of the optimizations getting implemented. Although that particular one, sending tx hashes rather than full tx's, is a really dangerous one because an attacker can use it to fork the network.
03:42 < warren> I don't know how that can be true though, given that each p2pool has its own bitcoind choose its own tx's to include.
03:43 < petertodd> p2pool would be making the assumption that tx's have propagated to the whole network
03:44 < warren> So if your patched your bitcoind to exclude SD spam, the sharechain block header upload won't succeed to reconstruct the block elsewhere?
03:44 < warren> OH!
03:44 < warren> the p2pool log shows BLOCK FOUND and sometimes INCOMPLETE BLOCK FOUND
03:44 < warren> That must be it.
03:44 < petertodd> Interesting, that may be exactly the case. I don't actually know the full details.
03:45  * petertodd has a BFL and can't use p2pool.
03:45 < warren> BFL FPGA with the 5 second work return latency?
03:45 < petertodd> yup
03:46 < warren> what kind of hash rate and power usage does that have?  just curious.
03:46 < petertodd> I forget exactly, but I remember it was exactly as advertised.
03:46 < petertodd> ~830 and 60W or something
03:47 < warren> nice. especially due to the ASIC delays that must have been good for you.
03:47 < petertodd> Slightly insane that my one unit was bringing in a theoretical $500/month at the very peak of the BTC price...
03:49 < warren> sigh, *this* might be the reason for the apparent "bad luck" of p2pool.  Just a few orphans here or there can make it look bad over short time scales especially since the pool finds blocks infrequently.
03:49 < petertodd> what do you mean '*this*'?
03:50 < warren> Lots of home users with asymetric bandwidth, uploading slowly.
03:50 < petertodd> ah, yes absolutely
03:50 < petertodd> people really don't realize how much bandwidth you need to upload blocks fast enough to keep orphans down
03:50 < warren> People have been commenting on the "bad luck" of p2pool forever and nobody mentioned this as a possibility.
03:51 < petertodd> Really? Shit, I thought I mentioned it publicly a few times in the small blocks stuff... I probably forget to mentiuon P2Pool specifically.
03:51 < warren> The block forwarding thing could be made better if the p2pool nodes also connected their bitcoind's together, so "INCOMPLETE BLOCK" won't happen.
03:51 < warren> They can't choose tx's anymore, but at least they're all reference.
03:52 < warren> or rather, INCOMPLETE BLOCK would happen less often
03:52 < petertodd> Yeah, it'd be a good idea. You'd need to come up with a central tx chosing algorithm, and at that point, you can actually semi-ditch bitcoind...
03:52 < warren> ah!  p2pool already has RPC access to bitcoind.  It could just addnode right?
03:52 < petertodd> Yup
03:53 < warren> do you have any way to read the bitcoind's IP:PORT from RPC?
03:53 < warren> it could addnode but it has no way of knowing *what* to connect to
03:53 < petertodd> No, but you don't need to. Just read ~/.bitcoin/bitcoin.conf
03:53 < warren> No, I mean the foreign node:port, which is what your addnode would need.
03:53 < petertodd> Ah, just have them tell you.
03:53 < warren> ahhh
03:54 < warren> Some people were talking about integrating p2pool-like functionality directly into bitcoind.  Perhaps this would be easier that way.
03:55 < warren> (not a great idea for other reasons, like you would have lots of extra code in the reference client)
03:56 < petertodd> Yeah, we're moving towards removing stuff from the bitcoinc lient not adding.
03:58 < warren> INCOMPLETE BLOCK FOUND seems to happen ~50% of the time here
03:58 < warren> so p2pool could be propagating the block faster in some cases, and slower in others
03:58 < petertodd> Interesting, have you read the p2pool code to figure out what's goign on?
03:58 < warren> no, never thought what INCOMPLETE meant
03:58 < petertodd> read it
03:59 < warren> and I heard the block forwarding thing was added because block submission on some nodes was failing entirely
03:59 < petertodd> lovely...
03:59 < warren> on LTC p2pool most of the nodes are actually failing block submission right now
03:59 < warren> and nobody noticed for months due to the block forwarding
04:00 < petertodd> nice
04:01 < warren>  block = share.as_block(self.tracker, self.known_txs_var.value)
04:01 < warren>             if block is None:
04:01 < warren>                 print >>sys.stderr, 'GOT INCOMPLETE BLOCK FROM PEER! %s bitcoin: %s%064x' % (p2pool_data.format_hash(share.hash), self.net.PARENT.BLOCK_EXPLORER_URL_PREFIX, share.header_hash)
04:01 < warren> self.known_txs_var.value ... that's probably it.
04:02 < petertodd> makes sense
04:02 < warren> so ugh... there is a *real* cost to decentralized mining
04:03 < petertodd> for sure, and that's the cost *now* with 1MB blocks, hell, more like 150KB blocks really...
04:03 < warren> which can be minimized with peer optimization, putting your nodes in nearby data centers or upgrading your upload bandwidth, connecting directly to other nodes, using reference clients that link to each other...
04:03 < petertodd> ....all things that cost money ultimately
04:04 < warren> some of that can be automated.  but yes, other things cost.
04:04 < warren> It's worthwhile for ASIC owners probably.
04:04 < petertodd> yup, it's the things that gavin and mike don't get: every cent spent on bandwidth is a cent not spent on hashing power
04:04 < petertodd> for now, only because ASICS bring in stupid amounts of money
04:05 < warren> the alternative for ASIC owners is to mine on a centralized pool, increasing risk to the network, and losing income from DDoS attacks on the centralized pool.
04:05 < petertodd> although, what that says is ASIC owners have reasons to just point it at BTC guild...
04:05 < petertodd> nah, a big ASIC miner should be mining solo
04:05 < warren> well, big in aggregate
04:05 < petertodd> but a small one should go for the biggest pool with hopefully the most resources to fend of dos attacks
04:06 < warren> DoS yes, but centralization is dangerous if the pool is compromised or http://xkcd.com/538/
04:06 < petertodd> but that's my point, the individual ASIC own doesn't care
04:06 < warren> You want to destroy Bitcoin?  forget fancy hardware.  Just kidnap two pool owners.
04:06 < warren> *done*
04:06 < petertodd> centralization costs everyone, not just them
04:06 < petertodd> absolutely
04:07 < petertodd> and it's most likely to happen when starting a new pool is hard... like with Mike's crazy world where just getting access to the UTXO set in it'sentirety is tough
04:07 < warren> yes, I really dislike those elitist arguments
04:08 < petertodd> he works at the biggest server *manufacturer* in the world, what do you expect?
04:08 < warren> oh, which?
04:08 < petertodd> Google. They make all their hardware from scratch
04:08 < warren> petertodd: this is largely why I'm interested in working on litecoin.  their users already accept anti-spam, so I have  lots of flexibility to try better anti-spam ideas there.  I don't need to fight the political battle in bitcoin.
04:08 < warren> oh
04:09 < warren> Right now their anti-spam mechanism is a blunt instrument, HIGH FEES ON EVERYONE AND EVERYTHING.  I aim to make those fees more targeted to encourage and discourage certain behaviors.
04:09 < petertodd> I had an interview there for a job with their hardware division actually, it's huge.
04:10 < petertodd> It was for a firmware testing position, and I'm analog electronics, so I'm not surprised I didn't get the job. :P
04:10 < warren> =)
04:11 < petertodd> If anything, it shows how desperate they were for people that they flew me down even after I made it clear I had no intention of a career change.
04:11 < petertodd> I like the "make it expensive" anti-spam rules myself, but they are only practical with small block sizes.
04:12 < warren> litecoin blocks are indeed small. =)
04:13 < petertodd> blockchain can grow 1MB per block, with 4x more blocks per hour
04:14 < warren> I intend on checking if any past blocks were bigger than 256KB or 512KB and if so shrinking the hard-limit.  It won't be risky after the majority of miners switch.
04:14 < petertodd> ah, see that would be a good thing
04:14 < petertodd> (I looked into attacking litecoin via spam, and figured it was too expensive, namecoin on the other hand is doable)
04:14 < warren> probably don't want to go all the way down to 4x smaller, since tx sizes aren't 4x smaller
04:15 < petertodd> more important though: work on off-chian tx systems, they'll help bitcoin and every alt-coin
04:15 < petertodd> even simple stuff like auditing is a big win
04:15 < petertodd> and multisig coin storage to reduce hacks
04:15 < warren> my interest in litecoin is primarily to prove that fee-based anti-spam incentives work, because I'm really angry about the bitcoin situation.
04:15 < petertodd> what, data in the chain?
04:16 < warren> No, the elitist arguments, and the blind belief that "fee competition" will somehow solve our problems.
04:16 < warren> bullshit.
04:16 < warren> Just throw more hardware at the problem
04:16 < warren> Satoshi's design is perfect.  Stop questioning it.
04:17 < petertodd> wait, so you think throwing more hardware is a solution, or the dumb way to fix it?
04:17 < warren> dumb
04:17 < petertodd> good
04:17 < warren> You can use fee-based incentives to encourage and discourage all kinds of behaviors that are beneficial to the overall network growth.
04:18 < warren> Discourage externalizing costs.
04:18 < petertodd> but see, I *do* have a blind belief in fee competition, so to speak, because I'm happy to spend $5 per tx if off-chain tx's work and are adopted widely
04:18 < petertodd> we can live with 1MB blocks basically forever even if every block is exactly 1MB
04:18 < petertodd> it's what we all signed up for, and in that scenario, spam doesn't bother me one bit
04:18 < warren> "what we signed up for" is a logical issue here
04:19 < petertodd> heh, well, it's what the source says
04:19 < petertodd> satoshi wasn't thinking too far ahead
04:19 < warren> satoshi got a LOT right
04:19 < petertodd> ...and a lot wrong too
04:19 < warren> He forgot to realize that UXTO cost isn't reflected in the fee formula.  Oops.
04:19 < petertodd> that's a BIG one he got wrong
04:20 < warren> yeah, and that's a key one a few devs fight
04:20 < petertodd> he never thought of fidelity bonds :P
04:20 < petertodd> (and I say that as someone who both invented them, and likes to make fun of them all the time)
04:20 < warren> anyway, I can't win this battle directly with bitcoin, so i'm going to prove it with litecoin.
04:21 < petertodd> well, good luck
04:21 < warren> I'm trying to push in arbitrary behavior incentives, including "punish uncompressed keys with yet another fee for no good reason"
04:21 < warren> excuse for all of this is "Hey you're unaffected with coin age."
04:22 < warren> and also "we're lowering the normal-sized tx fees.  Just don't spam and you're fine."
04:22 < petertodd> hmm... I'd suggest you focus on the UTXO business, and do so directly, don't get into the game of punishing specific tx types
04:22 < petertodd> if fees become valuable, miners will behave rationally
04:23 < warren> fees are already too valuable there, even the pool owners want fees to go down (for non-spam)
04:23 < petertodd> that just says litecoin isn't very valuable
04:23 < warren> yup
04:23 < petertodd> pool ops can always say they'll accept tx's directly iwth lower fees
04:24 < warren> they aren't smart enough to realize that
04:24 < petertodd> well then start your own pool that does that
04:26 < warren> Then I also figured out how to reduce the UXTO set by 50%.
04:26 < warren> at no cost
04:31 < warren> petertodd: err... does decreasing the hard limit really matter for them?  Even if mining pools remove the soft limit, the cost of including absurd sized tx's in the block is extremely high.
04:31 < petertodd> you mean increasing?
04:32 < warren> oh, misunderstood you later
04:32 < warren> err earlier
04:32 < petertodd> it must not be opposite day for you
04:33 < warren> very tired
04:33 < petertodd> go to bed
04:33 < warren> night =)
04:33 < warren> thanks for confirming my p2pool realization. this sucks. =(
04:33 < warren> well, it isn't THAT bad, block header forwarding can be improved
04:34 < petertodd> go to bed :)
04:34 < warren> night =)
05:43 < warren> went to bed, then my mtgox shenanigans alarm went off on my phone.  I look at the chart and *holy crap*
11:09 < gmaxwell> 00:14 < petertodd> >1MB blocks will without a doubt kill p2pool for that exact reason
11:09 < gmaxwell> no,
11:09 < gmaxwell> jesus
11:09 < gmaxwell> fucking fools, all of you :P
11:10 < gmaxwell> warren: I'm not sure how you missed the complicated design where p2pool nodes are forced to _pre send_ all the txn they're mining on to their peers
11:10 < gmaxwell> and then when they find a share the share contains the whole txn list
11:10 < gmaxwell> meaning that all the peers can recover the block cheaply
11:11 < gmaxwell> this means that (1) it doesn't take much bw for a p2pool node to transmit a block and (2) if its slow at doing this it will have a higher p2pool stale rate and naturally get paid less
11:14 < gmaxwell> The p2pool 'luck' is well within the expected norms, and it appears to have a _much_ lower orphan rate than eligius (appears because there is a lot of shot noise in any such measurement) now  (it didn't prior to the change to doing the share preforwarding)
11:16 < gmaxwell> 00:43 < petertodd> p2pool would be making the assumption that tx's have propagated to the whole network
11:16 < gmaxwell> No, it forces them to be propagated there is no assumption.
11:16 < gmaxwell> If you're mining on a txn you tell your peers about it first. If you don't your peers will discourage your shares.
11:50 < petertodd> oh, good to hear, although if you read the whole discussion warren and I did come to the conclusion that p2pool did that... in any case, still doesn't solve the real issue, which is the same as any other one: the bandwidth will get too expensive relative to the profit, but that's a long way off
11:50  * petertodd needs to stop writing about stuff at 4am when he's trying to do other things at the same time.
12:24 < gmaxwell> yea, sure but that issue isn't unique to p2pool. And sure, generally the bigger the blocks are the more genuine argument there is for cost saving existing for consolidating mining.
12:50 < petertodd> of course it's not unique, but p2pool is a good example where it very directly leads to centralization
--- Log closed Thu Apr 11 14:11:51 2013
--- Log opened Thu Apr 11 14:12:23 2013
15:14 < warren> gmaxwell: why do I see INCOMPLETE BLOCK so frequently then?
15:15 < warren> gmaxwell: whatever the issue is it appears it can be improved
15:18 < gmaxwell> warren: I _never_ got that on bitcoin p2pool. (just went to grep logs) Perhaps some genius litcoin p2pool users have modified their code in some daft way
15:18 < warren> how long do your logs go for?
15:18 < gmaxwell> warren: it's written so that it won't include a txn until it has relayed it, so other than right at startup or with modification, that should just not happen.
15:19 < gmaxwell> warren: The whole time share preforwarding existed while I was on p2pool.
15:20 < warren> hmm... weird.  INCOMPLETE BLOCK disappeared entirely four days ago.  it was like 50% before that.
15:20 < gmaxwell> it was probably some idiot who thought they could get some great advantage by turning off share preforwarding
15:21 < warren> OK, thank you for cluebatting me.
15:21 < warren> gmaxwell: btw, saw the post where ck claims to have reduced the avalon latency to 100ms?
15:22 < gmaxwell> warren: link?
15:22  * warren finds
15:24 < warren> https://bitcointalk.org/index.php?topic=18313.msg1761478#msg1761478  "I've spent quite a bit of time hacking on the Avalon(s) the last few days thanks to Aseras and then Xiangfu giving me access to them. I've come to the conclusion that any latency issues that prevent it mining on p2pool should be resolved in the next cgminer release incorporating the changes I've committed to the code. The max latency for restart should be on the order of 1
15:24 < warren> 00ms now which is fine even for 10s longpolls. Hopefully it will talk to p2pool better via stratum as well."
15:24 < warren> He goes on to say that p2pool has some other bugs that need to be fixed.
15:55 < warren> gmaxwell: are his latency improvements real?
--- Log closed Thu Apr 11 16:12:58 2013