--- Log opened Thu Mar 28 00:00:09 2013
01:23 < gmaxwell> I googled for 'access oblivious DHT' and got "Penis Extender With Topical DHT"
01:27 < warren> Keep up with the latest street lingo.
01:57 < amiller> gmaxwell i learned all about oblivious merkle trees, it's really interesting
01:58 < amiller> i tried to think of how that would apply to bitcoin but its strange because it hides access patterns for 'reads' as well as writes
01:58 < amiller> which means every time data is *read* from an untrusted service, it has to be rewritten back as well
01:58 < gmaxwell> yea, "oram"
01:59 < amiller> the data has to be encrypted for hiding the access patterns to make sense
01:59 < gmaxwell> I found a new cake-taker in this subfield for "paper doesn't do what it claims": http://www.cs.stonybrook.edu/~petertw/papers/usable.pir.williams.2008.pdf
02:00 < amiller> but maybe it would make sense for reads to be out of scope
02:01 < amiller> and it would only need to be oblivious regarding writes
02:08 < gmaxwell> (The paper claims that it prevents a system which makes PIR (like ORAM but for a public database) pratical (e.g. cheaper than just sending the whole database) — but half way through page 2 you find out that they need a trusted computing oracle (IBM4764) to do it.— not at all mentioned in the abstract.
04:59 < gmaxwell> gah. muggle hardly knows how bitcoin works at all, and his first reflex is to use it as a freeking broadcast medium:
04:59 < gmaxwell> 01:57 < Belxjander> topi`: well the only idea I have at the moment is to make an AppEngine python instance where I can pull ticker data from the blockchain...
05:05 < Graet> yeah, interesting concept, but who puts ticker data in for him to pull?
05:06 < gmaxwell> Graet: Where is your robe, wizard hat, and fitting scowl of disapproval?
05:06 < gmaxwell> :P
05:07 < Graet> i have 1 of 3 :P
05:20 < warren> I finally had time to read about the conference.  dang.  $300 registration fee.
05:20 < warren> roundtrip airfare at that time would be ~$700 right now
05:25 < Graet> only 7 and a bit btc
05:25 < Graet> will cost me ~12 in airfares each way
05:25 < Graet> ;)
05:26 < Graet> but i'd propbly fly in one side of the country and out the other ;)
05:27 < warren> I feel really stupid ... my BTC balance is 0.6
--- Log closed Thu Mar 28 07:23:15 2013
--- Log opened Thu Mar 28 07:23:27 2013
14:39 < gmaxwell> I wonder how two nodes could realize that they're both talking to a common third without disclosing who they're talking to each other.
14:40 < gmaxwell> On the subject of how do you prevent sociopaths from running nodes that connect to every other node simply because its cheap for them to do and may confer some imaginary benefit.
14:45 < petertodd> ..or less than imaginary benefit...
14:46 < petertodd> We've both independently come up with p2pool PoW's, so there is that.
14:53 < petertodd> oh, never mind
14:56 < gmaxwell> p2pool has a much stronger usecase for pows though, and a by-defintion way of geting them cheaply. :P
14:58 < petertodd> yeah, I'll punt and say "WITH SCIENCE! I mean CRYPTO!"
15:04 < warren> (It also helps that p2pool bogs down if you have too many connections.)
15:14 < gmaxwell> warren: the point for p2pool would be an outright attack... at least for a while it was very common for mining pools to get attacked.
15:15 < warren> Happens every day now for litecoin.
15:16 < warren> gmaxwell: I mean, "On the subject of how do you prevent sociopaths from running nodes that connect to every other node simply because its cheap for them to do and may confer some imaginary benefit."  p2pool fails badly if you tried to do that.
15:17 < gmaxwell> indeed, fair enough.
15:17 < warren> mostly an implementation issue
15:17 < warren> you can hack it to do more outgoing connections but it bloats and goes haywire
15:17 < warren> and really slow
15:20 < warren> gmaxwell: someone wrote a script that polls the entire p2pool address list and generates a web page that lists all public p2pool nodes and their fees.  He's afraid of releasing it though, for fear of making p2pool a DoS target.
15:27 < petertodd> It's not an absolute protection by any means, but a decent protection would be to encourage alternate network transports, including ones that rely on central services. twitter.com/blockheaders is a funny example, but seriously using Amazon EC2 message broadcast facilities and similar methods would be good
15:29 < gmaxwell> petertodd: agreed there— that was also a reason I thought a udp transport would be interesting, — though it seems jeff's work is still connection oriented.
15:31 < petertodd> gmaxwell: Yeah, I haven't looked at it in detail, but blockheader data seems particularly suited to UDP.
15:32 < petertodd> gmaxwell: The client should be able to store and compute stuff about raw block headers, some kind of "pending tx data" state.
15:33 < gmaxwell> petertodd: "new block with a verification level of 0"
15:33 < gmaxwell> well I suppose 1. Zero would be totally stateless validation I guess.
15:34 < petertodd> gmaxwell: Yup. "unknown chain with 10 bazillion knownwork"
20:04 < warren> "Bitcoin is a hedge against the entire global currency system." -- Bloomberg Businessweek
20:04 < warren> both funny and scary
21:11 < jgarzik> gmaxwell: UDP need not be connection oriented
21:11 < jgarzik> gmaxwell: That was just a convenient way to solve a few problems
21:12 < jgarzik> gmaxwell: UDP would be great for block headers, but you have to figure out how to know the membership list for receiving a broadcast.  You have to avoid amplification attacks inherent in many broadcast/subscription setups.
21:12 < jgarzik> petertodd: ^
21:14 < gmaxwell> jgarzik: sure, you can use cookies to do stateless bidi handshaking to setup an association with no state until the other side has shown its there, for example.
21:14 < jgarzik> gmaxwell: There were also issues with UDP-only CNode's that I did not want to step into, in the current implementation.  Needs-TCP-cxn was a cheat way to ensure there is always a CNode, even for UDP.
21:15 < jgarzik> cannot easily feed UDP messages into ProcessMessage() engine without a CNode
21:15 < jgarzik> etc.
21:15 < jgarzik> gmaxwell: agree it's possible, and the UDP implementation actually does a bit with cookies
21:16 < gmaxwell> ::nods:: there should probably be an always up dummy node for "all udp peers"  ... though another reason to build the udp stuff as a proxy first. :)
21:19 < BlueMatt> doesnt jgarzik have his own full-node implementation now? why didnt he code udp for that?
21:19 < jgarzik> I was thinking UDP broadcast of: block header + list of transactions
21:19 < BlueMatt> also, wouldnt that have been easier...
21:19 < jgarzik> BlueMatt: because bitcoind is more important? :)
21:19 < BlueMatt> well...ok fair enough
21:32  * BlueMatt seriously wishes the world would move off bitcoind, or...I suppose that the world could move off of bitcoind safely
21:34 < petertodd> jgarzik: bidi == bidirectional?
21:37 < gmaxwell> petertodd: yes.
21:38 < jgarzik> BlueMatt: why?  I think bitcoind is the best, most secure full node implementation out there
21:39 < BlueMatt> jgarzik: thats my point, I wish there was a library that was as secure so that we could get eaiser...stuff
21:39 < BlueMatt> something with reasonable code structure
21:39 < jgarzik> BlueMatt: what... stuff do you want?   :)
21:40 < jgarzik> Note that Java is no example of reasonable code structure ;p
21:40 < petertodd> ...and spaces apparently. :P
21:40 < sipa> i wonder
21:40 < BlueMatt> jgarzik: hell no, I'd like a C library...anyway, look at how hard it is to write your own network layer for UDP
21:40 < BlueMatt> jgarzik: you have limitations based on existing structure
21:41 < BlueMatt> that means something there is too interconnected
21:41 < BlueMatt> and it shouldnt be
21:41 < BlueMatt> also, yes, can we s/    /\t/
21:41 < jgarzik> BlueMatt: I had limitations based on what I could do in 30 minutes ;p
21:41 < sipa> wouldn't it be nice to fork bitcoind, and drop wallet and GUI
21:41 < sipa> and clean up the core
21:41 < BlueMatt> sipa: YES!
21:41 < petertodd> sipa: ACK
21:41 < BlueMatt> or...do that wallet protocol shit luke is always talking about
21:42 < sipa> wallet protocol is for talking with a wallet
21:42 < BlueMatt> nfc if its designed well, but implement the idea
21:42 < sipa> i just don't want a wallet in the first place
21:42 < BlueMatt> yes, pull out wallet and then give it a separate wallet
21:42 < petertodd> Everything related to validation and mining should be in one codebase, and nothing else in that codebase.
21:42 < sipa> to focus on what bitcoind imho should be: the core of the network
21:42 < jgarzik> sipa: certainly makes things easier :)  That's my goal with "brd", hidden inside picocoin.git
21:42 < gmaxwell> why do you think I keep trying to get etotheipi to hoist armory onto the RPC?  ... I want to dump the reference wallet into his lap. :P
21:42 < BlueMatt> yes, and it can also relay lists of txn and blocks so that wallets can attach
21:42 < petertodd> gmaxwell: sssh! he might be listening!
21:42 < sipa> BlueMatt: meh, wallets just do SPV
21:42 < sipa> done
21:42 < BlueMatt> well, ok fine
21:43 < sipa> or listen to events from a trusted bitcoind
21:43 < sipa> i don't care, really
21:43 < petertodd> Main thing wallets need is the searchable UTXO set. hint hint
21:43 < BlueMatt> how pissed would laanjw be if we did that?
21:43 < sipa> petertodd: hell no
21:43 < gmaxwell> There are some stats and such a wallet would want from its parent fullnode.
21:43 < sipa> petertodd: imho a wallet shouldn't need a thing
21:43 < gmaxwell> petertodd: thats SPV incompatible.
21:43 < jgarzik> sounds like an electrum server/client split ;p
21:43  * jgarzik runs
21:43 < petertodd> sipa: You expecting payment protocols?
21:43 < sipa> yes, i don't like that
21:43 < sipa> petertodd: yes
21:43  * BlueMatt beats jgarzik with a wet fish
21:43 < petertodd> sipa: Well... that's a long, long, long way off.
21:44 < sipa> petertodd: but what has that to do with anything?
21:44 < petertodd> sipa: After all, a UTXO set copy is a requirement for a validating node, which means the RPC should support it.
21:44 < gmaxwell> petertodd: funny multibit works just fine (cough on the fine part...) without any of that stuff.
21:44 < sipa> petertodd: wallet can perfectly keep track of their own transactions
21:44 < petertodd> gmaxwell: Right, by just downloading blocks.
21:44 < sipa> petertodd: no need to depend on any non-authorized data
21:45 < gmaxwell> petertodd: no, it uses bloom filterd blocks.
21:45 < petertodd> sipa: Remember, I did say UTXO in the *RPC*, not as a network visible thing.
21:45 < petertodd> sipa: Network visible is insane I agree.
21:45 < sipa> petertodd: i don't care, whatever protocol you use to let wallets and bitcoind communicate
21:46 < sipa> petertodd: and an address-indexed UTXO set isn't even enough for all wallet applications - you need the full transactions anyway to produce a ledher
21:47 < jgarzik> sipa: More seriously...  I proposed a similar split on #bitcoin-dev a few days ago.  Was looking into a fork(2)-based firewall between RPC/wallet/GUI and "everything else" (blockchain engine, really)
21:47 < jgarzik> sipa: would be trivial to split even further, once that happens
21:47 < sipa> agree
21:47 < petertodd> fork(2) is a clever way to do it, and easy to get started on
21:47 < gmaxwell> jgarzik: this has been said _lots_ of times in the past. By too too I think. I don't think anyone disagrees... doing that would let us do nice sandboxing eventually.
21:48 < jgarzik> I even researched Windows compatibility, RE fork+pipe: http://msdn.microsoft.com/en-us/library/edze9h7e%28v=vs.80%29.aspx
21:48 < sipa> no need for pipes; network sockets will work fine?
21:48 < jgarzik> sipa: fork+pipe is a nice existing model, can become sockets later
21:49 < jgarzik> sipa: but no big deal either way
21:49 < sipa> right
21:50 < jgarzik> in current code's context, The Program would fork off (re-run the exe, in Windows' case) the blockchain engine.  The Program would be RPC/wallet/GUI etc.
21:51 < jgarzik> gmaxwell: Well, from my perspective it seemed like everybody disagreed, when BlueMatt's big work was held in favor
21:51 < jgarzik> I prefer a more heavyweight messaging boundary (like pipe / network socket)
21:51 < petertodd> jgarzik: That also means you can make a very secure strictly non-wallet RPC interface that lives with the blockchain engine. - blockheaders and what not only
21:52 < petertodd> jgarzik: (or as a third process)
21:52 < gmaxwell> we don't need a zillion processes please. :P
21:53 < BlueMatt> jgarzik: meh, CBlockStore never worked out because it was never mergeable at the time I had free time to maintain it (along with many other issues)
21:53 < sipa> jgarzik: that was just making a hub structure for different components to communicate instead of ad-hoc... the idea was that some parts could move to other processes as well
21:53  * petertodd has shares in micron
21:53 < BlueMatt> realistically, it needs to happen in pieces, not at once
21:53 < sipa> yeah
21:53 < sipa> one of the things i'm "waiting" for, is CodeShark's split of main/core
21:54 < BlueMatt> CodeShark is working on this?
21:54 < gmaxwell> I don't see a reason for more than three proceeses  p2p network / block(+block rpc) / wallet(+wallet rpc / optionall gui)  and maybe the p2p and the block part are one process.
21:54 < petertodd> gmaxwell: right, that's exactly what I'm proposing
21:54  * jgarzik bets he can complete a fork()er before CodeShark ;p
21:54  * BlueMatt ponders writing CBlockStore part 1 version 4...
21:54 < jgarzik> mmmmm, competition
21:54 < BlueMatt> or am I on version 5 now?
21:54 < gmaxwell> then we can get to pick between pulls
21:55  * BlueMatt gives up and goes back to writing a bitcoinj full verification engine no one uses
21:56 < jgarzik> One "blockchain engine" process should manage p2p and block database, IMO.  Does not seem a need to split further (but who knows 'til ya get there)
21:56 < gmaxwell> BlueMatt: If I send you a fruit basket will you stop being sore about that? :P
21:56 < jgarzik> and that would be a natural splitting point for a further bitcoind/rest separation.
21:56 < sipa> jgarzik: core/main split has nothing to do with that; it's just that main right now has a) very low-level stuff (definitions of CTransaction/CBlock/...) and b) very high-level stuff (management of block db, verification, ...)
21:56 < petertodd> jgarzik: Keep in mind I'm specifically thinking of an RPC interface that would serve up data locally completely unauthenticated.
21:56 < jgarzik> petertodd: yes
21:56 < sipa> jgarzik: idea is that the low-level stuff moves to core.h/.cpp, so that +- everything loses its dependency on main
21:57 < jgarzik> petertodd: a blockchain engine would use something like that
21:57 < jgarzik> sipa: ah ok.  +1
21:57 < BlueMatt> gmaxwell: which one, CBlockStore or bitcoinj?
21:57 < BlueMatt> gmaxwell: and, no, I just like to bitch
21:58 < gmaxwell> BlueMatt: CBlockStore XXIVXI (the revenge) :P
21:58 < sipa> after that, i'd like to have a "block manager" or something, which just maintains CBlock's that are being worked on in memory, refcounts, and has a background thread for syncing them to disk
21:58 < gmaxwell> 2013-03-29 01:57:53  block index          716145ms
21:58 < gmaxwell> :P
21:58 < BlueMatt> valgrind?
21:58 < gmaxwell> yea.
21:59 < BlueMatt> gmaxwell: also, the amount of work that went into CBlockStore...
21:59 < BlueMatt> anyway...
21:59 < jgarzik> essentially all wallets, and other fun petertodd apps, are query clients for the public blockchain dataset
21:59 < BlueMatt> yes
21:59 < jgarzik> some apps might want additional indices we don't care about, to make things like searching for bitcoin address easier
22:00 < petertodd> yup, timestamp verification is you canonical example where a pure blockheader thing would be useful, fidelity bonded banking/ledgers needs searchable UTXO sets at the other extreme
22:00 < sipa> i do like to see a split (not necessary separate processes, but at least funcionality-wise independent) between archival block storage (with optionally some indexes) and UTXO maintainance (with optionally some indexes)
22:01 < petertodd> sipa: Good idea - needs to be done long-term conceptually for agressive SPV with a healthy network after all.
22:02 < sipa> SPV has nothing to do with that :p
22:02 < sipa> as it has neither
22:02 < petertodd> sipa: brainfart - s/SPV/pruning/
22:02 < sipa> ok
22:03 < petertodd> speaking of: blockchainbymail.com
22:04 < BlueMatt> hahahaha
22:04 < petertodd> It was going to be my April Fools joke, but then someone went off and did it so I gave them the domain. :P
22:09 < warren> petertodd: would be great if the only way to order the blockchainbymail is with bitcoin.
22:11 < petertodd> sigh, recursion...
22:12 < petertodd> related: I'm thinking for a merklized AST what makes sense is merklized forth. The forth dictionary concept is perfect for it, and means you have a simple, easy to implement language already used for embedded andother things (and bitcoin scripting) along with all the usual nice things like editor modes and what not
22:12 < sipa> did you mean: recursion?
22:12 < BlueMatt> sipa: no he meant recursion
22:12 < sipa> BlueMatt: the recursive kind?
22:13  * petertodd is an analog electronics designer, just so you know.
22:13 < sipa> petertodd: dude, what are you not?
22:13 < petertodd> sipa: well, I'm not an expert at anything...
22:19  * jgarzik ponders the bandwidth of QR codes
22:19 < jgarzik> Could a 1MB block fit on a single, printed 8.5" x 11" page?
22:20 < jgarzik> easy enough to have multiple QR codes
22:21 < sipa> jgarzik: there's even a standard for that
22:22 < petertodd> 400dpi works out to 1.87MB at 1 bit per dot
22:22 < petertodd> so I'm guessing no, but it's not far from possible
22:23 < petertodd> computer data storage on paper used to be a thing
22:32 < jgarzik> so
22:33 < jgarzik> header + coinbase tx + list of TX hashes is sufficient to recreate a full block byte-for-byte, assuming fully cached TX's
22:33 < jgarzik> correct?
22:33 < sipa> yes
22:42 < petertodd> Merkle Forth: So you've got your parameter stack and return stack, and are thus at the point where you can recreate Bitcoin scripting. Now the interesting thing to do is add TPM functionality, which means a PCR opcode and stack to allow you to select what you want to consider as the start of the current trusted block of code. Then add an encrypted stack, as expected encrypted with H(sec|PCR tip), and some sort of monotonic counter thing. That should give you enough to do trusted computing with an extremely stable API, and that API itself can be just AST heads of useful library function calls that may actually be implemented directly in C or whatever rather than the opcodes themselves.
22:43 < petertodd> Now off-chain tx's with trusted hardware is just a matter of agreeing on a common program that will manipulate the counters representing value attached to the private keys, as well as agreeing on what signatures sign the classes of hardware you can trust.
22:44 < petertodd> With some careful design you can probably even use the programs themselves to prove fraud/compromised trusted hardware, basically by just providing a program that should have run, and some kind of execution trace proving it didn't do that, at least in many cases.
22:45 < petertodd> Equally, that also makes designing redundent hardware easier, as you can reuse the execution traces to determine if two sealed up uC's runningt he code executed the code in the same way on the same data.
22:47 < gmaxwell> petertodd: I don't know that explicitly supporting that makes sense.. simply because you can just have the trusted hardware produce signed message regardless, even without support.
22:47 < petertodd> gmaxwell: I'm not sure either yet - strikes me that doing traces + monotonic counters could be very tricky, but it's worth looking into at least.
22:57 < petertodd> Anyway, I think the interesting part is more that with the model that you build up every part of the language from the forth primatives you make it very, very clear what code is actually being run. Equally, forth is already common in applications, IE spacecraft, where you need relatively bare metal languages with simple frameworks and symantics; note how with forth it's much easier to get to the level where you trust that the code being run is what you actually wrote than, say, C.
22:57 < petertodd> It's fundementally the same math wise as a tonne of other approaches, but forth makes what is going on very explicit.
22:58 < sipa> petertodd: type this: /script load splitlong.pl
22:58 < petertodd> ok, I typed rm -rf /, but it doesn't seem to be doing much
23:00 < petertodd> Anyway, I think the interesting part is more that with the model that you build up every part of the language from the forth primatives you make it very, very clear what code is actually being run. Equally, forth is already common in applications, IE spacecraft, where you need relatively bare metal languages with simple frameworks and symantics; note how with forth it's much easier to get to the level where you trust that the ...
23:00 < petertodd> ... code being run is what you actually wrote than, say, C.
23:03 < petertodd> The actual implementation can be some tiny and primitive C kernel with static memory layout. Just be clear what the maximum's are for the variou parts of the stack. Dunno yet what the stack datatype should be, MPI's are nice but there is the subtle issue that it'd be good to have some clear idea of how many operations an operation takes. Of course, really simple would be 32-bit ints and implement everything higher level in forth.
23:41 < jgarzik> go go Open Firmware
23:41 < jgarzik> the BIOS standard that should have won
23:42 < petertodd> for sure
23:43 < petertodd> With any luck some OpenFirmware TPM modules will become available and I won't actually make to write any code. :P
23:45 < jgarzik> OK
23:45 < jgarzik> Query results:
23:45 < jgarzik> Average block summary size: 8858.34
23:46 < jgarzik> That's block header + (vtx.size() * 32)
23:46 < jgarzik> does not include coinbase average size
23:48 < BlueMatt> ok?
23:49 < jgarzik> That's how big a UDP frame distributing block data would be
23:49 < sipa> you technically also need 1-3 bytes for the number of transactions
23:49 < jgarzik> nod
23:50 < jgarzik> call it 8858 + 4(n_tx) + 512(coinbase)
23:50 < jgarzik> I imagine 8-16 bytes of overhead would sneak in somewhere
23:50 < sipa> never 4
23:50 < BlueMatt> meh, average doesnt matter much since it new blocks are so much bigger than average
23:51 < sipa> 1,3,5 or 7 :)
23:51 < sipa> eh, 1,3,5 or 9
23:51 < sipa> but i doubt any block has over 65535 transactions :D
23:52 < jgarzik> 8858+3+512+16 == 9389 bytes
23:52  * jgarzik wonders how much droppage that would cause, given that it is well over size to be fragmented across WAN
23:53 < jgarzik> To avoid fragmenting, you could only looking at shipping smaller bits of useful data: block header only, small TX's, INV's, addresses perhaps
--- Log closed Fri Mar 29 00:00:11 2013