00:10:53Dizzle_:Dizzle_ is now known as Dizzle
00:54:36gmaxwell:Can anyone point me to an existant 0 value spendable txout (e.g. to save me from having to scan the utxo set myself to find one, perhaps one of you has done this recently.)
07:50:52Aquent1:Aquent1 is now known as Aquent
08:14:27lclc_bnc:lclc_bnc is now known as lclc
08:17:24lclc:lclc is now known as lclc_bnc
08:18:14lclc_bnc:lclc_bnc is now known as lclc
09:05:17hitchcock.freenode.net:topic is: This channel is not about short-term Bitcoin development | http://bitcoin.ninja/ | This channel is logged. | For logs and more information, visit http://bitcoin.ninja
09:05:17hitchcock.freenode.net:Users on #bitcoin-wizards: andy-logbot adam3us op_mul bosma_ vmatekole kristofferR paveljanik Aquent justanotheruser jtimon ryanxcharles hashtagg hashtagg_ catlasshrugged TheSeven NewLiberty Logicwax dgenr8 MoALTz_ jgarzik moa jaekwon_ samson_ Shiftos OneFixt e1782d11df4c9914 maaku Emcy roconnor Starduster copumpkin mortale tacotime DougieBot5000 stonecoldpat sl01 optimator atgreen [d__d] koshii rasengan adlai wiz shesek null_radix ebfull JonTitor pigeons pi07r_
09:05:17hitchcock.freenode.net:Users on #bitcoin-wizards: Fistful_of_Coins roasbeef_ SubCreative Keefe hktud0 grandmaster luny digitalmagus8 Transisto hollandais BigBitz [\\\] michagogo Muis_ mappum epscy gnusha poggy _Iriez Meeh otoburb mr_burdell s1w go1111111 bsm117532 HM2 Apocalyptic sdaftuar waxwing btcdrak CryptOprah kumavis btc__ jbenet nuke1989 nsh Guest19027 jcorgan wizkid057 PaulCapestany forrestv Luke-Jr tromp_ PRab c0rw1n LarsLarsen Greed Dyaheon lclc spinza harrow @gmaxwell
09:05:17hitchcock.freenode.net:Users on #bitcoin-wizards: gavinandresen NikolaiToryzin Cory isis iddo sadgit brand0 eric Krellan berndj @ChanServ jaromil petertodd tromp helo v3Rve midnightmagic espes__ fluffypony butters nickler lnovy ahmed_ warren fenn phantomcircuit Graet morcos kanzure gribble MRL-Relay Graftec bobke huseby BananaLotus amiller artifexd coryfields coutts BlueMatt cfields Anduck Eliel nanotube hguux_ AdrianG gwillen wumpus dansmith_btc heath bbrittain DoctorBTC EasyAt starsoccer
09:05:17hitchcock.freenode.net:Users on #bitcoin-wizards: danneu catcow TD-Linux ryan-c smooth mmozeiko Alanius asoltys_ K1773R a5m0 comboy Nightwolf andytoshi lechuga_ warptangent Guest38445 BrainOverfl0w phedny so crescendo throughnothing Taek burcin livegnik sipa harrigan sneak yoleaux azariah kinlo
09:15:44bosma_:bosma_ is now known as bosma
09:16:13bosma:bosma is now known as Guest40810
09:42:40wumpus:curious that I hadn't heard of this before re: open source CPU architectures, a 64-bit one http://www.lowrisc.org/, the discussion on the tagged memory page also mentions CHERI
09:43:30Guest40810:Guest40810 is now known as bosma
11:25:45Pan0ram1x:Pan0ram1x is now known as Guest52336
13:38:01e1782d11df4c9914:any serious implementations of MPC with respect to coinjoin anyone know of as being in development atm?
13:47:16op_mul:oh right.
13:49:02e1782d11df4c9914:sorry, multiparty computation with respect to coinjoin
13:49:38e1782d11df4c9914:are there any alts or bitcoin forks lying around in development? interested in the state of research
13:50:51op_mul:if you're looking for interesting cryptography altcoins really aren't the place for it. Monero/Bytecoin has some interesting stuff going on with ring signatures, which seems to be the exception.
13:51:44e1782d11df4c9914:right monero is what i was thinking of, but i'm not that interested in ring sigs, i understand its easier to implement than MPC, but i guess most of the research has been academic to date- just wondering if there were any whispers of ambitious implementers anywhere
13:55:24op_mul:I can't speak for what anybody else is doing.
13:56:10op_mul:as far as I can work out the trouble with coinjoin is always going to be peer discovery. no matter what you do, there's always going to be the danger of somebody flooding out your join with sybil joiners and discovering information about it.
13:59:22op_mul:you can't *lose* privacy that way, you just stand to gain none. you can't evaluate the quality of your privacy, either.
14:05:51e1782d11df4c9914:i havent researched the literature too deeply with respect to coinjoin specifically, but couldnt you construct a scheme that requires joiners to construct a proof towards a challenge to help discourage sybil joiners?
14:07:05e1782d11df4c9914:or at least construct some requirement that makes sybil joiners uneconomical with respect to inclusion within a join
14:08:05op_mul:I'm not aware of any way of doing that
14:08:30op_mul:any limiter you can come up with, I can scale to massive proportions
14:08:50op_mul:if it's burning money, sure, I won't be doing that, but people also won't want to use it
14:09:28e1782d11df4c9914:thats ok, just have to find some proof that makes it uneconomical to forge etc.
14:10:05e1782d11df4c9914:i'm sure some of the more dedicated minds have already thought of some alternatives, hopefully i can come back here with more discussion after some research
14:11:21op_mul:finding a bottleneck to exploit is nearly impossible, or impossible.
14:28:40wallet42:wallet42 is now known as Guest25452
14:28:40wallet421:wallet421 is now known as wallet42
14:31:32zooko``:zooko`` is now known as zooko
14:47:34NewLiberty_:NewLiberty_ has left #bitcoin-wizards
16:27:03Taek: Sorry if this has been covered, but is storj following the original idea? If so, why does it need its own blockchain?
16:27:11Taek: justanotheruser, it is not following the original idea at all
16:27:17Taek:What is the original idea?
16:34:52fanquake:fanquake has left #bitcoin-wizards
16:38:40sipa:ask gmaxwell
16:40:07Taek:relevant: http://garzikrants.blogspot.com/2013/01/storj-and-bitcoin-autonomous-agents.html
16:43:15Taek:seems like the original idea was not decentralized?
16:44:27Taek:as in, the storj code that's serving storage had to be run on trusted hardware, is only run in one place, and the customer has to figure out if the storj instance is trustworthy/
16:48:24jgarzik:Taek, the original idea has nothing to do with altcoins
16:49:47adam3us:Taek: I dont see why you need an alt. probably they dont, which means its an appcoin. ie why do you think the storj ICO holders can expect the p2p users on the network to pay economic rent to the ICO holders. they cant so what will happen in reality is someone will fork the network and pay for services at market rates in bitcoin.
16:50:50Taek:I don't claim to know very much about stroj the altcoin, I've never properly groked their whitepapers
16:51:17Taek:but a nice feature of storage is an automatically enforced contract that penalizes a host if they can't prove they have the file
16:51:30adam3us:Taek: havent read them. but sometimes when it makes your head hurt its because it doesnt compute :)
16:52:15op_mul:I don't understand why people would want to use storj in it's current form
16:52:54op_mul:you pay people to store files you already have on your disk, and you've got to keep uploading them again to the network to repair it. might as well just buy some storage somewhere proper.
16:53:10adam3us:Taek: its an observation that there are more things that it'd be nice to be able to robustly prove in a distributed system than we know how to enforce. (and when you put fungibility money on top of it people are going to sysmtematically abuse any gaps in enforceability)
16:54:14Taek:we know how to make someone prove they have a file in a decentralized context
16:54:42adam3us:Taek: i suspect most of these systems have that a) because its hard or in some cases impossible; and b) mostly they are enthusiastic or have their eye on marketing the ICO or coin pump more than coding; and c) they mostly dont know what they're doing. (hence the white papers are mostly technobabble as gmaxwell calls it)
16:55:25op_mul:Taek: well, that's sort of sticky. you can ask someone to prove *somebody* has a copy of a file, you can't prove a particular person has it.
16:55:32Taek:"these systems have that" --> what is that?
16:55:34adam3us:Taek: well can it be proxied for example? can i pay someone a tiny payment to answer the question i relay them?
16:55:43adam3us:Taek: technobabble
16:56:11Taek:do you care if the answer is proxied? As long as someone has your file you should be happy
16:56:27adam3us:Taek: well you do if you're paying for redundancy
16:56:39Taek:if you're worried about redundancy, you should encrypt a bunch of pieces after you encode them
16:56:42op_mul:Taek: that's where all of these systems of "pay full nodes" fall down, which is a similar sort of concept. people want to pay nodes for storing data, but there's no way to prove they actually are.
16:57:01Taek:that way someone can't cheat by pretending to have multiple copies - they don't know how to decode the redundant files
16:57:13adam3us:Taek: getting better, but now you're designing it for them.
16:57:18op_mul:that doesn't prove many people have it though.
16:57:28Taek:adam3us: designing it for storj?
16:57:34op_mul:they could all be just me, and therefor the "redundancy" is worthless.
16:57:53adam3us:op_mul: Taek means that you'd secret share it with high redundancy and hten store the unique shares once each
16:58:10op_mul:if I had some magic cheap storage, sybiling the storj network would presumably be profitable.
16:58:13adam3us:Taek: i dont know how storj works actually.
16:58:31op_mul:adam3us: they do something like that. encrypt the chunks uniquely for each host.
16:58:32adam3us:op_mul: bingo. the hard part is compactly proving storage.
16:58:53Taek:op_mul: sybil attacks would be a concern. There are ways to mitigate that but no golden bullet
16:58:58op_mul:yes, they do a challenge response system with "heatbeats".
16:59:03op_mul:Taek: no there's not :)
16:59:10adam3us:Taek: mitigate = systematically abused.
16:59:14Taek:ie each host locks coins away for a few moths to add weight to themselves
16:59:33Taek:adam3us: what do you mean by that?
16:59:38op_mul:* op_mul giggles
16:59:48op_mul:every solution is the masternode one now, eh? :)
17:00:17adam3us:Taek: say you want something to work but it doesn quite. so you patch it up a bit ofr obvious things that occur to you. you put fungibile money on top of it, some hacker makes swiss cheese of your "patch it up" stuff and brings the system down
17:00:31Taek:I don't think masternode was the first one to have that idea. You wouldn't trust consensus to people with more storage, only files
17:01:31adam3us:Taek: one way to compactly prove storage is to request a fiat-shamir transform over it with your challenge.
17:01:52adam3us:Taek: i presume thats what the jgarzik / gmaxwell post is about.
17:02:14Taek:I think that one requires someone who already has the data to pose the challenge right?
17:02:23jgarzik:adam3us, StorJ as envisioned in the blog post an autonomous agent
17:02:29adam3us:Taek: yes thats the problem
17:02:34jgarzik:adam3us, not much to do with storage proofs or decentralization
17:02:41jgarzik:*is an
17:03:07Taek:There's a weaker alternative using merkle trees. You can't make them prove they have the whole file but you can make them prove they have a segment, and anyone can pose the challenge
17:03:41Taek:Ask for 1 random segment a day, and after a few weeks you're pretty convinced they have 99%+ of the data
17:03:58op_mul:so? that's only proof for you
17:04:17op_mul:that's a hell of a lot of data if every node is asking for proof of every chunk
17:04:33adam3us:Taek: i presume you're talking about this http://storj.io/ and that yo're saying it has an alt or ICO behind it?
17:05:03op_mul:adam3us: yeah, they did some annoying crowd funding thing and made bank.
17:05:22Taek:what is ICO?
17:05:25op_mul:adam3us: http://storj.io/crowdsale.html
17:05:37sipa:Taek: initial coin offering
17:05:42adam3us:Taek: marketing/branding for premine / presale
17:07:10adam3us:Taek: there's a paper by coelho on a proof of work using storage. vitalik had a go at tweaking it as dagger. (still has tmto so he abandoned it)
17:07:19kanzure:so since nobody has figured out a way to prove redundancy or anything, isn't this a principle agent problem for people running data centers or hosting services (like s3)?
17:07:39gmaxwell:Taek: " you can make them prove they have a" no you can't. You can have someone prove they have a segment, but its fully delegatable.
17:07:40op_mul:time memory tradeoff
17:07:43adam3us:Taek: its related to proof of storage: http://hashcash.org/papers/merkle-proof.pdf
17:08:09op_mul:sipa: I like ToMaTO as a term for that though.
17:08:30sipa:TiMeTO, you mean?
17:08:42kanzure:gmaxwell: i wonder if you could make the delegation obvious, or in such a way that relays/delegation can only pass on the original response, and not modify the pay-to info
17:08:48Taek:you say TiMeTO I say ToMaTO
17:09:18op_mul:sipa: ToMaTO makes sense as a joke about the fruit though. TiMeTO is just odd.
17:09:22adam3us:kanzure: they could ip-forward your request.
17:09:29gmaxwell:kanzure: sure you can make things where you can't secretly delegate... but mining in bitcoin is a sybil resistance mechnism, which storage queries can't substitute.
17:10:00gmaxwell:kanzure: I mean you can't make it secret from the party recieving the request.
17:11:18Taek:why is it important if the service you are talking to is the service controlling the data?
17:11:22adam3us:gmaxwell: maybe you could prove the miner knows a private key. i think monero tries to do that maybe
17:11:26gmaxwell:kanzure: e.g. if the requests are always pseudorandom the request is H(nodeid||nonce) % range.
17:11:46gmaxwell:Taek: because the optimal solution otherwise is for there only to be one copy of the data.
17:12:47adam3us:you know one of these years an alt will invent something. maybe. perhaps.
17:12:57gmaxwell:Taek: and if you've made this part of the incentives in a consensus system, the optimal solution is that one-archive starts filling the network with 'files' that contain nothing but H(secret||counter) which it, knowing the secret, can answer super cheaply without storing.
17:12:58kanzure:how did that "proof of deletion" thing work? /me looks
17:14:16Taek:gmaxwell: only a problem if you're using storage as your means of consensus. Something I was trying to do a while ago but have since abandoned
17:14:22gmaxwell:adam3us: monero doesn't... and I don't think that helps, because the big disk could just be an unbounded number of identites.
17:15:15gmaxwell:Taek: I did say 'if', not because I had any knoweldge of you not doing that; but because it was promoted and I know it to be broken.
17:17:29lclc:lclc is now known as lclc_bnc
17:18:13gmaxwell:kanzure: that proof of deletion isn't what it says it is. It seems to be a "I promise I'm deleting key x -- bob" such that if you later see anything decrypted with key x, you know it lied.
17:19:02gmaxwell:should be called "promise of deletion" not "proof of deletion" :P
17:22:39Taek:so, assuming you can't solve delegation, you can still have worthwhile proof of storage by encrypting your redundant copies before uploading them
17:23:07Taek:you have to be around to make your own repairs, but if you pick a high enough redundancy you don't have to be around often
17:23:51Taek:and some redundancy schemes mean you don't even need the whole original file to make repairs: http://arxiv.org/pdf/1005.4178.pdf
17:25:12gmaxwell:Taek: yes, you can add redudancy but if there is no reason for it to be stored on multiple systems it may not be meaningful redundancy. (worse, it's hidden from you how non-redundant it is)
17:26:46adam3us:Taek: i imagine if you can repair it you can compact it while removing redundancy and then simulate the redundancy in answer to the proof of storage and store more cheaply (but less redundantly)
17:28:49gmaxwell:adam3us: yes, thats solved by making everyone unable to repair it.
17:28:50adam3us:Taek: and you need pretty high redundancy if you're storing in a p2p network with churn.
17:29:25adam3us:gmaxwell: but then he's back to having to be around to repair his own files.
17:36:01Taek:I imagine you'd want redundancy for any data you plan on storing, but it's true that you'd be uncertain about how much geographic redundancy you're getting. You'd know the phyiscal data existed N times but it could all be on the same platter
17:36:02gmaxwell:adam3us: which is what he said. :)
17:36:27gmaxwell:Taek: would be, because the system is vastly more efficient if there is only one copy.
17:37:30Taek:but also very unstable. I believe the general rule of thumb for corporations is 3 copies
17:38:47adam3us:gmaxwell: yeah that was a little circular.
17:39:50adam3us:gmaxwell: at least i guess its clear it would be desirable to have the network able to repair, and not able to reduce redundancy while still collecting full storage fees, tho we dont know how to do that or if its possible.
17:40:30adam3us:Taek: yeah what gmaxwell was saying up a bit was it'd all end up on one disk.
17:42:20op_mul:adam3us: I don't think it's possible, myself.
17:42:20Taek:I think in practice there would be enough ways for a client to get the data in multiple places. You'd likely be falling back on layer 8 processes, but since the client is manually choosing who to rent storage from, it's not consensus critical
17:49:59michagogo:Is it theoretically possible (yes, I know it's unlikely enough to the point that it may as well be considered impossible) to have 256 bits that SHA256 to themselves?
17:51:33op_mul:I don't see why not.
17:52:02op_mul:not sure we know if any given output is actually possible or not though
17:52:31adam3us:Taek: this is starting to sound less like a p2p system, with built in incentive or m2m fee based renting that you could let run without human intervention. a hacker is going to get in there and rob the renters blind :)
17:52:35sipa:michagogo: if it's a perfect random function, there is 63% chance for that :)
17:54:22adam3us:op_mul: well maybe.. if u admit enough snark, obfuscation & fully homomorphic encryption
17:54:46adam3us:op_mul: tho possibly not simply and not now
18:15:09Taek:adam3us: m2m?
18:24:17op_mul:adam3us: even if it worked I question the utility of such a service.
18:26:10adam3us:op_mul: people pay amazon for storage, they have some geo-redundant options also. there are some p2p storage things that are a bit like this, though they dont try to be byzantine secure nor necessarily central control-free.
18:26:26Taek:op_mul you could end up with somelike like Uber or Airbnb for storage, where lots of small parties collaborate to make something a lot more effective than a giant alternative
18:27:07adam3us:op_mul: i used to work for mozy (well emc/vmware who bought mozy) who had a 50PB cloud storage and analysed a number of related systems including p2p ones.
18:27:50adam3us:op_mul: eg spideroak (not p2p), then there's zooko's tahoeLAFS which is p2p, and another one or two who's names i forget
18:28:40op_mul:who cares if storage is p2p or not. peer to peer and decentralised systems are inefficient and expensive. I'd trust amazon glacier for backups more than I would any of these half baked IPO nonsense altcoins.
18:29:59op_mul:if you encrypt your backups, it doesn't matter where they are stored so long as they exist in some form.
18:30:56Taek:and as long as nobody decides to change their terms, jump their prices, or discontinue your service w/o refund for violating obscure TOS
18:30:57adam3us:op_mul: well indeed ICO nonsense coins. but tahoeLAFS for example is pure p2p and existed before bitcoin and has no alt associated. i presume its sort of tit-for-tat economics like bittorrent.
18:32:18adam3us:op_mul: the argument however may work in that amazon is arguably not that cheap in bulk if you're storing a lot and p2p could provide it cheaper. there was also spacemonkey.com by a few ex-mozy guys https://www.spacemonkey.com/ that has an economic model (no altcoin, you pay for a service and you get the hw discounted)
18:32:50adam3us:op_mul: with spacemonkey the p2p cloud is made by unused storage in the drives
18:34:52op_mul:adam3us: there's no reason to make things decentralised unless they absolutely need to be. the inefficiencies make it quite undesirable.
18:36:14Taek:op_mul: decentralized perhaps not, but p2p is a different story. If you want to build 10,000 small datacenters in the US it's going to be very expensive, but if you're doing p2p you get that as a side effect
18:37:57op_mul:I don't really follow there.
18:38:42Taek:In a system of p2p storage, you have hundreds to thousands of nodes distributed all over the world
18:39:03Taek:the distribution can be leveraged to get massive parallelism, and to keep files in a place that's geographically closest to you
18:39:12Taek:(disregarding byzantine attacks)
18:39:36op_mul:lots of small nodes to me just says inefficiency.
18:39:54Taek:everyone has multiple nodes within 50ms
18:40:40Taek:it's a lot cheaper to move bandwidth from idaho to idaho than it is from Seattle to idaho, or Ireland to idaho or wherever your massive datacenter is
18:41:07Taek:and you get much stronger geographic redundancy
18:42:09op_mul:I have no idea where those cities are. quite often you're not going to get people physically close to each other having a direct path. sometimes your file will go hundreds of KM off to some hub and then back again.
18:42:36op_mul:I don't think, in that sort of situation, bandwidth cost is even a factor.
18:42:40Taek:it's something that you can nonetheless optimize better if you have many small nodes as opposed to few large nodes
18:42:52gmaxwell:Taek: it's actually not always cheaper to move data short distances, it's cheaper to move data between hubs, because capacity costs scale by the muxing width. E.g. It's (much) cheaper to have 100GBit/s to one fairly far destination then it is to have 1GBit/s connectivity to 100 moderate distance desitnations.
18:43:26gmaxwell:Cost per megabit in the last mile is orders of magnitude higher than it is at the centers of networks.
18:43:48op_mul:and I'd have my gigabit port over 100 ADSL customers connections any day of the week.
18:44:00adam3us:gmaxwell: and that argues for what op_mul is saying - its inherently bandwidth inefficient to use p2p storage.
18:44:32adam3us:gmaxwell: the datacenter bw is way cheaper & you need higher redundancy because of peer churn. lose^2
18:45:16gmaxwell:yea, it is. It may achieve some 'useful' cost externalities (e.g. foolish providers who've sold all you can eat plans that are waiting to be exploited), but such solutions are not inherently stable since people often consider you a problem when you exploit the deal they gave you. :)
18:45:25adam3us:gmaxwell op_mul and yet still bittorrent is still using the most bandwidth of any protocol clogging those slow /expensive ast mile links
18:46:38Taek:is this wrong: with central nodes you travel the last mile once, but with p2p you travel it twice? Is it bounded at twice?
18:47:01gmaxwell:bramc probably does not recall, but many moons ago when bittorrent was new and I ran a public network; I wasted some number of cycles with him arguing that bittorrent should be more topology aware, because as designed it was very abusive of expensive last mile links, and it was going to cause providers to block/rate limit it because it'll break their networks. :P ("I told you so.")
18:47:22gmaxwell:(though I think providers turned out to be less agressive than I expected, to be fair.)
18:47:39adam3us:Taek: probably. and with bittorrent you travel it hundreds of times
18:47:53adam3us:gmaxwell: if the isps knew what was good for them they'd install a mega torrent cache.
18:47:56gmaxwell:Taek: no, it's not just twice, its that the fact that you could _choose_ to go down any of many paths means that all paths have to have the capacity.
18:48:58gmaxwell:adam3us: yes, **ahem** some people went into business selling tools for caching and shaping after it became an issue; though it seems throttling it cheaper to deploy.
18:49:23adam3us:gmaxwell: hearn was talking about micropayment hosted file sharing to kill bitcoin. i had mentioned to him that he should maybe talk to bram about using it to improve tit for tat in bittorrent.
18:49:23op_mul:Taek: the topography can be wildly different just depending on where it is. I know of one setup where houses in the same town have a 100ms, hundreds of kilometers round trip. in that case a P2P connection is going to have quite an impact.
18:49:34gmaxwell:Taek: costs in networks are related to the potential capacity, not the actual used capacity.
18:49:42wumpus:adam3us: content-addressable-storages at hubs would be a nice optimization of the internet, I vaguely remember some plans, but I guess the economics doesn't work out
18:51:12Taek:gmaxwell: in p2p is that a fair assessment? When you build a p2p network you assume the infrastructure is already in place and doing other things. You make use of the already existing capacity instead of building out more.
18:52:36Taek:op_null: the other advantage to p2p/decentralization is that (in theory) it forces the price much closer to at-cost
18:52:53Taek:right now, if I want to compete with s3, I need to build a reputation and grab customers
18:53:25Taek:but on Bitcoin, if I want to compete with a mining rig, I just have to plug in my cheaper hardware, no need to find customers
18:55:03gmaxwell:Taek: consider two cases: one where things are p2p and one where its client/server-ish. Say the usage increases and links are strained. In client/server there is one hotspot where you can put your upgrade costs, in p2p you have to upgrade many last mile links. If the p2p was a spherical cow of perfect topology awareness and unbounded adaptivity, then I suppose you could purely upgrade some serve
18:55:09gmaxwell:r exactly like in the client/server model and it would figure it out, but real systems aren't like that.
18:55:41wumpus:adam3us: I can't remember what it was called, but the realisation was that most of internet traffic is indeed the same files all over again, it involved clients requesting blocks of data by their hash, which are cached by intermediate parties as necessary
18:55:44gmaxwell:Worse, this traffic competes with other traffic which can't just move elsewhere... so for a low priority p2p thing it might be okay to just let the links saturate to all hell, thats not okay for the other traffic sharing the links.
18:57:00gmaxwell:(it doesn't help that BT doesn't use erasure coding, so you can very easily bottleneck on a single piece. :( )
18:57:20gmaxwell:(freenet does much better, ignoring the fact that it's freenet and thus hardly works.)
18:57:34op_mul:gmaxwell: ugh man not again. now I've got that song stuck in my head again.
18:58:34op_mul:https://www.youtube.com/watch?v=eSMeUPFjQHc < every time you say erasure coding.
18:59:54gmaxwell:I have a couple erasure albums around here someplace.
19:00:24Taek:gmaxwell: in my sphereical cow world, when two pieces of data are fighting for the same link, they compete on price and the one offering a higher payment gets higher priority. Humans can then tell where to add new lanes to maximize profit and can move p2p data to nodes with cheaper bandwidth
19:00:44Taek:perhaps it wouldn't work so well in practice
19:00:45zooko:Hey folks, Tahoe-LAFS is related to your apparent interests.
19:00:56zooko:However, I can't stay to read backlog carefully and chat because I have a date (!?!). :-)
19:01:38sipa:zooko: reading backlogs and chatting while having a date is totally in
19:02:06Taek:if it's anything like my last date you can be buried in #wizards while she's buried in her facebook feed
19:02:37fluffypony:hah hah Taek
19:04:29wumpus:adam3us: oh, found it: that project is called CCNx
19:20:59phantomcircuit:op_mul, how does one even construct a network in which there's 100ms rtt between physically close
19:21:18phantomcircuit:are there like 2 isps which dont have a peering point in town?
19:21:55op_mul:phantomcircuit: yep, exactly.
19:23:24gmaxwell:phantomcircuit: thats the topology you get when you block adjcent port traffic so you can enabled "lawful intercept", hurrah.
19:24:20phantomcircuit:btw maybe someone here knows of an easy way to do this
19:24:43phantomcircuit:i've been trying to get the metadata for a torrent using just the info hash (ie through dht)
19:25:04phantomcircuit:but i cant find any simple tools to do this
19:25:35op_mul:I had one at some point
19:26:07op_mul:there's some command line tool that will get the metadata from a .torrent, or download it via DHT if you just give the hash. let me see if I can find it.
19:29:43op_mul:phantomcircuit: duno. the tool does exist though.
20:21:21adam3us:about the btcd guys I finally replied to them (this is on a flame-laden bitcointalk thread where one of them replied in defence of why its not a problem for them to be risking breaking consensus) to suggest they listen to https://archive.org/details/The_Science_of_Insecurity_
20:22:05adam3us:about why language security between different implementations is a critical aspect of host security problems. (any encoding of messages is a language of some kind). the talk is academic and specifically talks about the problem.
20:22:19adam3us:and invited them to talk about it (here or wherever) we'll see...
20:23:28pigeons:kind of reminds me of that android bug where the code to check the apk signature was in c and behaved differently than the java code in the installer
20:24:17gwillen:pigeons: that was a wonderful bug, which I have personally exploited (it's pretty easy)
20:24:22adam3us:their specific example was hilarious, using this mode of thinking, they wer eable to find 10 different ways to fool a CA into signing a domain name that was displayed & interpreted differently to the CA than the browser
20:25:31adam3us:that was another research referred to briefly. but the actual talk is quite interesting and focussed on fundamental limits of security particularly undecidability, halting problem related concept, and what that means specifically for the way secure code should be written, and what things you should avoid.
20:27:12pigeons:yeah CAs is a good one, anything with ASN.1
20:27:43adam3us:pigeons: at some point she says *cough*ASN.1*cough* as an example of something with undecidable parsing
20:33:02gwillen:adam3us: does she actually cough in the presentation? That's hilarious.
20:35:22adam3us:gwillen: yep its at 28c3 so she's playing to the audience a bit about p0wnage.
21:04:57tacotime:adam3us: i'm here and from conformal too
21:05:23adam3us:tacotime: you are? i thought u were talking about some other project a few days ago
21:05:26tacotime:i mean, the number of btcd nodes running on the network right now is pretty minimal if i recall
21:05:40tacotime:yeah, i'm doing some non-specifically btcd-related stuff
21:06:07adam3us:tacotime: well so the impression people had through the backchaneel is conformal was actively trying to persuade people to use btcd
21:06:38adam3us:tacotime: davec on bitcointalk didnt seem to view that as a fork risk.
21:07:15tacotime:and i'm not sure any mining nodes are running it, which i think could be a risk if there was a fork of some kind. my overall opinion is that reimplementation of bitcoin in other languages can only strengthen core, and that if you're running a node that's consensus specific you may want to run bitcoind along side it to make sure they're both stuck on the same fork.
21:08:07tacotime:conformal would like people to use btcd, because they've tried to program their own user-demanded use cases into it and have spent a lot of money on the project. so it's sort of reasonable.
21:09:21adam3us:tacotime: well whats not reasonable is breaking bitcoin consensus.
21:09:31wumpus:any plans for btcd to use libbitcoinconsensus?
21:09:36adam3us:tacotime: here's what davec said on bitcointalk
21:09:45tacotime:but i mean; that people are going to make daemons with other use cases is inevitable, e.g. coinbase which is on the ruby impl. that doesn't mean you shouldn't make sure your daemon isn't on the main fork.
21:10:39lclc_bnc:lclc_bnc is now known as lclc
21:10:51adam3us:tacotime: (about fork risk) this is a complete red herring and exists just as much with Bitcoin Core as it does with anything else.  The exact same thing can easily happen (and already has as you've noted) with Bitcoin Core. This is a real problem with the technology that needs to be solved independent of the number of implementations on the network.
21:10:54wumpus:making daemons with other use cases is fine, but ideally they'd use the samme consensus code
21:11:07adam3us:tacotime: (davec contd) In fact, working toward solutions which address this fundamental issue is necessary for Bitcoin Core too since it means mistakes there don't lead to undesirable consequences either.  The current approach is "well let's just hope it doesn't happen!" and frankly, that just isn't good enough.
21:11:42adam3us:tacotime: which is not what i wanted to here, as thats outright confused and incorrect
21:13:09adam3us:tacotime: "exists just as much with Bitcoin Core as it does with anything else." the point is we dont even care if its btcd consensus or bitcoin consensus (so long as btcd isnt horribly broken)… the point is making two consensus implementations compatible is *undecidable*
21:13:57adam3us:tacotime: ie actually mathematically hard, bordering on impossible. and he needs to understand that.
21:18:32tacotime:well, yes, it's an impossibility. just as it's impossible to prove that bitcoind might not fork between versions or even the same versioned binary.
21:18:43adam3us:tacotime: anyway i wasnt trying to be hostile (to you nor davec) but encouraged him to listen to meredith patterson on language security as it applies to the fundamental limits of host security (lang sec as in parsing messages and equal processing when thats critical) https://archive.org/details/The_Science_of_Insecurity_
21:19:12tacotime:wumpus: not as of yet, we're retooling the consensus code but afaik we're on our own fork of that.
21:19:16adam3us:tacotime: ie listen to her because she knows what she's talking about and maybe there's some mistaken assumption that bitcoincore is trying to compete with btcd.
21:20:27wumpus:adam3us: it's fine to compete on everything else, but competing one the consensus code just doesn't make sense, we should aim to unify that between all node implementations
21:20:40tacotime:well, i think there'd just previously been a lot of cliquey-ness between conformal and the bitcoin core developers, and before we felt excluded from being considered part of core development. gmaxwell now hangs out on our servers and communicates with us on issues, which is good.
21:20:41adam3us:wumpus: exactly.
21:22:15adam3us:tacotime: a written and public confirmation that they would never encourage anyone to use btcd consensus critical code alone (without being overridden in case of dispute by libbitcoinconsensus) would go some way towards things. cos thats kind of not what davec who says he's the team lead was saying on bitcointalk a day ago
21:22:32tacotime:how does libbitcoinconsensus handle reorgs? or is that considered an implementation specific thing?
21:22:44wumpus:tacotime: it doesn't go that deep yet
21:22:56wumpus:tacotime: as of 0.10 it just verifies transactions
21:23:05tacotime:I think it's a bad idea to handle that in the consensus lib
21:23:10tacotime:because there are a lot of ways to do that
21:23:28tacotime:I think single-chain verification would be ideal
21:23:29adam3us:tacotime: what i said to davec was if he was being constructive he could help work on libbitcoinconsensus for example
21:23:55tacotime:adam3us: well, one of the goals of btcd is a complete reimplementation entire in golang
21:24:06wumpus:tacotime: the idea is to support the API at multiple levels; so the lowest has been implemented now, verifying transactions
21:24:19adam3us:tacotime: its a very very very bad idea to reimplement the consensus part.
21:24:20tacotime:wumpus: ah, ok
21:24:22wumpus:tacotime: the highest level will likely include utxo management of some kind
21:24:53adam3us:tacotime: i suggest he listen to the meredith patterson presentatoin on langsec and then come talk about it.
21:24:57wumpus:tacotime: handling reorganizations may be a step too far, although it needs to be roll-back-able
21:25:03tacotime:adam3us: well, we wanted partially for people to make their own alts with btcd, as experimental coins, though this hasn't really happened yet. in that use case it's not as big an issue.
21:25:39adam3us:tacotime: you can go fork alts all you want as far as i'm concerned. tho there is money on some of them. but if they're using btcd only, that seems fine.
21:25:50wumpus:tacotime: see https://github.com/bitcoin/bitcoin/blob/0.10/doc/release-notes.md#consensus-library
21:27:17tacotime:adam3us: well i can talk to them about external syncing to bitcoind to ensure consensus. there's nothing in there for that yet i think, but it shouldn't be hard to program btcd to call bitcoind's api and make sure something hasn't gone pete tong.
21:27:21tacotime:wumpus: thx
21:28:29tacotime:and i'll forward davec this conversation.
21:28:31adam3us:tacotime: i think its not safe to do otherwise. personally i would think if they are actually going around persuading people to use btcd in isolation it would be justified to protect bitcoin users to issue a warning to bitcoin businesses and to the press. its not a joking thing
21:29:35adam3us:tacotime: seems kind of a shitty thing to say, and i'm not being hostile to you (nor davec etc) but sometimes people just wont listen until they get bad press. eg coinvalidation seemed that way.
21:32:12tacotime:adam3us: well, i mean, soon we're not even using leveldb anymore, so the risk of a fork may yet be real.
21:32:56adam3us:tacotime: you might want to look at what davec is saying on bitcointalk https://bitcointalk.org/index.php?topic=68655.msg9998327#msg9998327
21:33:48wumpus:the specific database used as backend is unlikely to create a fork, although as was shown with berkeleydb it's possible for bugs in the database to create behavior differences
21:34:56justanotheruser:"It has been publicly stated many, many times that it is being coded because the ecosystem severely needs multiple diverse implementations if Bitcoin is ever going to truly flourish to the level we all want it to. " -- do you need to reimplement consensus to have multiple implementations?
21:34:58wumpus:but most likely a fork would be caused by a subtle difference in transaction validation, script interpretation, etc, which is likely if you reimplement in a different language
21:36:25wumpus:justanotheruser: consensus is just not something you can compete on, it's either the same or it's broken
21:37:02adam3us:wumpus: from the langsec presentation i'd imagine thats undecidable and hence extremely complex to avoid small differences. (and those even if tiny or boundary can of course be systematically abused to the detriment of whoever is on the wrong side of the fork)
21:38:23justanotheruser:wumpus: I agree for the most part. If we could somehow prove that one implementation would always evaluate the same as another and it was implementated more efficiently then that may be useful.
21:38:29wumpus:but re: leveldb, it is not a given that bitcoin core will keep using leveldb for all eternity either, though there are no plans to switch either, it works fine and isn't a bottleneck at the moment
21:38:39adam3us:i think the langsec talk is worth listening to, encourage people interested in this topic to watch, i found it altered my opinion for the worse about the hopes of making a compatible implementation when it really mattered. https://archive.org/details/The_Science_of_Insecurity_
21:39:09wumpus:adam3us: agreed
21:39:29tacotime:i would agree that whatever is currently your mainchain needs to evaluate and validate the same across the entire network.
21:39:39wumpus:justanotheruser: you mean to be able to sync the chain faster? that's dominated by secp256k1 verifications anyhow
21:39:41tacotime:by whatever the dominant implementation is.
21:40:26justanotheruser:wumpus: I qualified it with it being more efficient because a less efficient implementation is bad for consensus in a different way
21:40:48kanzure:perhaps it would be useful to insist on byte-by-byte compatibility in the binary that runs consensus code (because otherwise everyone misinterprets it as a "control freak" thing)
21:40:55kanzure:s/everyone/gullible people
21:41:03tacotime:i'm not sure if there's debate at all as to whether that should be c++. there seems to be a lot of issues as to what language is good for writing blockchain consensus things.
21:41:35wumpus:tacotime: there is no 'should be' debate, for better or worse the consensus code is specific c++
21:41:53tacotime:well, if you were to do it over, i mean.
21:42:19wumpus:tacotime: I'd rather have something else too, but the problem of potential incompatibility will exist in whatever language you decide to rewrite it in, that's the problem
21:42:27tacotime:satoshi made the choice for us w/r/t bitcoin.
21:42:30wumpus:tacotime: at least c++ is pretty low-level and can be bound to many other languages
21:42:38tacotime:wumpus: true
21:42:44wumpus:tacotime: would have been worse if it was, say, java
21:42:44tacotime:it's easy to hook into golang
21:43:08justanotheruser:I would bet that there are optimizations to the consensus code that can be done safely, but doing them safely would require a ton of research and auditing.
21:43:10tacotime:i would have reservations with satoshi if he had written it in java
21:43:16kanzure:right, mutual self-incompatibility is already a big enough problem, and then incompatibilities and differences between non-byte-exact implementations is an additional level of headache
21:43:32wumpus:justanotheruser: as I said above, the slowest part of validation is secp256k1, which is well defined
21:44:16justanotheruser:wumpus: are you saying that is a possible optimization?
21:44:25wumpus:justanotheruser: that's where optimization should be concentrated, not the tricky dangerous verification code
21:44:42justanotheruser:well that is part of the consensus code
21:45:48wumpus:justanotheruser: no argument there
21:46:30wumpus:kanzure: requiring it to be byte-exact the same may be a requirement too far, but I don't know
21:48:18kanzure:well, it needs to be something more specific for others to think about other than "they are not us" (i know that's not what you mean, but that's what it will be interpreted as)
21:48:28tacotime:ehm, don't we already have that by the deterministic builds?
21:48:30wumpus:kanzure: I've done some experiments in compiling the consensus code to a neutral, well-defined bytecode architecture, e.g. Moxie, maybe that will make a good immutable definition at some point
21:49:00kanzure:wumpus: i think that squawking about that sort of plan and direction would be much more useful than the alternative things to say to the press
21:49:31kanzure:(i may be misinterpreting what has been proposed as the messaging, for which i apologize, but also i'll bbl)
21:49:36wumpus:kanzure: the rest of the devs don't really agree with me, but I'd be happy to see the consensus code in a separate well-guarded repository, so that it can be easier shared between node implementations, and it's easier to keep track of changes
21:49:53kanzure:no that's not the contentious issue there
21:51:45wumpus:kanzure: alternative things? I'm not talking about anything to the press
21:52:04kanzure:ah i'm confusing a separate conversation from yesterday, forgive me
21:52:10wumpus:kanzure: keeping this thing on the rails is enough work without making a media circus
21:52:25kanzure:yep understood
21:56:41justanotheruser:Is there some btcd thing in the news are something?
22:06:07adam3us:justanotheruser: no i said something offhand on another bitcointalk thread about btcd, and someone who it turns out is the btcd lead davc replied so we were discussing davec reply with tacotime who apparently works with btcd
22:06:44adam3us:justanotheruser: eg https://bitcointalk.org/index.php?topic=68655.msg9998327#msg9998327
22:36:56justanotheruser:oh, thanks
22:37:52adam3us:justanotheruser: i guess if they inadvertenly forked bitcoin, that'd be negatively newsworthy :|
22:39:43justanotheruser:This is why there needs to be some Bitcoin best practices webpage that some of the core developers sign. For media purposes you can say "experts have agreed that doing this is a bad practice..."
22:40:32Taek:wumpus: I like the idea of setting a firm definition for consensus in something like moxie.
22:41:01adam3us:justanotheruser: i thought gmaxwell already spent a lot of time talkign with them, so they actually know. maybe davec was just irritated by the offhand nature of my remark (in an unrelated thread) and so said some not so advisable things in response. but what he said doesnt sound on the face of it like he agrees even yet.
22:41:33adam3us:Taek: i agree. moxie bytecode vm is a cool idea. for that and some other things also.
22:42:21adam3us:Taek: (other things in bitcoin) eg it can enforce deterministic execution of extensions that are coded in its bytecode.
22:42:23justanotheruser:adam3us: yeah, but news media isn't going to cite obscure forum posts and obviously the btcd crew isn't going to change their mind
22:43:37justanotheruser:even if davec backs down, reddit will probably have some quasi-activism to get btcd funded
22:44:16adam3us:justanotheruser: well i dont think it'd be wise to escalate it, but sure they would coindesk , even reddit noise, those rags are printign all kinds of alt-coin technobabble so they're obviously short of content
22:44:35adam3us:justanotheruser: cointelegraph, etc there's 1/2 a dozen of them.
22:45:17adam3us:justanotheruser: i think btcd is already funded, though its not public afaik (not that i tried to find out particularly hard short of asking davec - which he declined to answer)
22:46:02justanotheruser:I'm more concerned about forbes. If they could cite bitcoin.org/bestpractices as saying to not use a consensus reimplementation such as btcd, it is less negative
22:46:28justanotheruser:people would end up thinking "these people did something unsafe" rather than "bitcoin is unsafe"
22:51:53adam3us:justanotheruser: sounds like a good idea to me.
22:52:14adam3us:justanotheruser: could even cite the patterson presentation. probably she has some journal publication on it also.
22:55:49op_mul:justanotheruser: btcd is almost certainly funded.
22:55:59op_mul:they wouldn't go to all that trouble for no reason
22:57:06adam3us:tacotime: said further up that they'd spent a lot of money building btcd (including some user requestsed features) so now they wanted to promote it to people which he thought was fair enough. so that was the start of the discussion of well yes thats fair enough but please not the consensus part
22:57:31phantomcircuit:op_mul, im 90% sure someone working on it told me it was funded, but cant remember who...
22:57:31adam3us:justanotheruser: i meant. just sentence started with tacotime.
22:58:24adam3us:phantomcircuit: its still a bit of a mystery about who or why. seemingly to include user requested features into it and somehow monetize that i guess. but davec seem reluctant to say
22:58:53op_mul:it's an odd choice really.
22:59:03phantomcircuit:it wouldn't be unreasonable to build a server optimized for random access of various bits of the blockchain
22:59:12phantomcircuit:but i cant understand why you would bother with the consensus code
22:59:16op_mul:* op_mul rolls eyes
22:59:40op_mul:phantomcircuit: coinbase did it for you! 300GB of SQL, optimised everywhere. https://bitcoin.toshi.io/
22:59:52phantomcircuit:i got a nice laugh at the 300GB
23:00:05tacotime:good god
23:00:08phantomcircuit:they failed to take advantage of the indexing stuff postgresql can do
23:00:16phantomcircuit:completely and entirely failed
23:00:50tacotime:btcd performs better than that heh. i think we have some of the lowest mem usage amongst implementations iirc
23:01:21phantomcircuit:it's really easy to setup the address stuff as character arrays with indexes that can run efficiently against them
23:01:37phantomcircuit:but instead they have a weird normalized/denormalized structure
23:01:51phantomcircuit:(i presume because doing this right is basically impossible using the rails orm)
23:02:07adam3us:here's what davec said about funding and why on https://bitcointalk.org/index.php?topic=68655.msg9998327#msg9998327
23:02:11gmaxwell:op_mul: it's addressing a different application space; perhaps not a well advised one, but at least it's not using a ton of space because it straight up stupid.
23:02:29adam3us:phantomcircuit: "First, both of these are completely irrelevant.  No one knows who Satoshi was/is, yet that isn't a problem for Bitcoin Core.  So, even if every single contributor were unknown (they're not as you can find the names of every contributor on github), why would it be a problem for btcd when it isn't for Bitcoin Core?"
23:03:04adam3us:phantomcircuit: oh the why was separate "It has been publicly stated many, many times that it is being coded because the ecosystem severely needs multiple diverse implementations if Bitcoin is ever going to truly flourish to the level we all want it to.  ...
23:03:10adam3us:phantomcircuit: … Thinking that a single implementation controlled by a handful of people is a good scheme for a system that is supposed to take over the World's monetary supply is just not very forward thinking.  This has been covered ad naseum."
23:04:16adam3us:phantomcircuit: the first comment was "a) no one knows who's funding them" and the second one was "c) no one knows why they are coding it;" so you can see those are pretty much NON answers.
23:04:49phantomcircuit:which kind of makes sense... but only from a very very high level and only if restricted to being used only be people willing to accept a huge risk to themselves and who are incapable of projecting that risk onto others (ie miners)
23:05:40adam3us:phantomcircuit: exactly. my response was "about why consensus is hard, and more so for two different implementations, here are two presentations first on language security as it applies to host security and software compatibility and second on bitcoin consensus.  The first is quite eye opening even for security researchers (ie people who are experts in host security)."
23:05:54op_mul:gmaxwell: I don't really follow what, though. if it's meant to be the backend for services (for whatever reason they're not using a normal wallet), wouldn't just a handful of address based index make more sense?
23:05:54adam3us:phantomcircuit: language security https://archive.org/details/The_Science_of_Insecurity_
23:05:54adam3us:bitcoin consensus programming https://www.youtube.com/watch?v=on5ySFK0aoY#t=39m50s
23:06:16adam3us:phantomcircuit: "Have a listen, particularly to the first one and then we can discuss further (here or PM or bitcoin-dev or #bitcoin-wizards or whereever).  I am genuinely interested to help resolve the discrepancy of opinion around this, as its actually to my mind significantly misunderstood, and dangerous for bitcoin."
23:06:39op_mul:gmaxwell: if you're trying to do stats or something with it, you can do it with no additional storage at all with a block chain scanner.
23:07:05adam3us:btw the second link if you rewind it is an hour of BlueMatt presenting on bitcoin in general. at 39m50s he dives into "please dont reimplement consensus its a really, really bad idea" sub-topic
23:07:26gmaxwell:op_mul: the problem arises when you won't or can't make a clean boundary about what data is presented, and so you may have to efficiently service ramdom access to anything; and there is no compact way to index that.
23:07:32BlueMatt:adam3us: that talk is wayyyy below people in this room, I think
23:08:25adam3us:BlueMatt: arf, i thought it was pretty good. btw did the pebble guy tell you anything afterwards? (guy who jumped in with obscure consensus algorithm comment at one point)
23:09:20adam3us:BlueMatt: (lurkers might find it educational anyway.)
23:10:41lclc:lclc is now known as lclc_bnc
23:19:45midnightmagic:BlueMatt: your description of future-coding for internal redundancy in cooperative agents will benefit from the following result: 1. Blecher PM. On a logical problem. Discrete Mathematics. 1983;43(1):107-110. doi:10.1016/0012-365X(83)90026-2.
23:22:55midnightmagic:http://magicrulesunpaa6.onion/Blecher-1983-On_a_logical_problem.pdf for fulltext
23:25:49midnightmagic:Also note the massive cite tree hanging over that. The other (less pure IMO) results are a fascinating tree of similar results dating back to von Neumann: R. L. Dobrushin and S. I. Ortyukov, Lower bound for the redundancy of self-correcting arrangements of unreliable functional elements, Prob. Inf. Trans. 13 (1977), 59􏰈65.
23:28:01midnightmagic:and von Neumann's, for search a complete cite tree: J. von Neumann, Probabilistic logics and the synthesis of reliable organisms from unreliable components, in ``Automata Studies'' (C. E. Shannon and J. McCarthy, Eds.), pp. 329􏰈378, Princeton Univ. Press, Princeton, NJ, 1956.
23:28:56zer0veritas:zer0veritas has left #bitcoin-wizards
23:31:23midnightmagic:And I can get all the fulltext of this various stuff if you find something and can't find the fulltext wherever.
23:33:24phantomcircuit:adam3us, in general i think having alternative implementations which nobody relies upon for correctness is a good thing, because if done correctly they find a bunch of bugs
23:33:36phantomcircuit:however i dont really see this happening
23:33:52phantomcircuit:mostly just because that's a lot of work with little to no reward
23:33:52adam3us:phantomcircuit: agree. they might find bugs in bitcoind for example.
23:35:28midnightmagic:or they get too busy tracking down and eliminating bugs in their own code because they can't survive artificial memory pressures
23:35:31midnightmagic:* midnightmagic grumbles