--- Log opened Mon Mar 11 00:00:49 2013
18:31  * sipa validates the main chain using his own ECDSA implementation
18:33 < HM> >)
18:33 < HM> are you profiling ?
18:34 < sipa> i'm using -benchmark to measure validation speed
18:35 < sipa> seems my CPU can do some 20k validations per second
18:35 < sipa> but it's only the beginnen of the chain, so not too much parallellism possible yet
18:35 < sipa> oh, validation failed
18:37 < sipa> ok, just a failed signature parsing
18:37 < HM> :}
18:37 < sipa> see, i warned you about bug-for-bug conformance!
18:39 < HM> i haven't even got around to bootstrapping Bitcoin Wallet on my phone yet
18:55 < sipa> anyway, apparently one more usage of non-canonical sigs that even exists in the chain that i wasn't aware of that it was allowed
18:57 < HM> how different can they be?
18:58 < sipa> the 'errors' i knew about: negative R or S values (they're just interpreted as unsigned), values with excessive 0 padding in front
18:59 < sipa> the one in block 135105 that i didn't know: an extra 0 byte at the end, without increasing the length descriptors
18:59 < sipa> so just a valid sig, with a 0 bytes appended to it
18:59 < HM> right, well ignoring length is definitely a bug
19:00 < HM> how do signed values even work
19:00 < sipa> they're stored as 2's complement
19:00 < sipa> so if the highest bit of the first byte is set, it's a negative vale
19:00 < sipa> but OpenSSL just parses everything as unsigned
19:02 < HM> sure, but how did these transactions get accepted ?
19:02 < sipa> because OpenSSL parses everything as unsigned :)
19:02 < sipa> and every bitcoin full node has used OpenSSL to parse signatures
19:03 < sipa> that's what i mean with bug-by-bug conformance: every full node must mimic all 'errors' that OpenSSL allows, and no others
19:03 < HM> I don't follow, if they're always interpreted as unsigned then the first 1 bit means nothing
19:04 < sipa> imagine you want to store the value 0x9999
19:04 < HM> 0b10011001100110011001
19:04 < sipa> so the positive integer 39321
19:04 < sipa> the correct DER encoding is 0x009999
19:05 < HM> oh, I'm not familiar with DER
19:05 < sipa> as 0x9999 is interpreted as -26215
19:05 < HM> seemed like crufty nonsense to me
19:05 < sipa> well, that's the standard, and it's crufty indeed, but it's sane
19:06 < sipa> the problem is, because OpenSSL knows it expects an unsigned integer, even if you store 0x9999, it will interpret that as 39321 and not as -26215
19:06 < HM> how is sane to encode a perfectly reasonable 2 byte unsigned value in 3 bytes with 1 useless 0x00 byte?
19:06 < sipa> because it is not an unsigned value
19:06 < sipa> DER doesn't have an unsigned integer type
19:07 < HM> why didn't Bitcoin just use 32 byte unsigned big endian byte types
19:07 < HM> that's pretty straightforward
19:07 < sipa> because Satoshi just used OpenSSL to encode/decode pubkeys/sigs
19:07 < sipa> and probably knew nothing about the encoding itself
19:09 < sipa> anyway: bottom line: every implementation _must_ accept 0x9999 as 39321, even though a standards-compliant DER parser would interpret that as a negative number, which would cause ECDSA to reject the signature as out of range
19:10 < HM> don't you mean it must accept 0x00 0x99 0x99
19:10 < sipa> it must accept both
19:10 < sipa> as OpenSSL accepts both
19:11 < HM> OpenSSL is broken
19:11 < sipa> it is not - it is a tolerant parser
19:11 < sipa> which tries to accept anything that makes sense
19:11 < HM> Why does it use DER if it interprets it as unsigned and DER doesn't have an unsigned type?
19:12 < sipa> well in a way it makes sense: "ok you give me this signature *parse* ok, syntactically correct. wait... this R value is negative? i don't expect a negative number here... let's assume you just missed a 0 byte in front"
19:13 < sipa> the only problem is that bitcoin passes signatures directly to OpenSSL
19:13 < HM> ick
19:13 < sipa> and thus made OpenSSL's implementation an implicit hardforking network rule
19:13 < HM> so basically you have to read the byte string and if it's not already zero padded, add a 0x00 byte
19:14 < sipa> by the way: we did use this 'oversight' to our advantage as well: it allowed a completely backward-compatible implementation of compressed pubkeys
19:14 < sipa> as every old nodes silently already accepted compressed pubkeys
19:15 < HM> using one oversight to correct another :P
19:15 < jgarzik> speaking of (somewhat)...  petertodd already used 0x00 + 32 bytes for information purposes, so I think we should just do OP_DROP-as-standard
19:15 < jgarzik> i.e. purposefully invalid pubkey
19:15 < HM> compressed public keys made sense from the start
19:16 < sipa> jgarzik: i really dislike making something like that standard, but i have no problem with instant-pruning obviously-unspendable outputs
19:17 < jgarzik> interesting
19:17 < sipa> which was what he was asking for, i think
19:17 < jgarzik> a bit more limited than OP_DROP, but should suffice for per-transaction information purposes
19:17 < sipa> (make scriptPubKeys that start with an OP_RETURN insta-pruned)
19:18 < jgarzik> yeah
19:18 < sipa> 195k blocks verified
19:20 < HM> sipa: how is that AST idea for scripting going down?
19:20 < HM> along with P2SH
19:20 < sipa> HM: it exists
19:20 < sipa> in the abstract mathematical sense :p
19:21 < HM> anyone slapped together a grammar?
19:21 < sipa> roconnor and i have worked on something like that for a while about a year ago
19:22 < HM> any public documents?
19:22 < sipa> dunno
19:24 < sipa> we certainly never got to a point of defining a serialization, so i guess that'd count as 'no' to your question
19:26 < sipa> hmm, problem with such fast script verification: i can't get my parallel signature checking to use more than ~3 cores
19:28 < HM> i almost feel a register based script engine would suffice.
19:29 < HM> although long lists of pubkeys and such pose a problem
19:29 < HM> less flexible to expand as well
19:30 < sipa> 206k
20:02 < sipa> done!
21:06 < amiller> i'm learning to do proper semantics now
21:06 < amiller> wiht things that look like this: Γ ⊢₂ e : ⟪τ⟫ → ⟨τ⟩ + ⊤
21:07 < amiller> so i'm sure i'll have this sorted in no time
21:49 < gavinandresen> all righty…. who has done what?  I sent an alert to 0.8 nodes and tweeted, pointing to sipa's post on bitcointalk
21:49 < gavinandresen> gmaxwell: you sent email to the infrastructure list?
21:49 <@gmaxwell> Yes.
21:50 <@gmaxwell> Sorry it wasn't a great email, but I checked and one hadn't been sent— fast seemed better than perfect.
21:50 < gavinandresen> fast is good
21:51 < gavinandresen> I'm going to kiss the kids goodnight, and stop and take a breath.  We'll need a web page explaining to people what happened / what is happening; either on bitcoin.org or I could post to the foundation blog
--- Log opened Mon Mar 11 22:50:36 2013
--- Log closed Tue Mar 12 00:00:39 2013