1 2016-07-11 01:08:13	0|GitHub125|[13bitcoin] 15pstratem closed pull request #7940: [WIP] Fuzzing framework (06master...062016-04-20-fuzzing-framework) 02https://github.com/bitcoin/bitcoin/pull/7940
  2 2016-07-11 01:08:58	0|GitHub119|[13bitcoin] 15pstratem closed pull request #8277: Refactor CBlockHeaderAndShortTxIDs::GetShortID into CTransaction (06master...062016-06-26-ctransaction-getshortid) 02https://github.com/bitcoin/bitcoin/pull/8277
  3 2016-07-11 09:03:25	0|GitHub82|13bitcoin/06master 141ba3db6 15Christian von Roques: bash-completion: Adapt for 0.12 and 0.13...
  4 2016-07-11 09:03:25	0|GitHub82|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/67caef673089...26316ffa7dc5
  5 2016-07-11 09:03:26	0|GitHub82|13bitcoin/06master 1426316ff 15Wladimir J. van der Laan: Merge #8289: bash-completion: Adapt for 0.12 and 0.13...
  6 2016-07-11 09:03:35	0|GitHub140|[13bitcoin] 15laanwj closed pull request #8289: bash-completion: Adapt for 0.12 and 0.13 (06master...06completion) 02https://github.com/bitcoin/bitcoin/pull/8289
  7 2016-07-11 09:10:50	0|assder|When is the first release candidate for 0.13 expected?
  8 2016-07-11 09:13:49	0|gmaxwell|"Soon"
  9 2016-07-11 09:53:40	0|GitHub143|[13bitcoin] 15laanwj pushed 3 new commits to 060.12: 02https://github.com/bitcoin/bitcoin/compare/080457c4ee97...5c8438207661
 10 2016-07-11 09:53:40	0|GitHub78|[13bitcoin] 15laanwj closed pull request #8318: [0.12] Backport "Rename OP_NOP3 to OP_CHECKSEQUENCEVERIFY #7540" (060.12...06rename_nop3_0.12) 02https://github.com/bitcoin/bitcoin/pull/8318
 11 2016-07-11 09:53:41	0|GitHub143|13bitcoin/060.12 14ac5577b 15BtcDrak: Rename OP_NOP3 to OP_CHECKSEQUENCEVERIFY
 12 2016-07-11 09:53:41	0|GitHub143|13bitcoin/060.12 14c4e5688 15BtcDrak: Rename NOP3 to CHECSEQUENCEVERIFY in rpc tests
 13 2016-07-11 09:53:42	0|GitHub143|13bitcoin/060.12 145c84382 15Wladimir J. van der Laan: Merge #8318: [0.12] Backport "Rename OP_NOP3 to OP_CHECKSEQUENCEVERIFY #7540"...
 14 2016-07-11 10:29:25	0|GitHub157|13bitcoin/060.12 14fe98533 15Jonas Schnelli: [Qt] Disable some menu items during splashscreen/verification state...
 15 2016-07-11 10:29:25	0|GitHub157|[13bitcoin] 15laanwj pushed 2 new commits to 060.12: 02https://github.com/bitcoin/bitcoin/compare/5c8438207661...1233cb42dde9
 16 2016-07-11 10:29:25	0|GitHub27|[13bitcoin] 15laanwj closed pull request #8302: 0.12.2: [Qt] Disable some menu items during splashscreen/verification state (060.12...06Mf1607-012qtDebugSplash) 02https://github.com/bitcoin/bitcoin/pull/8302
 17 2016-07-11 10:29:26	0|GitHub157|13bitcoin/060.12 141233cb4 15Wladimir J. van der Laan: Merge #8302: 0.12.2: [Qt] Disable some menu items during splashscreen/verification state...
 18 2016-07-11 10:51:35	0|GitHub118|13bitcoin/06master 14477777f 15MarcoFalke: [rpcwallet] Don't use floating point
 19 2016-07-11 10:51:35	0|GitHub118|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/26316ffa7dc5...304eff3c614a
 20 2016-07-11 10:51:36	0|GitHub118|13bitcoin/06master 14304eff3 15Wladimir J. van der Laan: Merge #8317: [rpcwallet] Don't use floating point...
 21 2016-07-11 10:51:45	0|GitHub31|[13bitcoin] 15laanwj closed pull request #8317: [rpcwallet] Don't use floating point (06master...06Mf1607-rpcFloat) 02https://github.com/bitcoin/bitcoin/pull/8317
 22 2016-07-11 11:58:46	0|MarcoFalke|Who guessed it
 23 2016-07-11 11:58:55	0|sipa|?
 24 2016-07-11 11:58:56	0|MarcoFalke|Of course windows test are timing out, ow
 25 2016-07-11 11:59:41	0|MarcoFalke|I am not sure why they don't work in parallel "out of the box"
 26 2016-07-11 11:59:58	0|MarcoFalke|Maybe wine can't handle it?
 27 2016-07-11 12:00:00	0|sipa|reduce their parallelism?
 28 2016-07-11 12:01:41	0|MarcoFalke|There is already an overhead of about 2 for just using wine. Then, disabling parallel is another slowdown by 4.
 29 2016-07-11 12:01:51	0|MarcoFalke|So about 8 times longer rpc test for windows
 30 2016-07-11 12:01:58	0|sipa|fun.
 31 2016-07-11 12:06:26	0|luke-jr|what? why would WINE have any overhead?
 32 2016-07-11 13:10:06	0|wumpus|luke-jr: that's not really clear, I'd expect an overhead at startup/shutdown due to setting up the 'virtual windows environment', but not on networking
 33 2016-07-11 13:12:04	0|GitHub175|[13bitcoin] 15jtimon opened pull request #8328: Consensus: Rename: Move consensus code to the consensus directory (06master...060.12.99-consensus-rename) 02https://github.com/bitcoin/bitcoin/pull/8328
 34 2016-07-11 13:14:38	0|jtimon|ping kanzure ^^
 35 2016-07-11 13:15:01	0|jtimon|quite trivial to review, just touches the makefile and a bunch of includes
 36 2016-07-11 13:16:02	0|kanzure|thanks for the ping
 37 2016-07-11 13:16:20	0|jtimon|no problem, thanks for asking for it
 38 2016-07-11 13:42:25	0|jtimon|sipa: what's the advantage of validating this https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp#L3558 in ContextualCheckBlock() instead of ConnectBlock() ?
 39 2016-07-11 13:42:59	0|jtimon|and the same question kind of applies to the other 2 consensus checks in there
 40 2016-07-11 13:43:43	0|jtimon|in other words would it be crazy to move all those checks to just connectblock? why?
 41 2016-07-11 13:45:11	0|jtimon|or in other words, is it necessary or advantageous to call ContextualCheckBlock() in other places other than ConnectBlock() in some way I'm missing?
 42 2016-07-11 13:45:26	0|jtimon|answers from anyone welcomed
 43 2016-07-11 13:48:11	0|sipa|jtimon: contextualcheckblock runs before storing a block on disk, connectblock after
 44 2016-07-11 13:48:46	0|sipa|so everything that can be in contextualcheckblock should be, as it make attacks that cause disk space wasting harder
 45 2016-07-11 13:49:08	0|jtimon|I see, so the advantage is to discard offending blocks before storing them on disk for checks that are relatively cheap
 46 2016-07-11 13:49:29	0|jtimon|I didn't knew we stored invalid blocks at all
 47 2016-07-11 13:49:48	0|jtimon|why do we store invalid blocks?
 48 2016-07-11 13:51:07	0|sipa|because we can't know until we reorg the chainstate
 49 2016-07-11 13:51:27	0|sipa|we may accept a block on a side branch that does not overtake the best chai  yet
 50 2016-07-11 13:51:32	0|sipa|*chain
 51 2016-07-11 13:51:55	0|jtimon|but it's still valid, no?
 52 2016-07-11 13:52:10	0|sipa|we can't know that without reorg
 53 2016-07-11 13:52:22	0|sipa|because we need the utxo set for its parent
 54 2016-07-11 13:52:28	0|jtimon|we can know whether a block is valid in its chain without knowing whether it belongs to the "longest" chain or not, right?
 55 2016-07-11 13:52:39	0|sipa|in theory, yes
 56 2016-07-11 13:52:43	0|sipa|in practice, no
 57 2016-07-11 13:52:55	0|sipa|because we don't have its utxoset we can't validate
 58 2016-07-11 13:53:19	0|jtimon|oh, I see, we never build incompatible utxo histories
 59 2016-07-11 13:53:39	0|sipa|right, we just have a single utxo set for the tip
 60 2016-07-11 13:54:04	0|sipa|and delay full validation until we're forced to switch tips
 61 2016-07-11 13:54:15	0|jtimon|I thought that was part of the CCoinsViewCache purpose...
 62 2016-07-11 13:55:15	0|jtimon|but I guess yeah, it doesn't make sense to fully validate until you're forced to reorg by the accumulated pow, storing that block is cheaper I guess
 63 2016-07-11 13:55:19	0|sipa|no, that's just so we don't have to write everything to disk immediately
 64 2016-07-11 13:55:42	0|sipa|now, pow is the most expensive thing for an attacker
 65 2016-07-11 13:56:03	0|sipa|so everything that comes after pow validation and corruption checks does not matter very much
 66 2016-07-11 13:56:10	0|sipa|why are you asking, btw?
 67 2016-07-11 13:56:17	0|sipa|ah, i know
 68 2016-07-11 13:56:27	0|sipa|iswitnessenabled depends on versionbits
 69 2016-07-11 13:56:38	0|jtimon|mhmm, why do we validate any part of a block that we don't see as belonging to the longest chain at all?
 70 2016-07-11 13:57:28	0|sipa|because we at least need to know whether it is actually the right block
 71 2016-07-11 13:57:42	0|sipa|a peer could change one transaction before relay
 72 2016-07-11 13:57:43	0|jtimon|sipa: it was part of my latest consensus refactor branch to get rid of ContextualCheckBlock()
 73 2016-07-11 13:58:07	0|sipa|and in that case we must punish the peer, but not mark the block as invalid
 74 2016-07-11 13:58:33	0|sipa|so an invariant is that all blocks on disk are known to actually match the blocks whose headers we know
 75 2016-07-11 13:59:42	0|jtimon|if we know it's not making a longer chain, why bother validating BIP113 ?
 76 2016-07-11 14:00:15	0|sipa|validating bip113 is not necessary i think
 77 2016-07-11 14:00:31	0|sipa|but the segwit commitment check is
 78 2016-07-11 14:01:10	0|sipa|before storing on disk
 79 2016-07-11 14:03:42	0|jtimon|that's useful, what about the bip34 check in ContextualCheckBlock(), can it be moved out?
 80 2016-07-11 14:04:02	0|sipa|i believe so
 81 2016-07-11 14:04:13	0|jtimon|ok, thanks
 82 2016-07-11 14:04:31	0|jtimon|but not the new segwit one, fine
 83 2016-07-11 14:06:38	0|jtimon|well, I would prefer to pass the consensus validation flags instead of calling IsWitnessEnabled() from there
 84 2016-07-11 14:08:41	0|jtimon|but I should try to write that first before discussing it further
 85 2016-07-11 15:24:18	0|sdaftuar|reviewers, please review some open PRs for 0.13.0: #8295, #8305, #8312
 86 2016-07-11 16:34:27	0|BTCking|AMAZING! My bitcoin-grower is now ready. Send me some bitcoin (1btc, 2btc, etc.) and get MUCH more back INSTANTLY. This uses Block-chain explanding technology! Proven safe & secure. You cannot lose. PM me to begin!!
 87 2016-07-11 16:35:35	0|sipa|BTCking: fuck off
 88 2016-07-11 16:35:50	0|BTCking|AMAZING! My bitcoin-grower is now ready. Send me some bitcoin (1btc, 2btc, etc.) and get MUCH more back INSTANTLY. This uses Block-chain explanding technology! Proven safe & secure. You cannot lose. PM me to begin!!
 89 2016-07-11 16:47:08	0|GitHub65|[13bitcoin] 15jtimon opened pull request #8329: Consensus: MOVEONLY: Move functions for tx verification (06master...060.12.99-consensus-moveonly-tx) 02https://github.com/bitcoin/bitcoin/pull/8329
 90 2016-07-11 16:49:04	0|jtimon|kanzure: ping again, still quite simple to review and re-write
 91 2016-07-11 18:10:32	0|sipa|sdaftuar: done
 92 2016-07-11 18:10:40	0|sdaftuar|sipa: thanks!
 93 2016-07-11 18:13:46	0|sipa|github.com down?
 94 2016-07-11 18:15:02	0|eragmus|sipa: http://www.isup.me/github.com
 95 2016-07-11 18:15:40	0|eragmus|http://isitdownorjust.me/github-com/
 96 2016-07-11 18:16:04	0|sipa|thanks
 97 2016-07-11 18:16:08	0|sipa|it also works from my vps...
 98 2016-07-11 18:16:41	0|tadasv|working fine for me
 99 2016-07-11 18:33:29	0|gmaxwell|sdaftuar: maybe it doesn't matter because its a corner case, but #8305 makes it look like it will effectively discard the dangling header-- it pushmessages a getheaders and returns, without AcceptBlockHeader or UpdateBlockAvailability.
100 2016-07-11 18:33:52	0|sdaftuar|oh, UpdateBlocKAvailability, hmm
101 2016-07-11 18:34:39	0|gmaxwell|Is the expectation that it'll come back in via the getheaders, and that the remote end won't do anything smart like not send it because it already did?  (I guess thats reasonable?) though the lack of updateavailablity probably means we'll send a redundant announcement in that direction.
102 2016-07-11 18:35:02	0|sdaftuar|well there's no way to avoid getting the headers message back to you
103 2016-07-11 18:35:17	0|sdaftuar|if you set hashstop equal to the hash of that header, it'll still appear in the response
104 2016-07-11 18:35:41	0|sdaftuar|i don't think the remote end is allowed to not re-send, though i suppose lack of a spec makes this a difficult statement to concretely make
105 2016-07-11 18:35:59	0|sdaftuar|but we expect the contents of a headers message to be a linear connecting chain
106 2016-07-11 18:36:14	0|gmaxwell|K. That makes sense, I thought that was the assumption but wanted to be sure (I think it's correct.)
107 2016-07-11 18:37:38	0|sdaftuar|i guess there could be an advantage to setting hashLastUnknownBlock if a header doesn't connect... that would mean that if a different peer supplied the missing headers, we could download the block from the peer who made the announcement
108 2016-07-11 18:37:52	0|sdaftuar|but that would only matter if the peer didn't respond to the getheaders being pushed in this patch
109 2016-07-11 18:38:05	0|gmaxwell|so now let me ask, if I send you a header that doesn't connect because of timestamp stupidity. And I'm your only peer. If there are two blocks in a row during the time when the prior header will be rejected... When will it start connecting again?  I'll get the flag false on me, then it'll never be sent to true, since all further headers I send will also not connect.
110 2016-07-11 18:38:07	0|sdaftuar|which is a weird corner case
111 2016-07-11 18:38:38	0|gmaxwell|On the update front, I was mostly think in terms of not sending surpflous compact block messages.
112 2016-07-11 18:38:57	0|sdaftuar|when you respond to the getheaders, the bit is cleared
113 2016-07-11 18:39:30	0|midnightmagic|;;isitdown github.com
114 2016-07-11 18:39:31	0|gribble|github.com is up
115 2016-07-11 18:40:05	0|sdaftuar|and the headers will start connecting again
116 2016-07-11 18:40:09	0|sdaftuar|well,
117 2016-07-11 18:40:12	0|sdaftuar|unless the time stamp is still broken
118 2016-07-11 18:40:45	0|gmaxwell|hm. I was thinking that it wouldn't be set to true if the getheaders reply also didn't connect due to bad timestamps.
119 2016-07-11 18:41:26	0|xinxi|Hi guys
120 2016-07-11 18:41:43	0|xinxi|Anybody here?
121 2016-07-11 18:42:11	0|sdaftuar|gmaxwell: yeah, i can't solve the problem of a bunch of blocks in a row all being invalid to one peer but not another
122 2016-07-11 18:42:23	0|sdaftuar|but typically, the first block is too far ahead
123 2016-07-11 18:42:37	0|sdaftuar|and then by the time the next block comes in, you would be able to accept both
124 2016-07-11 18:42:46	0|sdaftuar|if you'd be willing to try
125 2016-07-11 18:42:51	0|sdaftuar|but we have no "try" mechanism currently
126 2016-07-11 18:43:40	0|sdaftuar|i guess the question is, what should we do if a peer is relaying many blocks in a row that are all actually too far in the future?
127 2016-07-11 18:43:52	0|sdaftuar|i just fall back to the current behavior of failing/disconnecting
128 2016-07-11 18:45:09	0|sdaftuar|i did briefly think about whether it would be feasible to somehow accept headers that are too far in the future, but i'm pretty sure that's a major rewrite to our consensus logic
129 2016-07-11 18:45:27	0|sdaftuar|(accept them, so that you could re-try them later)
130 2016-07-11 18:50:16	0|sdaftuar|gmaxwell: perhaps we could always send a getheaders if the peer is whitelisted?
131 2016-07-11 18:52:33	0|gmaxwell|I don't think that would be an awful thing, but I don't think it addresses the concern-- you shouldn't only fail to fail if you have whitelisted peers.
132 2016-07-11 18:53:26	0|sdaftuar|i guess i could do the thing you imply -- actually just check to see if the headers connect, and use that to re-set the bit
133 2016-07-11 18:55:04	0|gmaxwell|maybe I'm misunderstanding, but that situation I'm thinking about is if you're slow on time by a few seconds, rejected a block, then the next comes in, and it still doesn't connect... you're going to stop sending getheaders after that. Then no others will connect from that peer.  The time is a bit freaky, since it's a temporary failure.
134 2016-07-11 18:55:34	0|sdaftuar|yes, i agree your example is a correct one
135 2016-07-11 18:55:54	0|sdaftuar|i'm just trying to distinguish that from an example where an attacker sends you a chain that's far into the future
136 2016-07-11 18:56:59	0|gmaxwell|I think we award a little bit of misbehaving there. Ironically, for sync purposes it might be better to immediately disconnect: at least that will clear the flag! :)
137 2016-07-11 18:58:35	0|gmaxwell|relying on DOS disconnect to make progress is weird though, but I think thats the answer here: after some number non-connecting blocks I think you'll disconnect the peer.  When you reconnect, the flag will be true and you'll try to sync again.
138 2016-07-11 18:59:37	0|gmaxwell|sdaftuar: to prevent looping, why not store the most recent ID. And don't getheader again if it's the same?
139 2016-07-11 19:00:10	0|gmaxwell|e.g. block announce, doesn't connect, store the most recent ID and getheader. Get headers, still doesn't connect and most recent ID is the same. Stop.
140 2016-07-11 19:00:40	0|sdaftuar|sorry, what is most recent ID?
141 2016-07-11 19:01:21	0|gmaxwell|when the headers response comes in take the ID (hash) of the latest block in the reply. And do not keep requesting headers when that value hasn't changed for this peer.
142 2016-07-11 19:02:05	0|gmaxwell|I announce X, X doesn't connect. Remember X, request headers.  Headers come in, they tell you a bunch of junk ending at X. Still doesn't connect. Stop.
143 2016-07-11 19:02:39	0|sdaftuar|gmaxwell: i guess i just worry about assuming too much about the broken implementations out there
144 2016-07-11 19:02:55	0|sdaftuar|eg maybe the peer who was sending me 160kb of junk was sending me random junk
145 2016-07-11 19:04:07	0|gmaxwell|Well thats what misbehaving is for.
146 2016-07-11 19:04:33	0|sdaftuar|but in your example, wouldn't that behavior result in infinite loop?
147 2016-07-11 19:05:05	0|gmaxwell|The headers reply would end with the block X, which we already had remembered, so we would stop there.
148 2016-07-11 19:05:21	0|gmaxwell|no loop. then they later announce block y, and we'd getheaders again. and hopefully connect.
149 2016-07-11 19:05:33	0|gmaxwell|So basically  one headers retry per novel header they send.
150 2016-07-11 19:05:45	0|sdaftuar|i'm specifically referring to peers that are totally broken and on initial getheaders sync, they send us 2000 headers that don't connect
151 2016-07-11 19:06:40	0|sdaftuar|i don't want to assume too much about how that peer is doing that
152 2016-07-11 19:06:51	0|gmaxwell|sure, whatever, testnet node on mainnet or whatnot.
153 2016-07-11 19:06:59	0|sdaftuar|if getting it wrong means 160kb over and over again...
154 2016-07-11 19:07:33	0|gmaxwell|just don't assume it's malicious, since there are better ways to waste our bandwidth for malicious peers.
155 2016-07-11 19:08:15	0|sdaftuar|well you're suggesting we assume it's deterministic i think.  which is hopefully true, but if it's not, then sadness
156 2016-07-11 19:08:46	0|gmaxwell|yes, indeed, and actually the 2000 limit gets in the way too.
157 2016-07-11 19:09:16	0|sdaftuar|i mean, i think in practice your suggestion probably works, i was just hoping for something more robust to accidental disaster
158 2016-07-11 19:10:25	0|gmaxwell|speaking of 2000, I'm not seeing how what you suggest doesn't actually break sync.
159 2016-07-11 19:10:37	0|sipa|can i get a summary of the problem?
160 2016-07-11 19:11:20	0|gmaxwell|oh nevermind last comment, I was mistaking the order.
161 2016-07-11 19:11:32	0|sdaftuar|sipa: i noticed on testnet that i had a node that would fall out of sync, because its one peer would occasionally send a block header (using headers announcements) that was too far in the future
162 2016-07-11 19:11:39	0|gmaxwell|(I was thinking that we'd request headers, if there was more than 2000 missing, that they sent the later headers first)
163 2016-07-11 19:11:46	0|sdaftuar|so the peer thought the header was valid, but the local node would reject
164 2016-07-11 19:11:57	0|sdaftuar|this is a case i had never thought of when working on the headers announcements stuff
165 2016-07-11 19:12:21	0|sdaftuar|since the announcing peer doesn't realize the recipient rejects the header, they continue to announce blocks on top of the header
166 2016-07-11 19:12:36	0|sipa|sdaftuar: yes, i get that
167 2016-07-11 19:12:44	0|sdaftuar|resulting in headers announcements that don't connect for the recipient, and then eventually disconnect
168 2016-07-11 19:12:44	0|sipa|but you have a patch for that
169 2016-07-11 19:13:06	0|xinxi|hey guys, I want to make Bitcoin provably correct.
170 2016-07-11 19:13:08	0|sdaftuar|oh right.  so gmaxwell wonders if we can avoid failure in the case where two blocks come in quickly, where the first block is still invalid when the second one is announced
171 2016-07-11 19:13:18	0|sdaftuar|my patch doesn't address that
172 2016-07-11 19:13:19	0|xinxi|I am serious.
173 2016-07-11 19:13:32	0|sipa|xinxi: you'll need to be a bit more specific
174 2016-07-11 19:13:56	0|sipa|what part do you want to prove?
175 2016-07-11 19:14:02	0|gmaxwell|sipa: and sdaftuar is concerned that relaxations will leave it in a state where its constantly requesting headers over and over again.
176 2016-07-11 19:14:18	0|xinxi|sipa: thank you for your attention. Yeah, basically, I think bugs are still in the C++ implementation. And I want to make sure it's absolutely correct.
177 2016-07-11 19:14:27	0|xinxi|so there will be no bugs.
178 2016-07-11 19:14:45	0|sipa|xinxi: that implies you have a specification for correct to compare to
179 2016-07-11 19:14:47	0|gmaxwell|I think this could be addressed by just adding a small amount of DOS each time.
180 2016-07-11 19:15:10	0|gmaxwell|Correct with respect to a specification may well be meaningless if the specification is not "correct" in a broader sense.
181 2016-07-11 19:15:22	0|xinxi|sipa: yeah, I want to work out a formal specification as well.
182 2016-07-11 19:15:27	0|sipa|xinxi: for the consensus logic, the code implementation is the specification, so it is by definition correct
183 2016-07-11 19:15:38	0|xinxi|something like mathematical theorems about the code.
184 2016-07-11 19:15:56	0|sipa|xinxi: however, being able to verify that the behaviour of that code does not change between releases would be incredibly valuable
185 2016-07-11 19:16:38	0|sipa|unfortunately, due to historical reasons mostly, the consensus code is not well separated from the rest
186 2016-07-11 19:16:40	0|gmaxwell|sdaftuar: e.g. I think we could just trigger a getheaders on any header message with only a single header in it that doesn't connect, and add a small dos penalty.
187 2016-07-11 19:16:44	0|xinxi|sipa: I understand what you mean. The code, as a specification, may not be consistent with itself.
188 2016-07-11 19:16:52	0|gmaxwell|sdaftuar: then every new block that shows up on the network will cause us to retry to connect.
189 2016-07-11 19:16:54	0|morcos|i know this sounds kind of janky, but it seems to me that a lot of this sync logic is stuff that to a human is kind of obviously stupid.  i wonder if putting in a bit of belt and suspender check on things getting stuck might make sense.
190 2016-07-11 19:17:26	0|sipa|xinxi: indeed. not consistent with itself, or not consistent with other implementations/versions
191 2016-07-11 19:17:29	0|gmaxwell|sdaftuar: and eventually it will either connect, or disconnect the peer.
192 2016-07-11 19:17:43	0|xinxi|sipa: also, you may not want to treat bugs as part of the specification like the heartbleed bug in OpenSSL should not be part of OpenSSL.
193 2016-07-11 19:17:57	0|sipa|xinxi: we have to
194 2016-07-11 19:17:58	0|btcdrak|xinxi: there is a slow and arduous process of extracting the consensus code into a separate library which will make things a lot easier in the long run.
195 2016-07-11 19:18:11	0|morcos|for instance very 60 secs you clear out some state such as fLastHeadersConnected and perhaps nSyncStarted (maybe only if we're not actively receiving headers)
196 2016-07-11 19:18:20	0|morcos|i'm sure we can imagine other things that kind of get stuck
197 2016-07-11 19:18:24	0|xinxi|btcdrak: I've heard of that. It's libconsensus, isn't it?
198 2016-07-11 19:18:28	0|gmaxwell|morcos: you mean like a periodic, pick a random peer and restart our efforts to sync with it? and loudly logging that the sync state machine has failed if that accomplishes anything?
199 2016-07-11 19:18:38	0|sipa|xinxi: even if we would write down a specification for the behaviour, and every one would agree that that spec is indeed the behaviour we want
200 2016-07-11 19:18:55	0|sipa|xinxi: what if we then find a bug, and the code does something slightly different?
201 2016-07-11 19:18:56	0|sdaftuar|gmaxwell: i think that sounds pretty good, though the DoS score should decay back down once headers that do connect are received, right?  otherwise, over a long enough period of time, you eventually disconnect your peer
202 2016-07-11 19:19:13	0|morcos|gmaxwell: yes something along those lines, i like the idea of loudly announcing it so we try to make stuff work in general, but that next time we miss something, hopefully it kind of autorecovers (like power cycling)
203 2016-07-11 19:19:14	0|btcdrak|xinxi: the problem is it's quite disruptive to the codebase and causes everyone's pull-requests to need rebasing so it tends to get done slowly and in bursts. yes it's called libconsensus
204 2016-07-11 19:19:24	0|gmaxwell|sdaftuar: yea, :( the whole dos score thing is kinda lame.
205 2016-07-11 19:19:26	0|xinxi|btcdrak: but still, that probably reduces bugs in the consensus part, but still it does not guarantee the correctness.
206 2016-07-11 19:19:30	0|sipa|xinxi: the code would have a bug, arguably. but we can't tell everyone to change their code, so it will need to be the document that had to be changed
207 2016-07-11 19:20:00	0|gmaxwell|morcos: I agree with that in principle. We don't however, want to end up with a web of redundant mechenisms, potentially interacting.
208 2016-07-11 19:20:17	0|sipa|xinxi: so we've usually said that a specification for bitcoin would be descriptive but not prescriptive... the laws of the network are those implemented by the code that people choose to run, not by an abstract descriptio
209 2016-07-11 19:20:23	0|morcos|a bigger web you mean?
210 2016-07-11 19:20:28	0|gmaxwell|morcos: hah
211 2016-07-11 19:20:47	0|sdaftuar|unfortunately some of the bugs we've encountered (eg that hasHeaders thing that got reverted) wouldn't be fixed on restart either
212 2016-07-11 19:20:59	0|sipa|for 0.14 i think we need to refactor the sync code
213 2016-07-11 19:21:11	0|sipa|many parallel mechanisms and duplicated code
214 2016-07-11 19:21:17	0|gmaxwell|sdaftuar: you could instead have a simple counter, set to zero whenever you connect, incremented on every non-connecting header response. And dos score only when it hits a threshold to avoid that effect.
215 2016-07-11 19:21:25	0|sipa|no blame anywhere, but we've accumulated too much technical debt
216 2016-07-11 19:21:41	0|sdaftuar|gmaxwell: yep that's what i was thinking too
217 2016-07-11 19:21:49	0|gmaxwell|sdaftuar: I believe thats the first sync stuck bug I've ever seen that wasn't fixed on a restart.
218 2016-07-11 19:22:07	0|sdaftuar|i think i've seen 0.9 nodes not be able to recover from restart
219 2016-07-11 19:22:10	0|xinxi|sipa: we don't have to be like that. I was motivated by the project CompCert and SEL4. SEL4 is a OS kernel that's proved to be 100% consistent with the specification.
220 2016-07-11 19:22:28	0|sipa|xinxi: that implies you have a specification
221 2016-07-11 19:22:37	0|sipa|as i've said before, we can't have one
222 2016-07-11 19:22:47	0|sipa|we can describe the current behaviour
223 2016-07-11 19:22:53	0|gmaxwell|sdaftuar: well there was also the "stops on too many orphans" stuff between ~0.8 and 0.10.
224 2016-07-11 19:23:00	0|sipa|and then formally prove that future code matches that behaviour
225 2016-07-11 19:23:47	0|gmaxwell|(where it would start fetching blocks backwards, run into a 750 block orphan limit then stop, but even that could recover with enough restarts)
226 2016-07-11 19:24:26	0|xinxi|sipa: I just don't understand the factor that stops us from driving a formal specification from the code.
227 2016-07-11 19:24:43	0|gmaxwell|sdaftuar: in any case, I think doing get headers triggered on single header responses, and counting the failures, and DOSing on too many failures... would address both our concerns here.
228 2016-07-11 19:25:05	0|sipa|xinxi: deriving? yes, we could
229 2016-07-11 19:25:13	0|sdaftuar|gmaxwell: sounds right to me as well.  for 0.13.0?
230 2016-07-11 19:25:41	0|sipa|xinxi: i guess my point is that the only way to write a formal specification is by extracting it from the actually implemented and deployed code
231 2016-07-11 19:25:42	0|xinxi|sipa: yeah, that's where we can start from.
232 2016-07-11 19:25:59	0|btcdrak|xinxi: the difficulty is if the code differs from the written specification the code wins. Even if you class it as a bug, changing it is extremely hard because you have to change the network consensus. There are examples where this has happened and we've just had to live with the bug.
233 2016-07-11 19:25:59	0|sipa|xinxi: and not from behaviour that humans write
234 2016-07-11 19:26:09	0|gmaxwell|Yes, I think it wouldn't be more complex than that patch you currently have. (this is also not just a testnet problem-- we're just fortunate that miners have lately not be excessively rolling their timestamps forward...)
235 2016-07-11 19:26:40	0|btcdrak|xinxi: the other problem is unknown bugs which we might not be aware of, rare edge cases say, are actually part of the consensus rules even if we dont know it.
236 2016-07-11 19:26:46	0|sdaftuar|ok i'll see if i can whip up an improved patch
237 2016-07-11 19:26:57	0|gmaxwell|btcdrak: the kind of "formal specification" in the sense of SEL4 can be proven to agree with the code (effectively, the specification is at least as complex as the implementation)
238 2016-07-11 19:26:59	0|btcdrak|xinxi: so it's hard to document things you dont know
239 2016-07-11 19:27:05	0|xinxi|@btcdrak those rare edge bugs can be eliminated by formal proofs I think.
240 2016-07-11 19:27:27	0|btcdrak|xinxi: seems like a very worthwhile pursuit though.
241 2016-07-11 19:28:00	0|xinxi|so the motivation of this is to avoid catastrophic failures of Bitcoin.
242 2016-07-11 19:28:05	0|sipa|xinxi: for example, for a long time, bitcoin relied on OpenSSL's signature parsing code
243 2016-07-11 19:28:15	0|sipa|xinxi: people thought they knew what this code did
244 2016-07-11 19:28:24	0|sipa|xinxi: and reimplemented the logic in other places
245 2016-07-11 19:28:42	0|sipa|but nobody checked that what they thought actually matched what openssl was doing
246 2016-07-11 19:28:46	0|sipa|turns out, it didn't
247 2016-07-11 19:29:15	0|sipa|so now bitcoin's "specification" implicitly depended on OpenSSL's implementation
248 2016-07-11 19:29:30	0|xinxi|sipa: this kind of inconsistencies can be eliminated by formal proofs as well.
249 2016-07-11 19:29:35	0|sipa|xinxi: absolutely
250 2016-07-11 19:29:56	0|sipa|xinxi: but only if they would take OpenSSL's code into account and analyse it, and not treat it as a black box
251 2016-07-11 19:30:05	0|gmaxwell|xinxi: I'm not sure if you have a grasp of the difficulty of what you're thinking about, go look at the size of the SEL4 kernel. I believe it's about 4000 lines of C.  The isabelle specification for it is something like a half million lines, and though it proves many useful things about it, it falls far short of proving so much that agreement with the spec would be equivilent to correct in a hum
252 2016-07-11 19:30:11	0|gmaxwell|an sense.
253 2016-07-11 19:30:16	0|sipa|and this cross-layer effect on consensus makes it IMHO nearly impossible
254 2016-07-11 19:30:24	0|xinxi|gmaxwell: it's about 9000 lines of code.
255 2016-07-11 19:30:28	0|sipa|but i am not an expert on the state of the art of formal code proofs
256 2016-07-11 19:30:55	0|gmaxwell|xinxi: k, my figures are from memory thus approximate.
257 2016-07-11 19:31:31	0|xinxi|gmaxwell: SEL4 has to guarantee high efficiency. So the code has to be written in C which is difficult to prove.
258 2016-07-11 19:32:05	0|gmaxwell|xinxi: yes, and Bitcoin has to achieve high efficiency, efficiency is a security parameter for us.
259 2016-07-11 19:32:07	0|xinxi|For Bitcoin, the programming language is not a big issue, and using languages like Coq, which enables a unified way for both coding and proofs could make it a lot easier.
260 2016-07-11 19:32:12	0|gmaxwell|No. incorrect.
261 2016-07-11 19:32:33	0|gmaxwell|If nodes are not fast enough the system will stop converging, at some point this results in a total consensus failure.
262 2016-07-11 19:32:41	0|xinxi|I think OCaml is quite decent in terms of efficiency. Coq can generate OCaml code.
263 2016-07-11 19:32:59	0|sipa|well we're not going to rewrite the consensus code in another language!
264 2016-07-11 19:33:00	0|gmaxwell|Coq can generate C code via the compcert formalization (see VST).
265 2016-07-11 19:33:15	0|sipa|(or at least not without proving that the existing code matches the new code)
266 2016-07-11 19:33:15	0|xinxi|gmaxwell: here we go.
267 2016-07-11 19:33:44	0|xinxi|how many lines of code are there for the consensus part?
268 2016-07-11 19:34:06	0|xinxi|rewriting the consensus part is not infeasible after you separate it out.
269 2016-07-11 19:34:07	0|sipa|do you include the cryptography library? the c++ standard library? the kernel?
270 2016-07-11 19:34:25	0|gmaxwell|Far more than SEL4, the crypto part of it alone is that big.
271 2016-07-11 19:34:28	0|sipa|changes in any of those can affect consensus
272 2016-07-11 19:34:45	0|sipa|at some point you have to make an abstraction
273 2016-07-11 19:34:58	0|kanzure|"formal proofs" cannot eliminate edge cases, they can only prove that the edge case exists
274 2016-07-11 19:35:00	0|xinxi|sipa: we can leave cryptography part there first. C++ is just hopeless for verification. It's too complicated.
275 2016-07-11 19:35:12	0|sipa|xinxi: then i think there is nothing we can do
276 2016-07-11 19:35:13	0|kanzure|xinxi: how are you migrating the network precisely?
277 2016-07-11 19:35:19	0|kanzure|wtf?
278 2016-07-11 19:35:28	0|gmaxwell|chill chill.
279 2016-07-11 19:35:39	0|sipa|these are intesting things to discuss
280 2016-07-11 19:35:45	0|Chris_Stewart_5|xinxi: I think someone wrote a libsecp256k1 for ocaml if you are serious about implementing
281 2016-07-11 19:35:53	0|Chris_Stewart_5|using ctypes
282 2016-07-11 19:36:04	0|sipa|Chris_Stewart_5: what if the ocaml implementation doesn't match the C one?
283 2016-07-11 19:36:15	0|sipa|we won't know without analysing the one that is actually used
284 2016-07-11 19:36:28	0|sipa|(libsecp256k1, by the way, does have formal proofs for part of its operation)
285 2016-07-11 19:36:34	0|xinxi|I mean we can separate out libconsensus first, and then use Coq to rewrite it and generate a provably correct version of libconsensus.
286 2016-07-11 19:36:49	0|kanzure|xinxi: there are a lot of subtle side effects from the actual running network, but everyone in here still thinks it would be valuable to pursue formal verification anyway
287 2016-07-11 19:37:14	0|sipa|xinxi: how will you prove that the existing libconsensus (which only does script validation, not full block validation) matches that Coq specification?
288 2016-07-11 19:37:15	0|kanzure|libconsensus imho should be a priority similar to segwit
289 2016-07-11 19:37:54	0|sipa|kanzure: i agree. but we first need a plan for what 'libconsensus' means. people have been talking about it for ages, and have meant very different (and conflicting) things with it
290 2016-07-11 19:38:09	0|xinxi|sipa: we don't prove the existing libconsensus. We derive the formal specification based on the existing libconsensus, and then write another libconsensus in Coq.
291 2016-07-11 19:38:12	0|kanzure|sipa: maybe we can hijack the milan conference for these purposes (scaling bitcoin 3)
292 2016-07-11 19:38:15	0|GitHub72|[13bitcoin] 15JeremyRubin opened pull request #8330: WIP: Structure Packing Optimizations in CTransaction and CTxMemPoolEntry (06master...06structurepacking) 02https://github.com/bitcoin/bitcoin/pull/8330
293 2016-07-11 19:38:41	0|Chris_Stewart_5|sipa: It just binds to the C library, so unless there is a bug in ctypes you should be ok
294 2016-07-11 19:38:43	0|sipa|xinxi: does you derivation prove that the specification matches the current libconsensus?
295 2016-07-11 19:39:14	0|sipa|sorry; does you derivation result in a specification that provably captures the current behaviour?
296 2016-07-11 19:39:33	0|xinxi|sipa: no. do they have to match exactly?
297 2016-07-11 19:39:36	0|sipa|Yes.
298 2016-07-11 19:39:38	0|xinxi|we just take off there.
299 2016-07-11 19:39:48	0|kanzure|can coq do outputs re: "behavior is definitely undefined here"?
300 2016-07-11 19:40:03	0|sipa|xinxi: the specification for the network is the actually deployed code
301 2016-07-11 19:40:16	0|sipa|any divergence from it can result in a network fork
302 2016-07-11 19:40:29	0|xinxi|sipa: I think there can be slight difference?
303 2016-07-11 19:40:37	0|sipa|xinxi: such as?
304 2016-07-11 19:40:45	0|sipa|if you can't describe the current code exactly, we're done
305 2016-07-11 19:40:54	0|kanzure|once you announce a difference, soeone will probably try to trigger those circumstances anyway :)
306 2016-07-11 19:41:16	0|sipa|we won't rewrite the consensus layer in another language without certainty that it does not introduce any changes compared to the current code
307 2016-07-11 19:42:15	0|xinxi|have you guys talked about this before?
308 2016-07-11 19:42:20	0|kanzure|endlessly
309 2016-07-11 19:42:20	0|sipa|yes
310 2016-07-11 19:42:34	0|xinxi|many times?
311 2016-07-11 19:42:37	0|sipa|xinxi: i don't think you're understanding what i'm saying
312 2016-07-11 19:42:49	0|sipa|we do not prescribe the rules of the network
313 2016-07-11 19:42:50	0|kanzure|xinxi: there is high interest in our community in formal verification and libconsensus and coq.
314 2016-07-11 19:43:08	0|Chris_Stewart_5|^^^
315 2016-07-11 19:43:09	0|petertodd|sipa: along with the pragmatic requirement that the new language have a sufficiently large community of programmers to be maintainable - unusual languages don't need that criteria
316 2016-07-11 19:43:21	0|sipa|petertodd: agree, but i wasn't going to bring that up
317 2016-07-11 19:43:50	0|sipa|the first point is that our compatibility concerns are way beyond almost any other system
318 2016-07-11 19:44:19	0|xinxi|@petertodd that may not be a problem. Later the code will be proof driven. And the core team writes the specification, and the committers need to prove the correctness of their code, which makes them impossible to make any mistakes.
319 2016-07-11 19:44:23	0|kanzure|xinxi: this seems like something you would like to work on?
320 2016-07-11 19:44:38	0|sipa|xinxi: i'm getting annoyed
321 2016-07-11 19:44:46	0|sipa|xinxi: the core team does not write the specification
322 2016-07-11 19:45:15	0|xinxi|@kanzure yes, it is. And two professors in my school, i.e. National University of Singapore, are willing to work in it as well. They are experts on software verification.
323 2016-07-11 19:45:23	0|petertodd|...and the specification will change over time as new features are soft-forked in, and specification flaws are fixed
324 2016-07-11 19:45:24	0|sipa|xinxi: that's very interesting
325 2016-07-11 19:45:35	0|xinxi|and we may have big funding support.
326 2016-07-11 19:45:57	0|xinxi|the purpose is simple. we want to make bitcoin reliable.
327 2016-07-11 19:46:03	0|sipa|xinxi: but i first want to understand why you say "< xinxi> sipa: I think there can be slight difference?"
328 2016-07-11 19:46:06	0|Chris_Stewart_5|petertodd: Specifications can be changed overtime, and theoretically the proofs should reflect those changes
329 2016-07-11 19:46:14	0|xinxi|the impact could be huge.
330 2016-07-11 19:46:26	0|kanzure|xinxi: there are many bitcoin developers who want that as well. i think you could easily find collaborators from this conversation.
331 2016-07-11 19:46:48	0|petertodd|xinxi: it's not going to be huge - the reliability of the consensus specification/implementation hasn't been a major problem for bitcoin - other problems are far higher-impact (scaling)
332 2016-07-11 19:47:07	0|petertodd|xinxi: I'd suggest you focus your efforts on smart-contract use-cases, where specification flaws have been a huge problem (ethereum dao)
333 2016-07-11 19:47:14	0|sipa|hey
334 2016-07-11 19:47:26	0|kanzure|petertodd: nooo we do need some people thinking about libconsensus things
335 2016-07-11 19:47:39	0|xinxi|petertodd: I think we are just lucky that Bitcoin has not yet experienced any catastrophic attacks?
336 2016-07-11 19:47:43	0|Chris_Stewart_5|petertodd: I disagree, the consensus layer is the bedrock of bitcoin, if we screw that up in a major way we are done
337 2016-07-11 19:48:05	0|kanzure|smart contract stuff is already getting formal verification attention, i woudn't worry about that, it was fairly high profile and papers have been published already
338 2016-07-11 19:48:09	0|petertodd|kanzure: I'm not saying we don't, it's just that from the point of view of a researcher, they're going to get better return on their time by doing other things
339 2016-07-11 19:48:20	0|kanzure|disagree
340 2016-07-11 19:48:41	0|xinxi|petertodd: not all researchers work for papers.
341 2016-07-11 19:48:47	0|petertodd|xinxi: it's not luck... it's how we make changes - turns out that intense review works
342 2016-07-11 19:48:52	0|xinxi|impact is much more important.
343 2016-07-11 19:48:57	0|Chris_Stewart_5|petertodd: Until it doesn't
344 2016-07-11 19:49:06	0|Chris_Stewart_5|Bitcoin has already had issues with this in the past
345 2016-07-11 19:49:18	0|xinxi|petertodd: OpenSSL has been used by so many people. Still it has serious bugs.
346 2016-07-11 19:49:22	0|Chris_Stewart_5|Bip66 (or was it 62) comes to mind
347 2016-07-11 19:49:24	0|sipa|how many consensus failures in bitcoin can you name? :)
348 2016-07-11 19:49:28	0|sipa|yes, that's one
349 2016-07-11 19:49:44	0|sipa|the march 2013 fork was another one
350 2016-07-11 19:49:59	0|btcdrak|xinxi: you should probably also have a conversation with jtimon (who just pinged out). he's going a lot of the libconsensus stuff.
351 2016-07-11 19:50:03	0|petertodd|Chris_Stewart_5: there's no reason to think yet that formal proofs actually do any better in practice, because the pool of talent that can work on them is much smaller
352 2016-07-11 19:50:16	0|sipa|and bitcoin's scripting language before the removal of OP_VER was also a consensus problem
353 2016-07-11 19:50:21	0|sipa|i don't know of any others tbh
354 2016-07-11 19:50:28	0|petertodd|xinxi: indeed, which is why OpenSSL got replaced by libsecp256k1
355 2016-07-11 19:50:39	0|sipa|OP_VER was in 2010
356 2016-07-11 19:50:47	0|sipa|and all the others were due to supporting libraries
357 2016-07-11 19:50:59	0|sipa|xinxi: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009697.html <- read that
358 2016-07-11 19:51:02	0|xinxi|btcdrak: Yeah some one mentioned him to me.
359 2016-07-11 19:51:07	0|sipa|(shameless plug)
360 2016-07-11 19:51:26	0|sipa|xinxi: it gives you an idea of how complicated it was to find some of the actually known consensus failures in bitcoin
361 2016-07-11 19:51:44	0|sipa|the fact that no easier ones have been found may give an indication
362 2016-07-11 19:52:30	0|Chris_Stewart_5|petertodd: I think there actually has been some considerable interest in the formal verification community in bitcoin. I've talked to a handful of researchers myself and I don't think directing them away from the core protocol is a good idea
363 2016-07-11 19:52:31	0|xinxi|I mean the provably correct version of libconsensus may not be consistent with the existing one. The most serious result would be a hard fork, rigth?
364 2016-07-11 19:52:47	0|xinxi|But after that, we will take off. And Bitcoin will be much more secure than before.
365 2016-07-11 19:52:57	0|sipa|xinxi: but we'll need a hard fork to get there?
366 2016-07-11 19:53:19	0|xinxi|if the provably correct version is done, is it worth to have a hard fork?
367 2016-07-11 19:53:32	0|sipa|maybe
368 2016-07-11 19:53:38	0|petertodd|Chris_Stewart_5: I'm not saying there isn't interest, I'm saying that if you want to make a high impact with formal verification deployed in production, Bitcoin isn't interesting because the risks of deploying formal verification in production are higher than the theoretical benefits
369 2016-07-11 19:53:57	0|sipa|xinxi: but there are requirements beyond just correctness
370 2016-07-11 19:54:06	0|petertodd|Chris_Stewart_5: whereas with smart-contract systems, the risks of the existing approaches are known to be very high - likely too high to be practical - so the risks of alternate approaches are less in comparison
371 2016-07-11 19:54:32	0|sipa|xinxi: such as performance, and ease of maintenance
372 2016-07-11 19:54:35	0|btcdrak|xinxi: there are other things than correctness which can cause consensus failure.
373 2016-07-11 19:54:55	0|xinxi|petertodd: sorry, peter, I think current smart contract platforms are trying to solve some hypothetical problems. But this is a bit irrelevant.
374 2016-07-11 19:55:21	0|petertodd|xinxi: speaking of performance, will your formal verification techniques be able to guarantee bounds on execution time/space?
375 2016-07-11 19:55:28	0|kanzure|is jl2012 in singapore?
376 2016-07-11 19:55:34	0|btcdrak|hk
377 2016-07-11 19:55:37	0|kanzure|damn
378 2016-07-11 19:55:38	0|sipa|jl2012 is in hong kong
379 2016-07-11 19:55:44	0|sipa|what petertodd said
380 2016-07-11 19:56:01	0|petertodd|xinxi: *ethereum* is trying to solve a hypothetical problem; systems like R3's Corda are solving very real problems. Yes, they may be problems in centralized systems, but that doesn't make those problems any less real.
381 2016-07-11 19:56:43	0|sipa|i think proving time and space bounds (ideally on the current code!) is probably more useful in practice than a formal specification (which i expect to be much more complicated than you expect, if it is to have production value)
382 2016-07-11 19:56:55	0|xinxi|petertodd: I think most smart contracts should run on AWS. That's much more efficient and easy to use.
383 2016-07-11 19:57:25	0|xinxi|sipa: why do we need to prove time and space bounds?
384 2016-07-11 19:57:37	0|petertodd|xinxi: smart contracts are about verification, not computation; "should run on AWS" makes no sense
385 2016-07-11 19:58:09	0|sipa|xinxi: because if an attack exists that can make validation slow on some nodes, you can take advantage of that (a miner could knock out competition, or get a propagation advantage)
386 2016-07-11 19:58:15	0|petertodd|xinxi: if you exceed the time/space bounds allowed by the actual implementation, the system fails to reach consensus
387 2016-07-11 19:58:26	0|sipa|xinxi: if an attack exists that can make memory grow unbounded, you can knock out validation
388 2016-07-11 19:58:54	0|petertodd|xinxi: in fact, it's fair to say that much of the current development effort is ultimately about making the upper-bound time cost of a block lower
389 2016-07-11 19:58:58	0|sipa|xinxi: for bitcoin's security assumptions to hold, verification of blocks must be negligable in time compared to the block production rate
390 2016-07-11 19:59:41	0|xinxi|sipa: that sounds like the scalability issues that you guys are trying to solve.
391 2016-07-11 20:00:49	0|sipa|well, it's much more a problem today than fear of consensus failure
392 2016-07-11 20:01:28	0|xinxi|petertodd: a simpler way to do smart contracts is to have trustworthy organization to run a platform on AWS and then people can run their smart contracts on the platform. It's scalable, efficient, and can also be verified.
393 2016-07-11 20:02:10	0|petertodd|xinxi: the whole point of smart contracts in banking is to get away from that model
394 2016-07-11 20:02:23	0|btcdrak|xinxi: but we are interested in decentralised systems; there's special about yet another centralised system.
395 2016-07-11 20:02:43	0|sipa|petertodd, xinxi: i think centralized smart contract design is a bit off topic here
396 2016-07-11 20:02:44	0|petertodd|xinxi: verifying computaitons are done correctly is not easy
397 2016-07-11 20:02:47	0|xinxi|petertodd: yeah, we can use Bitcoin in that AWS based smart contract platform. It does not have to be decentralized.
398 2016-07-11 20:03:51	0|petertodd|sipa: point is, I think xinxi would be much better off solving problems in that problem space first, as they're easier to solve with more impact, and that'll give them tools to tackle bitcoin later; misunderstandings about that problem space are indicative of serious misunderstandings with how bitcoin works
399 2016-07-11 20:04:10	0|petertodd|xinxi: again, do you understand my point about verification vs. computation?
400 2016-07-11 20:06:22	0|xinxi|petertodd: you are right. A third party's platform cannot be fully trusted and thus full verification is impossible. But most people's smart contracts don't need that kind of strong verification.
401 2016-07-11 20:08:09	0|xinxi|To me, Bitcoin is great because it is solving a real problem.
402 2016-07-11 20:08:36	0|Chris_Stewart_5|xinxi: Having a formally verified library of smart contracts would be useful. As in any programming language you end up having design patterns, you will see the emergence of patterns in smart contracts imo
403 2016-07-11 20:09:04	0|xinxi|Chris_Stewart_5: it is truly useful in some extreme cases.
404 2016-07-11 20:09:39	0|petertodd|xinxi: I suspect you're unclear as to what smart contracts even do... the sane use-cases I'm talking about, are to formalize legal contracts, allowing adherence to them to be verified - that only works because you can take the state you're in - legally speaking - and verify that it matches the smart contracts (aka protocol definitions)
405 2016-07-11 20:10:00	0|xinxi|And the biggest concern of Bitcoin to me is not the lack of functions but its security. Many people still think it is too young and not reliable enough and could fail completely and thus don't want to adopt it.
406 2016-07-11 20:10:11	0|petertodd|xinxi: bitcoin is exactly the same, except that on top of that, PoW allows our state to converge to a single consensus history
407 2016-07-11 20:10:34	0|petertodd|xinxi: the most likely way bitcoin will fail right now is a failure of decentralization, which is very closely tied to scalability
408 2016-07-11 20:11:01	0|sipa|petertodd: agree, but i still think smart contracts is off topic here.
409 2016-07-11 20:11:08	0|sipa|of the type you're describing
410 2016-07-11 20:11:51	0|petertodd|sipa: again, I'm trying to explain to xinxi how this stuff works - not make smart contracts the topic...
411 2016-07-11 20:12:02	0|sipa|ok, great
412 2016-07-11 20:12:54	0|sipa|xinxi: i think it's unlikely that people will accept a hard fork to switch to a more formally provable consensus implementation
413 2016-07-11 20:13:01	0|xinxi|petertodd: to me, smart contracts are just contracts in code.
414 2016-07-11 20:13:14	0|sipa|i think you're talking about different things
415 2016-07-11 20:13:51	0|petertodd|xinxi: that's not any different from what I said... legal contracts are verification, not computation - they're about conditions that need to be met for something to be valid - aka a protocol specification
416 2016-07-11 20:14:39	0|petertodd|xinxi: equally, the definition of the bitcoin protocol is what the bitcoin implementation accepts as valid - the fact the code that does that is somewhat intermixed with code that records state is just a historical mistake
417 2016-07-11 20:14:42	0|xinxi|yeah, a coded contracts can be run on AWS and still legally binding.
418 2016-07-11 20:15:08	0|xinxi|you just write the legal part in terms of conditions.
419 2016-07-11 20:15:10	0|sipa|xinxi: but if it has very clear benefits (like performance/resource bounds), is easy to work with... maybe, but that's not something for us to decide
420 2016-07-11 20:15:12	0|btcdrak|xinxi: in this space smart contracts are self enforcing contract
421 2016-07-11 20:15:37	0|sipa|xinxi: i don't think people should ever expect a future hard forking change to succeed
422 2016-07-11 20:15:39	0|petertodd|xinxi: why do you keep saying AWS here? that's completely missing the point: what you want is to be able to show to someone else (e.g a judge, but hopefully it doesn't go that far) that the smart contract hasn't been upheld
423 2016-07-11 20:16:18	0|sipa|xinxi: so imho for formal verification to be practical in the short term, it has to be about the currently deployed code
424 2016-07-11 20:16:20	0|petertodd|btcdrak: self-enforcing because of the underlying proof-of-work layers ensuring consensus, and the fact that people widely choose to accept the outputs of the system - bitcoin being a perfect - albeit inflexible - example
425 2016-07-11 20:16:41	0|xinxi|My point is current smart contract platforms are mostly solving hypothetical problems.
426 2016-07-11 20:16:47	0|petertodd|sipa: yeah, pragmatically speaking, replacing the entire codebase in a hardfork is going to be incredibly controversial at best
427 2016-07-11 20:16:52	0|sipa|petertodd: indeed
428 2016-07-11 20:17:31	0|sipa|petertodd, xinxi: again, please take that discussion elsewhere. yes, there is some overlap that is relevant here. but discussion about what problems smart contracts are solving is not
429 2016-07-11 20:17:55	0|petertodd|sipa: ack
430 2016-07-11 20:17:56	0|xinxi|sipa: yeah. My focus is still formal verification.
431 2016-07-11 20:18:01	0|sipa|thanks
432 2016-07-11 20:18:30	0|sipa|xinxi: have you read my BIP66 writeup above?
433 2016-07-11 20:18:57	0|xinxi|So you are Pieter?
434 2016-07-11 20:18:59	0|sipa|yes
435 2016-07-11 20:19:02	0|xinxi|cool
436 2016-07-11 20:19:03	0|sipa|i think it is useful to illustrate the difficulty of consensus problems
437 2016-07-11 20:19:10	0|sipa|of finding
438 2016-07-11 20:19:39	0|kanzure|btcdrak: bitcoincore.org/en/team/ is lacking usernames
439 2016-07-11 20:20:45	0|btcdrak|kanzure: really could do with a bio page. people's user handles are not always consistent across platforms
440 2016-07-11 20:21:13	0|xinxi|sipa: please let met read it first.
441 2016-07-11 20:21:48	0|petertodd|xinxi: we'll, lets continue the discussion in another hour or two after you read it :)
442 2016-07-11 20:21:53	0|petertodd|bbl!
443 2016-07-11 20:23:12	0|sipa|xinxi: (i'll only mention this once): bitcoin core's cryptography library (libsecp256k1) does have some modest attempts at formally verifying pieces of the code, and help there would also be very welcome. It's much less ambitious than proving properties about bitcoin's consensus code, but also much easier to accomplish something in my opinion
444 2016-07-11 20:23:42	0|kanzure|sipa: don't you guys have that libsecp256k1 paper at some point? :D
445 2016-07-11 20:23:54	0|sipa|kanzure: at some point.
446 2016-07-11 20:24:52	0|btcdrak|xinxi: see https://github.com/bitcoin-core/secp256k1
447 2016-07-11 20:25:14	0|xinxi|btcdrak: thanks.
448 2016-07-11 20:25:32	0|xinxi|sipa: that's understandable. It takes huge amount of money to get started.
449 2016-07-11 20:27:25	0|sipa|also, #secp256k1 for that
450 2016-07-11 20:31:59	0|Chris_Stewart_5|sipa: I just read the email, however the hard fork last July wasn't related to the implementation of BIP66 , it was just spv mining correct?
451 2016-07-11 20:32:14	0|sipa|that was not a hard fork
452 2016-07-11 20:32:19	0|sipa|just miners creating invalid blocks
453 2016-07-11 20:32:41	0|sipa|and it triggered when bip65 activated
454 2016-07-11 20:32:48	0|sipa|not bip66, which was a year earlier
455 2016-07-11 20:35:32	0|Chris_Stewart_5|Seems that OP_CLTV was activated ~7 months ago?
456 2016-07-11 20:35:54	0|Chris_Stewart_5|I distinctly remember the issue being July 4th of last year
457 2016-07-11 20:35:55	0|btcdrak|yes, he means BIP66
458 2016-07-11 20:35:58	0|Chris_Stewart_5|oh ok
459 2016-07-11 20:36:06	0|sipa|oh, i'm misremembering
460 2016-07-11 20:36:19	0|btcdrak|CLTV went without a hitch.
461 2016-07-11 20:36:26	0|sipa|apologies
462 2016-07-11 20:36:31	0|Chris_Stewart_5|np
463 2016-07-11 20:36:55	0|btcdrak|I dont know, I think we must give a punishment.
464 2016-07-11 20:38:07	0|Chris_Stewart_5|Seven more years of reading openssl!
465 2016-07-11 20:38:21	0|btcdrak|too harsh. maybe no computer! and go to bed early!
466 2016-07-11 20:46:20	0|sipa|hah
467 2016-07-11 20:46:44	0|xinxi|sipa: so you wrote most of secp256k1?
468 2016-07-11 20:48:13	0|xinxi|The email seems fantastic.
469 2016-07-11 20:48:39	0|xinxi|It's truly a tough journey.
470 2016-07-11 20:49:26	0|sipa|xinxi: i started it; most of the fancier optimizations and verifications/tests were proposed or implemented by others
471 2016-07-11 20:57:14	0|xinxi|sipa: that's interesting. I saw it's written in C intead of C++. I thought you deliberately did that for verification?
472 2016-07-11 20:58:26	0|sipa|xinxi: it started as C++, actually, but gmaxwell suggested converting it to C to aide with formal verification
473 2016-07-11 20:58:36	0|sipa|as there are more analysis tools available for C than C++
474 2016-07-11 20:58:45	0|sipa|this was very early on
475 2016-07-11 20:58:57	0|xinxi|sipa: fantastic! I am so glad to see it's formally verified!
476 2016-07-11 20:59:11	0|sipa|i wouldn't say it's formally verified
477 2016-07-11 20:59:22	0|sipa|just some pieces
478 2016-07-11 20:59:29	0|xinxi|C++ is just too complicated to verify.
479 2016-07-11 20:59:33	0|sipa|i can discuss more on #secp256k1
480 2016-07-11 20:59:43	0|xinxi|Is it a channel?
481 2016-07-11 20:59:45	0|sipa|yes
482 2016-07-11 20:59:55	0|xinxi|sipa: let's talk there.
483 2016-07-11 22:35:16	0|GitHub44|[13bitcoin] 15dooglus opened pull request #8331: Fix three 'comparison between signed and unsigned integer expressions' warnings. (06master...06fix-compilation-warnings) 02https://github.com/bitcoin/bitcoin/pull/8331