1 2017-03-04 00:02:40	0|gmaxwell|no those figures are from using the scalar sha2 code.
  2 2017-03-04 00:03:07	0|gmaxwell|AFAIK the only real place we can make good use of the parallel sha2 SSE code would be in hash tree computation, but that is complicated as you know.
  3 2017-03-04 00:03:26	0|gmaxwell|IIRC the SIMD scalar sha2 is ~2x faster than ours, and the SIMD parallel sha2 is 3x faster than ours.
  4 2017-03-04 00:03:54	0|luke-jr|merkle trees probably aren't a significant amount of hashing I think?
  5 2017-03-04 00:05:28	0|gmaxwell|They're actually tons. because every node in it is three compression function runs. and there are txn*2 nodes in total.
  6 2017-03-04 00:05:48	0|gmaxwell|so a block can have something like 24000 compression function runs.
  7 2017-03-04 00:06:23	0|luke-jr|hmm
  8 2017-03-04 00:08:08	0|sipa|so 8ms?
  9 2017-03-04 00:08:12	0|sipa|that's significant
 10 2017-03-04 00:08:54	0|sipa|assuming 3 GHz and 15 cpb for sha256
 11 2017-03-04 00:09:15	0|TD-Linux|<wumpus> gmaxwell: it's a bit scary though as the external process will be able to keep a reference, and have all your key data :) <- you can seal the fd and verify the seal in the sandboxed process to eliminate this vulnerability
 12 2017-03-04 00:10:09	0|gmaxwell|yes, it's non-trivial in terms of validation latency.
 13 2017-03-04 00:11:19	0|TD-Linux|er actually disregard me, there is no read seal
 14 2017-03-04 00:52:29	0|jeremyrubin|hm Bitcoin Unlimited just added ~parallel block validation~ but I'm pretty sure it has no perforance benefit
 15 2017-03-04 00:52:51	0|jeremyrubin|(in case anyone is looking at what they implemented)
 16 2017-03-04 00:53:32	0|TD-Linux|jeremyrubin, correct, it is not a performance enhancement but an attempt to "fix" quadratic hashing
 17 2017-03-04 00:53:50	0|jeremyrubin|wait what
 18 2017-03-04 00:54:02	0|jeremyrubin|The parallel block validation?
 19 2017-03-04 00:54:05	0|TD-Linux|yes
 20 2017-03-04 00:54:19	0|sipa|it means that the node doesn't stall if your block takes a day to validate
 21 2017-03-04 00:54:27	0|TD-Linux|yup, it'll just burn a core for a day
 22 2017-03-04 00:54:52	0|TD-Linux|of course under the assumption that the block also gets orphaned.
 23 2017-03-04 00:55:12	0|jeremyrubin|Yeah isn't that block hopefully just going to orphan
 24 2017-03-04 00:55:18	0|jeremyrubin|if it takes that long to validate
 25 2017-03-04 00:55:20	0|jeremyrubin|hmm
 26 2017-03-04 00:55:53	0|sipa|also, since our validation is already parallel anyway, it makes both blocks slower
 27 2017-03-04 00:56:06	0|jeremyrubin|yeah that's why I thought it was useless
 28 2017-03-04 00:56:19	0|jeremyrubin|core can only do so many IPC
 29 2017-03-04 00:56:20	0|TD-Linux|jeremyrubin, thus the question "how long is too long"
 30 2017-03-04 00:56:41	0|jeremyrubin|Well... that seems to be a hard fork then?
 31 2017-03-04 00:57:10	0|jeremyrubin|Because now you'll partition old nodes trying to validate whatever monster block (assuming it crashes/kills your node on old hardware)
 32 2017-03-04 00:57:27	0|TD-Linux|jeremyrubin, it is likely to cause forks, yes.
 33 2017-03-04 00:57:31	0|TD-Linux|itself it is not a hard fork.
 34 2017-03-04 00:59:15	0|jeremyrubin|yeah, sorry was slightly imprecise with terminology
 35 2017-03-04 01:00:11	0|jeremyrubin|it seems they don't mention it being a quadratic hashing fix in the documentation
 36 2017-03-04 01:00:46	0|jeremyrubin|Also it seems quadratic hashing isn't really a problem before this either, just wait until the block gets orphaned?
 37 2017-03-04 01:01:41	0|gmaxwell|yea, it clearly has some unexplored interactions with selfish mining too.  E.g. if you mine an empty block, now you hold it locally for a bit comfortable that if there is a block race, you'll win even though you announced later... you can be up to the typical validation time late in your announcement.
 38 2017-03-04 01:02:00	0|gmaxwell|jeremyrubin: qudratic hashing is a _huge_ problem if you've ripped off the blocksize limit and done nothing about it.
 39 2017-03-04 01:02:31	0|gmaxwell|a block could take days to hash and shut down the network. :P (except for collaborating miners that know to 'optimize out' checking that transaciton) :)
 40 2017-03-04 01:03:19	0|jeremyrubin|Do we have a provision for abandoning a block mid validation if a longer header chain is seen?
 41 2017-03-04 01:03:31	0|jeremyrubin|Probably a tighter way to address same concern.
 42 2017-03-04 01:04:46	0|gmaxwell|it's not a concern for us. As it's not really easy to make excessively slow blocks in the concernsus rules and segwit completely fixes quadratic hashing.
 43 2017-03-04 01:05:16	0|gmaxwell|could basically worry about it once it _ever_ would have made a difference, rather than adding complexity now.
 44 2017-03-04 01:05:45	0|gmaxwell|(the complexity would be that that longer chain may be invalid, so you'll have to go back to validating the other thing, seems messy as heck)
 45 2017-03-04 01:06:05	0|jeremyrubin|well... it can be a good "defensive" code in case a new compelxity attaack is ever found.
 46 2017-03-04 01:06:29	0|gmaxwell|sure but has to be weighed against the complexity of the fix and the risk it implies.
 47 2017-03-04 01:06:45	0|jeremyrubin|I think it's a simple rule; always be trying to validate the largest-POW chain
 48 2017-03-04 01:07:03	0|TD-Linux|if such an attack was made, simply stalling is a pretty good failure option
 49 2017-03-04 01:07:24	0|gmaxwell|(maybe if someone implemented it they'd find it was easy... though just testing it makes me feel uneasy, concurrency is really hard to test well.)
 50 2017-03-04 01:07:51	0|jeremyrubin|fair!
 51 2017-03-04 01:08:41	0|gmaxwell|jeremyrubin: we do that, subject to the fact that while validating a block we're effectively non-concurrent, so we won't learn about the longer chain until we're done.  So really the complexity there is just in safely increasing the concurrency. Which might be an independant good--- e.g. a side effect of the changes for the block testing stuff we were talking about a day ago.
 52 2017-03-04 01:11:12	0|TD-Linux|certainly I'd hope for more tests than parallel validation has :^)
 53 2017-03-04 01:12:28	0|gmaxwell|we don't really have a good test harness for testing concurrency.  data race freeness doesn't mean that a parallel algorihim will yield expected results in all ordering sequences.
 54 2017-03-04 01:15:58	0|TD-Linux|some sort of framework that would cause all mutexes to block until the test explicitly lets them continue would be neat.
 55 2017-03-04 01:17:28	0|gmaxwell|TD-Linux: well RR actually has neat stuff for making threaded execution determinstic, that I think could be listed into being a concurrency fuzzing tool.
 56 2017-03-04 01:18:18	0|gmaxwell|e.g. replay from an to a given point... and then repeat the replay with many different values given to the RNG that schedules the threads, and see if you get different results.
 57 2017-03-04 01:18:19	0|TD-Linux|gmaxwell, well if you want to fuzz rather than be explicit, doesn't rr's chaos mode already count?
 58 2017-03-04 01:18:27	0|gmaxwell|oh does it already do this? lol
 59 2017-03-04 01:19:05	0|luke-jr|[01:01:39] <gmaxwell> yea, it clearly has some unexplored interactions with selfish mining too.  E.g. if you mine an empty block, now you hold it locally for a bit comfortable that if there is a block race, you'll win even though you announced later… you can be up to the typical validation time late in your announcement. <-- they may claim this is a good thing, since it incentivises smaller blocks
 60 2017-03-04 01:20:23	0|luke-jr|although in fact it incentivises blocks which meet relay-network policy even better
 61 2017-03-04 01:20:45	0|luke-jr|or rather, the most-limited relay policy
 62 2017-03-04 01:20:49	0|gmaxwell|the key point about selfish mining is that it gives excess returns to larger miners. So "incenticizes smaller blocks at the expense of decenteralization" ... missing the point. :P  Also,  not smaller but empty the validation time difference between a block and a slightly smaller one is neglgible, you have to make the block empty to reliably cut in front of others.
 63 2017-03-04 01:21:48	0|gmaxwell|the latest BIP152 stuff is much more policy durable than prior stuff, since it will retain transactions rejected for policy reasons and still use them to reconstruct blocks.
 64 2017-03-04 01:23:10	0|luke-jr|that's not a good thing IMO
 65 2017-03-04 01:23:33	0|luke-jr|the network policy putting pressure on miners is a desirable trait
 66 2017-03-04 01:23:51	0|luke-jr|although most-restrictive isn't the ideal either, so meh
 67 2017-03-04 01:24:12	0|gmaxwell|'network policy' doesn't matter, other miners policy matters. And moreover: doublespend is 'policy' that is not in the miners control.
 68 2017-03-04 01:24:24	0|gmaxwell|Without extra someone spamming doublespends can considerably slow propagation.
 69 2017-03-04 01:25:14	0|luke-jr|true, but if that's the only concern, we would want to limit the extra pool to just double spends
 70 2017-03-04 01:25:44	0|gmaxwell|luke-jr: consider, without it there is pressure to not increase the minimum feerate in your mempool.
 71 2017-03-04 01:26:04	0|gmaxwell|because it will make you slower to accept blocks from others with a lower minimum feerate.
 72 2017-03-04 01:26:33	0|gmaxwell|(though I think there should probably be seperate extra pools for different kinds of rejections, adding complexity would have delayed getting it in)
 73 2017-03-04 01:27:00	0|gmaxwell|luke-jr: also keep in mind that any time miners expirence delays the easiest solution for them is to just centeralize their pooling more.
 74 2017-03-04 01:27:20	0|gmaxwell|they're not going to sit and go "oh that sucks, I'll twiddle my policy."
 75 2017-03-04 01:28:45	0|gmaxwell|fair enough.
 76 2017-03-04 03:54:17	0|luke-jr|#8694 is finally ready for final reviewing
 77 2017-03-04 03:54:19	0|gribble|https://github.com/bitcoin/bitcoin/issues/8694 | Basic multiwallet support by luke-jr · Pull Request #8694 · bitcoin/bitcoin · GitHub
 78 2017-03-04 04:15:30	0|jeremyrubin|I'm debugging something; is there a threadsafe way to access pcoinstip during connectblock? I want to be able to access it from a scriptcheck
 79 2017-03-04 04:15:54	0|jeremyrubin|I'm guessing I would have to add locks around the usage
 80 2017-03-04 04:16:20	0|jeremyrubin|curious if anyone's done this before and if the performance decrease is bad
 81 2017-03-04 04:16:59	0|jeremyrubin|(I think it might be doable to parallelize checking the inputs)
 82 2017-03-04 04:30:57	0|jeremyrubin|hm I think i have something workable using shared_mutex
 83 2017-03-04 08:07:28	0|bitcoin-git|[13bitcoin] 15kobake opened pull request #9916: Fix msvc compiler error C4146 (minus operator applied to unsigned type) (06master...06fix-minus-operator-target-for-msvc-c4146) 02https://github.com/bitcoin/bitcoin/pull/9916
 84 2017-03-04 08:07:42	0|luke-jr|jonasschnelli: can always not support gitian builds with the first introduction of a new feature ;)
 85 2017-03-04 13:14:47	0|wumpus|whee I have RPC working over a UNIX socket
 86 2017-03-04 13:19:28	0|wumpus|needed no changes to libevent on the server side. however to make bitcoin-cli be able to http over a UNIX socket needs a small change.
 87 2017-03-04 15:46:34	0|bitcoin-git|[13bitcoin] 15laanwj opened pull request #9919: UNIX sockets support for RPC (06master...062017_03_unix_socket) 02https://github.com/bitcoin/bitcoin/pull/9919
 88 2017-03-04 18:00:42	0|sipa|wumpus: nice
 89 2017-03-04 21:13:11	0|Lightsword|wumpus, would it be easy to also do block notifications over the unix socket?
 90 2017-03-04 21:14:23	0|Lightsword|current ckpool local block notification method is basically to execute a binary that then writes to a unix socket for notification of a block
 91 2017-03-04 21:33:53	0|wumpus|Lightsword: everything that an be done over the current P2P port can also be done over the UNIX socket
 92 2017-03-04 21:33:58	0|wumpus|eh, RPC port
 93 2017-03-04 21:34:34	0|wumpus|I guess what you're looking for is #7949
 94 2017-03-04 21:34:36	0|gribble|https://github.com/bitcoin/bitcoin/issues/7949 | [RPC] Add RPC long poll notifications by jonasschnelli · Pull Request #7949 · bitcoin/bitcoin · GitHub
 95 2017-03-04 21:35:17	0|Lightsword|wumpus, not long polling, having bitcoind itself write to another app’s listening unix socket for notifications
 96 2017-03-04 21:43:18	0|wumpus|conceptually it's the same, apart from who calls who. In both cases the listeners get immediate notification. Longpolling is simpler as bitcoind doesn't need to keep track of who to notify, that's implied by who is listening
 97 2017-03-04 21:43:37	0|wumpus|block notifications can also be broadcast over zeromq
 98 2017-03-04 22:37:04	0|jeremyrubin|how does one CreateNewBlock s.t. a witness is added?
 99 2017-03-04 22:37:44	0|jeremyrubin|I'm having trouble writing a unit test using TestChain100 once segwit activates
100 2017-03-04 22:43:03	0|sipa|you need at least 3 retarget periods worth of blocks
101 2017-03-04 22:44:54	0|jeremyrubin|ah so I need to be at > 432
102 2017-03-04 22:48:48	0|sipa|right
103 2017-03-04 22:49:08	0|sipa|1 period before signalling starts, another before it's locked in, and a third before it is active
104 2017-03-04 22:49:23	0|jeremyrubin|hm ok
105 2017-03-04 22:49:32	0|jeremyrubin|that helped
106 2017-03-04 22:49:38	0|jeremyrubin|now I get witness mismatch
107 2017-03-04 22:59:17	0|jeremyrubin|CMutableTransaction n;
108 2017-03-04 22:59:18	0|jeremyrubin|{
109 2017-03-04 22:59:18	0|jeremyrubin|n.vin[0].prevout = COutPoint(txSpend.GetHash(), 0);
110 2017-03-04 22:59:18	0|jeremyrubin|n.vin.resize(1);
111 2017-03-04 22:59:18	0|jeremyrubin|std::vector<CMutableTransaction> txns;
112 2017-03-04 22:59:20	0|jeremyrubin|n.vout.resize(25);
113 2017-03-04 22:59:22	0|jeremyrubin|n.nVersion = 1;
114 2017-03-04 22:59:25	0|jeremyrubin|n.nLockTime = 0;
115 2017-03-04 22:59:27	0|jeremyrubin|n.vin[0].scriptWitness = CScriptWitness();
116 2017-03-04 22:59:30	0|jeremyrubin|n.vin[0].nSequence = CTxIn::SEQUENCE_FINAL;
117 2017-03-04 22:59:32	0|jeremyrubin|for (auto& v : n.vout) {
118 2017-03-04 22:59:35	0|jeremyrubin|v.scriptPubKey = CScript();
119 2017-03-04 22:59:37	0|jeremyrubin|v.nValue = txSpend.vout[0].nValue / n.vout.size();
120 2017-03-04 22:59:40	0|jeremyrubin|}
121 2017-03-04 22:59:43	0|jeremyrubin|txns.push_back(n);
122 2017-03-04 22:59:45	0|jeremyrubin|CreateAndProcessBlock(txns, scriptPubKey);
123 2017-03-04 22:59:47	0|jeremyrubin|}
124 2017-03-04 23:00:01	0|jeremyrubin|(that's the part of the test I'm trying to write)
125 2017-03-04 23:02:26	0|sipa|please don't paste more than 3 lines
126 2017-03-04 23:03:46	0|jeremyrubin|sorry; forgot