1 2016-07-29 00:17:32	0|cfields|NicolasDorier: hmm, I hadn't seen your hash cache stuff yet
  2 2016-07-29 00:18:01	0|cfields|I had been working on something very similar for the purpose of making the sigcache lock free
  3 2016-07-29 00:18:10	0|cfields|(after seeing morcos's numbers and discussing with jeremyrubin)
  4 2016-07-29 00:19:07	0|NicolasDorier|cfields: why is it so important to be lock free ?
  5 2016-07-29 00:19:10	0|cfields|I was going to wait to discuss with Jeremy since it was his idea, but I suppose I should push it up now for discussion. It'd be a shame if we collided with eachother and missed out on the possible speedup
  6 2016-07-29 00:19:44	0|cfields|NicolasDorier: it's not, but morcos reported very significant lock contention with 16 cores. and it's not actually necessary to lock it
  7 2016-07-29 00:20:34	0|NicolasDorier|for the cached hash make, the lock is tacken very briefly for a lookup in a map
  8 2016-07-29 00:20:40	0|cfields|NicolasDorier: sec, I'll push up my (not-fully-vetted) work
  9 2016-07-29 00:20:42	0|NicolasDorier|cached hash map
 10 2016-07-29 00:20:47	0|NicolasDorier|ok
 11 2016-07-29 00:25:49	0|cfields|NicolasDorier: https://github.com/theuni/bitcoin/tree/sigcache-speedup
 12 2016-07-29 00:26:13	0|cfields|NicolasDorier: much of it is quite similar to your work :)
 13 2016-07-29 00:27:39	0|cfields|NicolasDorier: if i can convince morcos to bench that and there's no real improvement, I'll drop it. Otherwise, we might want to merge approaches a bit
 14 2016-07-29 00:28:17	0|sipa|alternatively, for just this, we can just comoute the 3 cached hashes before doing any validation
 15 2016-07-29 00:28:21	0|NicolasDorier|oh cool I'll take a look, in my PR I have a benchmark code if you need
 16 2016-07-29 00:28:25	0|sipa|and then not have any locking at all
 17 2016-07-29 00:28:27	0|NicolasDorier|we can compare easily
 18 2016-07-29 00:30:55	0|cfields|that also has the advantage of making the sigcache usable by libbitcoinconsensus in a way that's thread-friendly, but I'm not sure how necessary/desirable that is
 19 2016-07-29 00:31:42	0|NicolasDorier|cfields: are we talkign about the same thing ? I was talking about midhash caches of segwit, not sig cache in fact
 20 2016-07-29 00:31:49	0|cfields|NicolasDorier: that's great. Sorry for not poking at it further before chiming in, making dinner atm. I'll review yours more carefully tomorrow
 21 2016-07-29 00:31:51	0|NicolasDorier|looking at your PR you are on the sig cache
 22 2016-07-29 00:32:35	0|cfields|NicolasDorier: yes, we're talking about different things. But I think we may want to approach them the same way.
 23 2016-07-29 00:33:02	0|NicolasDorier|ah yes indeed I'll check that.
 24 2016-07-29 00:34:27	0|cfields|like I said, I just took a quick glimpse and wanted to push it up for discussion before it was too late. I'll have a good look at the hash cache tomorrow :)
 25 2016-07-29 00:35:10	0|cfields|bbl
 26 2016-07-29 00:36:26	0|NicolasDorier|cool thanks !
 27 2016-07-29 00:47:06	0|morcos|NicolasDorier: cfields: jeremyrubin: I'm happy to bench anything.
 28 2016-07-29 00:48:00	0|morcos|What I've been working with so far is just a very small change to sigcache that does the erases at the end of block validation (since thats the only thing we need the lock for) and i'm using a boost lock free queue for that
 29 2016-07-29 00:48:15	0|morcos|of course it would be easy'ish to make a custom version
 30 2016-07-29 00:49:09	0|morcos|jeremy has been hard at work at removing the locking from checkqueue.  i know at least that the time to finish connecting all the transactions (and populating the checkqueue) is much slower due to lock contention..  so its a guess that possibly there is a lot of improvement in the actual validation too
 31 2016-07-29 00:49:25	0|morcos|cfields, if your code is ready i'm happy to try that out tomorrow
 32 2016-07-29 00:50:52	0|morcos|i've combined this with an approach to keep an always hot CCoinsViewCache on top of pcoinstip instead of creating a new view in ConnectBlock.  This both saves a lot of copying if things are already in pcoinstip and serves to keep things loaded from disk.
 33 2016-07-29 00:51:39	0|morcos|I do this by running CreateNewBlock once every 30 seconds.  and then pulling into that hot cache the inputs that would be needed to validate that block
 34 2016-07-29 00:53:35	0|morcos|I think this makes it unnecessary to ever have a dbcache bigger than about 300M (although you need up to about 150M additional for the hot cache, but some of that memory you would have been using anyway in block validation)
 35 2016-07-29 00:55:11	0|CodeShark|there seems to be a phishing attack going on where you get an email that seems to come from xapo and you can get 10 bitcoins immediately if you download and install a wallet
 36 2016-07-29 00:55:21	0|CodeShark|where by wallet, I mean malware
 37 2016-07-29 00:55:40	0|morcos|cfields: here is what i have been using: https://github.com/morcos/bitcoin/commit/3e8a10aa78a8a84a44eca9350178ecf7c308c21c
 38 2016-07-29 00:57:57	0|NicolasDorier|morcos: is there some kind of free lock map so I can get rid of lock in my hash cache map ?
 39 2016-07-29 00:58:22	0|NicolasDorier|ah also, those cache objects might find their way in libconsensus, we'll be able to reference boost ?
 40 2016-07-29 00:58:48	0|morcos|NicolasDorier: i don't think boost is the right end decision, and i think lock free coding is beyond the level i feel very comfortable doing myself
 41 2016-07-29 00:59:00	0|morcos|so thats why i'm offering to test other peoples implementations
 42 2016-07-29 00:59:11	0|morcos|but my goal here is to optimize current bitcoin core performance
 43 2016-07-29 00:59:21	0|morcos|how we make that work with libconsensus is a bit of a separate issue
 44 2016-07-29 01:00:12	0|morcos|without seeing a design document or us all agreeing on a roadmap for libconsensus, i don't feel able to help with that.  i've been saying that for a long time now.  i think its a worthy goal, but approaching piecemeal like this is frustrating.
 45 2016-07-29 01:04:19	0|cfields|morcos: mine's a little different....
 46 2016-07-29 01:05:05	0|cfields|morcos: jeremy made the observation that we only ever work with the sigcache from one thread at a time. ignoring the fact that _that_ thread spawns threads that trample on eachother
 47 2016-07-29 01:05:48	0|cfields|so the idea is for each of the workers to return what they would have cached, then batch cache/uncache after each block
 48 2016-07-29 01:06:35	0|morcos|cfields: thats what that commit i showed you does
 49 2016-07-29 01:06:38	0|cfields|rather, the workers return a list of cache hits/misses. the caller can then add the misses and optionally purge the hits (as in the case of connecting a real block)
 50 2016-07-29 01:06:45	0|morcos|you dont' actually cache anything in validation
 51 2016-07-29 01:06:49	0|morcos|its only uncaching
 52 2016-07-29 01:07:00	0|morcos|you dont' add misses in validation
 53 2016-07-29 01:07:08	0|morcos|only in ATMP, which isn't multithreaded
 54 2016-07-29 01:07:20	0|morcos|although good to make that work so we can make ATMP multi threaded
 55 2016-07-29 01:07:31	0|morcos|anyway, is that branch you have something thats ready to be tested
 56 2016-07-29 01:07:35	0|morcos|thats easy enough for me to do
 57 2016-07-29 01:08:47	0|morcos|got to go, will be online tomorrow
 58 2016-07-29 01:09:09	0|cfields|morcos: hmm, ok, I'll reread when I'm not preoccupied. I just saw it setting from get()
 59 2016-07-29 01:09:37	0|cfields|morcos: that'd be great, thanks. I'll read yours more carefully along with NicolasDorier's
 60 2016-07-29 01:10:30	0|NicolasDorier|cfields: wait tomorrow for reviewing my PR, I'm changing some stuff I got from sipa's feedback
 61 2016-07-29 01:15:30	0|cfields|NicolasDorier: roger
 62 2016-07-29 01:16:09	0|cfields|morcos: ah, heh, the diff threw me off
 63 2016-07-29 01:57:42	0|GitHub165|[13bitcoin] 15NicolasDorier opened pull request #8422: Cache hashes (06master...06cachedhashes2) 02https://github.com/bitcoin/bitcoin/pull/8422
 64 2016-07-29 01:58:37	0|GitHub146|[13bitcoin] 15NicolasDorier closed pull request #8259: Hash Cache (06master...06cachedhashes) 02https://github.com/bitcoin/bitcoin/pull/8259
 65 2016-07-29 01:58:48	0|NicolasDorier|cfields: when you feel like reviewing cached hashes : https://github.com/bitcoin/bitcoin/pull/8422
 66 2016-07-29 03:00:38	0|GitHub111|[13bitcoin] 15fanquake opened pull request #8423: [depends] expat 2.2.0, ccache 3.2.7 (06master...06expat-ccache-jul) 02https://github.com/bitcoin/bitcoin/pull/8423
 67 2016-07-29 03:01:40	0|fanquake|morning
 68 2016-07-29 05:13:08	0|luke-jr|jtimon: the issue is entirely giving bad advice to miners. regardless of what they might or might not configure in the end, we should not give bad advice
 69 2016-07-29 05:13:37	0|luke-jr|(btw, I see no reason the performance "hit" of using maxsize would be greater than the performance hit of using 0.13 over 0.12..)
 70 2016-07-29 06:06:27	0|GitHub151|[13bitcoin] 15PrinceofOrange opened pull request #8424: merge (060.8...06master) 02https://github.com/bitcoin/bitcoin/pull/8424
 71 2016-07-29 06:08:33	0|GitHub87|[13bitcoin] 15laanwj closed pull request #8424: merge (060.8...06master) 02https://github.com/bitcoin/bitcoin/pull/8424
 72 2016-07-29 06:09:59	0|luke-jr|O.o
 73 2016-07-29 06:12:54	0|wumpus|yes that's the only possible response - this seems deliberate spammyness, not merely an accident
 74 2016-07-29 06:21:38	0|GitHub182|[13bitcoin] 15laanwj pushed 3 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/ad087638ee48...842bf8d2c5af
 75 2016-07-29 06:21:39	0|GitHub182|13bitcoin/06master 141d06e49 15Suhas Daftuar: Ignore CMPCTBLOCK messages for pruned blocks...
 76 2016-07-29 06:21:39	0|GitHub182|13bitcoin/06master 141de2a46 15Suhas Daftuar: Ignore GETBLOCKTXN requests for unknown blocks...
 77 2016-07-29 06:21:40	0|GitHub182|13bitcoin/06master 14842bf8d 15Wladimir J. van der Laan: Merge #8408: Prevent fingerprinting, disk-DoS with compact blocks...
 78 2016-07-29 06:21:48	0|GitHub16|[13bitcoin] 15laanwj closed pull request #8408: Prevent fingerprinting, disk-DoS with compact blocks (06master...06cb-misbehaving) 02https://github.com/bitcoin/bitcoin/pull/8408
 79 2016-07-29 06:28:39	0|GitHub2|13bitcoin/060.13 14b7e2011 15Suhas Daftuar: Prevent fingerprinting, disk-DoS with compact blocks...
 80 2016-07-29 06:28:39	0|GitHub2|[13bitcoin] 15laanwj pushed 1 new commit to 060.13: 02https://github.com/bitcoin/bitcoin/commit/b7e201181bcc0f6328e0a499803f1dbb2c2dbd28
 81 2016-07-29 07:50:47	0|GitHub65|13bitcoin/060.13 14b06808c 15Wladimir J. van der Laan: doc: Release notes update for rc2
 82 2016-07-29 07:50:47	0|GitHub65|[13bitcoin] 15laanwj pushed 1 new commit to 060.13: 02https://github.com/bitcoin/bitcoin/commit/b06808c58eb7a997c42b55cba63688aec448a0ea
 83 2016-07-29 07:56:48	0|GitHub72|13bitcoin/060.13 14ced6c94 15Wladimir J. van der Laan: qt: Translations update pre-rc2
 84 2016-07-29 07:56:48	0|GitHub72|[13bitcoin] 15laanwj pushed 1 new commit to 060.13: 02https://github.com/bitcoin/bitcoin/commit/ced6c940da35fcf33160d1c7f2f54a99dc7eedb1
 85 2016-07-29 08:35:36	0|wumpus|* [new tag]         v0.13.0rc2 -> v0.13.0rc2
 86 2016-07-29 09:29:25	0|GitHub55|[13bitcoin] 15laanwj pushed 4 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/842bf8d2c5af...b77bb95b3cb4
 87 2016-07-29 09:29:26	0|GitHub55|13bitcoin/06master 14755aa05 15Cory Fields: httpserver: use a future rather than relying on boost's try_join_for
 88 2016-07-29 09:29:26	0|GitHub55|13bitcoin/06master 14d3773ca 15Cory Fields: httpserver: explicitly detach worker threads...
 89 2016-07-29 09:29:27	0|GitHub55|13bitcoin/06master 147e87033 15Cory Fields: httpserver: replace boost threads with std...
 90 2016-07-29 09:29:38	0|GitHub56|[13bitcoin] 15laanwj closed pull request #8421: httpserver: drop boost (#8023 dependency) (06master...06http-thread) 02https://github.com/bitcoin/bitcoin/pull/8421
 91 2016-07-29 10:33:11	0|GitHub12|[13bitcoin] 15laanwj pushed 3 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/b77bb95b3cb4...7a2d40272717
 92 2016-07-29 10:33:12	0|GitHub12|13bitcoin/06master 14695041e 15Wladimir J. van der Laan: util: Update tinyformat...
 93 2016-07-29 10:33:12	0|GitHub12|13bitcoin/06master 14a5072a7 15Wladimir J. van der Laan: util: Remove zero-argument versions of LogPrint and error...
 94 2016-07-29 10:33:13	0|GitHub12|13bitcoin/06master 147a2d402 15Wladimir J. van der Laan: Merge #8274: util: Update tinyformat...
 95 2016-07-29 10:33:17	0|GitHub181|[13bitcoin] 15laanwj closed pull request #8274: util: Update tinyformat (06master...062016_06_update_tinyformat) 02https://github.com/bitcoin/bitcoin/pull/8274
 96 2016-07-29 10:52:47	0|fanquake|Looks like sigs match @wumpus
 97 2016-07-29 10:53:39	0|fanquake|Have you noticed quite a lot of output during the Windows build now?
 98 2016-07-29 10:53:46	0|fanquake|Along the lines of warning: conflicts with previous declaration ‘void boost::signals2
 99 2016-07-29 11:03:50	0|wumpus|good that sigs match!
100 2016-07-29 11:04:07	0|wumpus|no, haven't noticed
101 2016-07-29 11:04:36	0|fanquake|I'll get the logs and open an issue.
102 2016-07-29 11:04:41	0|wumpus|thanks
103 2016-07-29 12:41:04	0|morcos|luke-jr: i was just discussing with sdaftuar yesterday, i'm not sure there is much of a perf hit, it just hasn't been benched yet.  i may try to do that today.  why does 0.13 have worse performance than 0.12, i don't think thats true.
104 2016-07-29 12:50:56	0|michagogo|The "native_biplist" has a confusing name
105 2016-07-29 12:51:14	0|morcos|luke-jr: ugh, sorry. i keep getting confused.  i guess there are two ways it could get slower, and my anecdotal evidence was only that the extra serialization wasn't that bad.  but the repeated failing to find a final tx is blockmaxsize is your constraining factor seems like it could be slow, especially on a big mempool
106 2016-07-29 12:51:26	0|michagogo|My first thought was, wtf? Why is a list of BIPs its own package?
107 2016-07-29 12:51:53	0|michagogo|Then I clicked it and saw it was downloading from PyPI, and that was even more confusing... until I looked up the package description
108 2016-07-29 12:52:48	0|morcos|anyway, too much annoyance to go test all these posibilities..  i think its a clearly correct recommendation that until segwit activates using only blockmaxweight is the desired configuration.  miners will be upgrading anyway for segwit, so there is a chance to argue something different for that
109 2016-07-29 12:52:56	0|morcos|lets concentrate on getting 0.13 right.
110 2016-07-29 13:04:49	0|sdaftuar|cfields: i'm looking at your crash from yesterday. seems odd! can you give some more color on what the environment was: mainnet/testnet/regtest? fresh node or already synced?  what commit were you running?
111 2016-07-29 13:44:23	0|instagibbs|is it expected behavior for a reindex to not show blocks being processed for while a while?
112 2016-07-29 13:44:39	0|instagibbs|in getinfo at least
113 2016-07-29 13:44:54	0|instagibbs|I seem to recall different behavior
114 2016-07-29 14:14:06	0|cfields|sdaftuar: uhmm, from my http-thread branch, which was merged today
115 2016-07-29 14:14:19	0|cfields|sdaftuar: so current master should be basically the same thing
116 2016-07-29 14:15:27	0|cfields|checking my backlog to see how i was running
117 2016-07-29 14:17:09	0|cfields|sdaftuar: ah, a funky config, forgot I was running that way. probably significant...
118 2016-07-29 14:17:16	0|cfields|gdb --args ./bitcoind -testnet -printtoconsole -reindex-chainstate -debug=http
119 2016-07-29 14:17:51	0|cfields|the -reindex-chainstate was accidental, left in there from testing something before.
120 2016-07-29 14:24:22	0|morcos|cfields: i have some results.  here are the average times per block for  Connect transactions/Verify txins/Connect block   (meausres are total time so far as reported in bench)
121 2016-07-29 14:24:31	0|morcos|in ms
122 2016-07-29 14:24:46	0|morcos|master: 26 / 80 / 111
123 2016-07-29 14:25:11	0|morcos|your sigcache: 39 / 46 / 82
124 2016-07-29 14:25:45	0|morcos|batched deletes using boost lock free (commit i showed you): 22 / 32 / 63
125 2016-07-29 14:26:53	0|cfields|morcos: hmm. that's surprising. that's 16 cores?
126 2016-07-29 14:27:12	0|morcos|yeah, 16 cores with 8GB dbcache
127 2016-07-29 14:27:41	0|morcos|for yours and the batched deletes test i also included your copy-move improvements and my hot tipcache, but those don't make much difference on master
128 2016-07-29 14:28:04	0|cfields|ok
129 2016-07-29 14:28:30	0|TomMc|In regards to signature hashtypes, if the hashtype is SIGHASH_ALL, is the last byte given a value of 0x01?
130 2016-07-29 14:28:31	0|cfields|i'm still working on the prevector btw, i've almost got that rewritten and specialized for the unsigned char case
131 2016-07-29 14:29:00	0|morcos|i'll dig into it more, so see if i can understand why your code slowed down Connect transactions
132 2016-07-29 14:29:01	0|cfields|i wonder what throws the connect time off...
133 2016-07-29 14:30:02	0|cfields|morcos: thanks a bunch for testing. that's very interesting. I'll look into the connect as well.
134 2016-07-29 14:31:37	0|cfields|morcos: one more interesting test, if you don't mind
135 2016-07-29 14:31:59	0|morcos|cfields: the connect time can be reduced to as low as 10ms if you eliminate the locking contention in checkqueue
136 2016-07-29 14:32:15	0|cfields|morcos: run with -par=1 as a baseline. I was surprised to see that it's actually faster on some of my machines that way
137 2016-07-29 14:33:06	0|morcos|:) i know!  how i found this stuff in the first place was running with blocks only mode was actually faster than running with tx, implying using the sigcache was slower than verifying the sigs!
138 2016-07-29 14:33:16	0|cfields|morcos: right. I'm looking forward to pairing those two, I should think they would improve eachother
139 2016-07-29 14:33:34	0|cfields|heh, right
140 2016-07-29 14:33:53	0|morcos|will try par=1 as well
141 2016-07-29 14:34:29	0|cfields|morcos: wait. current master actually _is_ faster with -par=1 on 16 cores?!
142 2016-07-29 14:34:56	0|morcos|oh, no i haven't tried that, i was just saying i know in that its not surprising to me that it could be in some configs
143 2016-07-29 14:35:22	0|cfields|oh, whew :)
144 2016-07-29 14:35:39	0|cfields|looks to me like it breaks even around ~3 as-is
145 2016-07-29 14:36:31	0|cfields|but I've tested so many configs now that I can't keep the numbers straight, so that could be wrong
146 2016-07-29 14:41:16	0|cfields|morcos: ah, looks like it's the overhead of creating a copy of the to-test CScriptCheck that slows down the connect. that could be much improved
147 2016-07-29 14:41:45	0|morcos|oh, that was my guess at first but then i didn't see where that happened b/c i thought it was just shared pointers
148 2016-07-29 14:42:16	0|sipa|cfields: before libsecp256k1-based verification, i benchmarked where the break even point was on a large-number-of-cores machine, and set the max value for -par based on that
149 2016-07-29 14:42:25	0|morcos|the method of batching deletes is extremely effective
150 2016-07-29 14:42:45	0|cfields|morcos: yea, those should be cheap, but I'm assuming it's the constant inserts() that suck
151 2016-07-29 14:42:46	0|sipa|when we switched to libsecp validation, the break even point probably dropped a lot
152 2016-07-29 14:43:02	0|sipa|cfields: copy? it should be using swap
153 2016-07-29 14:43:03	0|cfields|sipa: ah yes, that makes sense then.
154 2016-07-29 14:43:13	0|morcos|sipa: depends on whether you are taking advantage of the sigcache a lot or not though
155 2016-07-29 14:43:14	0|cfields|sipa: they're shared_ptrs
156 2016-07-29 14:43:51	0|sipa|cfields: really? when was that changed?
157 2016-07-29 14:44:01	0|morcos|:)
158 2016-07-29 14:44:04	0|morcos|its not in master
159 2016-07-29 14:44:07	0|cfields|sipa: in the branch we're discussing
160 2016-07-29 14:44:41	0|cfields|sipa: https://github.com/bitcoin/bitcoin/compare/master...theuni:sigcache-speedup
161 2016-07-29 14:44:53	0|morcos|back in a few
162 2016-07-29 14:45:37	0|sipa|ah, ok
163 2016-07-29 14:45:46	0|cfields|morcos: that whole vector business (including checkqueue) could be much improved with a list i think. Then we could splice all over the place instead.
164 2016-07-29 14:45:47	0|sipa|i thought you were comparing with master
165 2016-07-29 14:46:19	0|cfields|sipa: no. discussing different ways to reduce the sigcache lock contention
166 2016-07-29 14:46:27	0|sipa|ok ok
167 2016-07-29 14:46:34	0|sipa|looking forward to the results
168 2016-07-29 14:48:33	0|michagogo|cfields: are you by any chance in the process of building rc2?
169 2016-07-29 14:48:37	0|morcos|i can't remember if i mentioned this on channel before, but i think we made a mistake in the design of BIP 68
170 2016-07-29 14:48:39	0|michagogo|Looks like we have 4 sets of sigs already
171 2016-07-29 14:48:48	0|morcos|we should have designed that as a per input check
172 2016-07-29 14:49:17	0|cfields|morcos: another route would be to maintain a single list rather than a vector of per-txin hits/misses. But I was afraid that would slow down one of the checks. I'll whip that up for another data point
173 2016-07-29 14:49:22	0|morcos|that would have done away with that whole issue of calculating prevheights
174 2016-07-29 14:49:32	0|cfields|michagogo: yes, builds in progress
175 2016-07-29 14:49:48	0|morcos|cfields: keep in mind you never need both hits and misses
176 2016-07-29 14:49:50	0|michagogo|(i.e. are detached sigs imminent? If not, I'll just wait until Saturday night, but if they are I can leave the VM running)
177 2016-07-29 14:49:57	0|michagogo|Cool, thanks
178 2016-07-29 14:49:58	0|morcos|i made a couple comments on your commit
179 2016-07-29 14:51:22	0|morcos|and i actually haven't looked at how this interacts with the segwit style cachehashes that NicolasDorier is working on
180 2016-07-29 14:51:28	0|cfields|morcos: well you at least need each txin's misses shared with eachother. for the 2dup..checksig..2drop..checksig case, no?
181 2016-07-29 14:53:01	0|morcos|cfields: hmm..  explain a bit more.  you mean if you are checking the same sig multiple times you dont' want to have to redo the actual verfication?
182 2016-07-29 14:53:23	0|morcos|we don't solve that problem now do we?
183 2016-07-29 14:53:30	0|cfields|morcos: right
184 2016-07-29 14:53:41	0|cfields|morcos: i believe so, as they're all cached on-the-fly
185 2016-07-29 14:53:51	0|morcos|they aren't cached in ConnectBlock
186 2016-07-29 14:53:59	0|morcos|they are only cached in ATMP
187 2016-07-29 14:54:16	0|sipa|morcos: sorry, what is 'they' ?
188 2016-07-29 14:54:30	0|morcos|signature verification results
189 2016-07-29 14:54:41	0|morcos|whats in the existing sigcache
190 2016-07-29 14:54:52	0|sipa|ah, yes
191 2016-07-29 14:56:50	0|cfields|morcos: hmm, yes, you're right
192 2016-07-29 14:58:00	0|sipa|we store results in atmp, delete results in connectblock
193 2016-07-29 14:58:34	0|sipa|if you duplicate sigs i guess that indeed will hurt
194 2016-07-29 15:02:29	0|morcos|cfields: but you were talking about sharing a txin's misses with other txin checks()'s ?  thats not necessary right?  it's only within a given txin that you'll have a duplicated signature?
195 2016-07-29 15:03:17	0|morcos|seems like if thats a worthwhile performance improvement thats easy enough to add separately from the sigcache..  kind of a separate question
196 2016-07-29 15:04:01	0|sipa|cfields, morcos: with nicolasdorier's cache, we could add some sort of accumulator to the cache, which lists all the signatures that were valudated (or even sigcache iterators to them), and then delete them all at once after the whole block is validated
197 2016-07-29 15:04:26	0|cfields|morcos: right. atm, they're per-txin. But since there's the overhead of creating a vector/list/whatever for each, I was (thinking outloud) that it may end up quicker to do per-block instead
198 2016-07-29 15:04:45	0|cfields|sipa: that's exactly what this branch is doing :)
199 2016-07-29 15:04:55	0|cfields|sipa: and morcos's as well
200 2016-07-29 15:04:56	0|sipa|great
201 2016-07-29 15:05:32	0|morcos|sipa: but right that it should probably be combined with NicolasDorier's cache too, b/c you'll want to batch the deleting of the hashes as well
202 2016-07-29 15:05:36	0|cfields|sipa: no, it's helpful. The issue is that that means returning a list of hits/misses for each txin, which has a considerable overhead
203 2016-07-29 15:06:13	0|morcos|cfields: yes, but a lock free list is relatively straight forward right... we don't have to use the boost version
204 2016-07-29 15:06:58	0|morcos|but can accomplish the same thing ourselves..   so instead of returning the list, we just append to the shared lists of hits and misses
205 2016-07-29 15:08:17	0|cfields|morcos: sure, but i _really_ like the idea of moving the cache out so that it can be passed in without worry of threading semantics. that lets others (libbitcoinconsensus) use it as well
206 2016-07-29 15:08:41	0|cfields|but that's not the strongest argument, so I'd be fine with yours as well
207 2016-07-29 15:10:36	0|cfields|morcos: passing it in also means that ATMP can be threaded as well (as you mentioned yesterday). I suppose you could do the same with your approach by adding a similar batch-to-add structure
208 2016-07-29 15:10:46	0|morcos|right
209 2016-07-29 15:12:09	0|sipa|cfields: one way would be to have a simple preallocated vector in each validation thread
210 2016-07-29 15:12:21	0|sipa|and afterwards, all results are combined
211 2016-07-29 15:13:44	0|cfields|sipa: i started that way, but was hesitant to re-intruduce contention in the re-combining. Did you have something in mind to avoid that?
212 2016-07-29 15:14:25	0|sipa|cfields: i meant recombining only after all txn are validated
213 2016-07-29 15:15:41	0|morcos|sipa: yep, that was another approach jeremy and i discussed
214 2016-07-29 15:15:52	0|cfields|sipa: ah, so the threads would set a pointer to their thread_local vector at creation, then master could iterate when finished
215 2016-07-29 15:16:08	0|morcos|cfields: i'm not sure i understand why your branch is safe to remove all the locks...
216 2016-07-29 15:16:15	0|cfields|yes, that would be much better
217 2016-07-29 15:16:55	0|cfields|morcos: a const cache is passed in, it's not changing during validation
218 2016-07-29 15:16:59	0|morcos|ah, i guess its protected by cs_main?
219 2016-07-29 15:17:16	0|morcos|yes, but how did you know another thread isn't modifying it while you are reading from it
220 2016-07-29 15:17:36	0|cfields|morcos: right, it needs to be globally protected. atm nothing else touches it. so i suppose it's cs_main :)
221 2016-07-29 15:17:57	0|morcos|right a rpc call to submit a tx or something
222 2016-07-29 15:19:22	0|cfields|yes
223 2016-07-29 15:19:55	0|cfields|ok, I'll try the per-thread suggestion. should be easy to whip up i think
224 2016-07-29 15:39:58	0|GitHub38|13bitcoin/06master 1454af51d 15Jonas Schnelli: [QA] Add walletdump RPC test (including HD- & encryption-tests)
225 2016-07-29 15:39:58	0|GitHub38|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/7a2d40272717...bbcb8fd88433
226 2016-07-29 15:39:59	0|GitHub38|13bitcoin/06master 14bbcb8fd 15Wladimir J. van der Laan: Merge #8417: [QA] Add walletdump RPC test (including HD- & encryption-tests)...
227 2016-07-29 15:40:10	0|GitHub139|[13bitcoin] 15laanwj closed pull request #8417: [QA] Add walletdump RPC test (including HD- & encryption-tests) (06master...062016/07/dump_test) 02https://github.com/bitcoin/bitcoin/pull/8417
228 2016-07-29 15:42:06	0|luke-jr|morcos: CPFP requires updating stuff as parents get included in the block, which probably isnt *that* expensive, but a heck of a lot more than merely adding numbers (the overhead of maxsize); AFAIK it doesn't do any extra serialization, just adding sizes up
229 2016-07-29 15:42:53	0|morcos|luke-jr: i believe suhas's bench marks for CPFP actually showed it neglibly faster than the pre CPFP code.  this is b/c you can add a whole package of txs at a time i think.
230 2016-07-29 15:43:52	0|GitHub141|[13bitcoin] 15laanwj opened pull request #8427: net: Ignore `notfound` P2P messages (06master...062016_07_notfound) 02https://github.com/bitcoin/bitcoin/pull/8427
231 2016-07-29 15:44:42	0|GitHub72|[13bitcoin] 15laanwj closed pull request #8403: Process "notfound" messages, and safeguard against unreasonably long … (06master...06ProcessNotfound) 02https://github.com/bitcoin/bitcoin/pull/8403
232 2016-07-29 15:47:22	0|luke-jr|morcos: benchmarks with what tx sets? I suspect it would vary significantly depending on the input
233 2016-07-29 15:47:51	0|morcos|historical simulation over a default mempool with default policy i think
234 2016-07-29 15:52:39	0|luke-jr|what is a default mempool? anyhow, I guess the slower path is unlikely to be used historically
235 2016-07-29 15:52:51	0|morcos|300M
236 2016-07-29 15:53:08	0|luke-jr|probably not too likely to be used more with CPFP either
237 2016-07-29 15:53:29	0|luke-jr|morcos: oh, but we don't have historical mempool contents
238 2016-07-29 15:57:29	0|morcos|luke-jr: i'm not sure what you mean.  sdaftuar wrote code to save p2p messages to disk and be able to replay them back off disk to a node.  we've been using that for years to test and bench different changes.
239 2016-07-29 15:58:25	0|luke-jr|morcos: oh, I was not aware; so you have years of data like that? :o
240 2016-07-29 15:58:40	0|morcos|yep
241 2016-07-29 16:03:10	0|morcos|its not a perfect system by any means, but its good for benching and it was very useful for evaluating different fee estimation algorithms
242 2016-07-29 16:08:58	0|Chris_Stewart_5|Where do I add a new unit test file to have it included in test_bitcoin? I looked in test_bitcoin.cpp and didn't find an obvious spot, unless i'm blind
243 2016-07-29 16:09:46	0|sipa|Chris_Stewart_5: just add it to the makefile
244 2016-07-29 16:10:01	0|sipa|there is no explicit list of all tests
245 2016-07-29 16:11:09	0|Chris_Stewart_5|Hmm, the README seems a litle misleading then
246 2016-07-29 16:14:50	0|luke-jr|then clarify it too while you're at it :p
247 2016-07-29 16:18:35	0|Chris_Stewart_5|but it is so much more fun to complain! :-)
248 2016-07-29 16:43:06	0|cfields|gitian signers: v0.13.0rc2 sigs are pushed
249 2016-07-29 16:43:08	0|cfields|michagogo: ping ^^
250 2016-07-29 16:47:53	0|sdaftuar|cfields: i can reproduce your segfault.  kind of an interesting case really, though i believe unrelated to #8096.
251 2016-07-29 16:48:30	0|sdaftuar|cfields: looks like the issue is that if you ctrl-c on startup, while running with -reindex-chainstate, chainActive will not be initialized
252 2016-07-29 16:48:43	0|sdaftuar|because ActivateBestChain returns immediately on ShutdownRequested()
253 2016-07-29 16:48:52	0|sdaftuar|that violates an assumption in RewindBlockIndex
254 2016-07-29 16:53:37	0|GitHub108|[13bitcoin] 15Christewart opened pull request #8428: Update README.md (06master...06imporve_test_readme) 02https://github.com/bitcoin/bitcoin/pull/8428
255 2016-07-29 16:53:47	0|Chris_Stewart_5|luke-jr: ^ :-)
256 2016-07-29 17:14:18	0|cfields|sdaftuar: damn. Yea, completely forgot I was running with reindex-chainstate.
257 2016-07-29 17:14:27	0|cfields|sdaftuar: thanks for reproducing