1 2017-12-05 00:00:18	0|BlueMatt|achow101: give 11824 a spin?
  2 2017-12-05 00:00:23	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #11824:  Block ActivateBestChain to empty validationinterface queue (06master...062017-12-11822-debug) 02https://github.com/bitcoin/bitcoin/pull/11824
  3 2017-12-05 00:00:23	0|BlueMatt|seems to be working for me
  4 2017-12-05 00:00:47	0|achow101|pulling now
  5 2017-12-05 00:01:17	0|achow101|why's it so big?
  6 2017-12-05 00:05:03	0|BlueMatt|cause you cant call the callbacks themselves with cs_main held...I mean the first commit is just a nice-to-have that makes the queue drain faster, the third commit adds DEBUG_LOCKORDER annotations and removes spurious locks in tests, and the last two should probably be squashed
  7 2017-12-05 00:05:04	0|achow101|BlueMatt: build error ---> validation.cpp:2480:18: error: ‘promise’ is not a member of ‘std’
  8 2017-12-05 00:05:06	0|BlueMatt|its not as bad as it looks
  9 2017-12-05 00:05:37	0|BlueMatt|achow101: oops, try now, forgot to commit that line
 10 2017-12-05 00:07:36	0|achow101|what's the queue depth set to?
 11 2017-12-05 00:08:10	0|BlueMatt|max is 10
 12 2017-12-05 00:08:14	0|BlueMatt|feel free to play with it
 13 2017-12-05 00:09:34	0|achow101|well it compiled and it's running
 14 2017-12-05 00:10:00	0|sipa|ship it.
 15 2017-12-05 00:10:58	0|BlueMatt|someone should tag 11822 as 0.16
 16 2017-12-05 00:11:16	0|achow101|I'll let you know how it goes, so far so good.
 17 2017-12-05 00:11:47	0|BlueMatt|I mean it definitely fixes the obvious infite-queue issue, but there could be other gremlins hiding
 18 2017-12-05 00:11:50	0|BlueMatt|I doubt it
 19 2017-12-05 00:12:07	0|BlueMatt|my bigger worry with that patch is future lockorder violations, but not a ton to be done about it, honestly
 20 2017-12-05 00:12:11	0|sipa|tagged
 21 2017-12-05 00:15:42	0|cfields|BlueMatt: i think it'd make for a smoother transition if a fence was inserted after each block's callbacks had been queued, rather than stalling at an arbitrary depth
 22 2017-12-05 00:16:04	0|cfields|that way we could still work with the same assumptions as before, while we work out any remaining out-of-sync issues
 23 2017-12-05 00:16:30	0|BlueMatt|cfields: ugh, we started the fence-removal process two releases ago, I'd rather not add it back if we can avoid it :(
 24 2017-12-05 00:18:19	0|cfields|BlueMatt: the PR is the same thing, I'm just suggesting using a block-callback-depth (depth == 1 initially) rather than an individual callback depth.
 25 2017-12-05 00:18:27	0|BlueMatt|hmm, that patch may not be sufficient :/ achow101 you see it staying low?
 26 2017-12-05 00:18:57	0|achow101|BlueMatt: it's reasonable given my current dbcache of 8000
 27 2017-12-05 00:19:56	0|BlueMatt|cfields: well if you want to do that we should move the blocking to the end and not the beginning and set it to 1 right before release/on the backport branch, not on master, please
 28 2017-12-05 00:20:08	0|BlueMatt|cfields: 90% of the reason to merge this stuff sooner rather than later is to get testing cycles on it
 29 2017-12-05 00:20:19	0|BlueMatt|otherwise its never gonna happen
 30 2017-12-05 00:21:41	0|achow101|BlueMatt: I'd like this to be merged so I can run my fucking node :p
 31 2017-12-05 00:22:08	0|BlueMatt|achow101: lol, you dont carry patches on your node?
 32 2017-12-05 00:23:28	0|achow101|BlueMatt: my node is also what I test and develop on and I would like to be able to test the other stuff I'm working on and PR it without extra patches
 33 2017-12-05 00:24:33	0|BlueMatt|heh, well feel free to review
 34 2017-12-05 00:26:33	0|cfields|BlueMatt: to be clear, i was suggesting that we let it run a full block ahead (depth of single block's callbacks). I wasn't suggesting that we use an actual depth of 1.
 35 2017-12-05 00:26:36	0|BlueMatt|well res size seems to have stabilized on my reindex-chainstate node
 36 2017-12-05 00:26:56	0|BlueMatt|cfields: I dont know how much that'd do aside from hide bugs?
 37 2017-12-05 00:27:20	0|BlueMatt|if we want to neuter it cause we're worried, we should make ProcessNewBlock/ActivateBestChain block until callbacks finish for the latest action
 38 2017-12-05 00:27:38	0|BlueMatt|letting it run ahead but heavily limiting how much it can run ahead just sounds like it'll hide more bugs than anything, no?
 39 2017-12-05 00:29:40	0|cfields|well yes, but it'd ensure that we're only introducing one class of bugs at a time
 40 2017-12-05 00:33:17	0|achow101|for each extra item in the queue, how much extra memory does that use
 41 2017-12-05 00:35:27	0|sipa|a few MB, i expect
 42 2017-12-05 00:37:29	0|BlueMatt|achow101: (or not reindex)
 43 2017-12-05 00:37:52	0|BlueMatt|achow101: its (during IBD, mostly just) a shared_ptr<const CBlock>
 44 2017-12-05 00:38:06	0|achow101|BlueMatt: I'm not sure if the problem I saw was limited to reindex only though
 45 2017-12-05 00:38:06	0|BlueMatt|cfields: I'm not sure which class of bugs is resolved by your suggestion, tbh?
 46 2017-12-05 00:38:18	0|BlueMatt|achow101: it should almost entirely be
 47 2017-12-05 00:38:34	0|BlueMatt|your wallet shouldnt take 10 minutes to process a new block....
 48 2017-12-05 00:38:45	0|achow101|BlueMatt: the node crashed at some point before I had to reindex, and I think that was because of an OOM to (had to do the whole reboot computer stuff as I did when it crashed during reindex)
 49 2017-12-05 00:38:48	0|BlueMatt|cfields: blocking before return may fix some stuff, but....
 50 2017-12-05 00:39:03	0|achow101|that crash caused me to need to reindex it now
 51 2017-12-05 00:39:13	0|BlueMatt|achow101: thats.....weird
 52 2017-12-05 00:39:59	0|achow101|it's possible that that was unrelated, but I'm not sure
 53 2017-12-05 00:41:12	0|BlueMatt|though even that shouldnt use all that much memory....
 54 2017-12-05 00:41:12	0|BlueMatt|was that like right after a restart where you needed to catch up a few days blocks?
 55 2017-12-05 00:41:31	0|BlueMatt|i mean this has been on master a while...
 56 2017-12-05 00:42:02	0|achow101|no, it had been up for a few days at that point, and I had it running for several weeks with a build of master prior to that
 57 2017-12-05 00:42:19	0|achow101|although it is possible that the build I was running before did not have your PR merged
 58 2017-12-05 00:44:49	0|achow101|I can grep through my log file to see what was running before I started on this build. although it may take a while since that log file is 42 GB because I forgot to turn of debug=net
 59 2017-12-05 00:45:10	0|BlueMatt|hmm, I mean I did a read-through of the queue logic and didnt see any obvious races
 60 2017-12-05 00:45:42	0|BlueMatt|ls -lh ~/.bitcoin/debug.log
 61 2017-12-05 00:45:42	0|BlueMatt|-rw------- 1 matt matt 6.5T Dec  4 19:45 /home/matt/.bitcoin/debug.log
 62 2017-12-05 00:45:59	0|BlueMatt|errr, s/races/leaks/
 63 2017-12-05 00:46:28	0|achow101|BlueMatt: do you have your node patched to not shrink the debug.log at each restart?
 64 2017-12-05 00:46:44	0|BlueMatt|i believe it does not shrink by default if -debug? or something like that
 65 2017-12-05 01:03:09	0|jamesob|going to test #11824 with -reindex
 66 2017-12-05 01:03:11	0|gribble|https://github.com/bitcoin/bitcoin/issues/11824 | Block ActivateBestChain to empty validationinterface queue by TheBlueMatt · Pull Request #11824 · bitcoin/bitcoin · GitHub
 67 2017-12-05 02:15:03	0|promag|please give feedback to #11826, ty
 68 2017-12-05 02:15:04	0|gribble|https://github.com/bitcoin/bitcoin/issues/11826 | RFC: Activity feature · Issue #11826 · bitcoin/bitcoin · GitHub
 69 2017-12-05 04:44:15	0|bitcoin-git|[13bitcoin] 15kallewoof closed pull request #11489: [wallet] sendtoaddress style argument (06master...06201709_segwitwallet2_sendtoaddress) 02https://github.com/bitcoin/bitcoin/pull/11489
 70 2017-12-05 05:22:49	0|kallewoof|I've been asked about address index on top of bitcoin core (to replace the unmaintained fork that exists somewhere else). What are people's opinions on this? I know some people prefer a minimal code base, but having an address index option seems like it would alleviate a lot of problems in a lot of places.
 71 2017-12-05 05:27:16	0|gmaxwell|a history address index is unsustainable; anyone who uses it is eventually going to be forced to use trust a third party service.
 72 2017-12-05 05:27:46	0|kallewoof|Why?
 73 2017-12-05 05:28:01	0|kallewoof|Too much space, or..?
 74 2017-12-05 05:28:05	0|gmaxwell|I don't see any problems with having hooks to support such things, but directly incorporating an index of history wouldn't just be a waste of our review and maintaince resources, it would be actively harmful to the ecosystem.
 75 2017-12-05 05:28:28	0|gmaxwell|Yes too much space, too much seek load, and incompatible with pruning; or any kind of pruned sync.
 76 2017-12-05 05:29:17	0|gmaxwell|we've also already seen that behavior where people using such indexes constructed via abe or other tools that created them, failed back to blockchain.info and similar.
 77 2017-12-05 05:29:57	0|gmaxwell|They're also not required except for unusual cases (like making an explorer, which also requires spent-position indexes).   An index of unspent outputs would be far more reasonable.
 78 2017-12-05 05:30:44	0|kallewoof|Yeah, this would be used by explorers, I would expect. I think some (lots?) of them use the unmaintained fork I mentioned earlier.
 79 2017-12-05 05:30:59	0|gmaxwell|I sure hope not.
 80 2017-12-05 05:31:16	0|kallewoof|Rumor says it is so. Part of why I was asked to look into it.
 81 2017-12-05 05:31:23	0|gmaxwell|I know with absolute certanty that that old index patch could lose data, I just don't know _how_.
 82 2017-12-05 05:32:04	0|kallewoof|I think having hooks in bitcoin core that let such a system exist separately is a good plan. It would only rely on API breakage, which bitcoin core is very careful about.
 83 2017-12-05 05:32:32	0|gmaxwell|As in I had a node with it where it was faliing to actual spendable outputs connected to an address, and I tested that it was incorrect quite throughly. I assumed the bad data was some gotcha with reorgs or likewise, but no one investigated.
 84 2017-12-05 05:32:41	0|gmaxwell|Yes, hooks for such things seem fine to me.
 85 2017-12-05 05:32:59	0|kallewoof|Ah.. yeah, reorgs is definitely a challenge.
 86 2017-12-05 05:35:35	0|sipa|if i had infinite time i'd write a separate efficient indexing service, which just speaks the p2p protocol
 87 2017-12-05 05:36:19	0|sipa|which you then can connect to bitcoind
 88 2017-12-05 05:38:01	0|kallewoof|sipa: I guess that would have the obvious benefit of not actually needing a node at all if it could just sync from the network?
 89 2017-12-05 05:38:38	0|sipa|right
 90 2017-12-05 05:39:25	0|sipa|it would be nice to reuse some of the headers sync logic in core, but if you assume you're connected to a trusted peer anyway, i don't think it needs to be allthat complicated
 91 2017-12-05 05:39:29	0|sipa|just deal with reorgs
 92 2017-12-05 05:40:53	0|gmaxwell|from an arch perspective you'd even want it to just keep a SQL database in sync, unfortunately the performance would be horrible except on gold plated system. (massive nvem raid 1 arrays and gobs of ram or whatever)
 93 2017-12-05 05:42:06	0|kallewoof|SQL db sounds like it would be kinda slow yeah.
 94 2017-12-05 05:42:40	0|gmaxwell|thats how abe worked.
 95 2017-12-05 05:42:59	0|kallewoof|I was thinking of using redis, but I think I'm gonna have to do profiling to find a good solution.
 96 2017-12-05 05:43:07	0|gmaxwell|last I heard of anyone successfully running it, it required well over 1TB state and would take a month to sync or something.
 97 2017-12-05 05:43:18	0|kallewoof|Wow..
 98 2017-12-05 05:43:54	0|sipa|i think it's been unusably slow for the better part of 5 years now
 99 2017-12-05 05:44:10	0|gmaxwell|this was years ago, it would presumably be some multiple of that now.
100 2017-12-05 05:49:20	0|kallewoof|It would be address -> tx list, tx -> address list, block -> tx list.. tx'es could be compressed as block height + index (so ~8b). Not sure what else neat stuff you could do. sipa's work on tx format should be inspiring.
101 2017-12-05 05:50:12	0|esotericnonsense|blockheight + index doesn't work well at all, what about reorgs
102 2017-12-05 05:51:30	0|gmaxwell|works fine, so long as you're only tracking a single chain at a time.
103 2017-12-05 05:51:30	0|kallewoof|Maybe keep track of last 1000 entries and undo all based on shared fork point height?
104 2017-12-05 05:51:36	0|gmaxwell|you have to back out the index and reapply.
105 2017-12-05 05:51:49	0|kallewoof|Could just look at block again and undo, yeah.
106 2017-12-05 05:53:11	0|esotericnonsense|eh I suppose that's right. SQL is a bit abstracted for me i'm not sure how it stores repeated data internally. f.ex. if you have a ton of tables with foreign keys do they store the keys repeatedly or use a more space efficient method.
107 2017-12-05 05:55:46	0|esotericnonsense|for example on chain you might have a 256bit txid used hundreds of times if it has a bunch of outputs but you can probably just store that as a 32bit(?) incrementing integer on every tx you've seen since genesis.
108 2017-12-05 05:55:48	0|kallewoof|I really doubt SQL would work well for this, but I could be wrong.
109 2017-12-05 05:56:45	0|Randolf|esotericnonsense:  It depends on the SQL server software being used.  As far as I know, whatever you choose to store in the columns is what's stored there.
110 2017-12-05 05:57:18	0|Randolf|esotericnonsense:  Usually it's up to the DBA to choose to set up smaller IDs that reference more data in a separate row.
111 2017-12-05 05:58:41	0|kallewoof|We've used up 6.5% of a uint32 (278236762/4294967295), but it could work for awhile at least.
112 2017-12-05 05:58:51	0|Randolf|esotericnonsense:  For Bitcoin stuff, I'd create custom datatypes that save the data in binary format.
113 2017-12-05 05:59:21	0|esotericnonsense|basically what i'm thinking is that these indexes can easily multiply the blockchain size by five  and possibly more depending on how you do it if you store the full data.
114 2017-12-05 05:59:48	0|Randolf|It's possible.
115 2017-12-05 06:00:30	0|esotericnonsense|address -> txlist, block -> txlist, and tx (unless you don't store tx but pull it out of a stored blockchain) is 3x txid
116 2017-12-05 06:00:41	0|Randolf|An index normally refers to an internal reference that's usually smaller than the column data it's indexing.
117 2017-12-05 06:01:23	0|Randolf|Can arrays help?  In PostgreSQL a column type can be an array.
118 2017-12-05 06:01:31	0|Randolf|These can also be indexed efficiently.
119 2017-12-05 06:02:35	0|Randolf|While you'd still be storing TX data, you could cut down on the number of rows.
120 2017-12-05 06:08:05	0|gmaxwell|this sort of speculation isn't so useful.
121 2017-12-05 06:09:06	0|gmaxwell|IIRC the abe schema wasn't too offensively inefficienct, but there are overheads and billions of records add up... but nothing is really learned from this discussion.  just "I think this would be not efficienct" and "I think it would be, maybe" :P
122 2017-12-05 06:09:31	0|esotericnonsense|yeah agree.
123 2017-12-05 06:09:39	0|kallewoof|I'll just try a bunch of stuff I guess.
124 2017-12-05 06:10:52	0|esotericnonsense|it sounds to me like what might be useful is to have a seperate service that only maintains the set of things core doesn't
125 2017-12-05 06:11:22	0|esotericnonsense|so basically addrindex but as an external thing. all it needs is addr->[txids]. if you want the transactions just ask core rpc with txindex on
126 2017-12-05 06:12:24	0|esotericnonsense|that'd only need the core rpc api to remain (relatively) static and not have to be maintained as a thing on an ongoing basis if internals shuffle around.
127 2017-12-05 06:12:44	0|kallewoof|If txs are stored as height+index you don't even need txindex, at least if #10275 is merged.
128 2017-12-05 06:12:47	0|gribble|https://github.com/bitcoin/bitcoin/issues/10275 | [rpc] Allow fetching tx directly from specified block in getrawtransaction by kallewoof · Pull Request #10275 · bitcoin/bitcoin · GitHub
129 2017-12-05 06:13:37	0|kallewoof|And yeah, we agreed this should be an external thing that either uses hooks in bitcoin or implements the p2p protocol and talks to the network directly.
130 2017-12-05 06:15:22	0|esotericnonsense|i guess i was commenting on "address -> tx list, tx -> address list, block -> tx list" and how i don't think you need the latter two because bitcoind does it
131 2017-12-05 06:16:09	0|kallewoof|I wasn't assuming the person running had a local synced node. Maybe I should.
132 2017-12-05 06:17:01	0|esotericnonsense|eh? addrindex implies that no? (well that or that you have rpc access anyway).
133 2017-12-05 06:18:07	0|kallewoof|sipa had the idea of an efficient indexing service which speaks the p2p protocol.
134 2017-12-05 06:19:12	0|gmaxwell|that assumption saves you basically 1x copy of the blockchain at most, but then means you can't run the indexer on the seperate hardware from the node or fail over between multiple nodes, etc.
135 2017-12-05 06:19:29	0|gmaxwell|or upgrade the node independantly from the indexer
136 2017-12-05 06:22:24	0|kallewoof|Makes sense
137 2017-12-05 06:22:35	0|kallewoof|(to not assume local node)
138 2017-12-05 06:23:36	0|esotericnonsense|i'm confused. you can run the indexer on seperate hardware just fine. failover works in the same way anyway (you can have multiple bitcoinds if you want)
139 2017-12-05 06:24:49	0|esotericnonsense|height+index storing of transactions becomes complicated in that case sure.
140 2017-12-05 06:25:40	0|kallewoof|Ultimately it doesn't matter to the indexer if you have a local node or not. It won't use it directly, it will use the p2p network. The only time it needs the RPC is when it wants to convert e.g. block height+index into a tx, but it might just as well give you the height+index pairs and let you deal yourself.
141 2017-12-05 06:29:15	0|gmaxwell|also if the node the indexer is behind is pruned then the overhead of a redundant copy mostly goes away.
142 2017-12-05 06:38:16	0|kallewoof|If the node is pruned, the indexer will not be able to use it for any lookups of block height <= pruned height
143 2017-12-05 06:39:12	0|esotericnonsense|if it has p2p logic (basically it's a node) then what is the seperate bitcoind node being used for at all
144 2017-12-05 06:39:42	0|esotericnonsense|trusted verifier i guess?
145 2017-12-05 06:39:59	0|kallewoof|The indexer doesn't store anything but the indexing parts, so it needs a node to resolve the 8 byte (block height + tx index) into actual transactions. It could store the entire block chain while at it, but that seems a bit redundant.
146 2017-12-05 06:40:25	0|sipa|kallewoof: that seems inefficient
147 2017-12-05 06:40:46	0|sipa|relying on a node for storage won't gove the kind of performance a blockexplorer needs
148 2017-12-05 06:41:31	0|kallewoof|I thought you were suggesting a p2p indexer that doesn't depend on a local node. Why not use RPC then?
149 2017-12-05 06:42:41	0|sipa|rpc is slow
150 2017-12-05 06:42:43	0|kallewoof|Or are you saying store the entire transaction in the indexer db?
151 2017-12-05 06:42:47	0|sipa|yes
152 2017-12-05 06:42:52	0|kallewoof|Ah
153 2017-12-05 09:21:26	0|kallewoof|Hum. I could avoid a lot of reinventing wheels by simply requiring libbitcoin_[x].a in the indexer to link p2p stuff in. Potential minefield, but would be super easy to keep up to date with bitcoin core if it worked.
154 2017-12-05 09:23:21	0|kallewoof|Rewriting a minimal p2p codebase could be useful for lots of cases, but would need to be maintained to a greater degree.
155 2017-12-05 09:24:48	0|gmaxwell|also less important if its out of date... p2p protocol intentionally does not change frequently.
156 2017-12-05 09:25:16	0|gmaxwell|an indexer written against 0.2.10 would likely still work (though wouldn't index p2sh/segwit, but that may not matter if you're not using those things)
157 2017-12-05 09:25:52	0|kallewoof|That's a good point.
158 2017-12-05 09:27:20	0|kallewoof|I guess I'll try linking directly and see how far that gets me. I suspect there will be a lot of issues where the bitcoin code expects a fully fledged bitcoin instance running.
159 2017-12-05 09:28:09	0|kallewoof|If not, not regularly maintained would not be a critical issue as you noted.
160 2017-12-05 11:28:55	0|bitcoin-git|[13bitcoin] 15MeshCollider opened pull request #11829: Test datadir specified in conf file exists (06master...06201712_datadir_crash) 02https://github.com/bitcoin/bitcoin/pull/11829
161 2017-12-05 11:30:25	0|luihuidui|Hi core devs
162 2017-12-05 14:23:54	0|promag|ping jnewbery
163 2017-12-05 14:24:27	0|promag|is it possible to test .conf file?
164 2017-12-05 14:24:52	0|promag|ref #11829
165 2017-12-05 14:24:54	0|gribble|https://github.com/bitcoin/bitcoin/issues/11829 | Test datadir specified in conf file exists by MeshCollider · Pull Request #11829 · bitcoin/bitcoin · GitHub
166 2017-12-05 14:29:24	0|aj|promag: do stuff like https://github.com/bitcoin/bitcoin/pull/10996/files#diff-7814a1916d2f12f31509d33370931974 ?
167 2017-12-05 14:30:00	0|promag|aj: \m/
168 2017-12-05 14:54:03	0|bitcoin-git|[13bitcoin] 15hkjn opened pull request #11830: rpcuser.py: Use 'python' not 'python2' (06master...06rpcuser-py) 02https://github.com/bitcoin/bitcoin/pull/11830
169 2017-12-05 15:41:16	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #11831: Always return true if AppInitMain got to the end (06master...062017-12-startup-exit-return-code-race) 02https://github.com/bitcoin/bitcoin/pull/11831
170 2017-12-05 15:56:36	0|promag|BlueMatt: I didn't realise main thread was sitting there in WaitForShutdown
171 2017-12-05 15:56:57	0|BlueMatt|hmm?
172 2017-12-05 15:57:08	0|BlueMatt|ohoh, yea, we have a global thread that sits there and does nothing :(
173 2017-12-05 15:57:53	0|promag|actually it does.. are we there yet? are we there yet? are we there yet? ..
174 2017-12-05 15:58:28	0|BlueMatt|heh
175 2017-12-05 15:58:29	0|BlueMatt|yea...that
176 2017-12-05 15:59:29	0|promag|so zapwallettxes causes the process to exit with non-zero
177 2017-12-05 15:59:55	0|BlueMatt|Ive never seen that failure before, but I think its the race with ReacceptWalletTransactions
178 2017-12-05 16:00:03	0|BlueMatt|if nothing else that pr should be an improvement
179 2017-12-05 16:00:08	0|BlueMatt|its crazy to have that race
180 2017-12-05 17:33:43	0|sipa|off topic for at least 5 years: c++17 is now officia
181 2017-12-05 17:47:44	0|spudowiar|sipa: With these new C++ standards I'll soon playing fast and loose with the word "know" when I say that I know C++
182 2017-12-05 17:47:48	0|spudowiar|*soon be
183 2017-12-05 17:50:16	0|sipa|haha
184 2017-12-05 18:18:29	0|phantomcircuit|BlueMatt, wait doesn't WaitForShutdown wait in the kernel?
185 2017-12-05 18:20:04	0|BlueMatt|phantomcircuit: because signal handlers cant do "things"
186 2017-12-05 18:20:27	0|BlueMatt|so eg we cant do a cv notify in the SIGTERM signal handler
187 2017-12-05 18:20:32	0|BlueMatt|which is what we'd prefer to do
188 2017-12-05 18:21:04	0|BlueMatt|so *something* has to check for fShutdownRequested regularly
189 2017-12-05 19:14:24	0|sipa|BlueMatt: did i do something wrong when merging #10773? All travis master builds since report 'HEAD was not signed with a trusted key!'
190 2017-12-05 19:14:26	0|gribble|https://github.com/bitcoin/bitcoin/issues/10773 | Shell script cleanups by practicalswift · Pull Request #10773 · bitcoin/bitcoin · GitHub
191 2017-12-05 19:14:39	0|BlueMatt|almost certainly, then
192 2017-12-05 19:14:40	0|BlueMatt|:p
193 2017-12-05 19:14:52	0|BlueMatt|sipa: this is why you're supposed to use the verify-commits pre-push-hook :(
194 2017-12-05 19:15:54	0|sipa|well where is the error?
195 2017-12-05 19:16:07	0|sipa|it's signed with the same key as all my previous merges
196 2017-12-05 19:18:28	0|sipa|BlueMatt: also, please document the pre-push things in developer-notes.md or so
197 2017-12-05 19:18:52	0|BlueMatt|sipa: well the pre-push-hook is only for the committers...and I could have sworn I've yelled at all of y'all about it
198 2017-12-05 19:19:19	0|sipa|then create a committers.md or so
199 2017-12-05 19:20:23	0|BlueMatt|cp contrib/verify-commits/pre-push-hook.sh .git/hooks/pre-push
200 2017-12-05 19:20:43	0|BlueMatt|or, if you prefer, echo 'cp contrib/verify-commits/pre-push-hook.sh .git/hooks/pre-push' > committers.md
201 2017-12-05 19:21:10	0|sipa|please document it somewhere more permanently than IRC :)
202 2017-12-05 19:21:32	0|sipa|haha
203 2017-12-05 19:21:53	0|BlueMatt|as for travis, cant reproduce, so need to dig more, hold on
204 2017-12-05 19:24:34	0|sipa|thanks!
205 2017-12-05 19:58:02	0|aj|jnewbery: ping?
206 2017-12-05 19:58:12	0|BlueMatt|sipa: did you extend your key's (or subkey's) expiry period?
207 2017-12-05 19:58:42	0|BlueMatt|sipa: for some reason travis keeps getting a copy where the A636E97631F767E0 subkey expires on oct 19, 2017
208 2017-12-05 19:58:42	0|sipa|BlueMatt: no
209 2017-12-05 19:58:47	0|sipa|heh
210 2017-12-05 19:59:07	0|sipa|i can't remember changing that, at least
211 2017-12-05 20:17:31	0|BlueMatt|sipa: lol, did you know POSIX echo is poorly-defined?
212 2017-12-05 20:17:44	0|sipa|?
213 2017-12-05 20:18:02	0|BlueMatt|sec
214 2017-12-05 20:19:35	0|aj|BlueMatt: "echo -e hi" ?
215 2017-12-05 20:19:57	0|BlueMatt|aj: -e is implied for (some) \ things in dash, and you cant turn it off
216 2017-12-05 20:21:21	0|aj|BlueMatt: yeah. use printf or printf "%s" "what you actually want to print" ...
217 2017-12-05 20:21:28	0|spudowiar|BlueMatt: That is why I always use python -c "print(*__import__(\"sys\").argv[1:])" in my shell scripts
218 2017-12-05 20:21:29	0|BlueMatt|yes
219 2017-12-05 20:21:38	0|BlueMatt|spudowiar: lolwat
220 2017-12-05 20:22:41	0|spudowiar|BlueMatt: :)
221 2017-12-05 20:23:13	0|spudowiar|I wonder if we should discuss #11833 here as it's turned into a conversation that might be easier to have on IRC :P
222 2017-12-05 20:23:15	0|gribble|https://github.com/bitcoin/bitcoin/issues/11833 | [Net] WebSocket support · Issue #11833 · bitcoin/bitcoin · GitHub
223 2017-12-05 20:23:27	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #11834: [verify-commits] Fix gpg.sh's echoing for commits with '\n' (06master...062017-12-verify-commits-fix) 02https://github.com/bitcoin/bitcoin/pull/11834
224 2017-12-05 20:27:07	0|BlueMatt|spudowiar: lots of folks seem  like the "bitcoin has a p2p network, lets shove more features in it so that *everything* can use it!" approach, cause its easy, but it also doesnt scale....ultimately the bitcoin p2p network is full of garbage evil nodes anyway, its not magic...
225 2017-12-05 20:27:27	0|bitcoin-git|[13bitcoin] 15practicalswift opened pull request #11835: Add Travis check for unused Python imports (06master...06lint-python) 02https://github.com/bitcoin/bitcoin/pull/11835
226 2017-12-05 20:28:44	0|spudowiar|BlueMatt: Awww, but I was trying to run bitcoind in the browser though :P
227 2017-12-05 20:29:30	0|BlueMatt|spudowiar: please build second network of servers which provide blockchain data, it'd be useful to have redundancy for bitcoind to sync from them, even
228 2017-12-05 20:29:47	0|BlueMatt|(or...whats the wallet for that again? electrum?)
229 2017-12-05 20:30:24	0|spudowiar|huh?
230 2017-12-05 20:32:06	0|spudowiar|Does --with-sysroot= work with Bitcoin Core?
231 2017-12-05 20:32:14	0|achow101|spudowiar: why the hell would you want to run bitcoind in the browser
232 2017-12-05 20:32:23	0|spudowiar|achow101: Because I can, I guess?
233 2017-12-05 20:32:30	0|spudowiar|I'm a weird guy
234 2017-12-05 20:33:06	0|sipa|do you mean... compile with emscripten?
235 2017-12-05 20:34:03	0|spudowiar|sipa: Yeah, I'm using Emscripten and I have Bitcoin-Qt mostly working (except networking is disabled)
236 2017-12-05 20:34:08	0|spudowiar|And some graphics are acting wierd
237 2017-12-05 20:34:10	0|spudowiar|*weird
238 2017-12-05 20:34:24	0|instagibbs|neveraskedifheshould.gif
239 2017-12-05 20:34:46	0|sipa|spudowiar: that's pretty cool :)
240 2017-12-05 20:43:02	0|jonasschnelli|BlueMatt: https://github.com/bitcoin/bitcoin/pull/11740/commits/f9cc6dcc65a9cd9ce72c7fbb6bde9d2efb71a8f7#r154686982
241 2017-12-05 20:43:10	0|jonasschnelli|can you elaborate? Can't follow
242 2017-12-05 20:44:16	0|BlueMatt|jonasschnelli: the buffer comment?
243 2017-12-05 20:44:21	0|jonasschnelli|Yeah
244 2017-12-05 20:44:46	0|jonasschnelli|So accepting blocks up to NODE_NETWORK_LIMITED_MIN_BLOCKS(288)+2?`
245 2017-12-05 20:45:08	0|BlueMatt|yea, that
246 2017-12-05 20:47:08	0|jonasschnelli|BlueMatt: current impl.: if you tip is 1000 and someone requests height 713, you will disconnect because (1000-713 == 289)
247 2017-12-05 20:47:23	0|jonasschnelli|But you want disconnect 712 (==288 == within the treshold)
248 2017-12-05 20:47:35	0|jonasschnelli|BlueMatt: I don't see whats wrong with it
249 2017-12-05 20:48:21	0|jonasschnelli|BlueMatt: We already have a buffer of 144 (we have 144 additional blocks for the reorg buffer)
250 2017-12-05 20:48:44	0|jonasschnelli|(goal is 144 [one day], +144 reorg buffer = 288)
251 2017-12-05 20:48:45	0|BlueMatt|the service bit seems to imply that you can/will serve the top 288 blocks, but if you request a block, and then they get a new block before they respond to you, you get disconnected
252 2017-12-05 20:48:49	0|BlueMatt|even though you didnt violate the bip
253 2017-12-05 20:49:42	0|jonasschnelli|I guess if a client detects that he will request at the very bottom of NODE_NETWORK_LIMITED_MIN_BLOCKS he needs to expect a disconnect
254 2017-12-05 20:50:01	0|BlueMatt|I'd call the current implementation a violation of the BIP
255 2017-12-05 20:50:05	0|jonasschnelli|BlueMatt: what you propose would essentially increase the 144 block buffer leading to a similar event just two blocks higher?
256 2017-12-05 20:50:07	0|BlueMatt|"If signaled, the peer MUST be capable of serving at least the last 288 blocks "
257 2017-12-05 20:50:14	0|BlueMatt|the current implementation is clearly not capable of doign so
258 2017-12-05 20:50:38	0|jonasschnelli|BlueMatt: hmm? why? I can serve the last 288 blocks?
259 2017-12-05 20:50:40	0|BlueMatt|if I request block at 288 deep, I do not expect to get disconnected for it, even if there is a race and you get another block at that same time
260 2017-12-05 20:50:59	0|jonasschnelli|hmm..
261 2017-12-05 20:51:17	0|BlueMatt|while we're on the bip (cause I just pulled it up): "A safety buffer of additional 144 blocks to handle chain reorganizations SHOULD be taken into account when connecting to a peer signaling the NODE_NETWORK_LIMITED service bit." <-- I have absolutely no idea what in the hell that means
262 2017-12-05 20:51:27	0|jonasschnelli|https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki#counter-measures-for-peer-fingerprinting
263 2017-12-05 20:51:41	0|jonasschnelli|Okay... your right. Got your point!
264 2017-12-05 20:52:05	0|BlueMatt|I mean you're not *really* serving deeper, maybe one or two, but its certainly not a fingerprinting attack if everyone does it
265 2017-12-05 20:52:05	0|jonasschnelli|Though...
266 2017-12-05 20:52:28	0|jonasschnelli|But what if we use two blocks and other implementation would use a single block as additional buffer?
267 2017-12-05 20:52:41	0|jonasschnelli|But... meh,.. doesn't matter
268 2017-12-05 20:52:47	0|BlueMatt|then they'd already be fingerprintable because they have a different subver :p
269 2017-12-05 20:52:49	0|jonasschnelli|let me just add two blocks
270 2017-12-05 20:52:54	0|jonasschnelli|BlueMatt: indeed
271 2017-12-05 20:53:11	0|BlueMatt|I'm still confused as to what you meant by that sentence in the BIP
272 2017-12-05 20:53:18	0|BlueMatt|"A safety buffer of additional 144 blocks to handle chain reorganizations SHOULD be taken into account when connecting to a peer signaling the NODE_NETWORK_LIMITED service bit."
273 2017-12-05 20:53:23	0|BlueMatt|I have no idea what that means
274 2017-12-05 20:53:45	0|gmaxwell|144? I thought that was supposted to advise 12.
275 2017-12-05 20:53:47	0|sipa|i think it means you should only connect out to a LIMITED peer if you don't think you need more than 144 blocks (rather than 288)
276 2017-12-05 20:53:48	0|jonasschnelli|BlueMatt: assume you are a light client and what to fetch a block within the last 144
277 2017-12-05 20:54:01	0|jonasschnelli|what sipa said
278 2017-12-05 20:54:25	0|BlueMatt|sipa: elaborate? I'm still lost...
279 2017-12-05 20:54:31	0|BlueMatt|gmaxwell: wat?
280 2017-12-05 20:54:40	0|gmaxwell|oh jesus people.
281 2017-12-05 20:54:57	0|sipa|BlueMatt: i think it means that you should only signal LIMITED if you keep the last 288 blocks, but only fetch from a peer if you think you don't need more than 144
282 2017-12-05 20:55:17	0|BlueMatt|sipa: yea, sorry, that half I got, the *why* part I'm still missing...
283 2017-12-05 20:55:25	0|BlueMatt|gmaxwell: what'd we blow up?
284 2017-12-05 20:55:25	0|sipa|i don't know!
285 2017-12-05 20:56:05	0|jonasschnelli|BlueMatt: It's a kindly reminder for a reorg buffer...
286 2017-12-05 20:56:30	0|jonasschnelli|BlueMatt: if you have a light client app, you need tip-150, don't connect to LIMITED
287 2017-12-05 20:56:37	0|gmaxwell|BlueMatt: please read the prior discussions.
288 2017-12-05 20:56:44	0|jonasschnelli|BlueMatt: but do if you need block TIP-144
289 2017-12-05 20:57:03	0|jonasschnelli|(it's pure informative)
290 2017-12-05 20:57:13	0|sipa|it shouldn't be just informative
291 2017-12-05 20:57:19	0|jonasschnelli|Technically, it's always 288
292 2017-12-05 20:57:25	0|gmaxwell|gah
293 2017-12-05 20:57:52	0|sipa|gmaxwell: i vaguely remember discussions about shorter windows, but i wouldn't know where to find them
294 2017-12-05 20:57:55	0|gmaxwell|look, the BIP requires you keep your last 288.  Thats it, the rest is advice.
295 2017-12-05 20:58:03	0|jonasschnelli|"A safety buffer of additional 144 blocks to handle chain reorganizations SHOULD be taken into account when connecting to a peer signaling the NODE_NETWORK_LIMITED service bit."
296 2017-12-05 20:58:10	0|jonasschnelli|Is purely advice... you can't set a MUST there IMO
297 2017-12-05 20:58:23	0|sipa|it can be a SHOULD
298 2017-12-05 20:58:53	0|gmaxwell|If a client _guesses_ it's going to need blocks 285 ago, then a LIMITED peer is quite possibly useless to it: one, it's estimate of how many it will need will not be accurate (there could have been more blocks) and even it if is, they could be on a fork from where the peer is.
299 2017-12-05 20:59:22	0|BlueMatt|yes, that much I get, I'm just really confused as to where the N/2 came from there
300 2017-12-05 20:59:35	0|BlueMatt|why does the buffer need to be 2x what you want?
301 2017-12-05 20:59:58	0|gmaxwell|The actual correct behavior is complicated; if you're estimating your need based on nothing but time you should be pretty conservative (n/2 for example).  If you have headers and know what you need, that doesn't exist.
302 2017-12-05 21:00:27	0|BlueMatt|ok, it just seemed to be weird to be so specific...
303 2017-12-05 21:02:37	0|gmaxwell|I think the 144 number comes more out of "if we want to be able to reliably serve 144, then our target will need to be higher, e.g. 288)
304 2017-12-05 21:03:53	0|BlueMatt|ok, fair
305 2017-12-05 21:06:33	0|BlueMatt|it just reads funny, was my point :p
306 2017-12-05 21:06:57	0|BlueMatt|jonasschnelli: wanna just squash and I'll ack
307 2017-12-05 21:07:19	0|jonasschnelli|BlueMatt: Yeah. Just squashing (had to rebase anyways)
308 2017-12-05 21:07:23	0|jonasschnelli|Give me a min
309 2017-12-05 21:11:46	0|BlueMatt|suresure
310 2017-12-05 21:21:36	0|jonasschnelli|BlueMatt: done
311 2017-12-05 21:26:00	0|BlueMatt|lol, how's this for an idea: try to load libsensors and if it loads write cpu temperateure to debug.log on a regular basis
312 2017-12-05 21:33:10	0|bitcoin-git|13bitcoin/06master 14a38686c 15Matt Corallo: [verify-commits] Fix gpg.sh's echoing for commits with '\n'
313 2017-12-05 21:33:10	0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/91eeaa03354b...5bea05bc1d17
314 2017-12-05 21:33:11	0|bitcoin-git|13bitcoin/06master 145bea05b 15Pieter Wuille: Merge #11834: [verify-commits] Fix gpg.sh's echoing for commits with '\n'...
315 2017-12-05 21:33:45	0|bitcoin-git|[13bitcoin] 15sipa closed pull request #11834: [verify-commits] Fix gpg.sh's echoing for commits with '\n' (06master...062017-12-verify-commits-fix) 02https://github.com/bitcoin/bitcoin/pull/11834
316 2017-12-05 21:34:02	0|gmaxwell|BlueMatt: oh no, moar dependencies.
317 2017-12-05 21:34:13	0|jimpo|Is there a preferred implementation of union/variant types? I know about boost::variant, but I've heard Core is trying to use boost less.
318 2017-12-05 21:34:23	0|BlueMatt|gmaxwell: well you could do a if-you-cant-find-dep-dont-print-temp
319 2017-12-05 21:34:49	0|BlueMatt|gmaxwell: but it'd be so nice in debugging peoples' rejecting-chain issues to see debug.log entries that say "WARNING: My hardware is overheating garbage, dont trust output"
320 2017-12-05 21:34:50	0|sipa|jimpo: std::variant in c++17 ? :)
321 2017-12-05 21:35:06	0|jimpo|Oh, is C++17 allowed/enabled?
322 2017-12-05 21:35:11	0|BlueMatt|lol no
323 2017-12-05 21:35:16	0|jimpo|derp
324 2017-12-05 21:35:16	0|sipa|no
325 2017-12-05 21:35:28	0|sipa|it was just officially published... today
326 2017-12-05 21:35:31	0|BlueMatt|is it even in debian-testing?
327 2017-12-05 21:35:43	0|BlueMatt|i think clang supports it
328 2017-12-05 21:35:45	0|sipa|needs GCC 7
329 2017-12-05 21:36:23	0|sipa|and Clang 5
330 2017-12-05 21:36:30	0|jimpo|Ah, that explains why std::variant came up in searches today and I didn't see it a month ago
331 2017-12-05 21:37:02	0|sipa|to be fair, the last month or so was just formally for ISO to publish it
332 2017-12-05 21:37:11	0|sipa|it was exactly known what the spec would be
333 2017-12-05 21:37:17	0|jimpo|So is a ghetto struct with an enum + union preferred to boost::variant?
334 2017-12-05 21:38:27	0|sipa|i think you can use boost::variant, as we're already using it anyway
335 2017-12-05 21:38:32	0|jimpo|k
336 2017-12-05 21:38:48	0|BlueMatt|if you can keep it in a cpp, use boost::variant, if its in an infection header....ugh, but maybe ok
337 2017-12-05 22:37:32	0|luke-jr|jimpo: boost is fine outside of libconsensus I think
338 2017-12-05 22:38:55	0|GAit|patches to get core to build with the ndk (working on android shell with no deps or glibc/LD_LIBRARY_PATH hacks) if anyone is interested can be reviewed at https://github.com/greenaddress/bitcoin_ndk  - warning, just like abcore, these patches are also very alpha and currently I don't think there's anything worth making an upstream PR for. and fwiw later ndk releases seem to fail.
339 2017-12-05 22:40:12	0|GAit|mostly deals with setting up the toolchain and patching some different headers or things missing in android (ifaddrs for example)
340 2017-12-05 23:10:44	0|TD-Linux|BlueMatt, I think a built-in memtest would be a better use of time :)
341 2017-12-05 23:11:09	0|BlueMatt|TD-Linux: that is almost certainly true
342 2017-12-05 23:28:24	0|phantomcircuit|iirc there are user space memtest utilities but i dont see how they can actually be effective unless they use all available memory
343 2017-12-05 23:30:21	0|phantomcircuit|BlueMatt, ^
344 2017-12-05 23:31:18	0|BlueMatt|I think, or at least have a suspicion, that half our user-memory-corruption errors are due to cpu overheating and writing garbage
345 2017-12-05 23:31:58	0|BlueMatt|but that isnt based on much and may be very bogus
346 2017-12-05 23:32:19	0|phantomcircuit|hmm
347 2017-12-05 23:32:32	0|phantomcircuit|indeed that would be caught with a pretty simple test
348 2017-12-05 23:32:47	0|phantomcircuit|just load some test vectors and check siphash results
349 2017-12-05 23:33:36	0|BlueMatt|yea, I mean its a shame we have assumevalid now...otherwise I'd suggest just busywaiting one or two of the script check threads with something like that
350 2017-12-05 23:56:03	0|gmaxwell|we could make block validation fails sleep for a second then try again... and log&crash if successful the second time.