1 2017-08-03 00:00:42	0|jnewbery|#10882 is squashed and ready for reACK/merge
  2 2017-08-03 00:00:44	0|gribble|https://github.com/bitcoin/bitcoin/issues/10882 | Keypool topup by jnewbery · Pull Request #10882 · bitcoin/bitcoin · GitHub
  3 2017-08-03 00:01:32	0|sipa|jnewbery: thanks!
  4 2017-08-03 01:49:29	0|james0909|hi, i'm having an issue with core, could someone help?
  5 2017-08-03 01:51:34	0|instagibbs|james0909, #bitcoin please, someone can help there
  6 2017-08-03 02:47:19	0|praxeology|Would it be worth adding some ECC to chainstate and the block index?  Maybe running a memtest before the first synch?
  7 2017-08-03 02:47:58	0|praxeology|Memtest could auto start on the first synch from genesis... but be skippable
  8 2017-08-03 02:48:34	0|praxeology|Although... sure memtest is better to be done w/ something like Memtest86+
  9 2017-08-03 03:56:00	0|gmaxwell|praxeology: error correction wouldn't really help unless it was very very high overhead.
 10 2017-08-03 04:03:16	0|praxeology|I guess I wasn't really calling for error correction, more error detection.  But yea I guess it already does detect an error eventually
 11 2017-08-03 04:04:52	0|gmaxwell|There is error detection.
 12 2017-08-03 04:06:05	0|praxeology|Maybe if the client gets stuck on an old block, it should suggest to the user that maybe its database is corrupted... and give some suggestions on what to do about it
 13 2017-08-03 04:12:10	0|sipa|it does
 14 2017-08-03 04:12:15	0|sipa|and asks if you want to reindex
 15 2017-08-03 04:18:24	0|praxeology|ok.  james0909 in #bitcoin said his error log said...  "ERROR: invalid header received 2017-08-03 01:48:18 ProcessMessages(headers, 11422 bytes) FAILED peer=38"?  and his tip was stuck at 478645.
 16 2017-08-03 04:18:56	0|praxeology|But he didn't say anything about it asking him to reindex... although he did ask if he should reindex.
 17 2017-08-03 04:19:49	0|gmaxwell|praxeology: why do you think there is corruption...
 18 2017-08-03 04:20:27	0|gmaxwell|praxeology: the invalid headers messages are bcash nodes getting banned. And he may not catch up if he's been offline until the next block if the peer he picked for the initial header sync was broken or malicious.
 19 2017-08-03 04:21:18	0|praxeology|he said he tried connecting to other peers.
 20 2017-08-03 04:21:48	0|gmaxwell|what does that even mean
 21 2017-08-03 04:21:49	0|praxeology|I guess I didn't confirm that he actually connected to a known good peer
 22 2017-08-03 04:22:11	0|gmaxwell|doesn't matter f he is 'connected' to a good peer, he won't continue syncing until there is a new block.
 23 2017-08-03 04:22:32	0|praxeology|oh.  I didn't know that
 24 2017-08-03 04:22:46	0|gmaxwell|(if he was behind and the peer he elected for initial headers was bogus)
 25 2017-08-03 04:22:54	0|praxeology|So he needs to be connected to a good peer while a new block comes in
 26 2017-08-03 04:23:30	0|gmaxwell|praxeology: sure, just doing nothing will do that with very high likelyhood since it'll be connected to at least 8 peers.
 27 2017-08-03 04:24:02	0|praxeology|when bitcoin starts up, it only chooses one peer to ask for latest headers?
 28 2017-08-03 04:24:23	0|sdaftuar|praxeology: yes
 29 2017-08-03 04:24:30	0|praxeology|and if that fails... it doesn't ask another?
 30 2017-08-03 04:24:31	0|gmaxwell|praxeology: right, and if doesn't give all of them it won't learn that it's behind until someone else advertises a block.
 31 2017-08-03 04:24:37	0|sdaftuar|praxeology: in 0.15, there will be a timeout on that request
 32 2017-08-03 04:24:38	0|gmaxwell|it can't tell when it 'fails'
 33 2017-08-03 04:24:56	0|gmaxwell|except for a total timeout, which sdaftuar notes is detected in 0.15
 34 2017-08-03 04:25:14	0|praxeology|hm, well if the time of his last block is really old... or if the peer he is asking has a different tip...
 35 2017-08-03 04:25:44	0|gmaxwell|praxeology: different tip than what
 36 2017-08-03 04:26:25	0|praxeology|if your own node has a different tip than the one you are requesting the latest headers from
 37 2017-08-03 04:26:47	0|gmaxwell|praxeology: of course its different or fetching headers from it was pointless...
 38 2017-08-03 04:26:48	0|praxeology|like, their tip is in a different chain or behind or something
 39 2017-08-03 04:27:08	0|praxeology|bah ok you right
 40 2017-08-03 04:27:17	0|sdaftuar|you don't request headers from a peer who is on a less-work chain than your tip
 41 2017-08-03 04:27:45	0|gmaxwell|And any countermeasure has to not do things like terminate a long header sync from a valid peer just because it's a lot of data... otherwise a newly syncing node could get stuck if it's link was too low in bandwidth.
 42 2017-08-03 04:28:08	0|sdaftuar|actually, i said that wrong.  you don't request blocks from such a peer
 43 2017-08-03 04:30:12	0|gmaxwell|praxeology: there are a bunch of places we could improve responsiveness by punting less helpful peers, but such rules require just a lot of care because false positives are potententially severe attack vectors.
 44 2017-08-03 04:35:10	0|praxeology|alright... well, its unfortunate I misdiagnosed the guy's issue.  Thanks for the help.  Hopefully he will come back and ask again on #bitcoin and one of sees him... for his sake... before he -reindex
 45 2017-08-03 04:36:56	0|gmaxwell|it's also important to not let malicious peers cause users to get any kind of error to the greatest extent that we can avoid it.
 46 2017-08-03 04:37:25	0|gmaxwell|because unfortunately any warning you show users will cause a small percentage of them to jump off bridges.
 47 2017-08-03 04:38:17	0|gmaxwell|"I saw a warning that wouldn't go away so I deleted my wallet, and now I still have the warning and where are my bitcoins?"
 48 2017-08-03 04:47:00	0|mryandao|LOL
 49 2017-08-03 04:48:24	0|gmaxwell|not really funny though. :(
 50 2017-08-03 04:48:54	0|gmaxwell|it's not like people are stupid, first they try all the things they think are reasonable, then they try other things...
 51 2017-08-03 04:49:17	0|gmaxwell|and with enough users, someone is going to think they should try sawing off their hands or whatever.
 52 2017-08-03 04:49:40	0|Emcy_|throwing your wallet in a lake isnt reasonable
 53 2017-08-03 04:49:44	0|mryandao|i can't connect the dots between deleting wallet.dat and addressing the problem.
 54 2017-08-03 04:50:09	0|gmaxwell|mryandao: because it's state that could be causing the error.
 55 2017-08-03 04:50:20	0|gmaxwell|Emcy_: but they saved the addresses first!
 56 2017-08-03 04:50:33	0|gmaxwell|you and I know that doesn't help, but it's not exactly obvious
 57 2017-08-03 04:50:42	0|Emcy_|it was even called "wallet" as a kind of linguistic skeumorph to get people to understand what it is
 58 2017-08-03 04:50:46	0|mryandao|hmm, this sounds like motivation to decouple key from wallet.dat?
 59 2017-08-03 04:51:05	0|mryandao|so at least even if wallet.dat was removed out of a panic attack, at least keys is still somewhere safe?
 60 2017-08-03 04:52:06	0|Emcy_|youre right about explaining pubkey crypto to normies though oh boy
 61 2017-08-03 04:52:16	0|gmaxwell|well we've talked before about writing wallet backup files periodically but there is a counter argument that doing so will make it more likely for users to get their wallets stolen. e.g. clear wallet off a disk before giving it to someone else...
 62 2017-08-03 04:55:37	0|Emcy_|i meant to ask why the default datadir on windows in is the domain roaming profile folder, actually, instead of the local profile
 63 2017-08-03 04:55:46	0|Emcy_|since the subject came up
 64 2017-08-03 04:57:52	0|praxeology|does bitcoin still use windows registry?
 65 2017-08-03 04:58:21	0|praxeology|windows configuration/file system usage needs a rework :p
 66 2017-08-03 04:59:56	0|praxeology|HKEY_CURRENT_USER\Software\Bitcoin\Bitcoin-Qt
 67 2017-08-03 05:00:54	0|praxeology|Not that I am actually making demands for your time, sorry.  Have a good night!
 68 2017-08-03 05:17:30	0|sipa|praxeology: why?
 69 2017-08-03 05:59:46	0|goatpig|isn't that just the URI registration?
 70 2017-08-03 06:00:48	0|praxeology|windows registry is not very compatible with multiple installations in use on the same OS
 71 2017-08-03 06:02:33	0|praxeology|magic hidden settings that one wouldn't know to move to a new system
 72 2017-08-03 08:03:58	0|gmaxwell|I wonder if we should adopt a patch to instantly disconnect bcash nodes based on their service flags.  Sucks to burn a service bit forever to their recklessness though. :(
 73 2017-08-03 08:04:16	0|gmaxwell|and presumably the constant inadvertant dos attack of connecting to nodes on another network will move them onto another port eventually.
 74 2017-08-03 08:09:05	0|praxeology|potentially in the future the burned service code could be recovered in the future after the problem goes away
 75 2017-08-03 08:09:53	0|praxeology|are they refusing to move to a different port?
 76 2017-08-03 08:10:43	0|gmaxwell|they refused people people asked them previously.
 77 2017-08-03 08:11:00	0|gmaxwell|it's not all that easy to recover a service bit where its use results in instantly being disconnected!
 78 2017-08-03 08:11:13	0|praxeology|Maybe in a month bcash won't have anyone mining it anymore
 79 2017-08-03 08:11:52	0|gmaxwell|perhaps we shouldn't worry much about burning one because we know we're due for some other pretty substantial p2p revisions, and other updates can add new capabilities flags.
 80 2017-08-03 08:12:54	0|gmaxwell|I think we only use service flags now for things we absolutely need to have in addr messages, and if we create a addr message replacement (to support NG hidden services and I2P, we'd probably give it a different capabilities signaling tool)
 81 2017-08-03 08:14:06	0|praxeology|service flags... "flags" implies a 32 bit or 64 bit number?
 82 2017-08-03 08:14:51	0|praxeology|Maybe switch to a set of service tags instead?
 83 2017-08-03 08:15:08	0|gmaxwell|well, they need to be small because they're rumored everwhere in addr messages.
 84 2017-08-03 08:15:31	0|gmaxwell|making them fat sidechannels would likely have clowns using them for file trading. :)
 85 2017-08-03 08:17:17	0|praxeology|ok, well you could put a size constraint on the tag set... but whatever I'm just sleep dep and over engineering something I don't know enough about
 86 2017-08-03 09:27:09	0|Eliel|jonasschnelli: is bitcoind's code that does that too difficulty to understand?
 87 2017-08-03 09:27:19	0|Eliel|(never mind, was looking at the past)
 88 2017-08-03 09:27:23	0|jonasschnelli|Eliel: depends on you experience
 89 2017-08-03 09:27:39	0|Eliel|ah true
 90 2017-08-03 10:03:53	0|gmaxwell|Hi all, karelb is working on the trezor wallet, and they've been trying to use these patches to bitcoin that implement the bitcore address indexing stuff, but they're finding it really slow to the point where performance is problematic.
 91 2017-08-03 10:04:10	0|karelb|we are using it for a while now :)
 92 2017-08-03 10:04:29	0|karelb|but right now we are reindexing bitcoin blockchain and it takes foreved
 93 2017-08-03 10:04:34	0|gmaxwell|Maybe someone would be interested in giving them a bit of a hand at looking at it? (I need to get to bed); I've already provided the standard disclaimers about the inherent non-scalability of address-indexes-of-all-history. :)
 94 2017-08-03 10:05:11	0|karelb|(It is actually bitcoin-abc, I hope I won't be banned :D, but this issue popped up in bitcoin core too)
 95 2017-08-03 10:05:17	0|gmaxwell|karelb: one question would be what kind of system is this running on? e.g. is it on some VPS with remote storage that may have poor IO speed?
 96 2017-08-03 10:05:47	0|gmaxwell|karelb: we'll forgive you for your sins. though obviously can't help with any abc specific issues.
 97 2017-08-03 10:05:48	0|karelb|it is our local server that has SSD, lots of RAM and processors
 98 2017-08-03 10:06:00	0|gmaxwell|darn.
 99 2017-08-03 10:06:57	0|karelb|actually one issue was sort-of ABC related, but it was because of a commit ported from master from bitcoin core, so it will be relevant anyway
100 2017-08-03 10:07:05	0|karelb|https://github.com/Bitcoin-ABC/bitcoin-abc/issues/43 , https://github.com/satoshilabs/bitcoin-abc/commit/5337f8f210eaa34d1212103f700698dd4989f479
101 2017-08-03 10:07:26	0|gmaxwell|so since I doubt that addrindex has a useful caching layer, you could look for the leveldb::Options object for the database it creates and try increasing the options.block_cache and options.write_buffer_size to really large levells.
102 2017-08-03 10:07:39	0|karelb|it's because the bitcore patches do address index in ConnectBlock/DisconnectBlock, even on the start during the testing thing
103 2017-08-03 10:08:04	0|karelb|(jpochyla is my colleague)
104 2017-08-03 10:08:42	0|gmaxwell|karelb: probably code based on Bitcoin Core master (including ABC) will not be reliably compatible with that address indexing stuff until it is changed.
105 2017-08-03 10:08:50	0|karelb|And the introduction of ApplyBlockUndo somehow caused that
106 2017-08-03 10:09:35	0|karelb|yeah. We want to write our own indexing thing that you won't need to put inside the C++ code, since that is a little insane and we need to keep on rebasing that
107 2017-08-03 10:10:13	0|karelb|gmaxwell: just fyi, ABC is not based on master, but on 0.14.1, but they took that thing from master
108 2017-08-03 10:10:17	0|gmaxwell|karelb: bitcoin core master changed the atomiticity requirements for the backend database, but a side effect of this is that it needs special replay logic to handle crash recovery.  ABC has partially ported some of these changes. I am not sure, but I wouldn't be surprised if the address indexing would get corrupted until updated to have the right synchronization behavior.
109 2017-08-03 10:10:33	0|gmaxwell|I know it's based on 0.14.1, but they copied some of these database changes.
110 2017-08-03 10:10:37	0|karelb|ok
111 2017-08-03 10:11:22	0|gmaxwell|in any case, beyond the cache options I mentioned above, I am out of ideas for making it faster without substantial design changes.
112 2017-08-03 10:11:55	0|gmaxwell|I think you should probably also make an extra effort to always cleanly start and stop that node, because I wouldn't be confident that it is durable across crashes without corruption.
113 2017-08-03 10:13:37	0|Austindoggie|Did it take a long time to reindex because you went back a version of bitcoin core?
114 2017-08-03 10:14:34	0|Austindoggie|Sorry if im not allowed to talk here...
115 2017-08-03 10:14:47	0|karelb|gmaxwell: hah, that is not a good news. We also noticed the node randomly crashes once in a while and we are not sure why
116 2017-08-03 10:15:25	0|gmaxwell|::sigh::
117 2017-08-03 10:15:29	0|karelb|yeah
118 2017-08-03 10:16:32	0|karelb|block_cache and write_buffer_size can be set via conf, or only in code?
119 2017-08-03 10:16:40	0|karelb|wait I will have a look
120 2017-08-03 10:16:44	0|gmaxwell|karelb: run more nodes, reindex anytime one crashes?  At least until you can test if the address index is accurate across crashes?  I'm not completely confident that it won't be, but it would take some careful review and testing to be sure (especially in the context of ABC that has really scrambled things up a lot)
121 2017-08-03 10:17:03	0|gmaxwell|karelb: I think only in the code, I don't think there is an external way to override.
122 2017-08-03 10:17:22	0|gmaxwell|in any case, I have to go to bed. Hopefully someone else will wake up and have some other suggestions.
123 2017-08-03 10:17:33	0|karelb|(I really hate how ABC reformatted everything for no good reason, but that is another issue from this)
124 2017-08-03 10:18:26	0|gmaxwell|yes, it makes it really hard to see what exactly has changed because the reformats were heavily intermixed with real changes. :(
125 2017-08-03 10:19:59	0|karelb|@austindoggie nope we started to download blockchain from 0. And it is always stuck on IO for insanely long times
126 2017-08-03 10:20:16	0|karelb|and normal bitcoin without bitcore patches doesn't do it, on the same HW
127 2017-08-03 10:20:40	0|gmaxwell|karelb: gross final suggestion before I really go:  if you really have a lot of ram, create a tmpfs mount big enough for the entire datadir, and sync in there, when it finishes copy it to disk.
128 2017-08-03 10:21:44	0|gmaxwell|karelb: the challenge there is that we have insanely optimized bitcoin core's sync process.. we avoid writing to leveldb at all costs, basically, and a significant fraction of UTXO never hit the database at all during normal sync because they're spent before the dbcache fills)
129 2017-08-03 10:25:01	0|karelb|thanks a lot for your help
130 2017-08-03 10:25:18	0|karelb|going to IRC was a wild shot but it seems it might help :)
131 2017-08-03 10:26:51	0|bitcoin-git|13bitcoin/06master 1449d903e 15Alex Morcos: Eliminate fee overpaying edge case when subtracting fee from recipients
132 2017-08-03 10:26:51	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/659c09613408...2e857bb619f5
133 2017-08-03 10:26:52	0|bitcoin-git|13bitcoin/06master 142e857bb 15Wladimir J. van der Laan: Merge #10942: Eliminate fee overpaying edge case when subtracting fee from recipients...
134 2017-08-03 10:27:26	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10942: Eliminate fee overpaying edge case when subtracting fee from recipients (06master...06subtractfee) 02https://github.com/bitcoin/bitcoin/pull/10942
135 2017-08-03 10:34:41	0|karelb|Hm. options.block_cache and options.write_buffer_size are derived from dbcache
136 2017-08-03 10:35:17	0|karelb|dbcache option
137 2017-08-03 10:35:36	0|jonasschnelli|Updated to Debian 9 (stretch) and suddenly get gitian build errors: init.lxc: failed to mount /dev/shm : No such file or directory
138 2017-08-03 10:35:46	0|jonasschnelli|Anyone else experiences this?
139 2017-08-03 10:40:23	0|karelb|maximum dbcache is 16384 MB hard-capped in code; is there a reason for that?
140 2017-08-03 10:40:38	0|jonasschnelli|karelb: why would you need more?
141 2017-08-03 10:47:01	0|karelb|see the past discussion... we are running bitcore address index patches and they are very slow and stuck on disk operations. And it happens probably because it gets stuck on adding things to address index and commiting it to hard disk on every block
142 2017-08-03 10:47:12	0|karelb|probably
143 2017-08-03 10:48:14	0|karelb|it must be because of the address index somehow, because on the same PC, bitcoin core without bitcore (sigh) patches runs fine
144 2017-08-03 10:49:00	0|karelb|I am actually talking about ABC, but the same issue crops up in bitcoin, plus it might be because ABC added some db logic from master
145 2017-08-03 10:53:09	0|karelb|hm, we stopped the bitcoind and it got into some crashed state which is inconsistent and nothing happens
146 2017-08-03 10:53:11	0|karelb|:/
147 2017-08-03 10:53:22	0|karelb|this is hell
148 2017-08-03 10:54:16	0|gmaxwell|karelb: the leveldb caching isn't very useful for the leveldb databases in core because we've already cached the heck out of those things at a higher level...  but for the options set for your custom address index it may be very helpful.
149 2017-08-03 11:05:08	0|karelb|well it started chugging again, but is still gets stuck. With dbcache set at 16384. Memory has 1GB full and 63GB free
150 2017-08-03 11:06:16	0|gmaxwell|karelb: what does stuck mean exactly
151 2017-08-03 11:08:44	0|karelb|Log says nothing, and iotop shows 100% and doing a lot of reading/writing in the leveldb
152 2017-08-03 11:09:11	0|karelb|and after about 20 minutes, log messages start to appear again
153 2017-08-03 11:09:59	0|gmaxwell|dear lord. :(
154 2017-08-03 11:13:45	0|karelb|and the binary doesn't reply even to kill signals
155 2017-08-03 11:29:57	0|karelb|data/blocks/index is compacting like crazy when it is stuck
156 2017-08-03 11:30:34	0|karelb|https://pastebin.com/SvsyQyiL
157 2017-08-03 11:34:06	0|karelb|The bitcoind is stuck and at the same time when this is hapenning
158 2017-08-03 11:34:24	0|karelb|and it keeps writing compacting
159 2017-08-03 13:07:49	0|bitcoin-git|13bitcoin/06master 143498a8d 15Cory Fields: depends: fix fontconfig with newer glibc...
160 2017-08-03 13:07:49	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/2e857bb619f5...e222618a32a1
161 2017-08-03 13:07:50	0|bitcoin-git|13bitcoin/06master 14e222618 15Wladimir J. van der Laan: Merge #10851: depends: fix fontconfig with newer glibc...
162 2017-08-03 13:08:14	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10851: depends: fix fontconfig with newer glibc (06master...06fontconfig-bump) 02https://github.com/bitcoin/bitcoin/pull/10851
163 2017-08-03 14:46:02	0|bitcoin-git|[13bitcoin] 15NicolasDorier opened pull request #10980: [WIP] Decouple CKeyStore from CWatchOnlyStore (06master...06decouplewatchonly) 02https://github.com/bitcoin/bitcoin/pull/10980
164 2017-08-03 15:00:33	0|Mirobit|Doesn't anyone know BCash nodes aren't sending BCash transactions to Core peers? My node has only recieved 2 BCash tx and banned both peers. But the other ~5-10 peers don't seem to send any txs. Weird?
165 2017-08-03 15:59:04	0|BlueMatt|achow101: re: #10952: Do you have any idea *how* these folks' wallets got corrupted like that?
166 2017-08-03 15:59:06	0|gribble|https://github.com/bitcoin/bitcoin/issues/10952 | [wallet] Remove vchDefaultKey and have better first run detection by achow101 · Pull Request #10952 · bitcoin/bitcoin · GitHub
167 2017-08-03 15:59:16	0|BlueMatt|I'm highly skeptical that "just write a new default key" is the right solution here
168 2017-08-03 15:59:41	0|BlueMatt|their wallet got confused somehow, auto-fixing without telling them something may be wrong is probably not a good idea
169 2017-08-03 16:55:28	0|achow101|BlueMatt: absolutely no idea how they got corrupted
170 2017-08-03 16:55:40	0|achow101|I was not able to get a copy of their wallet files either so I couldn't examine them
171 2017-08-03 16:56:24	0|achow101|BlueMatt: the solution isn't to "just write a new default key". The solution is to change the first run checker from "there's a valid default key" to "there are keys in the wallet"
172 2017-08-03 16:57:10	0|achow101|so with 10952, those wallets would not be considered to be new first run wallets as they have keys but not a valid default key
173 2017-08-03 17:06:48	0|wumpus|default key should completely go (for post-0.15 though)
174 2017-08-03 17:08:42	0|BlueMatt|achow101: well my point is more broadly that their wallets clearly got corrupted
175 2017-08-03 17:08:45	0|wumpus|the check for a new allet should be replaced by a proper "is this an empty database" check
176 2017-08-03 17:08:56	0|BlueMatt|achow101: so silently continuing isnt really the right solution
177 2017-08-03 17:09:07	0|wumpus|a wallet without any keys yet could in principle be valid
178 2017-08-03 17:09:27	0|wumpus|though a bit strange as we initially generate a mempool
179 2017-08-03 17:09:34	0|karelb|ok we got abc node working, but it was some crazy solution of switching between various binaries and invalidating nodes; it did not crash (yet). But it seems that bitcore patches inside connectblock/disconnectblock are really not the way for the future; it will probably break again (both in abc and later in bitcoin core once you release the db changes)
180 2017-08-03 17:09:35	0|wumpus|(and we don't allow mempoolsize=0)
181 2017-08-03 17:10:17	0|achow101|BlueMatt: that would require a specific check for the case that a default key is invalid
182 2017-08-03 17:10:58	0|achow101|wumpus: a database is initially created when the wallet file is created. I'm not sure how to check it is empty except that when the database is then read in, there are no keys
183 2017-08-03 17:11:17	0|achow101|that's that 10952 does; it checks if there are any keys in the database
184 2017-08-03 17:11:36	0|wumpus|yes, that sounds sane
185 2017-08-03 17:12:15	0|wumpus|for a new format I'd suggest adding a "database-type" "wallet" k/v pair to distinguish it from other bdb databases, and check for that, but for backwards compatibility that works better
186 2017-08-03 17:12:38	0|wumpus|(and no one would be so stupid to put a random other berkeleydb database as wallet.dat? right? :-)
187 2017-08-03 17:15:16	0|Lightsword|anyone working on switching bdb out for something else like sqlite?
188 2017-08-03 17:15:31	0|achow101|Lightsword: there's a pr somewhere for that I think
189 2017-08-03 17:15:38	0|sipa|meh, overkill
190 2017-08-03 17:15:55	0|sipa|all we need is a key-value store that's effectively read into memory entirely anyway
191 2017-08-03 17:16:23	0|sipa|i had a patch years ago to switch it to an append-only flat file
192 2017-08-03 17:16:28	0|Lightsword|the main advantage to sqlite though is that it has good data integrity protection
193 2017-08-03 17:16:44	0|achow101|we could upgrade to bdb whatever latest and change how records are stored there
194 2017-08-03 17:16:55	0|wumpus|Lightsword: it's quite easy to swap the database for any k/v store, I have a local branch that uses leveldb
195 2017-08-03 17:17:12	0|BlueMatt|achow101: if you have a wallet with keys, and a default key that is invalid or missing, your wallet was clearly either corrupted or generated by something other than bitcoin core
196 2017-08-03 17:17:22	0|wumpus|achow101: bdb latest (6.x) has serious license issues
197 2017-08-03 17:17:23	0|BlueMatt|achow101: continuing silently in that case is not what we want
198 2017-08-03 17:18:36	0|Lightsword|sipa, would there not be some potential future use cases where having sql support be helpful?
199 2017-08-03 17:18:37	0|achow101|BlueMatt: if there is corruption in the keys themselves or other data, that will be caught elsewhere
200 2017-08-03 17:18:48	0|BlueMatt|achow101: we can also change the "first run" check to ignore vchDefaultKey, which we clearly should, but you're pointing to wallets in the wild that have been corrupted and suggesting we should silently continue
201 2017-08-03 17:18:50	0|achow101|BlueMatt: I don't think we should care that much about corruption in the default key as it has no use
202 2017-08-03 17:19:02	0|sipa|Lightsword: i don't wee how
203 2017-08-03 17:19:04	0|sipa|*see
204 2017-08-03 17:19:13	0|BlueMatt|achow101: yes we should!
205 2017-08-03 17:19:19	0|achow101|BlueMatt: only one of those wallets would silently continue, and the guy had no problems with older versions of core
206 2017-08-03 17:19:20	0|BlueMatt|the user should stop using that computer for a wallet!
207 2017-08-03 17:19:28	0|wumpus|sql could be useful for metadata kind of things, but meh, I don't think there's really an advantage to it for us we don't index anything
208 2017-08-03 17:19:31	0|achow101|BlueMatt: the other wallet ran into other corruption problems
209 2017-08-03 17:19:38	0|BlueMatt|that user should be told to throw out their hard drive and get a better one
210 2017-08-03 17:19:40	0|BlueMatt|or a new computer
211 2017-08-03 17:19:42	0|wumpus|although the current 'keep everything in memory' is kind of dumb
212 2017-08-03 17:19:43	0|BlueMatt|not continue
213 2017-08-03 17:20:30	0|Lightsword|yeah, I was thinking maybe metadata/multiple accounts or something along those lines could possibly make use of sql
214 2017-08-03 17:20:34	0|wumpus|if the wallet would keep things in the database instead and query them when needed, indexes could be useful
215 2017-08-03 17:20:49	0|wumpus|accounts are deprecated, if anything we're simplifying the wallet
216 2017-08-03 17:21:45	0|achow101|BlueMatt: sure they should be warned, but a corrupted default key should not be a reason to halt the software entirely as default key is useless
217 2017-08-03 17:21:54	0|Lightsword|wumpus, what about multiwallet?
218 2017-08-03 17:22:03	0|sipa|Lightsword: that uses multiple wallet files
219 2017-08-03 17:22:11	0|achow101|with older versions of core, IIRC the default key would have either been overwritten or ignored and allowed the user to continue
220 2017-08-03 17:22:14	0|BlueMatt|achow101: if someone's wallet is corrupted, we should exit with an error....if they wish to then restart with -salvagewallet or equivalent, that is also ok
221 2017-08-03 17:22:23	0|BlueMatt|achow101: and that is a bug!
222 2017-08-03 17:22:54	0|BlueMatt|if we read something and see that someone's hardware is silently corrupting their wallet, we should exit the same way we do with any other wallet corruption errors
223 2017-08-03 17:23:00	0|BlueMatt|silently fixing wallets is not ok
224 2017-08-03 17:23:30	0|sipa|wumpus: well if we expect to ever need indexes on the future because we won't keep everything in memory, i'd say that sqlite is a good choice
225 2017-08-03 17:23:33	0|wumpus|with older versions of core the lack of the default key record would make it assume it's a new wallet, which could do all kinds of bad things? I don't think that's abetter
226 2017-08-03 17:23:52	0|BlueMatt|there are clearly bugs here that achow101 identified (that i think need fixing for 15)
227 2017-08-03 17:24:00	0|sipa|wumpus: not sure if it's worthwhile though
228 2017-08-03 17:24:03	0|BlueMatt|I'm just not sure that silently correcting /anything/ is every good in wallets
229 2017-08-03 17:24:05	0|wumpus|for 0.15 it's too lte imo
230 2017-08-03 17:24:15	0|wumpus|we should do rc1 asap
231 2017-08-03 17:24:16	0|BlueMatt|wumpus: we've seen it in the wild, and it can be a simple fix
232 2017-08-03 17:24:17	0|BlueMatt|:(
233 2017-08-03 17:24:25	0|wumpus|not add new stuff
234 2017-08-03 17:24:28	0|sipa|have we even identified the bug?
235 2017-08-03 17:24:39	0|wumpus|but that's just my opinion (and being harried at all sides to do 0.15 asap)
236 2017-08-03 17:24:42	0|BlueMatt|sipa: as far as anyone knows its hardware/bdb silently corrupting things
237 2017-08-03 17:24:43	0|sipa|as opposed to "we have ween a wallet with no default key, somehow"
238 2017-08-03 17:24:44	0|achow101|sipa: I can replicate the problem, not necessarily the cause
239 2017-08-03 17:25:20	0|wumpus|but sure if it's a clearly defined problem, with a clearly defined fix, and it can be reviewed in the next days, we can include it
240 2017-08-03 17:25:21	0|sipa|achow101: elaborate
241 2017-08-03 17:25:46	0|achow101|sipa: encrypt a wallet, use db_dump to dump it, remove the default key, load with db_load, start core, runtime exception
242 2017-08-03 17:26:04	0|sipa|well, ok
243 2017-08-03 17:26:05	0|wumpus|that's what I expect if you just remove a record
244 2017-08-03 17:26:10	0|wumpus|don't do that.
245 2017-08-03 17:26:22	0|sipa|but if you permit random file changes, you can make anything happen
246 2017-08-03 17:26:26	0|BlueMatt|wumpus: yes, sorry, my understanding is its that kind of "simple fix" that would be easy to get in in a day or two
247 2017-08-03 17:26:29	0|achow101|if you do it unencrypted, it works fine
248 2017-08-03 17:26:30	0|wumpus|if you delete a random record from the utxo database you'll also be in for a world of pain
249 2017-08-03 17:27:00	0|BlueMatt|the issue is that this encourages people to downgrade wallets
250 2017-08-03 17:27:03	0|wumpus|we're not resistant to that kind of corruption, and one special case is not going to change that
251 2017-08-03 17:27:14	0|achow101|I suppose then a check for a valid default key can be added and that would just be a corruption warning?
252 2017-08-03 17:27:16	0|BlueMatt|just making it a nice error message and making sure -salvagewallet works correctly is the Correct (tm) fix, imo
253 2017-08-03 17:27:17	0|wumpus|could just as well have been a key record that disappears
254 2017-08-03 17:27:26	0|BlueMatt|achow101: yes
255 2017-08-03 17:27:32	0|achow101|fine
256 2017-08-03 17:27:35	0|wumpus|salvagewallet should work as long as there is any private key left
257 2017-08-03 17:27:45	0|achow101|btw the current corruption warnings are kinda bad
258 2017-08-03 17:27:56	0|wumpus|that's a known issue...
259 2017-08-03 17:28:19	0|wumpus|there's even a chance of running into asserts on db corruption
260 2017-08-03 17:29:31	0|wumpus|would be nice if it could detect corruption at run time and offer the user a chance to repair
261 2017-08-03 17:29:57	0|wumpus|though if a wallet gets corrupted that's a good indication you should stop using that computer for bitcoin, now
262 2017-08-03 17:30:21	0|wumpus|so I agree with BlueMatt in that regard
263 2017-08-03 17:31:32	0|achow101|the main problem I have with that is it encourages people who don't know what they are doing (i.e. don't ask, can't use -salvagewallet, etc.) to downgrade
264 2017-08-03 17:31:44	0|sipa|?
265 2017-08-03 17:31:55	0|wumpus|why would they think downgrading would work?
266 2017-08-03 17:32:01	0|achow101|wumpus: because it does.
267 2017-08-03 17:32:04	0|wumpus|say, your wallet is suddenly corrupted one day
268 2017-08-03 17:32:17	0|wumpus|why would you consider downgrading?
269 2017-08-03 17:32:27	0|wumpus|only if it happens on upgrade I suppose
270 2017-08-03 17:32:29	0|achow101|with this specific corruption problem, downgrading to non-hd wallet software (0.12.0- or -usehd=0) will ignore this problem
271 2017-08-03 17:32:31	0|sipa|if the corruption ransomly happens right after uograding, i can see why someone would think that
272 2017-08-03 17:32:42	0|sipa|*randomly
273 2017-08-03 17:32:46	0|sipa|not ransomly
274 2017-08-03 17:32:57	0|wumpus|yes, if it happens right after upgrading, but then it's probably a version issue and not a random corruption in any case, would be extremely unlikely otherwise
275 2017-08-03 17:33:14	0|sipa|achow101: update wallet version number?
276 2017-08-03 17:33:19	0|BlueMatt|anyway, this is why i think we should give an error message
277 2017-08-03 17:33:20	0|achow101|sipa: bleh
278 2017-08-03 17:33:29	0|sipa|oh, we can't because the optional stuff uses it
279 2017-08-03 17:33:41	0|BlueMatt|the crash is better than continuing silently, but if it makes people downgrade, we should instead tell them whats up
280 2017-08-03 17:33:41	0|sipa|we really need wallet features grr
281 2017-08-03 17:34:18	0|achow101|IIRC salvagewallet didn't fix their problems either, so that will need to be updated
282 2017-08-03 17:34:35	0|BlueMatt|sipa: arent you the one who added wallet versioning?
283 2017-08-03 17:34:45	0|BlueMatt|achow101: yes, lets do that :)
284 2017-08-03 17:35:05	0|wumpus|wallet versioning is used for *optional* features?
285 2017-08-03 17:35:13	0|BlueMatt|oh, no
286 2017-08-03 17:35:17	0|achow101|I think it would be fine to write some random default key right? it isn't used for anything
287 2017-08-03 17:35:30	0|achow101|wumpus: yes, hd and hd chain split or optional, but they have version numbers
288 2017-08-03 17:35:39	0|BlueMatt|achow101: its used for pay-to-IP :p
289 2017-08-03 17:36:03	0|BlueMatt|(I think)
290 2017-08-03 17:36:08	0|sipa|BlueMatt: yes, and now we can't use it
291 2017-08-03 17:36:19	0|sipa|because the version number is used for optional features
292 2017-08-03 17:36:59	0|sipa|like HD and split HD
293 2017-08-03 17:37:26	0|sipa|those should have been a separate record, rather the version number
294 2017-08-03 17:37:32	0|wumpus|would have been better to use a new key to indicate those (as well as bump the version number, but not in itself)
295 2017-08-03 17:37:47	0|wumpus|well teh version should be bumped too to make it incompatible, but yeah
296 2017-08-03 17:38:10	0|sipa|right
297 2017-08-03 17:39:36	0|BlueMatt|Make Bitcoin Great Again: Bring back Pay-2-IP
298 2017-08-03 17:40:32	0|sipa|BlueMatt: i tried :(
299 2017-08-03 17:40:54	0|wumpus|with .onion addresses or some other way to authenticate an IP at the transport layer (such as ipsec) it even could work securely!
300 2017-08-03 17:41:07	0|BlueMatt|wumpus: I'm currently working on tcpcrypt for linux patchset :p
301 2017-08-03 17:41:28	0|wumpus|BlueMatt: I have no idea what that is
302 2017-08-03 17:41:38	0|BlueMatt|wumpus: tcp-crypt :p
303 2017-08-03 17:42:19	0|sipa|en..crypt...TCP?
304 2017-08-03 17:42:25	0|Lightsword|http://www.tcpcrypt.org/
305 2017-08-03 17:42:26	0|BlueMatt|yes, that
306 2017-08-03 17:43:35	0|sipa|ah, nice
307 2017-08-03 17:45:18	0|wumpus|"and your network connections will continue to work even if the remote end does not support Tcpcrypt, in which case connections will gracefully fall back to standard clear-text TCP" - that screams downgrade attack :)
308 2017-08-03 17:46:11	0|achow101|oh damn, now default key can't be removed :(
309 2017-08-03 17:46:13	0|sipa|wumpus: read on
310 2017-08-03 17:46:21	0|sipa|achow101: ?
311 2017-08-03 17:46:41	0|sipa|wumpus: it only protects against passive attacks
312 2017-08-03 17:46:44	0|achow101|sipa: to check if default key is valid, it needs to be in CWallet
313 2017-08-03 17:46:52	0|sipa|achow101: ?
314 2017-08-03 17:46:57	0|BlueMatt|wumpus: yes, if the application layer does nto support it it only protects against passive mitm attacks, if the application layer supports it you can disconnect if tcpcrypt failed (but dont do that, cause middleware sucks)
315 2017-08-03 17:47:18	0|achow101|I need some way to get default key out of database into a checker, which also checks if default key exists
316 2017-08-03 17:47:29	0|sipa|achow101: you can just check when loading the wallet
317 2017-08-03 17:47:31	0|achow101|the only vehicle for that right now is cwallet I think
318 2017-08-03 17:48:06	0|sipa|i'm sure there is or can be a LoadDefaultKey function on CWallet, called from walletdb
319 2017-08-03 17:48:19	0|sipa|switch that to do a check, but not store
320 2017-08-03 17:48:58	0|achow101|I guess I could also just add more parameters too
321 2017-08-03 18:14:29	0|earlz|I'm trying to get gitian working with 0.14.0. Has anyone seen an error like this? lxcbr0: ERROR while getting interface flags: No such device
322 2017-08-03 18:14:32	0|earlz|SIOCSIFADDR: No such device
323 2017-08-03 18:14:57	0|earlz|just following the gitian-building.md document exactly and getting that error
324 2017-08-03 18:18:19	0|earlz|ifconfig
325 2017-08-03 18:18:23	0|earlz|er, woops
326 2017-08-03 18:19:48	0|earlz|do I need a kernel flag or something? It's failing at the sudo ifconfig lxcbr0 up 10.0.2.2 bit
327 2017-08-03 18:20:44	0|sipa|do you have the lxc kernel module loaded?
328 2017-08-03 18:21:20	0|earlz|I pretty much just followed the gitian build instructions and I don't see anywhere in them that the kernel module was explicitly installed unless the lxc package does that
329 2017-08-03 18:25:00	0|wumpus|looks like you don't have a lxcbr0 interface for bridging between the lxc and host
330 2017-08-03 18:25:10	0|wumpus|not sure how and where that gets created though
331 2017-08-03 18:25:38	0|earlz|yea, I'm not understanding where that gets created in the guide. Last time I setup gitian it was for Bitcoin 0.9 so not sure what all has changed since then
332 2017-08-03 18:27:54	0|earlz|maybe related: https://github.com/lxc/lxc/issues/1218
333 2017-08-03 18:28:17	0|earlz|LXC seems to be broken in some subtle way every time I try to do a new setup of gitian
334 2017-08-03 18:28:37	0|sipa|wumpus, BlueMatt: i think we really need to fix the versioning issues
335 2017-08-03 18:28:48	0|BlueMatt|wallet versioning?
336 2017-08-03 18:28:55	0|sipa|that would make it so much easier (you could say, past wallet version X, no default key is needed anymore)
337 2017-08-03 18:29:03	0|sipa|i want to move hd to a separate record
338 2017-08-03 18:29:13	0|sipa|rather than store it in the version number
339 2017-08-03 18:29:42	0|sipa|because we can't upgrade the wallet format, ever, now
340 2017-08-03 18:29:51	0|sipa|in a backward incompatible way
341 2017-08-03 18:29:56	0|BlueMatt|version number = backwards-incompatible change (ie hd), new-key is backwards-compatible extra features
342 2017-08-03 18:29:58	0|sipa|without turning the wallet into an hd wallet
343 2017-08-03 18:30:06	0|sipa|yes, that's what it's supposed to be
344 2017-08-03 18:30:11	0|sipa|but HD is a version number
345 2017-08-03 18:30:12	0|BlueMatt|so?
346 2017-08-03 18:30:23	0|sipa|we can't increment the version number to indicate something incompatible now
347 2017-08-03 18:30:26	0|BlueMatt|what do you want to add that needs to upgrade without being hd?
348 2017-08-03 18:30:34	0|sipa|anything
349 2017-08-03 18:30:38	0|BlueMatt|we should implement hd upgrade
350 2017-08-03 18:30:46	0|BlueMatt|thats something we very much need to do anyway
351 2017-08-03 18:30:52	0|sipa|what does that mean?
352 2017-08-03 18:31:13	0|BlueMatt|taking a non-hd wallet and adding an hd key
353 2017-08-03 18:31:34	0|sipa|yes, ok, that too
354 2017-08-03 18:31:44	0|sipa|but do you want to force everyone to have an hd wallet?
355 2017-08-03 18:31:51	0|BlueMatt|(probably also need an hd-key-rotate option, but thats separate and I think not related to hd)
356 2017-08-03 18:31:52	0|BlueMatt|^
357 2017-08-03 18:32:08	0|BlueMatt|I'm happy to force everyone into an hd wallet if we have an hd-master-key-rotate option
358 2017-08-03 18:32:17	0|sipa|yeah, ok...
359 2017-08-03 18:32:24	0|sipa|but those are all bigger features
360 2017-08-03 18:32:31	0|BlueMatt|well i think all of those are relatively limited code changes
361 2017-08-03 18:32:45	0|BlueMatt|and at least hd-master-key-rotate can happen with no format change, I think
362 2017-08-03 18:32:53	0|sipa|i am talking about 0.15
363 2017-08-03 18:32:56	0|BlueMatt|even if "big" features
364 2017-08-03 18:33:12	0|BlueMatt|do we need to add anything else? why rush the no vchdefaultkey thing?
365 2017-08-03 18:33:16	0|sipa|otherwise we'll complicate things further for ourselves
366 2017-08-03 18:33:24	0|sipa|to add a compatibility layer for split hd
367 2017-08-03 18:33:38	0|BlueMatt|ok, I'm missing something...need context
368 2017-08-03 18:33:40	0|sipa|no, ignore the vchdefaultkey thing
369 2017-08-03 18:34:07	0|sipa|hd and hdsplit, i think, are optional features - things that people can choose not to use
370 2017-08-03 18:34:32	0|BlueMatt|now, yes, but i have no problem with them not being optional in the future
371 2017-08-03 18:34:36	0|sipa|as they break existing wallet's expectation
372 2017-08-03 18:35:44	0|sipa|hmm, ok
373 2017-08-03 18:36:00	0|BlueMatt|not being optional in our traditional -walletupgrade sense, that is
374 2017-08-03 18:36:41	0|BlueMatt|i mean I'm happy for someone to disagree, I just dont see much downside to it (as long as we have an hd-master-key-rotate option and good documentation on it)
375 2017-08-03 18:36:52	0|sipa|i think i agree
376 2017-08-03 18:37:03	0|sipa|except i'm not sure we'll have that feature implemented by the time we need it
377 2017-08-03 18:37:19	0|BlueMatt|well hd-master-key-rotate is ~trivial with today's format
378 2017-08-03 18:37:25	0|sipa|okay
379 2017-08-03 18:37:30	0|BlueMatt|hd-upgrade may be slightly less so, I havent thought about it
380 2017-08-03 18:37:36	0|BlueMatt|but I expect it to be
381 2017-08-03 18:37:50	0|sipa|in that case, i guess it makes sense that those things are in fact in the version number
382 2017-08-03 18:39:17	0|sipa|i just wasn't seeing hd as a 'next version', and more as an optional but recommended feature
383 2017-08-03 18:40:55	0|BlueMatt|i mean its possible i did /because/ of our versioning scheme, but it is simpler to see it as such and there seem to be relatively limited downsides for it, given some code that doesnt exist yet :p
384 2017-08-03 18:53:32	0|wumpus|I'd say using hdsplit isn't really optional, given that you're using hd
385 2017-08-03 18:54:07	0|wumpus|hdsplit is a pure improvement on hd
386 2017-08-03 18:57:14	0|sipa|agree
387 2017-08-03 18:57:35	0|sipa|but do we support upgrading from hd to hdsplit?
388 2017-08-03 18:57:38	0|sipa|(right now)
389 2017-08-03 18:58:30	0|gmaxwell|ugh. how could we except via invalidating backups which people wouldn't expect...
390 2017-08-03 18:58:49	0|wumpus|no, we don't support that right now
391 2017-08-03 18:59:02	0|wumpus|hdsplit was sort-of rushed to make 0.15, so that at least new wallets would use it
392 2017-08-03 18:59:28	0|sipa|so, until we find a way to (forcefully) upgrade non-hd and hd to hdsplit, we should consider hdsplit an optional feature
393 2017-08-03 18:59:43	0|wumpus|but it's strictly superior to hd without hdsplit, so there should be no way to choose that for new wallets
394 2017-08-03 18:59:55	0|gmaxwell|I don't think forcefully upgrading is at all possible, because it will invalidate backups.
395 2017-08-03 19:00:12	0|wumpus|it wouldn't need to be 'forceful' upgrading, just a *way*
396 2017-08-03 19:00:35	0|wumpus|and in any case we don't ever automatically upgrade wallets
397 2017-08-03 19:00:41	0|wumpus|(because of backwards compatibility)
398 2017-08-03 19:00:46	0|gmaxwell|switch to using segwit, and that will upgrade you. :)
399 2017-08-03 19:00:55	0|lightningbot|Meeting started Thu Aug  3 19:00:54 2017 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
400 2017-08-03 19:00:55	0|lightningbot|Useful Commands: #action #agreed #help #info #idea #link #topic.
401 2017-08-03 19:00:55	0|wumpus|#startmeeting
402 2017-08-03 19:01:06	0|achow101|hi
403 2017-08-03 19:01:19	0|jonasschnelli|hi
404 2017-08-03 19:01:21	0|sipa|hi
405 2017-08-03 19:01:25	0|wumpus|#bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101
406 2017-08-03 19:01:37	0|kanzure|hi.
407 2017-08-03 19:01:49	0|cfields|hi
408 2017-08-03 19:01:49	0|jtimon|hi
409 2017-08-03 19:01:51	0|wumpus|0.15.0rc1 is planned for the 6th (next sunday), so let's go over the open issues again
410 2017-08-03 19:02:50	0|wumpus|there's not much anymore
411 2017-08-03 19:02:58	0|wumpus|https://github.com/bitcoin/bitcoin/milestones/0.15.0
412 2017-08-03 19:03:16	0|paveljanik|Hi
413 2017-08-03 19:03:18	0|wumpus|(and the scripted-diffs are option, and should be done at the last minute to not conflict with anything else)
414 2017-08-03 19:03:46	0|wumpus|Keypool topup #10882  is the most complicated PR open still, but should be almost ready
415 2017-08-03 19:03:50	0|gribble|https://github.com/bitcoin/bitcoin/issues/10882 | Keypool topup by jnewbery · Pull Request #10882 · bitcoin/bitcoin · GitHub
416 2017-08-03 19:04:02	0|BlueMatt|https://github.com/bitcoin/bitcoin/issues/10977
417 2017-08-03 19:04:03	0|BlueMatt|could go
418 2017-08-03 19:04:09	0|BlueMatt|(makes test_bitcoin valgrind-better)
419 2017-08-03 19:04:12	0|BlueMatt|and is trivial
420 2017-08-03 19:04:33	0|jnewbery|yeah, I've been working on 10882 today. Should be able to push my commits in the next hour or two
421 2017-08-03 19:04:35	0|wumpus|I'm not really fishing for new things to add to 0.15
422 2017-08-03 19:04:54	0|gmaxwell|jnewbery made a suggestion to fix my outstanding concern in review.
423 2017-08-03 19:05:01	0|wumpus|but if there are things that could be merged without affecting anything else that's ok
424 2017-08-03 19:05:08	0|jtimon|after #10758, #10919 seems simple to review, it's only +14-13
425 2017-08-03 19:05:09	0|gribble|https://github.com/bitcoin/bitcoin/issues/10758 | Fix some chainstate-init-order bugs. by TheBlueMatt · Pull Request #10758 · bitcoin/bitcoin · GitHub
426 2017-08-03 19:05:10	0|gribble|https://github.com/bitcoin/bitcoin/issues/10919 | Fix more init bugs. by TheBlueMatt · Pull Request #10919 · bitcoin/bitcoin · GitHub
427 2017-08-03 19:05:35	0|sipa|it's also already marked 0.15
428 2017-08-03 19:05:36	0|cfields|i think there's a one-liner that could be used to fix the issue in 10977, if it's deemed too much of a change
429 2017-08-03 19:05:41	0|gmaxwell|BlueMatt: I'd like to see 10977 fixed! but darn I wish that patch was smaller and easier to review.
430 2017-08-03 19:05:50	0|wumpus|yes, just needs some ACKs
431 2017-08-03 19:05:56	0|BlueMatt|could be smaller, but is easy to review imo
432 2017-08-03 19:05:58	0|sdaftuar|there's one commit in #10919 that i'm hoping others can review/ack
433 2017-08-03 19:05:59	0|gribble|https://github.com/bitcoin/bitcoin/issues/10919 | Fix more init bugs. by TheBlueMatt · Pull Request #10919 · bitcoin/bitcoin · GitHub
434 2017-08-03 19:07:15	0|wumpus|which one?
435 2017-08-03 19:07:21	0|BlueMatt|the first, i believe
436 2017-08-03 19:07:39	0|sdaftuar|yep
437 2017-08-03 19:08:14	0|wumpus|the threadgroup one? seems obviously sane to me, though it does mean things need to be interrupted too
438 2017-08-03 19:08:45	0|BlueMatt|well i think the point is that there is a comment there that notes we dont do it "because dragons"
439 2017-08-03 19:09:01	0|BlueMatt|i believe strongly that it is safe, and qt does it, but "dragons"
440 2017-08-03 19:09:05	0|wumpus|it is very bad practice not to wait for threads before exiting
441 2017-08-03 19:09:38	0|wumpus|yes, qt does it already, it's somewhat less scared of dragons it seems :)
442 2017-08-03 19:09:52	0|BlueMatt|isnt qt's logo a dragon or something?
443 2017-08-03 19:10:09	0|cfields|heh, think you're thinking of kde
444 2017-08-03 19:10:14	0|BlueMatt|oh, yea
445 2017-08-03 19:10:23	0|wumpus|(e.g. due to qt's handling of shutdown we also needed #10832)
446 2017-08-03 19:10:25	0|gribble|https://github.com/bitcoin/bitcoin/issues/10832 | init: Factor out AppInitLockDataDirectory and fix startup core dump issue by laanwj · Pull Request #10832 · bitcoin/bitcoin · GitHub
447 2017-08-03 19:10:35	0|wumpus|anyhow that commit looks good to me, I don't think there's any dragons left
448 2017-08-03 19:11:25	0|sdaftuar|sounds-good-to-me-ack
449 2017-08-03 19:12:02	0|wumpus|ok, wow, that seems to be all that is left between here and 0.15.0rc1
450 2017-08-03 19:12:13	0|BlueMatt|!
451 2017-08-03 19:12:20	0|cfields|:)
452 2017-08-03 19:13:11	0|morcos|wumpus: i had an assert crash this morning, i imagine it'll be a simple bug.. hopefully i'll have a PR this afternoon, just haven't had time to look at it yet
453 2017-08-03 19:13:11	0|wumpus|(there's another PR #10971 by cfields for fixing depends builds, but I don't think that needs disussion, it's a one-liner in the build system)
454 2017-08-03 19:13:13	0|gribble|https://github.com/bitcoin/bitcoin/issues/10971 | build: fix missing warnings and sse42 in depends builds by theuni · Pull Request #10971 · bitcoin/bitcoin · GitHub
455 2017-08-03 19:13:46	0|wumpus|morcos: ouch, can you open an issue to track it?
456 2017-08-03 19:13:48	0|cfields|yea, nothing major
457 2017-08-03 19:14:06	0|morcos|yes will open one or the other shortly
458 2017-08-03 19:14:30	0|wumpus|ok, thanks
459 2017-08-03 19:15:07	0|wumpus|do we need any updates to bips.md for 0.15?
460 2017-08-03 19:15:19	0|sipa|hmm, good question
461 2017-08-03 19:15:20	0|wumpus|(that's the item in the release process that still has a ? here)
462 2017-08-03 19:15:47	0|BlueMatt|is there a bip that recommends hd split?
463 2017-08-03 19:15:54	0|sipa|BlueMatt: bip32? :p
464 2017-08-03 19:16:16	0|wumpus|there's also "Update `src/chainparams.cpp` chainTxData with statistics about the transaction count and rate." left
465 2017-08-03 19:16:30	0|sipa|want me to do that?
466 2017-08-03 19:16:32	0|wumpus|and the BLOCK_CHAIN_SIZE, but that's straightforward
467 2017-08-03 19:16:50	0|wumpus|yes, if you know what's exactly to be done there that'd help :)
468 2017-08-03 19:17:01	0|sipa|sure
469 2017-08-03 19:18:38	0|wumpus|thanks
470 2017-08-03 19:18:56	0|sipa|short topic: bip173 unit tests issue
471 2017-08-03 19:19:10	0|wumpus|#topic bip173 unit tests issue
472 2017-08-03 19:19:14	0|jnewbery|There are a few more things for release notes
473 2017-08-03 19:19:44	0|sipa|so, bip173 specifies how to translate address strings to witness version/program, and defers to bip141 for encoding that to scriptPubKeys
474 2017-08-03 19:20:00	0|sipa|however, the unit tests actually test the whole step from address to scriptPubKey
475 2017-08-03 19:20:07	0|sipa|turns out, incorrectly
476 2017-08-03 19:20:27	0|sipa|the tests and reference implementation (of the tests) was wrong, and every reimplementation copied it
477 2017-08-03 19:21:05	0|gmaxwell|The the error was that it confused hex and dec values.
478 2017-08-03 19:21:07	0|sipa|i've made a PR to update the BIP, and all reference implementations i could find, but this is kind of scary
479 2017-08-03 19:21:17	0|cfields|corner-cases wrong? or in all cases?
480 2017-08-03 19:21:25	0|wumpus|jnewbery: agreed, but release notes need to be finished before -final, not -rc1, so it's not a blocker
481 2017-08-03 19:21:36	0|sipa|cfields: it assumed OP_n was encoded as 0x80 + n, rather than 80 + n
482 2017-08-03 19:21:57	0|BlueMatt|sipa: so they generate garbage scripts?
483 2017-08-03 19:22:02	0|jnewbery|got it. Thanks wumpus
484 2017-08-03 19:22:07	0|sipa|BlueMatt: the tests, yes
485 2017-08-03 19:22:27	0|gmaxwell|cfields: It was wrong for witness program versions other than 0
486 2017-08-03 19:22:27	0|sipa|the code itself doesn't contain a conversion to scriptPubKey at all, only a conversion to witness version/program
487 2017-08-03 19:22:35	0|cfields|yikes
488 2017-08-03 19:22:40	0|wumpus|oops
489 2017-08-03 19:22:43	0|gmaxwell|so this could happily have been deployed and started causing problems when v1 was used.
490 2017-08-03 19:22:47	0|sipa|but the tests contain a version/program -> scriptPubKey converter in order to be able the test
491 2017-08-03 19:23:04	0|BlueMatt|well if it generated garbage scripts, not much that can be done but fix it
492 2017-08-03 19:23:11	0|BlueMatt|are you asking if we should like change the prefix now?
493 2017-08-03 19:23:24	0|sipa|no
494 2017-08-03 19:23:29	0|sipa|just raising awareness
495 2017-08-03 19:23:33	0|BlueMatt|ok, cool
496 2017-08-03 19:23:38	0|gmaxwell|Also, it highlighes an implementation footgun, I suggested some warning text in the BIP itself.   One protection here is that the particular error in sipas' code would result in non-standard outputs.
497 2017-08-03 19:23:38	0|sdaftuar|perhaps an email to the -dev list would also be good?
498 2017-08-03 19:23:43	0|sipa|sdaftuar: yes
499 2017-08-03 19:24:07	0|gmaxwell|BlueMatt: I did make a suggestion that we consider changing it to break the checksum, but there doesn't appear to be reason to.
500 2017-08-03 19:24:19	0|BlueMatt|ok
501 2017-08-03 19:24:25	0|morcos|just to be clear what we are talking about, we're not talking about anything merged into Core, but code referenced from the BIP
502 2017-08-03 19:24:26	0|BlueMatt|awareness raised!
503 2017-08-03 19:24:28	0|gmaxwell|Especially since if someone had slavishly reimplemented the error in the reference, they'd produce non-standard outputs.
504 2017-08-03 19:24:32	0|sipa|morcos: indeed.
505 2017-08-03 19:24:53	0|morcos|still, a bit scary
506 2017-08-03 19:25:07	0|sipa|(though i'd like to PR it to core soon - apparently last week it was suggested to do that in 0.15.1?)
507 2017-08-03 19:25:22	0|gmaxwell|Don't start a debate about the name of the version. :P
508 2017-08-03 19:25:25	0|sipa|haha
509 2017-08-03 19:25:43	0|sipa|(though i'd like to PR it to core soon - apparently last week it was suggested to do that in some soon next version)
510 2017-08-03 19:25:51	0|gmaxwell|It also suggests that BIP173 support's test plan should include testing it with other witness version numbers. :)
511 2017-08-03 19:25:52	0|sdaftuar|prs welcome :)
512 2017-08-03 19:26:11	0|gmaxwell|sipa: well fix the testing shortfalls I found in the C++ version please. :)
513 2017-08-03 19:26:17	0|wumpus|PRs to improve the tests are always welcome anyhow
514 2017-08-03 19:26:18	0|sipa|gmaxwell: of course
515 2017-08-03 19:26:23	0|sipa|anyway, end topic
516 2017-08-03 19:26:29	0|wumpus|ok, other topics?
517 2017-08-03 19:28:04	0|gmaxwell|uh
518 2017-08-03 19:28:05	0|gmaxwell|yes.
519 2017-08-03 19:28:15	0|gmaxwell|So service bits and altcoins.
520 2017-08-03 19:28:22	0|wumpus|#topic service bits and altcoins
521 2017-08-03 19:28:30	0|BlueMatt|wait are altcoins using serice bits now?
522 2017-08-03 19:28:41	0|BlueMatt|oh, right 2x did
523 2017-08-03 19:28:44	0|gmaxwell|Bcash is using our port, p2pmagic, etc. They distinguish themselve with a blinking service bit.
524 2017-08-03 19:28:45	0|BlueMatt|what is wrong with people
525 2017-08-03 19:28:54	0|gmaxwell|(also 2x will do this too it seems)
526 2017-08-03 19:29:04	0|BlueMatt|gmaxwell: can someone open a pr to change this? or do they refuse to work properly?
527 2017-08-03 19:29:05	0|cfields|gmaxwell: i was under the impression that they were planning to change the magic soon
528 2017-08-03 19:29:10	0|gmaxwell|We mostly ban them when they send us transactions or headers.
529 2017-08-03 19:29:20	0|gmaxwell|cfields: not when I looked three days ago, maybe now.
530 2017-08-03 19:29:37	0|karelb|OK, maybe I will ask here. What format are the bitcoin .dat files in data/blocks/*.dat? is that leveldb? what is it exactly?
531 2017-08-03 19:29:48	0|gmaxwell|If so then the issue becomes moot, otherwise I was going to suggest we ban these bits on connect. The downside is we lose the bits basically forever.
532 2017-08-03 19:29:48	0|jonasschnelli|karelb: meeting, not now
533 2017-08-03 19:29:48	0|sipa|they are p2p network format
534 2017-08-03 19:29:51	0|karelb|ok sorry
535 2017-08-03 19:29:53	0|sipa|oops, yes, layer
536 2017-08-03 19:30:03	0|karelb|sorry going out
537 2017-08-03 19:30:18	0|BlueMatt|gmaxwell: yes, first should be someone bludgening them to work properly
538 2017-08-03 19:30:24	0|BlueMatt|gmaxwell: before we start burning service bits
539 2017-08-03 19:30:44	0|gmaxwell|BlueMatt: people have been since before they started. Obviously I'll go monitor but I'm not super confident.
540 2017-08-03 19:30:52	0|sdaftuar|gmaxwell: why not just ban for eg the next 3 months or something?
541 2017-08-03 19:30:55	0|achow101|BlueMatt: gmaxwell IIRC they will be changing their magic and port
542 2017-08-03 19:30:57	0|sdaftuar|if we need to do anything at all
543 2017-08-03 19:31:04	0|achow101|dunno about 2x
544 2017-08-03 19:31:05	0|BlueMatt|achow101: what about 2x?
545 2017-08-03 19:31:05	0|gmaxwell|One reason burning service bits may not be so bad is because we are due to replace the addr message for i2p and NG-HS support.
546 2017-08-03 19:31:22	0|gmaxwell|So we could at that point just define a new service flagging mechenism.
547 2017-08-03 19:31:27	0|BlueMatt|gmaxwell: yea, does anyone have a spec for that?
548 2017-08-03 19:31:33	0|gmaxwell|Not yet.
549 2017-08-03 19:31:36	0|BlueMatt|k
550 2017-08-03 19:31:52	0|gmaxwell|Well if they're finally going to change it, it becomes moot.. but the same issue arises with 2x.
551 2017-08-03 19:32:16	0|wumpus|how would a service bit help here?
552 2017-08-03 19:32:20	0|BlueMatt|well someone needs to bludgen the 2x folks to change it, otherwise we start banning for a few months
553 2017-08-03 19:32:21	0|wumpus|just ban everyone without NODE_SEGWIT? :p
554 2017-08-03 19:32:38	0|gmaxwell|wumpus: we still want to support non-upgraded nodes.
555 2017-08-03 19:32:48	0|wumpus|but they won't have any new version bit either
556 2017-08-03 19:32:59	0|wumpus|that was my point, not to suggest that seriously :)
557 2017-08-03 19:33:12	0|gmaxwell|wumpus: oh no, you misunderstand: ABC and 2x both set randomly generated service bits.
558 2017-08-03 19:33:23	0|cfields|gmaxwell: eh?
559 2017-08-03 19:33:27	0|BlueMatt|I think sdaftuar's suggestion is reasonable, assuming they refuse to do something sane
560 2017-08-03 19:33:28	0|gmaxwell|(which they've helpfully ignored the gigantic comment in the code that tells you to at least inform the list.)
561 2017-08-03 19:33:45	0|cfields|oh
562 2017-08-03 19:33:46	0|gmaxwell|sdaftuar: I hadn't considered a time limited ban. Good point.
563 2017-08-03 19:33:47	0|wumpus|oh you mean banning everything that sets their version bit?
564 2017-08-03 19:33:54	0|wumpus|yes, that'd be doable
565 2017-08-03 19:33:56	0|BlueMatt|wumpus: yes, with a time limit
566 2017-08-03 19:33:59	0|gmaxwell|wumpus: well disconnecting, not banning.
567 2017-08-03 19:34:07	0|BlueMatt|nah, ban for 3 months
568 2017-08-03 19:34:09	0|gmaxwell|Okay thanks, Time limit makes sense. duh.
569 2017-08-03 19:34:17	0|wumpus|temporarily, yes
570 2017-08-03 19:34:25	0|morcos|wumpus: opened issue #10981, easy fix, but i'll let someone else do the PR as i'm not in the office for next week.  please tag with 0.15.
571 2017-08-03 19:34:26	0|gribble|https://github.com/bitcoin/bitcoin/issues/10981 | resendwallettransactions asserts if walletbroadcast=0 · Issue #10981 · bitcoin/bitcoin · GitHub
572 2017-08-03 19:34:33	0|gmaxwell|BlueMatt: banning creates problems when you run multiple things on one machine.
573 2017-08-03 19:34:41	0|BlueMatt|gmaxwell: meh
574 2017-08-03 19:34:41	0|wumpus|morcos: thanks, will tag later (not logged in now)
575 2017-08-03 19:35:03	0|BlueMatt|gmaxwell: they refused to do something that wasnt astoundingly broken, if it means their users get fucked, its not really my problem
576 2017-08-03 19:35:13	0|wumpus|(or if someone else can do it)
577 2017-08-03 19:35:22	0|morcos|BlueMatt: so chaincode ip will be banned.  nice.
578 2017-08-03 19:35:37	0|BlueMatt|morcos: -connect=altcoin.dns.seed
579 2017-08-03 19:35:38	0|BlueMatt|:)
580 2017-08-03 19:36:08	0|wumpus|agree that banning goes too far, just not allow connections
581 2017-08-03 19:36:13	0|sipa|maye just disconnect?
582 2017-08-03 19:36:15	0|wumpus|there's no point in banning everything after that
583 2017-08-03 19:36:17	0|gmaxwell|what? no. matt, doing that will ban Bitcoin Core users when someone on the same IP ran crapware.
584 2017-08-03 19:36:20	0|jtimon|perhaps better for after the meeting, but I'm still not sure why #8498 wasn't suitable for 0.15 ...
585 2017-08-03 19:36:22	0|gribble|https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
586 2017-08-03 19:36:56	0|gmaxwell|To be clear this is important because these useless altcoin nodes waste connection slots, and are potentially at risk of gobbling up our initial headers fetch.
587 2017-08-03 19:36:57	0|BlueMatt|gmaxwell: ugh
588 2017-08-03 19:37:05	0|BlueMatt|fine, disconnect
589 2017-08-03 19:37:15	0|BlueMatt|at some point someone is gonna create some altcoin crapware that reconnects agressively, though
590 2017-08-03 19:37:17	0|wumpus|disconnect up until a certain date
591 2017-08-03 19:37:20	0|BlueMatt|some spv forks probably will
592 2017-08-03 19:37:24	0|wumpus|what would be that point?
593 2017-08-03 19:37:34	0|BlueMatt|because crapware
594 2017-08-03 19:37:35	0|wumpus|it would disconnect immediately after the version message
595 2017-08-03 19:37:38	0|gmaxwell|Also, looks like ABC has some kind of deadlocking bug, because I see a few of them just going unresponsive to anything but pings, which delays them getting banned for being on the wrong network.
596 2017-08-03 19:37:39	0|morcos|+1 disconnect up to certain date, but i think it should be 12 mos and not 3
597 2017-08-03 19:37:42	0|BlueMatt|do not make assumptions about crapware working in a sane way
598 2017-08-03 19:37:42	0|wumpus|banning would ban 1 message sooner
599 2017-08-03 19:37:47	0|morcos|no reason we'll need that next service bit right away
600 2017-08-03 19:37:59	0|sdaftuar|morcos: sure, that sounds fine
601 2017-08-03 19:38:05	0|wumpus|s/ban/disconnect
602 2017-08-03 19:38:06	0|BlueMatt|morcos: seems reasonable
603 2017-08-03 19:38:19	0|gmaxwell|ack on disconnect based on service bits for 12months or similar.
604 2017-08-03 19:38:38	0|wumpus|unless we start adding banned nodes to the local firewall, there's no serious difference between disconnecting on connect or after the version message
605 2017-08-03 19:38:52	0|gmaxwell|though in general, one of these clowns is going to squat service buts we're in the process of trying to use. :( I have no suggestion on dealing with that.
606 2017-08-03 19:39:13	0|morcos|but i think it'd be worth a quick email/github message to jgarzik to check that they aren't imminently changing their plan
607 2017-08-03 19:39:15	0|BlueMatt|lets deal with that when we get there
608 2017-08-03 19:39:16	0|gmaxwell|s/buts/bits/
609 2017-08-03 19:39:24	0|wumpus|well if they change the magic and port we can stop worrying about the service bits they claim
610 2017-08-03 19:39:31	0|BlueMatt|morcos: yes, as I stated previously we should tell these guys to change network magic *first*
611 2017-08-03 19:39:34	0|wumpus|also we could at that point check for NODE_SEGWIT + our service bit
612 2017-08-03 19:39:35	0|morcos|yes but what if they change to a different service bit
613 2017-08-03 19:39:42	0|gmaxwell|morcos: there is some PR where people have been arguing for ages, about this, so I'm doubtful but sure.
614 2017-08-03 19:39:55	0|morcos|might as well ask first and tell him what we're planning on doing
615 2017-08-03 19:39:58	0|wumpus|change to a different version bit? what would that accomplish?
616 2017-08-03 19:40:07	0|morcos|whooo knows?!!
617 2017-08-03 19:40:12	0|gmaxwell|At the end we're doing them a favor, there are a lot more bitcoin nodes than random altcoin nodes, so these incorrect connections tend to cause them a lot more problems than us.
618 2017-08-03 19:40:23	0|BlueMatt|yup
619 2017-08-03 19:40:28	0|BlueMatt|ok, probalem solved
620 2017-08-03 19:40:35	0|BlueMatt|who wants to go tell them that we're gonna disconnect them?
621 2017-08-03 19:40:40	0|wumpus|if avoiding detection is the point, they could better stop setting their version bti at that point is better than randomly cycling it according to moon phases
622 2017-08-03 19:41:00	0|gmaxwell|BlueMatt: perhaps we should just open the disconnect PR and ping them to comment on it?
623 2017-08-03 19:41:06	0|wumpus|bleh
624 2017-08-03 19:41:16	0|BlueMatt|gmaxwell: wfm, but seems like someone should open an issue
625 2017-08-03 19:41:18	0|BlueMatt|I vote morcos does it
626 2017-08-03 19:41:39	0|gmaxwell|throw him to the wolves... enh? what did he do so wrong?
627 2017-08-03 19:41:42	0|BlueMatt|actually, it was sdaftuar's idea, he can do it
628 2017-08-03 19:41:47	0|gmaxwell|throw him to the wolves... enh? what did he do so wrong?
629 2017-08-03 19:42:05	0|wumpus|seems we'd rather not invite certain discussions to our github but eh
630 2017-08-03 19:42:26	0|BlueMatt|gmaxwell: fine, I'll deal with it
631 2017-08-03 19:42:35	0|gmaxwell|BlueMatt: good, I know what you did so wrong. :P
632 2017-08-03 19:43:04	0|jtimon|but if the bits are selected randomly, how does burning them help?
633 2017-08-03 19:43:35	0|sipa|jtimon: s/randomly/arbitrarily/
634 2017-08-03 19:43:38	0|wumpus|they aren't selected randomly, they're not doing service bit hopping or something like that
635 2017-08-03 19:43:42	0|sipa|they're not different every time
636 2017-08-03 19:43:50	0|jtimon|sipa: I see, thanks
637 2017-08-03 19:43:50	0|sipa|they just arbitrarily picked one
638 2017-08-03 19:43:55	0|gmaxwell|jtimon: I don't follow your question. The altcoin efforts have selected randomly but hardcoded the result or their fair dice roll. :)
639 2017-08-03 19:44:13	0|morcos|+1 sdaftuar doing it...  i'm trying to pack
640 2017-08-03 19:44:14	0|jtimon|gmaxwell: yeah, got it
641 2017-08-03 19:44:17	0|gmaxwell|and just failed to follow the giant comment in the code to make a public announcement about it even.
642 2017-08-03 19:44:23	0|wumpus|e.g. they have monkeys throw darts to select one when they need it, not every connection
643 2017-08-03 19:44:52	0|morcos|oh nm, or bluematt
644 2017-08-03 19:45:22	0|BlueMatt|I think morcos is clearly just trying to throw anyone else under the bus, sounds like he should do it, then :p
645 2017-08-03 19:45:25	0|BlueMatt|anyway, next topic?
646 2017-08-03 19:45:50	0|gmaxwell|;;action bluematt goes under the bus
647 2017-08-03 19:46:03	0|gmaxwell|see, the robot overlords agree
648 2017-08-03 19:46:04	0|BlueMatt|;;action goes under the bus
649 2017-08-03 19:46:17	0|achow101|we can't/shouldn't ban 2x peers until they fork
650 2017-08-03 19:46:24	0|BlueMatt|achow101: yes we should
651 2017-08-03 19:46:40	0|wumpus|I think this was mainly about BCC which already forked
652 2017-08-03 19:46:52	0|achow101|BlueMatt: why? we won't be giving them invalid stuff until they fork, and vice versa
653 2017-08-03 19:47:14	0|gmaxwell|hash that out outside of the meeting plz.
654 2017-08-03 19:47:24	0|wumpus|achow101: on the other hand, adding logic to the code to check whether they've forked would complicate things more than just disconnecting on a service bit
655 2017-08-03 19:47:30	0|wumpus|yeah
656 2017-08-03 19:47:41	0|BlueMatt|next topic?
657 2017-08-03 19:47:46	0|gmaxwell|But in general if someone is going to make broken software, we can only go so far to accomidate it.
658 2017-08-03 19:47:49	0|wumpus|we've run out of topics
659 2017-08-03 19:48:01	0|achow101|wumpus: we know when they activate. at block X start banning them
660 2017-08-03 19:48:19	0|gmaxwell|I doubt we do.
661 2017-08-03 19:48:32	0|jtimon|perhaps better for after the meeting, but I'm still not sure why #8498 wasn't suitable for 0.15 ...
662 2017-08-03 19:48:33	0|BlueMatt|achow101: I'm not jumping through hoops to make sure altcoins stay in consensus until *right before* they fork...
663 2017-08-03 19:48:33	0|gribble|https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
664 2017-08-03 19:48:34	0|lightningbot|Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-08-03-19.00.log.html
665 2017-08-03 19:48:34	0|lightningbot|Meeting ended Thu Aug  3 19:48:33 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
666 2017-08-03 19:48:34	0|lightningbot|Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-08-03-19.00.html
667 2017-08-03 19:48:34	0|lightningbot|Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-08-03-19.00.txt
668 2017-08-03 19:48:34	0|wumpus|#endmeeting
669 2017-08-03 19:48:37	0|gmaxwell|since they still don't have a correct published specification and keep changing things.
670 2017-08-03 19:48:54	0|achow101|gmaxwell: they have stayed on using 12960 blocks after segwit activation
671 2017-08-03 19:50:30	0|gmaxwell|achow101: and what happens if they pull it up to 0 blocks, with a weeks notice? They're clearly being adversarial and trying to harm things, otherwise they'd change port/magic.
672 2017-08-03 19:50:37	0|BlueMatt|achow101: and what if they change it next week? No point, if they want to make sure they stay in consensus, they can run bridge nodes, its not hard
673 2017-08-03 19:51:42	0|achow101|we risk partitioning the network since there are miners that use btc1
674 2017-08-03 19:52:07	0|BlueMatt|achow101: that can be remedied otherwise
675 2017-08-03 19:52:10	0|BlueMatt|(and will be)
676 2017-08-03 19:52:23	0|BlueMatt|and, mostly, hopefully, they just change network magic
677 2017-08-03 19:52:27	0|BlueMatt|and then they can do whatever they want
678 2017-08-03 19:54:06	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #10982: Disconnect network service bits 6 and 8 until Aug 1, 2018 (06master...062017-08-bad-service-bits) 02https://github.com/bitcoin/bitcoin/pull/10982
679 2017-08-03 19:59:55	0|jnewbery|gmaxwell BlueMatt new commits in #10882 ready for review
680 2017-08-03 19:59:58	0|gribble|https://github.com/bitcoin/bitcoin/issues/10882 | Keypool topup by jnewbery · Pull Request #10882 · bitcoin/bitcoin · GitHub
681 2017-08-03 20:02:58	0|BlueMatt|thanks
682 2017-08-03 20:09:24	0|Cobra-Bitcoin|Any core dev around?
683 2017-08-03 20:09:35	0|sipa|many
684 2017-08-03 20:12:29	0|Cobra-Bitcoin|Anyone know what btcdrak means when he says "future releases will be linked" here https://github.com/bitcoin-core/bitcoincore.org/issues/33?
685 2017-08-03 20:12:49	0|Cobra-Bitcoin|So the release notes will point users to bitcoincore.org binaries?
686 2017-08-03 20:15:11	0|sipa|BlueMatt: ^
687 2017-08-03 20:17:53	0|BlueMatt|Cobra-Bitcoin: that is the intention, yes, I mailed you about this a while back :). I think the goal is to start pointing people to bitcoincore.org, but also sicne y'all wanted to keep mirroring on bitcoin.org (and cause no point telling people to go somewhere else if they're already used to bitcoin.org), they'd be in both places
688 2017-08-03 20:18:16	0|BlueMatt|ie cause security isnt helped by making people make hops
689 2017-08-03 20:18:52	0|BlueMatt|Cobra-Bitcoin: see-also #10651
690 2017-08-03 20:18:53	0|gribble|https://github.com/bitcoin/bitcoin/issues/10651 | Verify binaries from bitcoincore.org and bitcoin.org by TheBlueMatt · Pull Request #10651 · bitcoin/bitcoin · GitHub
691 2017-08-03 20:23:02	0|Cobra-Bitcoin|But is bitcoin.org not doing enough to already distribute the binaries effectively?
692 2017-08-03 20:23:50	0|Cobra-Bitcoin|We have no choice but to mirror, since we have a lot of pages structured around the binaries and Core
693 2017-08-03 20:25:21	0|achow101|wumpus: BlueMatt so what do we do about salvagewallet and this default key thing?
694 2017-08-03 20:25:52	0|achow101|write a new one?
695 2017-08-03 20:27:37	0|BlueMatt|Cobra-Bitcoin: see pm
696 2017-08-03 20:28:00	0|BlueMatt|achow101: I havent looked at salvage, I mean certainly making it return an error instead of throwing like it does is not too hard
697 2017-08-03 20:28:11	0|wumpus|Cobra-Bitcoin: hosting on both bitcoin.org and bitconcore.org is perfectly acceptable, preferable even IMO
698 2017-08-03 20:28:26	0|achow101|BlueMatt: I have it return a db corrupt error now (local change, not yet pushed)
699 2017-08-03 20:28:33	0|wumpus|Cobra-Bitcoin: more (trusted) places to get the binaries from gives some redundancy
700 2017-08-03 20:28:41	0|BlueMatt|plus what wumpus said
701 2017-08-03 20:29:24	0|achow101|but my concern is that people will downgrade because downgrading works. there's no way to prevent downgrade is to use a new wallet version, but that's a mess that I don't want to deal with right now
702 2017-08-03 20:30:02	0|BlueMatt|achow101: well a reasonable error message should help, then
703 2017-08-03 20:30:32	0|BlueMatt|I dont care about preventing users from doing X or Y, but giving them an error message informing them that their hardware appears to have silently corrupted their wallet is appropriate
704 2017-08-03 20:30:51	0|BlueMatt|ideally salvagewallet would handle this case (does it not, my understanding was it should?)
705 2017-08-03 20:31:17	0|achow101|currently it says "Error loading wallet.dat: Wallet corrupted" with a more specific "invalid default key" message in the debug.log
706 2017-08-03 20:31:28	0|BlueMatt|thats fine?
707 2017-08-03 20:31:41	0|achow101|salvagewallet doesn't handle uncorrupting keys, it only pulls data that it can find being valid and passes it through
708 2017-08-03 20:31:53	0|achow101|so any corrupted keys will get passed through to a salvaged wallet
709 2017-08-03 20:32:03	0|achow101|(not like you could uncorrupt a key anyways)
710 2017-08-03 20:32:07	0|wumpus|is there even such a thing as 'uncorrupting keys'?
711 2017-08-03 20:32:23	0|wumpus|it's not like we encode in some redundant way that allows recovery
712 2017-08-03 20:32:29	0|wumpus|we could do that, but we don't
713 2017-08-03 20:32:35	0|achow101|wumpus: well you could easily uncorrupt a default key by just writing a generic valid key like the generator
714 2017-08-03 20:32:55	0|wumpus|you could erase/ignore corrupted keys, though that's somewhat scary...
715 2017-08-03 20:33:06	0|BlueMatt|wumpus: isnt salvagewallet always scary?
716 2017-08-03 20:33:10	0|wumpus|(but not much different from what bdb salvage already does)
717 2017-08-03 20:33:18	0|wumpus|yes, sure, but this isn't black and white
718 2017-08-03 20:33:31	0|BlueMatt|true
719 2017-08-03 20:33:34	0|wumpus|there are certainly ways to make it even scarier :)
720 2017-08-03 20:33:35	0|sipa|so there is an added point here: any key that got corrupted ever probably resulted in the wallet just failing, and the user starting over or finding other solution
721 2017-08-03 20:33:55	0|sipa|but if it was the default key that was corrupted, it likely just meant that the wallet replenished the keypool at every startup, ever
722 2017-08-03 20:34:03	0|sipa|and everything kept working
723 2017-08-03 20:34:06	0|wumpus|yes
724 2017-08-03 20:34:26	0|sipa|so there may be a survivalship bias, resulting in currently a higher than expected number of people with a corrupted default key
725 2017-08-03 20:34:47	0|wumpus|there probably should be a way to salvage and ignore corrupt keys...
726 2017-08-03 20:34:54	0|sipa|yes.
727 2017-08-03 20:35:04	0|wumpus|(another salvage level!)
728 2017-08-03 20:35:18	0|sipa|unfortunately, not possible for encrypted keys
729 2017-08-03 20:35:25	0|sipa|at least not without passphrase
730 2017-08-03 20:35:36	0|wumpus|yeah...
731 2017-08-03 20:35:49	0|wumpus|repair for an encrypted wallet should ideally ask for the passphrase
732 2017-08-03 20:36:00	0|wumpus|it's the only way to recover most effectively
733 2017-08-03 20:38:52	0|achow101|it also seems like the two reported cases of this involve wallets that have corrupted keys, which salvagewallet doesn't help with
734 2017-08-03 20:39:14	0|achow101|it's just that the user doesn't notice those keys were corrupted until they try to spend
735 2017-08-03 20:39:26	0|achow101|then the wallet fails to decrypt
736 2017-08-03 20:42:56	0|wumpus|yes that's not good
737 2017-08-03 20:44:57	0|achow101|so i guess we could just leave this as a warning about wallet corruption and not do any salvaging which would exacerbate the problem?
738 2017-08-03 21:00:24	0|wumpus|agreed, any 'salvaging' should be at user initiative
739 2017-08-03 22:14:53	0|earlz|Is there any easy way to make gitian skip building dependencies? I'm having some trouble with a modification I made breaking the gitian build for bitcoind itself, but the 2 hours spent compiling dependencies is killing me
740 2017-08-03 22:21:33	0|cfields|earlz: the dependencies are cached, they'll only be built once
741 2017-08-03 22:27:38	0|earlz|cfields: that does not seem to be the case if gbuild fails
742 2017-08-03 22:29:50	0|earlz|is there some command line argument or something I'm missing to prevent it from saving the cache, cause right now I basiclaly do gbuild, it compiles all deps, then encounters failure, I'll apply a fix and commit/push it, then update the gbuild command for the new commit hash and I observe it compiles all deps again
743 2017-08-03 22:30:00	0|cfields|earlz: do a vanilla build, make sure it finishes, get the deps cached
744 2017-08-03 22:30:04	0|cfields|then rebuild with your changes
745 2017-08-03 22:30:19	0|earlz|oh I see, so it only caches if the build succeeds?
746 2017-08-03 22:30:41	0|cfields|yes
747 2017-08-03 22:48:44	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #10984: Allow 2 simultaneous (compact-)block downloads (06master...062017-08-paralell-block-downloads) 02https://github.com/bitcoin/bitcoin/pull/10984
748 2017-08-03 23:05:31	0|BlueMatt|cfields: when you do the next round of cleanups on #10756, can you not move InitializeNode/FinalizeNode? I'm starting work on building on top and have patches that conflict :/
749 2017-08-03 23:05:32	0|gribble|https://github.com/bitcoin/bitcoin/issues/10756 | net processing: swap out signals for an interface class by theuni · Pull Request #10756 · bitcoin/bitcoin · GitHub
750 2017-08-03 23:06:37	0|cfields|BlueMatt: heh, i thought we agreed that was the next step?
751 2017-08-03 23:06:53	0|BlueMatt|cfields: nono, i mean dont move the code down in the file
752 2017-08-03 23:06:54	0|BlueMatt|not dont move
753 2017-08-03 23:08:46	0|cfields|BlueMatt: oh, I see what you mean
754 2017-08-03 23:08:55	0|cfields|BlueMatt: they have to move out of the anon namespace though :(
755 2017-08-03 23:09:46	0|BlueMatt|ohh, didnt realize they were in a namespace
756 2017-08-03 23:09:48	0|BlueMatt|damn, ok, fine
757 2017-08-03 23:09:55	0|BlueMatt|I'll just have to do some dirty rebase
758 2017-08-03 23:10:08	0|cfields|yea, that's the only reason i moved them
759 2017-08-03 23:10:40	0|cfields|could do something ugly like closing/reopening the namespace, but i'd rather not :)
760 2017-08-03 23:26:16	0|bitcoin-git|[13bitcoin] 15sipa opened pull request #10985: Add undocumented -forcecompactdb to force LevelDB compactions (06master...0620170803_forcecompactdb) 02https://github.com/bitcoin/bitcoin/pull/10985
761 2017-08-03 23:40:55	0|bitcoin-git|[13bitcoin] 15sipa opened pull request #10986: Update chain transaction statistics (06master...0620170803_txstats) 02https://github.com/bitcoin/bitcoin/pull/10986
762 2017-08-03 23:47:30	0|gmaxwell|#10985: \0/
763 2017-08-03 23:47:31	0|gribble|https://github.com/bitcoin/bitcoin/issues/10985 | Add undocumented -forcecompactdb to force LevelDB compactions by sipa · Pull Request #10985 · bitcoin/bitcoin · GitHub
764 2017-08-03 23:47:50	0|sipa|gmaxwell: ^ untested