1 2017-10-05 03:36:37	0|bitcoin-git|[13bitcoin] 15jonasschnelli closed pull request #10517: Factor out CCoinsView based AreInputsStandard/IsWitnessStandard (06master...062017/06/policy_compile) 02https://github.com/bitcoin/bitcoin/pull/10517
  2 2017-10-05 03:37:43	0|bitcoin-git|[13bitcoin] 15jonasschnelli closed pull request #10238: Change setKeyPool to hold flexible entries (06master...062017/04/keypool_fix_a) 02https://github.com/bitcoin/bitcoin/pull/10238
  3 2017-10-05 04:05:31	0|bitcoin-git|[13bitcoin] 15jonasschnelli closed pull request #10251: Add balances cache / GUI: use a signal instead of a poll thread (06master...062017/04/gui_rm_locks) 02https://github.com/bitcoin/bitcoin/pull/10251
  4 2017-10-05 11:22:40	0|sturles|I have a problem with boost after upgrading to Debian 9.  I did an apt-get --reinstall install libboost-all-dev to be sure I have everything, but it still fails: https://0bin.net/paste/NjYBdILEZBChNmbI#rsDCuSOm7iKfR-UZj7ytJdxoAYMLRdB3szd43qbZmvH
  5 2017-10-05 11:24:03	0|sturles|Any ideas?
  6 2017-10-05 13:09:16	0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 5 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/167cef8082e2...e93fff1463ae
  7 2017-10-05 13:09:17	0|bitcoin-git|13bitcoin/06master 1435aeabe 15MeshCollider: Make fReindex atomic to avoid race
  8 2017-10-05 13:09:17	0|bitcoin-git|13bitcoin/06master 1458d91af 15MeshCollider: Fix race for mapBlockIndex in AppInitMain
  9 2017-10-05 13:09:18	0|bitcoin-git|13bitcoin/06master 14731065b 15MeshCollider: Consistent parameter names in txdb.h
 10 2017-10-05 13:09:50	0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #11107: Fix races in AppInitMain and others with lock and atomic bools (06master...06fix_mapBlockIndex_race) 02https://github.com/bitcoin/bitcoin/pull/11107
 11 2017-10-05 15:01:20	0|deltaT|simple question i hope, when i created my bitcoin core wallet i didnt write down the private key, how do i find it now?
 12 2017-10-05 15:06:44	0|jonasschnelli|deltaT: which Bitcoin Core version?
 13 2017-10-05 15:07:17	0|jonasschnelli|deltaT: did you lost you wallet.dat? file.... also, this question should be asked in #bitcoin (this here is the development channel)
 14 2017-10-05 15:07:27	0|BlueMatt|wumpus: #9572 looks relatively merge-able...it is consensus so if you want to ack first that'd also be nice, but its simple and has 4 already
 15 2017-10-05 15:07:27	0|deltaT|0.14.2
 16 2017-10-05 15:07:29	0|gribble|https://github.com/bitcoin/bitcoin/issues/9572 | Skip witness sighash cache for non-segwit transactions by jl2012 · Pull Request #9572 · bitcoin/bitcoin · GitHub
 17 2017-10-05 15:13:40	0|bitcoin-git|[13bitcoin] 15mess110 opened pull request #11455: CTxMemPool::GetMinFee should not return CFeeRate(0) (06master...06fix_mempool_GetMinFee_bug_returning_below_minRelayTxFee) 02https://github.com/bitcoin/bitcoin/pull/11455
 18 2017-10-05 15:18:38	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #11456: Replace relevant services logic with a function suite. (06master...062017-09-service-flags-cleanups) 02https://github.com/bitcoin/bitcoin/pull/11456
 19 2017-10-05 15:20:20	0|bitcoin-git|[13bitcoin] 15mess110 closed pull request #11410: [rpc] [tests] mempoolminfee should not drop below minRelayTxFee (06master...06add_minrelaytxfee_to_getmempoolinfo) 02https://github.com/bitcoin/bitcoin/pull/11410
 20 2017-10-05 15:31:18	0|tloriato|Good afternoon. I was looking into BIP45, that tries to standardize the Structure for Deterministic P2SH Multisignature Wallets, but I've read the emails discussing it and seems to have a little or disagreement between the need for this particular BIP. I'm wondering if there is another standard way to generate a Deterministic Multisig Wallet? I was inclined to generate a standard Mnemonic (BIP39) and split it using Shamir
 21 2017-10-05 15:36:29	0|BlueMatt|sipa: did you find time to write up the segwit wallet tradeoffs between the various ways of doing it and making an argument for why your pr is your preferred version?
 22 2017-10-05 15:36:37	0|BlueMatt|sipa: I believe you said you were gonna do that last meeting
 23 2017-10-05 15:50:20	0|jonasschnelli|tloriato: AFAIK BIP45 is the only "standard" to create multisig addresses with a set of given pubkeys.
 24 2017-10-05 15:50:28	0|jonasschnelli|*extended pubkeys
 25 2017-10-05 15:53:51	0|tloriato|jonasschnelli: thanks
 26 2017-10-05 16:00:52	0|BlueMatt|nothing else does anything close
 27 2017-10-05 16:06:41	0|bitcoin-git|[13bitcoin] 15laanwj pushed 3 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/e93fff1463ae...becbd71b0c16
 28 2017-10-05 16:06:42	0|bitcoin-git|13bitcoin/06master 144f890ba 15Donal OConnor: Add new step to clean $PATH var by removing /mnt specific Window's %PATH% paths that cause issues with the make system
 29 2017-10-05 16:06:42	0|bitcoin-git|13bitcoin/06master 14696ce46 15fanquake: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04
 30 2017-10-05 16:06:43	0|bitcoin-git|13bitcoin/06master 14becbd71 15Wladimir J. van der Laan: Merge #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04...
 31 2017-10-05 16:07:26	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 (06master...06windows-build-1704) 02https://github.com/bitcoin/bitcoin/pull/11437
 32 2017-10-05 16:08:16	0|bitcoin-git|13bitcoin/06master 14f3ba869 15practicalswift: [tests] Add libFuzzer support....
 33 2017-10-05 16:08:16	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/becbd71b0c16...9e8ef9d99179
 34 2017-10-05 16:08:17	0|bitcoin-git|13bitcoin/06master 149e8ef9d 15Wladimir J. van der Laan: Merge #10440: [tests] Add libFuzzer support...
 35 2017-10-05 16:08:31	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10440: [tests] Add libFuzzer support (06master...06libfuzzer) 02https://github.com/bitcoin/bitcoin/pull/10440
 36 2017-10-05 16:17:13	0|sipa|BlueMatt: i've been distracted by something else
 37 2017-10-05 17:02:55	0|bitcoin-git|[13bitcoin] 15sdaftuar closed pull request #10200: Mining: Skip recent transactions if fee difference is small (06master...062017-04-dont-mine-recent-tx) 02https://github.com/bitcoin/bitcoin/pull/10200
 38 2017-10-05 17:48:40	0|RealM9|Hello, how long until meeting starts? At what time it starts?
 39 2017-10-05 17:49:31	0|sipa|11 minutes from now
 40 2017-10-05 17:49:45	0|sipa|sorry, 71 minutes
 41 2017-10-05 17:50:12	0|sipa|it's at 7pm UTC
 42 2017-10-05 17:50:17	0|bitcoin-git|13bitcoin/06master 140da49b5 15Johnson Lau: Skip precompute sighash for transactions without witness
 43 2017-10-05 17:50:17	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/9e8ef9d99179...17f2acedbe07
 44 2017-10-05 17:50:18	0|bitcoin-git|13bitcoin/06master 1417f2ace 15Wladimir J. van der Laan: Merge #9572: Skip witness sighash cache for non-segwit transactions...
 45 2017-10-05 17:50:24	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9572: Skip witness sighash cache for non-segwit transactions (06master...06nocache) 02https://github.com/bitcoin/bitcoin/pull/9572
 46 2017-10-05 17:51:16	0|RealM9|Ok, thnx sipa
 47 2017-10-05 17:59:32	0|jl2012|why do we need a vector of PrecomputedTransactionData here? https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1755
 48 2017-10-05 18:00:29	0|sipa|jl2012: what alternative do you suggest?
 49 2017-10-05 18:00:39	0|jl2012|why couldn't we just have a PrecomputedTransactionData for each transaction?
 50 2017-10-05 18:00:49	0|sipa|we do?
 51 2017-10-05 18:00:58	0|sipa|that's just the vector that holds them
 52 2017-10-05 18:01:35	0|jl2012|yes, but why need to hold them all?
 53 2017-10-05 18:02:00	0|sipa|variables need to exist somewhere...
 54 2017-10-05 18:03:24	0|jl2012|yes, I mean, why not just have a line PrecomputedTransactionData txdata(tx); inside the loop "for (unsigned int i = 0; i < block.vtx.size(); i++)"?
 55 2017-10-05 18:04:00	0|sipa|jl2012: oh, i read "why need to them *at* all", sorry
 56 2017-10-05 18:04:19	0|sipa|the reason is that they're used from other threads during parallel validation
 57 2017-10-05 18:05:42	0|jl2012|ok...so without the vector, each thread will re-compute the hashes once?
 58 2017-10-05 18:06:08	0|sipa|depends how you do it
 59 2017-10-05 18:06:27	0|sipa|if you do what you suggest, you'd get undefined behaviour
 60 2017-10-05 18:06:50	0|jl2012|why?
 61 2017-10-05 18:06:54	0|sipa|as the precomputation data would be out of scope before it's accessed from the validation threads
 62 2017-10-05 18:08:16	0|sipa|most of the validation happens after that loop exits, and before the queue's wait call returns
 63 2017-10-05 18:10:36	0|JeremyRubin|jl2012: I have a patch that makes them shared_ptrs
 64 2017-10-05 18:10:46	0|BlueMatt|sipa: not anymore (tx script caching) :p
 65 2017-10-05 18:10:47	0|JeremyRubin|conceptually, that's what you want
 66 2017-10-05 18:10:56	0|JeremyRubin|performance wise, a vector is better
 67 2017-10-05 18:12:49	0|jonasschnelli|gmaxwell: for the notification system, you had concerns about the reliability.. would a long poll queue that only removes elements from the queue on an ack from the client be something that would defeat your concerns?
 68 2017-10-05 18:13:01	0|jonasschnelli|With that concept, loosing notifications would be very unlikely
 69 2017-10-05 18:13:05	0|sipa|BlueMatt: ok, i reformulate - most of the validation, if it happens, happens after that loop exits
 70 2017-10-05 18:15:34	0|jl2012|oh, I thought all script validation is done with CheckInputs?
 71 2017-10-05 18:15:56	0|sipa|jl2012: yes and no
 72 2017-10-05 18:16:18	0|sipa|CheckInputs returns a vector of CScriptCheck objects, which represent validation that will happen on another thread
 73 2017-10-05 18:16:35	0|sipa|those get pushed to the checkqueue, where validation threads pick them up
 74 2017-10-05 18:16:52	0|jl2012|ontrol.Add(vChecks); ?
 75 2017-10-05 18:16:59	0|jl2012|control.Add(vChecks); ?
 76 2017-10-05 18:17:00	0|JeremyRubin|yes
 77 2017-10-05 18:17:21	0|gmaxwell|jonasschnelli: it would but wumpus pointed out that that queue would potentially grow without bound if the client stops acking entirely.
 78 2017-10-05 18:17:59	0|sipa|jl2012: and the CScriptCheck objects have a pointer to the precomputation data
 79 2017-10-05 18:18:06	0|JeremyRubin|control.Add returns immediately
 80 2017-10-05 18:18:13	0|wumpus|yes, 100% reliable notification given any behavior of the receiver is not possible given finite space
 81 2017-10-05 18:18:25	0|jonasschnelli|gmaxwell: a queue limit is unavoidable... if you poll to lazy and the queue limit is to little, the only way to detect would be via the sequence number
 82 2017-10-05 18:18:32	0|sipa|jl2012: which means we must guarantee that that precomputation data remains alive as long as other threads may dereference the pointer
 83 2017-10-05 18:18:41	0|wumpus|some notification queue middleware logs events to disk in that case
 84 2017-10-05 18:18:59	0|jonasschnelli|wumpus: in long polling, the queue must be finite because we don't know when the client polls next,...
 85 2017-10-05 18:19:14	0|wumpus|e.g. rabbitmq etc, not by any means suggesting we take that up
 86 2017-10-05 18:19:54	0|wumpus|jonasschnelli: I agree there needs to be a limit realistically, it's enough if a client can detect missed events to resync
 87 2017-10-05 18:20:09	0|jonasschnelli|I personally think acking notifications is unnecessary,.. making it configurable seems also an overkill... so unsure.
 88 2017-10-05 18:20:33	0|JeremyRubin|jl2012: this is why I say a shared_ptr is what you really want, because the lifetime is automatically handled for you, whereas the vector is only by careful programming. However, that careful programming is currently correct :)
 89 2017-10-05 18:21:19	0|sipa|jonasschnelli: if you don't ask notifications, then your clients must manually deal with restarts... which implies having a way to ask for all events since some time... which, if you have it, removes the need for an event log entirely, as you can just use that all the time
 90 2017-10-05 18:21:29	0|jl2012|JeremyRubin, sipa: thanks. I'm thinking how I could do this: https://github.com/bitcoin/bitcoin/pull/9572#issuecomment-334282408
 91 2017-10-05 18:21:47	0|wumpus|what's important is to document the limitations of the notification system in that regard
 92 2017-10-05 18:22:11	0|sipa|jl2012: do what?
 93 2017-10-05 18:22:42	0|jl2012|"I'd be a little more comfortable with this if PrecomputedTransactionData were passed around as const, so nobody's tempted to mess with "ready" for some crazy reason. But that can always be done later."
 94 2017-10-05 18:22:54	0|gmaxwell|shared_ptrs are a way to produce a software engineering disaster, because they let you be sloppy with ownership. There are cases of frestanding objects that don't have organized ownership but they are relatively rare. They also have non-trivial overhad.
 95 2017-10-05 18:23:26	0|sipa|jl2012: seems trivial?
 96 2017-10-05 18:23:56	0|sipa|i think you're overthinking it
 97 2017-10-05 18:24:29	0|jl2012|I'm thinking something like "const PrecomputedTransactionData txdata(tx);"
 98 2017-10-05 18:25:31	0|sipa|jl2012: cfields is only asking to pass it around as const; not to make the entire object const
 99 2017-10-05 18:26:36	0|cfields|^^ yep
100 2017-10-05 18:27:18	0|cfields|though i think making the members const would be trivial too
101 2017-10-05 18:27:33	0|cfields|either way, it's not really important for that PR. just an observation.
102 2017-10-05 18:27:49	0|jl2012|it's even better if we could make the object const? There is no reason to make any modification
103 2017-10-05 18:28:21	0|JeremyRubin|gmaxwell: am aware, but strictly speaking that is the point here: they are designed to track the actual needed lifetime, and are a 'tighter fit' to the problem in this situation. That they have problems is why I didn't PR my patch.
104 2017-10-05 18:28:33	0|cfields|jl2012: just make the members const, then. use initializers rather than setting them in the function body
105 2017-10-05 18:29:05	0|JeremyRubin|jl2012: I think that we shouldn't make it const, because there will be in the future per-tx state that we'll modify
106 2017-10-05 18:29:38	0|JeremyRubin|e.g., if at some point in the future signature agg ends up needing a per-tx mutable struct that each input modifies
107 2017-10-05 18:29:53	0|gmaxwell|JeremyRubin: in the case of the validation the ownership is just a straght line though, the creating code owns it, then the ownership is passed to the queue, then the ownership passes to the verifier thread..
108 2017-10-05 18:30:38	0|JeremyRubin|Maybe it should be a different struct, but I think (to me), changing PreComputedTxData to PerTxData is straightforward.
109 2017-10-05 18:30:57	0|sipa|JeremyRubin: i think that should be separate
110 2017-10-05 18:31:08	0|sipa|if you want to avoid locking on the precomputed data
111 2017-10-05 18:31:33	0|JeremyRubin|gmaxwell: yes, and then they are kept alive for longer than strictly needed. After the last scriptcheck executes for that tx, they can be freed.
112 2017-10-05 18:31:34	0|cfields|agree. It'd be nice to keep the factual data separate from what's aggregated/stateful
113 2017-10-05 18:31:58	0|sipa|JeremyRubin: yes, so? it doesn't change the worst case resource consumption
114 2017-10-05 18:32:05	0|JeremyRubin|cfields: all the data should be factual in consensus?
115 2017-10-05 18:33:13	0|JeremyRubin|sipa: I'm really only trying to talk about lifetimes here, which is the core of what jl2012 is discussing
116 2017-10-05 18:33:13	0|jonasschnelli|sipa: persist the notification queue (with it's sequence numbers) to make it restart-safe?
117 2017-10-05 18:33:25	0|JeremyRubin|sipa: asking
118 2017-10-05 18:35:22	0|sipa|JeremyRubin: i'm just trying to point out that minimizing the time an object lives is not always wanted
119 2017-10-05 18:35:27	0|cfields|JeremyRubin: i just meant that it can be safely computed ahead of time and seen as immutible after that.
120 2017-10-05 18:35:46	0|cfields|factual was the wrong word, i suppose.
121 2017-10-05 18:37:03	0|JeremyRubin|sipa: of course
122 2017-10-05 18:38:32	0|sipa|jonasschnelli: that's one way (but it has pretty bad costs, if you want to make it durable - now you need synchronization across all files/databases)
123 2017-10-05 18:39:11	0|sipa|jonasschnelli: another model is that you just have RPCs like listsinceblock, where the client tells the server effectively what it already knows
124 2017-10-05 18:39:32	0|sipa|jonasschnelli: and then all you need is a notification like "there's something for you to look at"
125 2017-10-05 18:40:07	0|jonasschnelli|sipa: by a rolling hash?
126 2017-10-05 18:40:13	0|sipa|jonasschnelli: wut?
127 2017-10-05 18:40:30	0|sipa|no, just like listsinceblock
128 2017-10-05 18:40:31	0|jonasschnelli|how would the client tell the server what notfications it has... just the sequence number?
129 2017-10-05 18:40:37	0|jonasschnelli|(need to check that)
130 2017-10-05 18:40:44	0|jonasschnelli|thanks
131 2017-10-05 18:40:44	0|sipa|no, i'm saying there are no notification
132 2017-10-05 18:40:54	0|sipa|there are just state changes
133 2017-10-05 18:41:24	0|sipa|and the client can ask "i've synced up to block X" or "i've seen transactions up to timestamp Y", or "the last balance update i saw was Z"
134 2017-10-05 18:42:01	0|sipa|because most data has inherent sequencing anyway already
135 2017-10-05 18:42:23	0|jonasschnelli|could it also do long polling? ... because IMO that is what users want (not const. polling)
136 2017-10-05 18:42:42	0|sipa|yes, but the long poll just returns "there is something you should look at", not what
137 2017-10-05 18:43:02	0|jonasschnelli|ah. And then the state update call.
138 2017-10-05 18:43:09	0|sipa|... no
139 2017-10-05 18:43:29	0|sipa|i mean *exactly* like listsinceblock
140 2017-10-05 18:43:42	0|jonasschnelli|heh.. okay let me dive into there first
141 2017-10-05 18:43:51	0|sipa|it has no sequence numbers, or notifications
142 2017-10-05 18:44:16	0|sipa|it's a call where the client tells the server "hey, i've seen all transactions up to block X. what new transactions are there"
143 2017-10-05 18:44:39	0|sipa|the next time the client calls, it uses the block hash at the time of its previous call
144 2017-10-05 18:44:55	0|sipa|the server doesn't need to keep track of anything
145 2017-10-05 18:45:51	0|jonasschnelli|would that also work with non block/tx data? Like new wallet txns (locally injected) or bandwith watermarks?
146 2017-10-05 18:46:09	0|jonasschnelli|although not sure if there is a need for that
147 2017-10-05 18:46:11	0|sipa|perhaps - maybe it's not possible for everything
148 2017-10-05 18:46:46	0|sipa|bandwidth watermarks are easy... the client doesn't care if he misses one
149 2017-10-05 18:47:30	0|sipa|for wallet txn you can use the txid of the last seen tx
150 2017-10-05 18:48:21	0|sipa|jonasschnelli: my point is that you probably need something like that anyway, at least for when a new client starts up or went offline for a while
151 2017-10-05 18:49:02	0|jonasschnelli|sipa: Yes. Your idea make sense.
152 2017-10-05 18:49:13	0|sipa|and when you do, why bother with a separate event queue - all events could just be "hey something happened, you should check what"
153 2017-10-05 18:49:38	0|sipa|that's a big concern i had with ZMQ... as it passes the actual new tx and blocks, but without guarantees that it delivers all
154 2017-10-05 18:49:50	0|sipa|it was fixed with sequence numbers
155 2017-10-05 18:49:51	0|jonasschnelli|maybe you should be able to not get such notifications on new mempool txns
156 2017-10-05 18:50:11	0|sipa|but it could also have been fixed by not having it at all, and making clients query for the data they didn't have after every notifications"
157 2017-10-05 18:50:58	0|sipa|anyway, just an idea
158 2017-10-05 18:51:11	0|jonasschnelli|Thanks for sharing... need to think about it a bit more
159 2017-10-05 18:51:15	0|sipa|having events you can subscribe to individually that persist etc... is certainly more convenient for clients
160 2017-10-05 18:51:51	0|sipa|but it makes bitcoin core responsible for tracking who knows what, instead of leaving that up to clients (who are arguably in a better position to know what they already know)
161 2017-10-05 18:52:09	0|jonasschnelli|Indeed. And also the ressources.
162 2017-10-05 18:55:35	0|sipa|it would be pretty nice if there was a way to subscribe to "hey, let me know when txid X gets Y confirmations or gets reorged", but perhaps that could be done in a python script that just uses a simpler interface with bitcoidn
163 2017-10-05 19:00:08	0|achow101|meeting?
164 2017-10-05 19:00:44	0|instagibbs|yes
165 2017-10-05 19:01:13	0|sipa|yes
166 2017-10-05 19:02:18	0|jonasschnelli|#startmeeting
167 2017-10-05 19:02:19	0|lightningbot|Meeting started Thu Oct  5 19:02:18 2017 UTC.  The chair is jonasschnelli. Information about MeetBot at http://wiki.debian.org/MeetBot.
168 2017-10-05 19:02:19	0|lightningbot|Useful Commands: #action #agreed #help #info #idea #link #topic.
169 2017-10-05 19:02:23	0|wumpus|yes
170 2017-10-05 19:02:28	0|jonasschnelli|Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
171 2017-10-05 19:02:33	0|kanzure|hi.
172 2017-10-05 19:02:35	0|jonasschnelli|wumpus: sry: though your where OL
173 2017-10-05 19:02:43	0|cfields|hi
174 2017-10-05 19:02:43	0|wumpus|#bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101
175 2017-10-05 19:02:44	0|meshcollider|Hello
176 2017-10-05 19:02:45	0|achow101|hi
177 2017-10-05 19:02:50	0|luke-jr|hi
178 2017-10-05 19:03:11	0|wumpus|jonasschnelli: yes I was a bit late, sorry
179 2017-10-05 19:03:22	0|wumpus|#topic high-priority for review
180 2017-10-05 19:03:44	0|achow101|#10637 please?
181 2017-10-05 19:03:44	0|jonasschnelli|#chair wumpus
182 2017-10-05 19:03:44	0|michagogo|Huh?
183 2017-10-05 19:03:45	0|lightningbot|Current chairs: jonasschnelli wumpus
184 2017-10-05 19:03:48	0|gribble|https://github.com/bitcoin/bitcoin/issues/10637 | Coin Selection with Murchs algorithm by achow101 · Pull Request #10637 · bitcoin/bitcoin · GitHub
185 2017-10-05 19:04:16	0|michagogo|Oh, right
186 2017-10-05 19:04:20	0|michagogo|It's actually Thursday
187 2017-10-05 19:04:34	0|wumpus|achow101: ok
188 2017-10-05 19:05:16	0|wumpus|I also added #11389 today as it's blocking segwit wallet support
189 2017-10-05 19:05:18	0|gribble|https://github.com/bitcoin/bitcoin/issues/11389 | Support having SegWit always active in regtest by sipa · Pull Request #11389 · bitcoin/bitcoin · GitHub
190 2017-10-05 19:05:21	0|gribble|https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
191 2017-10-05 19:05:48	0|meshcollider|#11403 itself should be in there too probably?
192 2017-10-05 19:05:51	0|gribble|https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
193 2017-10-05 19:06:08	0|sipa|wumpus: i haven't had the time to work further on 11403, though concept review is certainly welcome
194 2017-10-05 19:06:17	0|jnewbery|I've been reviewing 11389 this afternoon. It looks generally good, but breaks assumevalid.py, which I'm trying to fix now
195 2017-10-05 19:06:29	0|jtimon|s/locks/looks/
196 2017-10-05 19:06:47	0|BlueMatt|sipa: I think we need a document on the various possible approaches, tbh
197 2017-10-05 19:06:59	0|sipa|BlueMatt: yes, i'll work on that soon
198 2017-10-05 19:07:01	0|BlueMatt|there are a few and talking through all of them is going to need something more formal
199 2017-10-05 19:07:02	0|BlueMatt|thanks
200 2017-10-05 19:07:22	0|morcos|achow101: does 10637 implement all the coin selection logic we discussed in SF or only BnB?  Is there a high level description somewhere of what the PR is purporting to accomplish and what else will need to be done before 0.16?
201 2017-10-05 19:07:33	0|achow101|morcos: only BnB
202 2017-10-05 19:07:49	0|achow101|morcos: IIRC Murch is working on all of the coin selection stuff that we discussed
203 2017-10-05 19:07:53	0|wumpus|btw I posted a proposed release schedule for 0.16.0 yesterday
204 2017-10-05 19:07:55	0|wumpus|#link https://github.com/bitcoin/bitcoin/issues/11449
205 2017-10-05 19:08:29	0|morcos|achow101: ok.. i've already forgotten what that is, so might be nice to have that written up in an issue or something so we remember the goal and can think about how this BnB implementation is going to fit into the big picture
206 2017-10-05 19:09:16	0|achow101|morcos: the description of what 10637 does is in the first comment.
207 2017-10-05 19:09:35	0|achow101|I can make an issue for coin selection changes in general
208 2017-10-05 19:09:42	0|achow101|*to keep track of
209 2017-10-05 19:10:54	0|wumpus|ok, I think that concludes high priority for review proposals
210 2017-10-05 19:10:58	0|wumpus|any other topics?
211 2017-10-05 19:11:21	0|achow101|topic suggestion: bad block interrogation/invalid block peer banning
212 2017-10-05 19:11:26	0|wumpus|#action achow101 make an issue for coin selection changes in general
213 2017-10-05 19:11:39	0|wumpus|#topic bad block interrogation/invalid block peer banning
214 2017-10-05 19:11:53	0|achow101|relevant PR is #11446 (I did this in class so it kinda sucks)
215 2017-10-05 19:11:55	0|gribble|https://github.com/bitcoin/bitcoin/issues/11446 | [WIP] Bad block interrogation by achow101 · Pull Request #11446 · bitcoin/bitcoin · GitHub
216 2017-10-05 19:12:07	0|Murch|hey, sorry, was still in a meeting
217 2017-10-05 19:12:34	0|achow101|basically the idea is gmaxwell's. when we receive an invalid block, we want to make sure that all of our peers would also reject that block as invalid. If they don't ban them
218 2017-10-05 19:12:38	0|Murch|I've been working on it, but since I do that in my free time in the evenings, it's been rather slow.
219 2017-10-05 19:12:54	0|luke-jr|wumpus: this feels delayed?
220 2017-10-05 19:13:07	0|gmaxwell|The general idea is that we aren't sufficiently agressive about punting peers on different consensus rules, so they can DOS attack us by sucking up slots, potentially hours per peer leaving us isolated... So there are number of things we can to do seek and destroy to speed up up.
221 2017-10-05 19:13:10	0|wumpus|luke-jr: what feels delayed?
222 2017-10-05 19:13:15	0|luke-jr|wumpus: 0.16
223 2017-10-05 19:13:17	0|Murch|@achow101: If you want to collaborate on a write-up, I'd make myself available for that.
224 2017-10-05 19:13:21	0|gmaxwell|luke-jr: release schedule is delayed because of 0.15.1
225 2017-10-05 19:13:24	0|achow101|Murch: ok
226 2017-10-05 19:13:24	0|luke-jr|i c
227 2017-10-05 19:13:31	0|wumpus|luke-jr: yes, two months extra added, I mention that in the issue
228 2017-10-05 19:13:53	0|achow101|what I wanted to discuss was the way to actually go about determining who to ban
229 2017-10-05 19:14:21	0|Murch|@achow101: Gonna be traveling the next three weeks, so I might actually have more time. ;)
230 2017-10-05 19:14:23	0|sipa|what is the issue with just looking at headers?
231 2017-10-05 19:14:24	0|gmaxwell|achow101: I was kinda hoping we could implement something just from the messages we already get, it's my belief (could be wrong) that effectively we always learn the peers best header chain-- so we can begin kicking off peers based on that, as a first pass.
232 2017-10-05 19:14:35	0|luke-jr|achow101: this contradicts the fixes in #10593
233 2017-10-05 19:14:37	0|gribble|https://github.com/bitcoin/bitcoin/issues/10593 | Relax punishment for peers relaying invalid blocks and headers by luke-jr · Pull Request #10593 · bitcoin/bitcoin · GitHub
234 2017-10-05 19:14:59	0|gmaxwell|achow101: I think we should be also drawing a distinction between inbound and outbound: the issue is what if we have a peer that accepts a broader set of blocks but would switch to our chain after learning of it.
235 2017-10-05 19:15:15	0|achow101|gmaxwell: that's what I am not sure about. I don't think we necessarily know our peer's best header chain. suppose both us and them are fully synced, how do we know their best header chain until a new block appears?
236 2017-10-05 19:15:47	0|wumpus|luke-jr: maybe two months is too much, but we'll see...
237 2017-10-05 19:16:04	0|luke-jr|wumpus: nah, that sounds reasonable
238 2017-10-05 19:16:07	0|sipa|achow101: when a new block appears, assuming it's PoW-valid to us, we'll learn about it through inv/headers/cb/...
239 2017-10-05 19:16:15	0|jonasschnelli|Yes. +2 M seems okay to me
240 2017-10-05 19:16:36	0|gmaxwell|sipa: but I believe he's right, we would have to wait for a new block, which is among the situations we're trying to resolve.
241 2017-10-05 19:17:07	0|achow101|sipa: right, but I'm concerned about before a new block appears. we just connected to them or they just connected to us. we want to know then if we should ban them or not
242 2017-10-05 19:17:21	0|luke-jr|IMO the desirable logic would be: for outbound connections, disconnect (don't ban) peers that aren't on the same chain; for inbound, tolerate it unless they reject a known-valid block
243 2017-10-05 19:17:23	0|sdaftuar|we send getheaders messages on connect, typically
244 2017-10-05 19:17:26	0|gmaxwell|For example say we are surrounded by ForkCoin peers, they are rejecting all bitcoin blocks.  There are few forkcoin miners so they only get blocks once per day.
245 2017-10-05 19:18:02	0|gmaxwell|We don't want to wait for them to get a new block just to figure out our current batch of peers are already on a chain we reject.
246 2017-10-05 19:18:03	0|achow101|sdaftuar: are you sure? all I could find is that we sometimes send getheaders, not all the time
247 2017-10-05 19:18:30	0|sdaftuar|achow101: we send getheaders messages to all our peers at some point after startup, but they might ignore them
248 2017-10-05 19:18:35	0|cfields|sdaftuar: not to incoming light clients, i think?
249 2017-10-05 19:18:36	0|sdaftuar|eg if they are doing ibd themselves or something
250 2017-10-05 19:18:45	0|sdaftuar|not to light clients, correct
251 2017-10-05 19:18:53	0|sipa|light clients don't matter here
252 2017-10-05 19:18:54	0|sdaftuar|but to inbound node_network ndoes we do
253 2017-10-05 19:19:08	0|gmaxwell|If we _always_ sent getheaders and then kicked outbound peers whos chain has a block we've rejected, then I think that is the best we can do per that concern (still not a perfect fix, since you're isolated until forkcoin finds at least one block)
254 2017-10-05 19:19:19	0|achow101|sdaftuar: if we are sending getheaders, if they are on a different chain, we still wouldn't necessarily know because our start block may not be on their best chain
255 2017-10-05 19:19:24	0|gmaxwell|oh hm. then perhaps we already do where it matters.
256 2017-10-05 19:19:43	0|sdaftuar|gmaxwell: the difficult part might be that you don't know the chain they're on is invalid
257 2017-10-05 19:19:48	0|sdaftuar|if it's got less work than yours
258 2017-10-05 19:19:54	0|RealM9|Topic suggestion: s2x
259 2017-10-05 19:20:05	0|jonasschnelli|RealM9: no
260 2017-10-05 19:20:07	0|luke-jr|sdaftuar: do you care?
261 2017-10-05 19:20:29	0|luke-jr|if they're rejecting your better chain, you want to disconnect them anyway
262 2017-10-05 19:20:34	0|gmaxwell|sdaftuar: seems like a seperate concern, we should also be kicking outbound peers that have less work than us, I think.
263 2017-10-05 19:20:39	0|RealM9|Ok, but community is pretty interested. Are you going to change POW?
264 2017-10-05 19:20:44	0|gmaxwell|But it would be silly to be overly agressive.
265 2017-10-05 19:20:44	0|sdaftuar|gmaxwell: i think that would be a good idea, yeah
266 2017-10-05 19:20:49	0|sipa|RealM9: us?
267 2017-10-05 19:20:58	0|achow101|sdaftuar: gmaxwell what I propose is that we send a getheaders for our earliest known invalid block (within a certain time frame) and see if they respond with invalid blocks
268 2017-10-05 19:21:19	0|luke-jr|RealM9: that's a decision for the community, not for developers. anyhow, ask on #bitcoin if you really want to discuss it
269 2017-10-05 19:21:35	0|gmaxwell|achow101: I don't think we need to do that: for sync purposes any outbound peer we should be makign sure we learn their headers chain period (they may have a better chain than us and we should sync up ASAP)
270 2017-10-05 19:21:37	0|sdaftuar|achow101: i'm not sure that's necessary?
271 2017-10-05 19:21:46	0|morcos|RealM9: we're in a meeting , but please see: https://bitcoincore.org/en/2017/08/18/btc1-misleading-statements/
272 2017-10-05 19:21:50	0|gmaxwell|achow101: if we're already doing that we'll notice known invalid block in their header chaip (well we will once we have code for that)
273 2017-10-05 19:21:53	0|sdaftuar|i think if we do gmaxwell's suggestion of booting inbound peers who are on less work chains, then we'd be in good shape
274 2017-10-05 19:21:59	0|sdaftuar|s/inbound/outbound/
275 2017-10-05 19:22:15	0|luke-jr|I think we may actually want to track the headers of invalid chains..
276 2017-10-05 19:22:22	0|achow101|what about inbound peers?
277 2017-10-05 19:22:32	0|gmaxwell|For _inbound_ I think we should be setting a flag that they're consensus inconsistent which excludes them from the inbound peer management connection reservation.
278 2017-10-05 19:22:36	0|sdaftuar|achow101: i think we should more aggerssively evict inbound peers if they appear to be on invalid chains
279 2017-10-05 19:22:44	0|gmaxwell|so they'll get kicked off in favor of other inbound peers.
280 2017-10-05 19:22:53	0|meshcollider|Agreed
281 2017-10-05 19:23:03	0|gmaxwell|so we don't need to be agressive: they'll just get pushed out by other inbound peers.
282 2017-10-05 19:23:03	0|luke-jr|consider: if an invalid chain has higher hashrate than the real chain, and then suddenly the invalid chain's hashrate drops off, without an equivalent increase on the main chain, we should consider that a possible attack and hold back on confirming transactions until it is resolved
283 2017-10-05 19:23:17	0|sdaftuar|gmaxwell: yes i agree with you
284 2017-10-05 19:23:50	0|achow101|my point is how do we know that an inbound peer is on an invalid chain?
285 2017-10-05 19:24:04	0|gmaxwell|luke-jr: I think there is some need for smarter wallet confirmation logic but I think thats a seperate matter. (there was a paper 6-ish months ago that also points out the the reorg probablity math in the whitepaper is somewhat incomplete)
286 2017-10-05 19:24:20	0|sdaftuar|achow101: set a flag if they relay an invalid block/blockheader
287 2017-10-05 19:24:33	0|luke-jr|gmaxwell: right, but this is relevant because we can't assume "relays invalid headers" means the other node *accepts* the invalid block
288 2017-10-05 19:24:45	0|gmaxwell|and we still interogate their headers if they're NODE_NETWORK/NODE_LIMITED
289 2017-10-05 19:24:59	0|luke-jr|sdaftuar: we intentionally relay blocks before checking validity now
290 2017-10-05 19:25:03	0|achow101|sdaftuar: that requires them to have a block to relay to us, which could take hours or days
291 2017-10-05 19:25:19	0|gmaxwell|luke-jr: the protocol does not have you realying a header of a block you haven't accepted. If you do that you risk dos attacking peers already.
292 2017-10-05 19:25:20	0|sdaftuar|achow101: i don't think we need to worry as much about inbound peers
293 2017-10-05 19:25:35	0|sdaftuar|achow101: for instance an attacker can already try to use all your inbound slots and not send you anything
294 2017-10-05 19:25:41	0|gmaxwell|luke-jr: the only place that happens in the protocol is HB BIP152 messages.
295 2017-10-05 19:25:52	0|achow101|sdaftuar: right, ok
296 2017-10-05 19:25:59	0|luke-jr|gmaxwell: which may be all you see from CB peers
297 2017-10-05 19:26:19	0|gmaxwell|sdaftuar: yes, for inbound we can just deprive them of reservations.
298 2017-10-05 19:26:50	0|sdaftuar|luke-jr: even with bip152 the headers need to be valid
299 2017-10-05 19:27:05	0|luke-jr|sdaftuar: yes, the header itself; but it can be a valid header for an invalid block
300 2017-10-05 19:27:17	0|gmaxwell|yes, though we'd catch it on the _next_ block.
301 2017-10-05 19:27:20	0|sdaftuar|luke-jr: if it builds on an invalid chain, i believe the header would be invalid
302 2017-10-05 19:27:26	0|achow101|so when we connect to an outbound peer, we will send them a getheaders so we know their best headers chain and ban accordingly
303 2017-10-05 19:27:30	0|luke-jr|(note I tried to keep track of peer bestblocks in #10512 and basically gave up)
304 2017-10-05 19:27:32	0|gribble|https://github.com/bitcoin/bitcoin/issues/10512 | Rework same-chain from abusing DoS banning, to explicit checks by luke-jr · Pull Request #10512 · bitcoin/bitcoin · GitHub
305 2017-10-05 19:27:42	0|gmaxwell|when they relay a CB message for a child of an invalid block.
306 2017-10-05 19:28:05	0|gmaxwell|achow101: yes, but based on the above I believe we already always send it.
307 2017-10-05 19:28:12	0|achow101|the other part of 11446 is to ban other peers for relaying us an invalid block for which we already know is invalid
308 2017-10-05 19:28:21	0|achow101|but I'm not sure how that interacts with compact blocks
309 2017-10-05 19:28:21	0|gmaxwell|achow101: because we send it to nodenetwork peers and outbound always are (or or disconnected)
310 2017-10-05 19:28:34	0|gmaxwell|achow101: FWIW, I think we should probably be just disconnecting and not banning.
311 2017-10-05 19:28:38	0|sdaftuar|achow101: oh that interaction might be tricky
312 2017-10-05 19:28:55	0|achow101|gmaxwell: why not ban?
313 2017-10-05 19:29:08	0|gmaxwell|I think the interaction isn't too bad, for this purpose a BIP152 CB HB block is relaying you the header of its parent.
314 2017-10-05 19:29:23	0|luke-jr|achow101: in a softfork, old nodes will send invalid blocks
315 2017-10-05 19:29:28	0|luke-jr|potentially
316 2017-10-05 19:29:43	0|gmaxwell|achow101: because it's hardly any better and it means that when some dimbulb tries running forkcoin it results in him being unable to run Bitcoin (perhaps concurrently) on the same host.
317 2017-10-05 19:30:03	0|achow101|gmaxwell: ok
318 2017-10-05 19:30:04	0|gmaxwell|it also blocks inbound from that peer, which we'd be find allowing.
319 2017-10-05 19:30:10	0|gmaxwell|s/find/fine/
320 2017-10-05 19:30:35	0|gmaxwell|In general we should be moving away from bans except when the thing we banned for was expensive for us.
321 2017-10-05 19:31:04	0|achow101|so 11446 can really just be reduced to an ~1 line change to disconnect on a header for a block we already know is invalid
322 2017-10-05 19:31:12	0|BlueMatt|yea, that
323 2017-10-05 19:31:19	0|sdaftuar|achow101: agree, though we have to be careful about compact blocks i think
324 2017-10-05 19:31:21	0|luke-jr|achow101: aka #10593 …
325 2017-10-05 19:31:23	0|gribble|https://github.com/bitcoin/bitcoin/issues/10593 | Relax punishment for peers relaying invalid blocks and headers by luke-jr · Pull Request #10593 · bitcoin/bitcoin · GitHub
326 2017-10-05 19:31:49	0|gmaxwell|achow101: yes, but for compact block interactions (HB mode will relay us blocks that are invalid).
327 2017-10-05 19:32:27	0|achow101|gmaxwell: so we would have to check the specific type of invalidness and whether it was a CB?
328 2017-10-05 19:32:32	0|gmaxwell|luke-jr: IIRC when you proposed that before you got squaked at that it would undermine our protection against isolation...
329 2017-10-05 19:33:05	0|gmaxwell|achow101: or just don't do it for the peers maked for HB CBs for now
330 2017-10-05 19:33:18	0|achow101|gmaxwell: isn't that likely to be most peers though
331 2017-10-05 19:33:20	0|achow101|?
332 2017-10-05 19:33:24	0|gmaxwell|No, it's at most three.
333 2017-10-05 19:33:34	0|luke-jr|gmaxwell: I don't see how. It's literally what achow101 was just describing.
334 2017-10-05 19:34:33	0|luke-jr|gmaxwell: maybe you're thinking of the predecessor 10512?
335 2017-10-05 19:34:36	0|achow101|luke-jr: it doesn't look like you are handling invalid duplicates?
336 2017-10-05 19:34:41	0|gmaxwell|probably.
337 2017-10-05 19:34:59	0|gmaxwell|achow101: in any case, we can pick this up on a PR and later discussion.
338 2017-10-05 19:35:10	0|achow101|gmaxwell: ok
339 2017-10-05 19:36:09	0|achow101|next topic then?
340 2017-10-05 19:36:09	0|wumpus|any other topics?
341 2017-10-05 19:36:10	0|luke-jr|achow101: IIRC it does, we can go over it later if you like
342 2017-10-05 19:36:14	0|achow101|luke-jr: ok
343 2017-10-05 19:37:10	0|jnewbery|luke-jr: any progress on multiwallet GUI without the rpcauth parts?
344 2017-10-05 19:37:22	0|wumpus|#topic multiwallet ui
345 2017-10-05 19:37:33	0|luke-jr|jnewbery: not yet, I'll plan to push the update later today
346 2017-10-05 19:38:05	0|jnewbery|great! I had a look myself, and I think it's just a one-line change to the debug console commit
347 2017-10-05 19:38:06	0|jonasschnelli|https://github.com/bitcoin/bitcoin/pull/11383
348 2017-10-05 19:38:13	0|sipa|topic suggestion: dealing with platform-specific code
349 2017-10-05 19:38:25	0|jonasschnelli|luke-jr: I can continue to work on 11383 if you want?
350 2017-10-05 19:38:33	0|jonasschnelli|(remove the auth stuff :P)
351 2017-10-05 19:38:51	0|luke-jr|jnewbery: certainly not that simple.. still need to resolve wallet name to CWallet earlier
352 2017-10-05 19:39:14	0|jnewbery|ok, well I've got a branch that works with just that change. Happy to share with you
353 2017-10-05 19:39:48	0|gmaxwell|Sounds good.
354 2017-10-05 19:40:01	0|luke-jr|jnewbery: push it and I'll take a look
355 2017-10-05 19:40:18	0|jnewbery|thanks
356 2017-10-05 19:41:03	0|wumpus|#topic dealing with platform-specific code (sipa)
357 2017-10-05 19:41:20	0|sipa|i've recently been looking into faster parallel hashing code
358 2017-10-05 19:41:37	0|wumpus|hashing as in sha256?
359 2017-10-05 19:41:51	0|sipa|in particular, for 8-way parallel SHA256 (which would be useful in merkle root computation and block deserialization), a 5x speedup is doable with AVX2
360 2017-10-05 19:42:05	0|sipa|and maybe 2.5x with SSE2
361 2017-10-05 19:42:23	0|wumpus|and parallel in this case means computing multiple hashes of multiple pieces of data at once?
362 2017-10-05 19:42:28	0|sipa|correct
363 2017-10-05 19:42:29	0|luke-jr|how much speedup is this for the entire IBD?
364 2017-10-05 19:42:58	0|gmaxwell|luke-jr: It saves something like 10 minutes on IBD.  But the greater impact is in block relay.
365 2017-10-05 19:43:01	0|wumpus|(I guess there are constraints there, do all the inputs need to be the same size?)
366 2017-10-05 19:43:06	0|gmaxwell|Where hash tree computation is most of the time.
367 2017-10-05 19:43:06	0|luke-jr|I imagine merkle root is a tiny fraction of the overall process, but otoh it's also possibly a blocker on parallelization
368 2017-10-05 19:43:26	0|luke-jr|gmaxwell: it is? :O
369 2017-10-05 19:43:33	0|luke-jr|oh, because the signature checks are cached?
370 2017-10-05 19:43:33	0|sipa|wumpus: yes and no; for now, it's just a primitive that you give a pointer to N 64-byte inputs, and produces 32-byte outputs
371 2017-10-05 19:43:38	0|BlueMatt|in terms of compact block relay, merkle root calculation and deserialize are about the only big timesinks before you can relay
372 2017-10-05 19:43:47	0|sipa|wumpus: which is specific for merkle root computation
373 2017-10-05 19:43:49	0|gmaxwell|luke-jr: yes for HB BIP152 we don't to validation except hashing!
374 2017-10-05 19:43:51	0|sipa|but it can certainly be adapted
375 2017-10-05 19:44:38	0|wumpus|sipa: ok
376 2017-10-05 19:44:39	0|cfields|sipa: i had a scare when reviewing some new leveldb crc32 changes that i thought (at first glance) could be a consensus issue. I was very angry at myself at that point for not adding a fallback un-optimized verification of the optimized path.
377 2017-10-05 19:44:44	0|sipa|anyway, there are multiple ways to integrate this: separate asm code, inline asm blocks, or code using intrinsics (my preference, it's much more easy to review, and has no OS-specific warts like the L label prefix...)
378 2017-10-05 19:44:57	0|gmaxwell|sipa has actually implemented the 8-way AVX2 sha2 and a hash tree computation that uses it... along with specialized implementation of 64-byte input double sha2.. which affords an addition 20%-ish speedup over generic sha2.
379 2017-10-05 19:45:10	0|cfields|very cool :)
380 2017-10-05 19:45:12	0|wumpus|I prefer intrinsics
381 2017-10-05 19:45:21	0|wumpus|(except for arm32 whre they suck)
382 2017-10-05 19:45:49	0|sipa|so, for intrinsics... do we want to have a separate LIBCRYPTO_AVX2 LIBCRYPTO_SSE2 LIBCRYPTO_... with different compile flags each?
383 2017-10-05 19:46:06	0|gmaxwell|Historically, For some code you cannot achieve equivilent performance w/ intrinsics because you must manage register allocation precisely for things to work, but that isn't the case here....
384 2017-10-05 19:46:06	0|sipa|or could we rely on __attribute__((target("avx2")))
385 2017-10-05 19:46:09	0|wumpus|but 64 bit platforms the SIMD instructions have been specially tweaked to work well with compilers and intrinsics
386 2017-10-05 19:46:13	0|sipa|(which works on both clang and gcc)
387 2017-10-05 19:46:37	0|cfields|sipa: i think we should test for the target attribute and use it if possible, but not completely rely on it
388 2017-10-05 19:46:45	0|cfields|iirc that improves dispatching time as well?
389 2017-10-05 19:46:50	0|sipa|cfields: no
390 2017-10-05 19:46:58	0|wumpus|different compile flags for different compile units, that's the only portable way
391 2017-10-05 19:47:02	0|luke-jr|sipa: I think we can't assume intrinsics for all platforms, so we want the separate lib route
392 2017-10-05 19:47:08	0|gmaxwell|dispatching is via a function pointer ultimately in all those cases.
393 2017-10-05 19:47:14	0|wumpus|luke-jr: you're confusing, that's not about intrinsics
394 2017-10-05 19:47:31	0|sipa|the only difference is avoiding the need for build system complication
395 2017-10-05 19:47:36	0|wumpus|intrinsics inthis case are headers like xmmintr.h which provides functions that work on vector types
396 2017-10-05 19:47:43	0|sipa|exactly
397 2017-10-05 19:47:52	0|luke-jr|wumpus: __attribute__((target("avx2"))) isn't an option for separate asm code, though, right?
398 2017-10-05 19:48:01	0|sipa|luke-jr: it also isn't needed for asm code
399 2017-10-05 19:48:03	0|cfields|gmaxwell: isn't there elf data that allows them to be setup at load time?
400 2017-10-05 19:48:14	0|wumpus|oh no no ELF magic please
401 2017-10-05 19:48:25	0|luke-jr|hmm
402 2017-10-05 19:48:37	0|cfields|not by hand, i thought __attribute__(target) did that behind the scenes
403 2017-10-05 19:48:42	0|sipa|target("avx2") just means "this function is compiled as if -mavx2 was passed on the command line
404 2017-10-05 19:49:03	0|sipa|cfields: GCC also has target("default"), where you can have multiple versions of the same function... which causes automatic dispatch to be added
405 2017-10-05 19:49:09	0|sipa|that's non-portable and has other issues
406 2017-10-05 19:49:10	0|gmaxwell|cfields: they're setup at load time, yes-- but they're still just a function pointer, which we could also have setup at load time.  Though it is nice that the function override trick can make them run before main.
407 2017-10-05 19:49:12	0|wumpus|I'm normally not scared of low level ELF hacking, but for bitcoin, let's try to keep it safe and portable
408 2017-10-05 19:49:13	0|luke-jr|sipa: how does it behave if I have an explicit -mno-avx2?
409 2017-10-05 19:49:16	0|sipa|(in particular clang doesn't have that)
410 2017-10-05 19:49:18	0|cfields|sipa: ah yes, that's what i was thinking of.
411 2017-10-05 19:49:34	0|sipa|cfields: so i'm not suggesting using that
412 2017-10-05 19:49:40	0|sipa|luke-jr: i assume it overrides it
413 2017-10-05 19:49:44	0|cfields|ok
414 2017-10-05 19:50:03	0|luke-jr|I suppose I can test it
415 2017-10-05 19:50:06	0|wumpus|yes, gcc can do it automatically on some platforms, but I'm afraid the only portable way is to make our own dispatch logic
416 2017-10-05 19:50:16	0|sipa|yes, we'll want our own dispatch logic anyway
417 2017-10-05 19:50:18	0|wumpus|we already have some CPUID bits checking
418 2017-10-05 19:50:21	0|sipa|so we can test things
419 2017-10-05 19:50:22	0|wumpus|so it's nothing new erally
420 2017-10-05 19:50:26	0|sipa|and report which version is being chosen
421 2017-10-05 19:50:30	0|cfields|np, i wasn't suggesting. just trying to understand the advantages of one vs the other.
422 2017-10-05 19:50:31	0|wumpus|yes, exactly
423 2017-10-05 19:51:03	0|sipa|but if possible i'd like to avoid the overhead of needing half a dozen libcrypto_XXX.a things that need to be linked in everywhere
424 2017-10-05 19:51:09	0|sipa|though that's really the only advantage
425 2017-10-05 19:51:39	0|wumpus|so I'd say: yes, use intrinsics instead of inline/offline asm,  and use our own dispatching, and compile units compiled with appropriate compiler flags
426 2017-10-05 19:51:56	0|sipa|okay.
427 2017-10-05 19:52:29	0|gmaxwell|can we say prefer intrinsics instead of use? :) I don't think we'd eschew inline asm if we thought it was better in a particular case.
428 2017-10-05 19:52:30	0|wumpus|yes regarding build system it's just verbose, not really complex
429 2017-10-05 19:52:42	0|cfields|sipa: see my point above about a fallback, though. In the case of mismatch hashes, i think it's worthwhile to re-check with a generic implementation before deciding it's failed.
430 2017-10-05 19:53:13	0|gmaxwell|cfields: we should be testing these things in startup tests.
431 2017-10-05 19:53:13	0|luke-jr|(yes, it seems to override -mno-*
432 2017-10-05 19:53:18	0|wumpus|gmaxwell: well if there's a case you can do much better than the compiler, sure...
433 2017-10-05 19:54:39	0|cfields|gmaxwell: an implementation bug in some branch of one optimized path is scary...
434 2017-10-05 19:54:57	0|gmaxwell|cfields: try differently if it fails is just not reasonable in a lot of cases; and often would add a lot of complexity (now you have to not cache hashes, but instead only use hash-verify methods) ... and we don't have expected values for thigns like single transaction hashes, just hash roots.
435 2017-10-05 19:54:58	0|cfields|gmaxwell: in particular, the crc issue had to do with incoming data alignment on x86_64
436 2017-10-05 19:55:25	0|wumpus|cfields: I agree
437 2017-10-05 19:55:51	0|wumpus|cfields: I think we should only do asm optimization in cases where it really makes a lot of difference, for that reason, ther risk has to be worth it
438 2017-10-05 19:56:02	0|gmaxwell|cfields: yes, thats something that always needs careful review and we should have unit tests that also stress alignment.
439 2017-10-05 19:56:34	0|wumpus|special-casing everything makes things a lot harder to review, and test, especially when it starts to need different kinds of hardware
440 2017-10-05 19:56:54	0|wumpus|but for testable low-level primitives like SHA256 I'd say it's ok
441 2017-10-05 19:57:12	0|gmaxwell|good thing no one is talking about special casing everything. :)
442 2017-10-05 19:57:30	0|gmaxwell|yea, sha2 etc have simple testable interfaces.
443 2017-10-05 19:57:37	0|wumpus|no, that's just one extreme, I've seen soome graphics drivers which are scary in that regard :)
444 2017-10-05 19:58:10	0|gmaxwell|But benchmarks!
445 2017-10-05 19:58:11	0|wumpus|oh let's special case 4x4 tiles, 4x5 tiles, 4x6 tiles, ... for 3 different architectures
446 2017-10-05 19:58:21	0|wumpus|right :)
447 2017-10-05 19:58:28	0|cfields|mmm. I don't see the harm in doing a quick re-check in a few certain cases (merkle mismatch is a good example)
448 2017-10-05 19:58:52	0|wumpus|special-casing benchmarks is a curious form of over-learning
449 2017-10-05 19:58:56	0|cfields|anyway, i've made my case
450 2017-10-05 19:59:02	0|gmaxwell|cfields: because it requires restructing the code to not return hashes but instead only have uncachable hash_Verify methods.
451 2017-10-05 19:59:07	0|wumpus|cfields: re-check in what case?
452 2017-10-05 19:59:25	0|luke-jr|although someone did manage to screw up xpub serialisation at one point IIRC
453 2017-10-05 19:59:25	0|wumpus|cfields: you mean re-run the validation w/ different implementations if an  incoming block fails?
454 2017-10-05 19:59:56	0|wumpus|(what about false positives?)
455 2017-10-05 20:00:01	0|cfields|wumpus: that's a big hammer, but yes-ish
456 2017-10-05 20:00:12	0|gmaxwell|cfields: and for small functions like a hash a check in an innerloop will measurably lower performance. ... and you also create the opposite problem, what if the alternative function is the wrong one?
457 2017-10-05 20:00:27	0|gmaxwell|(I'd actually consider whole block level more reasonable)
458 2017-10-05 20:00:30	0|wumpus|gmaxwell: I think he means on a high level
459 2017-10-05 20:01:03	0|wumpus|on the inner level it's just NASA-level crazy, let's run three implementations and see which ones agree
460 2017-10-05 20:01:07	0|sipa|i think re-checking a block if it fails is reasonable... but why switch hash functions? it's massively more likely your CPU is fried than that the hash function implementation is wrong all along and you never noticed
461 2017-10-05 20:01:14	0|gmaxwell|but then the dispatch is mutable not just set once at init. :(
462 2017-10-05 20:01:29	0|wumpus|yeah ... I think we're overdesigning this
463 2017-10-05 20:01:31	0|gmaxwell|right we have a constant slow stream of complaints from users whos hosts have falsely rejected the blockchain.
464 2017-10-05 20:01:38	0|wumpus|just continue with what you were doing sipa :)
465 2017-10-05 20:01:41	0|cfields|gmaxwell: just have a generic non-dispatchable one
466 2017-10-05 20:01:42	0|gmaxwell|I would like to see that improved somehow.
467 2017-10-05 20:01:44	0|wumpus|any other topics?
468 2017-10-05 20:01:47	0|wumpus|oh wait, it's time
469 2017-10-05 20:01:57	0|wumpus|#endmeeting
470 2017-10-05 20:01:58	0|lightningbot|Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.log.html
471 2017-10-05 20:01:58	0|lightningbot|Meeting ended Thu Oct  5 20:01:57 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
472 2017-10-05 20:01:58	0|lightningbot|Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.html
473 2017-10-05 20:01:58	0|lightningbot|Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.txt
474 2017-10-05 20:02:03	0|sipa|#lunch
475 2017-10-05 20:02:52	0|meshcollider|And for anyone who doesn't know yet, hacktoberfest = free t-shirt for 4 PRs in october
476 2017-10-05 20:02:57	0|meshcollider|https://hacktoberfest.digitalocean.com
477 2017-10-05 20:03:13	0|sipa|if someone is interested: https://github.com/sipa/bitcoin/commits/201709_dsha256_64 <- 64-byte specialzed SHA256 and AVX2 code
478 2017-10-05 20:03:15	0|gmaxwell|sneaky way to learn developers mailing addresses.
479 2017-10-05 20:03:21	0|sipa|(way too WIP to PR)
480 2017-10-05 20:03:28	0|achow101|ooh free t-shirts
481 2017-10-05 20:03:53	0|meshcollider|gmaxwell: Heh true
482 2017-10-05 20:03:55	0|cfields|sipa: very cool. didn't mean to rain on your parade.
483 2017-10-05 20:03:57	0|luke-jr|gmaxwell: never got any spam that I can tell came from last year's
484 2017-10-05 20:04:09	0|luke-jr|just don't use your home address for mailing address ;)
485 2017-10-05 20:04:24	0|wumpus|sipa: nice
486 2017-10-05 20:05:09	0|gmaxwell|cfields: so imagine that sha2 implementation isn't alignment safe. you get a block and miscompute one of the hashes due to alignment. ...  I don't see any way of efficiently accomplishing your 'try another function' approach that would stop the false rejection.
487 2017-10-05 20:05:57	0|sipa|wumpus: too bad the 64-byte specialized generic-x86 code is slower than the generic-data-size SSE4-specialized version
488 2017-10-05 20:06:08	0|sipa|wumpus: otherwise i'd straight up PR the 64-byte specialized code
489 2017-10-05 20:06:15	0|wumpus|the avx code looks very recognizable
490 2017-10-05 20:06:27	0|sipa|wumpus: it's pretty much search replace on SSE4 code
491 2017-10-05 20:06:35	0|wumpus|heh
492 2017-10-05 20:06:48	0|gmaxwell|it's only detectable at the hash root check at the end a long computation pipeline... when that fails do we just go back and re-deseralize the entire block with different code to compute new hashes in the CTransactions?
493 2017-10-05 20:07:44	0|gmaxwell|(and if we do, we magnify a DOS vector for someone sending us invalid blocks, though perhaps not enough to worry about)
494 2017-10-05 20:07:59	0|wumpus|that's another advantage of intrinsics, it's usually easier to review than straight up asm
495 2017-10-05 20:08:07	0|sipa|wumpus: absolutely
496 2017-10-05 20:08:17	0|wumpus|no need to keep track in your head where all the registers go
497 2017-10-05 20:08:31	0|cfields|gmaxwell: addmittedly that's ugly, but yes, i think that's worth considering
498 2017-10-05 20:08:37	0|sipa|which may be a good thing or a bad thing
499 2017-10-05 20:08:48	0|sipa|1) no need to keep in your head where the registers go!
500 2017-10-05 20:08:58	0|sipa|2) no way to tell the compiler to keep a certainly value in a register!
501 2017-10-05 20:09:40	0|wumpus|well combined with the pipelining/interleaving that optimized code needs, and the large number of registers that SIMD architectures have, that can be quite difficult
502 2017-10-05 20:09:41	0|gmaxwell|I wish you could assign variables to registers, and have the compiler yell at you if you tried to assign two live variables to the same register.
503 2017-10-05 20:10:40	0|wumpus|usually there's so many registers that it would be good to be able to tell that something is *not* worth storing in a register
504 2017-10-05 20:10:54	0|sipa|i believe there is actually a way (in GCC) to force a particular variable into a particular register
505 2017-10-05 20:11:05	0|sipa|int x asm("%edx");
506 2017-10-05 20:11:26	0|gmaxwell|cfields: and still doesn't help with accepting something we shouldn't, which is usually a more serious issue.
507 2017-10-05 20:12:33	0|gmaxwell|cfields: if we had some generic infrastructure to retry a failed validation (e.g. to cope with lossy hardware) then perhaps what you're thinking about could be dropped into it.
508 2017-10-05 20:12:48	0|cfields|gmaxwell: i'm not saying it's something we have to do, or something that wouldn't be ugly. I'm moreso coming from a place where I was in full-out panic for a few hours because I thought newer x86_64 machines were about to start diverging...
509 2017-10-05 20:12:54	0|sipa|https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
510 2017-10-05 20:13:12	0|cfields|so you're right, (good!) testing should negate those worries.
511 2017-10-05 20:13:38	0|gmaxwell|Well, in particular runtime tests... since they'll catch failures on the actual hardware the user has.
512 2017-10-05 20:13:56	0|cfields|gmaxwell: for reference: https://github.com/google/leveldb/commit/2964b803b857932ff7499d7bebb61dc5514dab7c
513 2017-10-05 20:13:59	0|cfields|yes
514 2017-10-05 20:19:23	0|morcos|I also don't want to rain on anyone's parade, and this is OSS so people work on what they find interesting, but I think we shoudl be careful to think about what are priorities for the project
515 2017-10-05 20:19:54	0|morcos|More importantly however, we shoudl be careful about making changes that are not easily reviewable by more than 1 or 2 people unless they are really warranted
516 2017-10-05 20:21:00	0|morcos|I've been thinking more about this since bech32.  I think bech32 is completely awesome, I'm super happy we're doing it and imo its a good priority project. But it essentially got no review.  Did anyone other than sdaftuar review it?
517 2017-10-05 20:21:45	0|morcos|Sometimes things will have to be like that, but it shoudl be a tradeoff we consider carefully...  how tricky are we trying to be vs how much is it warranted
518 2017-10-05 20:22:15	0|morcos|I haven't evaluated that in the context of the parallelized hashing, but its something I think we should
519 2017-10-05 20:25:13	0|gmaxwell|morcos: bech32 had more review then it appeared because we solicited extensive review before publishing the bip.. what didn't get review was the checksum itself other than myself and pieter, until sdaftuar.  ... but who reviewed the prior address checksum?
520 2017-10-05 20:25:31	0|gmaxwell|Clearly no one, because it needlessly sucked. :P
521 2017-10-05 20:25:56	0|sipa|well it was too late to review it by the time any of us were around
522 2017-10-05 20:26:28	0|luke-jr|I didn't review Bech32 because I figured it was over my head (especially magic checksum stuff).
523 2017-10-05 20:26:42	0|gmaxwell|There were many design changes to the earliest Bech32 proposal that arose out of review, e.g. the delimiter character.
524 2017-10-05 20:27:28	0|morcos|Yes and of course the 2 people that can't be blamed are you and pieter since you did the work.  But I don't want us to fall into a trap of just assuming if something is too difficult or outside of our field to review properly that someone else must be doign a good job with it
525 2017-10-05 20:28:04	0|morcos|gmaxwell: and yes i was referring to the checksum
526 2017-10-05 20:28:16	0|morcos|but thats a good point...
527 2017-10-05 20:29:39	0|gmaxwell|We deal with this for libsecp256k1 too, that fact that it's in a different repo is just enabling you to ignore it. :)
528 2017-10-05 20:29:55	0|gmaxwell|Though we do have effective review there too.
529 2017-10-05 20:30:14	0|gmaxwell|Though not as much as I'd like.
530 2017-10-05 20:30:42	0|sipa|but at least for secp256k1 it's clear what the goal is (implement secp256k1 EC correctly), so someone could review e.g. just the test and judge that they're sufficient for that goal
531 2017-10-05 20:30:55	0|cfields|morcos: yes, well said. I think that's why I find the asm changes scary. I spent a full day trying to learn and understand the sse42 optimized sha2, and only because it was failing some tests. At best, I agree that it looks right, but I could never say it with any degree of certainty. I deferred to "it passes all the tests".
532 2017-10-05 20:31:11	0|sipa|with bech32, the effort is in the design, not the implementation, and there is relatively little proof that the design actually has the properties it claims to have
533 2017-10-05 20:31:29	0|gmaxwell|But thats the same thing as reviewing the bech32 checksum. and fwiw, if feedback on bech32 we got was we needed more reviewers for the checksum, I would have gone and asked a libsecp256k1 contributor to do it.
534 2017-10-05 20:32:08	0|gmaxwell|because I know e.g. andrew (or peter dettman) aren't frightened off by math.
535 2017-10-05 20:32:14	0|morcos|I think my feedback is one meta level up.  There should have been more people that questioned how much reivew it got
536 2017-10-05 20:32:22	0|gmaxwell|yes, agreed.
537 2017-10-05 20:32:39	0|gmaxwell|well I was thinking that before you commented. Started thinking it as soon as I saw other bitcoin software had merged it.
538 2017-10-05 20:32:51	0|morcos|so i know i'm not going to review the hashing stuff, just want to make sure other peopel are going to ensure it is properly reviewed
539 2017-10-05 20:33:13	0|morcos|i'm frightened off by code. :)
540 2017-10-05 20:34:20	0|gmaxwell|had someone commented on that earlier, I would have agreed.  I reviewed the checksum work too, but pieter and I worked a lot togeather on it, and if he screwed up he probably would have tained me.
541 2017-10-05 20:34:25	0|sipa|the cool thing about hash functions is that they have essentially no branches... so it's very hard (though not impossible, see the alignment issue) to make it be incorrect only for a hard-to-detect small subset of inputs
542 2017-10-05 20:35:20	0|gmaxwell|yes on hash function correctness, but that doesn't help with BECH32 design, which I guess as your point above.  It's easy to be confident a new implementation of it is conformant... not so much that the design is good.
543 2017-10-05 20:35:51	0|sipa|right
544 2017-10-05 20:36:25	0|gmaxwell|and still no one has basically reviewed our decision to use a cyclic code -- perhaps if we were call coding theorists someone would have known an even better tool... but there is a limit to how far down a rabbit hole we can go.
545 2017-10-05 20:36:32	0|morcos|gmaxwell: but part of my point was the tradeoff on how important the feature is.  i am definitely +1 on bech32.  and not negative on parallel hashing just raising points for us to think about
546 2017-10-05 20:36:36	0|morcos|anyway, got to run
547 2017-10-05 21:33:38	0|BlueMatt|someone wanna close 11454?
548 2017-10-05 21:56:04	0|gmaxwell|morcos: I think you can decompose your concerns into two folks-- people will work on whatever they like, but if it's going to get merged it needs to deserve the required review attention, since that isn't just an indivigual decision.
549 2017-10-05 21:56:26	0|gmaxwell|forks*
550 2017-10-05 21:57:00	0|gmaxwell|morcos: and then unrelated, we shouldn't be in a state where people don't review or even provide meta review of things which are two mathy or too low level and so they assume that they won't be of use.
551 2017-10-05 21:57:33	0|gmaxwell|And the thing to encourage there is that even if its over your head you can still ask some of the right meta questions.
552 2017-10-05 23:15:01	0|bitcoin-git|[13bitcoin] 15theuni opened pull request #11457: Introduce BanMan (06master...06move-bandb) 02https://github.com/bitcoin/bitcoin/pull/11457
553 2017-10-05 23:29:30	0|luke-jr|jnewbery: your version seems to pass the wallet by name instead of CWalletRef