1 2017-11-30 01:27:17	0|promag|fanquake: extended tests too? https://github.com/bitcoin/bitcoin/pull/11791#issuecomment-348052591
  2 2017-11-30 01:31:59	0|fanquake|promag I can run them also
  3 2017-11-30 06:04:15	0|warren|MarcoFalke: https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-setup-gitian-fedora.md  nice!  how recently did you test this btw?
  4 2017-11-30 06:04:58	0|warren|MarcoFalke: I have had python-vm-builder RPM packages for years but it broke sometime this year and couldn't figure out how to fix it. although I use only qemu-kvm, not lxc.
  5 2017-11-30 07:19:05	0|bitcoin-git|[13bitcoin] 15Varunram opened pull request #11793: Docs: Bump OS X version to 10.13 (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/11793
  6 2017-11-30 07:19:49	0|Varunram|^ This needn't be a standalone PR in itself, can be merged with other fixes
  7 2017-11-30 07:37:41	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9254: [depends] ZeroMQ 4.2.2 (06master...06zeromq-4-2-0) 02https://github.com/bitcoin/bitcoin/pull/9254
  8 2017-11-30 10:06:54	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #11793: Docs: Bump OS X version to 10.13 (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/11793
  9 2017-11-30 10:27:54	0|aj|jnewbery: apparently github detects merge conflicts even when git merge and rebase both work fine, so i've split the preliminary changes from #11774 into #11796; hopefully 11796 can be easily merged once acked, since it shouldn't break any pending PRs
 10 2017-11-30 10:27:56	0|gribble|https://github.com/bitcoin/bitcoin/issues/11774 | [WIP] [tests] Rename functional tests by ajtowns · Pull Request #11774 · bitcoin/bitcoin · GitHub
 11 2017-11-30 10:27:57	0|gribble|https://github.com/bitcoin/bitcoin/issues/11796 | [tests] Functional test naming convention by ajtowns · Pull Request #11796 · bitcoin/bitcoin · GitHub
 12 2017-11-30 10:28:46	0|wumpus|github is really sensitive with regard to merges, more than the stock git commands
 13 2017-11-30 10:29:09	0|wumpus|might be so that at least no one blames them for merges going wrong
 14 2017-11-30 10:30:30	0|aj|wumpus: yeah, it does make sense. bit weird not being able to tell quite what github thinks the conflict is though
 15 2017-11-30 10:30:34	0|wumpus|at a company I used to work for the integration dept had a similar rule. Two changes affect the same file? let the devs sort it out
 16 2017-11-30 10:31:10	0|wumpus|yeah it could use some better reporting
 17 2017-11-30 10:33:49	0|wumpus|something I'd really like on github would be a graph of which open PRs collide with which other PRs
 18 2017-11-30 10:34:19	0|wumpus|would help a lot in finding the optimal merge order, as well as notify authors early if their PR is going to need to be rebased
 19 2017-11-30 10:35:05	0|sipa|that would be very useful
 20 2017-11-30 10:35:18	0|sipa|n^2 complexity though in the number of PRs, i think
 21 2017-11-30 10:37:50	0|wumpus|worst case, though the n^2 could be filtered by excluding PRs that affect completely distinct files
 22 2017-11-30 10:38:42	0|wumpus|also one'd want to avoid recomputing the whole thing, just incrementally take opens/closes into account
 23 2017-11-30 10:40:03	0|wumpus|(ugh, and PRs that are re-pushed)
 24 2017-11-30 10:40:46	0|aj|that just makes it O(n) for each PR change though
 25 2017-11-30 10:41:20	0|wumpus|which is reasonable, you can do it in the background, not at the time someone opens the graph
 26 2017-11-30 10:42:22	0|aj|yeah, n isn't /that/ large
 27 2017-11-30 10:43:19	0|wumpus|though yeah if github wants to do it for all projects it could get bad for them
 28 2017-11-30 10:43:43	0|wumpus|I don't really care if it would be a paid-only feature or such
 29 2017-11-30 11:00:14	0|wumpus|I've changed "Docs and output" label to just "Docs", it is only for documentation, for logging changes and such please use "Utils/logging/libs" instead.
 30 2017-11-30 11:01:00	0|wumpus|I think lumping everything output-related in one issue was a bit confusing
 31 2017-11-30 11:01:16	0|wumpus|in one label*
 32 2017-11-30 11:22:12	0|aj|-) > APPLY_${xa}_${xb} 2>&1 && echo "no conflict between $xa $xb" || echo "merge conflict $xa $xb ?"; done; else echo "conflict with $a and master"; fi; done > CONFLICT
 33 2017-11-30 11:22:12	0|aj|wumpus: had a quick play, and it seems like you can get somewhere with detecting PR conflicts using git locally. for xa in $tsts; do a=pull/origin/$xa; if (git reset --hard origin/master && git merge $a -m "merge $a to master") >/dev/null; then for xb in $tsts; do if [ "$xa" -ge "$xb" ]; then continue; fi; b=pull/origin/$xb; (git format-patch $(git merge-base $a $b)..$b --stdout | git apply --check
 34 2017-11-30 11:22:15	0|aj|gah
 35 2017-11-30 11:22:42	0|aj|wumpus: had a quick play, and it seems like you can get somewhere with detecting PR conflicts using git locally.  -- was what i was trying to say
 36 2017-11-30 11:22:53	0|aj|wumpus: https://gist.github.com/ajtowns/d0cf97678dc83efdf3f6cbf7083a35a0 -- was what i was trying to paste
 37 2017-11-30 11:22:56	0|wumpus|awesome!
 38 2017-11-30 11:24:11	0|wumpus|pretty clever that you manage to avoid touching the working tree
 39 2017-11-30 11:24:42	0|wumpus|but yes seems I won't get around to setting up a git tree that pulls all the PRs locally
 40 2017-11-30 11:25:45	0|aj|wumpus: hmm? pulling all the PRs locally is easy and awesome though?
 41 2017-11-30 11:26:43	0|wumpus|yeah no problem, just don't do it at the moment to avoid cluttering my tree further, I have 858 branches of my own at this point :)
 42 2017-11-30 11:27:09	0|fanquake|git branch -D *
 43 2017-11-30 11:28:22	0|aj|wumpus: the pull requests don't show up in git branch -av for me, so i don't even notice (actually, i don't even know how to get a list of them...)
 44 2017-11-30 11:28:53	0|wumpus|fanquake: git branch -D doesn't take a pattern :-) you'd need something like git branch | xargs git branch -D
 45 2017-11-30 11:29:22	0|wumpus|aj: interesting, I'd have expected them to show up
 46 2017-11-30 11:31:22	0|wumpus|(I regularly do 'git branch --list 'pull/*' | xargs git branch -D' to get rid of the temporary branches left behind by github-merge.py in some cases)
 47 2017-11-30 11:31:54	0|aj|ah, "git remote show origin" will at least give me a list of them
 48 2017-11-30 11:32:08	0|wumpus|apart from that I use a single repository with worktrees for persistent branches like 0.15
 49 2017-11-30 11:33:02	0|wumpus|anyhow it works pretty well, I think it will still work pretty will with 231 more branches; will have to find something to clean up closed PRs though
 50 2017-11-30 11:33:10	0|aj|updated that gist with the compatible and conflicts graphs for a handful of recent PRs
 51 2017-11-30 11:33:38	0|wumpus|or does that automatically remove removed branches at the gh side as well?
 52 2017-11-30 11:34:36	0|wumpus|aj: oh wow
 53 2017-11-30 11:36:29	0|aj|wumpus: the only thing i can think of to make it useful is dumping data into python to work out which collections of PRs have complete subgraphs of compatability
 54 2017-11-30 11:36:35	0|wumpus|so a connection means a conflict? why do 11790 and 11741 conflict?
 55 2017-11-30 11:37:28	0|wumpus|11741 only changes two files, yet conflicts with 9 PRs or so
 56 2017-11-30 11:37:36	0|wumpus|it's possible :)
 57 2017-11-30 11:40:44	0|aj|yeah, they merge fine manually; git apply --check complains about multiwallet.py, rpc/rawtransaction.cpp and test/txvalidation_tests.cpp
 58 2017-11-30 11:41:53	0|aj|ah, i'm trying to apply patches from master twice
 59 2017-11-30 11:41:54	0|wumpus|but 11741 touches neither of those
 60 2017-11-30 11:42:27	0|wumpus|hehe okay
 61 2017-11-30 11:45:33	0|fanquake|nice
 62 2017-11-30 11:48:42	0|wumpus|oope, so fetch = +refs/pull/*/head:refs/pull/origin/*  doesn't just fetch *open* pull requests
 63 2017-11-30 11:49:34	0|wumpus|whee tracking 8288 branches now
 64 2017-11-30 11:49:53	0|aj|https://gist.github.com/ajtowns/d0cf97678dc83efdf3f6cbf7083a35a0 -- conflict graph looks better now
 65 2017-11-30 11:50:08	0|wumpus|happy I did it in a temporary repo :)
 66 2017-11-30 11:52:04	0|wumpus|aj: great
 67 2017-11-30 11:56:44	0|Varunram|aj: thanks for the gist :)
 68 2017-11-30 11:58:14	0|aj|probably needs to work out which PR branched off from master most recently and use that as a base
 69 2017-11-30 14:19:30	0|wordsToLiveBy|I'm reading the bitcoin white paper, and in section 11 Satoshi gives the calculations for how you can prove with probability that an attacker can't outpace the combined processing power of the rest of the nodes, but I am not quite getting the math being used.
 70 2017-11-30 14:55:40	0|andytoshi|#bitcoin please, this is just about software development
 71 2017-11-30 15:08:18	0|jnewbery|aj: thanks. I've reviewed #11796. It should be an easy review/merge for others.
 72 2017-11-30 15:08:20	0|gribble|https://github.com/bitcoin/bitcoin/issues/11796 | [tests] Functional test naming convention by ajtowns · Pull Request #11796 · bitcoin/bitcoin · GitHub
 73 2017-11-30 15:24:57	0|aj|jnewbery: ugh, what sort of person points out a bug and says not to fix it /o\
 74 2017-11-30 15:27:02	0|saint_|any idea why can't bitcoin core download the blockchain on a remote NAS drive ?
 75 2017-11-30 15:27:19	0|wumpus|saint_: ask in #bitcoin please, will answer there
 76 2017-11-30 15:27:23	0|saint_|okay
 77 2017-11-30 16:16:02	0|jnewbery|aj: I wanted to get in before someone else pointed it out and told you to fix it :)
 78 2017-11-30 16:16:13	0|jnewbery|I don't think it matters since the next PR should remove that code anyway
 79 2017-11-30 17:25:37	0|achow101|is the github git notifier dead?
 80 2017-11-30 17:53:48	0|wumpus|achow101: intermittently, yes
 81 2017-11-30 17:54:01	0|wumpus|achow101: it seems incredibly unreliable
 82 2017-11-30 18:40:59	0|jonasschnelli|I wonder how the Twitter commit feed works... I expect them polling
 83 2017-11-30 18:42:33	0|wumpus|I think it must be; it isn't a webhook that we set up
 84 2017-11-30 18:43:09	0|wumpus|we only have a slack webhook, and the IRC integration
 85 2017-11-30 18:52:22	0|wumpus|so having ruled out that it listens on IRC, it must either be listening on slack or polling
 86 2017-11-30 18:52:39	0|wumpus|not sure whether the slack notiifications do go through
 87 2017-11-30 18:54:50	0|wumpus|if the normal webhook works we could set up our own IRC bot
 88 2017-11-30 18:56:56	0|jonasschnelli|wumpus: slack works...
 89 2017-11-30 18:57:33	0|wumpus|jonasschnelli: great, so we isolated it to a problem with their IRC bot, not their notifications in general
 90 2017-11-30 18:57:44	0|instagibbs|meeting in 3?
 91 2017-11-30 18:57:46	0|instagibbs|or am i off
 92 2017-11-30 18:57:46	0|jonasschnelli|Looks like
 93 2017-11-30 18:58:26	0|wumpus|you're right
 94 2017-11-30 18:59:24	0|wumpus|if the bot stays flaky like this tomorrow I'll contact github support, it's already been for a few days
 95 2017-11-30 19:00:04	0|lightningbot|Meeting started Thu Nov 30 19:00:03 2017 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
 96 2017-11-30 19:00:04	0|lightningbot|Useful Commands: #action #agreed #help #info #idea #link #topic.
 97 2017-11-30 19:00:04	0|wumpus|#startmeeting
 98 2017-11-30 19:00:23	0|wumpus|#bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag
 99 2017-11-30 19:00:32	0|cfields|hi
100 2017-11-30 19:00:35	0|sipa|hi
101 2017-11-30 19:00:36	0|CodeShark|hello
102 2017-11-30 19:00:44	0|achow101|hi
103 2017-11-30 19:00:47	0|Provoostenator|hi
104 2017-11-30 19:00:51	0|instagibbs|hi
105 2017-11-30 19:00:55	0|jonasschnelli|hi
106 2017-11-30 19:00:59	0|wumpus|#topic high priority for review
107 2017-11-30 19:01:01	0|meshcollider|hi
108 2017-11-30 19:01:17	0|luke-jr|hi
109 2017-11-30 19:01:21	0|wumpus|8 PRs have been tagged for https://github.com/bitcoin/bitcoin/projects/8
110 2017-11-30 19:01:39	0|BlueMatt|why does cfields have 2?
111 2017-11-30 19:01:45	0|wumpus|I added cfields's BanMan last week as it's blocking his further net refactoring
112 2017-11-30 19:01:59	0|cfields|BlueMatt: wasn't me, but i'll take it :)
113 2017-11-30 19:02:07	0|wumpus|ohh is it unfair?
114 2017-11-30 19:02:18	0|instagibbs|achow101, i see you rebased yours? we shooting for post-0.16?
115 2017-11-30 19:02:21	0|BlueMatt|i think we try to have only one per person
116 2017-11-30 19:02:25	0|jtimon|hi
117 2017-11-30 19:02:34	0|BlueMatt|well i suggested we include jonasschnelli's other one as its kinda a segwit-wallet-blocker imo
118 2017-11-30 19:02:38	0|BlueMatt|or at least in-same-release
119 2017-11-30 19:02:43	0|achow101|instagibbs: I'm shooting for 0.16
120 2017-11-30 19:02:47	0|achow101|but it may not make it
121 2017-11-30 19:03:04	0|instagibbs|ok, I can give it more love, though I think I've given a lot already, probably needs others....
122 2017-11-30 19:03:22	0|achow101|still some segwit related things to do and runn1ng wants simulations
123 2017-11-30 19:03:31	0|achow101|also been busy with exams and classwork
124 2017-11-30 19:03:38	0|wumpus|well, good solution for that, review and merge one of cfields's PRs and there will be only one left :)
125 2017-11-30 19:03:44	0|karelb|achow101: I came just in time :) yes I wanted some simulations
126 2017-11-30 19:03:51	0|jonasschnelli|wumpus: heh
127 2017-11-30 19:03:57	0|sipa|so i think we're maybe unintentionally moving from a "we release regularly with whatever is finished" to "X is a blocker for release Y"
128 2017-11-30 19:04:11	0|BlueMatt|oh, wait, i dont have one...hmmmm, I'd say #10279
129 2017-11-30 19:04:13	0|wumpus|sipa: segwit is the only exception to that
130 2017-11-30 19:04:15	0|cfields|i have no problem with removing one of mine if it's an issue. But yes, I think 11363 is ready/easy to review
131 2017-11-30 19:04:15	0|gribble|https://github.com/bitcoin/bitcoin/issues/10279 | Add a CChainState class to validation.cpp to take another step towards clarifying internal interfaces by TheBlueMatt · Pull Request #10279 · bitcoin/bitcoin · GitHub
132 2017-11-30 19:04:24	0|BlueMatt|sipa: I think we'll go back to that post-segwit-wallet, no?
133 2017-11-30 19:04:25	0|wumpus|sipa: for the rest, releases are only ever blocked on bugfixes not features
134 2017-11-30 19:04:29	0|achow101|sipa: I think 0.16 is the exception? because segwit wallet
135 2017-11-30 19:04:35	0|wumpus|yes
136 2017-11-30 19:04:37	0|jtimon|sipa: what is the blocker for 0.16 ?
137 2017-11-30 19:04:41	0|sipa|okay, that's fine - i'm not saying it's a bad thing
138 2017-11-30 19:04:50	0|instagibbs|i think it's openly acknowledged
139 2017-11-30 19:04:55	0|wumpus|and segwit wallet is not intended as a *blocker* for 0.16
140 2017-11-30 19:05:08	0|wumpus|we intend to release 0.16 early after that's in
141 2017-11-30 19:05:14	0|sipa|fair enough
142 2017-11-30 19:05:14	0|wumpus|so it's more like a trigger
143 2017-11-30 19:05:20	0|sipa|but it still changes prioritization a bit
144 2017-11-30 19:05:28	0|BlueMatt|indeed
145 2017-11-30 19:05:33	0|instagibbs|i think the segwit pr is shaping up nicely at least
146 2017-11-30 19:05:35	0|wumpus|if segwit wallet takes longer than the 0.16 release window, then IMO we should release 0.16 without it
147 2017-11-30 19:05:44	0|wumpus|but I don't expect that
148 2017-11-30 19:05:48	0|BlueMatt|wumpus: agreed
149 2017-11-30 19:06:10	0|jtimon|oh, I see, we want to release 0.16 soon after segwit wallet gets in
150 2017-11-30 19:06:15	0|wumpus|jtimon: yep
151 2017-11-30 19:06:26	0|sipa|i'm about to push some review fixes to #11403
152 2017-11-30 19:06:30	0|gribble|https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
153 2017-11-30 19:06:31	0|wumpus|jtimon: seems more advisable than trying to backport the whole lot + dependencies to 0.15
154 2017-11-30 19:06:34	0|cfields|sipa: not sure if you've clarified somewhere, do you consider all of the listed TODOs in 11403 blockers for release as well?
155 2017-11-30 19:06:54	0|jtimon|wumpus: yeah, I think I suggested that, eithe way it makes sense to me
156 2017-11-30 19:07:19	0|wumpus|jtimon: thanks in that case :)
157 2017-11-30 19:07:26	0|luke-jr|branch 0.15 into 0.16 ;)
158 2017-11-30 19:07:32	0|sipa|cfields: yes
159 2017-11-30 19:07:49	0|cfields|ok, thanks
160 2017-11-30 19:08:04	0|gmaxwell|So, segwit-wallet is done?
161 2017-11-30 19:08:15	0|jtimon|I'm not following review on the segwit wallet pr, how is it going?
162 2017-11-30 19:08:31	0|sipa|i think #11403 is pretty much done - just nits in command line option handling and output
163 2017-11-30 19:08:34	0|gribble|https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
164 2017-11-30 19:08:45	0|jtimon|great
165 2017-11-30 19:08:56	0|achow101|I tried reviewing it but it's big and scary
166 2017-11-30 19:09:10	0|wumpus|awesome
167 2017-11-30 19:09:23	0|sipa|we may want to discuss what to do with things like how dumpprivkey and signmessage should work in a post-segwit world
168 2017-11-30 19:09:30	0|cfields|i found it reasonable review on a per-commit basis
169 2017-11-30 19:09:37	0|instagibbs|cfields, same
170 2017-11-30 19:09:40	0|sipa|as i believe some projects have come up with formats on their own
171 2017-11-30 19:09:42	0|cfields|*to review
172 2017-11-30 19:09:56	0|achow101|sipa: trezor has implemented their own segwit message signing thing
173 2017-11-30 19:09:59	0|jtimon|I guess I need tofix the nits in #8994 and #10757 if I want them to have a chance to get merged before 0.16 then
174 2017-11-30 19:10:03	0|achow101|I think that might be the only one though
175 2017-11-30 19:10:03	0|gribble|https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
176 2017-11-30 19:10:07	0|gribble|https://github.com/bitcoin/bitcoin/issues/10757 | RPC: Introduce getblockstats to plot things by jtimon · Pull Request #10757 · bitcoin/bitcoin · GitHub
177 2017-11-30 19:10:22	0|sipa|achow101: perhaps we can just adopt that
178 2017-11-30 19:10:25	0|jtimon|what are we referring to by "soon after"?
179 2017-11-30 19:10:30	0|wumpus|sipa: well if other project's import/export formats make sense, it'd be good to use them for interchangeability
180 2017-11-30 19:10:45	0|achow101|sipa: I don't think it's documented anywhere though
181 2017-11-30 19:10:48	0|meshcollider|Should there be a BIP to formalise it
182 2017-11-30 19:10:49	0|achow101|at least not yet
183 2017-11-30 19:10:52	0|Provoostenator|Is there a good standard for message signing?
184 2017-11-30 19:10:53	0|sipa|longer term i think a script-based signing would nice, but i don't think we're ready to adopt something like that
185 2017-11-30 19:10:59	0|gmaxwell|I thought they did but backed it out because it wasn't standardized yet. What they did didn't seem to awful, though I feel a little uneasy about the mutability between keytypes.
186 2017-11-30 19:11:01	0|karelb|sipa: ad sign message - in trezor we changed v constant - https://github.com/bitcoin/bitcoin/issues/10542#issuecomment-316032523 - no BIP, no time :(
187 2017-11-30 19:11:14	0|sipa|karelb: ok, i'll have a look
188 2017-11-30 19:11:22	0|sipa|the original bitcoin core message signing doesn't have a bip either
189 2017-11-30 19:11:33	0|luke-jr|Provoostenator: no
190 2017-11-30 19:11:40	0|wumpus|a BIP is not a requirement for us implementing it, though in parallel it'd be nice
191 2017-11-30 19:11:43	0|BlueMatt|sipa: it probably should if we change it, though
192 2017-11-30 19:11:45	0|achow101|sipa: perhaps it should? retcon one :p
193 2017-11-30 19:11:56	0|sipa|seems reasonable
194 2017-11-30 19:11:57	0|wumpus|good to already have implementations and the BIP just formalizes it
195 2017-11-30 19:12:08	0|gmaxwell|Well, signmessage is generally not very good; but fixing it should be a longer term thing.
196 2017-11-30 19:12:09	0|Provoostenator|I vaguely remember trying to implement / refactor signing and being not amused by the lack of a standard.
197 2017-11-30 19:12:16	0|luke-jr|current signed messages get very little real use, and most "usage" is based on misconceptions
198 2017-11-30 19:12:23	0|karelb|sipa yes good point. I wanted to write up a document that describes both current and segwit message signing, but I got stuck on how ecdsa signing actually works :)
199 2017-11-30 19:12:24	0|wumpus|luke-jr: agree
200 2017-11-30 19:12:26	0|luke-jr|IMO it'd make sense to just deprecate it until a better replacement is made
201 2017-11-30 19:12:29	0|wumpus|luke-jr: it's not a priority at least
202 2017-11-30 19:12:33	0|instagibbs|luke-jr, it's used for airdrops :P
203 2017-11-30 19:12:40	0|sipa|luke-jr: i agree it's not all that useful
204 2017-11-30 19:12:44	0|Provoostenator|Can it be made coin-agnostic as well?
205 2017-11-30 19:12:48	0|wumpus|certainly shouldn't be a blocker for 0.16
206 2017-11-30 19:12:50	0|luke-jr|instagibbs: that's a misuse, since it doesn't prove you have funds ;)
207 2017-11-30 19:12:51	0|gmaxwell|Provoostenator: no.
208 2017-11-30 19:13:09	0|karelb|we didn't have segwit message signing in web wallet, but users repeatedly demanded it
209 2017-11-30 19:13:12	0|instagibbs|luke-jr, but i profit from it, how can it be misuse ;P
210 2017-11-30 19:13:20	0|instagibbs|+1 tho
211 2017-11-30 19:13:21	0|gmaxwell|The fact that it's inherently incompatible with multisig is a real bummer.
212 2017-11-30 19:13:22	0|luke-jr|karelb: for what, though?
213 2017-11-30 19:13:28	0|achow101|instagibbs: lol
214 2017-11-30 19:13:32	0|gmaxwell|luke-jr: airdrops
215 2017-11-30 19:13:36	0|luke-jr|XD
216 2017-11-30 19:13:50	0|wumpus|using bitcoin keys for anything else than signing transactions continues to make me uneasy
217 2017-11-30 19:13:51	0|jtimon|yeah, +1 to move to signing messages based on scripts
218 2017-11-30 19:14:09	0|luke-jr|MAST-based signmessage would be a sensible replacement
219 2017-11-30 19:14:14	0|Provoostenator|jtimon: what would that look like?
220 2017-11-30 19:14:44	0|wumpus|do hardware wallets even support it?
221 2017-11-30 19:14:51	0|luke-jr|Provoostenator: you could have a MAST if-branch that is always false for transactions, and then true for meta uses
222 2017-11-30 19:14:53	0|CodeShark|gmaxwell: a general solution for airdrops seems extremely difficult
223 2017-11-30 19:14:53	0|gmaxwell|wumpus: If I were to it again, I'd make signmessage work by just creating a transaction which is inherently unminable. It would make it a lot more compatible, though somewhat larger.
224 2017-11-30 19:15:30	0|jtimon|Provoostenator: like we sign txs but signing any message, in elements we sign blocks so perhaps you want to take a look (since there's generic functions to sign stuff, but they're not exposed and only used for blocks)
225 2017-11-30 19:15:31	0|gmaxwell|In elements sipa wrote a patch for a script based signmessage; but we ran into ... challenges.. with how softforks would be handled.
226 2017-11-30 19:15:50	0|wumpus|gmaxwell: that'd have been better; at least the keys would only be used to sign things in one format
227 2017-11-30 19:15:56	0|gmaxwell|Segwit script versioning would fix those challenges.
228 2017-11-30 19:15:58	0|CodeShark|yes, two chains might have very different redemption conditions for the same utxo
229 2017-11-30 19:16:02	0|Provoostenator|Hah, so you can do multisig messages? :-)
230 2017-11-30 19:16:07	0|sipa|Provoostenator: yes
231 2017-11-30 19:16:15	0|sipa|Provoostenator: and timelocked signatures :p
232 2017-11-30 19:16:16	0|Provoostenator|Or even partial messages?
233 2017-11-30 19:16:20	0|jtimon|gmaxwell: I think they should be handled with flags
234 2017-11-30 19:16:44	0|sipa|jtimon: yes, but the general property of softforks doesn't apply
235 2017-11-30 19:16:45	0|jtimon|oh, right, script versioning solves it too
236 2017-11-30 19:16:56	0|sipa|you don't want a "oh, i don't know this signature version! it's valid!!"
237 2017-11-30 19:17:01	0|gmaxwell|jtimon: the 'pubkey' needs to commit to the rules. pre-segwit style softforks don't.
238 2017-11-30 19:17:16	0|gmaxwell|Well you can tristate:  Valid, invalid, unknown public key version.
239 2017-11-30 19:17:48	0|wumpus|right
240 2017-11-30 19:17:50	0|luke-jr|sipa: that's a risk for txs too, though
241 2017-11-30 19:17:55	0|jtimon|sipa: sure, I mean, this is out of the blockchain, so you just need to know which flags to use when validating...what gmaxwell said, commit to the rules
242 2017-11-30 19:18:06	0|gmaxwell|prior to having the explicit versions we couldn't do that, which is why we never even merged the patch in elements, much less brought it to bitcoin.
243 2017-11-30 19:18:07	0|luke-jr|why sighash flags can't fail valid
244 2017-11-30 19:18:24	0|gmaxwell|luke-jr: attacker controls signature side flags.
245 2017-11-30 19:18:33	0|luke-jr|gmaxwell: right, that's my point
246 2017-11-30 19:19:15	0|gmaxwell|same reason sighash flags can fail valid in bitcoin: If I pick an out of range sighash flag I could forge signatures for your pubkeys.
247 2017-11-30 19:19:38	0|sipa|?
248 2017-11-30 19:20:24	0|CodeShark|I also don't follow
249 2017-11-30 19:20:49	0|luke-jr|[19:16:56] <sipa> you don't want a "oh, i don't know this signature version! it's valid‼" <-- this is a problem for transactions just as much as for signed messages
250 2017-11-30 19:20:50	0|wumpus|are there any topics we really want to discuss in this meeting? I think we're drifting off a bit
251 2017-11-30 19:21:00	0|gmaxwell|You cannot trigger "unknown behavior, I'll let it pass" based on sighash flags like you can based on the content of public keys.
252 2017-11-30 19:21:07	0|cfields|next (quick) topic suggestion: codesigning keys update
253 2017-11-30 19:21:07	0|jnewbery|Other PRs I'd love to see merged for v0.16 are #10583 #10579 #11415 . They're all removing wallet dependencies from server, but are API changes so have a one release deprecation lag time before the dependency can actually be removed.
254 2017-11-30 19:21:10	0|gribble|https://github.com/bitcoin/bitcoin/issues/10583 | [RPC] Split part of validateaddress into getaddressinfo by achow101 · Pull Request #10583 · bitcoin/bitcoin · GitHub
255 2017-11-30 19:21:14	0|gribble|https://github.com/bitcoin/bitcoin/issues/10579 | [RPC] Split signrawtransaction into wallet and non-wallet RPC command by achow101 · Pull Request #10579 · bitcoin/bitcoin · GitHub
256 2017-11-30 19:21:17	0|gribble|https://github.com/bitcoin/bitcoin/issues/11415 | [RPC] Disallow using addresses in createmultisig by achow101 · Pull Request #11415 · bitcoin/bitcoin · GitHub
257 2017-11-30 19:21:19	0|jtimon|yeah, how about what "shortly after" means for 0.16 ?
258 2017-11-30 19:21:25	0|jonasschnelli|Two topic proposals: 1) review "nits" reduction, 2) bitcoin core code signing assoc.
259 2017-11-30 19:21:39	0|jonasschnelli|2) goes in hand with cfields topicp.
260 2017-11-30 19:22:07	0|wumpus|jnewbery: thanks, I think adding all of them to high priority is a bit much, but we could add one
261 2017-11-30 19:22:09	0|wumpus|#topic codesigning
262 2017-11-30 19:22:18	0|cfields|I just wanted to point out that after researching for a bit, there's no painless way to renew our osx cert (without involving the btcf), so I think it's prudent that we explore other options.
263 2017-11-30 19:22:29	0|jonasschnelli|https://github.com/bitcoincore-codesigning-association
264 2017-11-30 19:22:44	0|achow101|wumpus: it seems like some of those are ready to be merged. they have utACKs and no comments left
265 2017-11-30 19:22:45	0|jonasschnelli|https://bitcoincorecodesigning.org
266 2017-11-30 19:22:46	0|jonasschnelli|The association stands and apple has accepted the entity
267 2017-11-30 19:22:57	0|jonasschnelli|Code sining certificates are ready (need to setup the key)
268 2017-11-30 19:23:12	0|cfields|jonasschnelli: oh that's fantastic! nice work :)
269 2017-11-30 19:23:15	0|wumpus|achow101: if they're ready for merge even better, please notify me of those things outside the meeting
270 2017-11-30 19:23:16	0|jnewbery|wumpus: thanks. I think they're all pretty much ready for merge. achow101 has been doing a great job maintaining and rebasing them - perhaps just a bit more ACKing and they can go in
271 2017-11-30 19:23:47	0|jonasschnelli|We need a windows code signing cert... have never done that but I'm ready to order if someone gives instructions where and how
272 2017-11-30 19:24:01	0|wumpus|PSA: if something is ready for merge just let me know here instead of waiting for me to stumble on it accidentally :)
273 2017-11-30 19:24:17	0|BlueMatt|jnewbery: afaict only maybe the last is ready? the second only has one ack?
274 2017-11-30 19:24:23	0|achow101|wumpus: ok!
275 2017-11-30 19:24:25	0|BlueMatt|jonasschnelli: that was fast!
276 2017-11-30 19:24:36	0|wumpus|jonasschnelli: oh great!
277 2017-11-30 19:24:37	0|achow101|jonasschnelli: https://docs.microsoft.com/en-us/windows-hardware/drivers/dashboard/get-a-code-signing-certificate
278 2017-11-30 19:25:15	0|cfields|achow101: note that a driver cert is different. More requirements.
279 2017-11-30 19:25:28	0|wumpus|getting a driver cert is much more difficult
280 2017-11-30 19:25:43	0|achow101|oh, didn't see that that was for drivers
281 2017-11-30 19:25:48	0|wumpus|I guess it should be, with the privilege level it operates at
282 2017-11-30 19:26:02	0|jonasschnelli|The question is if we want to try that distributed RSA key
283 2017-11-30 19:26:09	0|wumpus|we do
284 2017-11-30 19:26:26	0|BlueMatt|gmaxwell: mic?
285 2017-11-30 19:26:49	0|BlueMatt|or who was it who said they were gonna look into hooking it into a reasonable way to use it?
286 2017-11-30 19:26:54	0|cfields|I'll work on the CSR handling as promised. I assume we'll just have to shove the result back into the expected format.
287 2017-11-30 19:27:21	0|gmaxwell|I didn't think we were going to bother to do it for the first one. but this is going faster than expected! :)
288 2017-11-30 19:27:22	0|jonasschnelli|cfields: Yeah. Apple needs that -----BEGIN CERTIFICATE REQUEST-----
289 2017-11-30 19:27:23	0|achow101|BlueMatt: that was gmaxwell. I also wanted to take a look at it. it looked painful
290 2017-11-30 19:27:58	0|gmaxwell|it's not so painful but we need a process for converting raw RSA numbers into csrs and what not, which cfields was going to look into.
291 2017-11-30 19:28:30	0|gmaxwell|The easiest thing to do may be to just do trusted setup: generate the key normally on an offline machine, and we can split it after the the fact.
292 2017-11-30 19:28:49	0|BlueMatt|ugh
293 2017-11-30 19:28:50	0|gmaxwell|the MPC signing is radically simple and faster than MPC key generation.
294 2017-11-30 19:29:02	0|BlueMatt|i thought the mpc generation was implemented?
295 2017-11-30 19:29:08	0|gmaxwell|Though the stuff I linked to does both. (though it's setup for only three parties atm)
296 2017-11-30 19:29:17	0|BlueMatt|i mean thats probably fine?
297 2017-11-30 19:29:22	0|achow101|gmaxwell: can it be done faster than what is already implemented?
298 2017-11-30 19:29:22	0|BlueMatt|3/3, I assume?
299 2017-11-30 19:29:35	0|gmaxwell|achow101: of course, but do we care if it takes hours?
300 2017-11-30 19:30:12	0|wumpus|if it's a one-time thing we don't
301 2017-11-30 19:30:15	0|cfields|jonasschnelli: let's talk after the meeting. It'd be great if we could get a dummy to test with.
302 2017-11-30 19:30:19	0|gmaxwell|BlueMatt: yes 3/3. Generation you'd always do as n/n then potentially reshare to a different threshold.
303 2017-11-30 19:30:25	0|jonasschnelli|cfields: sure
304 2017-11-30 19:30:28	0|BlueMatt|its not like this is holding a zksnark trusted-setup for all btc
305 2017-11-30 19:30:37	0|gmaxwell|How big are the keys they use for these things? 2kbit or 4kbit?
306 2017-11-30 19:31:02	0|Provoostenator|BlueMatt: would be nice to get another blog from P
307 2017-11-30 19:31:03	0|Provoostenator|l
308 2017-11-30 19:31:11	0|sipa|?
309 2017-11-30 19:31:13	0|BlueMatt|?
310 2017-11-30 19:31:19	0|Provoostenator|Peter todd riding down Canada to generate his signing key.
311 2017-11-30 19:31:30	0|jonasschnelli|gmaxwell: 2048 is what my apple RSA keys are
312 2017-11-30 19:31:33	0|Provoostenator|And then bragging that's more secure than Zcash
313 2017-11-30 19:31:37	0|wumpus|indeed, it's just code signing, for one OS, if something is signed that is not validated by the gitian process that's discovered soon enough anyway
314 2017-11-30 19:31:39	0|gmaxwell|Provoostenator: it's totally unrelated.
315 2017-11-30 19:31:58	0|jonasschnelli|Code signing won't give a lot of additional security...
316 2017-11-30 19:32:02	0|jonasschnelli|It's just a UX thing IMO
317 2017-11-30 19:32:07	0|gmaxwell|zcash has a backdoor for the whole system, this is just some code signing crud.
318 2017-11-30 19:32:26	0|wumpus|right
319 2017-11-30 19:32:30	0|Provoostenator|I know.
320 2017-11-30 19:32:44	0|gmaxwell|the users shouldn't care AT ALL about the MPC we use for it, the purpose of the MPC is to protect the developers personally from being implicated in the unfortunate case their own computers get hacked.
321 2017-11-30 19:32:53	0|sipa|does the MPC generation have a trusted setup?
322 2017-11-30 19:33:00	0|gmaxwell|sipa: no.
323 2017-11-30 19:33:07	0|sipa|i assume not - if you're fine with trusted setup, just generate a single key and split it layer
324 2017-11-30 19:33:13	0|sipa|ok, so it is entirely unrelated
325 2017-11-30 19:33:25	0|wumpus|yes it's just to prevent the person doing the code signing from becoming a specific target
326 2017-11-30 19:34:12	0|jonasschnelli|review nit reduction topic?
327 2017-11-30 19:34:26	0|gmaxwell|sipa: it just uses paillier, and lots of roundtrips. it's pretty simple.
328 2017-11-30 19:34:48	0|wumpus|#topic review nit reduction
329 2017-11-30 19:35:10	0|jonasschnelli|I just feel that we spend a lot of time in finding code-style nits, fixing code-style nits and some of the PRs are almost a single wall of nits... It doesn't seem to be efficient... I think..
330 2017-11-30 19:35:31	0|jonasschnelli|We should enforce the rules... ( I know I already brought that up )
331 2017-11-30 19:36:03	0|jonasschnelli|The libcurl project does that... it won't compile (make) if the code style doesn't matches
332 2017-11-30 19:36:10	0|jonasschnelli|It would reduce the review time a lot...
333 2017-11-30 19:36:21	0|jonasschnelli|And we apperently fixing the nits anyways at some point in time
334 2017-11-30 19:36:53	0|jtimon|you mean getting travis to complain about style nits?
335 2017-11-30 19:36:56	0|Provoostenator|Currently we're only running whitespace linter?
336 2017-11-30 19:37:10	0|jonasschnelli|jtimon: travis already does this... (whitespace)
337 2017-11-30 19:37:18	0|BlueMatt|if there were an easy way to apply it to new code only, I'd say thats probably fine
338 2017-11-30 19:37:27	0|jonasschnelli|I just want to get the code styling nits away from github comments
339 2017-11-30 19:37:33	0|BlueMatt|but it was my impression we were not able to find a way to do so
340 2017-11-30 19:37:42	0|jonasschnelli|BlueMatt: only new code. Yes.
341 2017-11-30 19:37:47	0|jtimon|BlueMatt: that was my understanding too
342 2017-11-30 19:38:03	0|wumpus|so have e.g. clang-format check the diff of pull requests?
343 2017-11-30 19:38:12	0|sipa|jonasschnelli: (brainstorming) or we could have a bot that marks a PR ready for review only after all nits are fixed
344 2017-11-30 19:38:15	0|jonasschnelli|wumpus: Yes. Can we enforce that somehow?
345 2017-11-30 19:38:29	0|jonasschnelli|sipa: Would also be something.. yes.
346 2017-11-30 19:38:31	0|sipa|so that people know not to look at something while there is still automated review going on
347 2017-11-30 19:38:32	0|wumpus|jonasschnelli: I don't know...
348 2017-11-30 19:38:34	0|Provoostenator|It would also help to offer some hints for developers how to run linters in their editor. I'll add an entry for OSX Atom once I figure it out myself.
349 2017-11-30 19:38:36	0|jtimon|didn't someone wrote something like that at some point?
350 2017-11-30 19:38:41	0|wumpus|sipa: as long as it doesn't generate noise (posts) that's ok with me
351 2017-11-30 19:38:55	0|wumpus|sipa: so if it works like travis, just adds another cross to the status
352 2017-11-30 19:38:59	0|jonasschnelli|Would something speak against enforcing the clang-format-diff check during make?
353 2017-11-30 19:39:02	0|sipa|wumpus: right,t hat would be nice
354 2017-11-30 19:39:02	0|wumpus|w/ a link to the list of problems
355 2017-11-30 19:39:09	0|meshcollider|BlueMatt: adding new linters to Travis would only affect PRs now since #11699
356 2017-11-30 19:39:10	0|gribble|https://github.com/bitcoin/bitcoin/issues/11699 | [travis-ci] Only run linters on Pull Requests by jnewbery · Pull Request #11699 · bitcoin/bitcoin · GitHub
357 2017-11-30 19:39:15	0|wumpus|but I don't want any bots that generate notifications in my mail
358 2017-11-30 19:39:32	0|sipa|right
359 2017-11-30 19:39:34	0|jonasschnelli|wumpus: Yes, That would disturb even more
360 2017-11-30 19:39:46	0|jonasschnelli|Just an indication if its ready to review...
361 2017-11-30 19:39:59	0|jonasschnelli|And so we can have less of those "brackets, spaces missing blabla"
362 2017-11-30 19:40:38	0|BlueMatt|or was it always very version-dependant?
363 2017-11-30 19:40:47	0|jonasschnelli|It just feels some reviews are more or less a "clang-format diff output"
364 2017-11-30 19:40:53	0|wumpus|I like how golang handles that, they just have one true style
365 2017-11-30 19:41:13	0|wumpus|and their linter can be safely used in e.g. pull request checks, w/ no ambiguity
366 2017-11-30 19:41:15	0|jtimon|jonasschnelli: wouldn't we need to comply with clang format in the whole project before doing the clang check in make? has anyone seen haw many changes trying to apply clang-format to the whole project produces right now?
367 2017-11-30 19:41:30	0|morcos|do we have good developer documentation on how to do the right thing locally (not what the rules are, but how to check them?)
368 2017-11-30 19:41:32	0|sipa|you can apply clang-format on just the diffs
369 2017-11-30 19:41:37	0|meshcollider|jtimon: not if it's only run on the diff
370 2017-11-30 19:41:45	0|jonasschnelli|yes
371 2017-11-30 19:41:47	0|sipa|but clang-format only checks whitespace/newlines
372 2017-11-30 19:41:47	0|wumpus|morcos: good point!
373 2017-11-30 19:41:55	0|sipa|it doesn't check variable names etc
374 2017-11-30 19:41:56	0|jonasschnelli|morcos: we should focus on that
375 2017-11-30 19:41:57	0|gmaxwell|A small amount of code style nits on github are probably good for teamwork.
376 2017-11-30 19:42:04	0|wumpus|sipa: I think that's fine, those are the ones we want out of the way
377 2017-11-30 19:42:05	0|jonasschnelli|sipa: not sure if that is possible
378 2017-11-30 19:42:07	0|jtimon|meshcollider: right, only for pr diffs, yeah, I thought he meant every time you build locally
379 2017-11-30 19:42:11	0|wumpus|sipa: variable names are more of a human thing anyhow
380 2017-11-30 19:42:18	0|wumpus|sipa: so it's fine if humans comment on them
381 2017-11-30 19:42:30	0|morcos|i'd be happy to do more locally, i'm sure my PR's are disasters, but it would be nice if a good workflow was spelled out
382 2017-11-30 19:42:44	0|jonasschnelli|morcos: Indeed
383 2017-11-30 19:42:49	0|meshcollider|Scripted diffs will probably regularly require style change commits too to keep Travis happy
384 2017-11-30 19:42:57	0|jtimon|gmaxwell: there's always style nits that go beyond clang-format
385 2017-11-30 19:43:06	0|wumpus|I mean it could check basic things like is_this_snake_case for variable names in theory, but not whether it's a good name in the first place...
386 2017-11-30 19:43:13	0|sipa|sur
387 2017-11-30 19:43:16	0|jonasschnelli|yes
388 2017-11-30 19:43:36	0|jonasschnelli|It's more about the naming convenction (m_, snake_case, CONSTANTS, etc,)
389 2017-11-30 19:43:36	0|wumpus|and the latter is much more important for maintainablility :-)
390 2017-11-30 19:43:48	0|kanzure|do we have a clang-format file
391 2017-11-30 19:43:49	0|MarcoFalke|meshcollider: I'd prefer if we didn't use clang-format in scripted diffs. Makes it impossible to reproduce ...
392 2017-11-30 19:43:51	0|gmaxwell|jtimon: yes true enough, I guess my point was that they're not all bad.
393 2017-11-30 19:43:53	0|wumpus|kanzure: yes
394 2017-11-30 19:43:54	0|jonasschnelli|kanzure: yes
395 2017-11-30 19:44:16	0|wumpus|contrib/devtools/clang-format-diff.py
396 2017-11-30 19:44:16	0|wumpus|src/.clang-format
397 2017-11-30 19:44:26	0|kanzure|ah thanks.
398 2017-11-30 19:44:39	0|cfields|i assume it'd be possible to setup a git alias that shows the current diff, diff'd against the clang-format result
399 2017-11-30 19:44:45	0|jonasschnelli|MarcoFalke: are you up to write a higher level documentation for the clang-format-diff.py workflow?
400 2017-11-30 19:44:56	0|jtimon|wumpus: right, that's what I remember being written
401 2017-11-30 19:44:57	0|MarcoFalke|jonasschnelli: I did
402 2017-11-30 19:45:06	0|MarcoFalke|git diff | cfd
403 2017-11-30 19:45:19	0|wumpus|yapf?
404 2017-11-30 19:45:24	0|gmaxwell|general problem with clang format is that it isn't stable across versions, or at least hasn't been historically. In general we don't have a highly consistent enviroment htat people contribute from.
405 2017-11-30 19:45:30	0|ryanofsky|fwiw, i am not bothered by nits whatsoever. at the very least they mean somebody is actually looking at my pr. i also use clang-format and clang-format diff all the time
406 2017-11-30 19:45:36	0|wumpus|gmaxwell: right, that has always bothered about it
407 2017-11-30 19:45:37	0|jonasschnelli|gmaxwell: yeah. that
408 2017-11-30 19:45:39	0|MarcoFalke|also what gmaxwell said
409 2017-11-30 19:45:52	0|MarcoFalke|wumpus: The same thing for python
410 2017-11-30 19:45:59	0|jtimon|well, the version can be fixed on travis, though
411 2017-11-30 19:46:02	0|wumpus|MarcoFalke: that's not very shocking
412 2017-11-30 19:46:21	0|jonasschnelli|ryanofsky: nits should still remain... just not stuff that computers can find and are covered by our code styling rules
413 2017-11-30 19:46:23	0|sipa|jtimon: right, but that makes travis effectively part of your workflow, which is unfortunate
414 2017-11-30 19:46:43	0|wumpus|you need to be able to run any checks that travis does locally
415 2017-11-30 19:46:44	0|sipa|jtimon: as in, you won't really know a PR is acceptable before pushing and waiting for travis
416 2017-11-30 19:46:48	0|gmaxwell|We could give developers a VM image or equivilent, but it's asking a lot.
417 2017-11-30 19:46:49	0|jtimon|sipa: right, or the concrete version if I want to run it locally first
418 2017-11-30 19:47:19	0|luke-jr|Travis is slow, too
419 2017-11-30 19:47:31	0|sipa|maybe someone should investigate how stable clang-format/ clang-tidy/ iwyu/ ... are
420 2017-11-30 19:47:34	0|morcos|my issue with the nits on PR's is it leads to sloppier review.  When nits get fixed and potentially rebased.  Prior reviewers can get sloppy about assuming that the final version is well reviewed, especially if its happened repeatedly
421 2017-11-30 19:47:40	0|jonasschnelli|Could we not do a check with clang-format-diff.py during "make" and at least place a bold warning (or even refuse to compile *duck*)
422 2017-11-30 19:48:00	0|achow101|jonasschnelli: it would blow up on existing code
423 2017-11-30 19:48:08	0|sipa|jonasschnelli: the problem is that different clang-format versions say different things
424 2017-11-30 19:48:20	0|jonasschnelli|sipa: significant?
425 2017-11-30 19:48:26	0|sipa|jonasschnelli: irrelevant, i think
426 2017-11-30 19:48:30	0|wumpus|jonasschnelli: 'make lint' would be nice
427 2017-11-30 19:48:45	0|sipa|and making something work on your own system, and then seeing travis complain about the exact same things but requiring something else would be pretty demotivating
428 2017-11-30 19:48:49	0|wumpus|jonasschnelli: that would just run all the checks, for C/C++, for python, for whitespace in docs, etc
429 2017-11-30 19:48:50	0|cfields|wumpus: +1
430 2017-11-30 19:48:57	0|gmaxwell|morcos: that is more a problem with the intermixing of nits and serious review, and also how github handles tracking comments (them magically vanishing when the code changes)
431 2017-11-30 19:49:08	0|jonasschnelli|wumpus: Maybe it should by bypassable, but the novice contributor should get warned if he uses invalid code-styling
432 2017-11-30 19:49:15	0|sipa|gmaxwell: that's no longer true with the 'review' function
433 2017-11-30 19:49:28	0|gmaxwell|Obviously we should just pull clang into our code tree and ship it too. :P
434 2017-11-30 19:49:28	0|sipa|you can see all former review comments, and the code they apply on
435 2017-11-30 19:49:31	0|wumpus|jonasschnelli: well, travis and the person that merge can also check
436 2017-11-30 19:49:37	0|jtimon|sipa: oh, that's what is for...
437 2017-11-30 19:49:38	0|instagibbs|sipa, interesting, more reason to use it
438 2017-11-30 19:49:38	0|wumpus|jonasschnelli: it just should be possible to do it locally too
439 2017-11-30 19:49:49	0|morcos|gmaxwell: yes, i'd argue that once serious review stars, nits should not be squashed, but should be left as a cleanup commit at the end.  (at least of some kinds)
440 2017-11-30 19:49:51	0|sipa|jtimon: yes, you should use it :)
441 2017-11-30 19:49:54	0|wumpus|jonasschnelli: I'm not so much concerned with forcing people to run it but making it easy
442 2017-11-30 19:50:08	0|jonasschnelli|wumpus: yes. Travis code style check is not what we want in the first place (it helps,.. but you want to catch it before)
443 2017-11-30 19:50:34	0|sipa|something i have noticed is that sometimes 'nit' reviews start on PRs which are far from certain they'll be even concept ACKed
444 2017-11-30 19:50:34	0|wumpus|jonasschnelli: well if you don't do it before, travis will stop you and you need to wait longer, simple as that :)
445 2017-11-30 19:50:40	0|instagibbs|morcos, we might need to have guidelines for "ACK lockin"
446 2017-11-30 19:50:44	0|jonasschnelli|Okay... I'll have a look at lint and something simple within the make-process
447 2017-11-30 19:50:47	0|jonasschnelli|sipa: that also
448 2017-11-30 19:50:52	0|cfields|morcos: i have no problem with squashing nits as the pre/post-squash can be diff'd locally. It's rebasing to master that makes re-review a challenge imo.
449 2017-11-30 19:51:05	0|wumpus|sipa: yeah... then it adds even more nosie
450 2017-11-30 19:51:07	0|instagibbs|cfields, can be diffed if you locally checked
451 2017-11-30 19:51:20	0|sipa|cfields: right, i try to avoid rebasing, but i pretty liberally squash nits
452 2017-11-30 19:51:22	0|instagibbs|many times im just reading off of github(lazy)
453 2017-11-30 19:51:36	0|sipa|cfields: and i don't mind others doing that too, except for very complicated PRs
454 2017-11-30 19:51:38	0|instagibbs|unless it's a particularly intense series of commits
455 2017-11-30 19:51:42	0|achow101|instagibbs: that kind of breaks when there's lots of merge conflicts
456 2017-11-30 19:51:51	0|wumpus|sometimes I wish 'git fetch' could fetch by commit id to fetch individual commits, unfortunately that doesn't work
457 2017-11-30 19:52:00	0|cfields|pretty sure github can show the diff from a squash a well, you just have to come up with the url for it
458 2017-11-30 19:52:03	0|gmaxwell|well we have contributors who don't feel comfortable (or just don't have the expirence) to do much other than nit reviews. Their contributions are usually pretty helpful and I wouldn't want to ask most of them to stop. We could perhaps ask them to wait a day on new PRs so that there is at least time for Concept NAKs to show up first.
459 2017-11-30 19:52:06	0|wumpus|it can only be used with branch names
460 2017-11-30 19:52:39	0|wumpus|github can give you the patch for an arbitrary commit id though, but it's kind of annoying to do automatically
461 2017-11-30 19:52:55	0|gmaxwell|it is a little sad for someone to show up with a change, and then have two cycles on trivial changes before someone more expirenced comes in and says no-way-because-x.
462 2017-11-30 19:53:03	0|wumpus|gmaxwell: yeah...
463 2017-11-30 19:53:04	0|aj|gmaxwell: might be helpful to tag PRs as looking for concept acks vs looking for nits and style vs looking for tested acks and merging?
464 2017-11-30 19:53:08	0|MarcoFalke|wumpus: We could set up a bot to create branches for each utACK that is posted to gh
465 2017-11-30 19:53:13	0|instagibbs|not nitting a PR that doens't have any ACKs of any kind seems like a decent rule
466 2017-11-30 19:53:16	0|MarcoFalke|on a separate repo that is
467 2017-11-30 19:53:24	0|wumpus|MarcoFalke: that'd be kind of nice
468 2017-11-30 19:53:29	0|jtimon|cfields: at the same time rebasing is often good, for example, #8994 could unexpectedly start failing tests even if no new conflicts appear and github says it's all ready to merge it
469 2017-11-30 19:53:32	0|gribble|https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
470 2017-11-30 19:53:49	0|wumpus|rebasing is often necessary
471 2017-11-30 19:53:52	0|jonasschnelli|Yes. What aj says. The things is just, unenforced user rules tend to get ignored. :)
472 2017-11-30 19:53:54	0|Provoostenator|gmaxwell: as long as you learn something from the nits, it's not the end of the world having them concept-nacked later, imo
473 2017-11-30 19:54:00	0|MarcoFalke|then `git fetch bitcoin_reviewed_commits` will get you all the reviews
474 2017-11-30 19:54:02	0|gmaxwell|sometimes you don't know what you're going to do when you read it.. there are certantly PRs where I read them and then come away with no real opinions on it other than "you missed some braces here and there"
475 2017-11-30 19:54:13	0|wumpus|some files are hotspots and generate a lot of merge conflicts
476 2017-11-30 19:54:29	0|wumpus|although it's better now that main.cpp is dead
477 2017-11-30 19:54:33	0|gmaxwell|Provoostenator: no, but some people find it demotivating; 'they made me do more work then threw it out'. :)
478 2017-11-30 19:54:52	0|Provoostenator|True
479 2017-11-30 19:55:11	0|wumpus|Provoostenator: these are not the kind of nits that help you learn btter programming :)
480 2017-11-30 19:55:31	0|gmaxwell|Provoostenator: that can be resolved with a better perspective; bitcoin development is a distributed process, your effort is your contribution not the fact that it was merged, etc.
481 2017-11-30 19:55:48	0|jonasschnelli|gmaxwell: indeed
482 2017-11-30 19:55:52	0|gmaxwell|Provoostenator: but we can't really send every new contributor to a zen mastery class before they contribute.
483 2017-11-30 19:56:10	0|Provoostenator|Well, we could... but that would be flagged as spam :-)
484 2017-11-30 19:56:11	0|wumpus|Provoostenator: if it's a nit like 'it's better to use build in function X' or 'the loop can be more efficiently written like this' then I think it's great
485 2017-11-30 19:57:17	0|cfields|a slightly different work-flow might be: create a "fixed nits" commit on top of the branch, and squash all nits into that as they arise. Then merge that in without squashing.
486 2017-11-30 19:57:40	0|cfields|it makes for ugly commits getting merged in, but avoids the re-review after squash-for-nits issue.
487 2017-11-30 19:57:41	0|Provoostenator|So far I find most of the nits I received useful. They either taught me to look up coding practices, or motivated me to figure out how to run a linter.
488 2017-11-30 19:57:44	0|gmaxwell|a challenge with that is that we do get new contributions from people with zero git expirence.
489 2017-11-30 19:58:10	0|sipa|who think they need to open a new PR to fix a nit :)
490 2017-11-30 19:58:11	0|wumpus|the git basics stuff is explained pretty well in the contributing doc nowadays
491 2017-11-30 19:58:27	0|aj|do people think the +1, +0, -1, concept nak/ack thing from https://github.com/bitcoin/bitcoin/pull/11426#issuecomment-334091207 is good btw?
492 2017-11-30 19:58:29	0|wumpus|I've not gotten the question on how to squash a commit for a long time now
493 2017-11-30 19:58:41	0|gmaxwell|wumpus: I've noticed that, I wondered why.
494 2017-11-30 19:59:03	0|instagibbs|oh that's where -0 came from
495 2017-11-30 19:59:03	0|Provoostenator|Some projects make a giant squashed merge commit and just point to the original PR (e.g. AngularJS), but that's not ideal for other reasons.
496 2017-11-30 19:59:33	0|BlueMatt|aj: yes, I'm a fan
497 2017-11-30 19:59:38	0|wumpus|Provoostenator: we only want to encourage squashing when it's iterative changes, not atomic separate changes
498 2017-11-30 19:59:40	0|jtimon|aj: I would have preffered that BlueMatt had maintained a nack but answered my questions/rebute, to be honest
499 2017-11-30 19:59:43	0|BlueMatt|specifically caues we end up with lots of need for concept review
500 2017-11-30 20:00:13	0|BlueMatt|we have lots of prs where lots of folks are +0/-0 and dont think its worth review
501 2017-11-30 20:00:16	0|BlueMatt|but they sit open for months
502 2017-11-30 20:00:18	0|BlueMatt|which is bad
503 2017-11-30 20:00:24	0|MarcoFalke|+0 on end meeting
504 2017-11-30 20:00:29	0|BlueMatt|+1
505 2017-11-30 20:00:30	0|instagibbs|-0
506 2017-11-30 20:00:32	0|wumpus|oh, it's that time already
507 2017-11-30 20:00:34	0|lightningbot|Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-11-30-19.00.log.html
508 2017-11-30 20:00:34	0|lightningbot|Meeting ended Thu Nov 30 20:00:33 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
509 2017-11-30 20:00:34	0|lightningbot|Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-11-30-19.00.html
510 2017-11-30 20:00:34	0|lightningbot|Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-11-30-19.00.txt
511 2017-11-30 20:00:34	0|wumpus|#endmeeting
512 2017-11-30 20:01:00	0|instagibbs|so hold on, the numbers are for what? pre-ACK reflection on idea?
513 2017-11-30 20:01:13	0|BlueMatt|generally, ive been usint them as that, yea
514 2017-11-30 20:01:19	0|MarcoFalke|warren: I tested it in august. Should still work, as the vm-builder is compiled from source
515 2017-11-30 20:01:21	0|BlueMatt|see aj's link
516 2017-11-30 20:01:22	0|wumpus|did we just spend an hour talking about nits and code signing, oh - also the signmessage thing :-)
517 2017-11-30 20:01:28	0|BlueMatt|yes :(
518 2017-11-30 20:01:35	0|jtimon|BlueMatt: if they are worth merging I don't think it's bad they sit open for months, even if a couple of  folks said -0
519 2017-11-30 20:01:36	0|instagibbs|wumpus, 0.16 release too
520 2017-11-30 20:01:38	0|BlueMatt|spent an hour talking about how we waste time on nits
521 2017-11-30 20:01:42	0|MarcoFalke|warren: We do that on debian and ubuntu, so I sticked with it
522 2017-11-30 20:01:43	0|wumpus|BlueMatt: lol!
523 2017-11-30 20:01:46	0|wumpus|instagibbs: yes, you're right
524 2017-11-30 20:01:48	0|morcos|didn't you guys have a libevent topic
525 2017-11-30 20:01:51	0|aj|instagibbs: https://www.apache.org/foundation/voting.html#expressing-votes-1-0-1-and-fractions is the apache link
526 2017-11-30 20:01:54	0|instagibbs|BlueMatt, more irc meetings until throughput increases
527 2017-11-30 20:01:57	0|BlueMatt|wumpus: <BlueMatt> oh, wait, i dont have one...hmmmm, I'd say #10279
528 2017-11-30 20:02:00	0|gribble|https://github.com/bitcoin/bitcoin/issues/10279 | Add a CChainState class to validation.cpp to take another step towards clarifying internal interfaces by TheBlueMatt · Pull Request #10279 · bitcoin/bitcoin · GitHub
529 2017-11-30 20:02:11	0|jtimon|so, yeah, how long is "shortly after" for 0.16 ?
530 2017-11-30 20:02:12	0|Provoostenator|Is there a safe way to make Travis cache more stuff? It's insanely slow.
531 2017-11-30 20:02:28	0|BlueMatt|fuck, we forgot to talk about cfields' libevent question :(
532 2017-11-30 20:02:33	0|BlueMatt|that actually mattered
533 2017-11-30 20:02:42	0|instagibbs|feel free to talk on it
534 2017-11-30 20:02:48	0|morcos|well if you, wumpus and cfields are here, you can still talk
535 2017-11-30 20:02:49	0|cfields|it's ok, I'm still investigating something locally
536 2017-11-30 20:02:57	0|gmaxwell|too bad we're not allowed to talk about things except in meetings :(
537 2017-11-30 20:03:08	0|Dizzle|Has libevent merged pass-in-your-own-socket yet?
538 2017-11-30 20:03:10	0|cfields|heh
539 2017-11-30 20:03:32	0|cfields|The specific issue is how to deal with #11785
540 2017-11-30 20:03:33	0|gribble|https://github.com/bitcoin/bitcoin/issues/11785 | Prevent file-descriptor exhaustion from RPC layer by vii · Pull Request #11785 · bitcoin/bitcoin · GitHub
541 2017-11-30 20:03:43	0|wumpus|Dizzle: I don't think so
542 2017-11-30 20:04:08	0|wumpus|Dizzle: at least not for the http client, if you mean that, there's ton of ways to pass your own socket but not there
543 2017-11-30 20:04:29	0|cfields|I'm not convinced that we can fix it entirely, so it remains unclear what to do about it
544 2017-11-30 20:04:49	0|Dizzle|wumpus: thanks, that's what I was wondering about. stratum wallet and stratum mining projects are looking forward to unix socket RPC
545 2017-11-30 20:04:50	0|gmaxwell|well we could always merge some code that tries to increase our FD count, but thats not a fix.
546 2017-11-30 20:05:02	0|instagibbs|aj I still don't see the exact relationship between the numbers and ACKs.
547 2017-11-30 20:05:18	0|wumpus|BlueMatt: you mean #10279 for high priority for review?
548 2017-11-30 20:05:21	0|BlueMatt|yes
549 2017-11-30 20:05:21	0|gribble|https://github.com/bitcoin/bitcoin/issues/10279 | Add a CChainState class to validation.cpp to take another step towards clarifying internal interfaces by TheBlueMatt · Pull Request #10279 · bitcoin/bitcoin · GitHub
550 2017-11-30 20:05:22	0|instagibbs|seems to be concept ACK on sliding scale
551 2017-11-30 20:05:44	0|wumpus|Dizzle: do you need it to be in bitcoin-cli for that?
552 2017-11-30 20:05:52	0|cfields|gmaxwell: the second part of the PR does that, but that makes me really nervous :(
553 2017-11-30 20:06:00	0|aj|instagibbs: +1 -1 = concept ack / concept nak; in between is "leaning this way or not, but not decided either way"
554 2017-11-30 20:06:12	0|wumpus|Dizzle: we could just merge the server side, then you can use it from your own code, just not the command line
555 2017-11-30 20:06:13	0|Dizzle|wumpus: nope, most pools and electrum servers use json-rpc themselves for that.
556 2017-11-30 20:06:21	0|Dizzle|that would be neat!
557 2017-11-30 20:06:25	0|instagibbs|aj ok so it is a replacement for concept acks... that makes more sense
558 2017-11-30 20:06:31	0|BlueMatt|mostly I love -0 - I dont think this is worth anyone's time to review, but if others want to, fine
559 2017-11-30 20:06:55	0|instagibbs|wumpus, speaking of merge ready, 10677
560 2017-11-30 20:06:57	0|Dizzle|BTW, if anyone's curious how Electrum implements message signatures on native segwit addrs, we just iterate through the possible address types until there's match: https://github.com/spesmilo/electrum/blob/66cce115ef93c913d65ef5c7d781c8065f79bbaf/lib/bitcoin.py#L632
561 2017-11-30 20:07:00	0|gmaxwell|cfields: don't we have to eliminate the use of select before we go increasing the default maximum beyond FD_SETSIZE?
562 2017-11-30 20:07:10	0|wumpus|Dizzle: I don't think it's awfully useful in bitcoin-cli anyhow though it's useful for testing (though OTOH, the patch also changed the RPC testing framework to be able to use UNIX sockets)
563 2017-11-30 20:07:26	0|wumpus|instagibbs: ok
564 2017-11-30 20:07:40	0|cfields|gmaxwell: I believe the limit becomes FD_SETSIZE*2 if you're using 2 select loops?
565 2017-11-30 20:08:04	0|wumpus|cfields: no
566 2017-11-30 20:08:10	0|gmaxwell|to my great shame I seem to have forgotten how select is implemented internally. :P
567 2017-11-30 20:08:24	0|wumpus|cfields: select limits the max fd, using multiple selects doesn't change that
568 2017-11-30 20:08:46	0|wumpus|we should probably have switched to poll a long time ago
569 2017-11-30 20:09:24	0|wumpus|IIRC there have been PRs for that, but rejected because libevent P2P was just around the corner...
570 2017-11-30 20:09:27	0|gmaxwell|cfields: we've been waiting forever for the great networking refactors to eliminate the use of select, but perhaps this is a mistake-- select on windows scales fine, and switchin to poll on *nix is a couple line patch IIRC.
571 2017-11-30 20:09:29	0|cfields|well regardless, libevent is using epoll/kqueue/etc. here, so the select limits don't (or shouldn't) apply to rpc
572 2017-11-30 20:09:51	0|wumpus|cfields: no, but the fd's used might be in select's limit!
573 2017-11-30 20:10:06	0|wumpus|cfields: if 1024 fd's are used, then select is dead, no matter who claimed them
574 2017-11-30 20:10:13	0|gmaxwell|cfields: yes, but if we increase the process FD count, we're going to end up with FDs with high numbers which I thought broke select but as mentioned since I seem to have forgotten how it works internally I could be confused.
575 2017-11-30 20:10:25	0|wumpus|gmaxwell: you're right
576 2017-11-30 20:10:51	0|cfields|ok, i was confused about the limit then.
577 2017-11-30 20:10:51	0|wumpus|cfields: you're right that RPC itself won't be affected, but it still messes up P2P nevertheless
578 2017-11-30 20:11:18	0|gmaxwell|cfields: basically as-i-fake-recall the FD number itself is converted into a position in the array, so if you only have 8 FD's monitored but one of them has number 21318 you're going to be in trouble.
579 2017-11-30 20:11:37	0|wumpus|gmaxwell: yup the fd number is converted to a bit position in the array
580 2017-11-30 20:11:53	0|wumpus|which is also what makes it so inefficient
581 2017-11-30 20:12:00	0|wumpus|if it is a sparse array....
582 2017-11-30 20:12:35	0|gmaxwell|I believe bluematt wrote a very small patch to change bitcoin to poll.  We could take, that and wrap it in ifdefs so we still select on windows, and call that sub-issue done.
583 2017-11-30 20:12:56	0|BlueMatt|none of these things solve the problem....
584 2017-11-30 20:13:00	0|BlueMatt|the issue ends up being in eg leveldb
585 2017-11-30 20:13:10	0|gmaxwell|BlueMatt: no but it lets you increase the process limit above 1024.
586 2017-11-30 20:13:11	0|BlueMatt|i mean it'll blow up our p2p first, but mostly thats not a big deal
587 2017-11-30 20:13:14	0|wumpus|does leveldb use select?
588 2017-11-30 20:13:22	0|gmaxwell|since when does leveldb use select?
589 2017-11-30 20:13:25	0|wumpus|I don't hink so
590 2017-11-30 20:13:28	0|BlueMatt|wumpus: no, in this case we literally ran out of process fds
591 2017-11-30 20:13:34	0|wumpus|leveldb only croaks if you run out of all process fds
592 2017-11-30 20:13:50	0|wumpus|which is what vii's patch more or less solves
593 2017-11-30 20:13:53	0|gmaxwell|As I said above: increasing the count isn't a _fix_; ... but it's a pretty good mitigation.
594 2017-11-30 20:14:22	0|wumpus|at least the obvious 'attack RPC with tons of separate connections' doesn't work as an exhaustion attack anymore after that
595 2017-11-30 20:14:24	0|BlueMatt|increasing the count also mostly doesnt break p2p, p2p will just keep opening new sockets until it gets a low numbered one
596 2017-11-30 20:14:28	0|cfields|wumpus: unfortunately, it doesn't :(
597 2017-11-30 20:14:34	0|wumpus|cfields: oh?
598 2017-11-30 20:14:35	0|BlueMatt|its smart enough to handle it
599 2017-11-30 20:14:49	0|gmaxwell|BlueMatt: really? what? lol.
600 2017-11-30 20:15:00	0|cfields|wumpus: nah, i can still exhaust the FDs with little effort.
601 2017-11-30 20:15:03	0|BlueMatt|i mean its not *that* smart, eg it will fail each connection
602 2017-11-30 20:15:09	0|BlueMatt|but it wil lkeep trying to make connections
603 2017-11-30 20:15:16	0|wumpus|cfields: so we really need a fix at the libevent side?
604 2017-11-30 20:15:20	0|gmaxwell|BlueMatt: so if an incoming connection has a high FD number it just drops it?
605 2017-11-30 20:15:24	0|BlueMatt|yes
606 2017-11-30 20:15:46	0|wumpus|it's interesting as I did load tests when I started using libevent and never stumbled on this
607 2017-11-30 20:16:08	0|gmaxwell|thats really ugly, and doesn't let us increase the maximum, consider, we make the maximum 65536... and the most of your connections start getting dropped. :)
608 2017-11-30 20:16:15	0|cfields|wumpus: yes. Problem is that it does while(something){accept()...}
609 2017-11-30 20:16:16	0|wumpus|if I only had known its http server was so crappy :(
610 2017-11-30 20:16:33	0|cfields|so if you manage to make a shitload of connections while it's in that loop, it'll keep swallowing 'em
611 2017-11-30 20:16:34	0|BlueMatt|gmaxwell: i mean if your rpc is getting flooded you probably want to drop most connections :p
612 2017-11-30 20:16:46	0|BlueMatt|cause you're already overloaded
613 2017-11-30 20:16:51	0|wumpus|BlueMatt: yes, you want to drop them
614 2017-11-30 20:17:05	0|wumpus|BlueMatt: that would be the right solution, not hoard file descriptors
615 2017-11-30 20:17:15	0|gmaxwell|BlueMatt: yea sure but I think the FD assignment is next highest in range until it hits the maximum, not lowest available.
616 2017-11-30 20:17:38	0|wumpus|AFAIK fd assignment is lowest available on many OSes
617 2017-11-30 20:17:47	0|BlueMatt|hmm, my node doesnt seem to break and i run with high fd limit
618 2017-11-30 20:17:50	0|gmaxwell|okay, I needed to be wrong about at least one thing.
619 2017-11-30 20:17:52	0|BlueMatt|so....dont think so on linux?
620 2017-11-30 20:17:57	0|gmaxwell|BlueMatt: without the poll patch?
621 2017-11-30 20:18:00	0|BlueMatt|yes, without
622 2017-11-30 20:18:06	0|wumpus|try to close stdout and the next fd you open :-)
623 2017-11-30 20:18:12	0|BlueMatt|i stopped using it and you can still get something like 800 maxonncections
624 2017-11-30 20:18:23	0|gmaxwell|gessh comeone we should just switch to poll, if its *nix only it should be a dozen line patch.
625 2017-11-30 20:18:29	0|gmaxwell|wumpus: lol.
626 2017-11-30 20:19:10	0|wumpus|it's still relevant though
627 2017-11-30 20:19:13	0|gmaxwell|well increasing the fd count would change the urgency.
628 2017-11-30 20:19:38	0|BlueMatt|without poll
629 2017-11-30 20:19:52	0|wumpus|maybe not for this specific issue, but there's no good reason at all we're still using select, it only has drawbacks
630 2017-11-30 20:20:00	0|BlueMatt|fair
631 2017-11-30 20:20:03	0|gmaxwell|I really don't want to troubleshoot mysterious connection failures from "your FD was too high"
632 2017-11-30 20:20:10	0|Dizzle|Does select do ok on macOS? If not, would need to add kqueue polling to those 12 lines of code.
633 2017-11-30 20:20:45	0|gmaxwell|oh right osx poll is broken isn't it?
634 2017-11-30 20:21:03	0|wumpus|how is poll broken on osx?
635 2017-11-30 20:21:29	0|gmaxwell|https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/
636 2017-11-30 20:22:00	0|gmaxwell|sounds like they've fixed it, broken it again, and fixed it.
637 2017-11-30 20:22:08	0|wumpus|lol I'm not surprised, even validating the root password seems to be too difficult for them
638 2017-11-30 20:22:24	0|cfields|iirc they're just flip-flopping on undefined return values?
639 2017-11-30 20:23:09	0|gmaxwell|yea, it seems like that the case discussed there should be easy to avoid.
640 2017-11-30 20:23:16	0|wumpus|they took poor old freebsd and turned it to crap
641 2017-11-30 20:23:55	0|Dizzle|Of note, in addition to the issue that blog mentions, macOS' poll doesn't work on devices.
642 2017-11-30 20:24:06	0|gmaxwell|but shiny plastic crap
643 2017-11-30 20:24:12	0|gmaxwell|we only need it on sockets.
644 2017-11-30 20:24:53	0|wumpus|indeed, that doesn't matter
645 2017-11-30 20:25:20	0|wumpus|windows can't do select on devices or files either,we don't use that
646 2017-11-30 20:26:04	0|gmaxwell|BlueMatt: where is that poll patch?
647 2017-11-30 20:26:27	0|BlueMatt|uhhhh, lost?
648 2017-11-30 20:26:28	0|BlueMatt|i dunno
649 2017-11-30 20:27:29	0|Dizzle|I feel like ifdefing kqueue isn't a big stretch if we're already ifdefing poll. Python project ended up with a selector selector API on top of all this: https://docs.python.org/3/library/selectors.html#selectors.DefaultSelector
650 2017-11-30 20:28:11	0|gmaxwell|Dizzle: no probably not but whats the gain?
651 2017-11-30 20:28:28	0|Dizzle|performance and it works on those few versions of OS X with the broken poll
652 2017-11-30 20:28:46	0|gmaxwell|the performance differences are negligible for our sorts of usage.
653 2017-11-30 20:29:04	0|cfields|Dizzle: there's an abstraction that's done-ish, waiting on pre-requisites to go in. It uses epoll/kqueue/poll/etc. The poll discussion here arose because presumably it'd be simple.
654 2017-11-30 20:29:06	0|wumpus|on the fd assignment, this prints "0" on linux 4.10.0 https://gist.github.com/laanwj/8125d4971728dcfe85f6f5bd09dd572f , so at least anecdotally linux seems to assign the first available fd
655 2017-11-30 20:29:31	0|Dizzle|cfields: this abstraction on libevent or bitcoin core?
656 2017-11-30 20:29:31	0|wumpus|Dizzle: that's why the goal is to move P2P to libevent, it already handles all those polling methods
657 2017-11-30 20:29:38	0|cfields|yea
658 2017-11-30 20:29:46	0|Dizzle|OK, great. Sorry for the noise then :)
659 2017-11-30 20:30:06	0|wumpus|we really don't want to replicate all of that
660 2017-11-30 20:30:08	0|gmaxwell|the general problem with more ifdef paths is less coverage, the majority of the developers are are on linux, as are the majority of serious technical users who will try strange things and actually report results.
661 2017-11-30 20:30:09	0|cfields|#11227
662 2017-11-30 20:30:12	0|gribble|https://github.com/bitcoin/bitcoin/issues/11227 | WIP: switch to libevent for node socket handling by theuni · Pull Request #11227 · bitcoin/bitcoin · GitHub
663 2017-11-30 20:30:56	0|gmaxwell|so we should have a strong general preference to minimize platform ifdefs just on the basis that if we introduce a bug in one of them, it may take a long time to discover and fix.
664 2017-11-30 20:31:00	0|cfields|wumpus: as a data point, even with #11785, i can exhaust all handles via rpc in a minute or two.
665 2017-11-30 20:31:01	0|gribble|https://github.com/bitcoin/bitcoin/issues/11785 | Prevent file-descriptor exhaustion from RPC layer by vii · Pull Request #11785 · bitcoin/bitcoin · GitHub
666 2017-11-30 20:31:02	0|wumpus|at least the low-level libevent stuff is well tested
667 2017-11-30 20:31:19	0|cfields|So I'm not sure that it's worth adding work-around hacks
668 2017-11-30 20:31:19	0|gmaxwell|yes, at least libevent gets testing by other people.
669 2017-11-30 20:31:48	0|wumpus|cfields: so what can we do? (except say "don't do this")
670 2017-11-30 20:31:58	0|wumpus|I mean it's kind of sad to DoS yourself
671 2017-11-30 20:32:14	0|bitcoin-git|[13bitcoin] 15luke-jr opened pull request #11803: Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet (06master...06bugfix_dumpwallet_hdkeypath) 02https://github.com/bitcoin/bitcoin/pull/11803
672 2017-11-30 20:32:17	0|wumpus|and outside of localhost I doubt you'll ever accomplish such a rate
673 2017-11-30 20:32:31	0|gmaxwell|Though I think our general expirence with dependencies (including, sadly, libevent) is that we still run into bugs in them... in part because everything everywhere is broken, and small brokenness in most things is just background noise. ... so the fact that dependency foo is widely used helps less than we might guess.
674 2017-11-30 20:32:45	0|wumpus|that's just a fact of life, everything has bugs
675 2017-11-30 20:33:01	0|cfields|wumpus: yea, i'm not really sure
676 2017-11-30 20:33:18	0|gmaxwell|split rpc into another process. :P
677 2017-11-30 20:33:19	0|wumpus|I mean we even find gcc bugs
678 2017-11-30 20:33:42	0|wumpus|if anything should be well-tested...
679 2017-11-30 20:33:44	0|gmaxwell|wumpus: yea, well I used to comment that I was worried about our testing because we weren't finding GCC bugs. Glad thats fixed now. :)
680 2017-11-30 20:34:01	0|cfields|gmaxwell: yea, i expected some opposition, especially as these bugs crop up. I'd be ok with doing our own abstractions if it came down to it.
681 2017-11-30 20:34:13	0|wumpus|cfields: could we patch it at the libevent side?
682 2017-11-30 20:34:26	0|wumpus|cfields: I think trying to fix it in bitcoind is looking at it the wrong way
683 2017-11-30 20:34:33	0|gmaxwell|I think we should fix libevent.
684 2017-11-30 20:34:35	0|wumpus|cfields: it's a libevent bug
685 2017-11-30 20:34:45	0|cfields|wumpus: yea, pretty easily I believe
686 2017-11-30 20:35:11	0|wumpus|cfields: trying to work around upstream issues (at least permanently) instead of fixing them is pretty not done in open source
687 2017-11-30 20:35:21	0|cfields|through it might kill performance in some applications, so I'm not sure they'd want it upstream
688 2017-11-30 20:35:22	0|gmaxwell|and in the short term apply some mitigations in bitcoin, like increasing the FD count (ugh but I really don't like counting on FD magnitude sniffing)
689 2017-11-30 20:35:40	0|cfields|*though
690 2017-11-30 20:35:46	0|gmaxwell|cfields: surely exausting a processes' FDs isn't a cool move.
691 2017-11-30 20:35:47	0|wumpus|cfields: I don't think they care about performance in the http server
692 2017-11-30 20:36:31	0|cfields|wumpus: well it's in the listener, which is intended to be generic. The http server is just a user
693 2017-11-30 20:36:37	0|cfields|but yes, I'll do up a patch
694 2017-11-30 20:36:38	0|Randolf|[Syntax]:  That's a nice way to encourage people.  I like that.
695 2017-11-30 20:36:42	0|gmaxwell|among other things, it can contribute to amazing security vulns and remote crashes when it makes it impossible to open /dev/urandom to get randomness.
696 2017-11-30 20:36:46	0|Randolf|Whoops, sorry, wrong channel.
697 2017-11-30 20:37:16	0|wumpus|cfields: so the bug is deeper than just the http server, and woudl affect other clients (such as tor) as well?
698 2017-11-30 20:37:20	0|wumpus|cfields: now I'm interested :)
699 2017-11-30 20:37:46	0|cfields|wumpus: I'm unsure who/what else uses the listener
700 2017-11-30 20:37:49	0|BlueMatt|cfields: having a limited queue is a simple patch upstream *should* do
701 2017-11-30 20:38:04	0|cfields|i assume their dns server, as a quick example
702 2017-11-30 20:38:51	0|BlueMatt|so you can crash their dns server with enough tcp traffic?
703 2017-11-30 20:39:15	0|cfields|sec, checking
704 2017-11-30 20:39:32	0|gmaxwell|FWIW speaking of urandom, we'll assert if the open fails, so if you didn't know it this exhaust issue probably also causes (safe) crashes for us.
705 2017-11-30 20:40:08	0|wumpus|gmaxwell: yes luckily we check for that
706 2017-11-30 20:41:06	0|cfields|ok no, they only use the listener internally for http
707 2017-11-30 20:41:25	0|BlueMatt|what else uses libevent's http server?
708 2017-11-30 20:41:26	0|wumpus|cfields: I have a tor checkout here, anything I can easily grep for to see if they use it?
709 2017-11-30 20:41:29	0|BlueMatt|anything should be crashable
710 2017-11-30 20:41:40	0|cfields|wumpus: evconnlistener
711 2017-11-30 20:42:09	0|wumpus|cfields: no matches
712 2017-11-30 20:42:23	0|cfields|whew :)
713 2017-11-30 20:43:23	0|wumpus|they also don't use evhttp
714 2017-11-30 20:45:09	0|wumpus|they do have a http client in src/or/directory.c but apparently they rolled their own there
715 2017-11-30 20:45:31	0|wumpus|eh not that the client would suffer from this
716 2017-11-30 20:49:36	0|wumpus|still, there might be tons of things out there that use it
717 2017-11-30 20:50:09	0|wumpus|so if this can be triggered over the network it might still be reasonably serious
718 2017-11-30 20:52:00	0|wumpus|gah why are code search engines so useless
719 2017-11-30 20:52:50	0|wumpus|find 1000 forks of libevent, of course they have that string
720 2017-11-30 20:53:23	0|cfields|testing a patch
721 2017-11-30 20:54:14	0|wumpus|cfields: cool
722 2017-11-30 22:13:15	0|bitcoin-git|13bitcoin/06master 14680bc2c 15practicalswift: Use range-based for loops (C++11) when looping over map elements...
723 2017-11-30 22:13:15	0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/9e38d357447e...fbce66a98267
724 2017-11-30 22:13:16	0|bitcoin-git|13bitcoin/06master 14fbce66a 15MarcoFalke: Merge #10493: Use range-based for loops (C++11) when looping over map elements...
725 2017-11-30 22:23:11	0|Vision|where are the proxy settings stored for bitcoin-qt?   I cleared the proxy settings in the Options dialog, and now entering the settings dialog causes a crash.  definitely a bug.
726 2017-11-30 22:24:32	0|meshcollider|Vision: what operating system?
727 2017-11-30 22:24:42	0|Vision|meshcollider: windows
728 2017-11-30 22:25:59	0|meshcollider|Vision: in that case, the options will be in the registry, under HKEY_CURRENT_USER\Software\Bitcoin\Bitcoin-Qt iirc
729 2017-11-30 22:26:05	0|wumpus|Vision: what version?
730 2017-11-30 22:26:09	0|wumpus|(of bitcoin core)
731 2017-11-30 22:26:24	0|Vision|15.1 64-bit
732 2017-11-30 22:26:28	0|wumpus|there was a bug in 0.15.0 which caused crashes in the settings dialog, this was fixed in 0.15.0.1 and 0.15.1
733 2017-11-30 22:26:30	0|wumpus|ok
734 2017-11-30 22:26:34	0|meshcollider|Vision: please let us know what values are in the registry before you modify them
735 2017-11-30 22:26:39	0|Vision|will do
736 2017-11-30 22:26:48	0|Vision|cuz yeah this is kinda catastrophic
737 2017-11-30 22:26:55	0|wumpus|it's better to use the flag to clear the gui settings
738 2017-11-30 22:27:01	0|wumpus|it will automatically make a backup
739 2017-11-30 22:27:17	0|meshcollider|wumpus: depends if he wants to clear all settings or just fix the proxy issue manually though :)
740 2017-11-30 22:27:17	0|wumpus|run  bitcoin-qt with -resetguisettings
741 2017-11-30 22:27:51	0|Vision|looks like addrProxy is :10
742 2017-11-30 22:27:51	0|wumpus|this resets the settings and creates a guisettings.ini.bak with the old settings for troubleshooting
743 2017-11-30 22:27:58	0|Vision|lemme change that alone and see if that fixes
744 2017-11-30 22:28:00	0|achow101|start with -resetguisettings and then drop the backup file here
745 2017-11-30 22:28:12	0|wumpus|how did you manage to get :10 in there?
746 2017-11-30 22:28:20	0|meshcollider|just :10 with no IP?
747 2017-11-30 22:28:22	0|Vision|wumpus: clearing the boxes.
748 2017-11-30 22:28:33	0|Vision|I think :10 was the port
749 2017-11-30 22:28:51	0|Vision|maybe I left the port in
750 2017-11-30 22:28:56	0|Vision|and cleared just the IP
751 2017-11-30 22:29:03	0|Vision|meshcollider: indeed
752 2017-11-30 22:29:05	0|achow101|oh, I know how it crashed
753 2017-11-30 22:29:10	0|wumpus|strange, it shouldn't save in that case, but yeah possible
754 2017-11-30 22:29:20	0|Vision|oh it saved alright.
755 2017-11-30 22:29:39	0|achow101|it splits on the colon and does [0] and [1] to get the right params. that'll be an index out of bounds
756 2017-11-30 22:29:54	0|achow101|if there's no IP and just :port
757 2017-11-30 22:30:07	0|Vision|aha :D sounds like a culprit achow101.
758 2017-11-30 22:30:16	0|Vision|yeah I guess I just cleared the IP and not port
759 2017-11-30 22:30:26	0|Vision|and fixing that in the registry DID clear up the cache.
760 2017-11-30 22:30:36	0|Vision|CRASH*
761 2017-11-30 22:30:41	0|Vision|wtf. I swear I'm not retarded.
762 2017-11-30 22:30:43	0|achow101|I don't know how it let that save though
763 2017-11-30 22:31:18	0|Vision|once I finish what I was doing I may try to replicate it
764 2017-11-30 22:31:54	0|Vision|I swear I saw the '10' jump into the IP box before the dialog closed.
765 2017-11-30 22:31:57	0|Vision|if that helps
766 2017-11-30 22:33:54	0|Vision|aaaand I think it crashed instantly at that point.
767 2017-11-30 22:36:05	0|wumpus|it's clearly possible, though it might be that you stumbled on a very unlikely race condition. The parsing code really shouldn't crash on invalid input anyhow.
768 2017-11-30 22:36:54	0|Vision|aye
769 2017-11-30 22:38:04	0|meshcollider|Vision: were you using port 10 before you tried to clear it or is that number completely random
770 2017-11-30 22:38:37	0|Vision|yep I was
771 2017-11-30 22:38:40	0|Vision|not random
772 2017-11-30 22:40:37	0|Vision|oh yeah well that's easy to replicate
773 2017-11-30 22:40:56	0|Vision|have 'connect through socks5 proxy' checked, have an ip and port in the box, clear IP and hit OK
774 2017-11-30 22:40:57	0|Vision|instant crash
775 2017-11-30 22:41:42	0|Vision|and :port gets saved to registry
776 2017-11-30 22:44:06	0|wumpus|the code here https://github.com/bitcoin/bitcoin/blob/master/src/qt/optionsmodel.cpp#L231 for ProxyIP and ProxyPORT needs to catch the condition where the list is empty, or not long enough
777 2017-11-30 22:44:28	0|meshcollider|Yep I have replicated the issue on 0.15.1
778 2017-11-30 22:45:07	0|meshcollider|wumpus: I believe that is the part where it is loaded from the registry, we should deal with this when it is saved I think
779 2017-11-30 22:45:49	0|wumpus|meshcollider: yes but the parsing should be fixed too, it'd be two lines of code or so
780 2017-11-30 22:46:30	0|wumpus|meshcollider: this is the so-manieth time we run into crashes there for one reason or another
781 2017-11-30 22:46:46	0|meshcollider|wumpus: yeah :/
782 2017-11-30 22:47:13	0|meshcollider|why would this validator return true for an empty string: https://github.com/bitcoin/bitcoin/blob/master/src/qt/optionsdialog.cpp#L337
783 2017-11-30 22:47:13	0|Vision|[0] days since last crash
784 2017-11-30 22:50:37	0|Vision|why is 9050 hardcoded in that
785 2017-11-30 22:51:17	0|meshcollider|it is the default port
786 2017-11-30 22:51:23	0|meshcollider|if no port is specified
787 2017-11-30 22:51:34	0|Vision|ah.
788 2017-11-30 22:52:12	0|Vision|just seems an odd place for it
789 2017-11-30 22:52:20	0|wumpus|that's tor's default
790 2017-11-30 22:54:03	0|Vision|(cuz it looks like it's already in src/qt/optionsmodel.cpp as default)
791 2017-11-30 22:54:59	0|wumpus|yeah, feel free to send a patch that makes it a constant instead
792 2017-11-30 23:39:30	0|titim|Hi
793 2017-11-30 23:40:03	0|titim|lapoda lapoda