1 2016-05-12 00:00:28	0|GitHub127|13bitcoin/06master 1432114dd 15Wladimir J. van der Laan: bench: Add crypto hash benchmarks...
  2 2016-05-12 00:00:28	0|GitHub127|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/423ca302a3ee...69b3a6dd9d9a
  3 2016-05-12 00:00:29	0|GitHub127|13bitcoin/06master 1469b3a6d 15Pieter Wuille: Merge #8039: bench: Add crypto hash benchmarks...
  4 2016-05-12 00:00:34	0|phantomcircuit|wumpus, ^
  5 2016-05-12 00:00:41	0|GitHub131|[13bitcoin] 15sipa closed pull request #8039: bench: Add crypto hash benchmarks (06master...062016_05_benchmark_sha256) 02https://github.com/bitcoin/bitcoin/pull/8039
  6 2016-05-12 00:41:29	0|gmaxwell|sipa: the xor-linked-list-ish trick in this is clever: http://www.cs.cmu.edu/~binfan/papers/login_cuckoofilter.pdf
  7 2016-05-12 06:48:38	0|wumpus|jonasschnelli: but for the 1MB buffer the benchmark will get called more times, also flattening out variance
  8 2016-05-12 06:49:03	0|wumpus|I also reasoned like you but forgot to take into account that the benchmarking framework automatically runs for a (approximate) fixed amount  of time
  9 2016-05-12 06:50:22	0|wumpus|<Chris_Stewart_5> Is there a reason we don't have any logging inside of interpreter.cpp (or seemingly anything related to scripts)? <- for various reasons: it is inner-loop code, so logging would slow it down. Also it would be incredbily spammy. And further, any kind of logging complicates libconsensus initiatives by adding external dependencies
 10 2016-05-12 06:51:29	0|wumpus|can anyone please benchmark this on an actual recent Intel processor? https://github.com/laanwj/bitcoin/tree/2016_05_sha256_accel
 11 2016-05-12 06:51:58	0|wumpus|intel's 'fast' sha256 assembly implementations mostly turn out slower instead of faster here
 12 2016-05-12 06:52:00	0|wumpus|but that's on AMD
 13 2016-05-12 06:52:36	0|gmaxwell|doing do
 14 2016-05-12 06:52:38	0|gmaxwell|er so
 15 2016-05-12 06:59:55	0|gmaxwell|wumpus: this is on a haswell based xeon:
 16 2016-05-12 06:59:57	0|gmaxwell|SHA256_avx,319,0.00344849,0.00346613,0.0034525
 17 2016-05-12 06:59:57	0|gmaxwell|SHA256_basic,191,0.00531399,0.00533286,0.00532052
 18 2016-05-12 06:59:57	0|gmaxwell|SHA256_rorx,351,0.00290081,0.00290608,0.002903
 19 2016-05-12 06:59:57	0|gmaxwell|SHA256_rorx_x8ms,351,0.0028524,0.00286806,0.0028543
 20 2016-05-12 07:00:00	0|gmaxwell|SHA256_sse4,319,0.00342679,0.00345755,0.003443
 21 2016-05-12 07:04:57	0|gmaxwell|and a somewhat lower clocked broadwell-ep xeon:
 22 2016-05-12 07:04:59	0|gmaxwell|SHA256_avx,255,0.0039705,0.0040791,0.00397483
 23 2016-05-12 07:04:59	0|gmaxwell|SHA256_basic,175,0.00599706,0.00610304,0.00599851
 24 2016-05-12 07:04:59	0|gmaxwell|SHA256_rorx,319,0.00334549,0.00345182,0.00334802
 25 2016-05-12 07:04:59	0|gmaxwell|SHA256_rorx_x8ms,319,0.00328481,0.00339317,0.00328667
 26 2016-05-12 07:05:01	0|gmaxwell|SHA256_sse4,255,0.00395852,0.00404716,0.00396052
 27 2016-05-12 07:18:10	0|jonasschnelli|What does "OpenBlockFile failed for CBlockDiskPos(nFile=-1, nPos=0)" exactly mean? nFile=-1 looks after a undefined CDiskBlockPos?
 28 2016-05-12 07:18:42	0|jonasschnelli|I'd like to debug the corruption I've got on my AARCH
 29 2016-05-12 07:46:07	0|jonasschnelli|wumpus: my results:
 30 2016-05-12 07:46:09	0|jonasschnelli|SHA1,703,0.00148028,0.00152859,0.0015024
 31 2016-05-12 07:46:09	0|jonasschnelli|SHA256_avx,415,0.00256598,0.00258584,0.0025782
 32 2016-05-12 07:46:10	0|jonasschnelli|SHA256_basic,287,0.00375891,0.00400322,0.00383578
 33 2016-05-12 07:46:10	0|jonasschnelli|SHA256_sse4,415,0.00255489,0.00261028,0.0025789
 34 2016-05-12 07:46:10	0|jonasschnelli|SHA512,415,0.00251514,0.00256588,0.00252959
 35 2016-05-12 07:46:33	0|jonasschnelli|Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
 36 2016-05-12 07:47:47	0|jonasschnelli|bench results of master...
 37 2016-05-12 07:47:48	0|jonasschnelli|SHA1,703,0.00146294,0.00149019,0.00148072
 38 2016-05-12 07:47:48	0|jonasschnelli|SHA256,255,0.00396442,0.00401562,0.00398446
 39 2016-05-12 07:47:48	0|jonasschnelli|SHA512,415,0.00252354,0.0025878,0.00254187
 40 2016-05-12 07:48:32	0|jonasschnelli|Ah. I guess SHA256_basic is the non-tweaked one..
 41 2016-05-12 07:49:04	0|jonasschnelli|So... looks after a 150% performance increase on my xeon server
 42 2016-05-12 07:49:23	0|jonasschnelli|same on gmaxwell's results
 43 2016-05-12 07:51:52	0|gmaxwell|interesting that the only thing the avx seems to do is hurt performance on AMD. :)
 44 2016-05-12 07:52:36	0|gmaxwell|the rorx is somewhat faster though.
 45 2016-05-12 07:54:47	0|jonasschnelli|What about the Intel SHA extentension? I Guess sse4 is a different thing.
 46 2016-05-12 07:54:52	0|jonasschnelli|https://software.intel.com/en-us/articles/intel-sha-extensions
 47 2016-05-12 07:56:15	0|jonasschnelli|I have a Xeon E3 and it seems that intels sha extension is only built into E5 and E7 families.
 48 2016-05-12 07:57:24	0|gmaxwell|jonasschnelli: they don't exist yet.
 49 2016-05-12 07:57:58	0|gmaxwell|(they were announced for broadwell then quietly dropped, I assume because the silicon failed)
 50 2016-05-12 07:58:17	0|jonasschnelli|Also my "AArch64 Processor rev 4 (aarch64)" has Features	: fp asimd aes pmull sha1 sha2 crc32
 51 2016-05-12 07:58:30	0|jonasschnelli|mbedtls has some implementations for this: https://github.com/ARMmbed/mbedtls/pull/432/files
 52 2016-05-12 08:02:39	0|jonasschnelli|After this: https://github.com/CriticalBlue/mbedtls/wiki it could give a x5 performance boost for SHA256 on ARM with that extension
 53 2016-05-12 08:12:54	0|GitHub135|[13bitcoin] 15jonasschnelli opened pull request #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX (06master...062016/05/fix_quit) 02https://github.com/bitcoin/bitcoin/pull/8046
 54 2016-05-12 08:13:24	0|wumpus|thanks for the benchmark results! and yeah re SHA extensions: I'm trying to target hardware that actually exist in the wild :)
 55 2016-05-12 08:13:57	0|wumpus|the AARCH64 sHA256 extension sounds reasonably interesting though, too bad Odroid C2 doesn't have it
 56 2016-05-12 08:16:25	0|jonasschnelli|Yes. Not really an important thing.
 57 2016-05-12 08:16:49	0|jonasschnelli|But enabling AVX should probably be something we should do.
 58 2016-05-12 08:16:52	0|jonasschnelli|wumpus: Is the implementation directly copied from intel?
 59 2016-05-12 08:16:58	0|wumpus|yes
 60 2016-05-12 08:18:05	0|jonasschnelli|1.5* performance boost in sha256 for advance vector enabled processors should be something we should try to get in soon.
 61 2016-05-12 08:18:53	0|jonasschnelli|I guess Sandy and Ivy supports it.
 62 2016-05-12 08:21:33	0|wumpus|what you could try is see if it influences sync performance (a matter of adding CSHA256::SetImplementation(CSHA256::Impl::SSE4); in the inititalization)
 63 2016-05-12 08:22:50	0|gmaxwell|s/AVX/SSE4/
 64 2016-05-12 08:23:19	0|jonasschnelli|SSE4 is ~the same performance boost on my Xeon. But right. We should probably go for AVX
 65 2016-05-12 08:23:26	0|sipa|building
 66 2016-05-12 08:23:54	0|gmaxwell|jonasschnelli: AVX is no faster on anything we've tested on and it hurts performance on Wumpus' AMD chip.
 67 2016-05-12 08:24:10	0|jonasschnelli|gmaxwell: arg. Right. Mixed it up.
 68 2016-05-12 08:24:14	0|gmaxwell|:)
 69 2016-05-12 08:24:24	0|wumpus|yes, AVX seems useless
 70 2016-05-12 08:24:57	0|gmaxwell|BlueMatt: can you benchmark this on skylake?
 71 2016-05-12 08:25:16	0|sipa|SHA256_avx,255,0.00398244,0.00399226,0.00398497
 72 2016-05-12 08:25:16	0|sipa|SHA256_basic,175,0.00608169,0.00611341,0.00608469
 73 2016-05-12 08:25:16	0|sipa|SHA256_sse4,255,0.0039705,0.00398195,0.00397209
 74 2016-05-12 08:25:17	0|jonasschnelli|wumpus: so I just pass in a CSHA256::SetImplementation(CSHA256::Impl::SSE4) in init.cpp somewhere?
 75 2016-05-12 08:25:43	0|wumpus|jonasschnelli: yes, or alternatively change the default in src/crypto/sha256.cpp
 76 2016-05-12 08:25:57	0|wumpus|sipa: what processor?
 77 2016-05-12 08:26:14	0|sipa|Intel(R) Core(TM) i7-4800MQ, 2.6 GHz
 78 2016-05-12 08:26:33	0|wumpus|thanks
 79 2016-05-12 08:27:02	0|gmaxwell|sipa: thats another haswell.
 80 2016-05-12 08:27:41	0|sipa|ok
 81 2016-05-12 08:28:38	0|midnightmagic|I'm sure there's a reason for it, but is cpu detection and then dual codepaths, or even multiple-compile outputs not an option in the event a cpu family does benefit?
 82 2016-05-12 08:29:24	0|midnightmagic|or has a significant benefit to these features just not been found i guess
 83 2016-05-12 08:29:30	0|jonasschnelli|I could create a bench on a "2.6 GHz Intel Core i7" (guess this is Ivy Bridge) if it would be good to know.
 84 2016-05-12 08:29:47	0|wumpus|midnightmagic: runtime CPU detection is necessary to integrate this, yes
 85 2016-05-12 08:30:08	0|wumpus|(right now I don't bother as this is just a hack/experiment)
 86 2016-05-12 08:30:53	0|gmaxwell|midnightmagic: yes, it would be detected. But the test is important, e.g. it looks like we've learned not to use the AVX version even if supported.
 87 2016-05-12 08:31:00	0|wumpus|in any case it's nice that sse4 gives the best speedup; that's the most common
 88 2016-05-12 08:33:32	0|gmaxwell|well rorx was clearly faster in my benchmarks.
 89 2016-05-12 08:34:00	0|wumpus|but you actually have a cpu that supports it :)
 90 2016-05-12 08:34:37	0|wumpus|oh! you did post your results, I missed that
 91 2016-05-12 08:34:56	0|gmaxwell|I even highlit you. :)
 92 2016-05-12 08:35:03	0|midnightmagic|uh. I have a lemote laptop sitting next to me in a backpack. in the event someone might like me to use it for testing, I'd be cool with helping.
 93 2016-05-12 08:35:45	0|midnightmagic|(after I get it rebuilt)
 94 2016-05-12 08:38:12	0|gmaxwell|wumpus: so does sipa and jonasschnelli.
 95 2016-05-12 08:38:40	0|gmaxwell|They just didn't enable it.
 96 2016-05-12 08:45:38	0|gmaxwell|IIRC rorx was introduced with haswell so it's a couple arch generations old now and pretty widely spread. though I can't say if the modest performance increase is really worth it.
 97 2016-05-12 08:50:27	0|wumpus|ok added all of them to https://github.com/laanwj/bitcoin/tree/2016_05_sha256_accel
 98 2016-05-12 08:51:05	0|wumpus|agreed, at the least we should only select one of the rorx variants, they're so similar in performance
 99 2016-05-12 08:51:37	0|gmaxwell|sipa and jonasschnelli should test the rorx. (esp jonasschnelli since his chip is skylake)
100 2016-05-12 08:56:40	0|jonasschnelli|A simple reindex comparison:
101 2016-05-12 08:57:01	0|jonasschnelli|Reindex up to block 200'000:
102 2016-05-12 08:57:09	0|jonasschnelli|SSE4: 389 seconds
103 2016-05-12 08:57:16	0|jonasschnelli|master: 406 seconds
104 2016-05-12 08:58:04	0|jonasschnelli|But I guess the SHA256 performance will be mostly "visible" during a synced state
105 2016-05-12 08:59:03	0|wumpus|at least when larger blocks come into play it may be more indicative, up to 200,000 are mostly easy blocks IIRC
106 2016-05-12 08:59:10	0|wumpus|but it's a noticable speedup that's good
107 2016-05-12 08:59:55	0|jonasschnelli|here my bench with rorx:
108 2016-05-12 08:59:57	0|jonasschnelli|SHA256_avx,351,0.00267656,0.00406031,0.00287071
109 2016-05-12 08:59:57	0|jonasschnelli|SHA256_basic,191,0.00403945,0.00774813,0.00572075
110 2016-05-12 08:59:57	0|jonasschnelli|SHA256_rorx,415,0.00223376,0.00340772,0.00260254
111 2016-05-12 08:59:57	0|jonasschnelli|SHA256_rorx_x8ms,319,0.00224853,0.00372237,0.00329514
112 2016-05-12 08:59:57	0|jonasschnelli|SHA256_sse4,255,0.00273502,0.00505805,0.00409666
113 2016-05-12 09:03:19	0|wumpus|so sse4 139% rorx 173%
114 2016-05-12 09:04:25	0|jonasschnelli|Looks like...
115 2016-05-12 09:06:19	0|jonasschnelli|I try now to compare a full reindex with std sha against rorx.
116 2016-05-12 09:06:42	0|wumpus|is this on the same Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz?
117 2016-05-12 09:06:52	0|jonasschnelli|Yes.
118 2016-05-12 09:07:15	0|wumpus|the numbers are fairly different that's why I ask
119 2016-05-12 09:07:27	0|jonasschnelli|Hmm...
120 2016-05-12 09:08:22	0|jonasschnelli|sse4 is extremely different.. right.
121 2016-05-12 09:08:48	0|jonasschnelli|Hmm.. maybe the reindex ran in the background... let me redo it.
122 2016-05-12 09:08:59	0|wumpus|min and max differences are also huge; possibly the benchmark framework should ignore the first run to 'prime the cache'
123 2016-05-12 09:09:53	0|wumpus|not that precise benchmarks are super-imporant here, it's clear that there is a win to using sse4/rorx
124 2016-05-12 09:10:06	0|jonasschnelli|SHA1,703,0.00148106,0.00254279,0.00150908
125 2016-05-12 09:10:06	0|jonasschnelli|SHA256_avx,415,0.00256918,0.00271057,0.00260393
126 2016-05-12 09:10:06	0|jonasschnelli|SHA256_basic,287,0.00377643,0.00395856,0.00385546
127 2016-05-12 09:10:06	0|jonasschnelli|SHA256_rorx,479,0.00214249,0.00227334,0.00217936
128 2016-05-12 09:10:06	0|jonasschnelli|SHA256_rorx_x8ms,479,0.00212789,0.00233448,0.00219144
129 2016-05-12 09:10:06	0|jonasschnelli|SHA256_sse4,383,0.00263,0.00277644,0.00266623
130 2016-05-12 09:10:07	0|jonasschnelli|SHA512,415,0.00252406,0.00267003,0.0025598
131 2016-05-12 09:10:35	0|wumpus|that looks better
132 2016-05-12 09:10:46	0|jonasschnelli|Yes. The reindex was probably running in the background.
133 2016-05-12 09:11:09	0|jonasschnelli|The latest bench was on a quite system (like the first ones without rorx i have posted)
134 2016-05-12 09:14:03	0|wumpus|these are comparable: https://github.com/laanwj/bitcoin/tree/2016_05_sha256_accel#intelr-xeonr-cpu-e3-1275-v5--360ghz
135 2016-05-12 09:45:54	0|GitHub120|13bitcoin/06master 14fad60b3 15MarcoFalke: [qa] Fix bip9-softforks blockstore issue
136 2016-05-12 09:45:54	0|GitHub120|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/69b3a6dd9d9a...5b736ddaa1c1
137 2016-05-12 09:45:55	0|GitHub120|13bitcoin/06master 145b736dd 15Wladimir J. van der Laan: Merge #8041: [qa] Fix bip9-softforks blockstore issue...
138 2016-05-12 09:46:05	0|GitHub52|[13bitcoin] 15laanwj closed pull request #8041: [qa] Fix bip9-softforks blockstore issue (06master...06Mf1604-qaBip9Blockstore) 02https://github.com/bitcoin/bitcoin/pull/8041
139 2016-05-12 09:46:31	0|GitHub57|13bitcoin/06master 143262316 15Chirag Davé: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t...
140 2016-05-12 09:46:31	0|GitHub57|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/5b736ddaa1c1...2efe38b8323c
141 2016-05-12 09:46:32	0|GitHub57|13bitcoin/06master 142efe38b 15Wladimir J. van der Laan: Merge #8004: signal handling: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t...
142 2016-05-12 09:46:40	0|GitHub29|[13bitcoin] 15laanwj closed pull request #8004: signal handling: fReopenDebugLog and fRequestShutdown should be type sig_atomic_t (06master...06fix_signal_handler) 02https://github.com/bitcoin/bitcoin/pull/8004
143 2016-05-12 09:56:41	0|GitHub124|13bitcoin/06master 148b0e497 15Tyler Hardin: Qt: Add option to hide the system tray icon...
144 2016-05-12 09:56:41	0|GitHub124|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/2efe38b8323c...7c8558da362f
145 2016-05-12 09:56:42	0|GitHub124|13bitcoin/06master 147c8558d 15Wladimir J. van der Laan: Merge #8006: Qt: Add option to disable the system tray icon...
146 2016-05-12 09:56:52	0|GitHub140|[13bitcoin] 15laanwj closed pull request #8006: Qt: Add option to disable the system tray icon (06master...06disable-tray) 02https://github.com/bitcoin/bitcoin/pull/8006
147 2016-05-12 10:13:15	0|jonasschnelli|wumpus:do you have a tray icon menu on you Ubuntu?
148 2016-05-12 10:14:02	0|wumpus|yes, but it's messed up (ends up in the lower left of my screen), a known (upstream) issue with qt5 and ubuntu 14.04, so I'm thankful for the option to disable it
149 2016-05-12 10:16:41	0|wumpus|oh you mean if I can reproduce #8040 on ubuntu? I'll try
150 2016-05-12 10:18:27	0|wumpus|no, I can't: the tray icon only appears after the main window shows
151 2016-05-12 10:19:04	0|wumpus|it's not there during the splash screen
152 2016-05-12 10:19:29	0|wumpus|this would be an issue if we had a "launcher icon menu"
153 2016-05-12 10:19:45	0|wumpus|as the icon in the launcher appears as soon as the application loads
154 2016-05-12 10:20:06	0|jonasschnelli|wumpus: check https://github.com/bitcoin/bitcoin/issues/8043
155 2016-05-12 10:20:10	0|wumpus|then again that requires linking against ubuntu-specific libraries, so little chance
156 2016-05-12 10:20:15	0|jonasschnelli|I have this issue in 14.04 and 16.04
157 2016-05-12 10:20:36	0|MarcoFalke_|fedora core works fine
158 2016-05-12 10:20:49	0|wumpus|hmm, no I never had that problem
159 2016-05-12 10:20:53	0|MarcoFalke_|self-compiled, shows the tray icon after start
160 2016-05-12 10:21:45	0|wumpus|I've had this issue: https://github.com/bitcoin/bitcoin/issues/4826 and this issue: https://github.com/bitcoin/bitcoin/issues/5260
161 2016-05-12 10:21:56	0|wumpus|lots of fun with qt and the tray icon
162 2016-05-12 10:22:18	0|wumpus|the former doesn't seem to happen anymore though, so I think it was fixed upstream
163 2016-05-12 10:22:51	0|wumpus|the latter only happens with self-compiled bitcoin-qt against the system qt5, whichi s ancient
164 2016-05-12 10:23:13	0|wumpus|(5.2.1)
165 2016-05-12 10:23:33	0|wumpus|in any case not worrying. The missing menu is strange
166 2016-05-12 10:24:01	0|wumpus|adding the hide tray icon option is a good start, maybe at some point in the future it should be the default, too much trouble with it
167 2016-05-12 10:31:48	0|jonasschnelli|Yes. That's what I was thinking. Though, the menu itself can be helpful. But the global menu space problem and the issues we always see with the tray menu make me think so.
168 2016-05-12 10:34:08	0|wumpus|the right way to do this on ubuntu would be the 'launcher menu' route. But due to the practical concern with distributing GUI binaries on linux that'd only work, at most, for the ppa as that is custom-built for ubuntu
169 2016-05-12 10:38:49	0|wumpus|in any case I'll try on ubuntu 16.04 too and see if I can reproduce 8043
170 2016-05-12 10:53:07	0|jonasschnelli|wumpus: thanks.
171 2016-05-12 11:14:26	0|GitHub176|[13bitcoin] 15MarcoFalke opened pull request #8047: [qa] test_framework: Set wait-timeout for bitcoind procs (06master...06Mf1605-qaUtilTimeout) 02https://github.com/bitcoin/bitcoin/pull/8047
172 2016-05-12 11:16:06	0|GitHub130|13bitcoin/06master 1434ebceb 15Jonas Schnelli: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX
173 2016-05-12 11:16:06	0|GitHub130|[13bitcoin] 15jonasschnelli pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/7c8558da362f...169d379c9835
174 2016-05-12 11:16:07	0|GitHub130|13bitcoin/06master 14169d379 15Jonas Schnelli: Merge #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX...
175 2016-05-12 11:16:15	0|GitHub15|[13bitcoin] 15jonasschnelli closed pull request #8046: [Qt][OSX] Fix Cmd-Q / Menu Quit shutdown on OSX (06master...062016/05/fix_quit) 02https://github.com/bitcoin/bitcoin/pull/8046
176 2016-05-12 11:52:43	0|GitHub194|[13bitcoin] 15laanwj opened pull request #8048: doc: Remove outdated qt4 install information from README.md (06master...062016_05_doc_noqt4) 02https://github.com/bitcoin/bitcoin/pull/8048
177 2016-05-12 12:27:33	0|GitHub159|[13bitcoin] 15laanwj opened pull request #8049: Expose information on whether transaction relayed is enabled in `getnetwork` (06master...062016_05_rpc_relaytxes) 02https://github.com/bitcoin/bitcoin/pull/8049
178 2016-05-12 12:28:08	0|GitHub116|[13bitcoin] 15laanwj closed pull request #7841: Expose information on whether transaction relayed is enabled in getnetwork (06master...06patch-2) 02https://github.com/bitcoin/bitcoin/pull/7841
179 2016-05-12 13:50:05	0|BlueMatt|gmaxwell: still need it benchmarked?
180 2016-05-12 13:50:33	0|gmaxwell|BlueMatt: we got skylake numbers from jonasschnelli
181 2016-05-12 13:51:51	0|BlueMatt|kk
182 2016-05-12 14:28:54	0|jonasschnelli|wumpus: benchmark results of full reindex up to height 400'000
183 2016-05-12 14:29:08	0|jonasschnelli|master, std sha: 8'431 seconds
184 2016-05-12 14:29:13	0|jonasschnelli|rorx: 7'836
185 2016-05-12 14:37:35	0|instagibbs|jonasschnelli, what is this line doing? https://github.com/bitcoin/bitcoin/pull/8035/files#diff-fcc78df4b7178e5b09f83f38196fef8bR59
186 2016-05-12 14:37:40	0|instagibbs|I suppose I don't get the serialization code well enough
187 2016-05-12 15:15:16	0|jonasschnelli|instagibbs: i think this line is not required. I guess I copied this from another class. Will have a look soon.
188 2016-05-12 15:15:35	0|instagibbs|ok, that's my only ? left. Thanks
189 2016-05-12 15:15:54	0|luke-jr|it's doing something there
190 2016-05-12 15:16:07	0|luke-jr|nVersion is the method param, not the class var
191 2016-05-12 15:16:18	0|luke-jr|I think READWRITE uses it?
192 2016-05-12 15:16:38	0|luke-jr|definitely needs better docs
193 2016-05-12 15:16:44	0|instagibbs|... what
194 2016-05-12 15:16:55	0|luke-jr|+    inline void SerializationOp(Stream& s, Operation ser_action, int nType, int nVersion)
195 2016-05-12 15:16:56	0|instagibbs|oh
196 2016-05-12 15:16:56	0|instagibbs|right
197 2016-05-12 15:17:41	0|luke-jr|I suggest renaming one of the nVersions
198 2016-05-12 15:17:48	0|luke-jr|as-is, this is obfuscated code
199 2016-05-12 15:18:04	0|instagibbs|agreed
200 2016-05-12 15:18:07	0|jonasschnelli|Yes. Will rename it.
201 2016-05-12 15:18:35	0|jonasschnelli|I think I took it 1:1 from CKeyMetadata
202 2016-05-12 18:17:28	0|wumpus|jonasschnelli: nice, that's quite an improvement
203 2016-05-12 18:18:11	0|jonasschnelli|wumpus: Yes. I guess most reindex work in ECDSA and not SHA. So yes. It's an impressive improvement.
204 2016-05-12 18:18:24	0|jonasschnelli|I guess during a synced state the improvements are even more visible.
205 2016-05-12 18:18:40	0|jcorgan|are there still thursday weekly dev meetings here?
206 2016-05-12 18:18:47	0|helo|yes
207 2016-05-12 18:18:47	0|wumpus|yes
208 2016-05-12 18:19:11	0|gmaxwell|not for another 45 minutes.
209 2016-05-12 18:19:17	0|jcorgan|ah, off by an hour
210 2016-05-12 18:41:43	0|droark|Hi. Question about PR #7972. It adds a new file (qa/rpc-tests/create_cache.py) that creates a cache used, I assume, to assist with parallelizing the qa/rpc-tests code. I don't quite understand this. Can somebody help me out? TIA.
211 2016-05-12 18:44:00	0|jonasschnelli|droark: You might need to ask MarcoFalke. But IIRC, you need to create the cache before you run the script in parallel because you run into the risk of different cache versions...
212 2016-05-12 18:44:23	0|sdaftuar|jonasschnelli: droark: right, i assume it's to avoid having two jobs try to do it at the same time
213 2016-05-12 18:45:46	0|droark|Sounds reasonable. What exactly is being cached, though? The results? The runtime environment for each instance? Something else?
214 2016-05-12 18:46:01	0|sdaftuar|droark: it's a cached blockchain
215 2016-05-12 18:46:38	0|sdaftuar|take a look at initialize_chain in qa/rpc_tests/test_framework/util.py
216 2016-05-12 18:47:35	0|droark|Ahhh. Thanks! I'll ping Marco and double check the code but it sounds right.
217 2016-05-12 18:48:20	0|lclc|Is anyone of the devs who are coming to Zurich arriving the day before (Thursday) ?
218 2016-05-12 18:48:40	0|sipa|i'll be there on thursday evening
219 2016-05-12 18:49:49	0|jcorgan|i get there thu evening as well
220 2016-05-12 18:49:49	0|lclc|That doesn't surprise me :D   I'm asking because I'd like to organize a meetup the evening before so the community also has something from the event :)
221 2016-05-12 18:50:03	0|jcorgan|well, 6pm at the airport
222 2016-05-12 18:51:15	0|lclc|Would be cool if someone would give a talk - topic is up to you.   Our meetups are 30-110 people, free to attend, and recorded (www.bitcoinlectures.tv - YT Channel)
223 2016-05-12 19:00:16	0|wumpus|I'm also arriving in zurich thursday evening
224 2016-05-12 19:00:20	0|sdaftuar|meeting time?
225 2016-05-12 19:00:41	0|wumpus|#meetingstart
226 2016-05-12 19:00:42	0|petertodd|yup
227 2016-05-12 19:00:45	0|lightningbot|Meeting started Thu May 12 19:00:44 2016 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
228 2016-05-12 19:00:45	0|lightningbot|Useful Commands: #action #agreed #help #info #idea #link #topic.
229 2016-05-12 19:00:45	0|wumpus|#startmeeting
230 2016-05-12 19:01:13	0|BlueMatt|topics?
231 2016-05-12 19:01:38	0|wumpus|from last week:
232 2016-05-12 19:01:48	0|wumpus|add a file bip-0009/assignments.md in the bips repository:  btcdrak made a pull for that
233 2016-05-12 19:02:05	0|wumpus|discuss testnet activation date on bitcoin-dev mailing list
234 2016-05-12 19:02:15	0|wumpus|ead bluematt's compact block bip
235 2016-05-12 19:02:20	0|wumpus|+r
236 2016-05-12 19:02:59	0|wumpus|#link https://github.com/bitcoin/bips/pull/386
237 2016-05-12 19:03:31	0|luke-jr|on that note, it'd be nice if people didn't ACK/NACK bips they are not a listed author of ;)
238 2016-05-12 19:03:31	0|wumpus|any other topic proposals?
239 2016-05-12 19:04:15	0|sipa|was segwit testnet discussed on the ml?
240 2016-05-12 19:04:20	0|kanzure|luke-jr: please elaborate.
241 2016-05-12 19:04:25	0|BlueMatt|re: compact block, got good feedback from a few, made some slight tweaks but need to finish tweaks in the coming days, so no need to discuss here, I think?
242 2016-05-12 19:05:06	0|luke-jr|kanzure: when it comes to a BIP draft, the only ACK/NACK that matters is the Author(s); others posting ACK/NACK just makes me spend time looking at the Author header to confirm I need to ignore it
243 2016-05-12 19:05:13	0|wumpus|well in the case of #386 I felt I had the right to give my opinion because I was one of the people to come up with the idea last meeting, even though I'm not listed as author
244 2016-05-12 19:05:29	0|sipa|i guess the question is where BIPs are to be discussed
245 2016-05-12 19:05:38	0|luke-jr|sipa: bitcoin-dev ML
246 2016-05-12 19:05:41	0|sipa|and the idea is that that would be the mailinglist
247 2016-05-12 19:05:43	0|sipa|but...
248 2016-05-12 19:05:57	0|kanzure|luke-jr: wouldn't NACKs from non-authors still be useful? why should the authors be the only trusted source of updates ? i don't understand.
249 2016-05-12 19:06:17	0|instagibbs|oh, the compact blocks got a bip#
250 2016-05-12 19:06:30	0|paveljanik|I don't understand either. If the author dies in accident, we can no longer change the BIP? or what?
251 2016-05-12 19:06:31	0|morcos|luke-jr: i'd suggest that you just come up with a way to distinguish.   for instance ACK/NACK shoudl be interpreted by the author and the author shoudl say Approved or Ready or something for your knowledge on when to merge
252 2016-05-12 19:06:31	0|wumpus|I agree that ack/nack isn't very useful, in comparison to more detailed comments
253 2016-05-12 19:06:32	0|kanzure|luke-jr: i'm willing to believe you're right but you need better reasons yo.
254 2016-05-12 19:06:40	0|petertodd|kanzure: if I were author of a BIP an ACK would give me confidence; a NACK could be useful criticism
255 2016-05-12 19:06:42	0|sipa|paveljanik: an active BIP can't be modified anyway
256 2016-05-12 19:06:43	0|kanzure|wumpus: right, sure.
257 2016-05-12 19:06:48	0|luke-jr|kanzure: the BIP is a document by the Author(s). ideally, they should have exclusive merge access for changes, but GitHub doesn't support that
258 2016-05-12 19:06:54	0|paveljanik|text clarification, etc...
259 2016-05-12 19:06:59	0|kanzure|okay this is just trivial though, luke-jr isn't going to stop me from ACKing on BIPs i didn't author :)
260 2016-05-12 19:07:04	0|morcos|sipa: in some cases it should be, if for instance the BIP wording is incorrect ala 34
261 2016-05-12 19:07:18	0|luke-jr|paveljanik: BIP 1 technically allows me to reassign BIP Author in some cases IIRC
262 2016-05-12 19:07:19	0|sipa|morcos: yeah - makes sense
263 2016-05-12 19:07:39	0|jonasschnelli|We should also define a rule how we deal with links to implementations that have implemented the BIP.
264 2016-05-12 19:07:48	0|sipa|jonasschnelli: yes
265 2016-05-12 19:07:53	0|jonasschnelli|IMO we should not add any implementation links.
266 2016-05-12 19:07:58	0|jonasschnelli|(expect of sample impl.)
267 2016-05-12 19:08:00	0|wumpus|many bips have a list of implementations; what's wrong with that?
268 2016-05-12 19:08:08	0|jonasschnelli|Its just noisy
269 2016-05-12 19:08:12	0|sipa|wumpus: in BIP32 there are continuously pull requests to add links
270 2016-05-12 19:08:16	0|gmaxwell|it gets flooded with spammy updates.
271 2016-05-12 19:08:17	0|wumpus|I think it can be useful to have e.g. implementations in many languages
272 2016-05-12 19:08:18	0|jonasschnelli|And we don't control the implementations I guess
273 2016-05-12 19:08:21	0|sipa|wumpus: which imho function as nothing more than advertizement
274 2016-05-12 19:08:26	0|kanzure|and then we have to moderate the additions
275 2016-05-12 19:08:28	0|jonasschnelli|sipa: Yes!
276 2016-05-12 19:08:31	0|gmaxwell|And if we're not looking at it, we're eventually going to get a malware example BIP32 impl. :)
277 2016-05-12 19:08:33	0|wumpus|ok...
278 2016-05-12 19:08:35	0|luke-jr|BIP 2 comments would be a nice place to list implementations, but that was controverisal
279 2016-05-12 19:09:08	0|jonasschnelli|I could imaging more and more advertising like PR to come up in future.
280 2016-05-12 19:09:13	0|wumpus|I think the requirement should at least be to point to the code
281 2016-05-12 19:09:15	0|paveljanik|so what is the topic now? ;-)
282 2016-05-12 19:09:15	0|wumpus|not just the product
283 2016-05-12 19:09:22	0|wumpus|to prevent advertizement
284 2016-05-12 19:09:32	0|kanzure|i think the topic was "feedback about compact block relay BIP things"
285 2016-05-12 19:09:50	0|jonasschnelli|Which still intersects (a little bit)
286 2016-05-12 19:10:14	0|jcorgan|it is useful, however, when reading a BIP, to at least get pointed to reference code, but it need not be every implementation that anyone wants to list thereafter
287 2016-05-12 19:10:20	0|wumpus|I think the topic is how to handle BIP implementations, which is as good a topic as anything
288 2016-05-12 19:10:48	0|wumpus|but then you get the fight about what is reference code and what is not
289 2016-05-12 19:10:59	0|jonasschnelli|Links are always difficult. You need to check them... then what if a link diverges from the BIP specification?
290 2016-05-12 19:11:08	0|jonasschnelli|Do we check it?
291 2016-05-12 19:11:08	0|wumpus|in some cases it's clear if the BIP author wrote code themselves
292 2016-05-12 19:11:13	0|kanzure|yeah another threat vector is someone selling their github account in the future
293 2016-05-12 19:11:17	0|kanzure|and then a BIP links to someone's github
294 2016-05-12 19:11:21	0|jonasschnelli|Yes. These are the link we should keep there...
295 2016-05-12 19:11:33	0|jcorgan|or the BIP author themselves pick one or more...
296 2016-05-12 19:11:39	0|gmaxwell|If it's upfront then we can trust the bip author and review process to have had some chance to verify it.
297 2016-05-12 19:11:42	0|wumpus|yes, I'd say it's up to the BIP author
298 2016-05-12 19:11:46	0|wumpus|like other changes to the BIP
299 2016-05-12 19:11:47	0|kanzure|other problem is keeping track of upstream updates like security fixes, oops. anyway, this is a lot of work.
300 2016-05-12 19:11:58	0|kanzure|we should include hashes of the reference source code, in the BIP text
301 2016-05-12 19:11:58	0|luke-jr|maybe BIP PRs should be PGP signed
302 2016-05-12 19:12:03	0|wumpus|this is not something that should be globally decided
303 2016-05-12 19:12:16	0|gmaxwell|I could see the criteria being different for different BIPs.
304 2016-05-12 19:12:19	0|jonasschnelli|wumpus: Yes. This makes sense.
305 2016-05-12 19:12:20	0|kanzure|by hashing the source code we can at least have a way for readers to verify that the source code is still the part we meant to hyperlink to :)
306 2016-05-12 19:12:22	0|wumpus|gmaxwell: exactly
307 2016-05-12 19:12:33	0|wumpus|I mean the author of bip32 could say 'enough is enough'
308 2016-05-12 19:12:43	0|jcorgan|er, link to a URL and commit hash?
309 2016-05-12 19:12:44	0|wumpus|some other BIPs have far fewer implementations and the author may be happy to see one
310 2016-05-12 19:12:48	0|gmaxwell|(unfortunately, it's BIP32 that gets 95% of this-- key generation is an especially awkward place for random found on the internet code)
311 2016-05-12 19:12:53	0|petertodd|my BIP65 links to two separate demo repos that just give some sample code, which is probably something we can generally consider as acceptable (ignoring the issue of more complex implementations that aren't pure demos)
312 2016-05-12 19:13:09	0|jonasschnelli|I like jcorgan idea to link to sourcecode-only with a commit hash
313 2016-05-12 19:13:16	0|jonasschnelli|But kinda static.
314 2016-05-12 19:13:30	0|wumpus|Itend to agree with that. Link to the actual code, not the product
315 2016-05-12 19:14:12	0|wumpus|not 'blawallet implements bip32', no, *the code linked here shows how we implemented BIP32 in language Z*
316 2016-05-12 19:14:29	0|sipa|wumpus: agree
317 2016-05-12 19:14:30	0|luke-jr|+1
318 2016-05-12 19:14:34	0|petertodd|wumpus: +1
319 2016-05-12 19:14:39	0|paveljanik|yes
320 2016-05-12 19:14:43	0|jonasschnelli|ack
321 2016-05-12 19:14:49	0|wumpus|ok
322 2016-05-12 19:14:54	0|kanzure|we should review existing bips for source code links that do not include a commit hash. branch names are not OK.
323 2016-05-12 19:15:01	0|kanzure|i mean, branch names without a commit hash are not OK.
324 2016-05-12 19:15:02	0|wumpus|other topics?
325 2016-05-12 19:15:38	0|paveljanik|Revies Jonas' HD - #8035
326 2016-05-12 19:15:57	0|jonasschnelli|I have a tiny topic proposal that is very into the impl. teretorry..
327 2016-05-12 19:16:02	0|jonasschnelli|RPC long poll notifications
328 2016-05-12 19:16:07	0|kanzure|have we received an overview from sipa yet about areas of segwit that he feels should be most thoroughly reviewed
329 2016-05-12 19:16:12	0|sipa|kanzure: no, sorry
330 2016-05-12 19:16:23	0|kanzure|can we get 10 volunteers to heckle sipa about this?
331 2016-05-12 19:16:23	0|sipa|thanks for the reminder
332 2016-05-12 19:16:34	0|wumpus|#action Revies Jonas' HD - #8035
333 2016-05-12 19:16:52	0|jonasschnelli|https://github.com/bitcoin/bitcoin/pull/8035 ... very simple and easy to review
334 2016-05-12 19:17:12	0|paveljanik|jonasschnelli, can you add some after-merge TODO list there?
335 2016-05-12 19:17:19	0|jonasschnelli|I'm happy to add some RPC tests...
336 2016-05-12 19:17:36	0|jonasschnelli|There is no required after-merge PR... thats the nice thing.
337 2016-05-12 19:17:49	0|jonasschnelli|But most important we should enable flexible bip32 keypath
338 2016-05-12 19:17:50	0|paveljanik|import, dump... at least...
339 2016-05-12 19:17:52	0|instagibbs|jonasschnelli, a bit hard to rpc test without import/dump? (perhaps offline discussion)
340 2016-05-12 19:18:18	0|jonasschnelli|There are concerns with importing single keys into the bip32 wallet...
341 2016-05-12 19:18:32	0|jonasschnelli|They would not be covered by "a old backup"
342 2016-05-12 19:18:44	0|wumpus|but that's kind of obvious :)
343 2016-05-12 19:18:51	0|btcdrak|maybe need a sweep feature.
344 2016-05-12 19:19:07	0|wumpus|yes, a sweep feature would be nice
345 2016-05-12 19:19:09	0|jonasschnelli|There are plenty of possible features...
346 2016-05-12 19:19:25	0|jonasschnelli|But because of the lack of reviewers,.. we need babysteps
347 2016-05-12 19:19:28	0|sipa|let's not discuss all possible features
348 2016-05-12 19:19:29	0|luke-jr|sweep would be nice, but non-trivial (currently no way to iterate over the UTXO set I think)
349 2016-05-12 19:19:38	0|sipa|just what is necessary to work and test
350 2016-05-12 19:19:39	0|jonasschnelli|agree with sipa.
351 2016-05-12 19:19:46	0|jonasschnelli|Just review and ack it! :P
352 2016-05-12 19:19:48	0|luke-jr|so what about RPC longpoll?
353 2016-05-12 19:19:49	0|wumpus|I agree
354 2016-05-12 19:19:58	0|sipa|if import/export is necessary for testing, then maybe some functionality for that is warranted
355 2016-05-12 19:20:00	0|wumpus|scope creep agian
356 2016-05-12 19:20:23	0|instagibbs|well, it only came up due to testing
357 2016-05-12 19:21:11	0|wumpus|luke-jr: https://github.com/bitcoin/bitcoin/pull/7949
358 2016-05-12 19:21:19	0|jonasschnelli|RPC long poll would would allow remote GUI and remote wallet with a very easy setup. IMO it could allow third party developers to write simple remote GUIS, web-frontends, etc.
359 2016-05-12 19:22:07	0|jonasschnelli|I would even say RPC long poll has more potential then ZMQ for core.
360 2016-05-12 19:22:11	0|wumpus|I'm kind of divided on the notification thing to be honest - I'd prefer to stick to only one notification mechanism in bitcoin core (apart from the silly -Xnotify), and somehow zmq got a precedent there
361 2016-05-12 19:22:42	0|luke-jr|calling pollnotifications myself seems like it would disrupt an application relying on it?
362 2016-05-12 19:22:46	0|jonasschnelli|I also don't like multiple notification channels.
363 2016-05-12 19:22:49	0|luke-jr|ie, we need channels
364 2016-05-12 19:22:57	0|jonasschnelli|But how would you connect a remote GUI over the internet...
365 2016-05-12 19:23:04	0|luke-jr|wumpus: zmq is crazy though :<
366 2016-05-12 19:23:16	0|luke-jr|also note we already have longpolling
367 2016-05-12 19:23:18	0|wumpus|yes it should definitely have multiple channels, the current code supports only one client
368 2016-05-12 19:23:28	0|jonasschnelli|wumpus: no
369 2016-05-12 19:23:28	0|wumpus|luke-jr: where were you to NACK zmq when it was added?
370 2016-05-12 19:23:33	0|jcorgan|i'm happy to look at improving zmq
371 2016-05-12 19:23:35	0|jonasschnelli|I have added support for <n> clients
372 2016-05-12 19:23:35	0|luke-jr|jonasschnelli: no RPC over internet ever :<
373 2016-05-12 19:23:50	0|luke-jr|wumpus: just because zmq is crazy doesn't mean optional ZMQ support should be excluded ..
374 2016-05-12 19:23:51	0|jonasschnelli|RPC over internet over a reverse proxy is a wide use pratice.
375 2016-05-12 19:23:55	0|wumpus|(I don't think zmq is necessarily crazy)
376 2016-05-12 19:24:04	0|jtimon|sorry, what are the complaints with zmq? it is optional anyway
377 2016-05-12 19:24:19	0|wumpus|jtimon: I'm not sure either, it seems to be fashionable to complain about it
378 2016-05-12 19:24:21	0|luke-jr|jtimon: it's not optional if it is an excuse to exclude other systems
379 2016-05-12 19:24:33	0|luke-jr|zmq breaks protocol compatibility for minor bumps
380 2016-05-12 19:24:34	0|jonasschnelli|Working towards decoupling of the GUI and the wallet requires well defined channels/APIs
381 2016-05-12 19:24:39	0|luke-jr|ie, 4.1 won't work with 4.0 right
382 2016-05-12 19:25:00	0|jonasschnelli|ZMQ would require a tunnel (VPN, stunnel, etc.).
383 2016-05-12 19:25:06	0|luke-jr|also see all the Elements functionary problems due to ZMQ
384 2016-05-12 19:25:15	0|jcorgan|ZMQ 4.x is implementing CurveCP
385 2016-05-12 19:25:18	0|jtimon|I think it's a wonderful tool I would like to use more (although maybe the fact that its creator rewrote it completely in nanomsg may indeed indicate that some parts of zmq have become too complex)
386 2016-05-12 19:25:20	0|wumpus|zmq is basically only useful for local system, but so is RPC, it's not meant to be used over the internet
387 2016-05-12 19:25:45	0|jcorgan|see comment about curvecp
388 2016-05-12 19:25:55	0|wumpus|if you write tunneling for RPC - why not tunnel the notifications too?
389 2016-05-12 19:26:00	0|jonasschnelli|I just think Core users would love to have a GUI/wallet-frontend that can run on a different machine.
390 2016-05-12 19:26:11	0|jcorgan|http://curvezmq.org/
391 2016-05-12 19:26:17	0|jtimon|wumpus: exactly, it's for your intranet or at most inside a vpn (although I haven't tried that myself)
392 2016-05-12 19:26:19	0|luke-jr|jonasschnelli: yes, for years now that has been desirable
393 2016-05-12 19:26:26	0|wumpus|jonasschnelli: I agree, but does that need RPC notifications?
394 2016-05-12 19:26:28	0|jonasschnelli|wumpus: RPC does work extremly well in reverse-proxy mode.
395 2016-05-12 19:26:31	0|wumpus|jonasschnelli: what would you use it for?
396 2016-05-12 19:26:47	0|luke-jr|wumpus: right now it requires polling
397 2016-05-12 19:26:52	0|wumpus|e.g. to get notification of transactions/blocks the P2P protocol works fine
398 2016-05-12 19:26:53	0|jonasschnelli|Look at rtorrent or any other RPC daemon software.
399 2016-05-12 19:27:32	0|jonasschnelli|Polling is extremely inefficient. Long polling would allow clients to get "realtime" events while not require any other dependencies...
400 2016-05-12 19:27:37	0|jonasschnelli|And the code changes are tiny...
401 2016-05-12 19:27:38	0|wumpus|yes, agreed jonasschnelli
402 2016-05-12 19:27:47	0|wumpus|yes the code changes are tiny
403 2016-05-12 19:28:10	0|jonasschnelli|You could setup a save and secure remote bitcoind with a single apache/nginx config.
404 2016-05-12 19:28:18	0|wumpus|but I'm a bit afraid the same will happen as with REST, people will want to pile up things on top, and now for every notification people will want zmq and rpc longpull support
405 2016-05-12 19:28:19	0|jonasschnelli|Now do the same with ZMQ notifications... :)
406 2016-05-12 19:28:47	0|wumpus|just like people want every possible data item both through RPC and REST
407 2016-05-12 19:29:01	0|sipa|well ZMQ/P2P are suboptimal to write a remote GUI, as you can't filter for just wallet transactions
408 2016-05-12 19:29:04	0|wumpus|I believe thatthere is value to  keeping core limited to one interface
409 2016-05-12 19:29:08	0|luke-jr|wumpus: I don't see why this is a problem
410 2016-05-12 19:29:10	0|jonasschnelli|Yes. If i had a blank paper. I would drop ZMQ and REST in favor or RPC longpoll and the normal RPC.
411 2016-05-12 19:29:32	0|jcorgan|longpoll does indeed solve some of my original motivation for adding ZMQ
412 2016-05-12 19:29:32	0|sipa|jonasschnelli: i think you're biased because you're only thinking about one use case
413 2016-05-12 19:29:37	0|wumpus|well my iniitial idea for notifications was also something like longpoll, or a streaming socket
414 2016-05-12 19:29:42	0|wumpus|then a proxy from that to zmq
415 2016-05-12 19:29:45	0|wumpus|but zmq was first
416 2016-05-12 19:30:05	0|luke-jr|zmq is optional
417 2016-05-12 19:30:15	0|luke-jr|someone shouldn't need to install zmq for notifications
418 2016-05-12 19:30:18	0|jtimon|zmq's req/rep could replace both rpc and rest
419 2016-05-12 19:30:19	0|jonasschnelli|There is one big advantage of RPC long polling...
420 2016-05-12 19:30:26	0|jonasschnelli|you can have private notifications...
421 2016-05-12 19:30:31	0|jonasschnelli|secured behind auth
422 2016-05-12 19:30:34	0|wumpus|do we want private notifications?
423 2016-05-12 19:30:36	0|jonasschnelli|Like a new relevant wallet tx comes in
424 2016-05-12 19:30:43	0|sipa|wumpus: for a remote wallet gui you do
425 2016-05-12 19:30:52	0|jonasschnelli|It would be on the same level then the RPC security...
426 2016-05-12 19:30:57	0|jonasschnelli|just instead of poll you can have push
427 2016-05-12 19:31:06	0|wumpus|connect a remote wallet GUI through RPC?
428 2016-05-12 19:31:10	0|wumpus|wasn't RPC meant to be for local usage?
429 2016-05-12 19:31:15	0|jonasschnelli|I would not add this over ZMQ because of the unknown risks
430 2016-05-12 19:31:20	0|luke-jr|jonasschnelli: btw FYI https://en.bitcoin.it/wiki/Wallet_protocol
431 2016-05-12 19:31:32	0|jonasschnelli|wumpus: that is an argument.
432 2016-05-12 19:31:33	0|wumpus|I think the idea was to attach a *wallet*, not a wallet GUI
433 2016-05-12 19:31:47	0|wumpus|the wallet needs to be split from the core
434 2016-05-12 19:31:58	0|luke-jr|a wallet can be attached over p2p fine
435 2016-05-12 19:32:01	0|jonasschnelli|Another solution would be to provide a tiny daemon that would interact between core <-> remote GUI/wallet.
436 2016-05-12 19:32:15	0|wumpus|I'd prefer for core to handle semi-public data, then have a more restricted wallet
437 2016-05-12 19:32:17	0|jonasschnelli|But that tiny daemon would speak RPC with the outside/internet.
438 2016-05-12 19:32:21	0|luke-jr|ideally we should have node <-> wallet <-> GUI
439 2016-05-12 19:32:47	0|wumpus|luke-jr: yes
440 2016-05-12 19:32:52	0|jonasschnelli|luke-jr: But how would you see the communication channel between wallet <> GUI?
441 2016-05-12 19:32:57	0|jcorgan|another motivation for adding ZMQ was to allow bitcoind to be a trusted "border gateway", that spoke P2P and consensus, and then stuff behind it would be very simple ZMQ-based software that didn't need to know all about those things
442 2016-05-12 19:33:02	0|wumpus|but wallet<->GUI could be a competely different protocol than node<->wallet
443 2016-05-12 19:33:03	0|jonasschnelli|Needs to be bidirect.
444 2016-05-12 19:33:08	0|luke-jr|jonasschnelli: see wiki link; or something like what you're doing
445 2016-05-12 19:33:15	0|jonasschnelli|node <> wallet is probably p2p?
446 2016-05-12 19:33:19	0|sipa|yes
447 2016-05-12 19:33:21	0|luke-jr|yes
448 2016-05-12 19:33:45	0|morcos|node <-> wallet being p2p leaves a lot of missing pieces
449 2016-05-12 19:33:46	0|wumpus|maybe authentiated P2P
450 2016-05-12 19:33:47	0|jonasschnelli|But what direction do we want to go for <wallet> <-> <gui>?
451 2016-05-12 19:33:47	0|morcos|fee estimation
452 2016-05-12 19:33:48	0|wumpus|as you proposed
453 2016-05-12 19:33:55	0|wumpus|with some private extensions
454 2016-05-12 19:34:00	0|sipa|jonasschnelli: up to the wallet
455 2016-05-12 19:34:07	0|morcos|mempool actions (eviction, how close to top of mempool, whether it was accepted)
456 2016-05-12 19:34:12	0|jonasschnelli|morcos: fee estimations can be done with authentication/private extensions.
457 2016-05-12 19:34:14	0|BlueMatt|ehh, lets not layer everything onto p2p extensions
458 2016-05-12 19:34:23	0|luke-jr|jonasschnelli: XMPP! /s
459 2016-05-12 19:34:38	0|jtimon|why people want to remove zmq? I still don't undesrtand
460 2016-05-12 19:34:41	0|jonasschnelli|sipa: That's why I propose RPC long poll (as a "wallet" feature). :)
461 2016-05-12 19:34:55	0|sipa|jonasschnelli: i'm not sure our wallet should provide that
462 2016-05-12 19:35:01	0|sipa|jonasschnelli: at least not at this stage
463 2016-05-12 19:35:02	0|wumpus|BlueMatt: well if we have a autenticated+encryped P2P protocol, adding private extensions is attractive
464 2016-05-12 19:35:27	0|wumpus|jtimon: I don't want to remove zmq. But I do think we should have one notification mechanism.
465 2016-05-12 19:35:40	0|kanzure|notifications over zeromq would be nice
466 2016-05-12 19:35:43	0|jtimon|wumpus: why not it be zmq?
467 2016-05-12 19:35:43	0|luke-jr|jtimon: I just want to keep ZMQ as an optional feature, not necessary for this stuff
468 2016-05-12 19:35:56	0|wumpus|jtimon: I don't want to amintain an endless pile-up of different external notification mechanisms
469 2016-05-12 19:36:08	0|jonasschnelli|Yes. We don't want that.
470 2016-05-12 19:36:10	0|jtimon|wumpus: ok, why not use ONLY zmq?
471 2016-05-12 19:36:11	0|BlueMatt|wumpus: its quite attractive to shove everything into one monolithic protocol, indeed, but I do think there is a lot of value in splitting things out (though I wouldnt be upset if they were logically different code that just happened to have the same on-wire protocol or so)
472 2016-05-12 19:36:11	0|jcorgan|jtimon: i agree that zmq req/rep and pub/sub could provide the entirety of needed interfaces to bitcoind, but that's an argument lost years ago :)
473 2016-05-12 19:36:23	0|luke-jr|jtimon: then it's no longer optional
474 2016-05-12 19:36:24	0|kanzure|unfortunately the simplest private notification stuff that the rest of the community would understand would be.... web hooks. :(
475 2016-05-12 19:36:39	0|luke-jr|jcorgan: except ZMQ has no protocol compatibility
476 2016-05-12 19:36:44	0|wumpus|websocket would work
477 2016-05-12 19:36:49	0|morcos|wumpus: i think we can move towards deprecating some things that we view as redundant.  what we should do now though is decide what would work for a node <-> wallet communication protocol.  b/c that is something we defintiely want.
478 2016-05-12 19:36:57	0|jtimon|luke-jr: ok, either we have 1, we remove zmq or we make it non-optional
479 2016-05-12 19:37:00	0|luke-jr|wumpus: websocket doesn't define a protocol
480 2016-05-12 19:37:03	0|wumpus|but just as well as longpoll
481 2016-05-12 19:37:09	0|jonasschnelli|websockets have bad security
482 2016-05-12 19:37:24	0|sipa|wumpus: i think it may be reasonable to say that zmq is for node notifications (which are unauthenticated) and longpoll for wallet notifications
483 2016-05-12 19:37:26	0|luke-jr|jtimon: if we can only have 1, then IMO zmq can go
484 2016-05-12 19:37:30	0|wumpus|morcos: what would you propose to deprecate?
485 2016-05-12 19:37:33	0|jtimon|luke-jr: what do you mean by "protocol compatibility"?
486 2016-05-12 19:37:40	0|luke-jr|jtimon: but I don't think we should limit to 1
487 2016-05-12 19:37:50	0|wumpus|sipa: but what jonasschnelli wants is not wallet notifications
488 2016-05-12 19:37:52	0|luke-jr|jtimon: ZMQ 4.0 can't talk to ZMQ 4.1
489 2016-05-12 19:37:53	0|jtimon|luke-jr: and IMO, if we only want to have one, zmq should stay
490 2016-05-12 19:37:55	0|sipa|wumpus: yes it is
491 2016-05-12 19:38:01	0|wumpus|sipa: he wants the same stuff as is now offered over zmq
492 2016-05-12 19:38:08	0|jonasschnelli|wumpus sipa: What i want is going toward wallet notifications
493 2016-05-12 19:38:11	0|morcos|wumpus: well my argument is to first define the right way to do node <-> wallet and then do that (even if it means adding something) and then we can revisit and see what we have that is extraneous
494 2016-05-12 19:38:17	0|jonasschnelli|But also notifications that could enable a remote GUI
495 2016-05-12 19:38:21	0|sipa|i don't think so; nothing of what is offered over ZMQ is what you need for a remote wallet GUI
496 2016-05-12 19:38:22	0|jonasschnelli|(mempool / peers, etc9
497 2016-05-12 19:38:38	0|jonasschnelli|A Core node GUI on a smartphone....
498 2016-05-12 19:38:48	0|jonasschnelli|Which can't work over ZMQ
499 2016-05-12 19:39:05	0|sipa|i think we shouldn't discuss that in the current stage
500 2016-05-12 19:39:07	0|jcorgan|of course it *could*
501 2016-05-12 19:39:15	0|jcorgan|but not as it is written today
502 2016-05-12 19:39:25	0|wumpus|jonasschnelli: how would you protect the RPC connection to the smartphone?
503 2016-05-12 19:39:27	0|sipa|i don't want my bitcoin core full node software to be accessible by smartphones on the internet, i think
504 2016-05-12 19:39:29	0|kanzure|jcorgan: how much difference?
505 2016-05-12 19:39:35	0|wumpus|jonasschnelli: the same tunneling tool could route the notifications from zmq
506 2016-05-12 19:39:40	0|jcorgan|zmq is a transport
507 2016-05-12 19:39:40	0|jonasschnelli|wumpus: reverse proxy, SSL auth, mod_security
508 2016-05-12 19:39:44	0|sipa|except the p2p protocol, which is must provide by necessarily
509 2016-05-12 19:39:49	0|morcos|jonasschnelli: don't you think we shoudl concentrate on node <-> wallet now?  b/c then different people could build different wallet <-> gui protocols if we wanted.  our core responsiblity is the node
510 2016-05-12 19:39:58	0|jcorgan|you have to define a protocol/serialization/encapsulation to run over it
511 2016-05-12 19:39:59	0|luke-jr|jcorgan: except then you need to make sure your ZMQ lib versions match, which is just annoying
512 2016-05-12 19:40:15	0|jcorgan|you can say that about dozens of other libs
513 2016-05-12 19:40:17	0|jonasschnelli|morcos: I'm working on node <-> wallet. That's why I'm working on auth/enc. :)
514 2016-05-12 19:40:23	0|jtimon|jonasschnelli: how can something "not work over ZMQ"? can't you proxy the messages through some other protocol outside of bitcoind once you get the zmq messages?
515 2016-05-12 19:40:35	0|luke-jr|jcorgan: most protocols are compatible across lib versions
516 2016-05-12 19:40:37	0|jonasschnelli|I don't want to talk SPV over unencrypted channels...
517 2016-05-12 19:40:37	0|sipa|jtimon: because authentication
518 2016-05-12 19:40:49	0|kanzure|unencrypted...?
519 2016-05-12 19:40:51	0|jcorgan|anyway, if anyone wants to pursue making changes to the current zmq implementation, we can talk in zurich about it
520 2016-05-12 19:40:52	0|wumpus|wellt he same protocol that offers RPC over the internet needs authentication too
521 2016-05-12 19:40:54	0|jonasschnelli|jtimon: Sure can you. But can a normal user do that?
522 2016-05-12 19:40:57	0|wumpus|you have exactly the same problem there
523 2016-05-12 19:41:26	0|jonasschnelli|I just compared the requirements to setup a remote rtorrent GUI with a possible remote Core node GUI.
524 2016-05-12 19:41:46	0|wumpus|in any case, I'm not strongly against longpoll RPC notifications
525 2016-05-12 19:41:53	0|jtimon|jonasschnelli: well, what was your "normal user" using isntead of zmq?
526 2016-05-12 19:41:54	0|jonasschnelli|And i feal that rpc long polls would result in a easy setup...
527 2016-05-12 19:42:14	0|sipa|morcos, wumpus, jonasschnelli: i agree node <-> wallet is what we should be talking about first, and the apparent need to add private extensions is a sign of a deeper problem: an SPV wallet without trusted full node will be lacking in features
528 2016-05-12 19:42:23	0|kanzure|easiest setup for non-core-developers is going to be web hooks :(
529 2016-05-12 19:42:36	0|kanzure|not rpc
530 2016-05-12 19:42:50	0|d4de|0MQ supports GSSAPI and Curve auth, unencrypted?!
531 2016-05-12 19:42:55	0|wumpus|and indeed, RPC longpoll can be supported without linking against external dependencies
532 2016-05-12 19:42:57	0|morcos|sipa: good point, but maybe thats a separate problem to solve?
533 2016-05-12 19:42:58	0|wumpus|which is an advantage
534 2016-05-12 19:43:00	0|jonasschnelli|</rpc-long-poll-advertizing> I'm happy to keep the PR alive... I'll also try to work on a intermediate script that could -> ZMQ and does -> RPC
535 2016-05-12 19:43:17	0|sipa|morcos: well if that problem is solved (unsure how...), maybe we don't need private extensions :)
536 2016-05-12 19:43:38	0|morcos|yes, but it'll always be superior to some degree to have access to a trusted node
537 2016-05-12 19:43:38	0|sipa|jonasschnelli: you can't... ZMQ doesn't offer wallet-filtered events
538 2016-05-12 19:43:44	0|instagibbs|sipa, do you mean things like access to mempool, etc?
539 2016-05-12 19:43:50	0|wumpus|at the least we should make a clear division about *what* should be offered over RPC longpoll and what over zmq and what over (theoretic) P2P extensions
540 2016-05-12 19:43:51	0|sipa|instagibbs: yes, and fee estimation
541 2016-05-12 19:43:52	0|jonasschnelli|sipa: you still can talk RPC...
542 2016-05-12 19:43:52	0|morcos|so we shouldn't limit ourselves to things that you can't do without that
543 2016-05-12 19:43:58	0|wumpus|so not everything ends up on all three
544 2016-05-12 19:44:07	0|jonasschnelli|sipa: just the notifications need to be ZMQ to avoid polling..
545 2016-05-12 19:44:10	0|instagibbs|even if you have access to full node over p2p, that doesn't get you that (maybe that's what you meant)
546 2016-05-12 19:44:28	0|jonasschnelli|sipa: thats why I think adding long poll would not change the security aspect.
547 2016-05-12 19:44:43	0|sipa|jonasschnelli: but you don't want to send a ZMQ event for every transaction over the wire... that would consume as much bandwidth as just the p2p protocol
548 2016-05-12 19:44:56	0|wumpus|adding longpoll to RPC won't change any security aspect
549 2016-05-12 19:45:02	0|luke-jr|more, since ZMQ is ASCII :P
550 2016-05-12 19:45:07	0|sipa|luke-jr: wut?
551 2016-05-12 19:45:09	0|kanzure|rpc thread count exhaustion might change a security aspect
552 2016-05-12 19:45:17	0|wumpus|(except if it encourages opening up your RPC port to the internet)
553 2016-05-12 19:45:20	0|jonasschnelli|sipa: Right. Long poll could then be extended to relevant wallet txes.
554 2016-05-12 19:45:26	0|wumpus|kanzure: you need user/pass for that, so only the owner can attack it
555 2016-05-12 19:45:38	0|wumpus|kanzure: I would be against unauthenticated longpolls, that's for sure
556 2016-05-12 19:45:48	0|luke-jr|sipa: at least the main protocol design (it transmits binary data as-is, IIRC)
557 2016-05-12 19:45:53	0|wumpus|kanzure: and if you can authenticate you can do *much* worse things than hold up threads
558 2016-05-12 19:46:00	0|jtimon|sipa: of course you can filter stuff with zmq, you can do anything with zmq, is "network complete"
559 2016-05-12 19:46:17	0|jtimon|you may need more processes
560 2016-05-12 19:46:23	0|sipa|jtimon: that's like saying that x86 asm is better than http, because it can do everything
561 2016-05-12 19:46:35	0|jonasschnelli|Another + for RPC long poll: no dependencies...
562 2016-05-12 19:46:36	0|sipa|jtimon: of course you can filter, but the filter would be too late
563 2016-05-12 19:46:41	0|wumpus|so anyhow: I'd say continue your work jonasschnelli
564 2016-05-12 19:46:46	0|jtimon|sipa: I really don't undesrtand what you claim can't be done with zmq
565 2016-05-12 19:46:47	0|kanzure|my zeromq person just left the channel in a huff ("these people are retarded") hah...
566 2016-05-12 19:46:54	0|jonasschnelli|wumpus: Okay.. I keep the PR warm.
567 2016-05-12 19:47:03	0|luke-jr|again, we ALREADY have longpolling, so I don't see why make a big deal of it
568 2016-05-12 19:47:19	0|jtimon|sipa: you want to filter by a set of addresses or something?
569 2016-05-12 19:47:20	0|jonasschnelli|luke-jr: Yes. It's not a big deal...
570 2016-05-12 19:47:32	0|sipa|jtimon: we don't want ZMQ to be authenticated, so it can't provide wallet-specific information, which means the ZMQ client will need to do the filtering
571 2016-05-12 19:47:40	0|wumpus|people seem to have trouble to get zmq to work in practice, maybe if you can provide some easy examples for RPC longpolling then it will be the more popular way to do notifications soon
572 2016-05-12 19:47:42	0|jonasschnelli|sipa: Agree
573 2016-05-12 19:47:49	0|sipa|jtimon: which is duplicating logic, because that's something the wallet should do, not the gui
574 2016-05-12 19:47:58	0|jonasschnelli|wumpus: Okay. I'll add some examples... good point!
575 2016-05-12 19:48:02	0|wumpus|(I had so much trouble convincing joinmarket to use zmq notifications instead of wacky -notifyX scripts that I just gave u)
576 2016-05-12 19:48:11	0|luke-jr|LP is really simple. just make a normal request and wait ☺
577 2016-05-12 19:48:16	0|jonasschnelli|wumpus: hah .. yes. Another +1 for RPC long poll. :)
578 2016-05-12 19:48:28	0|wumpus|maybe it's the extra dependency, maybe it's unfamiliarity, I don't know.
579 2016-05-12 19:48:31	0|jonasschnelli|People will call exes on every transaction...
580 2016-05-12 19:48:38	0|jtimon|sipa: oh, you don't want authenticated connections going through zmq processes...I don't undesrtand why, but ok, that seems to be the piece I was missing
581 2016-05-12 19:48:40	0|sipa|jtimon: so yes, of course, you can do anything, if you reimplement the wallet in the client, but that was exactly what we were trying to avoid :)
582 2016-05-12 19:48:47	0|gmaxwell|luke-jr: lots of things time out.
583 2016-05-12 19:48:58	0|jonasschnelli|gmaxwell: time out doesnt matter
584 2016-05-12 19:48:59	0|gmaxwell|this discussion seems ratholed.
585 2016-05-12 19:49:07	0|jonasschnelli|Because you have a queue/state on the server
586 2016-05-12 19:49:15	0|wumpus|timeout doesn't matter with longpoll, you can just re-request
587 2016-05-12 19:49:20	0|jtimon|sipa: no, I would just use some kind of authentication
588 2016-05-12 19:49:22	0|kanzure|we need more user metric survey data about whehter rpc is really easy for folks. i think all the web app developers only know web hooks and HTTP.
589 2016-05-12 19:49:52	0|wumpus|kanzure: moving away from JSON-RPC as the main RPC API is out of the question at this stage
590 2016-05-12 19:50:03	0|wumpus|just too much software and libraries that assume it
591 2016-05-12 19:50:07	0|sipa|are there other topics?
592 2016-05-12 19:50:20	0|kanzure|er i did not mean to unintentionally advocate for moving away from json-rpc
593 2016-05-12 19:50:53	0|wumpus|doesn't seem to be
594 2016-05-12 19:51:11	0|wumpus|#endmeeting
595 2016-05-12 19:51:12	0|lightningbot|Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-05-12-19.00.log.html
596 2016-05-12 19:51:12	0|lightningbot|Meeting ended Thu May 12 19:51:11 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
597 2016-05-12 19:51:12	0|lightningbot|Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-05-12-19.00.html
598 2016-05-12 19:51:12	0|lightningbot|Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-05-12-19.00.txt
599 2016-05-12 19:51:54	0|jcorgan|do we have any grand plans or even wishlists for zurich?
600 2016-05-12 19:52:06	0|jtimon|kanzure: I did suggest replacing json-rpc with zmq json req/rep, but I knew very few people would consider it seriously
601 2016-05-12 19:52:19	0|sipa|i'd like to talk about p2p encryption, segwit, signature aggregation, ...
602 2016-05-12 19:52:33	0|jonasschnelli|Agree with sipa. We could also form groups to directly work on stuff.
603 2016-05-12 19:52:57	0|sipa|but the overall goal should just be to facillitate improvements that are in the pipeline by meeting in face... whatever those improvements are
604 2016-05-12 19:52:59	0|kanzure|i would like to do braindumps to get things written down
605 2016-05-12 19:53:18	0|jonasschnelli|+1 for writing down things...
606 2016-05-12 19:53:22	0|kanzure|i am good at this
607 2016-05-12 19:53:27	0|jcorgan|suggest doing as much brainstorming/dumping in advance to not use precious F2F time
608 2016-05-12 19:53:56	0|kanzure|would like to draw up list of too-often-unspoken proposals to write down more details about
609 2016-05-12 19:54:10	0|sipa|that'd be great
610 2016-05-12 19:54:15	0|jonasschnelli|I think we should't feel too much pressure. The fact that we are meeting face to face alone is a value.
611 2016-05-12 19:54:23	0|jcorgan|of course
612 2016-05-12 19:54:23	0|sipa|agree
613 2016-05-12 19:54:37	0|jonasschnelli|Even if there is no outcome. It would be a success.
614 2016-05-12 19:55:00	0|kanzure|some feedback from d4de- he recommends that future bitcoin core meetings shoud have someone that more actively guides discussion, e.g. make things more explicit rather than implicit
615 2016-05-12 19:55:07	0|wumpus|jtimon: right - there's no deep reason why zmq req/rep wouldn't work for the main RPC API, but it's .... unfamiliar for people, see my comment above about trying to get people to even use zmq for basic notifications
616 2016-05-12 19:55:15	0|jcorgan|the face time is extremely valuable, just want to make the most of it
617 2016-05-12 19:55:17	0|kanzure|jonasschnelli: gosh that sounds like one of those unreasonable metrics :) (kidding)
618 2016-05-12 19:55:35	0|jonasschnelli|kanzure: It's called positiv thinking. :)
619 2016-05-12 19:55:35	0|wumpus|kanzure: are you volunteering?
620 2016-05-12 19:56:01	0|jcorgan|so if there are things that can be done ahead of time that don't necessarily need the f2f time, it would make sense to get it out of the way beforehand
621 2016-05-12 19:56:06	0|jonasschnelli|kanzure: I think wumpus is doing a fantastic job and you easly overmoderate on IRC.
622 2016-05-12 19:56:06	0|kanzure|wumpus: for writing things down? yes. for brainstorming? sure... for meetings, i'll defer to our friendly neighborhod pointy-hair.
623 2016-05-12 19:56:10	0|gmaxwell|wumpus: execpt the no stable on the wire format, and no reliable delivery...
624 2016-05-12 19:56:43	0|jonasschnelli|jcorgan: Agree. You could start a discussion on the bitcoin-core-dev ML?
625 2016-05-12 19:56:48	0|kanzure|jonasschnelli: i don't have enough context on many of these topics to correctly guide meetings. so it's not something you should be concerned about. heh.
626 2016-05-12 19:56:53	0|jtimon|wumpus: yeah, it seems people forgot about socket-like things
627 2016-05-12 19:56:53	0|wumpus|gmaxwell: I think you can have reliable delivry over zmq, just not in the way we use it
628 2016-05-12 19:57:10	0|wumpus|jtimon: well it's a complete new API to learn
629 2016-05-12 19:57:40	0|wumpus|jtimon: even if you already know normal socket programming, zmq is like a whole new alien landscape
630 2016-05-12 19:57:55	0|jonasschnelli|jtimon: RPC long poll require no "real" dependencies, you just aim a http request and wait until you get response.
631 2016-05-12 19:58:02	0|jcorgan|wumpus: jtimon: people don't seem to grok message based vs. function call based
632 2016-05-12 19:58:14	0|kanzure|jonasschnelli: that can cause an unintentional DoS against yourself
633 2016-05-12 19:58:25	0|jonasschnelli|kanzure: explain?
634 2016-05-12 19:58:31	0|jtimon|wumpus: I hadn't used sockets in ages and it was very easy to me (but as said I just use xreq/xrep, push/pull and pub/sub, no dealers or other stuff)
635 2016-05-12 19:58:31	0|kanzure|rpc thread exhaustion, as mentioned above
636 2016-05-12 19:58:32	0|jcorgan|jonasschnelli: i'll make a suggestion on the ML then
637 2016-05-12 19:58:42	0|jonasschnelli|jcorgan:+1
638 2016-05-12 19:59:03	0|jonasschnelli|kanzure: no.. you only run the poll http request on a single thread...
639 2016-05-12 19:59:15	0|jonasschnelli|Only one request in parallel.
640 2016-05-12 19:59:19	0|kanzure|jonasschnelli: rpcthreads
641 2016-05-12 19:59:28	0|jonasschnelli|But thats the serverside.
642 2016-05-12 19:59:35	0|kanzure|yes.. that's still DoSing yourself.
643 2016-05-12 19:59:48	0|kanzure|you're not enforcing one request in parallel
644 2016-05-12 19:59:56	0|jtimon|wumpus: but apparently nanomsg simplifies things further removing the context and things like that
645 2016-05-12 20:00:01	0|wumpus|in principle you don't even need a rpc thread open for a long-poller, if you'd use the event based http, but that's a topic for the far future
646 2016-05-12 20:00:10	0|jonasschnelli|kanzure: sure. But that would not be different to the "normal" rpc calls?
647 2016-05-12 20:00:12	0|kanzure|jtimon: in addition to nanomsg, look at nq
648 2016-05-12 20:00:27	0|kanzure|jonasschnelli: well, not many of them are meant to be open for a while
649 2016-05-12 20:00:44	0|jonasschnelli|Sure. But you control them. Its secured behind auth.
650 2016-05-12 20:00:44	0|jtimon|kanzure: nq? do you have a link?
651 2016-05-12 20:01:02	0|jonasschnelli|kanzure: No strangers will connect to your RPC.
652 2016-05-12 20:01:08	0|kanzure|jtimon: actually no :) i think this means i owe you BTC... hah.
653 2016-05-12 20:01:10	0|wumpus|and if you need multiple long-pollers you can just increase the number of rpc threads anyhow, it's not like we need to support 1000 people in a chat
654 2016-05-12 20:01:19	0|wumpus|no need to overdesign/overcomplicate
655 2016-05-12 20:01:20	0|jonasschnelli|Indeed. :)
656 2016-05-12 20:01:46	0|jtimon|jonasschnelli: you can have an http server with auth that connects "locally" (intranet) to stuff via zmq
657 2016-05-12 20:01:49	0|kanzure|manual thread management, got it..
658 2016-05-12 20:02:01	0|luke-jr|we're moving the dev meetings to the p2p network and using longpolling with bitcoin-cli to read the chat? :P
659 2016-05-12 20:02:12	0|jtimon|jonasschnelli: but yeah, I guess that's one extra step you would be saving
660 2016-05-12 20:02:13	0|jonasschnelli|jtimon: Yes. Sure. But why the roundtrip when you can have RPC long poll?
661 2016-05-12 20:02:15	0|kanzure|we don't mention in our docs "don't connect to your bitcoin node from a thousand workers that listen for notifications"
662 2016-05-12 20:02:37	0|jonasschnelli|jtimon: you don't need the ZMQ dependencies if you want to serve a push not. over http
663 2016-05-12 20:02:38	0|wumpus|kanzure: some things are common sense, but feel free to add it explicitly
664 2016-05-12 20:03:04	0|wumpus|kanzure: and for that, zmq is indeed more suited as it has one-to-many broadcast
665 2016-05-12 20:03:44	0|jtimon|jonasschnelli: of course, strictly speaking "you don't need anything" to do anything, it's the "you can't do X using zmq" that nerves me
666 2016-05-12 20:04:30	0|jonasschnelli|I think ZMQ is useful....
667 2016-05-12 20:04:35	0|gmaxwell|jtimon: being excessively pedantic is not productive.
668 2016-05-12 20:04:39	0|jonasschnelli|Its just pain in the ass to get a notification to a remote end.
669 2016-05-12 20:05:40	0|jtimon|gmaxwell: sure, I was just misunderstanding some of the "you can't do this with zmq" claims, they were really just "you also need X" or similar things
670 2016-05-12 20:05:52	0|jtimon|mistery solved
671 2016-05-12 20:08:31	0|jtimon|and I'm sorry for being so biased in favor of zmq whenever it's an option for anything, I guess it may equilibrate some other biases against it.
672 2016-05-12 20:17:51	0|jcorgan|one suggestion--we seem to be going back and forth between different use cases and different means to accomplish them
673 2016-05-12 20:19:38	0|jcorgan|i'd rather clarify what all the use cases are first, then see what technology best suits them, then see how that maps to what exists and what doesn't in the client
674 2016-05-12 20:21:43	0|jcorgan|*best suits each of them
675 2016-05-12 20:25:45	0|jtimon|jcorgan: that sounds very reasonable
676 2016-05-12 20:26:15	0|jtimon|analysis first, arch/design decisions later
677 2016-05-12 22:54:36	0|BlueMatt|ugh, master is segfaulting trying to call rpc
678 2016-05-12 22:54:53	0|sipa|BlueMatt: elaborate?
679 2016-05-12 22:55:00	0|BlueMatt|gdb'ing
680 2016-05-12 22:55:08	0|sipa|thanks
681 2016-05-12 22:55:09	0|BlueMatt|but all i did was start and then try to call rpc :(
682 2016-05-12 22:57:01	0|BlueMatt|oh, false alarm...custom change blew up
683 2016-05-12 22:57:04	0|BlueMatt|sorry for the noise :/
684 2016-05-12 22:57:19	0|BlueMatt|fucking 1-line change segfaults :(
685 2016-05-12 22:57:41	0|sipa|we need a programming language with built-in reed-selomon code
686 2016-05-12 22:57:56	0|sipa|so that the compiler can correct 1-line errors :p
687 2016-05-12 22:58:05	0|BlueMatt|heh
688 2016-05-12 22:58:23	0|gmaxwell|sipa: and updating the code means you need to read all of it?
689 2016-05-12 22:58:41	0|sipa|gmaxwell: you mean you don't already do that?