1 2016-09-29 05:58:38	0|GitHub190|[13bitcoin] 15laanwj pushed 5 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/dc641415e75e...d675984fdfa4
  2 2016-09-29 05:58:39	0|GitHub190|13bitcoin/06master 143b4b6dc 15Luke Dashjr: Add MIT license to autogen.sh and share/genbuild.sh
  3 2016-09-29 05:58:39	0|GitHub190|13bitcoin/06master 14f4dffdd 15Luke Dashjr: Add MIT license to Makefiles
  4 2016-09-29 05:58:40	0|GitHub190|13bitcoin/06master 143f8a5d8 15Luke Dashjr: Trivial: build-aux/m4/l_atomic: Fix typo
  5 2016-09-29 05:58:53	0|GitHub162|[13bitcoin] 15laanwj closed pull request #8784: Copyright headers for build scripts (06master...06license_build) 02https://github.com/bitcoin/bitcoin/pull/8784
  6 2016-09-29 06:01:25	0|wumpus|this is strange, https://github.com/bitcoin/bitcoin/pull/8832   "waxmihs2902 approved these changes 10 hours ago"   clicking on that users name gives a 404 page
  7 2016-09-29 06:03:20	0|paveljanik|User deleted himself?
  8 2016-09-29 06:03:53	0|wumpus|I guess
  9 2016-09-29 06:04:03	0|wumpus|or github deleted him
 10 2016-09-29 06:06:58	0|wumpus|as it is impossible to remove approvals, random-approval-spamming trolls wouldn't be unthinkable. Note that for normal posts, the user gets changed to 'ghost' if they disappear, so the link doesn't break.
 11 2016-09-29 06:08:25	0|paveljanik|I think the approval stuff is half-baked...
 12 2016-09-29 06:09:21	0|paveljanik|So maybe someone is trying DoS on us...
 13 2016-09-29 06:09:49	0|wumpus|it's only on one issue AFAIK, so I doubt it'd be targetted on us :)
 14 2016-09-29 06:10:26	0|paveljanik|./DoS ... and then while :; do ./DoS; done ;-)
 15 2016-09-29 06:10:40	0|wumpus|or they got caught *really* early; more likely such a troll/bot would just randomly approve pulls all over the site
 16 2016-09-29 06:11:29	0|paveljanik|wumpus, search for the username on the github...
 17 2016-09-29 06:11:33	0|paveljanik|complete github...
 18 2016-09-29 06:12:09	0|wumpus|luckily they didn't add a message, the message added with review approval is also un-removable/un-editable IIRC
 19 2016-09-29 06:12:34	0|wumpus|yes its half-baked :)
 20 2016-09-29 06:12:46	0|wumpus|I'm sure they'll fix this eventually
 21 2016-09-29 06:13:13	0|GitHub131|13bitcoin/06master 14fa05cfd 15MarcoFalke: [rpc] throw JSONRPCError when utxo set can not be read
 22 2016-09-29 06:13:13	0|GitHub131|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/d675984fdfa4...7d563cc16d64
 23 2016-09-29 06:13:14	0|GitHub131|13bitcoin/06master 147d563cc 15Wladimir J. van der Laan: Merge #8832: [rpc] throw JSONRPCError when utxo set can not be read...
 24 2016-09-29 06:13:27	0|GitHub93|[13bitcoin] 15laanwj closed pull request #8832: [rpc] throw JSONRPCError when utxo set can not be read (06master...06Mf1610-rpcUtxoFail) 02https://github.com/bitcoin/bitcoin/pull/8832
 25 2016-09-29 07:19:48	0|GitHub3|13bitcoin/06master 1464047f8 15Wladimir J. van der Laan: depends: Add libevent compatibility patch for windows...
 26 2016-09-29 07:19:48	0|GitHub3|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/7d563cc16d64...489a6ab5073c
 27 2016-09-29 07:19:49	0|GitHub3|13bitcoin/06master 14489a6ab 15Wladimir J. van der Laan: Merge #8730: depends: Add libevent compatibility patch for windows...
 28 2016-09-29 07:19:58	0|GitHub165|[13bitcoin] 15laanwj closed pull request #8730: depends: Add libevent compatibility patch for windows (06master...062016_09_libevent_windows_gcc_531) 02https://github.com/bitcoin/bitcoin/pull/8730
 29 2016-09-29 07:21:13	0|GitHub162|[13bitcoin] 15laanwj closed pull request #7522: Bugfix: Only use git for build info if the repository is actually the right one (06master...06bugfix_gitdir) 02https://github.com/bitcoin/bitcoin/pull/7522
 30 2016-09-29 07:24:48	0|wumpus|cfields_: I've assigned some build-system issues to you, hope you don't mind
 31 2016-09-29 08:50:58	0|GitHub87|13bitcoin/06master 1454e5d7c 15jnewbery: Add bitcoin-tx JSON tests
 32 2016-09-29 08:50:58	0|GitHub87|[13bitcoin] 15MarcoFalke pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/489a6ab5073c...8ca69a2a88a7
 33 2016-09-29 08:50:59	0|GitHub87|13bitcoin/06master 148ca69a2 15MarcoFalke: Merge #8829: Add bitcoin-tx JSON tests...
 34 2016-09-29 08:51:08	0|GitHub21|[13bitcoin] 15MarcoFalke closed pull request #8829: Add bitcoin-tx JSON tests (06master...06test-bitcoin-tx-json) 02https://github.com/bitcoin/bitcoin/pull/8829
 35 2016-09-29 09:00:41	0|wumpus|weird, I'm on testnet and trying to rebroadcast a transaction using "sendrawtransaction" but it's not working, I don't see anything appear in my wireshark session
 36 2016-09-29 09:01:48	0|wumpus|maybe all the current nodes already know of it, otoh the "broadcast through 1 node(s)" in status doesn't increase either
 37 2016-09-29 09:03:27	0|sipa|could it be that your peers knew about it, but evicted it?
 38 2016-09-29 09:03:50	0|wumpus|but for they they'd have to request it first
 39 2016-09-29 09:04:26	0|MarcoFalke|What about resendwallettransactions
 40 2016-09-29 09:04:29	0|sipa|not necessarily from you
 41 2016-09-29 09:05:03	0|wumpus|MarcoFalke: this should work with sendrawtransaction
 42 2016-09-29 09:05:38	0|wumpus|could be that the net refactoring work changed the assumptions, will add some debugging...
 43 2016-09-29 09:06:31	0|sipa|i don't think so
 44 2016-09-29 09:06:42	0|MarcoFalke|its a noop when fHaveMempool?
 45 2016-09-29 09:07:01	0|sipa|no
 46 2016-09-29 09:07:05	0|wumpus|I don't hope so
 47 2016-09-29 09:07:25	0|MarcoFalke|sendrawtransaction will only put in in your mempool
 48 2016-09-29 09:07:36	0|sipa|and then loop over all peers
 49 2016-09-29 09:07:42	0|sipa|and call PushInventory
 50 2016-09-29 09:07:49	0|MarcoFalke|oh
 51 2016-09-29 09:08:31	0|sipa|but PushInventory is a noop if filterInventoryKnown for that peer already contains the tx
 52 2016-09-29 09:09:08	0|wumpus|but it can't be there unless the peer requested it from *us*right?
 53 2016-09-29 09:09:40	0|wumpus|hm or if they inved it for relay to us, I guess
 54 2016-09-29 09:10:46	0|wumpus|I don't have a capture of the whole session so we'll never know for sure
 55 2016-09-29 09:10:48	0|sipa|well you can enable -debug=net and see if any messages go out in response to sendrawtransaction
 56 2016-09-29 09:10:56	0|sipa|oh
 57 2016-09-29 09:11:00	0|sipa|you're already doing that
 58 2016-09-29 09:11:14	0|wumpus|no, no messages went out
 59 2016-09-29 09:11:30	0|wumpus|and according to the tx metadata apparently one peer ever requested the transaction
 60 2016-09-29 09:11:38	0|wumpus|but that was before I started logging
 61 2016-09-29 09:13:34	0|wumpus|it could also be that that counter is broken, of course, I don't think that functionality is tested anywhere it's UI only
 62 2016-09-29 09:33:17	0|sipa|i wonder if we should add some random memory/cpu intensive task occasionally during block validation (so it doesn't consume more than 1% cpu rate or so)
 63 2016-09-29 09:33:17	0|wumpus|ok, seems the behaviour was correct. After restart the transaction is no longer in the filter, so I do sendrawtransaction again, it sends out an inv to every node.
 64 2016-09-29 09:33:29	0|sipa|and if that fails, tell the user their hardware is too unreliable
 65 2016-09-29 09:33:47	0|wumpus|yes, some games have that, it's not a bad idea
 66 2016-09-29 09:33:56	0|wumpus|it allows distinguishing bugs from hw failures
 67 2016-09-29 09:34:58	0|wumpus|we have the same problem, 'support' is overflowed with issues that are probably hw failures but we can't be sure so it wastes a lot of time
 68 2016-09-29 09:38:42	0|paveljanik|We could ask users for the result of bitcoind -sanitychecks or something if we suspect HW issue...
 69 2016-09-29 09:39:14	0|sipa|paveljanik: the problem is that such a sanitycheck would need to run for several hours or so to be reliable
 70 2016-09-29 09:40:03	0|paveljanik|sure, but 90% rule...
 71 2016-09-29 09:40:46	0|sipa|if a failure is detectable within minutes, it also means their block validation like fails within minutes
 72 2016-09-29 09:40:51	0|sipa|and many other things
 73 2016-09-29 09:41:03	0|sipa|most random failure i hear about happen after several hours of sync
 74 2016-09-29 09:42:16	0|wumpus|the sanity check definitely needs to run automatically and periodically for it to be useful
 75 2016-09-29 09:42:26	0|wumpus|because otherwise it won't run in the right conditinos
 76 2016-09-29 09:43:19	0|wumpus|we should start selling a hardware quality certificates: has synced the bitcoin block chain succesfully
 77 2016-09-29 09:43:56	0|wumpus|if it can do that, it won't crash on games either. Well. Maybe after GPU secp256k1 verification is implemented ;)
 78 2016-09-29 09:44:18	0|sipa|maybe we should instead just market secp256k1 asics
 79 2016-09-29 09:45:13	0|wumpus|would be a very interesting project if there's a market for that
 80 2016-09-29 09:46:11	0|wumpus|hm, scrap that. THe market for that is key-cracking :(
 81 2016-09-29 09:50:45	0|wumpus|searching around a bit, apparently, many people *started* on a verilog/vhdl FPGA implementation of secp256k1, but there's no code to be found anywhere
 82 2016-09-29 09:58:40	0|wumpus|I'm pleasantly surprised how well the wireshark dissector for the bitcoin protocol works, can just do "tcp port 18333" then use a display filter of "bitcoin" and gets a running list of bitcoin packets easy to inspect using the tree structure
 83 2016-09-29 10:01:27	0|wumpus|compared to trying to mentally parse debug.log output this is a breeze
 84 2016-09-29 10:02:16	0|waxwing|wumpus: they've had that dissector for years, i remember being pleasantly surprised in like 2013/14
 85 2016-09-29 10:02:30	0|waxwing|not that i need it or anything, but it's cool :)
 86 2016-09-29 10:02:50	0|wumpus|yeah yeah it's probably not new
 87 2016-09-29 10:03:15	0|wumpus|but just in case someone didn't discover it 20 years ago yet, there you go...
 88 2016-09-29 10:03:31	0|waxwing|sorry wasn't trying to be a hipster :)
 89 2016-09-29 10:03:44	0|wumpus|:-)
 90 2016-09-29 10:03:48	0|sipa|damn. i believe this is a blocker for bip151.
 91 2016-09-29 10:04:26	0|wumpus|agree sipa
 92 2016-09-29 10:08:25	0|wumpus|though it is possible to have dissectors with parameters, such as a key, but it's so much less user friendly!
 93 2016-09-29 10:12:29	0|wumpus|a more practical way to go at this, post-bip151, would be add functionality to dump packets to disk after decryption on receive and before encryption on send, it's for debugging anyhow not snooping
 94 2016-09-29 10:14:04	0|waxwing|iirc there are per-dissector settings, for ssl you can enter the session key there etc.
 95 2016-09-29 10:14:26	0|wumpus|yes
 96 2016-09-29 10:14:31	0|waxwing|there's an env variable you can set when running firefox that dumps the keys for you, and you can import it in
 97 2016-09-29 10:14:40	0|sipa|implementing bip151 in wireshark will be fu
 98 2016-09-29 10:14:40	0|waxwing|and chrome i think
 99 2016-09-29 10:14:49	0|wumpus|waxwing: but if you deal with lots of different sessions, or sessions that are created on the fly
100 2016-09-29 10:15:21	0|waxwing|right, which i guess might be of particular import in your case (/me hasn't read bip151 tho)
101 2016-09-29 10:15:44	0|wumpus|it's certainly possible, the only remark I was making was in regard to what was the most practical way :)
102 2016-09-29 10:16:45	0|wumpus|one advantage of doing the decryption in the dissector would be that it could detect crypto problems
103 2016-09-29 10:18:32	0|sipa|i don't know what language wireshark uses... but reimplementing sha256, chacha20 and poly1305 doesn't sound trivial
104 2016-09-29 10:18:48	0|sipa|(unless those are already available as primitives)
105 2016-09-29 10:18:49	0|wumpus|C
106 2016-09-29 10:19:10	0|sipa|oh, the filters too?
107 2016-09-29 10:19:20	0|wumpus|it's also possible to write dissectors in lua, but that's only recommended for one-off projects, as performance is abysmal
108 2016-09-29 10:19:23	0|sipa|i thought those would be in a plugin language like lua or so
109 2016-09-29 10:19:24	0|wumpus|yes, most of them
110 2016-09-29 10:19:32	0|sipa|ah.
111 2016-09-29 10:20:28	0|sipa|the bitcoin dissector is in c currently?
112 2016-09-29 10:20:43	0|wumpus|let me see
113 2016-09-29 10:21:31	0|sipa|i vaguely remember seeing the code for it, years ago
114 2016-09-29 10:22:52	0|wumpus|https://github.com/wireshark/wireshark/blob/master/epan/dissectors/packet-bitcoin.c
115 2016-09-29 10:24:24	0|GitHub156|[13bitcoin] 15MarcoFalke opened pull request #8834: [qa] blockstore: Switch to dumb dbm (06master...06Mf1610-qaBlockstoreDumb) 02https://github.com/bitcoin/bitcoin/pull/8834
116 2016-09-29 10:24:52	0|wumpus|apparently it's not yet updated for 0.12.0+, no handler for sendheaders, feefilter etc
117 2016-09-29 10:27:30	0|mmeijeri|The same environment variable (SSLKEYLOGFILE) also works for Chrome.
118 2016-09-29 10:29:18	0|wumpus|oh it's a dynamic file? that's a good idea
119 2016-09-29 10:29:20	0|waxwing|mmeijeri: it's coming back to me, i have a feeling that it handles multiple sessions transparently, by just checking which key material works for which session. not that this matters, sorry for OT :)
120 2016-09-29 10:30:32	0|waxwing|ah yes that was it, i think it stores the premaster secrets for all sessions, then in each handshake it tries them out until it finds what works. something like that.
121 2016-09-29 11:10:28	0|GitHub26|13bitcoin/06master 14a0f8482 15Suhas Daftuar: [qa] Split up slow RPC calls to avoid pruning test timeouts
122 2016-09-29 11:10:28	0|GitHub26|[13bitcoin] 15MarcoFalke pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/8ca69a2a88a7...cc9e8aca5f95
123 2016-09-29 11:10:29	0|GitHub26|13bitcoin/06master 14cc9e8ac 15MarcoFalke: Merge #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts...
124 2016-09-29 11:10:48	0|GitHub122|[13bitcoin] 15MarcoFalke closed pull request #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts (06master...06fix-pruning-timeout) 02https://github.com/bitcoin/bitcoin/pull/8827
125 2016-09-29 11:19:44	0|luke-jr|any reason for me to NOT switch to 64-bit after I sleep?
126 2016-09-29 11:22:46	0|phantomcircuit|luke-jr: something about  #8828 results in a null pointer dereference
127 2016-09-29 11:22:58	0|phantomcircuit|afaict i just moved code around basically
128 2016-09-29 11:23:04	0|phantomcircuit|you wrote the original method
129 2016-09-29 11:23:07	0|phantomcircuit|can you take a look?
130 2016-09-29 11:23:54	0|luke-jr|phantomcircuit: too sleepy now, can you ask me after I finish switching into 64-bit (probably tomorrow)? :x
131 2016-09-29 11:24:10	0|phantomcircuit|luke-jr: whoa now
132 2016-09-29 11:24:13	0|phantomcircuit|you're moving to 64bit?
133 2016-09-29 11:24:23	0|luke-jr|phantomcircuit: unless someone stops me before I wake up
134 2016-09-29 11:24:56	0|phantomcircuit|also the line gdb thinks it's segfaulting on makes no sense
135 2016-09-29 11:25:01	0|luke-jr|?
136 2016-09-29 11:25:02	0|phantomcircuit|nOrderPosNext = 0;
137 2016-09-29 11:25:10	0|luke-jr|so this is NULL? O.o
138 2016-09-29 11:25:47	0|luke-jr|+    int64_t& nOrderPosNext = nOrderPosNext;
139 2016-09-29 11:25:48	0|luke-jr|wtf is this
140 2016-09-29 11:26:13	0|luke-jr|bet that's your problem, it makes no sense
141 2016-09-29 11:26:16	0|luke-jr|prob undefined behaviour
142 2016-09-29 11:26:35	0|luke-jr|but I'm half asleep, so who knows
143 2016-09-29 11:55:05	0|luke-jr|oh well, hope that helped, otherwise ping me tomorrow I guess :x
144 2016-09-29 11:55:06	0|luke-jr|night
145 2016-09-29 12:01:36	0|phantomcircuit|luke-jr: that's weird but apparently correct
146 2016-09-29 12:10:17	0|midnightmagic|that's a weird scoping problem
147 2016-09-29 12:10:26	0|midnightmagic|lol
148 2016-09-29 13:00:43	0|GitHub52|13bitcoin/06master 1464d9507 15Pavel Janík: [WIP] Remove unused statement in serialization
149 2016-09-29 13:00:43	0|GitHub52|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/cc9e8aca5f95...9b94cca41f3e
150 2016-09-29 13:00:44	0|GitHub52|13bitcoin/06master 149b94cca 15Wladimir J. van der Laan: Merge #8658: Remove unused statements in serialization...
151 2016-09-29 13:00:53	0|GitHub41|[13bitcoin] 15laanwj closed pull request #8658: Remove unused statements in serialization (06master...0620160902_nVersion_serialization_cleanup) 02https://github.com/bitcoin/bitcoin/pull/8658
152 2016-09-29 13:02:25	0|kanzure|wumpus: tcpdump if you want to generate .pcap files without wireshark's gui
153 2016-09-29 13:05:44	0|GitHub166|[13bitcoin] 15jgarzik closed pull request #6451: BIP 102: Increase block size limit to 2MB (06master...062015_2mb_blocksize) 02https://github.com/bitcoin/bitcoin/pull/6451
154 2016-09-29 13:36:14	0|GitHub180|[13bitcoin] 15MarcoFalke opened pull request #8835: [qa] nulldummy.py: Don't run unused code (06master...06Mf1610-qaNulldummyUnused) 02https://github.com/bitcoin/bitcoin/pull/8835
155 2016-09-29 14:22:25	0|GitHub8|[13bitcoin] 15jnewbery opened pull request #8836: bitcoin-util-test.py should fail if the output file is empty (06master...06bitcoin-tx-no-empty-outputs) 02https://github.com/bitcoin/bitcoin/pull/8836
156 2016-09-29 14:27:05	0|GitHub100|[13bitcoin] 15jnewbery opened pull request #8837: allow bitcoin-tx to parse partial transactions (06master...06bitcoin-tx-partial-transactions) 02https://github.com/bitcoin/bitcoin/pull/8837
157 2016-09-29 14:34:24	0|GitHub16|[13bitcoin] 15jnewbery opened pull request #8838: Only log block size if block size is being accounted (06master...06dont_log_size) 02https://github.com/bitcoin/bitcoin/pull/8838
158 2016-09-29 14:46:15	0|GitHub177|13bitcoin/06master 14fa156c6 15MarcoFalke: [qa] nulldummy: Don't run unused code
159 2016-09-29 14:46:15	0|GitHub177|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/9b94cca41f3e...2dd57e4f9f58
160 2016-09-29 14:46:16	0|GitHub177|13bitcoin/06master 142dd57e4 15Wladimir J. van der Laan: Merge #8835: [qa] nulldummy.py: Don't run unused code...
161 2016-09-29 14:46:30	0|GitHub194|[13bitcoin] 15laanwj closed pull request #8835: [qa] nulldummy.py: Don't run unused code (06master...06Mf1610-qaNulldummyUnused) 02https://github.com/bitcoin/bitcoin/pull/8835
162 2016-09-29 15:04:34	0|GitHub44|[13bitcoin] 15laanwj opened pull request #8839: test: Avoid ConnectionResetErrors during RPC tests (06master...062016_09_freebsd_rpctest_fix) 02https://github.com/bitcoin/bitcoin/pull/8839
163 2016-09-29 15:08:54	0|GitHub42|13bitcoin/06master 1416f8823 15fanquake: [depends] Boost 1.61.0
164 2016-09-29 15:08:54	0|GitHub42|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/2dd57e4f9f58...c84181665f34
165 2016-09-29 15:08:55	0|GitHub42|13bitcoin/06master 14c841816 15Wladimir J. van der Laan: Merge #8819: [depends] Boost 1.61.0...
166 2016-09-29 15:09:08	0|GitHub98|[13bitcoin] 15laanwj closed pull request #8819: [depends] Boost 1.61.0 (06master...06depends-boost-1-61-0) 02https://github.com/bitcoin/bitcoin/pull/8819
167 2016-09-29 15:23:48	0|GitHub53|13bitcoin/06master 14fa9cd25 15MarcoFalke: [qa] blockstore: Switch to dumb dbm
168 2016-09-29 15:23:48	0|GitHub53|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/c84181665f34...c9d7b0de2fc9
169 2016-09-29 15:23:49	0|GitHub53|13bitcoin/06master 14c9d7b0d 15Wladimir J. van der Laan: Merge #8834: [qa] blockstore: Switch to dumb dbm...
170 2016-09-29 15:23:59	0|GitHub36|[13bitcoin] 15laanwj closed pull request #8834: [qa] blockstore: Switch to dumb dbm (06master...06Mf1610-qaBlockstoreDumb) 02https://github.com/bitcoin/bitcoin/pull/8834
171 2016-09-29 15:27:49	0|GitHub17|13bitcoin/06master 147e5fd71 15Pavel Janík: Do not include env_win.cc on non-Windows systems
172 2016-09-29 15:27:49	0|GitHub17|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/c9d7b0de2fc9...f560d9564f74
173 2016-09-29 15:27:50	0|GitHub17|13bitcoin/06master 14f560d95 15Wladimir J. van der Laan: Merge #8826: Do not include env_win.cc on non-Windows systems...
174 2016-09-29 15:28:00	0|GitHub75|[13bitcoin] 15laanwj closed pull request #8826: Do not include env_win.cc on non-Windows systems (06master...0620160928_leveldb_no_win) 02https://github.com/bitcoin/bitcoin/pull/8826
175 2016-09-29 16:05:25	0|GitHub51|[13bitcoin] 15laanwj opened pull request #8840: test: Explicitly set encoding to utf8 when opening text files (06master...062016_09_textfiles_locale) 02https://github.com/bitcoin/bitcoin/pull/8840
176 2016-09-29 16:23:13	0|GitHub57|[13bitcoin] 15jl2012 opened pull request #8841: [qa] fix nulldummy test (06master...06nulldummytest) 02https://github.com/bitcoin/bitcoin/pull/8841
177 2016-09-29 17:16:21	0|cfields_|wumpus: no problem
178 2016-09-29 17:17:05	0|cfields_|jonasschnelli: i had a look at your circular dep issue. No luck here. The (hack) solution is to use grouping libs, but I'd really rather not go down that road. I've started untangling dependencies instead.
179 2016-09-29 17:25:56	0|wumpus|which dependencies are the problem there? the circular dependency between libbitcoin_server and libbitcoin_wallet?
180 2016-09-29 17:26:46	0|wumpus|that would "just" require solving https://github.com/bitcoin/bitcoin/issues/7965
181 2016-09-29 17:26:53	0|arubi|jnewbery, :)  I was just about to whine about 'if (!DecodeHexTx(txDecodeTmp, strHexTx)'
182 2016-09-29 17:26:58	0|arubi|thanks!
183 2016-09-29 17:57:33	0|arubi|too bad, I wanted to ask him about it here
184 2016-09-29 18:20:37	0|jonasschnelli|wumpus, cfields_: Yes. I think we need to complete 7965 first
185 2016-09-29 18:37:36	0|jl2012|Travis test failed for #8841 with an unrelated test
186 2016-09-29 18:37:45	0|jl2012|https://www.irccloud.com/pastebin/pWEgvccJ/
187 2016-09-29 18:38:14	0|jl2012|https://travis-ci.org/bitcoin/bitcoin/jobs/163788152
188 2016-09-29 18:40:22	0|luke-jr|phantomcircuit: correct for what?
189 2016-09-29 18:40:38	0|MarcoFalke|jl2012: We should probably create an issue for those failures
190 2016-09-29 18:44:19	0|MarcoFalke|and a minor nit: I think the if(msg) is redundant. (The empty string already evaluates to false in python)
191 2016-09-29 18:47:47	0|jl2012|indeed
192 2016-09-29 19:00:16	0|jonasschnelli|bingbong
193 2016-09-29 19:00:35	0|luke-jr|hi
194 2016-09-29 19:00:42	0|sipa|ovatobat
195 2016-09-29 19:00:45	0|CodeShark|yo
196 2016-09-29 19:01:20	0|lightningbot|Meeting started Thu Sep 29 19:01:19 2016 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
197 2016-09-29 19:01:20	0|lightningbot|Useful Commands: #action #agreed #help #info #idea #link #topic.
198 2016-09-29 19:01:20	0|wumpus|#startmeeting
199 2016-09-29 19:01:26	0|BakSAj|hi
200 2016-09-29 19:01:37	0|CodeShark|meeting time!
201 2016-09-29 19:01:47	0|MarcoFalke|meeting!
202 2016-09-29 19:01:57	0|wumpus|#bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
203 2016-09-29 19:02:04	0|MarcoFalke|(oh already started)
204 2016-09-29 19:02:07	0|btcdrak|here
205 2016-09-29 19:02:08	0|cfields_|hi
206 2016-09-29 19:02:12	0|kanzure|hi
207 2016-09-29 19:02:20	0|wumpus|any proposed topics?
208 2016-09-29 19:02:24	0|jonasschnelli|topic proposal: pruning and blockrelay
209 2016-09-29 19:02:57	0|petertodd|hi
210 2016-09-29 19:02:59	0|sipa|policy against uncompressed keys or not
211 2016-09-29 19:03:11	0|wumpus|#topic pruning and blockrelay
212 2016-09-29 19:03:38	0|jonasschnelli|I think we should add a service flag for block relay with a min-height
213 2016-09-29 19:03:52	0|jonasschnelli|NODE_PRUNENETWORK or something
214 2016-09-29 19:03:58	0|sipa|there have been multiple ideas around that
215 2016-09-29 19:04:35	0|petertodd|IMO whatever we do, we should recognise that w/ segwit's larger blocks we can expect a lot of full nodes to run out of disk space quite soon
216 2016-09-29 19:04:36	0|jonasschnelli|I guess people slowly start to prune the blockchain to a max of 80GB or similar... but I guess not everyone is aware of the fact that you don't relay then
217 2016-09-29 19:04:36	0|sipa|the easiest is just to add a flag that says you relay valid blocks and transactions, but not historical blocks more than a few deep
218 2016-09-29 19:04:36	0|wumpus|it would be nice to support more than one range, e.g. also archive nodes that host part of the old blocks
219 2016-09-29 19:04:50	0|sipa|it becomes harder when you want multiple ranger
220 2016-09-29 19:04:58	0|petertodd|do we have a reason for more than one range?
221 2016-09-29 19:05:00	0|jonasschnelli|We could introduce another message type... blockrange  or so
222 2016-09-29 19:05:02	0|sipa|it becomes even harder when you want to support sharding in an efficient way
223 2016-09-29 19:05:08	0|wumpus|I'm not sure why itb ecomes hard, just add a query message that returns what ranges are supported
224 2016-09-29 19:05:11	0|petertodd|sipa: what do you mean by sharding exactly?
225 2016-09-29 19:05:26	0|sipa|petertodd: you'd configure your node to maintain a certain % of blocks
226 2016-09-29 19:05:45	0|jonasschnelli|wumpus: query, yes, why not, or just inform like we do with sendheaders
227 2016-09-29 19:05:47	0|petertodd|sipa: see, given that the bitcoin protocol can't be safely sharded right now, I think we can safely say that we don't need to support sharding in block relay yet
228 2016-09-29 19:05:55	0|petertodd|sipa: doing so might even be dangerous if people start using it
229 2016-09-29 19:06:04	0|sipa|petertodd: not in block relay
230 2016-09-29 19:06:08	0|sipa|petertodd: for block archival
231 2016-09-29 19:06:17	0|petertodd|sipa: but why shard vs. keep ranges?
232 2016-09-29 19:06:24	0|petertodd|sipa: (ranges of full blocks)
233 2016-09-29 19:06:25	0|luke-jr|BitTorrent already does this. Surely we can learn from that?
234 2016-09-29 19:06:38	0|petertodd|luke-jr: I don't think so - bittorrent is a very different problem than bitcoin
235 2016-09-29 19:06:39	0|wumpus|this is for letting other peers know what ranges of blocks are hosted, I don't think this should affect releay
236 2016-09-29 19:06:42	0|sipa|so, i've been running statistics on what block depths are being requested from nodes
237 2016-09-29 19:06:47	0|luke-jr|petertodd: learn from it, not use it directly
238 2016-09-29 19:07:01	0|luke-jr|petertodd: BitTorrent's problem isn't very different from IBD
239 2016-09-29 19:07:08	0|jonasschnelli|sipa: interesting.. do you have the stats public available somewhere
240 2016-09-29 19:07:16	0|jonasschnelli|I wanted to do this a long time
241 2016-09-29 19:07:20	0|petertodd|luke-jr: so, the thing is bittorrent has the problem of a diverse set of files, we just don't have that problem and can optimise differently because everyone needs access t othe same set of data
242 2016-09-29 19:08:03	0|sipa|there are something like 4 meaningful 'ranges' 1) the top 2 blocks (just relay) 2) up to ~2500 blocks deep... requested very often 3) up to ~10000 deep... requested a few times more than the next range 4) the rest
243 2016-09-29 19:08:15	0|wumpus|otoh bittorrent has a fixed block size :)
244 2016-09-29 19:08:23	0|sipa|wumpus: so do we *ducks*
245 2016-09-29 19:08:38	0|petertodd|sipa: that probably corresponds to how long people leave their nodes offline :)
246 2016-09-29 19:08:51	0|btcdrak|inb4 Bittorrent XT
247 2016-09-29 19:09:04	0|sipa|jonasschnelli: they're not available, and the ranges i gave above are just from me quickly glancing over the result
248 2016-09-29 19:09:05	0|petertodd|btcdrak: I use Bittorrent Unlimited myself
249 2016-09-29 19:09:06	0|jonasschnelli|What about fingerprinting issued in conjunction with available ranges?
250 2016-09-29 19:09:16	0|jonasschnelli|*issues
251 2016-09-29 19:09:21	0|petertodd|jonasschnelli: make them powers of two?
252 2016-09-29 19:09:34	0|sipa|well 4 ranges can be done with 2 service bit flags
253 2016-09-29 19:09:45	0|sipa|gmaxwell: you've worked on these ideas before, comments?
254 2016-09-29 19:09:48	0|jonasschnelli|But would that work with the flexible pruning option based on MB?
255 2016-09-29 19:10:06	0|petertodd|jonasschnelli: sure, just find the biggest range less than the pruning amount
256 2016-09-29 19:10:12	0|sipa|jonasschnelli: you'd change your service bits on the fly
257 2016-09-29 19:10:20	0|wumpus|why would the ranges need to be in the flags?
258 2016-09-29 19:10:30	0|gmaxwell|sorry, I missed that the meeting started.
259 2016-09-29 19:10:32	0|jonasschnelli|Yes. Why? Better add an explicit message for the range
260 2016-09-29 19:10:34	0|sipa|how would you otherwise discover what nodes to connect to?
261 2016-09-29 19:10:45	0|sipa|just randomly try?
262 2016-09-29 19:10:46	0|petertodd|jonasschnelli: oh right, you mean if MB != blocks... sorry.
263 2016-09-29 19:10:50	0|wumpus|I think you'll need a service flag to show support for the protocol, but not what ranges you have
264 2016-09-29 19:10:52	0|jonasschnelli|query or inform the other node if proto-ver > NODE_PRUNENETWORK
265 2016-09-29 19:11:00	0|wumpus|well that can be negotiated later, like bittorrent does I guess
266 2016-09-29 19:11:08	0|sipa|wumpus: well you do want addr messages to contain this information
267 2016-09-29 19:11:18	0|petertodd|sipa: so the nice thing about bitcoin, is just randomly try will probably work fairly often due to the low number of ranges out there
268 2016-09-29 19:11:18	0|wumpus|I doubt bitcoin has 'service flags' in its tracker what blocks nodes have
269 2016-09-29 19:11:26	0|wumpus|as that changes all the time anyhow
270 2016-09-29 19:11:30	0|gmaxwell|I was strongly of the view that we needed to signal at least two ranges. Sipa's latest measurements make me think at least three are needed.
271 2016-09-29 19:11:32	0|wumpus|s/bitcoin/bittorrent/
272 2016-09-29 19:11:37	0|jonasschnelli|I think informing other nodes ranges over addr is another thing...
273 2016-09-29 19:11:46	0|jonasschnelli|A first step would be a information after connect
274 2016-09-29 19:11:55	0|wumpus|yes, addr is another thing
275 2016-09-29 19:12:22	0|gmaxwell|I think ranges in service bits are no big deal, the harder question is what to do about the history. having nodes with 150GB of history in order to serve the last range is not very viable.
276 2016-09-29 19:12:24	0|wumpus|could be done later if an efficient way is needed to *locate* peers with certain ranges
277 2016-09-29 19:12:32	0|wumpus|but that seems premature optimization
278 2016-09-29 19:12:36	0|gmaxwell|We will need to redo addr sometime relatively soon in any case, as our messages are not compatible with HS-NG.
279 2016-09-29 19:12:53	0|petertodd|gmaxwell: oh, you mean Tor's new hidden services standard right?
280 2016-09-29 19:12:57	0|gmaxwell|petertodd: yes.
281 2016-09-29 19:13:02	0|gmaxwell|(also I2P though thats not new)
282 2016-09-29 19:13:18	0|wumpus|I think the number of ranges should be variable
283 2016-09-29 19:13:24	0|wumpus|redesigning addr is a different topic
284 2016-09-29 19:13:35	0|wumpus|also necessary, but again, doesn't need to be on one heap
285 2016-09-29 19:13:45	0|gmaxwell|wumpus: when I'm saying ranges I am specifically referring to the top-N zomes.
286 2016-09-29 19:14:14	0|petertodd|well, so if we add service bits for recent history ranges, that should be possible to implement as a separate feature to archival history ranges, and it'd be a big first step
287 2016-09-29 19:14:23	0|wumpus|I think it should be possible to, say, only host the first 20GB of blocks
288 2016-09-29 19:14:37	0|jonasschnelli|historic only nodes
289 2016-09-29 19:14:38	0|petertodd|I don't think it's likely we'll see the two different features collide, so maybe implement recent history ranges first
290 2016-09-29 19:14:38	0|wumpus|I don't see why it should be restricted to only recent history
291 2016-09-29 19:14:55	0|wumpus|or I mean first 20GB + last 144 blocks
292 2016-09-29 19:15:07	0|gmaxwell|For history storage, I was previously working on a proposal where nodes could signal a small (32 bit) seed and a size and from that everyone would know what parts of the history they would store.  I was so far unable to unify two different schemes, one which was computationally efficient to figure out who had what, and one which never required a peer to fetch a block it had previously deleted.
293 2016-09-29 19:15:15	0|sipa|so very quick breakdown: out of 7M requested blocks, 100k were for the tip, range 2-2500 has around 200-2000 requests per block, and from 10000 to genesis deep there are around 20 per block
294 2016-09-29 19:16:19	0|gmaxwell|I think for now we should not worry about the old history part and only worry about Top-n vs everything, as that fits into the pruning we already have and can be accomplished purely with service bits.
295 2016-09-29 19:16:22	0|wumpus|the bittorrent problem is different in that there the goal of each node is to have everything
296 2016-09-29 19:16:26	0|petertodd|so a social consideration here, is we can think in terms of recent history as "if there's a flaw, how much would we ever reorg w/o just saying bitcoin has failed?"
297 2016-09-29 19:16:44	0|gmaxwell|petertodd: thats partly why we have the 288 block maximum amount of pruning.
298 2016-09-29 19:17:14	0|petertodd|gmaxwell: indeed, and that's only two days...
299 2016-09-29 19:17:37	0|jonasschnelli|Using multiple service bits for 4 ranges seems to be a hackish-design IMO
300 2016-09-29 19:17:49	0|gmaxwell|at 100 blocks any reorg will _necessarily_ cause unrecoverable losses. So 288 basically gives a day plus an extra day for overhead.
301 2016-09-29 19:17:50	0|petertodd|there's also a natural time criteria from how the difficulty adjustments reduce your resistance to 51% attack - if your node is offline longer, the minimum attacker size to fool you goes down
302 2016-09-29 19:17:53	0|sipa|strangely enough: i see much more requests around 1000 deep than around 100 deep
303 2016-09-29 19:18:04	0|gmaxwell|jonasschnelli: I don't see anything hackish.
304 2016-09-29 19:18:05	0|wumpus|jonasschnelli: I also think it's a strange use of service bits
305 2016-09-29 19:18:07	0|jonasschnelli|I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
306 2016-09-29 19:18:13	0|petertodd|sipa: probably because people don't turn their nodes on and off every day
307 2016-09-29 19:18:26	0|gmaxwell|sipa: you probably want to filter out the bitnodes spider, as I believe it requests a block to check the node is working.
308 2016-09-29 19:18:36	0|sipa|gmaxwell: ah.
309 2016-09-29 19:18:53	0|gmaxwell|petertodd: someone who hasn't turned their node on will request all of 0 to -1000. so it will not make 1000 greater.
310 2016-09-29 19:19:07	0|gmaxwell|jonasschnelli: NAK.
311 2016-09-29 19:19:08	0|petertodd|gmaxwell: oh! I didn't know we did that
312 2016-09-29 19:19:09	0|sipa|i'm a bit surprised people think there is no need to have the available block ranges indicated in addr messages
313 2016-09-29 19:19:30	0|sipa|(whether through service bits, or some extension)
314 2016-09-29 19:19:31	0|jonasschnelli|I think there is a need... but it could be a second step
315 2016-09-29 19:19:38	0|wumpus|jonasschnelli: appending to version should be unnecessary, that's also a hack :)
316 2016-09-29 19:19:50	0|sipa|jonasschnelli: if it's a second step, we need to extend addr, and the whole management of addresses
317 2016-09-29 19:19:52	0|jonasschnelli|Okay. Agree. What about a new message type?
318 2016-09-29 19:19:55	0|jonasschnelli|blockrange
319 2016-09-29 19:19:58	0|sipa|jonasschnelli: you don't understand.
320 2016-09-29 19:20:18	0|gmaxwell|jonasschnelli: look at pieter's request figures, if nodes are effectively forced to go to peers that have everything whenever they connect becuase if they don't know they'll be able to fetch any blocks at all, then it will put lots more load on them.. causing people to stop offering blocks... causing more pressure on what remains.
321 2016-09-29 19:20:56	0|sipa|jonasschnelli: the point of having it in service bits is so nodes can find peers that have the range they need
322 2016-09-29 19:20:58	0|wumpus|but addr information gets old really fast
323 2016-09-29 19:21:17	0|sipa|wumpus: much less so with feeler connections
324 2016-09-29 19:21:26	0|wumpus|nodes may dynamically change what blocks they have, so there will always be cases of nodes connecting and realizing they have nothing to offer each other
325 2016-09-29 19:21:27	0|jonasschnelli|Okay. I see the point.
326 2016-09-29 19:21:30	0|sipa|(presumably, i don't have numbers)
327 2016-09-29 19:21:54	0|wumpus|just like currently nodes will try to connect into black holes that no longer host a node
328 2016-09-29 19:22:25	0|petertodd|so another interesting thing here is that ranges are queried linearly - you download blocks in a roughly linear fashion - so we could take advantage of that by making sure that nodes with one range keep track of nodes with adjacent ranges
329 2016-09-29 19:22:29	0|wumpus|sipa: sure, feeler connections make it somewhat better
330 2016-09-29 19:22:35	0|gmaxwell|wumpus: yes, sometimes the data is wrong. But there is a big difference between having 80% of the nodes on the network giving you no idea if they'll be useful at all until after you connect, vs a suggestion that might sometimes be wrong.
331 2016-09-29 19:22:41	0|wumpus|but I don't think addr is a very up-to-date information source
332 2016-09-29 19:22:47	0|petertodd|thus, as you sync the first time, ask nodes with the range you're syncing at this moment for the next range you need
333 2016-09-29 19:23:08	0|luke-jr|wumpus: if ranges are deterministic, they don't need to be up to date
334 2016-09-29 19:23:16	0|sipa|petertodd: yes, any sharding plan wouldn't randomly distribute the kept blocks, but keep randomly distributed ranges
335 2016-09-29 19:23:32	0|gmaxwell|wumpus: I don't know if you realize that sipa and I are not thinking in terms of absolute ranges here. but nodes saying "I keep the last 288" or "I keep the last 2016" or "I have all of history".
336 2016-09-29 19:23:38	0|wumpus|gmaxwell: but indeed this is a different problem from the bittorrent problem where everyone's goal is to have everything
337 2016-09-29 19:23:56	0|sipa|gmaxwell: well that's sharding... maybe that is something to postpone for later
338 2016-09-29 19:24:01	0|petertodd|sipa: sure, I'm more talking about how the linearity affects the network p2p design - prefentially peering with peers with the adjacent range may even be a reasonable design
339 2016-09-29 19:24:03	0|luke-jr|wumpus: eh, everyone needs to get everything
340 2016-09-29 19:24:04	0|gmaxwell|wumpus: and I wouldn't expect that data to go out of date fast.. pretty much only when nodes go up and down.
341 2016-09-29 19:24:04	0|wumpus|there, nodes can just connect randomly and have a high change the other nodes has something to offer them
342 2016-09-29 19:24:05	0|sipa|oh, nvm, i'm misreading
343 2016-09-29 19:24:21	0|wumpus|luke-jr: only initially
344 2016-09-29 19:24:29	0|luke-jr|oh, I see the distinction
345 2016-09-29 19:24:41	0|wumpus|luke-jr: bittorrent nodes don't throw away blocks, generally
346 2016-09-29 19:24:53	0|luke-jr|f(best-height, seed-in-addr) -> ranges
347 2016-09-29 19:25:13	0|gmaxwell|for the spreading the history around, as mentioned I came up with concrete schemes (based on consistent hashes) that have nice properties.
348 2016-09-29 19:25:30	0|sipa|i wonder whether we need to have that in the first go at this
349 2016-09-29 19:26:04	0|jonasschnelli|I think a first simple solution that allow to extend it further would be appriciated.
350 2016-09-29 19:26:10	0|sipa|even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
351 2016-09-29 19:26:15	0|wumpus|making the ranges deterministic makes some sense, on the other hand, it does restrict the flexibilty of nodes to choose what ranges they host, it means everything has to be got right in first try
352 2016-09-29 19:26:23	0|gmaxwell|sipa: thats what I am saying.
353 2016-09-29 19:26:27	0|jonasschnelli|sipa: agree
354 2016-09-29 19:26:30	0|gmaxwell|I do not think we can do better immediately anyways.
355 2016-09-29 19:26:46	0|sipa|21:18:07 < jonasschnelli> I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
356 2016-09-29 19:26:47	0|gmaxwell|sipa: though your latest figures suggest that the 2016 depth is important too.
357 2016-09-29 19:26:49	0|sipa|21:19:07 < gmaxwell> jonasschnelli: NAK.
358 2016-09-29 19:27:10	0|petertodd|if nodes attempt to maintain a few connections to peers that have the next range after they have, maybe it doesn't matter exactly what the ranges actually are? any given node would have a few connections to the next range, and anyone syncing from them could ask for those connections
359 2016-09-29 19:27:15	0|gmaxwell|sipa: my understanding of jonasschnelli comment was there should be a bit that says "I relay blocks but don't have history" I am NAK on that.
360 2016-09-29 19:27:36	0|wumpus|as there is no scope for later optimization, because all nodes have to agree what ranges are implied
361 2016-09-29 19:27:40	0|jonasschnelli|We could add a service bit that says "I relay only the last 288 blocks"
362 2016-09-29 19:27:51	0|wumpus|jannes: yes that would be the initial idea
363 2016-09-29 19:27:54	0|wumpus|jonasschnelli*
364 2016-09-29 19:28:09	0|sipa|gmaxwell: how is that different from what i suggested?
365 2016-09-29 19:28:12	0|sipa|21:26:10 < sipa> even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
366 2016-09-29 19:28:22	0|jonasschnelli|I think my initial idea with the general pruning sevice bit and a new message type is to complex and inflexible
367 2016-09-29 19:28:28	0|gmaxwell|jonasschnelli: yes, that would be better, though pieter's data suggests that there are a LOT of requests at 1000. I think if I had that data I would have been suggesting the maximum pruning should be 2016, and then had the bit at that dep.
368 2016-09-29 19:28:51	0|gmaxwell|sipa: the ability to relay blocks at depth -10.
369 2016-09-29 19:29:05	0|sipa|gmaxwell: less than 2% of blocks requested from my node are at the tip
370 2016-09-29 19:29:22	0|sipa|(but the tip is still 100x more frequent than any other individual depth)
371 2016-09-29 19:29:50	0|sipa|gmaxwell: "a service bit to indicate pruned blockchain" implies you can serve 288 deep :)
372 2016-09-29 19:30:08	0|petertodd|gmaxwell: re: maximum pruning depth, it's reasonable for that to be a similar % of the total data that storing the UTXO set takes - if you have 10GB of UTXO, 2GB of block data isn't a big change
373 2016-09-29 19:30:11	0|wumpus|yes, you could define it as that
374 2016-09-29 19:30:15	0|gmaxwell|I don't think there is any remaining disagreement on using bit(s) to signal I have a top-n.  But I have some doubt on N. it needs to capture the largest amount of the block realy bandwidth without being unduely pruning incompatible.
375 2016-09-29 19:30:37	0|wumpus|288 is the minimum pruning amount in bitcoin core already so it'd be a valid choice
376 2016-09-29 19:30:41	0|morcos|as a first pass, i wonder if you preferentially downloaded from pruned peers whenever you were behind by less than 288 blocks, that would take enough load of peers serving full history?
377 2016-09-29 19:30:51	0|gmaxwell|morcos: absolutely.
378 2016-09-29 19:31:01	0|jonasschnelli|Good idea
379 2016-09-29 19:31:09	0|wumpus|yes, that would make sense
380 2016-09-29 19:31:24	0|gmaxwell|unfortunately, sipa's data suggests that 288 sheds less traffic than measurements years ago suggested.
381 2016-09-29 19:31:44	0|sipa|maybe i should compute statistics in bytes rather than blocks
382 2016-09-29 19:31:45	0|morcos|gmaxwell: it wasn't clear to me what the integral from 1 to 288 was compared to 288 to inf
383 2016-09-29 19:31:46	0|wumpus|well it is a compromise
384 2016-09-29 19:31:54	0|wumpus|putting the threshold higher makes some peers completely useless
385 2016-09-29 19:31:57	0|sipa|to see what percentage of bandwidth is needed in 1-288
386 2016-09-29 19:31:58	0|wumpus|which reduces morcos 's argument
387 2016-09-29 19:32:04	0|jonasschnelli|Yes. I guess you convinced me to use two service bits then. -288 and -2016
388 2016-09-29 19:32:16	0|gmaxwell|which is why it might be useful to use two bits and be able to signal 1-288, 1-2016... and perhaps start encouraging people to not prune shorter than 2016.
389 2016-09-29 19:32:37	0|sipa|i think we're getting into a design discussion here
390 2016-09-29 19:32:48	0|sipa|my number are very premature and not well analysed
391 2016-09-29 19:32:51	0|wumpus|it'd also be possible to add a 288-flag now, and then consider a 2016 flag later
392 2016-09-29 19:32:54	0|gmaxwell|sipa: indeed, thought that was the input you requested from me.
393 2016-09-29 19:32:58	0|morcos|wumpus: yes, thats what i'm saying
394 2016-09-29 19:33:00	0|gmaxwell|wumpus: yes! indeed.
395 2016-09-29 19:33:03	0|jonasschnelli|Agree with wumpus
396 2016-09-29 19:33:05	0|wumpus|if it turns out to be necessary
397 2016-09-29 19:33:10	0|petertodd|wumpus: ACK
398 2016-09-29 19:33:15	0|sipa|yes, i think just a 1-288 one seems useful
399 2016-09-29 19:33:16	0|jonasschnelli|Start with a simple tip-288 relay, and get some experience
400 2016-09-29 19:33:16	0|wumpus|good :)
401 2016-09-29 19:33:23	0|gmaxwell|wumpus: it looks pretty clearly necessary but no need to do everything at once.
402 2016-09-29 19:33:30	0|petertodd|wumpus: basically advice is, turn your node on at least once every two days
403 2016-09-29 19:33:54	0|wumpus|petertodd: yes
404 2016-09-29 19:33:56	0|gmaxwell|petertodd: we really should have cron mode for the daemon where it just syncs up and shuts off. :P
405 2016-09-29 19:34:06	0|gmaxwell|bitcoind -oneshot
406 2016-09-29 19:34:08	0|gmaxwell|:P
407 2016-09-29 19:34:17	0|petertodd|gmaxwell: heh, that's not a crazy idea - I'd use it on my laptop
408 2016-09-29 19:34:19	0|jonasschnelli|didn't we once had a proposal for the pause option?
409 2016-09-29 19:34:24	0|wumpus|right, there's a flag that quits after reindex, but none that exits after sync
410 2016-09-29 19:34:31	0|wumpus|would be easy to add tho
411 2016-09-29 19:34:42	0|morcos|we could just ask for the utxo set, shoudl we discuss ideas how to do that
412 2016-09-29 19:34:51	0|CodeShark|^ yes :)
413 2016-09-29 19:34:59	0|petertodd|make -oneshot run in the foreground with a progress bar :)
414 2016-09-29 19:34:59	0|wumpus|without utxo commitment that's a no-go
415 2016-09-29 19:35:06	0|morcos|thanks codeshark
416 2016-09-29 19:35:21	0|petertodd|wumpus: +1
417 2016-09-29 19:35:42	0|gmaxwell|morcos: pointless when we were unable to get past the discussion for the security model change to not validate the past history based on proof of work.
418 2016-09-29 19:35:49	0|petertodd|and lets not underestimate how dangerous UTXO commitments can be - I'm very dubious about committing to the (U)TXO set more recently than maybe a month or two
419 2016-09-29 19:35:52	0|CodeShark|would be great to query utxo for quick sync, then go backwards in time fetching blocks to increase security...but yes, this is a design discussion
420 2016-09-29 19:35:53	0|morcos|i was making a joke, sorry
421 2016-09-29 19:36:18	0|CodeShark|alas, quick sync doesn't look feasible in the nearterm
422 2016-09-29 19:36:20	0|wumpus|ok, next topic?
423 2016-09-29 19:36:36	0|gmaxwell|but since that was brought up... Can we talk about removing checkpoints?
424 2016-09-29 19:36:55	0|wumpus|#topic removing checkpoints
425 2016-09-29 19:36:59	0|sipa|what % of transactions are before the last checkpoint
426 2016-09-29 19:37:01	0|sipa|does anyone know?
427 2016-09-29 19:37:04	0|morcos|someone should write up a design proposal for that to be evaluated
428 2016-09-29 19:37:18	0|gmaxwell|Right now they're used for two things, preventing header flooding with low difficulty headers; and skipping signatures in earlier blocks.
429 2016-09-29 19:37:22	0|petertodd|gmaxwell: just removing checkpoints, or assuming sigs are valid if buried deep enough?
430 2016-09-29 19:37:27	0|sipa|gmaxwell: and 3) estimating progress
431 2016-09-29 19:37:46	0|wumpus|keeping something for estimating progress would make sense
432 2016-09-29 19:37:47	0|sipa|i think 1) remains needed and 3) remains useful
433 2016-09-29 19:37:50	0|wumpus|that doesn't need to be checkpoints
434 2016-09-29 19:38:06	0|gmaxwell|because very few percentage of the transactions are below the checkpoint .. since libsecp256k1 (and I expect the checkqueue)-- my point two is basically pointless, and I think it could just be removed
435 2016-09-29 19:38:29	0|gmaxwell|I think on a desktop it only adds 15-20 minutes to the sync.
436 2016-09-29 19:38:29	0|petertodd|gmaxwell: I'd ACK simply removing checkpoints entirely; I'm not happy to see them replaced with another scheme to skip sig checking
437 2016-09-29 19:38:33	0|wumpus|a block-height-to-relative-difficulty map would have much less of a stigma
438 2016-09-29 19:38:46	0|wumpus|eh, verification difficulty that is
439 2016-09-29 19:38:52	0|sipa|gmaxwell: really?
440 2016-09-29 19:38:54	0|gmaxwell|petertodd: I think we could remove CP from reason two without implementing the replcement.
441 2016-09-29 19:39:07	0|gmaxwell|petertodd: morcos is right that needs a design proposal outside of the meeting.
442 2016-09-29 19:39:13	0|sdaftuar|i'm a bit confused about how to think about checkpoints for signature skipping
443 2016-09-29 19:39:23	0|gmaxwell|sipa: I benchmarked before but I'm going off of memory, I could be wildly wrong. I will test again if there is interest.
444 2016-09-29 19:39:53	0|jonasschnelli|Removing checkpoints would slow down (maybe insignificant) a scan in a possible SPV hybrid mode?
445 2016-09-29 19:39:54	0|gmaxwell|For reason (1) the only answer I have is that I think we should proposal a bit to perpetually increase the minimum difficulty from 1 to something else.
446 2016-09-29 19:40:00	0|sdaftuar|for instance the recent ISM change caused us to do less validation for certain blocks in our history (blocks in a softfork between the 75% and 95% thresholds)
447 2016-09-29 19:40:10	0|sipa|jonasschnelli: SPV mode won't validate *anything* at all
448 2016-09-29 19:40:29	0|gmaxwell|(with a checkpoint like bypass of that new rule, for existing blocks that break it) As little as 100,000 would eliminate the header flooding vulenrablity.
449 2016-09-29 19:40:34	0|jonasschnelli|Yes. But assume we would add an SPV hibrid mode in oder to received payment during IBD
450 2016-09-29 19:40:50	0|jonasschnelli|One would need to download 400k headers without a checkpoint at h400k
451 2016-09-29 19:40:53	0|luke-jr|maybe checkpoints should just be disabled by default before complete removal?
452 2016-09-29 19:41:01	0|sipa|jonasschnelli: i think you're confused
453 2016-09-29 19:41:07	0|gmaxwell|for Sipa's (3) reason for 'checkpoints' I don't give a darn, use chicken bones for progress estimation for all I care. :P it's historical accident that checkpoints and progress use the same data structure.
454 2016-09-29 19:41:24	0|morcos|gmaxwell: :) +1
455 2016-09-29 19:41:28	0|wumpus|gmaxwell: yes, my point too
456 2016-09-29 19:41:35	0|sipa|gmaxwell: agree, those could be completely separated
457 2016-09-29 19:41:36	0|petertodd|gmaxwell: ACK checken bones
458 2016-09-29 19:41:37	0|gmaxwell|Might as well fit a cubic spline to the height vs txn count... and store the parameters.
459 2016-09-29 19:41:53	0|wumpus|right
460 2016-09-29 19:42:10	0|petertodd|gmaxwell: heh, if we do that with floating point math that has the advantage that it _can't_ be used for consensus :)
461 2016-09-29 19:42:21	0|gmaxwell|so my proposal, if there is interest, is that I'll measure the performance impact of removing the signature skippingentirely (esp post checkqueue). And if it's not awful, we'll remove.
462 2016-09-29 19:42:37	0|wumpus|+1
463 2016-09-29 19:42:43	0|sipa|gmaxwell: i'm unconvinced
464 2016-09-29 19:42:52	0|wumpus|it doesn't hurt to benchmark
465 2016-09-29 19:43:00	0|gmaxwell|and maybe I'll tender a proposal to up the minimum difficulty, but I'd like to know what people think about that.
466 2016-09-29 19:43:04	0|wumpus|measuring is always better than making assumptions
467 2016-09-29 19:43:11	0|sipa|with a replacement for sig skipping that isn't based on checkpoints we could significantly improve things
468 2016-09-29 19:43:39	0|petertodd|sipa: I don't think such a replacement can exist without changing the security assumptions; I'd *rather* have checkpoints than trusting hashing power for that
469 2016-09-29 19:43:44	0|sipa|the last checkpoint currently is very old for the very reason that we've been planning to replace it
470 2016-09-29 19:43:48	0|gmaxwell|sipa: would you like to help work on a proposal for that? it has been controversial in the past. I'd like to do something good, because otherwise imprudent attempts will be adopted instead.
471 2016-09-29 19:44:15	0|sipa|so it's unfair to use the "the last checkpoint is old" as a given; it's something we've affected indirectly
472 2016-09-29 19:44:19	0|petertodd|sipa: though what checkpoints should do is say "Something big has changed; you can disable checkpoints with --no-checkpoints, but you should find out what this means before doing so."
473 2016-09-29 19:44:29	0|gmaxwell|(for example Bitcoin Classic's current behavior simply looks at block header timestamps and ignores signatures when they're more than 24 hours (*par) old by the local clock. It's easily exploited and makes me sad.
474 2016-09-29 19:44:40	0|sipa|petertodd: it's my opinion that on a timescale of months, it doesn't matter
475 2016-09-29 19:44:53	0|sipa|IF you can guarantee it's actually a timescale of months
476 2016-09-29 19:44:55	0|wumpus|yes that makes me sad too
477 2016-09-29 19:44:58	0|petertodd|sipa: on a timescale of months, checkpoints shouldn't matter either...
478 2016-09-29 19:45:06	0|wumpus|anything based on time seems very brittle
479 2016-09-29 19:45:27	0|sipa|petertodd: look at the current hashrate; what's 3 months worth of chain work at that hashrate
480 2016-09-29 19:45:31	0|petertodd|wumpus: and anything based on work isn't much better if you're running an old client, and mining has advanced significantly
481 2016-09-29 19:45:37	0|jonasschnelli|sipa: I (hope) I'm not confused. If we would add a SPV hybrid mode directly fetch blocks at the tip (in order to received payments), no available checkpoint would result in downloading all headers *losing* maybe 3-4mins before you can start using SPV... minor issue though, I agree
482 2016-09-29 19:45:38	0|petertodd|sipa: that assumes you know what the current hashrate is
483 2016-09-29 19:45:41	0|gmaxwell|wumpus: the prior proposals were based on work,  e.g. skip if the best chain you see dominates the next conflicted chain at that hight by N months of work.
484 2016-09-29 19:45:45	0|Chris_Stewart_5|gmaxwell: How have we solved the problem that checkpoints were originally created for? You have an excerpt in here: https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_5):_Initial_Block_Download#Checkpoints
485 2016-09-29 19:45:45	0|petertodd|sipa: your node might be surrounded by sybils
486 2016-09-29 19:45:53	0|gmaxwell|wumpus: with a 'minimum total work' coded in as part of the release proces.
487 2016-09-29 19:46:01	0|sipa|Chris_Stewart_5: headers first sync
488 2016-09-29 19:46:03	0|sipa|Chris_Stewart_5: 0.10
489 2016-09-29 19:46:05	0|gmaxwell|Chris_Stewart_5: headers first sync.
490 2016-09-29 19:46:09	0|wumpus|gmaxwell: right. Well, at the least it should be measured whether such a change is really worth it.
491 2016-09-29 19:46:17	0|sipa|petertodd: yes, i know...
492 2016-09-29 19:46:22	0|sipa|so, let's measure.
493 2016-09-29 19:46:25	0|sipa|and discuss later
494 2016-09-29 19:46:37	0|gmaxwell|Chris_Stewart_5: and the signature skipping behavior in checkpoints was actually a result of a bug fixed years ago.. mlock being used on all allocations making script validation INSANELY slow.
495 2016-09-29 19:46:42	0|wumpus|so much of the verification overhead is looking up UTXOs
496 2016-09-29 19:46:44	0|gmaxwell|sipa: okay.
497 2016-09-29 19:46:48	0|wumpus|something you'll not avoid
498 2016-09-29 19:47:21	0|gmaxwell|Chris_Stewart_5: but then with chain growth we became dependant on it to keep sync times reasonable. but libsecp256k1 made signature validation >5x faster.
499 2016-09-29 19:47:22	0|wumpus|especially for recent blocks
500 2016-09-29 19:47:37	0|wumpus|if you do any benchmarking please look at the recent blocks, not the first N
501 2016-09-29 19:48:04	0|gmaxwell|wumpus: it's still a major speed up on existing blocks.
502 2016-09-29 19:48:08	0|sipa|in a side node: i've already updated my logging to measure bandwidth vs blockdepth instead of just count.
503 2016-09-29 19:48:11	0|Chris_Stewart_5|So header sync solves the attack of flooding disk space, but not having your entire network hijacked, correct?
504 2016-09-29 19:48:33	0|wumpus|Chris_Stewart_5: huh?
505 2016-09-29 19:48:42	0|wumpus|gmaxwell: sure, could be
506 2016-09-29 19:48:43	0|gmaxwell|Chris_Stewart_5: isolation can be resolved by simply knowing what the total work of the best chain was at release.
507 2016-09-29 19:49:00	0|gmaxwell|Chris_Stewart_5: sorry, this was discussed prior times removing checkpoints had come up, I haven't completely described the background.
508 2016-09-29 19:49:19	0|Chris_Stewart_5|gmaxwell: Thanks for the explanation, i'll keep digging.
509 2016-09-29 19:49:46	0|wumpus|Chris_Stewart_5: ah, you mean being isolated and being fed a wrong chain, sorry I was imaginging some wacky things at having your network hijacked :)
510 2016-09-29 19:50:38	0|wumpus|ok, next topic?
511 2016-09-29 19:50:39	0|gmaxwell|wumpus: just the "you got a faithful bitcoin core download but the attacker controls your network"... but that doesn't need a checkpoint to fix, a simple partitioning detction that knows the total work of the best chain at releast time is sufficient.
512 2016-09-29 19:50:45	0|gmaxwell|Thanks for the discussion.
513 2016-09-29 19:51:14	0|wumpus|#topic segwit against uncompressed keys or not
514 2016-09-29 19:51:17	0|wumpus|(10 minutes to go)
515 2016-09-29 19:51:25	0|wumpus|(9 minutes to go)
516 2016-09-29 19:51:28	0|petertodd|so to be clear, *just* segwit right?
517 2016-09-29 19:51:31	0|CodeShark|does anyone still use uncompressed keys?
518 2016-09-29 19:51:34	0|wumpus|yes, only segwit
519 2016-09-29 19:51:39	0|achow101|CodeShark: armory does
520 2016-09-29 19:51:43	0|luke-jr|seems uncontroversial
521 2016-09-29 19:51:46	0|petertodd|I'm happy to ACK that given just segwit
522 2016-09-29 19:51:57	0|achow101|having segwit enforce uncompressed keys would delay segwit adoption for armory users
523 2016-09-29 19:52:04	0|achow101|*compressed
524 2016-09-29 19:52:05	0|jl2012_|it's in #8499
525 2016-09-29 19:52:09	0|luke-jr|achow101: why? just compress them
526 2016-09-29 19:52:13	0|wumpus|gmaxwell: yes, though we had a lot of trouble with partitioning detection, I remember some code being stripped out and such. But anyhow, yes that's the better approach if it can be gotten to work.
527 2016-09-29 19:52:22	0|sipa|achow101: sigh, does armory still not do that?
528 2016-09-29 19:52:30	0|achow101|luke-jr: we have to change the whole wallet structure (it's still going to happen anyways)
529 2016-09-29 19:52:31	0|wumpus|gmaxwell: without too much false positives
530 2016-09-29 19:52:36	0|luke-jr|achow101: why?
531 2016-09-29 19:52:39	0|sipa|achow101: alan said somewhere in 2013 he was implementing it...
532 2016-09-29 19:52:49	0|achow101|alan's gone now..
533 2016-09-29 19:52:50	0|luke-jr|afaik the only downside to using compressed keys is it changes the address, which segwit is changing anyway
534 2016-09-29 19:52:52	0|CodeShark|it's not a very complicated change
535 2016-09-29 19:52:55	0|wumpus|armory still uses uncompressed keys?!
536 2016-09-29 19:53:07	0|luke-jr|there's no reason you'd need to change the wallet structure I can see
537 2016-09-29 19:53:16	0|wumpus|in any case this only applies to segwit, not to old transactions
538 2016-09-29 19:53:34	0|achow101|the plan is to have a new wallet structure with bip32 that supports segwit and compressed keys
539 2016-09-29 19:53:41	0|gmaxwell|wumpus: "you're partitioned until you see a header chain with at least work X" is a pretty simple critera. :P
540 2016-09-29 19:53:44	0|sipa|luke-jr: it had fixed size records in its wallet format for pubkeys
541 2016-09-29 19:54:05	0|sipa|achow101: well if a new wallet format is needed for segwit anyway, it doesn't matter right?
542 2016-09-29 19:54:10	0|gmaxwell|achow101: oh god please do not use uncompressed keys with segwit. why would you do that?
543 2016-09-29 19:54:13	0|luke-jr|sipa: zero-pad it?
544 2016-09-29 19:54:35	0|achow101|sipa: well no, we don't need a new wallet for segwit as it could still work with the old one with a little bit of hacking
545 2016-09-29 19:54:48	0|achow101|that was the original plan
546 2016-09-29 19:54:48	0|luke-jr|achow101: no less than compressed could
547 2016-09-29 19:55:08	0|luke-jr|sipa: or store the uncompressed key, and compress it at address-generation/signing
548 2016-09-29 19:55:27	0|gmaxwell|achow101: why cant the same hack that indicates segwit is in use indicate compressed.. you just chop off some bytes of the key pretty much.
549 2016-09-29 19:55:43	0|sipa|btw, uncompressed keys account for 0.7% of used keys in succesful sigs on the network (in the past 2 hours)
550 2016-09-29 19:55:44	0|gmaxwell|it could be done entirely inside the process that seralizes the segwit scriptpubkey.
551 2016-09-29 19:55:58	0|achow101|gmaxwell: idk. ask goatpig
552 2016-09-29 19:56:06	0|gmaxwell|achow101: okay
553 2016-09-29 19:56:10	0|CodeShark|I think we should encourage all wallets to use compressed keys - achow101, if you need help with this I'd be willing to help
554 2016-09-29 19:56:26	0|sipa|agree - we should help
555 2016-09-29 19:56:27	0|gmaxwell|yes, lots of people would be glad to help.
556 2016-09-29 19:56:32	0|sipa|instead of just yell
557 2016-09-29 19:56:51	0|gmaxwell|well I offered to help armory move off uncompressed keys to alan several times, including offering to pay to do it.
558 2016-09-29 19:56:56	0|gmaxwell|so please don't say anyone just yelled.
559 2016-09-29 19:58:39	0|CodeShark|I initially designed my account structures to only use compressed keys - but later added a compressed bit to support legacy stuff
560 2016-09-29 19:59:06	0|petertodd|CodeShark: what legacy stuff specifically? legacy armory users?
561 2016-09-29 19:59:08	0|wumpus|CodeShark: bah,it's kind of sad that to hear some things seem to be going back instead of forward :)
562 2016-09-29 19:59:19	0|CodeShark|yes, to support other wallets
563 2016-09-29 19:59:27	0|wumpus|it's time
564 2016-09-29 19:59:50	0|CodeShark|but I think we really do need to prod all wallets to move to compressed keys
565 2016-09-29 20:00:07	0|CodeShark|there's really no reason to continue to support uncompressed keys - other than perhaps some migration tools
566 2016-09-29 20:00:08	0|wumpus|#endmeeting
567 2016-09-29 20:00:15	0|gmaxwell|CodeShark: as pieter notes, virutally nothing is already.
568 2016-09-29 20:00:15	0|lightningbot|Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.log.html
569 2016-09-29 20:00:15	0|lightningbot|Meeting ended Thu Sep 29 20:00:15 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
570 2016-09-29 20:00:15	0|lightningbot|Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.html
571 2016-09-29 20:00:15	0|lightningbot|Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.txt
572 2016-09-29 20:00:29	0|gmaxwell|0.7% percent
573 2016-09-29 20:00:38	0|anchow101|Yes, help would be appreciated
574 2016-09-29 20:01:05	0|wumpus|well supporting it in consensus for the normal network keeps making sense, but segwit is just such a great oppertunity to get rid of it
575 2016-09-29 20:01:31	0|petertodd|wumpus: yeah, I don't see how we can remove backwards compatibility for it w/o confiscating funds, but no reason to not remove support in new addresses
576 2016-09-29 20:01:40	0|wumpus|petertodd: indeed
577 2016-09-29 20:01:56	0|gmaxwell|yes, thats why its important to get rid of now. otherwise I wouldn't care if action were taken n months later.
578 2016-09-29 20:01:56	0|luke-jr|if anything, we should be discussing whether to make it a consensus rule rather than a policy ;)
579 2016-09-29 20:02:20	0|gmaxwell|luke-jr: I like to but many people feel that addining an additional consensus rule for segwit now wouldn't be prudent.
580 2016-09-29 20:02:37	0|gmaxwell|making it non-standard is sufficient in my view, such that we'd be able to make it a consensus rule later.
581 2016-09-29 20:02:42	0|btcdrak|achow101 seems to be having connection problems
582 2016-09-29 20:02:45	0|luke-jr|sure
583 2016-09-29 20:02:55	0|achow101_|btcdrak: just a little bit. switching computers
584 2016-09-29 20:02:58	0|CodeShark|gmaxwell: having to deal with the additional case complicates implementations
585 2016-09-29 20:02:59	0|luke-jr|just saying the rule shouldn't be controversial itself really
586 2016-09-29 20:03:02	0|CodeShark|not very much, but still
587 2016-09-29 20:03:08	0|petertodd|gmaxwell: well, so long as we loudly warn that this is intended to become unspendable later if you bypass the standardness
588 2016-09-29 20:03:28	0|morcos|gmaxwell: if so, we should be as clear about it being not allowed now as if we were to make it a consensus rule now.
589 2016-09-29 20:03:55	0|gmaxwell|for those thinking that we have to verify all the old stuff for all time, that might be true for bitcoin core, but in the future I could imagine some implementations just not bothering to verify old stuff.
590 2016-09-29 20:04:09	0|gmaxwell|morcos: petertodd: agreed for sure.
591 2016-09-29 20:04:34	0|jtimon|I agree with luke-jr on not seeing the controversy in it being consensus rule with the rest of segwit
592 2016-09-29 20:04:34	0|sdaftuar|petertodd: ntoe that it's standard to create an output with an uncompressed pubkey hash, as we can't detect the issue until a spend attempt (right?)
593 2016-09-29 20:04:43	0|petertodd|gmaxwell: I don't mean verifying old stuff, I mean verifying new txs spending old coins
594 2016-09-29 20:04:55	0|petertodd|sdaftuar: yes, nothing we can do about that though
595 2016-09-29 20:05:16	0|gmaxwell|jtimon: well in the case of low-S which we also wanted to make a consensus rule, jl2012 discovered that there were corner conditions we wanted to think about more carefully before making it a consensus rule.
596 2016-09-29 20:05:23	0|petertodd|for example, we've made hybrid pubkeys non-standard, and given basically no-one's ever used them in production for anything I'd have no issues with making them unspendable in a soft-fork
597 2016-09-29 20:05:23	0|sdaftuar|petertodd: right; just want to make sure we're all on the same page as i think communicating this widely/loudly is important
598 2016-09-29 20:05:43	0|jtimon|gmaxwell: thanks
599 2016-09-29 20:06:30	0|gmaxwell|jtimon: I don't think the same applies to uncompressed keys, because the criteria there is even simpler. but the lowS reason is part of why we punted this collection of improvements to policy for now.
600 2016-09-29 20:06:50	0|jtimon|mhmm
601 2016-09-29 20:06:57	0|michagogo|I saw a movie that depicts a form of distributed/decentralized system, to avoid it getting shut down. Or in the words of the character that explains it, "everyone that logs on is a server". It's said to be "open source", but then that's explained as "anyone can edit the code, like Wikipedia.
602 2016-09-29 20:07:18	0|achow101_|so if anyone wants to help armory with segwit support, bip32, compressed keys, we accept PRs. All our work happens in the dev branch, not master
603 2016-09-29 20:07:19	0|michagogo|And the code is deployed when a majority of users approve it
604 2016-09-29 20:08:01	0|wumpus|michagogo: heh, open source in some weird twisted mirror world
605 2016-09-29 20:08:05	0|gmaxwell|achow101: is there a IRC channel where things are discussed? E.g. where should I ask goatpig about compressed pubkeys in segwit.
606 2016-09-29 20:08:12	0|sipa|michagogo: i believe they're mistakingly not describing a computer network, but politics.
607 2016-09-29 20:08:13	0|achow101_|#bitcoin-armory
608 2016-09-29 20:08:24	0|jtimon|michagogo: that is, when sybil decides so...
609 2016-09-29 20:08:50	0|luke-jr|achow101_: meh, should just collapse that into #bitcoin-dev :p
610 2016-09-29 20:09:27	0|michagogo|And it's completely vulnerable to Sybil attacks…
611 2016-09-29 20:09:43	0|michagogo|Gah, lagging
612 2016-09-29 20:09:57	0|michagogo|Yeah
613 2016-09-29 20:10:17	0|michagogo|And of course, when the last user logs off, it doesn't just stop working
614 2016-09-29 20:10:44	0|michagogo|The sybil attackers are able to watch it dramatically implode with special effects, "graphical corruption" type stuff
615 2016-09-29 20:11:50	0|michagogo|And there's the obligatory "they're blocking all our foreign IPs" and that kind of stuff, with no explanation of who "they" are
616 2016-09-29 20:12:05	0|jtimon|so what was there a conclusion for the range service bits? nothing/top-288/everything?
617 2016-09-29 20:12:19	0|jtimon|what about the getrange message and "sharding"
618 2016-09-29 20:12:39	0|GitHub159|[13bitcoin] 15laanwj opened pull request #8843: rpc: Handle `getinfo` client-side in bitcoin-cli w/ `-getinfo` (06master...062016_09_getinfo_clientside) 02https://github.com/bitcoin/bitcoin/pull/8843
619 2016-09-29 20:12:41	0|wumpus|conclusion was to add one service bit: last-288-served
620 2016-09-29 20:13:04	0|wumpus|and maybe later one for last-1000-served
621 2016-09-29 20:13:14	0|jtimon|wumpus: I see, and leave the rest for later, thanks
622 2016-09-29 20:13:17	0|luke-jr|1024 would be rounder. ☺
623 2016-09-29 20:13:30	0|wumpus|and a jackpot for whoever enabled both at once
624 2016-09-29 20:13:52	0|luke-jr|if you set both, does it mean last 288000? :P
625 2016-09-29 20:14:22	0|jtimon|would it be crazy to just have last-1024 without last-288 and just change prunning's default?
626 2016-09-29 20:14:48	0|wumpus|288 is not just the default, it's the minimum
627 2016-09-29 20:15:05	0|wumpus|I'd be okay with changing the default not the minimum, but that'd keep some nodes completely useless
628 2016-09-29 20:15:19	0|wumpus|whereas by far most requests are in the last 288
629 2016-09-29 20:15:25	0|luke-jr|wumpus: useless for syncing*
630 2016-09-29 20:15:58	0|luke-jr|frankly, there are enough full-archive nodes out there that we really don't *need* to do anything right now, so meh :p
631 2016-09-29 20:16:06	0|sipa|wumpus: actually, not true.
632 2016-09-29 20:16:12	0|jtimon|well, the users know what to do to stop being useless...
633 2016-09-29 20:16:17	0|wumpus|which, as morcos remarked, preferentially downloading the last blocks from would take a lot of load of nodes that do keep more blocks
634 2016-09-29 20:16:22	0|sipa|there are more requests in 101-1000 deep then 2-100 deep
635 2016-09-29 20:16:29	0|wumpus|ok...
636 2016-09-29 20:16:34	0|sipa|*than
637 2016-09-29 20:16:37	0|wumpus|I misremembered apparently, never mind
638 2016-09-29 20:16:45	0|luke-jr|an unsyncable-from node is still more useful than a syncable node that isn't used for a wallet
639 2016-09-29 20:17:09	0|luke-jr|syncable-from*
640 2016-09-29 20:17:26	0|jtimon|maybe we can changethe prunning minimum if that simplifies things?
641 2016-09-29 20:17:30	0|sipa|wumpus: well, sample of 1 long-term-running node over the course of a few weeks of data
642 2016-09-29 20:17:36	0|sipa|wumpus: more samples welcome
643 2016-09-29 20:17:57	0|wumpus|sipa: do you have a special patch for statistics collection?
644 2016-09-29 20:18:00	0|gmaxwell|sipa: need to filter out bitnotes.
645 2016-09-29 20:18:13	0|sipa|gmaxwell: right; how do you suggest to do that?
646 2016-09-29 20:18:26	0|wumpus|sipa: or a script for parsing logs?
647 2016-09-29 20:18:39	0|sipa|wumpus: both; i'll publish them after a little cleanup
648 2016-09-29 20:18:46	0|wumpus|I could put it up on a few nodes, no problem
649 2016-09-29 20:19:05	0|sipa|it just logs an extra line with depth and block size for each requested block
650 2016-09-29 20:19:13	0|wumpus|nice
651 2016-09-29 20:19:17	0|jtimon|I guess it's not completely crazy, but nobody seem to specially like it
652 2016-09-29 20:19:36	0|sipa|en then
653 2016-09-29 20:19:37	0|sipa|S=0; fgrep DEEP ~/.bitcoin/debug.log | cut -d ' ' -f 4 | sort -g | uniq -c | tac | while read C D; do S=$(($S+$C)); echo "$D $C $S"; done | tac | less
654 2016-09-29 20:19:40	0|sipa|to inspect :)
655 2016-09-29 20:20:14	0|wumpus|jtimon: no, it's not completely crazy, using only one service bit is kind of elegant
656 2016-09-29 20:21:12	0|jtimon|:)
657 2016-09-29 20:22:20	0|wumpus|jtimon: if 1000 is really one-size-fits-all, and <1000-keeping nodes may as well be ignored. It's just hard to say without better statistics.
658 2016-09-29 20:22:35	0|wumpus|also statistics about what pruning sizes people prefer
659 2016-09-29 20:23:00	0|wumpus|I mean if everyone prefers the minimum and no one sets 1000 in practice
660 2016-09-29 20:23:19	0|gmaxwell|sipa: just do Satoshi:(recent) useragents.
661 2016-09-29 20:23:35	0|jtimon|well, independenlty of the statistics we will eventually need a more generic solution for flexible sharding, right?
662 2016-09-29 20:23:45	0|sipa|jtimon: maybe
663 2016-09-29 20:24:06	0|sipa|"need" is a big word imho
664 2016-09-29 20:24:13	0|sipa|but i agree it would be nice
665 2016-09-29 20:24:14	0|gmaxwell|jtimon: I think we do, would you like to finish the solution for that I started on?
666 2016-09-29 20:24:17	0|wumpus|jtimon: well there needs to be a different solution for historical block hosting IMO, but that's a different thing
667 2016-09-29 20:25:25	0|gmaxwell|sipa: I think excepting participants to keep around hundreds of gigs of blockchain is not conducive to the surival of the network, the alternative I see is a hardfork that drops off the history past some point. (e.g. just restarts the chain from a utxo commitment made a year before)
668 2016-09-29 20:26:27	0|sipa|gmaxwell: well, or just stop supporting historical block fetching more than 1 year or whatever number back on the p2p protocol, and use http
669 2016-09-29 20:26:41	0|wumpus|or bittorrent *ducks*
670 2016-09-29 20:26:51	0|jtimon|wumpus: yeah, historical hosting is what I mean
671 2016-09-29 20:27:15	0|jtimon|gmaxwell: maybe, but it sounded deterministic like luke-jr proposed instead of flexible like wumpus wanted
672 2016-09-29 20:27:22	0|wumpus|it could be anything that supports downloading ranges of data...
673 2016-09-29 20:27:47	0|brainwave|Under overview, balances, on the right side of  available, pending, total, add ~ exchange rate for dollars, pounds, euro
674 2016-09-29 20:28:00	0|sipa|brainwave: bitcoin core does not and cannot know exchange rates
675 2016-09-29 20:28:37	0|sipa|(because it would require contacting a centralized service, which we don't do by design)
676 2016-09-29 20:29:09	0|wumpus|yes or someone would need to commit them to the chain, but that'd still be trusting a central issuer/signer of the information
677 2016-09-29 20:29:12	0|wumpus|it's just a no-go
678 2016-09-29 20:35:52	0|gmaxwell|well if the users of bitcoin accepted that kind of security model change, what I would suggest is something like every 26280 blocks the block is required to have a commitment to the utxo set (could be a linear hash) as of 2016 blocks prior. and then six months of work after that, that commitment becomes usable for initial sync. and so then no one need process more than a year of blocks at sync.
679 2016-09-29 20:35:58	0|gmaxwell|.. though you would have to store three copies of the utxo set (though perhaps deduplicated)
680 2016-09-29 20:36:46	0|gmaxwell|jtimon: I don't know why anyone would find determinstic less desirable.
681 2016-09-29 20:37:21	0|sipa|gmaxwell: well i expect the controversy to not be about the change in security model, but about the perpetual requirement of having a utxo set
682 2016-09-29 20:40:07	0|wumpus|gmaxwell: I explained that: if you make it deterministic you have to be sure of the parameters in advance, there is no room for tweaking or optimizing later on
683 2016-09-29 20:42:12	0|gmaxwell|wumpus: well you simply extend the protocol to have a new signaling mechenism for the tweaked thing.
684 2016-09-29 20:42:15	0|wumpus|sipa: yes the bigger problem is the ever-growing UTXO set
685 2016-09-29 20:42:25	0|wumpus|gmaxwell: but then it loses backwards compatibility every time
686 2016-09-29 20:43:03	0|gmaxwell|something that just signals absolute heights has the problem that the communicated information will always be out of date. .. or if nodes don't change the ranges they host, we will end up with highly irregular distributions of information.
687 2016-09-29 20:43:42	0|sipa|the type of tweaking needed, and the potentially aging problem depend on the specific proposal
688 2016-09-29 20:44:10	0|sipa|i'm sure we can come up with something that seems reasonable to all
689 2016-09-29 20:44:28	0|wumpus|agree, there may be a compromise that is somewhat flexible and still deterministic
690 2016-09-29 20:44:51	0|gmaxwell|well what I suggested might not be viable after all too. I'm not sure, I wasn't successful in achieving all my goals at once.
691 2016-09-29 20:45:26	0|wumpus|I just don't think setting it all in stone in advance is a good idea, for the whole reason that it's so hard to achieve all your goals all at once
692 2016-09-29 20:45:40	0|wumpus|especially if you don't know some of those goals yet
693 2016-09-29 20:45:47	0|gmaxwell|I wanted a scheme that would result in a uniform distribution of blocks, that didn't depend on peers to look to see what other peers had (because that could be spoofed), required minimal communication (not a long list of blocks in an addr message).. and retainined uniformity as the chain grew, without causing peers to redownload blocks they already forgot.
694 2016-09-29 20:47:13	0|gmaxwell|So I had found two schemes, one where peers had a ID and the amount of blocks they would store, and from that they could determine which they would store, and as new blocks came in they might store then and drop some group of old ones.  The problem with it was that to figure out if a particular peer had block X you had to do computation linear in the number of blocks in the chain.
695 2016-09-29 20:47:37	0|wumpus|darn, also fingerprinting will be hard to avoid
696 2016-09-29 20:47:46	0|gmaxwell|Then I had another scheme that was sublinerar work, BUT a peer might drop a block but later have to go fetch it again.
697 2016-09-29 20:47:55	0|gmaxwell|wumpus: thats unavoidable with any split up scheme.
698 2016-09-29 20:48:14	0|wumpus|yes
699 2016-09-29 20:48:15	0|sipa|make the IP address part of the seed
700 2016-09-29 20:48:26	0|sipa|if your dhcp changes, you have to resync, sorry.
701 2016-09-29 20:48:30	0|wumpus|unless a substantial part of nodes are the same
702 2016-09-29 20:48:31	0|gmaxwell|sipa: then when you change IP, you have to go download a different set of blocks.. :P hah
703 2016-09-29 20:48:47	0|wumpus|e.g. there are only 8 IDs, pick one
704 2016-09-29 20:48:54	0|gmaxwell|wumpus: well I was thinking 32 bits, but perhaps a smaller collection would be fine.
705 2016-09-29 20:49:18	0|gmaxwell|but that gives you at best only 1/8th spitting storage. :( maybe fine now, but not in the long term.
706 2016-09-29 20:49:47	0|wumpus|maybe the number of groups can grow over time, a doubling every so many blocks :)
707 2016-09-29 20:50:03	0|sipa|hah: if you get a request through an IP that doesn't correspond to your local storage, just proxy all requests through to another node which does, and use that to gradually resync for the new seed.
708 2016-09-29 20:50:05	0|gmaxwell|Part of why I haven't given this that much more thought is because I think bitcoin will need to move to the commit state and forget history model; the ever growing sync time is too big a tide to stand against.
709 2016-09-29 20:50:16	0|gmaxwell|sipa: lol!
710 2016-09-29 20:50:32	0|gmaxwell|sipa: I think thats actually how the freenet location swapping works, funny enough.
711 2016-09-29 20:50:36	0|wumpus|hehe
712 2016-09-29 20:50:46	0|sipa|downside: if you want this to be fingerprint resistant, you have no way to determine how many proxies your blocks actually went through
713 2016-09-29 20:50:49	0|sipa|=> instant mixnet
714 2016-09-29 20:51:37	0|gmaxwell|sipa: freenet nodes change position over time, and they do it by swapping their location with a direct neighbor, when that location swap makes them both closer to where they want to be, ... when requests come in for the new location, they don't have the data, but it's only one hop away..
715 2016-09-29 20:52:55	0|wumpus|gmaxwell: I've always thought that, it's hard to imagine this continuing for 10's of years, but where to put the anchor...
716 2016-09-29 20:53:00	0|gmaxwell|in any case. if there were only 8 flavors of nodes, then it all becomes simple, block_height//1000 % 8 = flavor.
717 2016-09-29 20:54:18	0|wumpus|that seems kind of elegant and straightforward, there must be a catch
718 2016-09-29 20:59:48	0|jtimon|gmaxwell: sorry, well the deterministic seems to come at the cost of less flexibility
719 2016-09-29 21:00:17	0|sipa|wumpus: i'm trying to think about why 8 isn't enough
720 2016-09-29 21:00:26	0|wumpus|if you want to automatically scale the number of flavors up with height you could divide height 0..N into X flavors, the N..3N into 2*N flavors, and so on, where each flavor gets flavor (x<<1)+randbit()
721 2016-09-29 21:01:08	0|jtimon|only 8 flavors requires you to store 1/8 of the blockchain
722 2016-09-29 21:01:38	0|sipa|and we could have names for the first 8 top-level flavours or so... so your wallet could report "Looking for a bittersweet node..."
723 2016-09-29 21:01:40	0|wumpus|(well those numbers are arbitrary but the idea is that if a doubling of the # is needed, the new flavor, a member of a twice as big set, would contain the previous one)
724 2016-09-29 21:02:21	0|wumpus|hehe, yes assigning names would be nice
725 2016-09-29 21:46:24	0|GitHub131|[13bitcoin] 15jnewbery opened pull request #8844: change sigops cost to sigops weight (06master...06sigops_weight) 02https://github.com/bitcoin/bitcoin/pull/8844
726 2016-09-29 21:49:09	0|GitHub72|[13bitcoin] 15jnewbery opened pull request #8845: Don't return the address of a P2SH of a P2SH (06master...06trivial-P2SH-P2SH) 02https://github.com/bitcoin/bitcoin/pull/8845
727 2016-09-29 21:51:55	0|gmaxwell|jtimon: "less flexible" -- everything is less flexible short of sending someone arbritary x86 bytecode that they run.
728 2016-09-29 21:54:00	0|jtimon|less flexible in the amount of data you store, but maybe 8 flavors can be subidivided in 16 flavors half the size as wumpus was suggeting, then 16 to 32, etc. That may be flexible enough
729 2016-09-29 21:54:31	0|gmaxwell|jtimon: I was recommending 2^32 'flavors' but wumpus was concerned about reducing fingerprinting.
730 2016-09-29 21:55:10	0|gmaxwell|the whole reason to reduce the amount was to make it more difficult to follow a node around as it changes network identity.
731 2016-09-29 21:55:31	0|gmaxwell|sipa: 8 isn't enough if the chain is perpetually growing.
732 2016-09-29 21:56:05	0|jtimon|I see
733 2016-09-29 21:56:15	0|sipa|yeah, increasing number, the further back you go, may make sense
734 2016-09-29 21:56:35	0|gmaxwell|a year from now the chain will be 200 gb, a year after 300 gb-- at that size it is larger than the most common ssd size currently. a year after that 400gb.... and at that point an 8 way split is again running common hosts out of disk even if the common ssd size has moved up to 500gb by then.
735 2016-09-29 21:57:00	0|jtimon|well, maybe archive nodes that don't want to store everything have to get a privacy hit
736 2016-09-29 21:57:32	0|gmaxwell|who will bother running one if it takes speical effort above and beyond running a node, and draws more resources?
737 2016-09-29 21:57:39	0|sipa|well if only we'd have a separate network for archivsl
738 2016-09-29 21:57:54	0|sipa|there are no privacy issues at all then
739 2016-09-29 22:06:25	0|gmaxwell|and no one run them.
740 2016-09-29 22:06:58	0|gmaxwell|s/run/running/
741 2016-09-29 22:07:27	0|sipa|i was about to say that separate network doesn't need to imply separate nodes
742 2016-09-29 22:07:42	0|sipa|but of course, that doesn't work because you'd get a privacy leak from correlating
743 2016-09-29 22:08:20	0|sipa|however, you can reconcile those by only having nodes with a long-term IP provide archival further back than some threshold
744 2016-09-29 22:10:03	0|gmaxwell|sipa: not just that, but if it's a special very resource intensive mode.. few will do it, pliling more resources onto it... causing fewer to do it...
745 2016-09-29 22:10:37	0|sipa|it's true that it's resource intensive, but it's a different kind of resources than most of the rest of running a node
746 2016-09-29 22:10:43	0|sipa|it needs disk space and bandwidth
747 2016-09-29 22:10:46	0|gmaxwell|I might think it's not over the threshold of that, except already people don't run regular nodes due to costs.
748 2016-09-29 22:11:12	0|sipa|rather than memory and cpu
749 2016-09-29 22:11:13	0|gmaxwell|which are what people usually complain about.
750 2016-09-29 22:11:31	0|sipa|then why aren't we seeing more pruned nodes?
751 2016-09-29 22:11:57	0|sipa|one reason may be that pruned nodes don't advertize, so we just don't know about them
752 2016-09-29 22:12:33	0|gmaxwell|because you have to edit a config file or change an obscure setting, we don't advertise it, and it breaks rescan and reindex. (which is part of why we don't really advertise it)
753 2016-09-29 22:14:43	0|sipa|well people mostly complain about the sync time for a node
754 2016-09-29 22:15:15	0|gmaxwell|yes, though I think thats most because so many stop there and give up before they get a chance to complain about the rest.
755 2016-09-29 22:15:21	0|sipa|perhaps
756 2016-09-29 22:18:17	0|TD-Linux|a first-run dialog box with a slider for disk usage and an estimated sync time would be very nice
757 2016-09-29 22:18:49	0|sipa|except the sync time does not depend on the value of the slider
758 2016-09-29 22:19:57	0|TD-Linux|yes, I meant it there so it'd appear at start. I guess having it in the status bar is sufficient
759 2016-09-29 22:20:36	0|sipa|ah
760 2016-09-29 22:20:48	0|sipa|well there will be an overlay with sync time indication in 0.14
761 2016-09-29 22:21:10	0|gmaxwell|doesn't it still incorrectly say you can't transact while syncing?
762 2016-09-29 22:21:30	0|sipa|we still have a lot time until 0.14
763 2016-09-29 22:23:09	0|gmaxwell|:)
764 2016-09-29 22:23:15	0|wumpus|well you can, but most people probably shouldn't do so
765 2016-09-29 22:23:35	0|gmaxwell|yes they should
766 2016-09-29 22:23:52	0|wumpus|during the initial sync they won't have any coins to send anyway, and receiving them is a bad idea as they'll only see them when the entire thing is done
767 2016-09-29 22:23:55	0|wumpus|oh?
768 2016-09-29 22:24:02	0|wumpus|why?
769 2016-09-29 22:24:18	0|gmaxwell|initial sync isn't my concern there:
770 2016-09-29 22:24:19	0|gmaxwell|probably one of the most common usage patterns for a wallet user is that you start your wallet up in order to pay someone, and it's three weeks behind. You can go ahead and pay, no problems.. why wouldn't you?
771 2016-09-29 22:24:37	0|gmaxwell|during initial sync you just won't have any coins, indeed. :)
772 2016-09-29 22:24:48	0|wumpus|the biggest problem is people giving out addresses during initial sync
773 2016-09-29 22:24:52	0|wumpus|then realizing how long it takes
774 2016-09-29 22:25:10	0|wumpus|this is what the overlay is designed to prevent
775 2016-09-29 22:25:27	0|wumpus|sure, you can send coins if you're three weeks behind, no problem, although fee computation could be off
776 2016-09-29 22:25:39	0|gmaxwell|yes, that is a large source of complaints, but we shouldn't tell people that they cant send funds already in their wallet when they start up and they're a bit behind, it's already a common mistaken belief that they cant (and then they complain about how long it takes to catch up a month of blcks)
777 2016-09-29 22:25:40	0|TD-Linux|the warning could be conditional on having zero funds
778 2016-09-29 22:26:13	0|gmaxwell|TD-Linux: the earlier warning text was fine-- saying that you won't see payments to you yet, but for some reason it was changed to say that you cannot send funds.
779 2016-09-29 22:26:17	0|wumpus|yeah fix one thing and they'll start complaining about another, it's a never ending source of fun...
780 2016-09-29 22:26:34	0|sipa|i don't think anyone will read the text anyway
781 2016-09-29 22:26:47	0|gmaxwell|I also complained that the text is now too long and won't get read.
782 2016-09-29 22:26:51	0|wumpus|of course people will read it
783 2016-09-29 22:26:55	0|sipa|the important thing is that it's in the way, and gives accurate (by then, hopefully) information
784 2016-09-29 22:26:57	0|wumpus|heck, users aren't stupid
785 2016-09-29 22:26:59	0|gmaxwell|The first text was better.
786 2016-09-29 22:27:11	0|sipa|gmaxwell: PR welcome
787 2016-09-29 22:27:15	0|wumpus|maybe some are, but not all of them, some will actually read and understand
788 2016-09-29 22:27:18	0|gmaxwell|Well I'm stupid, and looked at the notice in its updated state and didn't read the list line.
789 2016-09-29 22:27:24	0|gmaxwell|first*
790 2016-09-29 22:27:49	0|gmaxwell|because when there is too much text many people go a bit banner blind and skim past headings and such.
791 2016-09-29 22:27:51	0|wumpus|if we don't believe peopel actually pay attention then why do anything at all
792 2016-09-29 22:28:14	0|gmaxwell|saying that a wall of text is too much is not saying that people don't pay attention.
793 2016-09-29 22:28:20	0|wumpus|I think it's an improvement to what was there, indeed, if you want to imrpvoe further then pulls are welcome
794 2016-09-29 22:28:49	0|sipa|right, that's what i'm saying - having there being an overlay at all is more important than what the text says
795 2016-09-29 22:28:51	0|gmaxwell|and re: being able to send, people already complain that they have to wait a long time after starting to send because they already frequently mistakingly believe they can't.
796 2016-09-29 22:28:54	0|sipa|and we have time to improve the latter
797 2016-09-29 22:29:06	0|wumpus|but I'm a bit tired of people always saying "users won't read anyway" to everything that adds documentation , help or warnings
798 2016-09-29 22:29:29	0|wumpus|a lot of users are definitely looking for more help and guidance when they first open the program, and a bit of text helps there
799 2016-09-29 22:29:41	0|gmaxwell|wumpus: why should I waste my time when I point out that THE TEXT IS OUTRIGHT UNTRUE and your response is to accuse me of thinking users are stupid?  my comment was that the earlier version of the text which was simple and NOT UNTRUE was better.
800 2016-09-29 22:29:54	0|sipa|please guys
801 2016-09-29 22:30:01	0|sipa|gmaxwell: go propose something
802 2016-09-29 22:30:05	0|gmaxwell|I did!
803 2016-09-29 22:30:16	0|wumpus|gmaxwell: well if the text is wrong then it should be fixed obviously, change it to a better text
804 2016-09-29 22:30:49	0|wumpus|I don't know what the previous version of the text was
805 2016-09-29 22:31:56	0|sipa|it's been changed a dozen times in the lifetime of the pull
806 2016-09-29 22:33:04	0|sipa|also, it says "Spending bitcoins may not be possible until synchronization has finished."
807 2016-09-29 22:33:09	0|sipa|which is not untrue.
808 2016-09-29 22:34:44	0|gmaxwell|okay, it was changed after I last saw it.
809 2016-09-29 22:35:13	0|wumpus|ok that was useless :)
810 2016-09-29 22:35:19	0|gmaxwell|by saying 'may' which is still misleading, but worse, that text is the bold.
811 2016-09-29 22:35:25	0|gmaxwell|er is the only bold part.
812 2016-09-29 22:35:43	0|sipa|well, improvements welcome
813 2016-09-29 22:35:54	0|gmaxwell|So now it says "mumble mumble mumble  Spending bitcoins may not be possible during that phase!" :-/
814 2016-09-29 22:36:14	0|gmaxwell|it's a waste of my time, I already raised these issues and it was then merged.
815 2016-09-29 22:36:46	0|wumpus|it had to be merged at some point, with the idea it could be improved later
816 2016-09-29 22:36:58	0|gmaxwell|well to be fair the last change did improve it, its true.
817 2016-09-29 22:37:38	0|gmaxwell|but created the problem that if you skim it is that all you extract is that you can't spend, .. which misses the really critical thing: which is that you wallet may look empty when it isn't.
818 2016-09-29 22:37:58	0|wumpus|that doesn't mean it's final, most will only see the message when it is merged, and can improve it then, there are already some pulls open to improve that overlay
819 2016-09-29 22:38:00	0|gmaxwell|but okay, I can open a PR.
820 2016-09-29 22:38:17	0|wumpus|(but I don't think they change that message)
821 2016-09-29 22:39:09	0|sipa|gmaxwell: i think people didn't really understand the point of your concern (i didn't): if you're looking at it from a point of view that this would be be mostly seen (and intended to convey information) during IBD, it's perfectly reasonable to warn users they won't be able to spend the money they're still to receive... and a simplification to reduce the length of the text may be warranted
822 2016-09-29 22:39:31	0|sipa|it's a good point that it's also seen during non-IBD
823 2016-09-29 22:39:35	0|gmaxwell|It will mostly not be seen during IBD.
824 2016-09-29 22:39:54	0|gmaxwell|during IBD sure someone will see it then, say of course I knew that (even if they didn't) minimize and go on with life. :P
825 2016-09-29 22:40:13	0|gmaxwell|but then users will see it every single time they start.
826 2016-09-29 22:40:22	0|sipa|i'm aware, you don't need to argue about this
827 2016-09-29 22:40:30	0|sipa|i'm just explaining why maybe you felt misunderstood
828 2016-09-29 22:40:31	0|gmaxwell|sorry, not arguing-- clarifying.
829 2016-09-29 22:40:43	0|gmaxwell|Yes, I see that and I didn't before.
830 2016-09-29 22:41:19	0|gmaxwell|when I first saw this PR I even took the time to go through the code carefully to check to see if there was anything that made it IBD only.
831 2016-09-29 22:42:02	0|gmaxwell|because I couldn't understand why people wanted the text that it had.
832 2016-09-29 22:42:18	0|gmaxwell|it did not occure to me that other people might be only thinking about IBD.
833 2016-09-29 22:42:26	0|gmaxwell|sorry for being thoughtless there.
834 2016-09-29 22:42:47	0|wumpus|#8805 fixed a few minor grammar nits, #8821 fixes a blocking problem with the overlay, there are no pulls yet that improve the message
835 2016-09-29 22:43:18	0|sipa|sorry, we (including me) aren't being careful with terminology here... IBD is also used for syncup when you were previously synced to a month ago
836 2016-09-29 22:43:48	0|wumpus|it's very easy to forget about catching up nodes
837 2016-09-29 22:44:18	0|wumpus|but yes we shouldn't
838 2016-09-29 22:44:21	0|sipa|well it's mostly designed to help with that first sync
839 2016-09-29 22:45:07	0|gmaxwell|more obvious to me just by chance of hering more people complain about it, also I've stopped running a node 24/7 on my laptop because I've been watching the battlestar galactica series in evenings and bitcoin interupts video playback. :)
840 2016-09-29 22:45:15	0|wumpus|during catch-up it's reasonably useful too, people may not know they won't see transactions newer than their sync point and worry, but yes it's mostly important for the initial IBD (lol)
841 2016-09-29 22:45:21	0|gmaxwell|so every time I go to use bitcoin I'm stuck waiting for it to catch up.
842 2016-09-29 22:45:42	0|gmaxwell|yes, we should have this message during catch up. But it's important to not make people think they can't spend funds that they can see.
843 2016-09-29 22:46:04	0|gmaxwell|The important message is that you may not see all payments to you yet (and you can't spend what you can't see).
844 2016-09-29 22:46:31	0|wumpus|bitcoin interrupts video playback? even in steady state mode?
845 2016-09-29 22:46:34	0|TD-Linux|one option would be to put it on the payment request generation page instead. but even in its current state it's far better than what was there before (nothing)
846 2016-09-29 22:47:29	0|gmaxwell|wumpus: yes. on my laptop... playback from a local file. The issue is IO or cpu related, probably the former but I haven't tested extensively to know for sure.
847 2016-09-29 22:48:03	0|wumpus|that's very strange. I'd expect that during intial sync when it maxes out CPU and I/O usage, but not when it's up to date
848 2016-09-29 22:48:13	0|gmaxwell|TD-Linux: the big problem we should be solving here is that people see a balance of zero then delete the wallet. I think thats the priority because any other issue doesn't cause irrecoverable loss.
849 2016-09-29 22:48:43	0|gmaxwell|wumpus: I notice it during ordinary computer use.. causes IO hangs, but its not irritating except when watching video.
850 2016-09-29 22:49:29	0|wumpus|do you have a lot of mlocked memory? is it swapping?
851 2016-09-29 22:50:01	0|gmaxwell|no, not swapping 8gb ram. I think that when a bunch of random writes happen it causes long delays for garbage collection in the SSD.
852 2016-09-29 22:50:03	0|wumpus|swapping seems to be the foremost cause of I/O related hangs here, as essentially the memory subsystem has to wait for I/O to complete
853 2016-09-29 22:50:41	0|wumpus|heh as if 8gb ram means no swapping these days :)
854 2016-09-29 22:50:59	0|gmaxwell|well on my laptop its enough most of the time.
855 2016-09-29 22:51:29	0|gmaxwell|The stalls seemed to get better for a while after I freed up a bunch of space and trimmed the drive, but got worse after which is why I think SSD GC plays a roll.
856 2016-09-29 22:51:50	0|gmaxwell|but in any case, while watching the show every block arrival causes a second-long pause in playback.
857 2016-09-29 22:51:51	0|wumpus|maybe someone is requesting a lot of blocks from you with a bloom filter? :-) it would be interesting to find out what your node is actually doing at those times
858 2016-09-29 22:52:02	0|gmaxwell|nah, outbound only.
859 2016-09-29 22:52:10	0|gmaxwell|I know its at the same time as blocks showing up.
860 2016-09-29 22:52:19	0|TD-Linux|gmaxwell, easy way to verify that would be to increase your video player's lookahead cache
861 2016-09-29 22:52:22	0|wumpus|ok so it's block verification, leveldb seeks
862 2016-09-29 22:52:58	0|gmaxwell|TD-Linux: think mpv uses non-blocking reads of the disk?
863 2016-09-29 22:53:21	0|TD-Linux|gmaxwell, yup it does. I've increase the setting to 10s when using sshfs and it works fine
864 2016-09-29 22:54:04	0|gmaxwell|in any case, performance distraction aside, when this happens I shut down bitcoind then it may stay off for a week before I need to do something with it, then waiting for it to catch up is irritating.
865 2016-09-29 22:55:24	0|gmaxwell|(and of course my systems performance is seriously impacted while it catches up)
866 2016-09-29 22:55:28	0|wumpus|yes, nothing to do about that, I guess if hybrid SPV mode is implemented it could also work during catch-up
867 2016-09-29 22:55:55	0|wumpus|indeed, it's either slow down the catch up or tolerate it hogging the whole system
868 2016-09-29 22:58:09	0|gmaxwell|I have wondered if it might be useful to split the chainstate into two parts, one with txouts created in the most recent N blocks, and one with the rest. Then on start we could just load the whole first one into the cache.
869 2016-09-29 22:58:11	0|wumpus|the default setting of hogging all cores during IBD/catch-up is a bit rude, certainly if it is a background process
870 2016-09-29 22:59:04	0|gmaxwell|if we did that much of the cost would then be signature valdation instead of random IO, and signature validation could run in the background, following behind the blocks.. and at lower priority.
871 2016-09-29 22:59:51	0|wumpus|so that would be like a 'prefer keeping recent UTXOs' cache policy?
872 2016-09-29 23:00:14	0|gmaxwell|I guess thats a stat that I still haven't collected.  "what is the average age-in-blocks of inputs that are consumed" (/what is the distribution of that age)
873 2016-09-29 23:00:29	0|gmaxwell|we know recent ones are spent more often but I don't have good numbers on it.
874 2016-09-29 23:00:32	0|gmaxwell|wumpus: yes.
875 2016-09-29 23:02:56	0|gmaxwell|probably not the highest priority improvement in any case.
876 2016-09-29 23:02:58	0|wumpus|well, currently the whole cache is emptied at a write, I think there are many eviction policies that would do better
877 2016-09-29 23:03:33	0|gmaxwell|right, also the in memory representation of the cache entries is quite inefficient.
878 2016-09-29 23:04:03	0|gmaxwell|so its effective size could potentially be doubled if its entries were flat allocated.
879 2016-09-29 23:04:08	0|wumpus|though that helps it actually being an efficient cache, a more efficient representation shouldn't come at a higher access cost
880 2016-09-29 23:06:48	0|wumpus|I did an experiment once with storing the UTXOs in serialized form in memory: https://github.com/laanwj/bitcoin/tree/2016_04_dummy_db
881 2016-09-29 23:09:31	0|gmaxwell|interesting!
882 2016-09-29 23:09:34	0|wumpus|although that's for the entire UTXO set, not just a limited cache
883 2016-09-29 23:10:53	0|gmaxwell|yea, the on disk seralization is most efficient, but not fast.
884 2016-09-29 23:11:26	0|gmaxwell|I wouldn't be surprised though for the current in memory representation if there wasn't more bytes spend in malloc/container overhead than actual transaction data though.
885 2016-09-29 23:11:29	0|wumpus|but given that most time is wasted on disk seeks anyway, it may not make too much difference in practice
886 2016-09-29 23:11:38	0|wumpus|depends on the system...
887 2016-09-29 23:12:01	0|wumpus|yes, the malloc overhead is somewhat bad
888 2016-09-29 23:14:08	0|wumpus|but not more than the actual data size, from what I remember
889 2016-09-29 23:15:16	0|wumpus|in any case improvements are certainly possible there, without any rocket science, it's just that it's such risky code to change
890 2016-09-29 23:15:53	0|wumpus|if it was any other project people would have optimized the shit out of it by now
891 2016-09-29 23:21:06	0|wumpus|unfortunately the damages of a bug there are unfathomable, not just skipping a few video frames
892 2016-09-29 23:22:24	0|gmaxwell|well the hope I had there was that with making the cache more efficient, it could be increased in size and avoid more disk IO. :)
893 2016-09-29 23:24:30	0|gmaxwell|on the earlier subject of 'alt' implementations doing inadvisable things, some of them have a genius performance optimization which they're crowing about, -- where they only validate transactions that weren't already in the mempool; something we explicity decided not to do because of the long history of subtle mempool corruption issues.
894 2016-09-29 23:25:36	0|wumpus|exactly, there are tons of ways to optimize and get things just slightly wrong
895 2016-09-29 23:25:45	0|gmaxwell|I worry that there will be a race to the bottom, where by making risky / security reducing optimizations implementations will gain significant performance advantages, and suffer no cost until the inevitable spectacular failure that results.
896 2016-09-29 23:26:06	0|wumpus|which doesn't matter if no one runs your code anyway, but we have to be really careful
897 2016-09-29 23:26:33	0|gmaxwell|and being safe doesn't matter if peopel don't run it in favor of things that are faster.
898 2016-09-29 23:26:43	0|gmaxwell|people*
899 2016-09-29 23:26:53	0|wumpus|we also shouldn't overestimate how important the performance is to most users, many just run it on a server or otherwise unused computer
900 2016-09-29 23:27:51	0|wumpus|well you'd say safeness is really important, if the inevitable spectacular failure happens you don't want to be at the center of it
901 2016-09-29 23:28:33	0|gmaxwell|well sure. indeed.
902 2016-09-29 23:28:39	0|wumpus|better slow than dead :)
903 2016-09-29 23:29:44	0|gmaxwell|but for things like the security model change to use a 6 month old utxo commitment instead of syncing the history... the potential for a spectacular failure there which a more conservative approach could have stopped is negligible.
904 2016-09-29 23:29:53	0|TD-Linux|gmaxwell, still think you should verify that it's actually IO that's the problem before going too deep :)
905 2016-09-29 23:29:56	0|gmaxwell|And if we don't investgate things like that, someone will do something dumber.
906 2016-09-29 23:30:18	0|wumpus|TD-Linux: yes, measuring is better than assumptions :)
907 2016-09-29 23:30:28	0|gmaxwell|TD-Linux: Oh IO is an issue for sure regardless of whats causing my mpv stalls. I'll find out tonight (I'm not going to sit here and watch video for an hour just to check)
908 2016-09-29 23:31:54	0|gmaxwell|TD-Linux: SSD vs a fast spinning disk with small dbcache here is a <4 hour sync (from a local peer) vs a >9hour sync.
909 2016-09-29 23:32:27	0|wumpus|and sure, it's good to investigate things like that
910 2016-09-29 23:33:00	0|wumpus|I/O is absolutely a problem sometimes, leveldb generates *tons* of seeks and small reads, better caching would help avoid some of that
911 2016-09-29 23:34:27	0|wumpus|I had no luck with other databases, I remember trying with lmdb at some point, which was faster in reading, but instead... does tons of seeks and small writes at write, so it just moves the problem
912 2016-09-29 23:35:21	0|TD-Linux|yeah, on a SSD at least, the former is much less detrimental to system latency
913 2016-09-29 23:35:28	0|wumpus|indeed!
914 2016-09-29 23:35:53	0|gmaxwell|for operation at the tip, using the mempool instead of validating would be a big aid... but the safty of that remains dubious. :)
915 2016-09-29 23:36:07	0|sipa|gmaxwell: i believe for the mempool it's approximately a factor 2 overhead
916 2016-09-29 23:36:24	0|sipa|gmaxwell: that that is also indexes, orderings, accounting, ...
917 2016-09-29 23:38:07	0|wumpus|yes that remains dubious, especially with regard to isFinal and such
918 2016-09-29 23:38:28	0|wumpus|I mean there is a part of transaction validation that can be obviously cached, and a part that may change in time
919 2016-09-29 23:39:03	0|gmaxwell|fortunately the MTP change made that much safer.
920 2016-09-29 23:41:49	0|gmaxwell|e.g. before a block could come in with a time before your local time, and contain txn which are isfinal invalid according to the block but okay with respect to the local time, and you'd accept it. I wonder if the alt implementations had that bug.
921 2016-09-29 23:48:12	0|wumpus|I remember that one, tricky... and there may be more problems of that kind not yet found
922 2016-09-29 23:49:43	0|gmaxwell|looks like their change dodged it, because the finality test is in the ContextualCheckBlock, and the bypass patch only bypasses checkinputs...
923 2016-09-29 23:49:56	0|gmaxwell|(which also means that it doesn't manage to avoid accessing the utxo cache entries)
924 2016-09-29 23:51:17	0|wumpus|I just realized, if the problem is that the block validation hiccups other things happening on your PC, the solution may be actually to slow it down :)
925 2016-09-29 23:52:01	0|wumpus|put a small sleep between each UTXO lookup, limit the validation to one thread
926 2016-09-29 23:52:38	0|wumpus|not something you'd want to do during initial sync if you're waiting for it, but if you don't care and it runs in the background...
927 2016-09-29 23:54:14	0|wumpus|after all you run it to keep up, you don't need to outrace it
928 2016-09-29 23:56:05	0|gmaxwell|I've thought before that if we have bandwidth limiting enabled we should delay announcement of new blocks to reduce the number of peers that request them from us... but slowing down the validation would work as well.
929 2016-09-29 23:56:19	0|gmaxwell|small sleeps perhaps aren't so good because it may busy spin. :P
930 2016-09-29 23:57:44	0|wumpus|heh, not that small
931 2016-09-29 23:58:57	0|wumpus|or use some OS-dependent way to reduce the I/O priority
932 2016-09-29 23:59:35	0|wumpus|as long as it's done by the time the next block comes in, so taking 10 minutes would take it too far :)