1 2016-04-13 02:20:56	0|GitHub177|[13bitcoin] 15mrCertified opened pull request #7867: deleted Configure.ac restore bits to all networks(%master%masterCode[{rLi}]) (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/7867
  2 2016-04-13 02:54:59	0|GitHub91|[13bitcoin] 15theuni opened pull request #7868: net: Split DNS resolving functionality out of net structures (06master...06net-cleanup-resolve) 02https://github.com/bitcoin/bitcoin/pull/7868
  3 2016-04-13 06:23:52	0|cryptocoder|hi everyone
  4 2016-04-13 06:24:37	0|cryptocoder|not sure if this is the right place for this, but how come the windows release for core 0.12 does not seem to have zmq support in it?
  5 2016-04-13 06:38:14	0|jonasschnelli|cryptocoder: IIRC there where problems with static linking...
  6 2016-04-13 06:38:50	0|jonasschnelli|cryptocoder: https://github.com/bitcoin/bitcoin/issues/6681
  7 2016-04-13 06:40:21	0|cryptocoder|ah! thank you jonasschnelli.  I was hoping i’m not missing something obvious
  8 2016-04-13 06:40:58	0|jonasschnelli|cryptocoder: I think you could hack the depends/ build system to link it dynamic.
  9 2016-04-13 06:41:28	0|jonasschnelli|Or compile it on a window machine (not cc), but not sure how this exactly works.
 10 2016-04-13 06:43:14	0|jonasschnelli|sipa, wumpus: reindex with LMDB took ~11h (same machine where a full sync with master took 2h20'). Now reindexing the master levelDB node.
 11 2016-04-13 09:12:50	0|jonasschnelli|wumpus: LMDB reindex with -dbcache=9000: >11h, levelDB with -dbcache=8000: 2h25'.
 12 2016-04-13 09:13:07	0|jonasschnelli|(the IBD from random peers was a couple of minutes faster... i'm confused)
 13 2016-04-13 09:13:33	0|jonasschnelli|I wonder where we the performance bottleneck with LMDB is.
 14 2016-04-13 09:14:22	0|sipa|that's very strange!
 15 2016-04-13 09:28:54	0|btcdrak|Completely backwards
 16 2016-04-13 09:30:32	0|sipa|especially since neither should be touched the database at all
 17 2016-04-13 09:30:41	0|sipa|with such dbcache
 18 2016-04-13 10:35:05	0|shangzhou|sipa:http://bitcoin.sipa.be/ data is not up to date
 19 2016-04-13 10:38:57	0|sipa|shangzhou: thanks, fixing
 20 2016-04-13 10:54:06	0|sipa|shangzhou: done
 21 2016-04-13 11:04:02	0|shangzhou|thanks @sipa
 22 2016-04-13 11:04:42	0|sipa|i upgraded my node to 0.12 and the new rpcauth mechanism, but didn't give the credentials to the script generating the website
 23 2016-04-13 11:43:40	0|wumpus|jonasschnelli: interesting - so a reindex is slow with LMDB, but a sync from another node is fast? Greg did a benchmark with a sync from another node and LMDB came out much faster: https://github.com/laanwj/bitcoin/tree/2016_04_mdb#x86_64  . Also wonder where the bottleneck is, but will add it to the performance results
 24 2016-04-13 11:44:18	0|jonasschnelli|wumpus: Both are slower. IBD and reindex.
 25 2016-04-13 11:44:25	0|wumpus|okay
 26 2016-04-13 11:44:41	0|wumpus|have you tried with the default dbcache as well?
 27 2016-04-13 11:44:57	0|wumpus|maybe the 5GB write transaction is what gets it
 28 2016-04-13 11:44:58	0|jonasschnelli|No... you mean a reindex with default dbache?
 29 2016-04-13 11:45:24	0|wumpus|yes. You've done so with dbcache 9000, which means it fills to about ~5gb without using the database, then it writes everything at once
 30 2016-04-13 11:45:36	0|wumpus|lmdb seems to shine in read latency, but it could be the big write is slow
 31 2016-04-13 11:45:50	0|wumpus|with such a large dbcache it never reads so that part isn't measured
 32 2016-04-13 11:46:23	0|jonasschnelli|wumpus: From looking at the "log speed" (very efficient benchmark technique :) ), lmdb seems to be slower during non-write operations.
 33 2016-04-13 11:46:43	0|jonasschnelli|But i'll do now compare a reindex with no other options.
 34 2016-04-13 11:46:56	0|wumpus|but with such a high dbcache, syncing from scratch, it doesn't touch the database at all
 35 2016-04-13 11:47:12	0|jonasschnelli|That is what is really strange...
 36 2016-04-13 11:47:25	0|jonasschnelli|maybe another change from your lmdb branch causes this.
 37 2016-04-13 11:47:33	0|jonasschnelli|Haven't had time to debug it tough.
 38 2016-04-13 11:47:35	0|wumpus|if everyone used dbcache=9000 we wouldn't need a database at all, we could just store the utxo set in a linear file
 39 2016-04-13 11:48:41	0|jonasschnelli|Indeed. Just hope there is no crash before the big write. :)
 40 2016-04-13 11:48:49	0|sipa|jonasschnelli: can you run with debug=bench, and show the resulting debug.log?
 41 2016-04-13 11:49:01	0|jonasschnelli|sipa: ah. Right. Let me do that.
 42 2016-04-13 11:50:22	0|sipa|--- 3 hours later ---
 43 2016-04-13 11:51:10	0|jonasschnelli|Hei! There is a SSD! :)
 44 2016-04-13 11:51:35	0|jonasschnelli|Example bench (lmdb): height: 82804, Connect total: 0.14ms [22.08s]
 45 2016-04-13 11:51:48	0|jonasschnelli|Verify 0 txins: 0.03ms (0.000ms/txin) [15.22s], Index writing: 0.05ms [3.19s]
 46 2016-04-13 11:51:51	0|jonasschnelli|Maybe index writing?
 47 2016-04-13 11:52:01	0|wumpus|height 82804 isn't very interesting yet :)
 48 2016-04-13 11:52:14	0|jonasschnelli|Yeah. Can't start top down. :)
 49 2016-04-13 11:53:23	0|wumpus|but anyhow in one light I like your result jonasschnelli, it would mean that leveldb is ok+ and there's no need to even spend time investigationg switching to something else
 50 2016-04-13 11:54:15	0|wumpus|if both sqlite and lmdb came out slower - then again Greg's benchmark showed something completely different that's curious
 51 2016-04-13 11:54:50	0|wumpus|you're using a SSD? don't know what his storage device was
 52 2016-04-13 11:55:39	0|sipa|except if lmdb's final full UTXO set dump took ~9h longer than leveldb's, i can't explain jonasschnelli's result
 53 2016-04-13 11:56:15	0|jonasschnelli|Yes. Lets first find out where the time/cycles is/are consumed.
 54 2016-04-13 11:57:45	0|jonasschnelli|wumpus: Yes. I'm using SSD. The write speed is somewhere around 1GB/s.
 55 2016-04-13 14:01:27	0|jonasschnelli|Verify 1245 txins: 131.58ms (0.106ms/txin) [6447.02s]
 56 2016-04-13 14:01:34	0|jonasschnelli|Is the value in the [] the total value so far?
 57 2016-04-13 14:03:06	0|sipa|yes
 58 2016-04-13 14:05:25	0|jonasschnelli|Running two hours so far. Looks like most time is consumed for verify (as expected). But still not clear why lmdb takes ~5 times longer...
 59 2016-04-13 14:22:01	0|jonasschnelli|Current Master at block 200'000: Connect total: 33.82ms [331.59s]
 60 2016-04-13 14:22:14	0|jonasschnelli|LMDB at block 200'000: Connect total: 125.55ms [1466.35s]
 61 2016-04-13 14:22:43	0|sipa|are you sure the LMDB is with dbcache set high?
 62 2016-04-13 14:23:00	0|jonasschnelli|no. Now its with default dbcache
 63 2016-04-13 14:23:00	0|sipa|jonasschnelli: show me the verify lines
 64 2016-04-13 14:23:09	0|sipa|both leveldb and lmdb?
 65 2016-04-13 14:23:13	0|jonasschnelli|yes.
 66 2016-04-13 14:23:16	0|sipa|oh, ok
 67 2016-04-13 14:23:25	0|jonasschnelli|LMDB block 200k: Verify 1231 txins: 121.49ms (0.099ms/txin) [1385.86s
 68 2016-04-13 14:23:49	0|jonasschnelli|leveldb: Verify 1231 txins: 28.88ms (0.023ms/txin) [262.67s]
 69 2016-04-13 14:24:09	0|sipa|are they running with the same -par?
 70 2016-04-13 14:24:19	0|jonasschnelli|Yes. Passed only -debug=bench
 71 2016-04-13 14:24:52	0|jonasschnelli|nononono!
 72 2016-04-13 14:25:22	0|jonasschnelli|lmdb node -> git branch -> master!
 73 2016-04-13 14:25:32	0|sipa|?
 74 2016-04-13 14:25:55	0|jonasschnelli|I'm actually not testing against LMDB i'm testing again non-libsecp-master from nov 1st!
 75 2016-04-13 14:26:01	0|jonasschnelli|Damit!
 76 2016-04-13 14:26:06	0|sipa|lol
 77 2016-04-13 14:26:19	0|sipa|libsecp was merged on nov 5th :)
 78 2016-04-13 14:26:43	0|sipa|thank for benchmarking the improvements we've made the past 5 months, then... they turn out to be significant
 79 2016-04-13 14:26:47	0|jonasschnelli|Somewhere during git re-set I forget to pass a "git checkout 2016_04_mdb".
 80 2016-04-13 14:29:22	0|GitHub190|[13bitcoin] 15mrbandrews opened pull request #7871: Manual block file pruning. (06master...06ba-manual6) 02https://github.com/bitcoin/bitcoin/pull/7871
 81 2016-04-13 15:04:18	0|sdaftuar|so, MAX_OPS_PER_SCRIPT includes op codes not executed?  i didn't expect that.
 82 2016-04-13 15:05:26	0|Chris_Stewart_5|^^^^
 83 2016-04-13 15:05:55	0|Chris_Stewart_5|even more interesting, if MAST is implemented what does that mean???
 84 2016-04-13 15:06:22	0|Chris_Stewart_5|if I understasnd correctly MAST only reveals branches of our control structure that are actually executed
 85 2016-04-13 15:06:27	0|sipa|exactly
 86 2016-04-13 15:06:36	0|sipa|and for the ones not executed, you give their hash
 87 2016-04-13 15:07:01	0|Chris_Stewart_5|sipa: Does MAST change the data structure from a List to a Tree inside of interpreter?
 88 2016-04-13 15:07:09	0|sipa|that depends on the implementation
 89 2016-04-13 15:07:21	0|sipa|it's just a generic idea
 90 2016-04-13 15:07:31	0|sipa|i haven't looked at jl2012's specific proposal
 91 2016-04-13 15:08:26	0|Chris_Stewart_5|sipa: Going off what we were talking about yesterday, are we constrained to realistically implementing this as a list for fear of unintended consensus changes?
 92 2016-04-13 15:08:37	0|sipa|no
 93 2016-04-13 15:08:49	0|sipa|no script versions can easily use a completely independent interpreter
 94 2016-04-13 15:08:54	0|sipa|*new script versions
 95 2016-04-13 15:08:57	0|Chris_Stewart_5|ahh ok
 96 2016-04-13 15:10:54	0|jl2012|sipa: It's like your tree signature. I just compact everyone to become 3 arguments: position, path, script . (Actually I borrowed your original segwit code when the commitment was a Merkle tree)
 97 2016-04-13 15:13:05	0|jl2012|and the depth is implied by the size of path
 98 2016-04-13 15:14:59	0|jl2012|Chris_Stewart_5: it's very similar to P2SH and P2WSH. Just with 2 extra arguments
 99 2016-04-13 15:16:04	0|Chris_Stewart_5|jl2012: I understand that part, I for specific implementation details I"m getting caught up on 'Position' and 'Path' and how they are different
100 2016-04-13 15:16:28	0|Chris_Stewart_5|It seems that if you have the path that the script takes in the tree you could derive its position..
101 2016-04-13 15:17:17	0|jl2012|First you divide the size of Path by 32, which is the Depth of the tree
102 2016-04-13 15:17:52	0|instagibbs|Chris_Stewart_5, I believe path is the hashes in the tree, the position will tell you how to build the branch
103 2016-04-13 15:18:08	0|instagibbs|jl2012, you might want to make it explicit, as I had trouble understanding it first go around as well
104 2016-04-13 15:18:16	0|instagibbs|and if I'm wrong, doubly so :P
105 2016-04-13 15:18:33	0|jl2012|instagibbs, yes, you are right
106 2016-04-13 15:18:38	0|Chris_Stewart_5|hmm ok
107 2016-04-13 15:18:57	0|instagibbs|I had to read between the lines tbh, since path had to be multiple of 32 bytes, i inferred it was sha hash
108 2016-04-13 15:19:25	0|jl2012|for Depth (d), you may have at most 2^d possible Position
109 2016-04-13 15:19:45	0|jl2012|Position = 0 means the leftmost position in the tree
110 2016-04-13 15:20:11	0|Chris_Stewart_5|jl2012: Leftmost... what exactly doe that mean? THe left most leaf node if you were to draw the tree out?
111 2016-04-13 15:20:17	0|instagibbs|yes
112 2016-04-13 15:20:21	0|jl2012|yes
113 2016-04-13 15:20:45	0|jl2012|same as the Merkle Root in the block header
114 2016-04-13 15:21:07	0|instagibbs|jl2012, mind if I write a clarification text? do I PR directly against the bip repo or yours?
115 2016-04-13 15:21:47	0|jl2012|instagibbs, please feel free, just a direct PR to BIP repo, thanks
116 2016-04-13 15:23:32	0|Chris_Stewart_5|So path is essentially a vector of sha256 hashes.. why do we exactly need the position arg again? instagibbs said for building the branch, can you be more explicit than that? Some how reconstructing the script from the hashes?
117 2016-04-13 15:25:07	0|sipa|Chris_Stewart_5: if your leaves are (a,b,c,d) then root=H(H(a,b),H(c,d)), right?
118 2016-04-13 15:25:15	0|Chris_Stewart_5|yes
119 2016-04-13 15:25:27	0|sipa|if you want leaf b, you need to reveal a and H(c,d), right?
120 2016-04-13 15:25:39	0|Chris_Stewart_5|yes
121 2016-04-13 15:25:40	0|sipa|so your path would be (a,H(c,d))
122 2016-04-13 15:26:06	0|sipa|if you want leaf c, however, you reveal (d,H(a,b))
123 2016-04-13 15:26:34	0|Chris_Stewart_5|so position = 3?
124 2016-04-13 15:26:49	0|sipa|if you reveal b, position = 1
125 2016-04-13 15:26:52	0|sipa|if you reveal c, position = 2
126 2016-04-13 15:26:57	0|sipa|etc
127 2016-04-13 15:26:58	0|Chris_Stewart_5|ahh zero based index
128 2016-04-13 15:27:01	0|Chris_Stewart_5|gotcha. Thanks.
129 2016-04-13 15:27:03	0|sipa|of course :p
130 2016-04-13 15:28:27	0|Chris_Stewart_5|lol
131 2016-04-13 15:29:11	0|jl2012|programmers count from 0
132 2016-04-13 15:31:05	0|Chris_Stewart_5|I think I forgot to turn on the CS part of my brain this morning... it was almost like I was a normal person for a while :P
133 2016-04-13 15:31:40	0|Chris_Stewart_5|*looks around for more coffee*
134 2016-04-13 15:49:33	0|instagibbs|jl2012, https://github.com/bitcoin/bips/pull/369
135 2016-04-13 15:49:35	0|instagibbs|let me know of quibbles etc
136 2016-04-13 15:51:34	0|jl2012|thanks!
137 2016-04-13 15:52:52	0|instagibbs|the merkle branch also has to be "minimal", but I figure anyone who skims the merkle root function would arrive at that
138 2016-04-13 15:53:04	0|instagibbs|and really should just copypasta that function if need be
139 2016-04-13 15:53:41	0|instagibbs|maybe branch singular already means that, no idea
140 2016-04-13 15:59:54	0|wumpus|jonasschnelli: hahah oops! good to know that's the issue, that was an old branch
141 2016-04-13 16:02:12	0|wumpus|I remember having made a similar mistake at least once, testing another branch then I thought I was testing, then spending quite some time debuggin why it didnt work as expected. I think around the time of the boost to evhttpd switch.
142 2016-04-13 16:09:00	0|jl2012|instagibbs: what do you mean by minimal?
143 2016-04-13 16:11:18	0|instagibbs|it only deals with the nodes it needs to compute a single path
144 2016-04-13 16:14:47	0|jl2012|I think it's implied by the design
145 2016-04-13 16:32:52	0|Chris_Stewart_5|probably better to be explicit..
146 2016-04-13 16:34:33	0|instagibbs|Copying and pasting the tiny function call is pretty explicit and no work
147 2016-04-13 16:55:50	0|jl2012|Chris_Stewart_5, instagibbs: more comments added to the reference implementation https://github.com/jl2012/bips/blob/bip114ref/bip-0114.mediawiki#Reference_Implementation
148 2016-04-13 16:56:24	0|instagibbs|ACK :)
149 2016-04-13 19:54:32	0|jonasschnelli|Now correct: LMDB branch sync from random peers took 2h 30min up to progress=1
150 2016-04-13 19:54:43	0|jonasschnelli|Now comparing reindex with default dbcache
151 2016-04-13 19:56:44	0|jonasschnelli|First shutdown of LMDB IBDed node with dbcache=9000 took just a couple of seconds (write speed didn't felt different to leveldb).
152 2016-04-13 20:20:23	0|GitHub79|[13bitcoin] 15morcos opened pull request #7874: Improve AlreadyHave (06master...06speedAlreadyHave) 02https://github.com/bitcoin/bitcoin/pull/7874
153 2016-04-13 23:38:26	0|cfields_|mm, what's the real-world use-case for getaddednodeinfo rpc?
154 2016-04-13 23:39:08	0|cfields_|the resolving logic is kinda wonky, and i'm not sure it's worth trying to maintain compatibility with the net refactor
155 2016-04-13 23:42:34	0|cfields_|BlueMatt_: ^^. Looks like you added it. In particular, the issue is that it (for dns entires) it does a resolve in the rpc, though that doesn't represent the ips that the network thread will end up trying
156 2016-04-13 23:44:55	0|cfields_|so I'm not sure that it's really worth trying to enumerate them. seems like keeping a map of dns->resolved would be enough to determine if a dns entry is connected or not, which i think is the useful info there?
157 2016-04-13 23:57:24	0|sipa|cfields_: i doubt it is important to keep its exact semantics
158 2016-04-13 23:59:43	0|cfields_|ok