1 2016-06-25 00:01:26 0|randy-waterhouse|ok, another 'devil's advocate' tester on-board segwit test smoothly ... thnx gmaxwell, sipa wumpus, etc ... and many others, good job.
2 2016-06-25 00:04:54 0|Lauda|Does the dbcache=N parameter only start using up additional memory during block processing?
3 2016-06-25 00:05:54 0|gmaxwell|Lauda: I think what you're asking is if it will use it right away, no it won't-- only has there is more data read into the cache
4 2016-06-25 00:06:28 0|Lauda|gmaxwell all that is required is "dbcache=3000" in bitcoin.conf right (since mine is practically empty) and I want to run another reindex over night?
5 2016-06-25 00:06:41 0|Lauda|since*
6 2016-06-25 00:06:42 0|gmaxwell|right!
7 2016-06-25 00:06:49 0|Lauda|Okay great. Thanks!
8 2016-06-25 00:07:25 0|Lauda|The reindex issue is indepdant of the blockchain, i.e. it doesn't matter whether one tests on testnet or mainnet (someone asked me this)?
9 2016-06-25 00:07:31 0|Lauda|independant*
10 2016-06-25 00:16:26 0|midnightmagic|i can convert my testnet node(s) to segwitty. is it time?
11 2016-06-25 03:36:48 0|jl2012|What would happen if I upgrade my node after segwit is active and there are already segwit blocks on the chain?
12 2016-06-25 06:21:40 0|GitHub136|[13bitcoin] 15paveljanik opened pull request #8261: The bit field is shown only when status is "started" (06master...0620160625_sw_getblockchaininfo_bit) 02 https://github.com/bitcoin/bitcoin/pull/8261
13 2016-06-25 06:30:52 0|gmaxwell|jl2012: it reorgs back to before segwit activated.
14 2016-06-25 06:31:06 0|gmaxwell|jl2012: then will download blocks as needed.
15 2016-06-25 06:32:49 0|jl2012|Same for other soft forks like csv?
16 2016-06-25 06:33:02 0|jl2012|I think it has to be
17 2016-06-25 06:33:53 0|gmaxwell|jl2012: it arguably should be, but we haven't done that before.
18 2016-06-25 06:34:09 0|gmaxwell|for segwit it had to go get the witness data in order to serve blocks
19 2016-06-25 06:34:46 0|gmaxwell|we'd talked about doing it in the past for other soft forks but I think we thought it would be harder to implement than it turned out to be.
20 2016-06-25 06:35:40 0|gmaxwell|(and unlikely to really matter that much unless the reason you were upgrading was to fight a network split, in which case the invalidateblock rpc can be used)
21 2016-06-25 06:36:36 0|jl2012|But theoretically I may be following an invalid chain
22 2016-06-25 06:41:32 0|gmaxwell|Potentially; though if there were a large invalid chain we would make loud announcements to recommend the use of the invalidateblock command. I think now that the code exists, we'd likely use it in the future.
23 2016-06-25 07:54:20 0|wumpus|segwit testnet node: 213.46.222.31:18333
24 2016-06-25 10:23:52 0|Lauda|What is the main bottleneck for reindex, storage speed or CPU processing?
25 2016-06-25 10:29:25 0|gmaxwell|probably depends on the hardware, on my laptop I think it's IO. on systems with faster IO, I think its cpu inside leveldb code... at least with default dbcache. with dbcache cranked, its likely cpu elsewhere in bitcoin.
26 2016-06-25 10:30:20 0|Lauda|Hmm, seems like the last 20-30 weeks are taking forever
27 2016-06-25 10:30:44 0|wumpus|Lauda: unless you have a very fast disk/ssd, or increase the dbcache, i/o will be your main bottleneck
28 2016-06-25 10:30:58 0|Lauda|I've just checked in on my node, and some bans say e.g. until June 19th but the nodes are still banned. Do these get removed after a restart or something went wrong?
29 2016-06-25 10:31:00 0|wumpus|Lauda: you can easily check though: is CPU maxed out?
30 2016-06-25 10:31:06 0|Lauda|It isn't
31 2016-06-25 10:31:23 0|Lauda|DBcache 3GB
32 2016-06-25 10:31:40 0|wumpus|(unless you have changed the number of script verification threads, bitcoind will max out your CPU cores in initial sync when it's not i/o bound)
33 2016-06-25 10:32:48 0|Lauda|Okay so even with 3GB dbcache that's still not enough
34 2016-06-25 10:33:05 0|wumpus|I have a branch to run bitcoind db-less: https://github.com/laanwj/bitcoin/tree/2016_04_dummy_db it does mean it loses all data when e.g. bitcoind crashes
35 2016-06-25 10:34:49 0|wumpus|but the utxo and block index is simply stored in a flat file, which is loaded at startup and written at shutdown
36 2016-06-25 10:35:02 0|Lauda|That's interesting and those times are amazing in comparison to leveldb
37 2016-06-25 10:35:14 0|wumpus|if you can afford the memory :)
38 2016-06-25 10:35:59 0|wumpus|though it uses less memory than keeping everyting in the dbcache, and doesn't have the issue that the cache is not seeded at startup
39 2016-06-25 10:37:15 0|wumpus|I think research and experimentaitno how to best store the utxo set is in order
40 2016-06-25 10:38:06 0|Lauda|The move towards SSDs should definitely help with this, but the industry is not there yet..
41 2016-06-25 10:38:45 0|Lauda|I can afford 4 GB on this machine, but it still takes a fair amount of time.
42 2016-06-25 10:39:32 0|wumpus|memristor would be nice
43 2016-06-25 10:40:24 0|wumpus|but no matter what, also at some point, unrestrained growth of the utxo set needs to be addressed
44 2016-06-25 10:40:54 0|Lauda|^
45 2016-06-25 10:45:23 0|sipa|wumpus: we should try switching to a model where all utxos are stored as separate db entries, rather than in a vector of unspends per txid
46 2016-06-25 10:46:48 0|wumpus|but it may well be we're running against the limits of what databases can (with good performance) handle, which means there is no room for scaling there at all
47 2016-06-25 10:47:33 0|gmaxwell|gigantic cuckoo hash table. with a update log. :P
48 2016-06-25 10:47:35 0|wumpus|sipa: yes, that would be an interesting experiment too
49 2016-06-25 10:48:27 0|wumpus|also the access pattern is essentially random, so the only type of caching that helps very well is keep everything
50 2016-06-25 10:48:43 0|gmaxwell|sipa: the ripple people have claimed that leveldb performance falls off a cliff with more than some threshold number of entries (I believe they were storing every transaction in it)
51 2016-06-25 10:49:20 0|sipa|gmaxwell: i think they don't have application level caching
52 2016-06-25 10:49:31 0|wumpus|well I'm not actually sure how random the access pattern is, but it looks like that from a disk perspective with the current organization
53 2016-06-25 10:49:57 0|wumpus|it's very possible for optimizations to be possible based on sorting utxos smartly which are expected to be accessed together? I don't know
54 2016-06-25 10:50:18 0|gmaxwell|sipa: sure, but that wouldn't change the performance of the underlying database.
55 2016-06-25 10:50:34 0|wumpus|it's not like the other databases that we tried perfomed better
56 2016-06-25 10:50:49 0|wumpus|leveldb still seems, all in all, the best perforing on-disk database for utxo storage
57 2016-06-25 10:51:04 0|gmaxwell|Ripple folks created their own.
58 2016-06-25 10:51:17 0|gmaxwell|(and also suggested we might be interested in using it)
59 2016-06-25 10:51:22 0|wumpus|lmdb looked promising but it has it's own performance cliff
60 2016-06-25 10:51:31 0|wumpus|(depending on the amount of memory in the system, it seems)
61 2016-06-25 10:52:00 0|wumpus|what license is it under?
62 2016-06-25 10:52:56 0|wumpus|I could give it a try pretty easily
63 2016-06-25 10:53:03 0|GitHub74|[13bitcoin] 15btccode opened pull request #8262: Forgetaddress 0.12 (060.12...06forgetaddress-0.12) 02 https://github.com/bitcoin/bitcoin/pull/8262
64 2016-06-25 10:53:46 0|gmaxwell|I think it's permissively licensed, looking for it now
65 2016-06-25 10:53:55 0|GitHub2|[13bitcoin] 15btccode closed pull request #8262: Forgetaddress 0.12 (060.12...06forgetaddress-0.12) 02 https://github.com/bitcoin/bitcoin/pull/8262
66 2016-06-25 10:54:42 0|sipa|gmaxwell: well is it read performance or write performance that is bad?
67 2016-06-25 10:55:40 0|wumpus|with leveldb it's read performance, and also latency
68 2016-06-25 10:56:23 0|wumpus|write performance of leveldb is quite good, I suppose because it writes consecutive files
69 2016-06-25 10:56:57 0|gmaxwell|https://bitcointalk.org/index.php?topic=1004943.0
70 2016-06-25 10:56:59 0|wumpus|but no database likes huge databases + random seek patterns for reads
71 2016-06-25 10:57:53 0|wumpus|https://github.com/vinniefalco/nudb
72 2016-06-25 10:57:56 0|gmaxwell|(thats why I made the half serious suggestion of a gigantic hash table)
73 2016-06-25 10:58:56 0|wumpus|lmdb read latency seems to be - on average- better, but its writing is worse than leveldb, I think it does more random writing
74 2016-06-25 10:59:45 0|GitHub121|13bitcoin/06master 14b0be3a0 15Wladimir J. van der Laan: doc: Mention Windows XP end of support in release notes...
75 2016-06-25 10:59:45 0|GitHub121|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/5cdc54b4b62d...63fbdbc94d76
76 2016-06-25 10:59:46 0|GitHub121|13bitcoin/06master 1463fbdbc 15Wladimir J. van der Laan: Merge #8240: doc: Mention Windows XP end of support in release notes...
77 2016-06-25 10:59:50 0|GitHub49|[13bitcoin] 15laanwj closed pull request #8240: doc: Mention Windows XP end of support in release notes (06master...062016_06_windows_xp) 02 https://github.com/bitcoin/bitcoin/pull/8240
78 2016-06-25 11:00:09 0|gmaxwell|as every read would simply be one or two random disk accesses... and its hard to do better than that. it's just writing is awful. (e.g. end up with read-write-write to update a log, with both sequential reads and writes, and if the table needs to be resized woe is you).
79 2016-06-25 11:00:40 0|wumpus|going to try nudb when I have some time
80 2016-06-25 11:01:17 0|wumpus|unfortunately we also do a lot of writing, at least during initial sync, every utxo read is updated and written back
81 2016-06-25 11:01:36 0|wumpus|so making reading much faster at the expense of writing is going to give yo mixed results
82 2016-06-25 11:02:52 0|wumpus|has research been done on utxo access patterns? e.g. are more recent blocks more often accessed, or the other way around, or are there other regularities that could be used?
83 2016-06-25 11:04:10 0|gmaxwell|Spending is more freuqntly from recently created utxo.
84 2016-06-25 11:04:55 0|wumpus|interesting
85 2016-06-25 11:06:14 0|gmaxwell|I would expect naievely that the expected lifetime of a utxo is how long it's lived so far. If something had made it a year without being spent, you should expect it to last another year. But beyond knowing that an unusually large number of utxo have short lives, I've not done anything to try to verify this hypothesis.
86 2016-06-25 11:07:32 0|gmaxwell|we could probably construct fairly elaborate predictions using other features like how many txouts were in the creating transactions, reuse of the pubkey, and the amount of the coin.
87 2016-06-25 11:10:17 0|gmaxwell|(or even using non-fungibility-- a coin is likely to be spent soon if its recent ancestors were spent soon)
88 2016-06-25 11:10:56 0|sipa|wumpus: that's why the "fresh" optimization helps a lot... we create utxo entries indide the database, and fully spend them before they even hit disk
89 2016-06-25 11:11:07 0|sipa|s/inside the database/inside the cache/
90 2016-06-25 11:11:41 0|gmaxwell|with the cache turned way up, the whole initial sync runs without writing the chainstate until the end.
91 2016-06-25 11:14:26 0|gmaxwell|oh, seems nudb is a big hashtable (uses external storage for values)
92 2016-06-25 11:14:43 0|sipa|it keeps the entire keyset in memory?
93 2016-06-25 11:15:10 0|gmaxwell|no, the keys are an a file. sounds like it's chunked so it can independantly resize sub tables.
94 2016-06-25 11:20:28 0|wumpus|isn't the fact that nudb is insert-only a problem? we delete and change entries a lot
95 2016-06-25 11:22:52 0|wumpus|gmaxwell: would be a good research project to investigate that hypothesis in detail, and see if it is possible to optimize storage based on those predictions/assumptions. Maybe one huge key/value store is not the best way to handle this
96 2016-06-25 11:23:00 0|gmaxwell|hm. I thought it could delete keys but not the values in external storage.
97 2016-06-25 11:24:28 0|gmaxwell|Oh I see what you mean there... I hadn't caught that implication before.. that effect is more or less why caching smaller than the utxo set in memory is still effective, but depending on the geometry of the effect it might make sense to have two databases.. so that the high access parts are in something with low log(n) costs.
98 2016-06-25 11:29:44 0|gmaxwell|LOL, totally offtopic: https://lkml.org/lkml/2016/3/31/1109
99 2016-06-25 11:30:57 0|btcdrak|inb4 linus splats him
100 2016-06-25 11:35:48 0|gmaxwell|it's old, just turned up in a random google search
101 2016-06-25 11:36:08 0|btcdrak|oh it's an april fool!
102 2016-06-25 11:38:48 0|Lauda|Error reading from database, shutting down 15 weeks left :<
103 2016-06-25 11:39:22 0|Lauda|I think I can still get detect results if I test another version and break at 15 weeks left?
104 2016-06-25 11:39:25 0|Lauda|decent*
105 2016-06-25 11:42:41 0|gmaxwell|do you know why it failed?
106 2016-06-25 11:42:57 0|gmaxwell|if you're benchmarking you can just compare to a common height.
107 2016-06-25 11:43:05 0|gmaxwell|e.g. use the timestamps in the log
108 2016-06-25 11:44:11 0|Lauda|I can't be sure. It's possible that my HDD disconnected for a second (the cable seems a bit unstable if touched).
109 2016-06-25 11:44:39 0|spudowiar|Lauda: check dmesg for I/O errors?
110 2016-06-25 11:45:18 0|Lauda|Okay then, I'll compare the timestamp of the same height. Running a test on a revision before the reindex changes now
111 2016-06-25 11:45:32 0|Lauda|I'll check system error log (these tests are on Windows not Unix).
112 2016-06-25 11:47:57 0|Lauda|http://pastebin.com/Zau75AHY it doesn't tell me much.
113 2016-06-25 12:36:33 0|sipa|Lauda: you have debug.log.
114 2016-06-25 12:36:34 0|sipa|?
115 2016-06-25 12:36:40 0|Lauda|Yes
116 2016-06-25 12:38:48 0|Lauda|The last entry is just an UpdateTip.
117 2016-06-25 12:39:29 0|Lauda|Comparing the partial data shows that re-index is much faster on the newer version than one before the re-index changes (at least on custom dbcache).
118 2016-06-25 12:42:27 0|sipa|yes, for a fair comparison you need to disable checkpoints
119 2016-06-25 12:42:44 0|sipa|before the reindex changes, signatures were always checked
120 2016-06-25 12:42:57 0|Lauda|How do I disable checkpoints?
121 2016-06-25 12:43:02 0|sipa|after thry're only checked past the last checkpoints
122 2016-06-25 12:43:09 0|sipa|-nocheckpoints i think
123 2016-06-25 12:43:38 0|Lauda|So I should delete this data (version before re-index changes) and run it again with that flag?
124 2016-06-25 12:47:09 0|MarcoFalke|no need to delete data
125 2016-06-25 12:47:52 0|Lauda|It's still re-indexing the build from 16-05.
126 2016-06-25 12:48:18 0|sipa|you can start over
127 2016-06-25 12:48:51 0|Lauda|How would I add that within the .conf file?
128 2016-06-25 12:49:19 0|sipa|checkpoints=0
129 2016-06-25 12:49:29 0|sipa|or nocheckpoints=1
130 2016-06-25 12:52:02 0|sipa|but please consult the help
131 2016-06-25 12:52:06 0|sipa|(bitcoind -help)
132 2016-06-25 12:55:55 0|Lauda|Okay thanks!
133 2016-06-25 13:12:24 0|Lauda|sipa is it normal that the wallet shows weird/non-existing transactions (date-wise) during reindex?
134 2016-06-25 13:14:11 0|sipa|example?
135 2016-06-25 13:14:25 0|Lauda|http://i.imgur.com/J7YeKdu.png
136 2016-06-25 13:14:54 0|Lauda|e0a871f4897af619c4e0d8ab91d6c6f81e25d23f4dea421439b60e9c9dd8cb83
137 2016-06-25 13:15:01 0|Lauda|Received Time 2015-09-09 20:57:49
138 2016-06-25 13:15:04 0|Lauda|wallet shows yesterday
139 2016-06-25 13:15:45 0|Lauda|I don't even recognize these transactions, a (big) list of microtransactions (incoming and outgoing) for 24/06
140 2016-06-25 13:18:30 0|sipa|heh
141 2016-06-25 13:18:39 0|sipa|did you import some common brainwallet
142 2016-06-25 13:19:03 0|Lauda|No. I didn't do anything. I'm running this on my wallet machine (since my other one is down)
143 2016-06-25 13:19:10 0|Lauda|I have never used anything besides QT.
144 2016-06-25 13:19:35 0|Lauda|I think it wasn't showing on 24-06 nightly build.
145 2016-06-25 13:20:05 0|sipa|that seems unlikely :)
146 2016-06-25 13:21:38 0|Lauda|The dates seem correct for all transactions up to this point. There are surely a few hundred TX's stamped at this date now
147 2016-06-25 13:21:40 0|Lauda|Hmm..
148 2016-06-25 13:23:18 0|Lauda|My wallet.dat grew. I made a backup before I started reindex tests. It was ~400kb, now it is 4.6MB