1 2018-04-06 01:20:32	0|aj|right, different VMs, but same test_bitcion.exe, under wine coinselector_tests/knapsack_solver_test takes 581.8 seconds, under windows server 2016 w/ updates it takes 31.4s
  2 2018-04-06 01:21:30	0|sipa|is there a way to do profiling in wine?
  3 2018-04-06 01:22:50	0|aj|i figure i'll try a newer version of wine than what's in trusty first
  4 2018-04-06 01:39:31	0|aj|same version of wine on xenial seems no better, unsurprisingly; but, wine 3.0 at home seems much faster
  5 2018-04-06 01:43:15	0|aj|total of 236s for the entire test suite under wine 3.0 vs 193s under windows server; different hardware though
  6 2018-04-06 02:07:50	0|aj|wine1.8 from some random ppa on xenial also gets the entire test suite in 377s or so, different hardware yet again
  7 2018-04-06 02:08:48	0|aj|oh, bugger, that's wine3.0 as well, should have paid more attention when installing
  8 2018-04-06 02:10:13	0|bitcoin-git|[13bitcoin] 15lachlangreenbank opened pull request #12898: Comment cleanup and consistency (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/12898
  9 2018-04-06 05:02:44	0|karl2|sipa: you can use gprof with wine binaries as normal, I found out the other day. It wasn't particularly exciting stuff when I tried to track down job #2 timeout thing the other day, though. Maybe running different wine versions will trigger it, like aj is saying!
 10 2018-04-06 05:02:48	0|gribble|https://github.com/bitcoin/bitcoin/issues/2 | Long-term, safe, store-of-value · Issue #2 · bitcoin/bitcoin · GitHub
 11 2018-04-06 05:08:22	0|bitcoin-git|[13bitcoin] 15AkioNak opened pull request #12899: macOS: Prevent Xcode 9.3 build warnings (06master...06preventxcodebuildwarnings) 02https://github.com/bitcoin/bitcoin/pull/12899
 12 2018-04-06 06:17:08	0|bitcoin-git|[13bitcoin] 15martintibor40 opened pull request #12900: Update README.md (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/12900
 13 2018-04-06 06:17:44	0|bitcoin-git|[13bitcoin] 15fanquake closed pull request #12900: Update README.md (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/12900
 14 2018-04-06 09:11:30	0|bitcoin-git|[13bitcoin] 15practicalswift opened pull request #12901: build: Show enabled sanitizers in configure output (06master...06print-enabled-sanitizers-in-configure-output) 02https://github.com/bitcoin/bitcoin/pull/12901
 15 2018-04-06 10:17:50	0|maxmatics|hello world.
 16 2018-04-06 13:10:35	0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #12898: Comment cleanup and consistency (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/12898
 17 2018-04-06 13:49:55	0|bitcoin-git|[13bitcoin] 15sdaftuar opened pull request #12902: [qa] Handle potential cookie race when starting node (06master...062018-04-improve-dbcrash-restarts) 02https://github.com/bitcoin/bitcoin/pull/12902
 18 2018-04-06 14:36:16	0|stevenroose|Does bitcoin core have a utxo cache? If so, could anyone point me to the file where it is defined?
 19 2018-04-06 14:38:45	0|stevenroose|Aha, I found -dbcache, which is the size in MiB for the utxo db cache
 20 2018-04-06 14:39:54	0|stevenroose|I thought the cuckoocache was used as a sigcache.
 21 2018-04-06 14:40:06	0|stevenroose|Is it also used for the utxo cache?
 22 2018-04-06 14:41:21	0|sdaftuar|stevenroose: no, it is not.  see src/coins.h and src/coins.cpp for more information about the utxo cache.
 23 2018-04-06 14:44:42	0|stevenroose|sdaftuar: thanks!
 24 2018-04-06 14:45:07	0|stevenroose|from the init code, I see that the -dbcache is split in 3, of which one is "chain state cache", what is that one for?
 25 2018-04-06 15:09:16	0|bitcoin-git|[13bitcoin] 15sdaftuar opened pull request #12904: [qa] Ensure bitcoind processes are cleaned up when tests end (06master...062018-04-always-kill-bitcoind) 02https://github.com/bitcoin/bitcoin/pull/12904
 26 2018-04-06 15:36:10	0|sipa|stevenroose: for the UTXO set the layers are: (1) disk (with OS cache etc) (2) LevelDB's cache (3) Bitcoin Core's pcoinsTip (a CCoinsViewCache object)
 27 2018-04-06 15:36:31	0|setpill|is there any way to actively check for conflicting transactions in the mempool for a given tx?
 28 2018-04-06 15:37:29	0|sipa|stevenroose: there is another data set, the block index which is loaded entirely in memory, but is stored in a separate LevelDB database
 29 2018-04-06 15:37:59	0|sipa|so the 3 pieces -dbcache is split over is the blockindex leveldb cache, the chainstate leveldb cache (=utxo), and pcoinTip
 30 2018-04-06 15:38:18	0|setpill|i suppose trying to rebroadcast would give an error in some cases, but i am not sure how reliable that is, and i would prefer just simply checking without broadcasting
 31 2018-04-06 18:28:35	0|bitcoin-git|[13bitcoin] 15sdaftuar opened pull request #12905: [rpcwallet] Clamp walletpassphrase value at 100M seconds (06master...062018-04-wallet-encryption-timeout) 02https://github.com/bitcoin/bitcoin/pull/12905
 32 2018-04-06 18:50:47	0|sdaftuar|setpill: unfortunately i don't think we have great tools right now. in the next release, we'll have an rpc called "testmempoolaccept" which you could use to determine whether a given transaction would be accepted to your mempool, which might be along the lines of what you'd want?
 33 2018-04-06 18:50:59	0|sdaftuar|but it's tricky in general, because dealing with transaction chains is not easy
 34 2018-04-06 18:52:28	0|sdaftuar|for instance, if someone sends you a transaction that depends on another unconfirmed transaction, and then a third transaction conflicts with the parent and evicts it from the mempool, it's hard to tell that your transaction is indirectly conflicted as well
 35 2018-04-06 19:03:40	0|stevenroose|sipa: the block index I understand
 36 2018-04-06 19:03:52	0|stevenroose|"the chainstate leveldb cache (=utxo), and pcoinTip" -> so what exactly is the difference there?
 37 2018-04-06 19:04:06	0|stevenroose|they are both used to cache the utxo set, right?
 38 2018-04-06 19:10:49	0|jamesob|does POTENTIAL DEADLOCK DETECTED being logged by a node during functional test runs indicate something is definitely out of the ordinary?
 39 2018-04-06 19:21:22	0|stevenroose|sipa: oh you meant with (=utxo) that it's just the leveldb cache for the store that has the utxo data
 40 2018-04-06 19:21:48	0|stevenroose|ok, so the only thing core maintains itself is the CCoinsViewCache, right?
 41 2018-04-06 19:47:05	0|sipa|stevenroose: yes, but that's thr most important source of speedups
 42 2018-04-06 19:47:25	0|sipa|the leveldb cache helps ob systems with very slow i/o
 43 2018-04-06 19:47:58	0|stevenroose|sipa: the coinsCache map is from outpoint to utxo entry, right?
 44 2018-04-06 19:48:22	0|stevenroose|Doesn't that mean that for a new tx, the txhash is potentially added a lot of times?
 45 2018-04-06 19:49:12	0|stevenroose|in btcd, we have a structure where we map txid to utxoentry that has a (potentially sparse) map from index to output
 46 2018-04-06 19:49:22	0|stevenroose|but it
 47 2018-04-06 19:49:43	0|stevenroose|'s currently not cached, and indeed, that's one of the mayor performance bottlenecks
 48 2018-04-06 19:51:24	0|drexl|when an opcode has inputs, do these come from the stack?
 49 2018-04-06 19:52:31	0|sipa|stevenroose: i believe btcd's design was based on bitcoin core's previous
 50 2018-04-06 19:52:47	0|sipa|we switched from per-tx to per-txout in 0.15
 51 2018-04-06 19:53:27	0|sipa|leveldb deduplicates multiple key-value pairs with keys that share a prefix anyway, so on disk it's not all that impactful
 52 2018-04-06 19:54:04	0|sipa|and it simplifies the in-memory cache and serialization overhead for read/writes significantly
 53 2018-04-06 19:54:43	0|stevenroose|simplifies it a lot indeed
 54 2018-04-06 19:55:11	0|stevenroose|any numbers on extra memory usage /utxo stored?
 55 2018-04-06 19:55:18	0|sipa|500 MB on disk extra
 56 2018-04-06 19:55:52	0|stevenroose|is disk more important than memory?
 57 2018-04-06 19:55:59	0|sipa|imho, no
 58 2018-04-06 19:56:07	0|sipa|it's a small constant factor extra disk
 59 2018-04-06 19:56:20	0|sipa|but it makes memory usage for the cache faster, more effective, and more efficient
 60 2018-04-06 19:56:20	0|stevenroose|I mean for initila sync mostly, a big memory cache can be very significant, no?
 61 2018-04-06 19:56:38	0|stevenroose|hmm
 62 2018-04-06 19:57:05	0|sipa|because now we don't need to load unrelated other unspent outputs into memory
 63 2018-04-06 19:57:09	0|sipa|when one output is being spent
 64 2018-04-06 19:59:06	0|stevenroose|oh, but you don't have to do that anyway, right?
 65 2018-04-06 19:59:16	0|stevenroose|the entry map (index -> output) is sparce
 66 2018-04-06 19:59:40	0|stevenroose|also on disk, you don't store entire txs, only the unspent outputs
 67 2018-04-06 19:59:45	0|sipa|yes
 68 2018-04-06 19:59:55	0|sipa|but then you need a complex operation to write changes to disk
 69 2018-04-06 20:00:07	0|sipa|and i don't see how you can easily perform the freshness optimization on that
 70 2018-04-06 20:00:58	0|sipa|(that's the idea that if you create a utxo in memory, and then later spend it, in memory, it can be deleted from the cache, and nothing ever needs to hit disk, because both the create-txout and spent-txout operations are idempotent)
 71 2018-04-06 20:02:04	0|stevenroose|hmm, I only just started looking into this more deeply, but I don't see how storing utxos grouped per tx change that
 72 2018-04-06 20:02:28	0|sipa|well it means you can only do that optimization if all utxos of a tx are spent before a flush
 73 2018-04-06 20:04:07	0|sipa|i guess you can have a hybrid where you store them per-txout on disk, but with shared txids in memory
 74 2018-04-06 20:04:17	0|sipa|i looked into doing that, but the memory usage savings are tiny
 75 2018-04-06 20:05:07	0|stevenroose|I don't know what optimization you talk about tbh. let's say you add a tx with two outputs (you always add whole txs right? I don't think single outpoints make much sense), so you git txid -> (o1, o2)
 76 2018-04-06 20:05:32	0|sipa|yup
 77 2018-04-06 20:05:51	0|stevenroose|then, before a flush, o2 gets spent, so you keep txid -> (o1) and then when you flush, I don't see the overhead over writing txid1 -> o over txid -> (o1)
 78 2018-04-06 20:06:08	0|stevenroose|(opposite overhead)
 79 2018-04-06 20:06:28	0|sipa|that's actually a pretty significant overhead
 80 2018-04-06 20:06:43	0|sipa|you need a dynamically allocated structure for variable-length output array
 81 2018-04-06 20:07:37	0|sipa|it's hard to compare what we're talking about because there are so many variations, depending on how you do things in memory vs on disk
 82 2018-04-06 20:10:48	0|stevenroose|I realize, trying to see if it's worth turning the whole structure here upside down :)
 83 2018-04-06 20:11:09	0|sipa|https://github.com/bitcoin/bitcoin/pull/10195
 84 2018-04-06 20:18:18	0|stevenroose|Hmm, not reassuring :)
 85 2018-04-06 20:20:11	0|sipa|what could be done is change the in-memory representation slightly where you have two maps, one txid->(int,coinbase,height,#unspents) and another (int,vout)->(amount,script)
 86 2018-04-06 20:20:33	0|sipa|rather than the single map we currently have (txid,vout)->(amount,script,coinbase,height)
 87 2018-04-06 20:21:03	0|sipa|but it becomes a lot more complicated in the presence of more complicated cache flushing strategies
 88 2018-04-06 20:22:01	0|sipa|perhaps if average txn over time gain more txouts this becomes better
 89 2018-04-06 20:23:32	0|stevenroose|well, mostly to avoid changing all the db code etc, I will probably start with a cache on the current system
 90 2018-04-06 20:23:55	0|stevenroose|the argument of txs becoming bigger could make sense, though
 91 2018-04-06 20:24:37	0|stevenroose|if we want actual unlinkability with CT, coinjoin-like structures will become increasingly common
 92 2018-04-06 20:25:35	0|stevenroose|but yeah the simplicity of a txout based structure is also very compelling
 93 2018-04-06 20:25:52	0|sipa|in any case, i'm very skeptical that any attempts to share the txids and other tx metadata in memory are worthwhile
 94 2018-04-06 20:26:40	0|sipa|and unfortunately the CCoinsViewCache code is pretty complicated as it takes advantage of a bunch of tricks that are specific to utxo data
 95 2018-04-06 20:27:05	0|sipa|so it's not trivial to just drop in another cache design
 96 2018-04-06 20:27:20	0|sipa|you may want to talk to eklitzke
 97 2018-04-06 20:28:11	0|stevenroose|"a bunch of tricks that are specific to utxo data" hmm
 98 2018-04-06 20:28:46	0|sipa|well in particular the freshness optimization
 99 2018-04-06 20:28:50	0|sipa|that i mentioned above
100 2018-04-06 20:28:58	0|stevenroose|I was going through it a bit, will certainly do some more consideration before i dive into coding
101 2018-04-06 20:29:11	0|stevenroose|yeah, I'll look into that
102 2018-04-06 20:29:27	0|sipa|it seems like a very hard first project if you're not already somewhat familiar with the codebase :)
103 2018-04-06 20:29:41	0|stevenroose|how does it handle crashes? keeping latest flushed height or so and rebuilding newer blocks from disk in  case of crash?
104 2018-04-06 20:29:48	0|sipa|ah!
105 2018-04-06 20:31:08	0|sipa|https://github.com/bitcoin/bitcoin/pull/10148
106 2018-04-06 20:32:27	0|stevenroose|thanks!
107 2018-04-06 20:32:52	0|stevenroose|when talking about cached "chain state", I suppose new blocks are always directly written to disk, no?
108 2018-04-06 20:33:02	0|sipa|the chain state is the utxo set
109 2018-04-06 20:33:12	0|sipa|blocks are stored completely independently
110 2018-04-06 20:33:13	0|stevenroose|ok, that clears it up
111 2018-04-06 20:34:42	0|sipa|basically on disk there is a marker with a hash of a block that means "utxos created or spent after this block MAY be present" and another that means "all utxos created or spent before this block MUST be on disk"
112 2018-04-06 20:35:02	0|sipa|and at startup all blocks before the first and the second hash are replayed and applied to the UTXO set
113 2018-04-06 20:35:32	0|sipa|this approach means we don't need to write the whole cache utxo set in one atomic operation, and also don't need to remember old deleted entries
114 2018-04-06 20:37:58	0|ThinkOfANick|sipa: Wait, why not remember old entries?
115 2018-04-06 20:38:17	0|sipa|ThinkOfANick: because you want to save memory
116 2018-04-06 20:38:49	0|stevenroose|sipa: I'm having difficulty to see how those two block locators are not the same
117 2018-04-06 20:39:06	0|setpill|sdaftuar: perhaps ill have to resort to adding all the "from" addresses of the entire chain as watch-only addresses...
118 2018-04-06 20:39:19	0|sipa|stevenroose: you have 1 million modified UTXO entries in memory
119 2018-04-06 20:39:40	0|sipa|stevenroose: you can't construct a single batch to write them all at once to memory, as that batch would be gigabytes in side
120 2018-04-06 20:39:46	0|sipa|so you write part of it
121 2018-04-06 20:39:48	0|sipa|*size
122 2018-04-06 20:40:10	0|sipa|say you last full flush was at block 400000
123 2018-04-06 20:40:11	0|sipa|the current tip is 450000
124 2018-04-06 20:40:26	0|sipa|that means your cache may include entries from up to block 450000
125 2018-04-06 20:40:29	0|stevenroose|(partial flush, right?)
126 2018-04-06 20:40:38	0|stevenroose|ah full
127 2018-04-06 20:40:42	0|stevenroose|srry didn't see
128 2018-04-06 20:41:01	0|sipa|you're now doing a partial flush, writing just some subset of UTXO cache entries to disk
129 2018-04-06 20:41:29	0|sipa|that may include UTXOs created up to block 450000 (or may miss things that were created between 400000 and 450000)
130 2018-04-06 20:41:48	0|sipa|but it's not guaranteed to contain everything up to 450000 (in fact, because you know it's partial, it can't be)
131 2018-04-06 20:41:57	0|sipa|so the range you write is 400000..450000
132 2018-04-06 20:42:02	0|sipa|and if a crash happens then and there
133 2018-04-06 20:42:14	0|sipa|at startup you'll need to replay everything between 400000 and 450000
134 2018-04-06 20:42:57	0|sipa|because all utxo creation and spending operations are idempotent, it never hurts to replay an operation that was already applied
135 2018-04-06 20:43:11	0|stevenroose|I see, I just don't see how the 450000 is not just the same as the tip.. you also said "utxos created or spent after this block MAY be present" that must have been "before" then?
136 2018-04-06 20:43:28	0|sipa|450000 is the tip
137 2018-04-06 20:43:31	0|sipa|in this scenario
138 2018-04-06 20:43:33	0|stevenroose|yeah
139 2018-04-06 20:43:40	0|sipa|no
140 2018-04-06 20:43:50	0|sipa|utxos created or spent after 400000 MAY be present
141 2018-04-06 20:43:58	0|sipa|utxos created or spent before 450000 MUSt be present
142 2018-04-06 20:44:03	0|sipa|so the range is 400000..450000
143 2018-04-06 20:44:25	0|sipa|eh, i guess this is vague
144 2018-04-06 20:44:25	0|stevenroose|> utxos created or spent before 450000 MUSt be present
145 2018-04-06 20:44:34	0|stevenroose|yeah that's where you lost me
146 2018-04-06 20:44:36	0|sipa|let me reformulate
147 2018-04-06 20:44:56	0|sipa|all operations up to block 400000 are guaranteed to be on disk
148 2018-04-06 20:44:57	0|stevenroose|you said "MUST be before" and "MAY be after" so I assume that is the same one
149 2018-04-06 20:45:10	0|sipa|all operations between 400000 and 450000 may be present on disk, but are not guaranteed to be
150 2018-04-06 20:45:25	0|sipa|my "may" and "must" were very confusing before
151 2018-04-06 20:45:34	0|stevenroose|yeah, that's what confused me
152 2018-04-06 20:45:46	0|sipa|but things after 450000 are guaranteed to not be on disk
153 2018-04-06 20:45:55	0|stevenroose|so now let's assume there is already a persistent chain tip indicator, then you only need to keep one, right?
154 2018-04-06 20:46:07	0|sipa|well this is the chain tip indicator
155 2018-04-06 20:46:14	0|sipa|instead of a tip, it's a range
156 2018-04-06 20:46:33	0|stevenroose|that's why I was confused for it to be two. ok ok, yeah I got it. I just assumed you would always need a chaintip anyways
157 2018-04-06 20:46:45	0|sipa|in the ideal scenario the two are the same
158 2018-04-06 20:46:48	0|sipa|after a full flush
159 2018-04-06 20:46:53	0|stevenroose|one last thing I'm missing
160 2018-04-06 20:47:25	0|stevenroose|how to know when to update that consistence height
161 2018-04-06 20:47:28	0|stevenroose|(the first one of the range)
162 2018-04-06 20:47:35	0|sipa|ah, right now it's very simple
163 2018-04-06 20:48:09	0|sipa|when we start a flush operation, we check what the previous lower-height (the 400000) was, and update it to (that lower height, current tip)
164 2018-04-06 20:48:24	0|sipa|after a flush operation completes, it's replaced with (current tip, current tip)
165 2018-04-06 20:49:00	0|stevenroose|oh, but then you still need full flushes?
166 2018-04-06 20:49:06	0|stevenroose|so you can't update in partial writes?
167 2018-04-06 20:49:19	0|sipa|well we only have full flushes now
168 2018-04-06 20:49:26	0|sipa|but they're implemented as a sequence of partial flushes
169 2018-04-06 20:49:43	0|sipa|longer term i want a system where we have a background thread that's constantly flushing
170 2018-04-06 20:49:59	0|sipa|and is always "running behind" on the tip
171 2018-04-06 20:50:15	0|sipa|to give the memory db a chance to cache creates/spends that cancel each other out before writing
172 2018-04-06 20:50:36	0|sipa|but keeping track of which is the lower hash in the range in that system is more complicated
173 2018-04-06 20:51:08	0|sipa|it's basically the lowest height of which you either have an unwritten create or unwritten spend
174 2018-04-06 20:51:23	0|sipa|but it's more tricky in the presence of reorganizations
175 2018-04-06 20:51:34	0|sipa|https://github.com/bitcoin/bitcoin/blob/master/src/txdb.cpp#L102L137 <- current flushing to disk logic
176 2018-04-06 20:52:27	0|stevenroose|oooooooh
177 2018-04-06 20:52:55	0|stevenroose|ok, I thought you were doing a partial flush when the cache was full
178 2018-04-06 20:53:08	0|stevenroose|(like LRI fashion or so)
179 2018-04-06 20:53:12	0|sipa|nope
180 2018-04-06 20:53:19	0|stevenroose|but it's just a way to reduce the size of the transaction
181 2018-04-06 20:53:28	0|stevenroose|ldb transaction
182 2018-04-06 20:53:39	0|sipa|i've experimented with various approaches for MRU eviction from the cache etc
183 2018-04-06 20:53:41	0|stevenroose|oh yeah than you can just do all batches and update the pointer
184 2018-04-06 20:53:53	0|sipa|but they're basically all slower than what we're doing now (on fast hw at least)
185 2018-04-06 20:54:51	0|stevenroose|sipa: yeah you'd have the problem of knowing to what hash it's consistent. you'd need to keep heights in entries and iterate over all entries once and a while to see what the most recent dirty one is
186 2018-04-06 20:55:16	0|sipa|the *least* recent dirty one, yes
187 2018-04-06 20:55:30	0|sipa|thankfully, utxo entries already have a height
188 2018-04-06 20:55:33	0|stevenroose|excuseer :p
189 2018-04-06 20:55:57	0|sipa|unfortunately, that's the creation height and not really the modification height (which may differ in the case of a spend or a reorg)
190 2018-04-06 20:56:44	0|stevenroose|yeah I keep having a hard time picturing reorg handling there
191 2018-04-06 20:57:00	0|stevenroose|because when you don't have a txindex and you delete an entry, it's impossible to get it back :D
192 2018-04-06 20:57:11	0|sipa|oh, we have undo data
193 2018-04-06 20:57:14	0|sipa|the *.rev files
194 2018-04-06 20:57:16	0|stevenroose|do you keep like "revert objects" for the last X blocks
195 2018-04-06 20:57:23	0|stevenroose|ah
196 2018-04-06 20:58:10	0|stevenroose|how many are there? (thinking hardfork races here where two chains constantly catch up with each other and fuck old nodes)
197 2018-04-06 20:58:29	0|sipa|there is one per block
198 2018-04-06 20:58:36	0|sipa|and we prune them along with the blocks themselves
199 2018-04-06 20:58:54	0|stevenroose|(technically wouldnt be a hardfork in that case though)
200 2018-04-06 20:59:34	0|stevenroose|wait, why keep for all? or are they only like pointers to the actual data?
201 2018-04-06 20:59:55	0|sipa|because we need to be able to reorg?
202 2018-04-06 21:00:12	0|stevenroose|I haven't seen any of that data in btcd's codebase, let me dig to that tomorrow :)
203 2018-04-06 21:00:30	0|stevenroose|yeah I know, but well, reorging over half the chain is kinda unlikely, isn't it?
204 2018-04-06 21:00:38	0|sipa|yes
205 2018-04-06 21:00:45	0|stevenroose|I'd say at least before the last checkpoint makes no sense..
206 2018-04-06 21:00:54	0|sipa|checkpoints need to go away
207 2018-04-06 21:00:59	0|stevenroose|oh
208 2018-04-06 21:01:18	0|sipa|but yes, sure, it's unlikely that deep reorgs happen
209 2018-04-06 21:01:32	0|stevenroose|what's the fundamental problem with checkpoints?
210 2018-04-06 21:01:42	0|sipa|they confuse people
211 2018-04-06 21:01:44	0|sipa|:)
212 2018-04-06 21:01:59	0|sipa|they're seen as a security measure
213 2018-04-06 21:02:32	0|stevenroose|more as an efficiency tool :p I mean you can skip verification up to that point
214 2018-04-06 21:02:52	0|sipa|we have assumevalid for that now, which is far less invasive
215 2018-04-06 21:03:02	0|stevenroose|(f.e. when syncing a node I always "ask my friends for the latest block they trust" (i.e. check some explorers) and do --addcheckpoint
216 2018-04-06 21:03:06	0|sipa|it doesn't restrict which chain is valid
217 2018-04-06 21:03:20	0|sipa|it just skips validation for any block that is an ancestor of a known valid block
218 2018-04-06 21:03:34	0|sipa|but if the best chain we see is different than the assumevalid one, we'll accept it (after validating)
219 2018-04-06 21:03:55	0|stevenroose|ok yeah that's a better version of a checkpoint
220 2018-04-06 21:04:21	0|sipa|yes, assumevalid is updated from time to time, but we haven't modified checkpoints in years
221 2018-04-06 21:04:29	0|stevenroose|but checkpoints are also usefull against eclipse attacks when you're just getting started
222 2018-04-06 21:04:36	0|sipa|no
223 2018-04-06 21:05:00	0|stevenroose|at least they let you know something is up, no?
224 2018-04-06 21:05:06	0|sipa|they're useful against being spammed with low difficulty headers
225 2018-04-06 21:05:15	0|sipa|but that's independent of eclipse attacks
226 2018-04-06 21:05:38	0|sipa|we need backward headers sync to remove that dependency on checkpoints
227 2018-04-06 21:06:02	0|stevenroose|backward header sync?
228 2018-04-06 21:06:17	0|sipa|first learn the best header, and only then download headers along that path
229 2018-04-06 21:06:41	0|sipa|as opposed to downloading whatever header people give you, hoping that indeed it'll turn out to be one with more work than your current one
230 2018-04-06 21:06:51	0|sipa|(that's how it works now)
231 2018-04-06 21:07:43	0|stevenroose|how can you learn the best header?
232 2018-04-06 21:08:07	0|sipa|using a yet to be devised protocol :)
233 2018-04-06 21:08:08	0|stevenroose|asking all peers? (heighest checkpoint? :p)
234 2018-04-06 21:08:12	0|stevenroose|oh
235 2018-04-06 21:08:56	0|sipa|there are some ideas about random sampling, where someone can send you a merkle sum tree over all their headers, and then you randomly query it a number of times to see if they indeed have the distribution of pow they claim
236 2018-04-06 21:08:59	0|stevenroose|I recently thought about a backwards sync mechanism for initila utxo building. but I guess it's kinda not worth the effort when there is a good utxo cache
237 2018-04-06 21:10:10	0|sipa|and once you've done enough queries, you know they actually have a chain with a certain amount of work
238 2018-04-06 21:10:21	0|sipa|and if that amount of work is good enough, you can start downloading the actual headers
239 2018-04-06 21:10:49	0|setpill|sipa: wouldn't just believing the claimed amount of accumulated work + ban on lie work?
240 2018-04-06 21:11:00	0|stevenroose|are checkpoints really that bad?
241 2018-04-06 21:11:26	0|sipa|stevenroose: people seem to misunderstand that if checkpoints ever have an effect, bitcoin is broken
242 2018-04-06 21:11:46	0|sipa|i'm much more comfortable to have much weaker assumptions about correctness of the code
243 2018-04-06 21:11:56	0|sipa|(which includes the checkpoints)
244 2018-04-06 21:12:07	0|sipa|setpill: how is that better than what we have now?
245 2018-04-06 21:13:05	0|sipa|stevenroose: i don't think they're terrible, but we also don't really need them anymore, except for this tiny DoS concern
246 2018-04-06 21:13:06	0|setpill|sipa: last blocks are likely to have more pow behind them so are more expensive to maliciously craft
247 2018-04-06 21:13:19	0|stevenroose|"if checkpoints ever have an effect, bitcoin is broken" you mean that when code breaks validity of an old block that no one will validate because checkpoints?
248 2018-04-06 21:14:00	0|sipa|stevenroose: i mean that if checkpoints ever prevent the network from reorging to an attacker chain, it's clear that the concept of PoW itself is brokenb
249 2018-04-06 21:15:31	0|stevenroose|and what if they prevent *a new node* from syncing a wrong chain? that's what their main use is imho
250 2018-04-06 21:16:00	0|sipa|stevenroose: that's not what they do
251 2018-04-06 21:16:08	0|sipa|stevenroose: they just prevent OOM
252 2018-04-06 21:16:20	0|stevenroose|I mean even as a spam vector, a decent miner right now can prob create a quite significantly long chain that is use in size (1MB blocks) and has legit work
253 2018-04-06 21:16:36	0|sipa|yes, absolutely - that's exactly the one thing they still do
254 2018-04-06 21:16:36	0|stevenroose|s/use/huge/
255 2018-04-06 21:16:50	0|stevenroose|OOM?
256 2018-04-06 21:16:56	0|sipa|out of memory
257 2018-04-06 21:17:05	0|sipa|also, not actually blocks, just headers
258 2018-04-06 21:17:21	0|sipa|we don't download block data until a chain of validated headers is actually the best chain
259 2018-04-06 21:18:08	0|stevenroose|yeah true, so that would only work if eclipsed
260 2018-04-06 21:18:15	0|sipa|oh, and there is a known min amount of work
261 2018-04-06 21:18:21	0|sipa|independently of checkpoints
262 2018-04-06 21:18:42	0|stevenroose|"min amount of work at height x"?
263 2018-04-06 21:18:43	0|sipa|so we never accept a headers chain until it passes that point
264 2018-04-06 21:18:50	0|stevenroose|oh llike that
265 2018-04-06 21:18:52	0|stevenroose|cumulative
266 2018-04-06 21:18:59	0|stevenroose|thats neat
267 2018-04-06 21:19:49	0|setpill|sipa: interesting, i hadnt heard of that; is that documented somewhere?
268 2018-04-06 21:21:41	0|sipa|-minimumchainwork cmdline option
269 2018-04-06 21:22:49	0|stevenroose|well, thanks for the insights :)
270 2018-04-06 21:23:44	0|sipa|yw!
271 2018-04-06 21:26:12	0|setpill|sipa: ahh, so its another "checkpoint-esque" thing, as in a hardcoded value that gets updated periodically?
272 2018-04-06 21:26:55	0|setpill|for a second i was under the impression pow inflation somehow had a lower bound ^^'
273 2018-04-06 21:27:45	0|sipa|setpill: there is a minimum difficulty, but it's trivial
274 2018-04-06 21:28:09	0|setpill|yeah and wont help much against a malicious chain
275 2018-04-06 21:29:08	0|sipa|a single modern CPU thread can create a minimum-difficulty block in a few minute
276 2018-04-06 21:29:31	0|sipa|modern HW miners can make 1000s per second
277 2018-04-06 21:30:45	0|setpill|yeah, so i thought there was a higher-than-that, actually useful lower bound on difficulty based on something i was unaware of
278 2018-04-06 21:31:11	0|sipa|there is also the max-divide-by-4 rule for difficulty changes
279 2018-04-06 21:33:24	0|stevenroose|sipa: is that a consensus rule??
280 2018-04-06 21:33:30	0|sipa|yes
281 2018-04-06 21:33:32	0|stevenroose|was not aware of that
282 2018-04-06 21:33:37	0|sipa|always has been
283 2018-04-06 21:33:41	0|stevenroose|is there a max-multiply-by-x one?
284 2018-04-06 21:33:45	0|sipa|yes, by 4
285 2018-04-06 21:33:56	0|stevenroose|aha
286 2018-04-06 21:33:57	0|sipa|neither rule has ever been hit
287 2018-04-06 21:34:06	0|sipa|on mainnet at least
288 2018-04-06 21:34:23	0|stevenroose|I hate the zero-diff-after-20-mins testnet rule
289 2018-04-06 21:34:32	0|sipa|haha
290 2018-04-06 21:34:58	0|stevenroose|I used to try and testnet mine with an old butterfly labs jalapeno but somehow was never hitting one
291 2018-04-06 21:35:16	0|sipa|you need to set your timestamp in the future :)
292 2018-04-06 21:35:17	0|stevenroose|btcd didn't notify gbt long polls on the 20 minute hit
293 2018-04-06 21:35:42	0|stevenroose|also
294 2018-04-06 23:05:34	0|jtimon|sipa: re protocol to know the best chain: couldn't compact pow proofs be better for that than random sampling?
295 2018-04-06 23:06:10	0|jtimon|hi stevenroose
296 2018-04-06 23:08:38	0|sipa|jtimon: compact pow proofs need consensus rules
297 2018-04-06 23:08:42	0|sipa|bitcoin doesn't have those
298 2018-04-06 23:09:44	0|jtimon|sure, I mean assuming a compact proofs sf
299 2018-04-06 23:16:15	0|intcat|is there any (u)txo commitment design (close to becoming) a BIP?
300 2018-04-06 23:16:35	0|intcat|ive read some ml interaction between peter todd and bram cohen but it's a few years old and im not sure how much has happened on that since