1 2017-11-27 10:45:52	0|wallet42|What are the counter arguments to have a switch where the miner can set a couple of vbytes (10-50K) aside in a getblocktemplate that will include lowfee, oldest first transactions instead of highest paying?
  2 2017-11-27 11:08:40	0|sipa|wallet42: complicates the block construction code significantly, would likely not be used by a significant portion of the hashrate
  3 2017-11-27 11:09:46	0|sipa|further, having it available even if it was used wouldn't really benefit anyone; it's essentially introducing a lottery in the mining construction from the perspective of wallets
  4 2017-11-27 11:13:20	0|Provoostenator|I was thinking about creating an RPC method to add a new destination + amount to an existing RBF transaction. Or to add a "append-tx" argument to sendtoaddress. Has anyone worked on that in past?
  5 2017-11-27 11:13:52	0|Provoostenator|(or explained why that's a terrible idea)
  6 2017-11-27 11:14:13	0|sipa|Provoostenator: i've thought a bit about a more invasive idea that's related
  7 2017-11-27 11:14:51	0|sipa|i think it would be nice to have in the wallet, independent from the sendtoaddress/sendmany/... RPCs, a list of "outputs i would like to pay to" (with address/amount)
  8 2017-11-27 11:15:22	0|sipa|and any time that list is changed, or a transaction confirms, a new single transaction is constructed that pays out to all the entries in that list that haven't been paid to already
  9 2017-11-27 11:15:42	0|Provoostenator|And then have the wallet figure out if a new transaction is cheaper than amending one?
 10 2017-11-27 11:15:57	0|sipa|no, there is always just a single outstanding unconfirmed transaction
 11 2017-11-27 11:16:07	0|sipa|for any change, that transaction is modified
 12 2017-11-27 11:16:16	0|Provoostenator|That makes sense.
 13 2017-11-27 11:16:33	0|sipa|the cost of bumping for RBF reasons is usually trivial compared to its actual fee
 14 2017-11-27 11:16:47	0|sipa|(just add 1sat/vbyte)
 15 2017-11-27 11:17:14	0|Provoostenator|Right, unless it's a very large transaction. But that would be trivial to check.
 16 2017-11-27 11:17:19	0|sipa|there are complications to this... for example you need coin selection that guarantees conflicts with all previously attempted unconfirmed transaction
 17 2017-11-27 11:17:47	0|sipa|or you may need to pre-sign multiple candidate follow-up transactions based on which of your earlier attempts confirms
 18 2017-11-27 11:17:59	0|sipa|(as you can't automatically get access to encrypted keys later)
 19 2017-11-27 11:18:27	0|sipa|and it likely conflicts with how people actually use bitcoin in the real world now - where they expect a txid to be something you can track
 20 2017-11-27 11:18:28	0|Provoostenator|Oh, so in that case you'd use SIGHASH flags?
 21 2017-11-27 11:18:58	0|sipa|just predict the possible sceniarios, and sign a full follow-up transaction for each (or at least for the few most likely ones)
 22 2017-11-27 11:22:48	0|aj|sipa, wallet42: can't you just use prioritisetransaction rpc on the 50 oldest/whatever transactions to get that behaviour anyway?
 23 2017-11-27 11:23:25	0|Provoostenator|Although it sounds more elegant, I don't think I can pull that off, yet :-)
 24 2017-11-27 11:24:14	0|sipa|aj: yes
 25 2017-11-27 11:24:37	0|aj|sipa: predict the possible scenarios? should be called fund-xanatos-transaction...
 26 2017-11-27 11:24:49	0|sipa|xanatos?
 27 2017-11-27 11:25:05	0|aj|http://tvtropes.org/pmwiki/pmwiki.php/Main/XanatosGambit
 28 2017-11-27 11:27:44	0|sipa|Provoostenator: well i think this would be pretty much independent from all existing transaction logic
 29 2017-11-27 11:27:56	0|sipa|it'd reuse coin selection, but that's it :)
 30 2017-11-27 11:30:48	0|wallet42|sipa: are the wallet RPC capabilities currently enough to pull of something like that with an external python app?
 31 2017-11-27 11:31:12	0|wallet42|As a prototype
 32 2017-11-27 11:31:54	0|sipa|yes, just use prioritisetransaction
 33 2017-11-27 11:32:03	0|sipa|and getrawmempool
 34 2017-11-27 11:42:19	0|sipa|ugh, master allocates enormous amounts of memory for me while syncing from scratch from network
 35 2017-11-27 11:42:38	0|sipa|0.15.1 with the same configuration works fine
 36 2017-11-27 11:43:02	0|sipa|(it made my system with 32G RAM OOM)
 37 2017-11-27 11:44:28	0|Provoostenator|sipa: any way Travis could catch something like that?
 38 2017-11-27 11:44:46	0|Provoostenator|Or does it take too long before memory usage gets out of hand?
 39 2017-11-27 11:45:06	0|sipa|it OOMed when my debug.log said the dbcache was 1.7G
 40 2017-11-27 11:45:17	0|sipa|which is substantial but not excessive
 41 2017-11-27 11:45:31	0|sipa|and the overhead on top of what the dbcache predicts should only be a few 100 MB
 42 2017-11-27 12:24:04	0|Provoostenator|sipa: the second log link in the channel topic seems a bit stale...
 43 2017-11-27 12:24:33	0|aj|Provoostenator: yeah, the cron job broke and i haven't updated it since botbot seems better
 44 2017-11-27 12:32:53	0|Provoostenator|Any idea why "test/functional/test_runner.py --loglevel=INFO bumpfee" doesn't result in "Mining blocks..." appearing my console?
 45 2017-11-27 12:33:09	0|Provoostenator|It should according to: https://github.com/bitcoin/bitcoin/blob/master/test/functional/bumpfee.py#L52
 46 2017-11-27 12:38:07	0|aj|Provoostenator: okay cron job fixed, that wasn't so hard
 47 2017-11-27 12:39:17	0|Provoostenator|aj: nice, that seems to work
 48 2017-11-27 12:42:13	0|Provoostenator|Is anyone timestamping them?
 49 2017-11-27 12:48:58	0|aj|Provoostenator: each line has a timestamp :) but if you mean cryptographically, no
 50 2017-11-27 13:16:54	0|morcos|sipa: that sounds bad
 51 2017-11-27 13:17:11	0|morcos|re: wallet replacement, i think privacy concerns would also be substantial
 52 2017-11-27 13:17:56	0|morcos|you can assume chain analysis companies would start tracking unconfirmed txs, and use that to further group outputs as related
 53 2017-11-27 13:20:59	0|Provoostenator|That's one reason I would make this RPC-only initially.
 54 2017-11-27 13:22:25	0|Provoostenator|morcos: but does it really matter much privacy wise if you spend the next transaction from your change or by amending the original transaction?
 55 2017-11-27 13:26:30	0|morcos|Provoostenator: well if you spend from your change, it MAY be unclear that it was your change or the person you paid spending (agreed that right now change is probably pretty easily identifiable though)
 56 2017-11-27 13:26:56	0|morcos|also if you spend unconfirmed, then you have a good point
 57 2017-11-27 13:28:57	0|Provoostenator|I can see a use case for low priority and smaller transactions, e.g. paying back friends, where you intentionally set the fee too low, so you can keep amending it. A poor mans payment channel. Obviously there's a privacy cost.
 58 2017-11-27 13:29:27	0|morcos|or rich mans payment channel as the case may be.  :)
 59 2017-11-27 13:33:06	0|Provoostenator|"TestFramework (INFO): Mining blocks..." does show up in the test log file.
 60 2017-11-27 13:35:59	0|Provoostenator|Even stranger given that the default is INFO.
 61 2017-11-27 13:43:43	0|Provoostenator|Ah, so I need to call "test/functional/bumpfee.py --loglevel=INFO" for this to work.
 62 2017-11-27 14:48:31	0|bitcoin-git|[13bitcoin] 15joemphilips opened pull request #11770: [REST] add a rest endpoint for estimatesmartfee, docs, and test (06master...06rest_fee) 02https://github.com/bitcoin/bitcoin/pull/11770
 63 2017-11-27 16:04:51	0|promag|archaeal_: it it was me, I would have the PR title: Add REST endpoint blockhash
 64 2017-11-27 16:05:32	0|promag|then one commit for the implementation, one for the test and other for the doc
 65 2017-11-27 16:06:29	0|promag|^ 11765
 66 2017-11-27 16:07:06	0|archaeal_|i see...yeah that makes sense
 67 2017-11-27 16:09:05	0|Lauda|sipa I can confirm very high memory usage. I had a fresh sync (albeit a fork of 0.15.x) and it was using over 6 GB with 4GB dbcache and swap was 100% utilized.
 68 2017-11-27 16:35:32	0|bitcoin-git|[13bitcoin] 15jnewbery opened pull request #11771:  [tests] Change invalidtxrequest to use BitcoinTestFramework (06master...06refactor_invalidtxrequest) 02https://github.com/bitcoin/bitcoin/pull/11771
 69 2017-11-27 17:37:31	0|bitcoin-git|[13bitcoin] 15jnewbery closed pull request #11047: [tests] rename functional tests (06master...06rename_functional_tests) 02https://github.com/bitcoin/bitcoin/pull/11047
 70 2017-11-27 17:42:46	0|morcos|sipa: have a second to talk about SW wallet?
 71 2017-11-27 17:43:37	0|morcos|i thought one of the cases we wanted to cover (which is mostly a version of 2 i think) is that you have a backup of a 0.15.0 wallet now, and you addwitnessaddress, but then later you have to restore from backup
 72 2017-11-27 17:43:50	0|morcos|this is what the 2(a) case was meant to handle
 73 2017-11-27 17:44:46	0|morcos|i dont think i particularly care if we don't handle that in 0.16, but it does seem like at some point we should handle that...  with SW out, people are going to be doing that work flow.
 74 2017-11-27 17:46:31	0|morcos|never mind, i didn't read carefully enough..  the 1(a) case handles that if you are running 0.16, its only the downgrade case that isn't handled
 75 2017-11-27 17:46:36	0|morcos|i think that is good enough
 76 2017-11-27 17:48:28	0|morcos|i'm fine with 1(a), 2(c), 3(a).  But I wonder if there is some way we can warn the user about Problem 2, like could the backup have a higher walelt version (ick)
 77 2017-11-27 17:48:28	0|sipa|morcos: ok
 78 2017-11-27 17:48:53	0|sipa|right, that would be one solution
 79 2017-11-27 17:49:13	0|sipa|but it's hard to prevent backups by just file copying
 80 2017-11-27 17:51:23	0|sipa|you could have some system where a wallet is 'loaded' into a node, and the wallet file itself becomes marked as a backup, but something else in the node's configuration contains knowledge to avoid treating it as a backup
 81 2017-11-27 17:51:28	0|sipa|but... yuck
 82 2017-11-27 17:54:40	0|morcos|well the good thing is there is a workaround right.. so maybe  warnings are enough, saying that if you want to downgrade and you only have a backup, you first need to run 0.16 with the backup and make a fresh backup, and then you can downgrade
 83 2017-11-27 17:55:13	0|morcos|and that is true regardless of whether you backup is from 0.15 or 0.16
 84 2017-11-27 18:11:59	0|BlueMatt|sipa: I really cant say I'm a fan of 2c
 85 2017-11-27 18:14:56	0|sipa|the alternatives aren't great either though
 86 2017-11-27 18:17:23	0|BlueMatt|sipa: what did you think of my suggestion of a segwit-wallet-bestblockhash field
 87 2017-11-27 18:17:28	0|BlueMatt|in addition to wallet-bestblockhash
 88 2017-11-27 18:17:52	0|BlueMatt|which allows you to rescan for segwit addresses if you downgraded to 15 for a while (and havent fully pruned those blocks)
 89 2017-11-27 18:20:00	0|sipa|BlueMatt: sounds doable, but it's not really something we can continue
 90 2017-11-27 18:20:35	0|sipa|it's a bit of an accident that segwit was introduced in 0.13.1, but will only be actually supported 3 major versions later
 91 2017-11-27 18:20:47	0|BlueMatt|hmm?
 92 2017-11-27 18:21:09	0|sipa|i don't think we should continue to expect that it's at all feasible that a new feature introduced will actually work at all after downgrading
 93 2017-11-27 18:21:24	0|BlueMatt|I didnt claim it would/should?
 94 2017-11-27 18:21:38	0|sipa|right, but the approach you suggest really only works for segwit
 95 2017-11-27 18:21:46	0|sipa|not for anything later
 96 2017-11-27 18:21:49	0|sipa|(likely)
 97 2017-11-27 18:21:59	0|BlueMatt|yes, this is specifically a segwit thing in large part cause we're not doing a full -walletupgrade for it
 98 2017-11-27 18:22:21	0|BlueMatt|if we were doing a -walletupgrade (which I still think is the "right" way to do this, but obviously with a ton more complication) then I dont think we'd need to do this
 99 2017-11-27 18:22:49	0|BlueMatt|a non-default clear statement of "I'm moving my wallet to 0.16" should be sufficient, but absent that, we should be more careful to handle some downgrade cases
100 2017-11-27 18:24:14	0|gmaxwell|how is walletupgrade right when it leaves current wallets people are using today unrecoverable?
101 2017-11-27 18:24:21	0|BlueMatt|hmm?
102 2017-11-27 18:24:45	0|gmaxwell|people use addwitnessaddress, then their backups are instantly no good-- no lookahead, no nothing.
103 2017-11-27 18:25:07	0|gmaxwell|that screup is fixed by 'upgrading' by default
104 2017-11-27 18:25:22	0|sipa|gmaxwell: i don't see how that has anything to do with explicit upgrade or not
105 2017-11-27 18:25:24	0|BlueMatt|I mean addwitnessaddress is barely supported and largely unsupportable, at least without something like upgradewallet
106 2017-11-27 18:25:35	0|BlueMatt|I mean you can auto-upgrade a user if they've previously used addwitnessaddress
107 2017-11-27 18:25:46	0|BlueMatt|cause an addwitnessaddress wallet is already a mess (at least wrt backups)
108 2017-11-27 18:26:04	0|sipa|explicit upgrade just guarantees that old versions will give you an error rather than silently missing things _for addresses generated in the future_
109 2017-11-27 18:26:16	0|sipa|obviously any addresses generated in the must should always continue to work
110 2017-11-27 18:26:52	0|BlueMatt|of course, but explicit upgrade also has a clear definition for the user: you cannot open this wallet anymore in old versions
111 2017-11-27 18:27:12	0|BlueMatt|which if we *dont* do an explicit upgrade, we end up in this quagmire of backward compatibility/downgrade hell
112 2017-11-27 18:27:15	0|gmaxwell|BlueMatt: if you don't autoupgrade someone then you won't make their backups good again.
113 2017-11-27 18:27:34	0|BlueMatt|"make their backups good again"?
114 2017-11-27 18:28:06	0|BlueMatt|I do not think we should *ever* auto-upgrade someone, with the possible exception of folks who've previously addwitnessaddress'd, cause they're kinda fucked if we dont, or at least we should nag them
115 2017-11-27 18:28:30	0|sipa|i think at some point we should do a required uograde
116 2017-11-27 18:28:36	0|gmaxwell|Right now, joe blow believes our claims of supporting segwit, runs the software, follows instructions to use it... unknown to him, his backup will miss coins if recovered.  Post upgrading to a release that autoupgrades his backup will no longer miss coins when recovered.
117 2017-11-27 18:28:56	0|gmaxwell|_You cannot tell if they've used addwitnessaddress because they could be loading a backup which hasn't seen it_
118 2017-11-27 18:29:27	0|sipa|well 1a solves that
119 2017-11-27 18:30:03	0|gmaxwell|if you try, you get even more awesome effects like one backup works for address X and another doesn't for reasons entirely unrelated to address X.
120 2017-11-27 18:30:08	0|BlueMatt|yes, I think for that reason 1a is probably good
121 2017-11-27 18:30:48	0|sipa|the only case that is not covered is downgrading + restoring a backup simuktaneously
122 2017-11-27 18:31:15	0|sipa|which is inherently impossible to solve completely
123 2017-11-27 18:31:31	0|BlueMatt|yes, and hence my suggestion to do a segwit-wallet-bestblockhash, cause then we at least fix it for non-pruned or only-briefly-run-as-downgraded nodes
124 2017-11-27 18:31:32	0|sipa|but some solutions are possible for the case where the backup was made after uograding
125 2017-11-27 18:31:38	0|BlueMatt|sidestepping the version rabbithole
126 2017-11-27 18:32:32	0|BlueMatt|whats default keypool size? its not *that* bad
127 2017-11-27 18:32:39	0|sipa|2000 keys
128 2017-11-27 18:32:51	0|BlueMatt|and, when we go to do things "the right way", we can clean those out, maybe
129 2017-11-27 18:32:55	0|sipa|+ more logic and edge cases
130 2017-11-27 18:33:02	0|BlueMatt|with a -walletupgrade can clean out those entries
131 2017-11-27 18:33:24	0|sipa|= something that still only works as long as you don't actually use the wallet too much after downgrading
132 2017-11-27 18:34:44	0|BlueMatt|hmm? no?
133 2017-11-27 18:34:57	0|BlueMatt|your backups will still work cause they have the witnesses added
134 2017-11-27 18:35:08	0|BlueMatt|and the upgraded wallet obviously wont cause you set the version to be un-openable with the old version
135 2017-11-27 18:35:36	0|sipa|right
136 2017-11-27 18:36:11	0|sipa|but if the wallet was used a lot in the new version after the backup was made, you'll hit problems when the backup's keypool runs out
137 2017-11-27 18:36:29	0|BlueMatt|yes, but that is always true
138 2017-11-27 18:36:39	0|BlueMatt|if your keypool runs out you cant run from a backup or you lose funds
139 2017-11-27 18:36:40	0|BlueMatt|period.
140 2017-11-27 18:37:03	0|BlueMatt|(I suppose with the exception of HD, but....ugh)
141 2017-11-27 18:37:28	0|sipa|no, not period
142 2017-11-27 18:37:43	0|sipa|this is exactly the assumption we don't need to make with HD wallets anymore
143 2017-11-27 18:38:04	0|sipa|relying on the keypool to deal with downgraded backups reintroduces that assumption
144 2017-11-27 18:38:32	0|BlueMatt|yes, with the exception of HD
145 2017-11-27 18:38:40	0|sipa|that's not a small exception
146 2017-11-27 18:38:48	0|sipa|that's the norm
147 2017-11-27 18:39:13	0|BlueMatt|ok, so now we're back to segwit-wallet-bestblockhash
148 2017-11-27 18:39:17	0|BlueMatt|thats the only way to fix that issue
149 2017-11-27 18:39:52	0|wallet42|sipa: ah yes. prioritisetransaction is exactly what im looking for.
150 2017-11-27 18:40:09	0|sipa|BlueMatt: right, but only in the case the user actually uogrades again
151 2017-11-27 18:40:21	0|BlueMatt|well if they dont upgrade again there is literally nothing we can do
152 2017-11-27 18:40:27	0|sipa|that's my point
153 2017-11-27 18:40:32	0|wallet42|spia: my python prototype question was in the context of the output list wallet amend tx amend idea you talked about with provoost
154 2017-11-27 18:40:41	0|BlueMatt|(and I think thats reasonable, if you use segwit in a version with a headline of "SegWit Wallet" and then downgrade, you'd kinda expect to not have your segwit funds)
155 2017-11-27 18:41:04	0|BlueMatt|well the current proposal does *not* fully work in the case you upgrade again, without a rescan
156 2017-11-27 18:41:04	0|sipa|BlueMatt: so why do we need to support the downgrade + restore case again?
157 2017-11-27 18:41:13	0|sipa|it's without guarantees at best
158 2017-11-27 18:41:23	0|BlueMatt|if you are non-pruned its guaranteed
159 2017-11-27 18:41:23	0|sipa|in whatever case
160 2017-11-27 18:41:51	0|sipa|it's without guarantees at best as long as you don't upgrade
161 2017-11-27 18:41:57	0|BlueMatt|huh?
162 2017-11-27 18:42:18	0|sipa|don't upgrade after downgrading, i mean
163 2017-11-27 18:42:46	0|BlueMatt|yes, if you dont ever switch back to 0.16+, then you are maybe not safe in this case
164 2017-11-27 18:42:56	0|sipa|not maybe
165 2017-11-27 18:42:58	0|BlueMatt|but so what? we can still improve things
166 2017-11-27 18:43:03	0|sipa|in that case nothing can be guaranteed
167 2017-11-27 18:43:15	0|BlueMatt|doing a partial-rescan is a very, very big improvement for many of the cases we're talking about
168 2017-11-27 18:43:19	0|BlueMatt|even if its not a guarantee
169 2017-11-27 18:43:35	0|BlueMatt|assuming you may never upgrade to 0.16+ and trying to support that seems like nonsense
170 2017-11-27 18:43:38	0|BlueMatt|but thats not what I'm arguing here
171 2017-11-27 18:43:41	0|sipa|ok, so let's walk through yhis
172 2017-11-27 18:43:48	0|sipa|you upgrade to 0.16
173 2017-11-27 18:43:57	0|sipa|there is no segwitwallet record
174 2017-11-27 18:44:30	0|sipa|you do what? assume that the existing non-segwit wallet bestblock record is accurate?
175 2017-11-27 18:45:14	0|BlueMatt|a) ignore, b) rescan from segwit activation date if you have P2SH-P2WPKH entries c) always rescan from segwit activation date
176 2017-11-27 18:45:57	0|sipa|b assumes the file is recent enough to have those records
177 2017-11-27 18:46:00	0|BlueMatt|d) rescan in case -findmissingsegwitoutputs=true is set
178 2017-11-27 18:46:09	0|BlueMatt|indeed, b kinda sucks
179 2017-11-27 18:46:32	0|sipa|i think the best solution is just telling people to run with 0.16 and -rescan in that case
180 2017-11-27 18:46:42	0|BlueMatt|do we have a rescan from date rpc?
181 2017-11-27 18:46:45	0|BlueMatt|i think we do now, right?
182 2017-11-27 18:47:06	0|sipa|unsure
183 2017-11-27 18:47:23	0|BlueMatt|I mean in that case we also need 2a, I think
184 2017-11-27 18:48:00	0|sipa|how so?
185 2017-11-27 18:48:13	0|sipa|1a solves everything during a rescan
186 2017-11-27 18:49:00	0|BlueMatt|yea, sorry, 1a obviously fixes it for rescan, maybe I'm just more worried about problem 2
187 2017-11-27 18:49:39	0|BlueMatt|more like "oops, da server with wallet blew up, bring in the one we upgraded from last week, running 0.15.1, will have to do for now"
188 2017-11-27 18:49:39	0|gmaxwell|unrelated to this, through some combination of multiwallet, -rescan, restarting my node.. I ended up in a situation with a bunch of wallets that lost all their transactions but think they're in sync.
189 2017-11-27 18:50:02	0|BlueMatt|gmaxwell: thats....frightening....is it 0.15.1 with the copied-db issue?
190 2017-11-27 18:50:07	0|morcos|i basically agree with sipa
191 2017-11-27 18:50:16	0|sipa|my solution to problem 2 is tell people to avoid downgrading and restoring, and if in exceptional cases that misses something, rescan
192 2017-11-27 18:50:18	0|gmaxwell|w/ recent-ish master.
193 2017-11-27 18:50:40	0|BlueMatt|sipa: but that doesnt solve the problem of not upgrading right away? at least 2a will help significantly
194 2017-11-27 18:50:46	0|morcos|i think 1a, 3a and a guide for acceptable workflows and how to recover is the way to go..  we should not make things too difficult.
195 2017-11-27 18:51:32	0|morcos|however i think we will need to do something like 2a when we have the upgrade wallet option for 0.17.  but then it will i think be fully optional as to whether you upgrade or not.
196 2017-11-27 18:51:45	0|gmaxwell|I _think_ what I did was start with two dozen -wallets and -rescan then realized it would sequentially rescan each, killed the process, then went on to try using them directly.
197 2017-11-27 18:52:12	0|sipa|BlueMatt: but even if we do 2a, there is no solution for 4a
198 2017-11-27 18:52:27	0|sipa|recovery options are great, but they exist
199 2017-11-27 18:52:45	0|BlueMatt|yea, issue 4 is unsolveable, I agree
200 2017-11-27 18:52:52	0|morcos|sorry, not actually 2a for 0.17, but something that takes all your keys in your keypool and stores them as things that implicitly might have any style address
201 2017-11-27 18:52:56	0|BlueMatt|with the exception of 1a + rescan
202 2017-11-27 18:53:14	0|sipa|2a is essentially trying to avoid a recovery procedure in a situation which already is something that should never be part of normal workflow
203 2017-11-27 18:54:21	0|gmaxwell|if we make rescan faster we could be more liberal in expecting its use.
204 2017-11-27 18:55:01	0|gmaxwell|not tooo liberal, since pruned nodes can't rescan...
205 2017-11-27 18:55:48	0|BlueMatt|gmaxwell: do you have an intuition in why rescan is slow? I havent looked into it
206 2017-11-27 18:56:58	0|sipa|BlueMatt: we should do a benchmark for reindex with and without wallet
207 2017-11-27 18:58:43	0|sipa|if there is a significant different between the two, there is a totally obvious improvement from not calling Solver on each output in order to match it with the wallet
208 2017-11-27 18:58:54	0|gmaxwell|well we fully read and deseralize each block and each transaction then pass them through the wallet code, so that implhes 1001 memory allocations plus sha256ing each transaction (and reserializing to do so)... looking at IO we also seem to create a lot of excess  IO with opening the file, seeking to the block closing it.
209 2017-11-27 18:58:54	0|sipa|(which construct a vector of vectors ffs)
210 2017-11-27 18:59:29	0|sipa|then indeed there is the cost of just loading blocks, which could be improved by having a block deserializer that doesn't build/verify the merkle tree
211 2017-11-27 18:59:57	0|gmaxwell|at least from rough looking at it, we do both a bunch of dump computation and IO and all of it serialized.
212 2017-11-27 19:00:21	0|gmaxwell|it's worth noting that it's not remarkably faster with the blockchain all in cache.
213 2017-11-27 19:01:18	0|BlueMatt|sipa: ok, so I guess I feel a lot more comfortable about things if we do #11281 and recommend rescanblockchain segwit_activation_height
214 2017-11-27 19:01:20	0|gribble|https://github.com/bitcoin/bitcoin/issues/11281 | Avoid permanent cs_main/cs_wallet lock during RescanFromTime by jonasschnelli · Pull Request #11281 · bitcoin/bitcoin · GitHub
215 2017-11-27 19:02:32	0|BlueMatt|sipa: I'd still be more comfortable with 2a...its far from a hard guarantee of anything, but I really could see someone ending up in the state where that'd save them
216 2017-11-27 19:02:48	0|BlueMatt|if for no other reason than someone restoring a backup may load in 0.15, realize they need 0.16, then open in 0.16 right away
217 2017-11-27 19:02:51	0|BlueMatt|and end up screwing themselves
218 2017-11-27 19:04:06	0|BlueMatt|the segwit-wallet-best-block-hash would also similarly solve the same issue even if you do my option a where you just ignore it if it isnt there
219 2017-11-27 19:04:36	0|BlueMatt|though I agree that kinda feels gross given its solving such a specific edge case without any hard guarantees
220 2017-11-27 19:08:43	0|BlueMatt|can we at least high-priority #11281 for 0.16?
221 2017-11-27 19:08:45	0|gribble|https://github.com/bitcoin/bitcoin/issues/11281 | Avoid permanent cs_main/cs_wallet lock during RescanFromTime by jonasschnelli · Pull Request #11281 · bitcoin/bitcoin · GitHub
222 2017-11-27 19:08:52	0|BlueMatt|so that rescan is at least slightly less painful from the qt command window
223 2017-11-27 19:09:01	0|sipa|sgtm
224 2017-11-27 19:09:28	0|BlueMatt|and add a "I lost my segwit coins" button in the debug window
225 2017-11-27 19:13:19	0|gmaxwell|any kind of scanning uncertanty will cause untold amounts of prophylactic scanning.
226 2017-11-27 19:13:40	0|BlueMatt|yea, but I dont think we can really solve that......
227 2017-11-27 19:14:06	0|BlueMatt|in part cause 0.16 docs are going to have to have a bunch of "in case you did X, do this rescan"
228 2017-11-27 19:15:11	0|gmaxwell|thats partly why I was commenting about making rescanning faster.
229 2017-11-27 19:15:38	0|BlueMatt|yes, though dunno about making that a requirement for 0.16 :p
230 2017-11-27 19:15:44	0|gmaxwell|oh another point on speed-- I opened an issue for this-- right now if you load up a dozen wallets and -rescan. ... it rescans a dozen times, one for each wallet by itself...
231 2017-11-27 19:15:58	0|BlueMatt|yes, that should be fixed as well
232 2017-11-27 19:16:14	0|gmaxwell|my first idea about not spending the rest of my life rescanning was thwarted by that.
233 2017-11-27 19:30:51	0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #11775: Move fee estimator into validationinterface/cscheduler thread (06master...062017-09-background-feeest) 02https://github.com/bitcoin/bitcoin/pull/11775
234 2017-11-27 20:11:56	0|jonasschnelli|BlueMatt: added 11281 to the project 7
235 2017-11-27 20:13:02	0|BlueMatt|jonasschnelli: thanks!
236 2017-11-27 20:17:16	0|cluelessperson|BlueMatt: are you updating the fee estimator?
237 2017-11-27 20:20:27	0|BlueMatt|cluelessperson: it doesnt change the fee estimator logic, except to handle a few edge cases better
238 2017-11-27 20:42:54	0|sipa|Lauda: using 6GB with 4GB dbcache sounds bad but not nearly the same thing i was seeing
239 2017-11-27 20:43:20	0|sipa|i was seeing 32GB usage with 1.7G dbcache
240 2017-11-27 21:07:21	0|Lauda|Well, add 4 gb of SWAP to that so probably ~10ish (I think I allocated 7 maximum for the VM). However, yes. That is much worse :<
241 2017-11-27 21:07:49	0|Lauda|Any ideas as to what is causing that?
242 2017-11-27 21:10:01	0|sipa|by "using" i mean RSS size, not physical usage
243 2017-11-27 21:10:32	0|sipa|it's pretty hard to map VM usage to physical usage, as processes can share memory etc
244 2017-11-27 21:11:49	0|sipa|(by VM i mean virtual memory, not virtual machine - it's probably confusing in this context)
245 2017-11-27 21:12:17	0|Lauda|Oh, then 6 GB yeah.
246 2017-11-27 21:14:14	0|Lauda|Your behavior is only in current master, not in 0.15.1?
247 2017-11-27 21:14:45	0|sipa|indeed
248 2017-11-27 21:15:28	0|sipa|in 0.15.1 memory usage stayed nicely within a few 100 MB of the cache size
249 2017-11-27 21:15:39	0|sipa|though i now fail to reproduce with master
250 2017-11-27 21:16:56	0|Lauda|That's strange.
251 2017-11-27 21:59:32	0|cluelessperson|BlueMatt: do you know who might be best to redo the logic?
252 2017-11-27 21:59:42	0|cluelessperson|reason being that fees seem to jump up faster than volume
253 2017-11-27 21:59:53	0|cluelessperson|https://jochen-hoenicke.de/queue/#2h
254 2017-11-27 21:59:57	0|gribble|https://github.com/bitcoin/bitcoin/issues/2 | Long-term, safe, store-of-value · Issue #2 · bitcoin/bitcoin · GitHub
255 2017-11-27 22:00:32	0|BlueMatt|cluelessperson: if there were a clear and easy fix it would have been done already :/
256 2017-11-27 22:00:39	0|BlueMatt|sadly good fee estimation is super non-trivial
257 2017-11-27 22:01:06	0|BlueMatt|*especially* for the non-economical (eg doing non-rbf so want high confidence) case
258 2017-11-27 22:01:30	0|cluelessperson|BlueMatt: agreed.  I could only recommend experimenting with "lower estimates slightly" and see how the mempool responds.
259 2017-11-27 22:01:38	0|cluelessperson|that would take >years to tune.
260 2017-11-27 22:01:44	0|BlueMatt|doing it with rbf in economical mode could probably get improvements and go down faster, etc
261 2017-11-27 22:02:06	0|BlueMatt|"lower estimates slightly"? what does that mean?
262 2017-11-27 22:02:26	0|BlueMatt|in steady-state the fee estimator does incredibly well, it also happens to do incredibly well for very-high-confidence-of-inclusion
263 2017-11-27 22:02:40	0|cluelessperson|BlueMatt:   (existing_fee_estimate) - 1 sat,    -5 sat,   -10 sat
264 2017-11-27 22:02:53	0|BlueMatt|though fee estimation also ends up being a bit of a self-fuffilling prophecy
265 2017-11-27 22:03:05	0|BlueMatt|yea, I dont think that is a good approach :p
266 2017-11-27 22:06:20	0|cluelessperson|BlueMatt: I'm taking classes.  Hopefully I can help in the future, but right now I'm a moronw. :/
267 2017-11-27 22:18:15	0|mlz|lol
268 2017-11-27 23:22:26	0|bitcoin-git|[13bitcoin] 15MarcoFalke opened pull request #11779: qa: Combine logs on travis by default (06master...06Mf1711-travisQaLogs) 02https://github.com/bitcoin/bitcoin/pull/11779