1 2016-07-01 00:35:03	0|gmaxwell|as your only connections from core nodes will be inbound and after segwit activates core will avoid making outbound connections to non-segwit enabled hosts; already that behavior is dangerous because there are relatively few hosts that it is willing to outbound connect to, making chance partitioning and vulnerability to sybil attacks worse.
  2 2016-07-01 01:38:33	0|luke-jr|fwiw, there appears to be no changes affecting bitcoin-cli between 0.12.0 and 0.12.1
  3 2016-07-01 04:59:42	0|gmaxwell|Can anyone think of why we'd get a "CommitTransaction(): Error: Transaction not valid"  on a new transaction being created that paid twice the nodes minrelay fee, ... and then managed to broadcast when the rebroadcast ran?
  4 2016-07-01 04:59:48	0|gmaxwell|it looks like it was spending an unconfirmed input.
  5 2016-07-01 05:10:31	0|gmaxwell|okay I think the issue was he managed to make a 25 deep unconfirmed chain, and got the last txn in it rejected.
  6 2016-07-01 05:10:38	0|gmaxwell|Doesn't look like the wallet handles that well.
  7 2016-07-01 05:13:00	0|gmaxwell|I think we should avoid using coins that are at the unconfirmed limit maximum unless not otherwise possible.
  8 2016-07-01 05:13:16	0|gmaxwell|Also ooRBF to sendmany would have eliminated 25 transactions here.
  9 2016-07-01 07:52:45	0|GitHub68|[13bitcoin] 15laanwj pushed 3 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/6a87eb0e4b47...da50997a3ee7
 10 2016-07-01 07:52:46	0|GitHub68|13bitcoin/06master 140ce8e99 15Wladimir J. van der Laan: windows: Add testnet link to installer
 11 2016-07-01 07:52:46	0|GitHub68|13bitcoin/06master 14975a41d 15Wladimir J. van der Laan: windows: Add testnet icon for testnet link...
 12 2016-07-01 07:52:47	0|GitHub68|13bitcoin/06master 14da50997 15Wladimir J. van der Laan: Merge #8285: windows: Add testnet link to installer...
 13 2016-07-01 07:52:52	0|GitHub129|[13bitcoin] 15laanwj closed pull request #8285: windows: Add testnet link to installer (06master...062016_06_testnet_link_windows) 02https://github.com/bitcoin/bitcoin/pull/8285
 14 2016-07-01 08:10:00	0|instagibbs|gmaxwell, yes the maximum ancestor/descendant stuff means that the transaction will be marked invalid and the funds deducted from your wallet until rescan :/
 15 2016-07-01 08:10:09	0|instagibbs|err reindex or something
 16 2016-07-01 08:11:33	0|gmaxwell|instagibbs: in this case the transaction when through after the parents confirmed.
 17 2016-07-01 08:11:47	0|gmaxwell|the wallet rebroadcast accepted it the next time around and tada.
 18 2016-07-01 08:12:09	0|gmaxwell|probably the coinselection process, when it considers 0conf inputs should only consider thoes with a low enough depth.
 19 2016-07-01 08:12:40	0|gmaxwell|(perhaps two below the maximum to leave room for CPFP), and only if it's unable to meet that should it fall back to considering all inputs.
 20 2016-07-01 08:14:39	0|instagibbs|gmaxwell, oh so even in failure it rebroadcasts?
 21 2016-07-01 08:14:59	0|gmaxwell|yea, it saves it in the wallet before it tries to mempool it.
 22 2016-07-01 08:15:08	0|gmaxwell|and since its saved, it keeps trying to rebroadcast.
 23 2016-07-01 08:15:08	0|wumpus|as long as the transaction is in the wallet it rebroadcasts
 24 2016-07-01 08:15:49	0|gmaxwell|I dunno if the depth of an unconfirmed coin is easily discernable though from the wallet.
 25 2016-07-01 08:20:31	0|wumpus|not at is, it would need another (ugly) mempool dependency
 26 2016-07-01 08:21:05	0|gmaxwell|bleh
 27 2016-07-01 08:22:48	0|wumpus|then again we already do an InMempool() check in IsTrusted, which determines whether the outputs of an transaction are considered spendable, so...
 28 2016-07-01 08:23:16	0|wumpus|if looking up the depth in the mempool is relatiely cheap that could be added too
 29 2016-07-01 08:24:17	0|wumpus|but yes I agree with the 'bleh' sentiment
 30 2016-07-01 08:24:46	0|wumpus|ideally mempool implementation details shouldn't matter to the wallet
 31 2016-07-01 08:25:03	0|wumpus|"if it cannot be submitted yet due to depth, try again later" seems a better approach
 32 2016-07-01 08:25:28	0|wumpus|the wallet already tries to avoid generating chains of unconfirmed
 33 2016-07-01 08:25:34	0|wumpus|so if it does, it really needs to
 34 2016-07-01 08:26:35	0|gmaxwell|it doesn't really: so it tries to avoid spending unconfirmed coins, but if it must it treats short and long chains the same.
 35 2016-07-01 08:26:52	0|gmaxwell|So in the example with the user I was supposting today, they had plenty of unconfirmed coins that were only 1 deep.
 36 2016-07-01 08:27:17	0|wumpus|yes I don't mean to imply that it looks at the mempool depth, it obviously doesn't
 37 2016-07-01 08:27:20	0|gmaxwell|and one chain of 25.
 38 2016-07-01 08:27:58	0|wumpus|yes that would make sense - count *unconfirmed* depth
 39 2016-07-01 08:28:08	0|wumpus|then prefer coins that are shallow
 40 2016-07-01 08:28:17	0|gmaxwell|it ended up that way because they had multiple unconfirmed few-bitcoin outputs, and then a 1 bitcoin output and were making many not huge payments... so it decided to keep reusing the change because it considered them all equal.
 41 2016-07-01 08:28:19	0|wumpus|that wouldn't require asking the mempool at all
 42 2016-07-01 08:28:49	0|gmaxwell|yea, don't' really care about the mempool, the ismine tracing could count the depth until confirmed I guess.
 43 2016-07-01 08:29:13	0|wumpus|coins with a smaller unconfirmed depth could get some advantage in selection
 44 2016-07-01 08:29:40	0|wumpus|another thing in the long list of things that could be better in coin selection
 45 2016-07-01 08:31:54	0|gmaxwell|well the way to do the advantage is the same way it prefers confirmed coins.
 46 2016-07-01 08:32:17	0|gmaxwell|try the selection first with any too-deep coins excluded, and only if it fails attempt without that restriction.
 47 2016-07-01 08:32:37	0|wumpus|that's a possiblity, yes
 48 2016-07-01 08:36:18	0|wumpus|though at some point the 'exclude these coins and do the selection again' list would become very long, and is less suited for non-boolean properties like input size (https://github.com/bitcoin/bitcoin/issues/7664)
 49 2016-07-01 08:36:38	0|wumpus|but sure, this could be bolted on too :)
 50 2016-07-01 08:37:32	0|gmaxwell|well, it fits here since the failure results in an (temporarily) invalid transaction. ... just leaving them out is the right call for max depth, at least.
 51 2016-07-01 08:37:33	0|wumpus|I think at some point it would be nice to have a per-coin scoring system, then make the coin selection use that
 52 2016-07-01 08:38:08	0|gmaxwell|yes, for some things, but a score for this should be infinite unless there is no other choice... since it guarentees the txn will not relay until its ancestors confirm some. :)
 53 2016-07-01 08:38:30	0|wumpus|but you make assumptions about other people's mempool depth
 54 2016-07-01 08:38:42	0|wumpus|I think a better approach would just be to *prefer* shallower transactions
 55 2016-07-01 08:38:51	0|wumpus|without an absolute theshold
 56 2016-07-01 08:39:21	0|gmaxwell|They're not exclusive, prefer shorter but absolute threshold at the point where you won't relay the thing yourself anymore.
 57 2016-07-01 08:39:54	0|gmaxwell|this usecase would actually have been better off with some kind of replacement, as they could have saved a good 20 transactions.
 58 2016-07-01 08:39:55	0|wumpus|and if it does manage to generates a temporarily invalid transaction, handle that more friendly, e.g. a warning instead of an error
 59 2016-07-01 08:41:31	0|wumpus|having the wallet have a hard dependency on a property of the mempool seems really brittle
 60 2016-07-01 08:42:12	0|wumpus|and it's not a solution that wallets that don't have such a close coupling to a node could use
 61 2016-07-01 08:42:13	0|gmaxwell|I don't think this is really a mempool dependency.
 62 2016-07-01 08:42:44	0|wumpus|in principle it's a matter of optimizing the speed at which the transaction can be submitted, longer chains result in slower confirmation times
 63 2016-07-01 08:42:55	0|wumpus|very long chains even in temporary rejection
 64 2016-07-01 08:42:59	0|gmaxwell|If it's spending here the transaction is ismine, which means all unconfirmed ancestors are wallet txn.
 65 2016-07-01 08:43:10	0|gmaxwell|So it just needs to know the longest depth.
 66 2016-07-01 08:43:14	0|wumpus|but long chains, even if they are accepted, are not a good thing in itself
 67 2016-07-01 08:44:34	0|gmaxwell|(er not IsMine, but IsFromMe)
 68 2016-07-01 08:46:04	0|gmaxwell|wumpus: they're not that bad, esp with ancestor feerate mining; and relay being not totally broken for them in 0.13.
 69 2016-07-01 08:46:28	0|wumpus|but I hope you agree that they are worse than shallow transaction chains
 70 2016-07-01 08:46:49	0|wumpus|'not that bad' yes there are always much worse things :)
 71 2016-07-01 08:46:51	0|gmaxwell|all things equal... but compared to a choice that is less private or pays more fees?
 72 2016-07-01 08:47:17	0|wumpus|well the individual scoring weights would have to be determined, yes, or even depend on some setting
 73 2016-07-01 08:47:51	0|wumpus|paying more fees may be ok if it means the transaction has a larger chance of being picked
 74 2016-07-01 08:47:53	0|gmaxwell|because we never split large change output, a lot of usage patterns result in no real choice in any case.
 75 2016-07-01 08:48:36	0|wumpus|less private, well long transaction chains are a great way to say 'hey this is a wallet sending out transactions serially'
 76 2016-07-01 08:49:59	0|gmaxwell|sure, but often better to just respend change than join a dozen otherwise unrelated inputs.
 77 2016-07-01 08:51:15	0|wumpus|if it requires joining them, yes I agree
 78 2016-07-01 08:55:45	0|gmaxwell|Would we do anything with it?... a while back someone proposed a change (to remove extranious inputs), I suggested that it might result in wallets grinding down coins into small amouts more often. He made a simulator that showed it would, then we took the change. Then later users showed up complaining about the wallet grinding down inputs when they didn't use to...
 79 2016-07-01 08:56:39	0|wumpus|well the problem is that we're too busy running from issue to issue to look at a higher level
 80 2016-07-01 08:56:46	0|wumpus|well at least I am
 81 2016-07-01 08:56:54	0|gmaxwell|right. I agree with that.
 82 2016-07-01 08:57:13	0|gmaxwell|just collectively we are.
 83 2016-07-01 08:57:29	0|wumpus|so now long chains are an issue, long chains are fixed by adding yet another special case, but without considering impact on other things
 84 2016-07-01 08:57:47	0|gmaxwell|You're right that someone looking at it more holistically would be good, part of the problem in that issue was that even there it was only asking a very narrow question.
 85 2016-07-01 08:57:54	0|wumpus|at some point I just worry all those hacks make things worse, instead of picking some simple algorithm that does fine in most cases
 86 2016-07-01 08:58:13	0|gmaxwell|I don't really think there is much to consider on the subject of "avoid going over your own maximum chain depth if at all possible".
 87 2016-07-01 08:58:16	0|wumpus|but I don't know, I don't have the ovierview
 88 2016-07-01 08:58:36	0|gmaxwell|since producing a txn even you won't relay when you could have avoided it seems clearly wrong.
 89 2016-07-01 08:58:50	0|phantomcircuit|gmaxwell: the wallet should probably have all of the unconfirmed depdencies of a transaction
 90 2016-07-01 08:59:08	0|wumpus|so dropping of extraneous inputs was bad?
 91 2016-07-01 08:59:09	0|phantomcircuit|and then remove the dependencies after some high number of confirms which aren't relevant
 92 2016-07-01 09:00:06	0|gmaxwell|phantomcircuit: in these cases all the unconfirmed dependencies will be wallet transactions-- IsFromMe otherwise it won't spend the coin!
 93 2016-07-01 09:00:07	0|wumpus|normally you'd say that choosing a minimum set that covers the value to be spent would be optimal
 94 2016-07-01 09:00:09	0|phantomcircuit|but yeah you're right it has to pass IsMine which means the wallet already has all the info you need to calculate depth with just coins view
 95 2016-07-01 09:00:38	0|wumpus|but yes, I'm sure there are lots of other contraints and scores that should be taken in to account
 96 2016-07-01 09:00:39	0|phantomcircuit|reading top down so my comments might be already worked out :P
 97 2016-07-01 09:00:40	0|gmaxwell|wumpus: the result ends up being the smallest possible change. so it breaks your wallet into lots of tiny little coins.
 98 2016-07-01 09:00:41	0|wumpus|which was my point before
 99 2016-07-01 09:01:43	0|wumpus|so why wasn't that reverted?
100 2016-07-01 09:02:03	0|gmaxwell|I don't know/understand why it wasn't.
101 2016-07-01 09:02:47	0|gmaxwell|I think when it went in there was some expectation that further improvements would come, and they didn't. Then it was released. Then people showed up and complaints (opening an issue) and we figured out that the behavior change was due to removing extranious inputs.. and then?  I don't know
102 2016-07-01 09:02:52	0|wumpus|we just have too much on our plate
103 2016-07-01 09:02:56	0|gmaxwell|then something else caught fire.
104 2016-07-01 09:03:11	0|wumpus|sometimes I really want to run around screaming with my hands on my head
105 2016-07-01 09:03:44	0|phantomcircuit|wumpus: that sounds like fun
106 2016-07-01 09:03:48	0|wumpus|it's not possible to handle this all anymore
107 2016-07-01 09:04:29	0|wumpus|we really need someone that focuses on wallet improvements
108 2016-07-01 09:04:31	0|gmaxwell|(privacy wants you to spend all payments to a single address at once, so that you don't get a rolling linkage that eventually cross taints every address in your wallet--- so maybe just attempting to do this would counteract most of the grinding. thats the kind of thing you want tne step back and overview to answer)
109 2016-07-01 09:06:01	0|gmaxwell|There are a number of positive wallet things going on, at least. But there is a lot of time spent twiddling things in the weeds rather than setting up prioties and identifying larger scale issues.
110 2016-07-01 09:06:01	0|wumpus|someone that has an overview of what the heck is happening in the wallet
111 2016-07-01 09:06:12	0|wumpus|but how to sort 'bad' improvements from good ones?
112 2016-07-01 09:06:22	0|wumpus|I don't know anymore
113 2016-07-01 09:07:40	0|phantomcircuit|i've been trying to improve the wallet
114 2016-07-01 09:07:55	0|wumpus|yes, thanks for that
115 2016-07-01 09:08:14	0|phantomcircuit|unfortunately yeah it's a lot of running around in the weeds trying to fix things up... to make it easier to do bigger things
116 2016-07-01 09:08:30	0|wumpus|all in all the utxo/coin based approach does cause a lot of non-trivial difficulties, both at the node (utxo set sprawl) level as in the wallet
117 2016-07-01 09:08:38	0|gmaxwell|in any case, I didn't bring any of this up here to complain-- I brought it up because initially I couldn't figure out the cause, and then updated when I did.
118 2016-07-01 09:09:31	0|gmaxwell|it avoids a lot of replay problems however, which avoids other kinds of sprawl
119 2016-07-01 09:09:49	0|wumpus|no, I'm not trying to complain either, but sometimes I just don't know anymroe
120 2016-07-01 09:10:52	0|gmaxwell|well, from another perspective-- we have no deadline. These same issues existed in 2011 (actually, much worse), but it didn't matter because hardly anyone used the system. :)
121 2016-07-01 09:11:15	0|wumpus|but that's part of what worries me, yes these same issues existed in 2011
122 2016-07-01 09:11:58	0|wumpus|no, no deadline, but e.g. the utxo growth is worrying
123 2016-07-01 09:12:02	0|wumpus|we're running against (soft) walls
124 2016-07-01 09:12:10	0|gmaxwell|yes, I'm concerned about utxo growth.
125 2016-07-01 09:12:37	0|gmaxwell|I happened to chart data from all those reindexing tests I just did, and the time to verify a block is increasing (for the same amount of txn data).
126 2016-07-01 09:13:15	0|wumpus|and all the while joe user is complaining about lack of *scaling*, while the system is already seemingly bursting at its seems
127 2016-07-01 09:13:24	0|gmaxwell|I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
128 2016-07-01 09:13:51	0|gmaxwell|so I'm going to try to figure out if thats the case.
129 2016-07-01 09:14:39	0|gmaxwell|(by increasing, I mean increasing enough that it was clearly visible on a last-8064 block plot)
130 2016-07-01 09:15:16	0|wumpus|"the time to verify a block is increasing (for the same amount of txn data" yes, indeed, I also intended to plot time per block, but that much was clear just from looking at timestamps :(
131 2016-07-01 09:16:23	0|gmaxwell|well the difference between 1MB and smaller blocks was expected, but the increase still existed when looking at only large blocks.
132 2016-07-01 09:16:35	0|wumpus|yes
133 2016-07-01 09:16:49	0|wumpus|recent blocks verify slower
134 2016-07-01 09:17:03	0|wumpus|even of the same size
135 2016-07-01 09:17:21	0|wumpus|so *let alone* what bigger blocks would do
136 2016-07-01 09:17:53	0|gmaxwell|At least segwit doesn't increase the worst case amount of utxo growth or accessing...
137 2016-07-01 09:56:48	0|pedrobranco|sipa: hi, when you have the time can you help on the doubts in https://github.com/bitcoin/bitcoin/pull/7551? Thanks :)
138 2016-07-01 10:04:14	0|sipa|pedrobranco: i answered quickly
139 2016-07-01 10:04:58	0|sipa|pedrobranco: feel free to disagree by the way... if my view is not obvious here, perhaps the semantics should be different and more clear
140 2016-07-01 10:09:06	0|pedrobranco|sipa: thanks.
141 2016-07-01 10:38:00	0|wumpus|wow, that wallet drop-unnecessary-inputs-from-best-subset change had been open from sep 2014
142 2016-07-01 10:38:07	0|wumpus|https://github.com/bitcoin/bitcoin/pull/4906
143 2016-07-01 10:40:46	0|wumpus|it has not yet made it into any release
144 2016-07-01 10:41:04	0|MarcoFalke|it has
145 2016-07-01 10:41:14	0|MarcoFalke|it was backported to .12
146 2016-07-01 10:41:19	0|wumpus|fuck
147 2016-07-01 10:41:35	0|wumpus|yes I see: 96e8d120336cf4312cd5f42ba2f9aff17d4ad414
148 2016-07-01 10:42:04	0|wumpus|that was my own stupid idea probably
149 2016-07-01 10:43:37	0|MarcoFalke|Up to now I have not seen proof that this made things considerably worse
150 2016-07-01 10:44:07	0|MarcoFalke|Someone should come up with a framwork that mimics common wallet use cases
151 2016-07-01 10:44:17	0|wumpus|ok
152 2016-07-01 10:44:26	0|MarcoFalke|e.g 'exchange': something with a huge in-out volume/count
153 2016-07-01 10:44:56	0|MarcoFalke|then a "normal user": always waits for confirmations, low volume
154 2016-07-01 10:45:21	0|wumpus|right, that would make sense, for a project taking the wallet part seriously
155 2016-07-01 10:45:45	0|MarcoFalke|and then different behaviors like 'always send small coins' (pay for coffee each day)
156 2016-07-01 10:46:12	0|MarcoFalke|and 'receive large coins' (get payed in btc)
157 2016-07-01 10:46:14	0|MarcoFalke|etc
158 2016-07-01 10:47:58	0|wumpus|right.
159 2016-07-01 10:49:19	0|wumpus|gmaxwell mentioned above that he did see some reports of worse behavior after the change
160 2016-07-01 10:49:29	0|wumpus|now it's not clear whether to revert it or not
161 2016-07-01 10:50:14	0|gmaxwell|We recieved an issue with a business with a large wallet reporting specifically the expected behavior there.
162 2016-07-01 10:50:46	0|wumpus|why don't businesses with large wallets never help with the wallet development?
163 2016-07-01 10:51:04	0|wumpus|it seems it would be in their best interest
164 2016-07-01 10:51:10	0|gmaxwell|because there are very few left, most are using fully hosted things that have custom software.
165 2016-07-01 10:51:14	0|wumpus|but I don't think this was even reported on github
166 2016-07-01 10:51:20	0|gmaxwell|I thought it was.
167 2016-07-01 10:51:23	0|wumpus|it's just frustrating
168 2016-07-01 10:51:26	0|MarcoFalke|it was
169 2016-07-01 10:51:31	0|wumpus|oh?
170 2016-07-01 10:51:39	0|gmaxwell|And utxo growth coincided with it. This was previously discussed in meetings IIRC and then ... I'm not sure what happened.
171 2016-07-01 10:52:13	0|wumpus|that's not important. What is important is what to do now
172 2016-07-01 10:52:17	0|wumpus|revert it?
173 2016-07-01 10:52:38	0|MarcoFalke|#7664 #7657
174 2016-07-01 10:53:02	0|gmaxwell|maybe it was dropped for good reason. I just can't recall.
175 2016-07-01 10:54:02	0|gmaxwell|At the end of the day, the pruning itself was correcting a bug. The potential issue is that the bug was covering up that the non-bugged behavior is bad (and the simulations showed that it, as I said, would grind down wallets into lots of tiny coins)
176 2016-07-01 10:54:10	0|MarcoFalke|Imo it shouldn't matter if we revert it or not. I haven't yet seen simulations which prove that either would place an advatage/disatvantage
177 2016-07-01 10:54:38	0|gmaxwell|MarcoFalke: IIRC there were simulations posted that showed that the behavior caused far more utxo.
178 2016-07-01 10:55:08	0|MarcoFalke|No one verified that the simulations were implemented according to the current coin selection code
179 2016-07-01 10:55:18	0|MarcoFalke|I have seen single satoshi outputs in those simulations
180 2016-07-01 10:55:26	0|MarcoFalke|which does not happen with our code
181 2016-07-01 10:55:42	0|MarcoFalke|Also the author found a bug in the implementation about two weeks ago or so
182 2016-07-01 10:55:44	0|gmaxwell|The result is also intutive to me. It results in making the smallest possible outputs. This is pessimal when you consider that avoiding utxo size wants, for a given users balance, the largest value outputs.
183 2016-07-01 10:56:41	0|MarcoFalke|Ideally, we'd implement the wallet benchmark framework in cpp and have it just use our wallet code?
184 2016-07-01 10:57:00	0|wumpus|if it would be modular enough for that :)
185 2016-07-01 10:57:45	0|wumpus|possibly separate the coin selection logic out to a separate file, which could also be used by the simulation framework
186 2016-07-01 10:58:04	0|wumpus|it could use an abstraction of a list of coins with some properties instead of a wallet as input
187 2016-07-01 10:58:41	0|wumpus|it would make some things a lot easier, like trying out what the algo does in certain circumstances, without actually having to fake a wallet
188 2016-07-01 10:59:46	0|gmaxwell|An even better behavior would be to add all other inputs to the same addresses being spent, up to some limit to prevent very large transactions, to sweep them up. This is a privacy improving strategy, as well as a fee minimizing strategy under an assumption that fees will increase in the future.   My guess is that I didn't NO DONT the pruning change because in and of itself it was right, and the
189 2016-07-01 10:59:52	0|gmaxwell|grinding should be fixed by something like this.
190 2016-07-01 11:00:12	0|gmaxwell|but it went off my radar, also changes to coinselection are a pain because a bunch of th tests freeking hard code the behavior.
191 2016-07-01 11:00:40	0|wumpus|if that was the only thing making changes to coin selection a pain :)
192 2016-07-01 11:01:18	0|wumpus|it's just almost impossible to agree what is better behavior, which is what talled #4906 for so long and resulted in it eventually being merged, probably wrongly
193 2016-07-01 11:02:06	0|wumpus|so a) it should be better behavior b) it needs to be implemented correctly, and indeed, we have no good tests or simulation framework
194 2016-07-01 11:04:32	0|wumpus|sweeping up as many as possible inputs to the same address would make sense
195 2016-07-01 11:04:48	0|wumpus|there is no privacy benefit from having more spends to one address
196 2016-07-01 11:04:56	0|wumpus|then again it assumes bad-case behavior from the user, reusing addresses
197 2016-07-01 11:05:29	0|gmaxwell|I don't want to be hard on you, but Xekyo ran simulations which showed 25% to 400% increase in the UTXO set size under some simulation loads.
198 2016-07-01 11:05:38	0|gmaxwell|We had simulations, they might not have been ideal.
199 2016-07-01 11:05:38	0|wumpus|I think it would make sense to design a standard interface for wallet coin selection algorithms, separately from any sepecific wallet
200 2016-07-01 11:05:57	0|wumpus|so that research and simuation can be done outside of the specific scope of bitcoin core
201 2016-07-01 11:06:08	0|gmaxwell|They showed bad behavior from this change. From intution I predited specifically this bad behavior and asked for the simulations.  Our issue is not that we need more simulations.
202 2016-07-01 11:06:16	0|gmaxwell|(well we do, but that didn't help here!)
203 2016-07-01 11:06:30	0|wumpus|ok, never mind then
204 2016-07-01 11:06:38	0|gmaxwell|and still MarcoFalke is not agreeing that there is a potential issue.
205 2016-07-01 11:06:47	0|wumpus|just trying to think of some things that might help, out of the blue
206 2016-07-01 11:06:53	0|gmaxwell|yea.
207 2016-07-01 11:06:56	0|wumpus|if it's all hopeless I'll just shut up
208 2016-07-01 11:07:27	0|gmaxwell|xekyo also identifyied a useful strategy, making the coinselection target double the value instead of the value. This results in change of useful sizes.
209 2016-07-01 11:07:41	0|wumpus|*more* simulations may not be a solution, but better, correctly implemented ones would
210 2016-07-01 11:08:32	0|wumpus|if the answer of this is up to whether the simulation is implemented correctly, then we're not really helped a bit, I agree
211 2016-07-01 11:09:27	0|gmaxwell|re: assumes bad-case behavior from the user, well, indeed, it's no effect if the user doesn't reuse. But no harm, and reuse is ubiquitious. The (vast?) majority of bitcoins in circulation are held in reused addresses.
212 2016-07-01 11:09:57	0|wumpus|the vast majority of those may not  be using bitcoin core in the first place
213 2016-07-01 11:10:38	0|wumpus|many wallets encourage address reuse, or can only do address resuse
214 2016-07-01 11:10:42	0|wumpus|that rule may help them a lot :)
215 2016-07-01 11:11:15	0|gmaxwell|pretty much every active user has some reuse though, consider that thing that sends tips for commits.
216 2016-07-01 11:11:36	0|wumpus|anyhow that speaks too of generalizing the coin selection algorithm beyond the specific software level
217 2016-07-01 11:11:37	0|gmaxwell|means that all of us have reuse, even if we otherwise act perfectly ourselves. :)
218 2016-07-01 11:11:51	0|wumpus|yes, that is true, ideally that should use some BIP32 construction
219 2016-07-01 11:12:13	0|wumpus|to be honest I always use manual coin selection
220 2016-07-01 11:12:48	0|wumpus|(apart from testing)
221 2016-07-01 11:13:13	0|gmaxwell|I do too. (and always spend all the coins connected to an address, when I spend any)
222 2016-07-01 11:14:00	0|wumpus|and that hould ideally coincide with changing the donation address
223 2016-07-01 11:14:07	0|gmaxwell|they've gotta get spent someday or the coins are lost, fees are likely lower now than in the future... and spending at once avoids privacy harm from constantly interlinking inside the wallet.
224 2016-07-01 11:14:36	0|gmaxwell|yea, I do that with my bct address. change the one on the site, when I spend all the coins.
225 2016-07-01 11:15:02	0|gmaxwell|for that tip commit thing, I just haven't been spending its payments, I think... as I dunno how to change it.
226 2016-07-01 11:16:10	0|wumpus|they have a little web interface where you can log in and change the address IIRC
227 2016-07-01 11:18:19	0|gmaxwell|I wonder what the average coin value is in the utxo set and how thats evolved over time. (it's not quite as simple as the set size evolution over time, since more coins have also been introduced)
228 2016-07-01 11:19:32	0|gmaxwell|"errors": "WARNING: abnormally high number of blocks generated, 4477 blocks received in the last 4 hours (24 expected)"
229 2016-07-01 11:19:35	0|gmaxwell|lol
230 2016-07-01 11:20:12	0|wumpus|yes that would be an interesting utxo statistic
231 2016-07-01 11:20:48	0|wumpus|gmaxwell: https://github.com/bitcoin/bitcoin/pull/8275 :p
232 2016-07-01 11:27:15	0|GitHub78|[13bitcoin] 15laanwj opened pull request #8298: wallet: Revert input selection post-pruning (06master...062016_06_revert_wallet_input_postprune) 02https://github.com/bitcoin/bitcoin/pull/8298
233 2016-07-01 11:37:34	0|sturles|I can see some HD related pull requests with no milestone set.  Any chance of HD support in 0.13, or is it 0.14 material?
234 2016-07-01 11:37:47	0|wumpus|basic HD support has been merged for 0.13
235 2016-07-01 11:38:11	0|wumpus|the feature freeze for 0.13 is long past now though, so everything else is for 0.14 soonest
236 2016-07-01 11:38:18	0|wumpus|feel free to help testing them though, that helps a lot
237 2016-07-01 11:38:58	0|wumpus|if anything there's a lack of involvement and testing and review for wallet PRs, especially UI oriented ones
238 2016-07-01 11:39:40	0|gmaxwell|we certantly had wallet features slip due to lack of love.
239 2016-07-01 11:42:38	0|sturles|I haven't used the UI for years..  Will start testing HD stuff.
240 2016-07-01 11:51:12	0|wumpus|at least for BIP32, in contrast to coin selection, the goal is well-defined and delimited :)
241 2016-07-01 11:56:46	0|murch|wumpus: I'm working on the latter. :)
242 2016-07-01 11:57:36	0|wumpus|murch: awesome!
243 2016-07-01 11:58:28	0|murch|I'm writing a Master thesis on Coin Selection. I've been talking about that to some people here before.
244 2016-07-01 11:58:42	0|murch|(Pieter, Greg, Greg, Marco)
245 2016-07-01 11:59:02	0|wumpus|great, some more general reserach there would be very welcome
246 2016-07-01 11:59:38	0|murch|One focus will be trying to analyze goals, prerequisites and constraints, then the results of varying approaches for selection
247 2016-07-01 12:00:15	0|wumpus|yea exactly, what I posted here https://github.com/bitcoin/bitcoin/pull/8298#issuecomment-229928583  I think what is needed first is a list of criteria to judge coin selection algorithms by
248 2016-07-01 12:00:48	0|wumpus|right now we can't tell up from down
249 2016-07-01 12:01:11	0|wumpus|(unless it is obvious breakage)
250 2016-07-01 12:03:54	0|murch|Thanks, I hadn't seen that one yet.
251 2016-07-01 12:04:16	0|murch|I was actually the one that provided the simulation in 4096
252 2016-07-01 12:05:05	0|wumpus|I mean there are obvious concerns such as CPU usage and memory usage, fee minimization, but also more 'tragedy of the commons' issues such as privacy concerns, utxo growth concerns (though that also coincides with performance a bit, keeping the wallet's utxo set small keeps the global set also smaller)
253 2016-07-01 12:06:05	0|murch|Actually, I think that the pruning should have had little effect, as there should only be anything to prune when a second pass is needed. Otherwise, since the last added UTXO would be the smallest, there should not be any UTXO prunable.
254 2016-07-01 12:06:45	0|wumpus|it has very little effect
255 2016-07-01 12:06:55	0|wumpus|(but apparantly visible in some cases)
256 2016-07-01 12:08:00	0|wumpus|though of course it's not entirely certain whether the reported problems were due to this change, or another change, or a change in the general usage of bitcoin not directly related to a change in our wallet
257 2016-07-01 12:08:51	0|wumpus|or a combination of factors including this change
258 2016-07-01 12:09:59	0|wumpus|in any case, if it has 'little effect' that's also enough reason to revert, I think. If it achieves hardly anything good, it shouldn't have been changed.
259 2016-07-01 12:10:13	0|murch|wumpus: Completely agree.
260 2016-07-01 12:11:47	0|murch|Well, I hope that I'll be able to provide some real improvements in the following months. :)
261 2016-07-01 12:12:41	0|murch|Although I was surprised how well the current implementation does. It's not trivial to improve on it, and not make it deterministic in some fashion. :)
262 2016-07-01 12:13:41	0|murch|Oh, and providing some metrics to compare Coin Selection approaches is the focus of the work
263 2016-07-01 12:13:46	0|wumpus|I hope so too!
264 2016-07-01 12:14:15	0|wumpus|good to hear the current approach is fairly good, on the other hand, that is going to make it even harder to replace by something better :)
265 2016-07-01 12:14:46	0|wumpus|I think the most common complaints are that it does bad with very large wallets, or containing lots of small inputs
266 2016-07-01 12:14:52	0|murch|By the way, wumpus, do you know of any other wallet usage data? I got the moneypot.com wallet from #4096, but more good testcases would be grand.
267 2016-07-01 12:14:58	0|wumpus|apart from that it's not something people usually stumble upon
268 2016-07-01 12:15:46	0|wumpus|no, unfortunately not
269 2016-07-01 12:16:00	0|wumpus|I don't have any realistic big wallets
270 2016-07-01 12:16:13	0|murch|Okay, too bad. :)
271 2016-07-01 12:16:58	0|wumpus|(this is also an issue which has existed from 2011, which affects both coin selection and general wallet scaling work, people with big wallets are very reluctant to share them even with developers :-) )
272 2016-07-01 12:17:35	0|murch|Yeah, I'd imagine. :-/
273 2016-07-01 12:18:15	0|wumpus|which is also why I always encourage companies which have to cope with them to get involved in development themselves, that avoids having to share anything with third parties
274 2016-07-01 12:18:27	0|murch|The code in wallet.cpp is pretty hard to understand, too. I'd love to refactor it, maybe even to just understand it properly…
275 2016-07-01 12:18:46	0|wumpus|phantomcircuit is working on that too
276 2016-07-01 12:18:53	0|murch|oh really?!
277 2016-07-01 12:19:21	0|murch|mh, just refactoring it, or also improving?
278 2016-07-01 12:19:31	0|wumpus|in a way it's kind of a chicken and egg problem though, people rarely dare to change things *becuase* they're not sure how they work exactly, but evaluating refactors is also made hard by that because it's non-trivial to see that behavior stays the same
279 2016-07-01 12:20:35	0|wumpus|both, but not coin selection AFAIK
280 2016-07-01 12:20:36	0|murch|Yeah, on the other hand it's in a state where it is really hard to add unittests to check whether the behavior remains unchanged.
281 2016-07-01 12:20:46	0|murch|okay, excellent
282 2016-07-01 12:20:49	0|wumpus|yes, that too.
283 2016-07-01 12:23:54	0|wumpus|and years of proposals of doing things differently have made me think that it being hard to understand is not so much a result of the organizatin of wallet.cpp, but that the underlying subject matter is hard
284 2016-07-01 12:24:46	0|wumpus|people tend to read the code and then blame that the code is difficult, but it's not entirely clear whether it is *unnecessarily* difficult
285 2016-07-01 12:25:23	0|wumpus|it's clearly a complex problem too
286 2016-07-01 12:25:41	0|wumpus|maybe it could be divided up into better manageb;e sub-problems though
287 2016-07-01 12:25:47	0|wumpus|manageable*
288 2016-07-01 12:27:47	0|wumpus|the thing is, before wallet.cpp was written, cryptocurrency wallets was a problem no one ever considered before, it was necessarily ad-hoc
289 2016-07-01 12:28:25	0|wumpus|this is very different from say, a web server, where everyone knows what is expected from it, how it usually should be structured, and so on
290 2016-07-01 12:30:46	0|murch|wumpus: It's surely complicated, but there are some methods that could be extract, and some things are just somewhat obfuscated by very brief variable names. E.g. I've been looking at how and when the fee gets decided for the transaction and it's still baffling me.
291 2016-07-01 12:30:49	0|wumpus|so the refactor is also a discovery process, learning how to best structure this novel application
292 2016-07-01 12:32:28	0|wumpus|well yes it's complicated - but does it need to be complicated, is there a less complex way that would be just as good ,or better? would that simpler way still satisfy the requirements? (and in some cases: what are the requirements, even?)
293 2016-07-01 12:34:02	0|murch|wumpus: I think it could be more comprehensible if key management and coin selection were separated into different classe for example. They are pretty far apart for being part of one class.
294 2016-07-01 12:34:04	0|wumpus|things like variable names are trivialities, sure, on first reading of the code it helps to have better names, but once you learn what they are it doesn't really matter what they are anymore (e.g. mathematicians do fine with names such as a b c :)
295 2016-07-01 12:34:36	0|murch|Most of Coin Selection could actually be completely static, as it basically is just reliant on the spending target, the UTXO set and the desired fee level.
296 2016-07-01 12:34:49	0|wumpus|yes coin selection is a clear unit
297 2016-07-01 12:35:18	0|wumpus|I think it would be nice to factor it out to a separate unit with a separate interface, that just gets the information that it needs and returns the selected list
298 2016-07-01 12:35:29	0|wumpus|this would also be useful for simulations
299 2016-07-01 12:35:31	0|murch|well, wallet.cpp is just a bit of a moloch to delve into at 3500 LOC. ;)
300 2016-07-01 12:35:46	0|murch|wumpus: exactly
301 2016-07-01 12:35:49	0|wumpus|I've seen so much worse at some companies I've worked at :-)
302 2016-07-01 12:36:40	0|murch|Also, adding, watching and tracking UTXO could probably be separated off
303 2016-07-01 12:36:54	0|wumpus|yes, that would make sense
304 2016-07-01 12:37:19	0|wumpus|did you see https://github.com/bitcoin/bitcoin/pull/7823 ?
305 2016-07-01 12:38:07	0|murch|nope, but I just subscribed.
306 2016-07-01 12:38:27	0|murch|I am not nearly as familiar with the github repository as I'd like. :-/
307 2016-07-01 12:38:40	0|wumpus|the idea is quite nice, but it's *hard* to see what its interaction with different things would be, such as reorganizations and conflicts
308 2016-07-01 12:38:46	0|murch|I should probably check out all issues tagged with "wallet" soon.
309 2016-07-01 12:38:52	0|wumpus|yes :)
310 2016-07-01 12:39:45	0|wumpus|in any case, coin selection in itself is enough of a subject to fill a master's thesis I think, I'd warn against scope creep, or trying to fix the world at once :)
311 2016-07-01 12:39:53	0|murch|hehe
312 2016-07-01 12:41:03	0|murch|yeah, it's a bit overwhelming at times. I've been reading about Subset Sum solvers the past days, and delving through wallet.cpp to understand how fees are handled.
313 2016-07-01 12:42:36	0|murch|Anyway, back to work. ;) TTYL
314 2016-07-01 12:43:09	0|wumpus|later :) hope you manage to make head and tail of it
315 2016-07-01 12:43:24	0|wumpus|hm the new option -blockmaxcost is going to give user understanding problems: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L455
316 2016-07-01 12:43:52	0|wumpus|should this be a hidden/debug option? if not, it should be documented better
317 2016-07-01 13:18:24	0|jonasschnelli|sipa, wumpus: Do I get this right: -reindex does re-create the block index and recreates the utxo set (including all signature validation)    while -reindex-chainstate will only recreate the utxo set with the current block index?
318 2016-07-01 13:20:27	0|sdaftuar|wumpus: -blockmaxcost is supposed to be the recommended way to configure the mining code, going forward
319 2016-07-01 13:21:25	0|sdaftuar|wumpus: and -blockmaxsize is in the we-want-to-deprecate-in-the-future category
320 2016-07-01 13:21:44	0|sdaftuar|(the mining code is optimizing for fee-per-block-cost, not fee-per-serialized-byte)
321 2016-07-01 13:22:05	0|sdaftuar|but yes, we need to document all this before release
322 2016-07-01 13:22:55	0|sdaftuar|heh, yeah that --help message text is not very informative
323 2016-07-01 13:30:55	0|wumpus|sdaftuar: a translator was wondering what kind of cost this was, I think they assume that it's something with fee
324 2016-07-01 13:31:10	0|wumpus|jonasschnelli: correct
325 2016-07-01 13:31:12	0|sdaftuar|oops.  yeah that is bad language
326 2016-07-01 13:31:58	0|sdaftuar|what's the timeline for improving that text, is it too late for making changes now that affect translation?
327 2016-07-01 13:32:06	0|sdaftuar|i guess we can just document in the release notes?
328 2016-07-01 13:32:31	0|wumpus|well given that translators are confused by it (and hence unable to translate it effectiely) I wouldn't mind changing the message
329 2016-07-01 13:33:20	0|wumpus|there also needs to be mention in the release notes, but release notes aren't really documentation, just 'news'
330 2016-07-01 13:33:46	0|wumpus|e.g. you wouldn't assume that someone that is looking for documentation for an option to go through all previous release notes to find it documented
331 2016-07-01 13:33:55	0|sdaftuar|a simple change might just be to reference the BIP where it's defined; that wouldn't necessarily impose an additional burden on translators if it's just "(BIP 141)" or something at the end of it
332 2016-07-01 13:34:13	0|wumpus|what needs to be made clear is that this is just an abstract cost
333 2016-07-01 13:34:30	0|wumpus|referencing the BIp would make sense, yes
334 2016-07-01 13:34:53	0|wumpus|in any case, even if it's too late for this to be translated, a better english documentation message would go further than a confusing translated one :)
335 2016-07-01 13:35:17	0|sdaftuar|agreed!
336 2016-07-01 13:35:44	0|sdaftuar|perhaps block cost shouldn't be translated at all actually, if we're referencing the BIP
337 2016-07-01 13:39:32	0|sdaftuar|i updated #8294 so we don't forget this
338 2016-07-01 13:43:22	0|MarcoFalke|gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
339 2016-07-01 13:43:38	0|MarcoFalke|Right now they save us from introducing accidental regressions, which is nice
340 2016-07-01 13:44:09	0|MarcoFalke|If someone comes up with a new idea, the unit test need to go anyway and will be replaced by new ones, so not really an issue
341 2016-07-01 13:45:26	0|MarcoFalke|Also the 25% to 400% performance loss may be flawed, as the coin generator was not adjusted to what happens in the real network
342 2016-07-01 13:45:34	0|MarcoFalke|(re simulation)
343 2016-07-01 13:47:20	0|MarcoFalke|I remember there was a site which put every address which was ever in a wallet. (Combining whenever two addresses happen to be used in the inputs of the same tx)
344 2016-07-01 13:47:46	0|MarcoFalke|This could be useful to verify coin generators in simulations are doing their job properly
345 2016-07-01 15:00:57	0|morcos|wumpus: gmaxwell: i don’t feel strongly enough to make a big argument about it, but if it was up to me i wouldn’t bother reverting 4906.  I agree that we didn’t have sufficient justification to merge it in the first place, but we already crossed that bridge, and discussed it after the fact (more than once).   I’m not sure we’re confident enough that its clearly worse to risk making the same mistake in the other directi
346 2016-07-01 15:01:10	0|morcos|direction by reverting it
347 2016-07-01 15:01:23	0|morcos|I suppose my point is I’d err on the side of being conservative with changes to coin selection and only making them when someone has put in the effort to really study them.  Since 0.12 has been out for a while running with 4906, it now seems like a change to me to revert it.
348 2016-07-01 15:14:51	0|jonasschnelli|sipa: A full IBD (against random peers) with current master took ~3h4min, a -reindex-chainstate took ~1h10min
349 2016-07-01 15:15:07	0|jonasschnelli|http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.full.txt
350 2016-07-01 15:15:29	0|jonasschnelli|IBD: http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.full.txt
351 2016-07-01 15:15:40	0|jonasschnelli|now doing a full -reindex
352 2016-07-01 15:15:58	0|jonasschnelli|Used -dbcache=8000 for both runs
353 2016-07-01 15:37:07	0|wumpus|morcos: I don't think 'we crossed that bridge' is a good argument, if this was no improvement, it should not have been merged, and it should be reverted (it should have been reverted a long time ago!)
354 2016-07-01 15:37:21	0|wumpus|morcos: I don't see any reason to keep the code if it i not an improvement
355 2016-07-01 15:37:52	0|wumpus|and if we should be conservative about changes to coin selection, which we should have been in the first place, then again we shouldn't have this change without understanding it
356 2016-07-01 15:38:20	0|morcos|wumpus: ok like i said i won't argue, it just makes me nervous that somehow reverting could be worse (interaction with something else that changed that we're not considering), but i don't have any reason to believe that
357 2016-07-01 15:38:20	0|wumpus|I don't think having the mistake out in 0.12 is  a reason to keep it now
358 2016-07-01 15:38:25	0|wumpus|don't get attached to your mistakes :)
359 2016-07-01 15:38:50	0|morcos|i won't get attached to somebody's mistakes  :)
360 2016-07-01 15:39:08	0|wumpus|then again I don't feel strongly about it either, but I'm tired of it coming up every time
361 2016-07-01 15:39:17	0|wumpus|let's just make a damn decision about it
362 2016-07-01 15:39:29	0|wumpus|I'm sure if we decide not to revert it now, then someone will bring it up again after a month or so
363 2016-07-01 15:39:54	0|wumpus|I don't want this following me around forever
364 2016-07-01 15:41:00	0|wumpus|I don't see how this could interact with anything else
365 2016-07-01 15:44:17	0|bsm1175321|Woah.  I just built github master and on testnet something has gone horribly wrong: I have a negative balance!
366 2016-07-01 15:45:03	0|morcos|bsm1175321: checked how?
367 2016-07-01 15:45:14	0|bsm1175321|-56.85287200
368 2016-07-01 15:45:14	0|bsm1175321||mcelrath@jane:~> bitcoin-cli -testnet getbalance ''
369 2016-07-01 15:45:24	0|sipa|drop the ''
370 2016-07-01 15:45:28	0|bsm1175321|It seems to have suddenly fixed itself.
371 2016-07-01 15:45:31	0|sipa|you're computing an account balance
372 2016-07-01 15:45:36	0|sipa|not the wallet balance
373 2016-07-01 15:45:50	0|wumpus|you used to wrong way to query your balance! woe you!
374 2016-07-01 15:46:07	0|bsm1175321|0.00000000
375 2016-07-01 15:46:07	0|bsm1175321||mcelrath@jane:~> bitcoin-cli -testnet getbalance
376 2016-07-01 15:46:07	0|bsm1175321|Without the quotes I got:
377 2016-07-01 15:46:10	0|morcos|i hesitate to ask this question, but can we finally remove accounts for 0.14
378 2016-07-01 15:46:15	0|wumpus|(we're crazy that there are so many, including wrong ways to query your balance)
379 2016-07-01 15:46:22	0|morcos|talk about having an issue follow us around
380 2016-07-01 15:46:22	0|wumpus|morcos: I tried to push for a label API to replace it
381 2016-07-01 15:46:30	0|bsm1175321|Which is also wrong.  A few minutes later it seems to have synced up...and is now showing correct balances.  What happened?
382 2016-07-01 15:46:42	0|sipa|there was a 4000 deep reorg
383 2016-07-01 15:46:48	0|wumpus|morcos: unnecessary to say, it didn't get enough attention, and is still lingering, I hope to pick it up again for 0.14
384 2016-07-01 15:47:12	0|bsm1175321|+1 for getting rid of accounts...
385 2016-07-01 15:47:26	0|wumpus|morcos: https://github.com/bitcoin/bitcoin/pull/7729   I think the release after that we can really remove the account functionality
386 2016-07-01 15:47:57	0|morcos|wumpus: ok 3rd topic. i thought i head you guys mention that something has gotten slower recently.  is it just reindex, or actually something with connecting blocks?
387 2016-07-01 15:48:12	0|bsm1175321|Next question: I need to fund transactions and be sure that their txid is not malleable.  I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not.  What's the best way to achieve this?
388 2016-07-01 15:48:20	0|wumpus|morcos: AFAIK the summary there was: gmaxwell accidentally -txindex
389 2016-07-01 15:48:46	0|sipa|morcos: and nobody recently tested -reindex with default -dbcache
390 2016-07-01 15:48:56	0|morcos|wumpus: oh i definitely missed that conclusion. :)
391 2016-07-01 15:48:57	0|wumpus|morcos: yes, more recent blocks are slower to validate, comapred to older blocks of the same size - but this isn't a reversion in the code
392 2016-07-01 15:49:12	0|morcos|wumpus: wait, yes its that last thing i'm asking about
393 2016-07-01 15:49:14	0|morcos|why is that?
394 2016-07-01 15:49:25	0|wumpus|it's likely a by-product of increasing utxo size
395 2016-07-01 15:50:06	0|morcos|and that affecting cache hit rate?
396 2016-07-01 15:50:08	0|wumpus|and the utxo database is reaching the size that leveldb can handle w/ good performance, e.g. now lots of crazy seeking and reading all over your disk, can't be good for performance
397 2016-07-01 15:50:14	0|morcos|oh
398 2016-07-01 15:50:37	0|wumpus|yes that was another hypothesis
399 2016-07-01 15:50:51	0|wumpus|<gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
400 2016-07-01 15:51:00	0|morcos|eyeballing the numbers i'm not seeing a noticeable slowdown since march
401 2016-07-01 15:51:04	0|wumpus|which probably means it's a combination of both
402 2016-07-01 15:52:08	0|wumpus|in any case the increased default dbcache should alleviate either problem, at least for a while
403 2016-07-01 15:53:26	0|morcos|ok, yeah my numbers are with a big dbcache (2G)
404 2016-07-01 15:53:28	0|wumpus|there may be a better way to structure the utxo set on disk - but having looked into other databases it seems that leveldb has the best performance for our use, at least during sync
405 2016-07-01 15:54:17	0|wumpus|if you set a huge dbcache you won't notice much, as sipa says it's probably why the surprise that indexing was so slow with default dbcache
406 2016-07-01 15:54:30	0|sipa|for 0.14 we should prioritize working on chainstate backups
407 2016-07-01 15:55:06	0|sipa|(and vigorously fight the potential push for servers with 'helpful' backups to download...)
408 2016-07-01 15:58:53	0|bsm1175321|I need to fund transactions and be sure that their txid is not malleable.  I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not.  What's the best way to achieve this?  (I'm willing to write such an option if there isn't a better way) @sipa?
409 2016-07-01 15:59:19	0|sipa|bsm1175321: patches welcome :)
410 2016-07-01 15:59:29	0|sipa|seems like a very useful feature
411 2016-07-01 15:59:31	0|bsm1175321|Ok so the answer is that this isn't possible currently?
412 2016-07-01 15:59:47	0|sipa|for now, you'll need listunspent
413 2016-07-01 15:59:48	0|bsm1175321|Cool, I'll see what I can do.
414 2016-07-01 16:00:02	0|sipa|i believe i've heard someone else ask for the same feature
415 2016-07-01 16:09:30	0|bsm1175321|One complication with such an option is that it's possible for your wallet to have enough funds to fund the transaction, but not enough segregated witness inputs.  A solution would be to create an intermediate transaction whose outputs have segregated witness.  Thoughts?
416 2016-07-01 16:09:49	0|sipa|fundrawtransaction shouldn't do such thing
417 2016-07-01 16:09:51	0|sipa|imho
418 2016-07-01 16:10:05	0|sipa|but it can fail with a nice error code, instructing you to clean up your wallet
419 2016-07-01 16:10:13	0|bsm1175321|You'd rather it fail in such a case?  That's fine with me too...
420 2016-07-01 16:10:24	0|sipa|yes, insufficient funds :)
421 2016-07-01 16:10:43	0|sipa|(or perhaps you should use separate wallets if dealing with that situation is hard)
422 2016-07-01 16:10:51	0|bsm1175321|What would you expect a user to do in such a case?  Is there a procedure to "clean up your wallet"?  That people will know?
423 2016-07-01 16:11:26	0|sipa|it's no different from the situation where you have a wallet with some watchonly coins, and the non-watchonly together are not enough to pay
424 2016-07-01 16:12:03	0|bsm1175321|True, sort of, except you have the funds.  I'll make a descriptive error message.
425 2016-07-01 16:15:48	0|sipa|i think the real solution is that if you have a need for segwit-only inputs, you run with a segwit-only wallet
426 2016-07-01 16:17:23	0|bsm1175321|In effect that's what I'll do.  But have to start from a non-segwit wallet.  Or is there a spent-to-myself-segwit-out wallet RPC call?
427 2016-07-01 16:18:49	0|sipa|you can just use sendtoaddress to send from the old wallet to the new?
428 2016-07-01 16:19:29	0|sipa|also, wallet integration of segwit is really preliminary at this point
429 2016-07-01 16:19:40	0|sipa|for example, change outputs won't be segwit
430 2016-07-01 16:19:55	0|bsm1175321|ooohhh...hmmm...
431 2016-07-01 16:20:09	0|sipa|(which they should be when it becomes ready for production, but that's not done yet)
432 2016-07-01 16:21:30	0|bsm1175321|Well if you 'fundrawtransaction ... segwitOnly' I can make it use a segwit change address.
433 2016-07-01 16:21:50	0|bsm1175321|Or you can add an explicit change address
434 2016-07-01 16:22:03	0|sipa|true
435 2016-07-01 16:22:31	0|bsm1175321|If no objections, I'll make the 'segwitOnly' option take only segwit inputs, and generate segwit change.
436 2016-07-01 16:35:13	0|instagibbs|what is/is there a way to link to an external secp library during compilation?
437 2016-07-01 16:41:43	0|bsm1175321|sipa: do you have a "wallet wishlist" for segwit?  I'm seeing that a lot of infrastructure is missing here.
438 2016-07-01 16:43:52	0|sipa|bsm1175321: thinks like default witness addresses
439 2016-07-01 16:43:58	0|sipa|post softfork
440 2016-07-01 16:53:45	0|bsm1175321|I'm confused regarding addresses.
441 2016-07-01 16:54:31	0|bsm1175321|What is the output of 'addwitnessaddress'?  (what kind of address is that?)
442 2016-07-01 16:57:33	0|sipa|p2sh
443 2016-07-01 16:58:29	0|bsm1175321|Ok, duh.  (hadn't made a testnet p2sh before, was confused by the 2)
444 2016-07-01 17:04:43	0|bsm1175321|Ok I think I understand.  A wallet bitcoin.conf really should have 3 possible values then: non-segwit, segwit, and segwit-in-p2sh.
445 2016-07-01 17:04:50	0|bsm1175321|*wallet setting
446 2016-07-01 17:05:17	0|bsm1175321|Do I understand correctly: The point of P2SH nesting is that non-upgraded wallets will check the destinations (though not the signatures)?
447 2016-07-01 17:15:56	0|jl2012|bsm1175321: I'd say the point of P2SH nesting is to allow non-upgraded wallets to send money to upgraded wallet
448 2016-07-01 17:20:11	0|bsm1175321|Can't they always do that though?  Do you mean the other way around?
449 2016-07-01 17:21:36	0|arubi|it's made for upgraded nodes to be able to accept payment from unupgraded nodes to a segwit scriptpubkey, yes
450 2016-07-01 17:23:09	0|jl2012|for upgraded -> not upgraded, you just use P2PKH address (1xyz......)
451 2016-07-01 17:25:37	0|arubi|bsm1175321, I'm curious, what kind of malleability are you trying to avoid?  are you the one signing the transaction?  can't you use normal inputs too and just sign it properly?
452 2016-07-01 17:26:41	0|bsm1175321|arubi: I'm going to be handing out txids, including txids that may be in the mempool, not mined yet.
453 2016-07-01 17:27:59	0|arubi|signed by you and you only?  you could still "maul"(?) if you wanted to, even with segwit
454 2016-07-01 17:28:32	0|jl2012|arubi: without BIP62, that's not reliable
455 2016-07-01 17:28:47	0|bsm1175321|The non-upgraded nodes that receive funds from an upgraded node doesn't verify signatures though.  (correct?)
456 2016-07-01 17:29:32	0|arubi|jl2012, talking about the anyonecanpay|single sighash, sign once, permute how many you'd like
457 2016-07-01 17:30:15	0|arubi|bsm1175321, the script that they see doesn't talk about checksigs, so they don't check any signatures
458 2016-07-01 17:30:21	0|jl2012|arubi: there are still other forms of third party malleability, e.g. non-canonical push
459 2016-07-01 17:30:37	0|bsm1175321|arubi: signed by me and me only.
460 2016-07-01 17:31:26	0|arubi|jl2012, sure, using standard scripts is a must here if you don't want 3rd parties doing things, but as a single signer, you could still do it to your own txs, even with segwit
461 2016-07-01 17:33:12	0|jl2012|well, if you want to guarantee non-malleability to only yourself, segwit is the only way
462 2016-07-01 17:33:16	0|bsm1175321|jl2012: can you elaborate?  Malleation of the script (which is part of the witness data) doesn't change the txid, and I don't think I care about that.
463 2016-07-01 17:34:06	0|arubi|jl2012, true.  I am asking because bsm1175321 explains that he will hand out txids (not the transactions themselves, as I understand)
464 2016-07-01 17:34:30	0|arubi|so as a single signer, he could still cause malleability, even if he manages to convince the other party all inputs are segwit
465 2016-07-01 17:34:53	0|arubi|*and standard scripts, or even better, even only with p2wpkh
466 2016-07-01 17:34:54	0|jl2012|arubi: that's double-spending
467 2016-07-01 17:35:03	0|jl2012|not malleability
468 2016-07-01 17:35:26	0|arubi|how so?  payments are still paid, but it's the order of the inputs outputs that's changed
469 2016-07-01 17:35:33	0|arubi|you're using the same inputs
470 2016-07-01 17:36:06	0|bsm1175321|arubi: Are you saying I could change the order of outputs, keeping the same segwit txid?
471 2016-07-01 17:36:18	0|arubi|no, you're changing it
472 2016-07-01 17:37:54	0|bsm1175321|Ok then someone looking for one txid I handed them won't see it at all, and there's no point in me giving the wrong txid to someone.  (A txid which never get mined because I malleated it is useless)
473 2016-07-01 17:38:32	0|arubi|right
474 2016-07-01 17:39:01	0|arubi|they should be checking their addresses for incoming transactions, and that's it.  txid is something that's only relevant when spending
475 2016-07-01 17:39:11	0|arubi|*should only be, I guess
476 2016-07-01 17:39:19	0|bsm1175321|However if they act on a txid in the mempool, and then I broadcast a malleated version which gets mined.  That's your usual double spend.
477 2016-07-01 17:40:16	0|arubi|how is it a double spend?  there is no spending from the malleated version
478 2016-07-01 17:40:23	0|pigeons|well if you only malleate the signature its not a double spend
479 2016-07-01 17:41:00	0|bsm1175321|Let's say I malleate the output order.  This is a weird thing to do.  I'll have to think on it...
480 2016-07-01 17:41:42	0|arubi|same can be done for inputs
481 2016-07-01 17:42:12	0|pigeons|if you have someone taking action on the txid of the unconfirmed transaction they will be affected, otherwise, not right
482 2016-07-01 17:43:09	0|arubi|a spender still has to reference a txid as input, so even if the outputs order isn't changed, the input is invalid in case of malleability
483 2016-07-01 17:43:56	0|arubi|oh you mean if the bad one wasn't mined, right
484 2016-07-01 17:48:21	0|bsm1175321|In this case, I'm proving control of coins.  Malleating my own txn is shooting myself in the foot.
485 2016-07-01 17:58:43	0|arubi|I'm guessing, you're proving control by announcing the txid before the transaction is seen on the network?
486 2016-07-01 18:08:51	0|bsm1175321|Not *before* but it could be simultaneous.  And I'm wondering what the consequences are of fiddling around...
487 2016-07-01 18:09:21	0|bsm1175321|But If I claim to prove something with a txid, and you look for it on the network, I'm only hurting myself by messing with it.  I'm having trouble thinking of any negative consequences here.
488 2016-07-01 18:11:24	0|arubi|why not use signed messages to prove ownership of addresses?  as long you commit to an address, which is kinda like commiting to specific coins
489 2016-07-01 18:17:38	0|arubi|hm.  I guess if it's a p2sh that's not so simple, because you'd have to disclose the redeemscript
490 2016-07-01 19:45:36	0|adiabat|hey, I'm spamming testnet and have some unexpected behavior
491 2016-07-01 19:46:06	0|adiabat|I think I get what "size", "cost", "strippedsize", "Vsize" all mean
492 2016-07-01 19:46:50	0|adiabat|"cost" is basically "Vsize" * 4
493 2016-07-01 19:47:05	0|adiabat|so a block "cost" must be < 4M
494 2016-07-01 19:47:11	0|adiabat|(<=, whatever)
495 2016-07-01 19:48:07	0|adiabat|so in .conf, blockmaxsize should set the cost limit of created blocks
496 2016-07-01 19:48:24	0|adiabat|but it seems to be targeting "size" instead
497 2016-07-01 19:49:08	0|bsm1175321|arubi: I want to be able to commit to coins regardless of the address type, and as you say, I'd need the redeemScript.
498 2016-07-01 19:54:34	0|kvnn|Hello everyone. I'm offering a 1BTC bounty for 3&4 here: https://github.com/drivechain-project/docs . Please let me know if you are interested. (if this is not okay in this channel I apologize & will discontinue posting about it)
499 2016-07-01 20:02:55	0|sdaftuar_|adiabat: -blockmaxsize continues to refer to total serialized bytes of a block, counting witness and non-witness parts the same. -blockmaxcost is the option that specifies the limit you probably want here.
500 2016-07-01 20:04:44	0|adiabat|hmmm ok
501 2016-07-01 20:04:48	0|adiabat|so there's both
502 2016-07-01 20:09:19	0|sdaftuar_|Yeah.  The hope is that -blockmaxsize is deprecated in the future.
503 2016-07-01 20:10:38	0|sdaftuar_|(My preference was to change the semantics of -blockmaxsize to refer to segwit's virtual size, but that's not what we ended up with)
504 2016-07-01 20:11:02	0|luke-jr|well, depends what you want to limit
505 2016-07-01 20:11:23	0|roasbeef|segwit miners on testnet seem to be limiting to ~750k cost
506 2016-07-01 20:11:31	0|luke-jr|sdaftuar_: then miners would need to set blockmaxsize to 250k to avoid making blocks >1M
507 2016-07-01 20:11:55	0|gmaxwell|06:43 < MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
508 2016-07-01 20:12:01	0|luke-jr|the goal of blockmaxsize is to avoid blocks larger than the size. cost/vsize doesn't matter here
509 2016-07-01 20:12:20	0|gmaxwell|I can tell you that personally I've written improvements that I haven't PRed because they required throwing out all the existing tests.
510 2016-07-01 20:12:21	0|sdaftuar_|My preference was to introduce a new option that would control serialized bytes
511 2016-07-01 20:12:51	0|luke-jr|sdaftuar_: why a new oppppption when we have one for it already?
512 2016-07-01 20:13:27	0|sdaftuar_|Because the mining code doesn't optimize for it
513 2016-07-01 20:13:40	0|sdaftuar_|So it's misleading to suggest it's supported
514 2016-07-01 20:14:05	0|luke-jr|it does what it's supposed to do.
515 2016-07-01 20:14:08	0|gmaxwell|I don't think it makes sense to even have a seralized size setting, we don't have a setting to limit the size of the sighashed bytes, or a setting to limit the size of the block's CTransaction encoding.  We don't have a limit to control the size of the compactblock form, or a limit to control the size of a zlib compressed block.
516 2016-07-01 20:14:46	0|gmaxwell|But I don't bother complaining because luke was going pyric over this before, and it shouldn't hurt that much having it either.
517 2016-07-01 20:14:51	0|luke-jr|gmaxwell: at this time, serialised size is a critical factor in practice
518 2016-07-01 20:15:23	0|sdaftuar_|gmaxwell: +1
519 2016-07-01 20:15:32	0|gmaxwell|luke-jr: critical for what? it doesn't reflect block propagation time especially well.
520 2016-07-01 20:15:58	0|luke-jr|on the p2p network it sure does..
521 2016-07-01 20:16:06	0|gmaxwell|validation time is more closely related to the number of utxo operations and how well relayed the txn in hte block were.
522 2016-07-01 20:16:24	0|gmaxwell|luke-jr: we have BIP152 now.
523 2016-07-01 20:16:36	0|luke-jr|not deployed, unfortunately.
524 2016-07-01 20:17:14	0|luke-jr|to be clear, I'm all for removing blockmaxsize once it doesn't matter and isn't used.
525 2016-07-01 20:17:23	0|luke-jr|hence why I don't think optimising block creation for it is important
526 2016-07-01 20:17:29	0|gmaxwell|(but even without it, seralized size is still not the ideal predictor of propagation time.)
527 2016-07-01 20:18:20	0|gmaxwell|K. indeed 0.12 doesn't have BIP152. Perhaps that does suggest that it should default to four million in 0.13?
528 2016-07-01 20:18:20	0|luke-jr|I just worry that we'll end up with >1 MB blocks before the network can handle it sanely
529 2016-07-01 20:18:28	0|gmaxwell|luke-jr: thats inevitable...
530 2016-07-01 20:18:48	0|gmaxwell|see also the utxo comments by wumpus last night.
531 2016-07-01 20:47:40	0|sturles|Am I correct that the 0.13 HD wallet implementation will only support new wallets, and only support hardened keys?
532 2016-07-01 20:47:49	0|gmaxwell|sturles: correct.
533 2016-07-01 20:48:09	0|sturles|Will it support importing private keys from an old style wallet?
534 2016-07-01 20:48:37	0|gmaxwell|I believe it does, I've not actually tested that!
535 2016-07-01 20:53:28	0|sturles|A HD wallet won't work with older versions, right?  Should use the opportunity to switch DB for the wallet to something > libdb 4.8?
536 2016-07-01 20:54:08	0|gmaxwell|Do you want software that never makes progress? Thats how you get software that never makes progress.
537 2016-07-01 20:54:45	0|gmaxwell|It can't use a later libdb or it will make all non-HD wallets also unreadable by other versions.
538 2016-07-01 20:55:46	0|helo|i believe a hd wallet should work with older versions
539 2016-07-01 20:56:18	0|gmaxwell|helo: hows that?
540 2016-07-01 20:56:44	0|sturles|Yes, I know.  I guess it is impossible to use different versions depending on wallet type, e.g. via dlopen?
541 2016-07-01 20:57:17	0|gmaxwell|sturles: in theory possible but that would be a lot of work without any benefit.
542 2016-07-01 20:57:31	0|helo|it will work to a limited extent... hd keys already added will be visible
543 2016-07-01 20:57:49	0|helo|untested, ofc
544 2016-07-01 20:57:59	0|gmaxwell|if so, thats a bug. it should get rejected due to having a too new version.
545 2016-07-01 20:58:02	0|Lightsword|can we just migrate to sqlite?
546 2016-07-01 20:58:09	0|gmaxwell|if it isn't then you could put yourself in a weird state.
547 2016-07-01 20:58:18	0|gmaxwell|I will have to start stabbing people.
548 2016-07-01 20:59:23	0|helo|nah, that's not necessary. the version check probably functions as intended :)
549 2016-07-01 20:59:40	0|gmaxwell|hah
550 2016-07-01 21:01:34	0|sturles|I'd say it is benificial to switch to a newer, supported version of libdb
551 2016-07-01 21:02:43	0|gmaxwell|the latest versions have incompatible licenses.
552 2016-07-01 21:04:40	0|sturles|Oh.
553 2016-07-01 21:05:06	0|gmaxwell|there is newer than 4.8 that is okay, but the latest stuff has a much stronger copyleft than we'd normally use.
554 2016-07-01 21:07:19	0|gmaxwell|really the 'database' isn't actually used for much--- the wallet is largely entirely in memory, and the database is only really used for persistance.
555 2016-07-01 21:14:07	0|luke-jr|eh, -walletupgrade won't work? O.o
556 2016-07-01 21:36:54	0|Lightsword|would working on migrating the wallet to sqlite be something worthwhile at this point?
557 2016-07-01 21:39:50	0|CubicEarth|Block validation is, theoretically, very amenable to parallelization, correct?  Is the main serial component the depth of the merkel tree?
558 2016-07-01 21:43:50	0|gmaxwell|signature validation is, and bitcoin core runs that in paralle... but normally almost all of the signatures in a block are already validated before it shows up.
559 2016-07-01 21:44:22	0|gmaxwell|parallel*
560 2016-07-01 21:44:27	0|gmaxwell|the general database handling ends up taking much of the time, which isn't really paralizable.
561 2016-07-01 22:05:09	0|CubicEarth|gmaxwell: so thinking about the initial chain sync and validation, it could (with lots and lots of careful work) be made to harness GPU's, and spread the signature validation across *many* cores?  I'm not familiar with the nature of the general database handling you are referring to, but for the initial sync is that mainly keeping track of the utxo's as it increments tough the blocks?
562 2016-07-01 22:07:54	0|gmaxwell|I've seen no reason to believe that a gpu would help at all, GPU cores aren't particularly good at the work of validating. (64 bit arithmetic helps a lot!)
563 2016-07-01 22:08:26	0|gmaxwell|and nodes don't even verify the signatures far back in the chain.
564 2016-07-01 22:09:45	0|gmaxwell|but sure more work could be done to extract more parallelism out of it.
565 2016-07-01 22:11:17	0|gmaxwell|but even with signature validation turned off completely, with default settings a reindex takes almost 9 hours currently.
566 2016-07-01 22:13:03	0|gmaxwell|increasing the db cache to a really huge size (such that the sync runs entirely out of ram) lets the whole sync _with_ signature checking run in about 3.5 hours.
567 2016-07-01 22:16:12	0|CubicEarth|Is the default 100MB?  I never knew that!!  I'll be curious too see what kind of speedup I can achieve on an i5 - dual core.  Are we talking 100GB of ram here, or would setting it to 4 or 8 GB make a substantial difference?
568 2016-07-01 22:17:11	0|sipa|setting it to 2 GB is more than sufficient
569 2016-07-01 22:17:33	0|sipa|higher numbers don't matter much
570 2016-07-01 22:19:02	0|CubicEarth|Is it a rolling-window thing?
571 2016-07-01 22:19:17	0|sipa|no
572 2016-07-01 22:19:24	0|sipa|it's a cache
573 2016-07-01 22:19:32	0|sipa|when the cache is full, we write it to disk
574 2016-07-01 22:19:39	0|sipa|and start over
575 2016-07-01 22:19:39	0|sipa|very silly
576 2016-07-01 22:22:11	0|gmaxwell|I think you actually need more than 2GB now to get full performance, but 8GB is more than enough.
577 2016-07-01 22:22:31	0|midnightmagic|maybe an iovec scatter write might help some systems write faster
578 2016-07-01 22:24:25	0|midnightmagic|or is it just a plain sequential write thing and the rest is leveldb
579 2016-07-01 22:24:41	0|CubicEarth|Well it's easy enough to set it higher.  I just never knew.  A default of 100MB can make sense for older systems, but if there was place for 'performance tips', that would be a good place to let people know.  Maybe everyone already (node operators) knows except for me.
580 2016-07-01 22:25:15	0|gmaxwell|I think a lot of people don't realize what a big difference it makes.
581 2016-07-01 22:25:46	0|gmaxwell|looks like we'll increase the default to 300MB.
582 2016-07-01 22:26:46	0|gmaxwell|wumpus: on the leveldb stuff, without txindex, I'd suggest making that 4mb instead of 2. 4MB was something like 1% faster for me when testing leveldb in isolation.
583 2016-07-01 22:27:30	0|gmaxwell|maybe 2 is a win at the 300mb limit, but with it bumped 4mb would be... and probably is more conservative overall.
584 2016-07-01 22:39:58	0|CubicEarth|I did a fresh sync a week ago, 2013-era i5 dual core laptop.  Ubuntu, 4GB of ram, spinning HD.  75Mbps cable internet.  It took about 20 hours to finish.  What was interesting though was how the last 20 weeks just crawled.  Seemed less than linear, even estimating about the effects of fuller blocks.
585 2016-07-01 22:42:36	0|sipa|yeah, try dbcache 4000 or so
586 2016-07-01 22:43:18	0|CubicEarth|after I order more ram :)
587 2016-07-01 22:47:43	0|CubicEarth|gmaxwell: earlier you wrote "and nodes don't even verify the signatures far back in the chain.".  That's due to checkpoints, right?  They don't seem like a good thing to rely on.  I understand their practicality, but yeah.
588 2016-07-01 22:49:17	0|CubicEarth|I'm just trying to understand the theoretical performance limits... I'm not crusading against checkpoints
589 2016-07-01 22:52:44	0|gmaxwell|well you'd be welcome here to do that, I want them gone and have for years.
590 2016-07-01 22:53:51	0|gmaxwell|But even with them gone, one can avoid skipping signature validation during the initial sync for blocks burried by thousands of additional blocks with negligible risk-- consider, if miners went rogue enough to reorg thousands of blocks the system is already screwed... and if its only during initial sync they'd only trick new nodes in any case.
591 2016-07-01 22:54:11	0|gmaxwell|if it makes a difference between a user running a node or not-- the best choice is clear.
592 2016-07-01 22:54:54	0|gmaxwell|Meanwhile bitcoinxt and classic have disabled signature checking for any block where the miner supplied timestamp is a day back or more... meaning they can be fooled without a reorg at all. And no one seems to care.
593 2016-07-01 22:56:57	0|CubicEarth|Arn't miners only afforded the option to have their timestamp be off by more than 2-hours?
594 2016-07-01 22:57:13	0|CubicEarth|(as an aside)
595 2016-07-01 22:57:28	0|gmaxwell|no, not in the past direction. In the future direction yes.
596 2016-07-01 22:58:05	0|gmaxwell|The past direction is limited by the median of the last 11 blocks-- but that median can be arbritarily far back.
597 2016-07-01 22:59:42	0|CubicEarth|So practically speaking, the way to detect the issue is when the client said it was 'synced' and you noticed that is was reporting yesterdays date?
598 2016-07-01 22:59:57	0|midnightmagic|..  did they actually commit that change? the >24hr non-t-verify change?
599 2016-07-01 23:01:32	0|gmaxwell|CubicEarth: no, because miners can simply add a correctly timed block immediately after.
600 2016-07-01 23:02:55	0|CubicEarth|That would appear as if the hashrate had fallen precipitously... not blocks for 24hr!
601 2016-07-01 23:03:22	0|gmaxwell|midnightmagic: XT did and released, it's merged in classic, but I don't know if it made it into a release yet.
602 2016-07-01 23:03:31	0|gmaxwell|These folks are extremely and dangerously incompetent.
603 2016-07-01 23:04:03	0|gmaxwell|CubicEarth: no, just a wonky timestamp, miners provide the timestamps and sometimes they're pretty wonky.
604 2016-07-01 23:04:30	0|gmaxwell|and while _syncing_ you have no idea that the timestamps were wonky.
605 2016-07-01 23:05:19	0|gmaxwell|e.g. someone can produce blocks that are -24h, -23h, -22h .... normal.
606 2016-07-01 23:05:23	0|midnightmagic|so.. the peer preference thing means some of the alt-branch software will soon start disconnecting the only nodes that are immune to the attack, making themselves *more* vulnerable to it?
607 2016-07-01 23:07:30	0|gmaxwell|midnightmagic: I don't think that matters that much, but right now bitcoin "xt" is only not totally partitioned due to inbound connections from core to XT, but part of segwit is that segwit capable nodes will strongly prefer to connect out to only non-segwit nodes (because they must in order to get blocks once segwit is activated)-- the result will be making that partition complete.
608 2016-07-01 23:08:43	0|midnightmagic|i thought -classic was now disconnecting non-classic nodes
609 2016-07-01 23:09:22	0|gmaxwell|no, Xt. I don't think classic is doing that yet.
610 2016-07-01 23:10:17	0|slackircbridge1|<alp> let them lose money for people
611 2016-07-01 23:10:20	0|slackircbridge1|<alp> its the only way theyll learn
612 2016-07-01 23:10:38	0|gmaxwell|It's not that simple. See also: Ethereum and DAO.
613 2016-07-01 23:11:23	0|gmaxwell|which adversely impacted the bitcoin price even though many of us have been pointing out the related risks and distinguishing bitcoin on the basis of them for some time.
614 2016-07-01 23:13:18	0|slackircbridge1|<brg444> :O
615 2016-07-01 23:13:23	0|CubicEarth|gmaxwell: well I'm pretty sure the rise of ETH was hurting the bitcoin price to some extent.  I think this event was a decoupling, where Bitcoin shakes the monkey off its back.
616 2016-07-01 23:13:31	0|slackircbridge1|<brg444> did he read your post :O
617 2016-07-01 23:24:28	0|slackircbridge1|<alp> Short term pain, long term gain
618 2016-07-01 23:27:57	0|slackircbridge1|<moli> guys, we can read your posts on IRC lol
619 2016-07-01 23:33:10	0|btcdrak|wait that's wrong. I'm trying to set slackircbridge1 +q
620 2016-07-01 23:34:00	0|gmaxwell|lol
621 2016-07-01 23:34:03	0|gmaxwell|undo the damage first.