1 2017-07-05 02:33:05	0|bitcoin-git|[13bitcoin] 15luke-jr opened pull request #10745: Be consistent in using "opt_into_rbf" parameter for Opt-In RBF (06master...06opt_in_rbf-param) 02https://github.com/bitcoin/bitcoin/pull/10745
  2 2017-07-05 11:23:39	0|bitcoin-git|[13bitcoin] 15jnewbery opened pull request #10747: [rpc] fix verbose argument for getblock in bitcoin-cli (06master...06fix_getblock_verbose_argument) 02https://github.com/bitcoin/bitcoin/pull/10747
  3 2017-07-05 13:27:38	0|bitcoin-git|[13bitcoin] 15jnewbery opened pull request #10748: [config] Help text cleanup (06master...06helptextcleanup) 02https://github.com/bitcoin/bitcoin/pull/10748
  4 2017-07-05 14:52:11	0|bitcoin-git|[13bitcoin] 15practicalswift opened pull request #10749: Use compile-time constants instead of unnamed enumerations (remove "enum hack") (06master...06enum-hack) 02https://github.com/bitcoin/bitcoin/pull/10749
  5 2017-07-05 15:32:00	0|bitcoin-git|[13bitcoin] 15instagibbs closed pull request #10333: [wallet] fee fixes: always create change, adjust value, and p… (06master...06fixfeefinal) 02https://github.com/bitcoin/bitcoin/pull/10333
  6 2017-07-05 17:38:09	0|earlz|Is there a know minimum gcc version for compiling Bitcoin Core?
  7 2017-07-05 17:51:21	0|sipa|4.8
  8 2017-07-05 18:43:06	0|bitcoin-git|[13bitcoin] 15instagibbs opened pull request #10750: [wallet] Change bumpfee totalFee option to feeRate option (06master...06bumpfeerate) 02https://github.com/bitcoin/bitcoin/pull/10750
  9 2017-07-05 19:11:30	0|morcos|instagibbs: Just wanted to jot down some of the thoughts on future improvements to bumpfee
 10 2017-07-05 19:11:55	0|morcos|So right now bumpfee doesn't let you bump a tx which has any of its outputs spent by other mempool txs (i.e. has any descendants)
 11 2017-07-05 19:12:02	0|morcos|I think there are several reasons for this
 12 2017-07-05 19:12:40	0|morcos|One you might unintentionally pay way more in fee than you were expecting b/c you're paying for a lot of descendants
 13 2017-07-05 19:12:52	0|instagibbs|yes, rhavar brought this up on the mailing list recently
 14 2017-07-05 19:13:08	0|instagibbs|someone does a mega-sweep on top, even low fee, and you can pay lots
 15 2017-07-05 19:13:16	0|morcos|Two is the tricky issue of you aren't sure which transaction is actually going to get confirmed, the original or the replacement
 16 2017-07-05 19:14:31	0|morcos|So if your replacement isn't doing something pretty smart, you may now end up in a confused state as to whether your descendant payees should be repaid and making sure they are repaid using conflicting in puts so you don't end up double paying them
 17 2017-07-05 19:15:01	0|morcos|We should probably write up a plan for the next step in improving bumpfee functionality
 18 2017-07-05 19:15:57	0|morcos|I think the ability to bump a chain of txs where all the descendant txs are also yours ought to be feasible as a next step
 19 2017-07-05 19:17:07	0|morcos|Bumping a tx which someone else has spent seems riskier..  Perhaps if there are no new inputs added in the original chain and you have enough original change, then you could pay all their payees for them?
 20 2017-07-05 19:17:15	0|instagibbs|Are you saying the local logic cares that downstream someone double-counted payments?
 21 2017-07-05 19:18:40	0|morcos|Not exactly.. But I'm saying we don't want to design our wallet software such that in an ecosystem of people using it, they end up actually double paying other people b/c they aren't properly conflicting inputs on double spends
 22 2017-07-05 19:19:07	0|morcos|I think this also touches on why it's tricky to add new inputs on just the simple case of bumping your own tx
 23 2017-07-05 19:19:15	0|sipa|morcos, instagibbs: i wonder if an alternative to deal with rhavar's problem is to keep a counter in every tx "bytes_replaced", which accumulates any time a replacement happens through acceptance of a tx (both for rbf or eviction reasons)... and then you're required to pay that number times the relay margibal feerate
 24 2017-07-05 19:19:26	0|instagibbs|I'm not grokking tbh, I might need specific examples of what's wrong here
 25 2017-07-05 19:19:52	0|sipa|the requirement of strictly larger fee on replacement is not actually necessary to make the economics work out
 26 2017-07-05 19:20:35	0|morcos|sipa: hmmmm....  i'm not sure that's completely correct
 27 2017-07-05 19:21:13	0|sipa|the requirement is that 1) mining the new tx is economically better for miners and 2) the new tx pays for the relay of all previous txn it caused to be evicted
 28 2017-07-05 19:21:21	0|morcos|it seems to me there is a relationship between the min relay rate and the min rate which is accepted in a block, which is dictated by the size of the mempool and the decay of the mempool min fee
 29 2017-07-05 19:21:46	0|morcos|it is a slightly broken relationship to be sure
 30 2017-07-05 19:22:38	0|morcos|but i dont' think we have enough confidence that the min relay fee alone is sufficient to prevent spam that we should sort of allow "downgrading" the fees paid as long as your are still over the cumulative bytes times min relay
 31 2017-07-05 19:22:42	0|instagibbs|morcos, can you give me an example of problematic behavior which doesnt just boil down to "don't do unconfirmed chaining on top of bip125 transaction"?
 32 2017-07-05 19:23:00	0|instagibbs|s/behavior/example
 33 2017-07-05 19:23:26	0|morcos|sipa: certainly it's also not clear that would be in the best interest of miners
 34 2017-07-05 19:24:56	0|morcos|instagibbs: let me think for a min
 35 2017-07-05 19:25:47	0|sipa|morcos: i what way wouldn't it?
 36 2017-07-05 19:28:47	0|morcos|sipa: hmm..  i suppose only if blocks aren't full  (hopeful smiley face)
 37 2017-07-05 19:29:20	0|morcos|but no, not even exactly that
 38 2017-07-05 19:29:42	0|morcos|if the feerates in question are all above average, then miners have lost right?
 39 2017-07-05 19:30:39	0|morcos|you could replace 10200 bytes of something paying 100 sat/B with 200 bytes of something paying 201 sat/B in your example right?
 40 2017-07-05 19:31:11	0|morcos|if the feerate at the bottom of a block ever drops below 100 sat/B then miners lost out didn't they?
 41 2017-07-05 19:33:00	0|morcos|instagibbs: ok back to your questions.  i think i said two things, the second was that your previous idea of adding inputs to bumpfee might have issues
 42 2017-07-05 19:33:10	0|morcos|i think i agree that it should not have issues
 43 2017-07-05 19:33:18	0|instagibbs|it'se self-invalidating
 44 2017-07-05 19:33:31	0|morcos|although i think we'll need to carefully examine the code for handling conflicts and make sure we're not making any edge cases worse
 45 2017-07-05 19:33:38	0|morcos|but i agree we should be able to make it work
 46 2017-07-05 19:33:45	0|instagibbs|i think for descendants in mempool, the two easiest cases to think about: 1) all yours 2) none of yours
 47 2017-07-05 19:34:26	0|morcos|yes, so i think we can handle 1.
 48 2017-07-05 19:34:44	0|sipa|morcos: right, ok, i'm assuming a more rational market than currently exists, i guess
 49 2017-07-05 19:34:58	0|morcos|i think we can handle any cases that aren't 1 (including mix of yours and not yours) as long as no inputs that aren't yours are added
 50 2017-07-05 19:35:52	0|morcos|volatile != irrational does it?
 51 2017-07-05 19:36:27	0|morcos|instagibbs: but i agree we should separate those into two separate cases...  and handling all yours first seems easier
 52 2017-07-05 19:37:02	0|sipa|morcos: i'm assuming near infinite demand at various fee rate levels
 53 2017-07-05 19:37:15	0|sipa|reality is much more complicated, i agree
 54 2017-07-05 19:37:22	0|rhavar|There's minor privacy implications of doing that automatically, you clearly identify which descendants are yours
 55 2017-07-05 19:37:27	0|morcos|instagibbs: see this https://github.com/bitcoin/bitcoin/pull/8456#issuecomment-264734557 for background
 56 2017-07-05 19:38:01	0|morcos|rhavar: you mean of only having the ability to do it when all are yours?
 57 2017-07-05 19:38:41	0|rhavar|yeah
 58 2017-07-05 19:39:43	0|rhavar|(although I'm just jumping in this conversation a bit late, as I got some pings on slack i was being mentioned). But you're talking about automatically resigning and resending invalidated descendants?
 59 2017-07-05 19:39:48	0|morcos|yeah, i mean we could do both steps, but you will still be able to differentiate b/c if any of the descendant txs added inputs, those will be identified as being your descendant txs
 60 2017-07-05 19:39:58	0|instagibbs|rhavar, no just simple bump case
 61 2017-07-05 19:40:26	0|rhavar|oh, never mind then
 62 2017-07-05 19:40:31	0|morcos|rhavar: well just replacing the whole chain with a single tx that pays all the necessary payees and conflicts all the txs which pull in new in-chain inputs
 63 2017-07-05 19:40:49	0|rhavar|That's not often a sane thing to do, as the fee rates will differ
 64 2017-07-05 19:41:41	0|morcos|instagibbs: i suppose if you look at it the way i just stated it, then it doesn't matter to break it in two cases... you just wont be able to do it if any of the non-you descendnats add inputs (assuming no mixed txs)
 65 2017-07-05 19:42:13	0|morcos|and now we have to start looking at stuff in a per wallet world
 66 2017-07-05 19:42:19	0|morcos|from you has to mean from this wallet
 67 2017-07-05 19:43:04	0|morcos|which raises an interesting point...  if you have wallets which overlap with other wallets, then you can run into problems conflicting only part of a tx
 68 2017-07-05 19:43:15	0|morcos|did people envision overlapping wallets?
 69 2017-07-05 19:43:27	0|BlueMatt|lol, morcos writes out "smiley face"
 70 2017-07-05 19:43:59	0|instagibbs|can you rephrase "just wont be able to do it if any of the non-you descendnats add inputs (assuming no mixed txs)" (so sorry, lots of terminology)
 71 2017-07-05 19:44:31	0|morcos|instagibbs: mixed tx = tx which spends some of your inputs and some of someone elses  (can't remember exactly what we call those)
 72 2017-07-05 19:44:47	0|instagibbs|ok that cannot be replaced because we don't know fees?
 73 2017-07-05 19:45:40	0|morcos|instagibbs: that's not what i meant, but maybe i'm looking at it backwards
 74 2017-07-05 19:45:54	0|morcos|yeah i was looking at it backwards
 75 2017-07-05 19:46:19	0|morcos|what i was trying to avoid is making it so you accidentally have two pays to the same payee get confirmed that spend different inputs
 76 2017-07-05 19:46:23	0|instagibbs|that's a check feebumper does, fwiw :P
 77 2017-07-05 19:46:41	0|morcos|we do that in our own wallet by making sure we conflict at least one of the inputs between the two txs
 78 2017-07-05 19:47:21	0|morcos|so now i'm back to your original division and thinking we should distinguish between a chain of only our txs and a chain which includes someone elses child spends
 79 2017-07-05 19:47:57	0|morcos|it's just going to be a bit unexpected if they see their child spend evicted and they won't know their payee has been paid (if thats what we do by bumping the package)
 80 2017-07-05 19:48:12	0|rhavar|Is it really even worth while to support chains? :P
 81 2017-07-05 19:48:20	0|morcos|so lets just forget about that (at least for now) and only bump chains of ourself
 82 2017-07-05 19:48:49	0|morcos|support bumping them or support them?  unfortunately supporting them is a long lost cause, but i do think that is still valuable
 83 2017-07-05 19:49:03	0|morcos|supporting bumpng them makes a lot of sense b/c you can save money by consolidating
 84 2017-07-05 19:49:34	0|rhavar|support bumping them. There's a lot of edge cases, like the descendant having a lower fee rate
 85 2017-07-05 19:49:42	0|rhavar|which you might not want to consolidate
 86 2017-07-05 19:50:30	0|BlueMatt|and if there are "your" descendants, we should probably only support bumping at the bottom of the chain
 87 2017-07-05 19:50:57	0|BlueMatt|should also probably support explicit cpfp, which handles some other cases, too
 88 2017-07-05 19:51:21	0|morcos|BlueMatt: the advantage of not bumping at the bottom is you can consolidate
 89 2017-07-05 19:51:31	0|morcos|you can always choose to bump at the bottom
 90 2017-07-05 19:51:37	0|BlueMatt|well ok, but that seems like a rather separate thing, no?
 91 2017-07-05 19:51:52	0|BlueMatt|auto-merging transactions sounds like very surprising behavior for "bumpfee"
 92 2017-07-05 19:52:06	0|morcos|but yeah having a smart algo that determines whether CPFP or bump or CPFP by bumping the bottom is the best choice would be nice
 93 2017-07-05 19:52:07	0|sipa|i wonder if we should just have a set of outputs that require being paid, and just have RPCs that change that set, where every change just results in a new RBF doing the whole thing
 94 2017-07-05 19:52:38	0|sipa|and stop seeing unconfirmed transactions as transactions
 95 2017-07-05 19:52:41	0|BlueMatt|that sounds like reasonable behavior...for users who only need limited privacy
 96 2017-07-05 19:52:55	0|BlueMatt|but it sounds like a separate set of RPCs?
 97 2017-07-05 19:53:11	0|morcos|i think that at the Bitcoin Core level it's always going to be a highly advanced application, and its better not to abstract away too much about how it actually works
 98 2017-07-05 19:53:29	0|morcos|Let higher level software build on top of it that type of functionality
 99 2017-07-05 19:53:32	0|sipa|BlueMatt: yeah, it'd need to be separate... and not compatible with any receiver that wants a txid ahead of time etc
100 2017-07-05 19:53:48	0|sipa|what do you mean with limited privacy?
101 2017-07-05 19:53:53	0|morcos|Of course if the wallet software was separate, we could just have both
102 2017-07-05 19:54:14	0|sipa|morcos: right, perhaps this is more something for a new wallet rather than an add-on to the existing one
103 2017-07-05 19:54:23	0|BlueMatt|sipa: well my biggest concern with auto-merging in bumpfee is that you have people who manually selected inputs or created raw txn and all of a sudden those txn change structure massively
104 2017-07-05 19:54:25	0|sipa|it just seems so much easier to reason about
105 2017-07-05 19:54:33	0|BlueMatt|potentially merging outputs that they wanted to keep "separate"
106 2017-07-05 19:54:53	0|sipa|well this wouldn't support self-selecting inputs...
107 2017-07-05 19:55:03	0|BlueMatt|it doesnt, but Core does
108 2017-07-05 19:55:12	0|rhavar|If merging was desired, wouldn't it be better to have done that in the first place when sending the 2nd transaction?
109 2017-07-05 19:55:16	0|BlueMatt|in the future maybe want to push people to multiwallet, but thats far away
110 2017-07-05 19:55:55	0|sipa|yeah, my suggestion isn't really on topic in this discussion
111 2017-07-05 19:56:02	0|morcos|BlueMatt: I think bumpfee could take a list of txs to merge, and then could by default merge descendants of those txs, but take an option to not include descendants.  In all cases, all txs must be from you.  That ought to be pretty intuitive and cover all possibilites.
112 2017-07-05 19:56:13	0|sipa|but all the reasoning about CPFP and bumping and chains of transactions makes my head hurt
113 2017-07-05 19:56:29	0|sipa|and i think there is a possible way of how things could have worked if not for legacy, that is much easier
114 2017-07-05 19:56:41	0|BlueMatt|morcos: maybe we should rename it "dothings" if it grows to do all kinds of magic :p
115 2017-07-05 19:56:50	0|morcos|if blocks were bigger and came every 10 secs instead of every 10 mins, we wouldn't really have these problems
116 2017-07-05 19:57:05	0|BlueMatt|lol
117 2017-07-05 19:57:12	0|rhavar|I'm actually a huge fan of bumpfee taking a *list* of transactions to fee bump   (and consolidate them)   .. but i think all the descendant stuff is way too insanely annoying
118 2017-07-05 19:57:14	0|BlueMatt|bitcoin also wouldnt function, but thats a separate issue
119 2017-07-05 19:57:17	0|instagibbs|getting back to my original thing: absolute fee argument is brittle if we want to do anything dynamic with our replacements
120 2017-07-05 19:57:37	0|morcos|instagibbs: dynamic?
121 2017-07-05 19:57:40	0|instagibbs|maybe it can just get the fee wanted, but perhaps grab too many inputs in order to pay
122 2017-07-05 19:57:44	0|instagibbs|adding inputs for example
123 2017-07-05 19:57:50	0|rhavar|because if you're a high volume sender, you probably don't have a single transaction to bump ... but have a list of them
124 2017-07-05 19:58:02	0|instagibbs|I was really hoping to avoid doing insane loops guessing how many outputs to select
125 2017-07-05 19:58:05	0|BlueMatt|hmm, well maybe I take that back, maybe a list of txids is ok and not too huge
126 2017-07-05 19:58:21	0|morcos|instagibbs: ah, i see
127 2017-07-05 19:58:25	0|rhavar|the main difficulty i guess is that you'll have now multiple change addresses
128 2017-07-05 19:58:32	0|rhavar|but that should be easy to prune
129 2017-07-05 19:58:55	0|morcos|instagibbs: i would recommend just adding a new option which is payTxFeeRate
130 2017-07-05 19:59:16	0|morcos|I think if you wait until my PR's that use coin control get merged, it'll be trivial to add that to bumpfee
131 2017-07-05 19:59:25	0|instagibbs|and disable anything nice when using totalFee? :P
132 2017-07-05 19:59:39	0|morcos|No
133 2017-07-05 19:59:59	0|morcos|Hmm
134 2017-07-05 20:00:05	0|instagibbs|or just be ok grabbing too many inputs
135 2017-07-05 20:00:11	0|instagibbs|and shove the excess fee into a change output?
136 2017-07-05 20:00:29	0|instagibbs|(thinking along lines of using effective value)
137 2017-07-05 20:00:35	0|morcos|In what cases do we grab extra inputs?
138 2017-07-05 20:00:49	0|morcos|Only when we have no change or it's not big enough?
139 2017-07-05 20:00:53	0|morcos|s/do/will/
140 2017-07-05 20:01:08	0|instagibbs|no change or not enough yeah
141 2017-07-05 20:01:24	0|instagibbs|otherwise no reason to
142 2017-07-05 20:01:34	0|BlueMatt|morcos: in any case, I kinda like the "multiple txn to bump and auto-merge" option now that I think of it...but still think we should just go with only supporting bottom chunks of chains by default, cause thats almost always what you want to do anyways (ie cpfp, effectively)
143 2017-07-05 20:02:00	0|rhavar|and i don't think it has any fragile and annoying logic
144 2017-07-05 20:02:27	0|morcos|BlueMatt: For starters you can do that by bumping the bottom tx yourself...   I think if you have a chain, you'll probably save more by consolidating
145 2017-07-05 20:03:12	0|morcos|instagibbs: Yeah I think that gets tricky.  It's actually going to get a bit tricky even without thinking about totalFee, I think.
146 2017-07-05 20:03:41	0|BlueMatt|morcos: yes, agreed, but you can only support consoladating at the bottom of the chain
147 2017-07-05 20:04:03	0|BlueMatt|if you want to consolodate mid-chain, things might break, and thats less of our issue
148 2017-07-05 20:04:11	0|morcos|Why don't you just get it to work right only in the event that it is using automatic fee calculation or a txconfirmtarget was passed in
149 2017-07-05 20:04:31	0|morcos|Supporting it in the totalFee case (if possible) can be later.
150 2017-07-05 20:04:46	0|morcos|Adding a payTxFeeRate is orthogonal and as simple as above.
151 2017-07-05 20:05:47	0|instagibbs|morcos, if we don't have that constraint, should be a fairly simple effective value selection, no?
152 2017-07-05 20:05:47	0|instagibbs|well, we have to decide how much we want to grab, just has to be enough to pay for fee delta... if you get exact match no change, otherwise make change.
153 2017-07-05 20:06:14	0|instagibbs|Fair enough, was just hoping it was a useless arg to have less code
154 2017-07-05 20:06:19	0|morcos|instagibbs: don't have what constraint?  that it has to work with totalFee?
155 2017-07-05 20:07:27	0|instagibbs|Yeah. Previously it's dead-easy because you just adjust the change.
156 2017-07-05 20:12:03	0|rhavar|or if you don't mind rabbit-holes, the "create transaction" stuff could be extended with an array of set of transaction inputs in which at least 1 must be picked
157 2017-07-05 20:12:13	0|rhavar|and then all that logic can be reused
158 2017-07-05 20:13:42	0|rhavar|the part that gets messy is that you need to make sure your new transaction not only conflicts with the transaction you're fee bumping -- but all previous fee bumps too
159 2017-07-05 20:14:21	0|rhavar|so you can avoid that by only adding new inputs  (at the cost of worse coin selection)
160 2017-07-05 20:15:00	0|instagibbs|oh please, not that rabbit hole
161 2017-07-05 20:15:29	0|sipa|instagibbs: BnB isn't hard to give a constraint "only accept combinations which include at least one input of each of these previous transactions
162 2017-07-05 20:15:48	0|instagibbs|right nothing in principle is wrong, just the versioning thing
163 2017-07-05 20:16:05	0|instagibbs|it's obviously correct, just hard
164 2017-07-05 20:16:22	0|morcos|instagibbs: yeah to that point, i'd argue we should concentrate on the BnB and effective value logic first
165 2017-07-05 20:16:37	0|instagibbs|our functionary stack uses a version of that logic to not doublespend
166 2017-07-05 20:16:46	0|morcos|adding inputs to bumpfee sounds nice but very non-critical and might as well only do it once on top of the new coin selection logic
167 2017-07-05 20:17:00	0|instagibbs|morcos, agreed, I was building a speculative PR on top of achow's
168 2017-07-05 20:18:55	0|instagibbs|I'm supporting BnB as trojan horse to get effective value in (kidding, sorta)
169 2017-07-05 20:20:39	0|bitcoin-git|[13bitcoin] 15instagibbs closed pull request #10750: [wallet] Change bumpfee totalFee option to feeRate option (06master...06bumpfeerate) 02https://github.com/bitcoin/bitcoin/pull/10750
170 2017-07-05 20:23:43	0|gmaxwell|instagibbs: you keep complaining about EV but it doesn't seem like anyone is working to fix the bloat concern!
171 2017-07-05 20:24:05	0|instagibbs|gmaxwell, hopefully it doesn't cause bloat for bumpfee!
172 2017-07-05 20:24:09	0|instagibbs|:)
173 2017-07-05 20:25:39	0|instagibbs|not trying to be flippant, you can just take negative EV outputs anyways
174 2017-07-05 20:26:33	0|gmaxwell|it's not clear to me that being willing to take negative ev outputs is sufficient to be as de-bloating as the current stuff, maybe it is.
175 2017-07-05 20:27:11	0|gmaxwell|rhavar: the easiest thing to do is always conflict all the earlier inputs, then you don't have to worry about failing to conflict an earlier bump.  It's also better for privacy.
176 2017-07-05 20:28:00	0|gmaxwell|BlueMatt: we can't really do the multiple txn bump and auto-merge without segwit, I think.  Handling malleability is too gnarly.
177 2017-07-05 20:29:30	0|rhavar|it's not clearly better for privacy, if the new coin selection result avoided creating change (and thus the means to cluster your wallet further)
178 2017-07-05 20:30:39	0|rhavar|hmm never mind, i'm stupid
179 2017-07-05 20:33:32	0|instagibbs|gmaxwell, would simulations suffice? What kind of data are people looking for?
180 2017-07-05 20:51:34	0|gmaxwell|instagibbs: simulations would sufficice in my view.  Basically just a clear argument that the new procedure won't tend to select fewer inputs than the old one.
181 2017-07-05 21:02:46	0|morcos|gmaxwell: I'm all for putting together a simulation. But at the end of the day, I think we're going to have to wing it a bit.  It's almost by definition going to put together fewer inputs.
182 2017-07-05 21:02:57	0|morcos|We're doing stupid things now by spending uneconomical inputs
183 2017-07-05 21:03:17	0|morcos|To fix that and prevent utxo bloat getting worse takes a more involved solution I think
184 2017-07-05 21:03:32	0|morcos|Being smarter about what size outputs we create is one starting point
185 2017-07-05 21:03:49	0|morcos|But the key is also being willing to purposefully pick multiple small outputs to consolidate sometimes
186 2017-07-05 21:04:03	0|morcos|The tricky thing is when to do that
187 2017-07-05 21:04:43	0|morcos|But I think if we start on this right after 0.15, we ought to be able to get it all done.  BnB, effective value, better output creation and smart consolodiation
188 2017-07-05 21:05:19	0|morcos|Whether we'll be 100% convinced its at least as good as before, I don't think thats going to be easy...  But as long as it's not a COMPLETE disaster, we can refine it over a couple of releases.
189 2017-07-05 21:05:36	0|morcos|Simulations are just so hard when we don't have a good sense of what the user population looks like.
190 2017-07-05 21:06:02	0|instagibbs|morcos, you can at least compare apples to apples, which might be useful
191 2017-07-05 21:06:20	0|morcos|Depends on what an apple is I guess
192 2017-07-05 21:06:39	0|morcos|but like i said, i'm all for tryingsimulations out
193 2017-07-05 21:17:39	0|gmaxwell|morcos: I don't think that is acceptable-- you don't even know what the magitude of the change is. And yes, but definition it will use fewer inputs: thats the problem.  It's not like we haven't made this kind of mistake before, and it had a tremendously negative and lasting effect.
194 2017-07-05 21:18:01	0|morcos|gmaxwell: what do you propose as an alternative?
195 2017-07-05 21:18:53	0|morcos|my argument is that TX A which consolidates the UTXO more but contains an input with negative effective value is not clearly superior to TX B which is identical but does not contain the negative effective value input
196 2017-07-05 21:19:08	0|gmaxwell|There isn't an alernative. We need to simulate and make compensating changes and adjust things until we have good reason to believe there won't be a seriously bad effect.
197 2017-07-05 21:19:41	0|morcos|Simulate how?
198 2017-07-05 21:19:55	0|gmaxwell|I disagree, especially considering that we will make UTXO which immediately have negative effective value, that sequence alone basically guarentees runaway utxo growth (I just don't know the runaway constant)
199 2017-07-05 21:20:25	0|morcos|gmaxwell: well the "especially .." is the part i think we need to address first
200 2017-07-05 21:21:42	0|morcos|the problems i have with simulation is they tend to simulate over a single large wallet making and receiving a series of txs
201 2017-07-05 21:22:18	0|morcos|I don't know if there is any reason to believe that this has the same net utxo affect as an ecosystem of wallets (many probably much smaller) all making/receiving txs to each other
202 2017-07-05 21:22:46	0|gmaxwell|If it were addressed first I would worry somewhat less.  Similarly, if we had some mechenism to sweep up utxo even when they have low or slightly negative EV then there would be less cause for concern.  We know that pruning unnecessary inputs caused phenominal UTXO set inflation, avoiding low EV inputs seems to me like it would do the same.
203 2017-07-05 21:22:51	0|morcos|without understanding some structure of how tx flows look, we're at risk of producing useless results
204 2017-07-05 21:23:06	0|gmaxwell|morcos: well murch's simulation had traces of real payment in, payment out amounts.
205 2017-07-05 21:23:08	0|bitcoin-git|[13bitcoin] 15practicalswift opened pull request #10752: Use quoted form when including primitives/transaction.h (06master...06transaction-h) 02https://github.com/bitcoin/bitcoin/pull/10752
206 2017-07-05 21:23:35	0|gmaxwell|morcos: I think it is still a big step forward to say that in at least one realistic situation the changes don't produce UTXO runaway.
207 2017-07-05 21:23:35	0|morcos|did you read my whole spiel..  i thought we should do those other things as well
208 2017-07-05 21:24:03	0|morcos|but it's actually easier to build those other things properly on top of an effective value framework
209 2017-07-05 21:24:43	0|rhavar|Has anyone suggested raising the "is dust" check to something more sane?
210 2017-07-05 21:24:49	0|gmaxwell|I'm not saying they should be done, I'm saying that they must be done. And that we must have at least _some_ measurement that suggests that it won't go nuts. It doesn't have to be perfect.
211 2017-07-05 21:24:51	0|instagibbs|rhavar, yep
212 2017-07-05 21:24:56	0|rhavar|the current dust threshold is ludicrously low
213 2017-07-05 21:25:11	0|instagibbs|rhavar, the BnB branch does just that, to minimize review surface, but generally raising it is a goal too
214 2017-07-05 21:25:21	0|sipa|rhavar: using effectively output value effectively does that
215 2017-07-05 21:25:26	0|gmaxwell|I think rhavar is talking about something different instagibbs, sipa.
216 2017-07-05 21:25:30	0|morcos|rhavar: yeah, changing dust levels is a bit annoying, but changing the levels at which we'll create dust is a no-brainer!
217 2017-07-05 21:25:32	0|rhavar|i'm talking about is standard
218 2017-07-05 21:25:35	0|gmaxwell|rhavar is talking about what the network allows people to do.
219 2017-07-05 21:25:38	0|sipa|oh
220 2017-07-05 21:25:40	0|instagibbs|oh, then no
221 2017-07-05 21:25:41	0|rhavar|not the coin selection itself
222 2017-07-05 21:25:56	0|gmaxwell|rhavar: I think we should get rid of it, unfortunately it's ineffective since some large miners bypass it.
223 2017-07-05 21:26:23	0|sipa|no sane coin selection strategy should produce anything near the current dust threshold
224 2017-07-05 21:26:30	0|rhavar|some miners do bypass it, but it's still quite effective as a network policy
225 2017-07-05 21:26:33	0|morcos|gmaxwell: i agree to the extent that we shouldn't raise it, but getting rid of it seems risky.  it's at least somewhat of an impediment to other implementations creating stupidly small utxos
226 2017-07-05 21:26:46	0|sipa|but there are quite a few not-so-sane coin selection algorithms out there...
227 2017-07-05 21:26:49	0|rhavar|it stops a lot of fee attacks
228 2017-07-05 21:26:54	0|gmaxwell|morcos: perhaps.
229 2017-07-05 21:27:12	0|gmaxwell|rhavar: I don't think it does, you can reliably get txn mined that violate it.
230 2017-07-05 21:27:12	0|rhavar|I saw a service that recently (3 month ago?) got spammed with 3000 sat (?) outputs  and ended up needing to spend over a bitcoin to clean it up
231 2017-07-05 21:27:30	0|rhavar|if they could've spammed with 1 sat outputs, it would've been a lot more effective
232 2017-07-05 21:27:44	0|rhavar|and at a certain point people wouldn't even bother trying to clean it up
233 2017-07-05 21:27:47	0|rhavar|which leads to permanent bloat
234 2017-07-05 21:28:18	0|gmaxwell|Really the weighting in segwit is intended to be a better way of dealing with this stuff-- make the fees on creating those outputs that never get spent higher.
235 2017-07-05 21:28:18	0|rhavar|Unless you imagine a future where spending dust has  <=  0 weight :P
236 2017-07-05 21:28:41	0|gmaxwell|Constant amounts of bloat don't concern me, bloat blowup does. :)
237 2017-07-05 21:29:45	0|gmaxwell|I think on the order of 30% of hashpower doesn't enforce it, so I think really the only effect it has is that ignorant/lazy wallet authors get forced by relay policy to avoid creating dust, where otherwise they might not care.
238 2017-07-05 21:30:19	0|rhavar|it's also reasonably effective at stopping spammers (who are trying to attack a specific wallet, not the network itself)
239 2017-07-05 21:30:27	0|rhavar|they send to 1 peer who rejects it, so they use a higher amount
240 2017-07-05 21:30:35	0|gmaxwell|rhavar: why? are they just too stupid to give their txn directly to antpool?
241 2017-07-05 21:30:42	0|rhavar|probably
242 2017-07-05 21:30:45	0|gmaxwell|(or whomever else is bypassing at the moment)
243 2017-07-05 21:31:02	0|gmaxwell|fair but kind of a fragile justification.
244 2017-07-05 21:31:24	0|rhavar|not sure how they even construct the spam, tbh. it's possible they just use a wallet (like core?) that rejects it immediately
245 2017-07-05 21:31:36	0|rhavar|they probably aren't aware of that some miners don't enforce
246 2017-07-05 21:32:52	0|rhavar|it's a pretty big attack vector though, i'm not sure a sane way to deal with it
247 2017-07-05 21:33:15	0|rhavar|having wallets not spend uneconomical outputs might reduce the amounts of attacks though (as they know they can't harm someone by doing it)
248 2017-07-05 21:33:50	0|rhavar|if i know you're using a wallet that spends the uneconomical dust, i'm a lot more likely to create it in the first place
249 2017-07-05 21:33:53	0|gmaxwell|rhavar: segwit substantially deals with it, because it moves fees from spending outputs to creating them.  (not as much as I'd like, but there are constraints on how far you can go with that)
250 2017-07-05 21:34:10	0|rhavar|yeah i'm aware =)
251 2017-07-05 21:34:18	0|instagibbs|he wants infinite discount :P
252 2017-07-05 21:34:24	0|gmaxwell|And having wallets be sensible about not spending insanely uneconomical dust would be good.
253 2017-07-05 21:34:53	0|rhavar|instagibbs: nah, i want a constant negative weight for each byte you remove from the utxo    :P
254 2017-07-05 21:35:03	0|gmaxwell|Yea, infinite discount has problems, alas... byte bloat goes up hyperbolically with the discount.   Signature costs are much less of a concern than utxo but only finitely so. :)
255 2017-07-05 21:35:16	0|gmaxwell|negative weight is even more problematic than infinite discount.
256 2017-07-05 21:35:18	0|gmaxwell|:(
257 2017-07-05 21:35:25	0|sipa|just surcharge for outputs
258 2017-07-05 21:35:36	0|rhavar|unless you join the dark size and embrace almost unbounded size blocks 8)
259 2017-07-05 21:35:52	0|rhavar|(they'd still be bounded to the size of the utxo or something lolz)
260 2017-07-05 21:35:56	0|gmaxwell|because then you can pad up outputs then produce a yottabyte block. which then in practice you end up with multiple constraints and intractable fee estimation.
261 2017-07-05 21:38:28	0|rhavar|I wonder how harmful it really would be if someone mined a 1GB block that also removed 1GB from the utxo :P
262 2017-07-05 21:39:07	0|sipa|short-term, terrible
263 2017-07-05 21:39:11	0|sipa|long-term, great
264 2017-07-05 21:39:28	0|rhavar|as it'd kind of be a "one off" style block, as it's obviously not sustainable
265 2017-07-05 21:40:11	0|sipa|if we as a community feel that such a UTXO reduction is necessary, the proper way would be aim for a softfork that does that, not with a massive block
266 2017-07-05 21:40:34	0|morcos|the biggest problem with too big a discount is that fees are still sometimes effectively nil, so you can preload the utxo now for "freeish"
267 2017-07-05 21:40:39	0|rhavar|I just meant that if you came up with a weighting scheme that made it possible
268 2017-07-05 21:40:46	0|sipa|rhavar: ah, i see
269 2017-07-05 21:40:52	0|gmaxwell|rhavar: the problem is that the block would get orphaned because it would take forever to propagate. It would blow up all fast propagation methods (or to avoid blowing them up we'd have to uncap relay of txn, which would make the network DOS vulnerable).
270 2017-07-05 21:41:22	0|rhavar|fair point
271 2017-07-05 21:41:41	0|morcos|but i'm with gmaxwell, segwit didn't go quite far enough, and if we ever do have a HF, we should definitely go a bit further.
272 2017-07-05 21:41:44	0|gmaxwell|rhavar: so in practice miners would impose a second limit, which would often be the operable limit, and now efficient fee computation becomes intractable, because you don't know what limit you're competing for when you make the transaction. ... plus all the above just degrades network stability.
273 2017-07-05 21:42:15	0|gmaxwell|yea, with a HF we could go a bit further than segwit. E.g. counting the txin:vout data like witness data.
274 2017-07-05 21:42:51	0|gmaxwell|I still don't think a factor much beyond 4 is well advised, but there are some other accounting changes that would be reasonable along these lines.
275 2017-07-05 21:43:02	0|morcos|BlueMatt has a pAtent on that though, even more problematic than segwit pAtents
276 2017-07-05 21:43:15	0|gmaxwell|lol
277 2017-07-05 21:43:34	0|rhavar|I can't imagine it actually causing a problem with fee estimation. If one of the limits was just an extreme thing that was sustainably being able to hit (due to it requiring continual utxo decrease)  fee estimation could just ignore it
278 2017-07-05 21:43:48	0|rhavar|although multiple limits is nasty for transaction selection :(
279 2017-07-05 21:44:16	0|rhavar|that wasn't* able to be continually hit
280 2017-07-05 21:45:01	0|gmaxwell|At least when I've played through these things, it seems to work out such that both limits should always be near getting hit the reason is that if the one limit isn't being hit, miners should pat their blocks creating UTXO to charge up to allow larger blocks later.
281 2017-07-05 21:45:39	0|gmaxwell|In general I think anything where it can go negative immediately leads to non-trivial stratigies for income maximization.
282 2017-07-05 21:46:40	0|morcos|gmaxwell: one easy change we could still make for 0.15 is if we think the amount you should be willing to discard entirely (just move to fees) should be higher than the DustThreshold
283 2017-07-05 21:47:01	0|rhavar|Imagine you had a weight of: -2 for each byte removed from the utxo. 1 weight for each byte in the transaction. And 100 weight for each byte added to the utxo. Say you have two limits: block weight limit of 1e6 and  block size limit of 100MB
284 2017-07-05 21:47:08	0|morcos|I like these changes that can be made without redoing coin selection.
285 2017-07-05 21:47:10	0|rhavar|it'd be impossible for the block size limit to be hit frequently
286 2017-07-05 21:47:59	0|morcos|It would be easy to add a new calculation.  GetDiscardRate  =  max(dustrate, min(discardrate, estimatesmartfee(1000)))
287 2017-07-05 21:48:51	0|morcos|then if we configured discardrate default to something > 1 sat/Byte .
288 2017-07-05 21:49:22	0|morcos|i'm the king of adding new rate variables though
289 2017-07-05 21:51:02	0|morcos|say it was 5 sat/byte
290 2017-07-05 21:51:35	0|morcos|at $3000 bitcoin, i think that means if you create an output thats less than 8 cents (or a bit smaller for segwit) that you'll just throw it away, and instead of redoing coin selection, you'll just add it to fee
291 2017-07-05 21:52:32	0|morcos|you guys say sane wallets shouldn't create anything close to the dust threshold, but i think that number, which is just 5x dust would be hard to convince people is good idea
292 2017-07-05 21:53:18	0|rhavar|what's the alternative though? redoing coin selection?
293 2017-07-05 21:53:23	0|morcos|taking the min with estimatesmartfee(1000) though helps avoid it becoming bad if BTC denominated fees
294 2017-07-05 21:54:00	0|morcos|drop
295 2017-07-05 21:54:01	0|morcos|rhavar: the problem with coin selection, is you don't know if you actually could get a better result
296 2017-07-05 21:54:56	0|rhavar|with "redoing coin selection" i mean it would contain the same logic about omitting change
297 2017-07-05 21:55:59	0|instagibbs|the coin selection could pick a similar sized set and get to keep the change, instead of dump it
298 2017-07-05 21:56:41	0|morcos|i suppose when we redo coin selection you'll first aim for at least min_desired_change (which is now a CENT and we aim for it exactly, but screw that, just at least)
299 2017-07-05 21:56:43	0|rhavar|yeah, i think it's strictly better or equal (from a fee perspective, not privacy)
300 2017-07-05 21:57:27	0|morcos|if you can't get to min_desired_change, then you'll throw everything you have in there and just take what you get, and if its less than discard (right now just dust) just throw away change
301 2017-07-05 21:57:33	0|gmaxwell|morcos: I'm fine with discarding more.  One way we could answer your concenr is making it configurable. Then people who want it different could adjust it.
302 2017-07-05 21:58:31	0|morcos|i'll write it up.. it's small
303 2017-07-05 21:59:12	0|gmaxwell|morcos: you could use the BNB hurestic: you can throw away up to the cost of creating and spending the change output.
304 2017-07-05 22:00:05	0|morcos|gmaxwell: but then you're throwing it away twice
305 2017-07-05 22:00:19	0|rhavar|"spending the change output"  how does it know the cost of spending the output? At what fee rate does it use?
306 2017-07-05 22:00:46	0|gmaxwell|rhavar: achow101's implementation uses a 1008 block feerate estimate.
307 2017-07-05 22:00:56	0|gmaxwell|rhavar: as the feerate.
308 2017-07-05 22:00:58	0|rhavar|ah nic
309 2017-07-05 22:00:59	0|rhavar|e
310 2017-07-05 22:01:10	0|rhavar|same as my code then :D
311 2017-07-05 22:01:18	0|gmaxwell|morcos: I don't get the 'throwing it away twice'?
312 2017-07-05 22:01:35	0|BlueMatt|lol, morcos' troll game is strong today
313 2017-07-05 22:01:42	0|instagibbs|We should never be keeping change that doesn't hit that threshold, the real Q is larger but still not great.
314 2017-07-05 22:01:47	0|Murch|gmaxwell: One of the Trezor guys did some more experiments this weekend and found that he got better results by allowing a smaller window of just the change output size
315 2017-07-05 22:02:01	0|morcos|well you've already done the coin selection and fee calculation to pay for the change, and now you're also throwing away the change output itself,so you are potentially overpaying the newly needed fee by double the cost of creating the change
316 2017-07-05 22:02:04	0|BlueMatt|gmaxwell: what am I being dense about? how does merging txn during a bump cause issues w/ malleability? shouldnt it go the other way if anything?
317 2017-07-05 22:02:20	0|BlueMatt|or are you suggesting we should explicitly not support chains in your wallet without segwit (which is true)
318 2017-07-05 22:02:32	0|gmaxwell|morcos: ah okay, don't do that.
319 2017-07-05 22:03:42	0|gmaxwell|BlueMatt: because you don't know what will get confirmed, will the orignal or the bump get confirmed?  So what you should do is make two transactions: a child and a bump. sign both, announce the child if the bump loses.  (and apply this recursively for more spends)
320 2017-07-05 22:04:09	0|gmaxwell|BlueMatt: as soon as malleability enters the picture you can't do this anymore; and you're stuck hoping the user stays online and reenters their keys so you can resign when a malleation happens.
321 2017-07-05 22:04:20	0|BlueMatt|what do you care if the child gets confirmed?
322 2017-07-05 22:04:29	0|BlueMatt|great! you've saved the attempted-bump in fees?
323 2017-07-05 22:04:50	0|BlueMatt|now the user is a bit confused, and may need to bump the child again, but thats ok
324 2017-07-05 22:07:23	0|morcos|i'm confused now too.  i agree we should not attach a child to a bumper or bumpee.  but what is wrong bumping something that already has a child if the bump is the merge of the parent and child?
325 2017-07-05 22:07:52	0|morcos|if the bump loses, i don't think you've done too much harm to the child have you?
326 2017-07-05 22:09:48	0|gmaxwell|When you create a _new_ child or a _new_ bump that adds outputs, you must also make children of all the existing ones.
327 2017-07-05 22:10:23	0|gmaxwell|Perhaps I'm talking about something a bit more broad than bluematt.
328 2017-07-05 22:12:42	0|BlueMatt|I believe you are talking about a very full-featured bumpfee, much more than I am thinking?
329 2017-07-05 22:12:58	0|BlueMatt|Though I missed some of the above discussion, just waiting for them to reopen the airport after they closed it due to weather :(
330 2017-07-05 22:18:35	0|rhavar|I think it's pretty sane to support  the case of fee bumping a list of transactions, where all of them have no children.
331 2017-07-05 22:18:46	0|BlueMatt|would those then get merged, or no?
332 2017-07-05 22:18:48	0|rhavar|yeah
333 2017-07-05 22:18:58	0|BlueMatt|yea, I agree, but I dont think thats an issue pre-segwit
334 2017-07-05 22:19:11	0|rhavar|yeah, i don't see how malleability matters at all
335 2017-07-05 22:19:33	0|rhavar|(at least if nothing has descendants)
336 2017-07-05 22:20:15	0|gmaxwell|it doesn't matter if nothing has decendants. But BlueMatt was talking about decendants, unless I've misunderstood.
337 2017-07-05 22:20:31	0|BlueMatt|i was, but I'm still confused
338 2017-07-05 22:21:08	0|rhavar|if there's descendants, there's some super annoying cases. But i don't think anyone is ever going to do that, so i don't think it's worth worrying about
339 2017-07-05 22:21:36	0|rhavar|i think it's only worth considering bumping shit without descendants, and i don't think bumping a list of them at once adds much more complexity :D
340 2017-07-05 22:21:59	0|gmaxwell|BlueMatt: If you allow decendants comes up.   You pay A  then you bump A.  Then you go to pay B,  if you're going to chainbumb you now need to compute and sign three transactions   A->B, A'->B and AB.
341 2017-07-05 22:22:52	0|gmaxwell|BlueMatt: and you can keep applying this for any number of chains and bumps and its all great. until you consider malleability.  E.g. maybe A gets mined but in malleated form, and now your payment to B is invalidated until you restart the wallet and enter your phassphrase.
342 2017-07-05 22:23:31	0|gmaxwell|rhavar: I think without malleability there is no big deal on the bumps, just takes some code.  I also think greenaddress does bumps with decendants.
343 2017-07-05 22:23:33	0|BlueMatt|well that has nothing to do with bumping
344 2017-07-05 22:23:44	0|BlueMatt|thats just the general-case spending-unconfirmed
345 2017-07-05 22:24:06	0|BlueMatt|and I dont see why you need to sign A'->B, you're just merging?
346 2017-07-05 22:24:08	0|BlueMatt|what is A'?
347 2017-07-05 22:29:28	0|morcos|gmaxwell: take a look at the comment I linked above: https://github.com/bitcoin/bitcoin/pull/8456#issuecomment-264734557
348 2017-07-05 22:29:31	0|morcos|it's your comment
349 2017-07-05 22:30:05	0|morcos|i think what we are talking about is a tx with descendants (or multiple txs each with descendants) and then bumpfee bumping all of them at once
350 2017-07-05 22:30:30	0|morcos|the thing we already don't allow, which i agree we should not change, is adding a child to a tx that is a bumper or has been bumped
351 2017-07-05 22:31:02	0|morcos|i don't see any issue with bumping a tx with children (presumably if the children are all yours) as long as you also pay all their outputs as well
352 2017-07-05 22:32:58	0|gmaxwell|morcos: no, it's a problem with making a child of a transaction that has a bump.
353 2017-07-05 22:33:17	0|morcos|ok agreed
354 2017-07-05 22:33:25	0|gmaxwell|you can bump something that has a child without any big complexity, but you can't produce a child of anything that has been bumped.
355 2017-07-05 22:35:34	0|rhavar|sounds good to me ;D