1 2017-01-21 00:07:53	0|bitcoin-git|[13bitcoin] 15fanquake closed pull request #9452: Use TravisCI to enforce copyright header rules for source files (06master...06PR-travisci-copyright-enforce) 02https://github.com/bitcoin/bitcoin/pull/9452
  2 2017-01-21 00:24:18	0|bitcoin-git|[13bitcoin] 15fanquake closed pull request #8587: Provide bloom services to whitelisted nodes. (06master...06WhitelistedBloom) 02https://github.com/bitcoin/bitcoin/pull/8587
  3 2017-01-21 00:27:40	0|bitcoin-git|[13bitcoin] 15fanquake closed pull request #9451: CScript: remove redundant bounds check (06master...06remove_extra_bounds_check) 02https://github.com/bitcoin/bitcoin/pull/9451
  4 2017-01-21 01:06:49	0|bitcoin-git|[13bitcoin] 15fanquake closed pull request #9315: Request announcement by cmpctblock AFTER requesting cmpctblock/blocktxn (06master...06RequestAnnounceAfter) 02https://github.com/bitcoin/bitcoin/pull/9315
  5 2017-01-21 05:51:24	0|bitcoin-git|[13bitcoin] 15jtimon opened pull request #9608: WIP: split processmessages (06master...062017-01-split-processmessages) 02https://github.com/bitcoin/bitcoin/pull/9608
  6 2017-01-21 06:54:39	0|bitcoin-git|[13bitcoin] 15theuni opened pull request #9609: net: fix remaining net assertions (fixes #9212) (06master...06net-version) 02https://github.com/bitcoin/bitcoin/pull/9609
  7 2017-01-21 18:29:27	0|gmaxwell|should UpdatePreferredDownload be checking ffeeler?
  8 2017-01-21 18:36:39	0|BlueMatt|probably?
  9 2017-01-21 18:37:44	0|BlueMatt|I'm hugely concerned about current fee estimation algorithms - they have effectively put floors on what they will return, when things an order of magnitude or less fee will easily get confirmed :(
 10 2017-01-21 18:38:19	0|gmaxwell|huh?
 11 2017-01-21 18:38:44	0|BlueMatt|eg currently, because no one creates txn with things less than 50 sat/byte, no fee estimation will ever return anything less than that
 12 2017-01-21 18:39:10	0|BlueMatt|but things with 10 often easily get confirmed after 6 or 10 blocks (now, though that can vary to a full day sometimes)
 13 2017-01-21 18:43:15	0|BlueMatt|eg i just tested a tx with 0.1 sat/byte fee, and it had issues propagating, but was even still confirmed in 5 blocks
 14 2017-01-21 18:43:40	0|BlueMatt|but every fee estimation anywhere returns >50 sat/byte
 15 2017-01-21 18:45:02	0|sipa|0.1 sat/byte? you mean 25 sat for a full transaction?
 16 2017-01-21 18:45:23	0|sipa|are you sure you're not off by an order of magnitude or 2?
 17 2017-01-21 18:45:33	0|BlueMatt|https://btc.com/1b4c848a30a73c805b36ec19142b9407fb74ffe9e41000ee87fadea8c6f582b5
 18 2017-01-21 18:45:38	0|BlueMatt|yes, 25 sat for the whole thing
 19 2017-01-21 18:46:17	0|gmaxwell|1s/b is absurdly low, I don't think having a floor around that point is a problem.
 20 2017-01-21 18:46:36	0|gmaxwell|we do need to have a meaningful relay floor to prevent DOS attacks.
 21 2017-01-21 18:46:38	0|BlueMatt|sure, 1s/b whatever, but the floor is currently like 50
 22 2017-01-21 18:46:53	0|BlueMatt|i'm not complaining about relay issues, i dont care it wasnt relayed
 23 2017-01-21 18:47:08	0|BlueMatt|the issue is that all fee estimation stuff is currently off by an order of magnitude
 24 2017-01-21 18:50:53	0|BlueMatt|ironically the only one i could find that has reasonable recommendations is by a miner: https://btc.com/stats/unconfirmed-tx
 25 2017-01-21 18:51:04	0|BlueMatt|(ofc its looking at mempool state, which is also kinda shit, but...)
 26 2017-01-21 18:54:18	0|gmaxwell|mempool state is great, if you're getting it from a miner.
 27 2017-01-21 18:54:48	0|TD-Linux|shouldn't overpaying be easy to verify by running the estimator against historical data?
 28 2017-01-21 18:54:56	0|BlueMatt|gmaxwell: well, fair
 29 2017-01-21 18:54:58	0|gmaxwell|(thus have no issues with non-standard txn, and no its complete)
 30 2017-01-21 18:55:01	0|gmaxwell|er know
 31 2017-01-21 18:55:52	0|gmaxwell|part of the challenge is miners that @#$@# set their blocksize to less than the maximum... hard for an estimator to know if the block wasn't full because there wasn't supply or because the size was artifically restricted.
 32 2017-01-21 18:56:12	0|BlueMatt|TD-Linux: current algorithms are all of the "look at what we had already seen on the public network get into a block to avoid any trickery by miners" variety....the problem is these algorithms tend to have no way to correct downwards if everyone is using them
 33 2017-01-21 18:56:42	0|BlueMatt|gmaxwell: yea, and lots of shit that pays 0.1 sat/byte that gets mined :p
 34 2017-01-21 18:59:33	0|gmaxwell|BlueMatt: lots of things that get pushed through miner priority interfaces and pays god knows what gets mined.
 35 2017-01-21 18:59:44	0|BlueMatt|yea, that too
 36 2017-01-21 19:00:21	0|BlueMatt|I dont have a solution, just noting that things are currently Fucked (tm)
 37 2017-01-21 19:00:39	0|gmaxwell|which is why you can't simply use the existance of low fee txn getting mined as a measurement of anything. You need to use things not getting mined as your measurement. :)
 38 2017-01-21 19:01:48	0|BlueMatt|lol, yes, thank you, I'm well aware of restrictions, but also noting things like mempools on many public nodes being <1MB and miners who post their mempool having the same thing is pretty strong supporting evidence of things being Fucked
 39 2017-01-21 19:02:05	0|TD-Linux|BlueMatt, I don't see why they can't correct downward. they would just be delayed one block
 40 2017-01-21 19:02:26	0|TD-Linux|err nevermind I see.
 41 2017-01-21 19:02:27	0|BlueMatt|TD-Linux: well /someone/ has to be generating such transactions for that to happen, and in sufficient volume
 42 2017-01-21 19:02:46	0|sipa|i nominate BlueMatt to do the low fee tx creation job
 43 2017-01-21 19:03:03	0|BlueMatt|lol, i nominate petertodd...doesnt he already?
 44 2017-01-21 19:04:21	0|gmaxwell|we might want to consider renaming the commandline arguments for minimum relay fee... so as to throw off settings people set back during the flood attacks before there was mempool limiting that have just been forgotten.
 45 2017-01-21 19:04:36	0|BlueMatt|yea, i was wondering about that
 46 2017-01-21 19:04:48	0|gmaxwell|petertodd does create low fee txn, IIRC the openstransactions stuff produces 1s/b transactions and just RBFs them until they get mined.
 47 2017-01-21 19:04:56	0|BlueMatt|yup
 48 2017-01-21 19:05:32	0|BlueMatt|yea, I'm now wondering how much of this comes from stuff failing to relay quickly due to shit like that
 49 2017-01-21 19:06:35	0|TD-Linux|does anyone collect public historical mempool data?
 50 2017-01-21 19:06:49	0|gmaxwell|what does public mean there?
 51 2017-01-21 19:07:32	0|TD-Linux|posted publicly
 52 2017-01-21 19:08:30	0|sipa|i sort of do
 53 2017-01-21 19:09:31	0|gmaxwell|there are some websites with stats but they're all (?) borken because they're all (?) running the aformentioned adjusted settings.
 54 2017-01-21 19:09:55	0|petertodd|gmaxwell: opentimestamps starts with the lowest fee bitcoin core will accept with default settings, and increments it by the minimum fee necessary to relay a replacement each time
 55 2017-01-21 19:09:58	0|BlueMatt|or displaying miles of garbage that has been floating around for months :(
 56 2017-01-21 19:11:01	0|gmaxwell|BlueMatt: that would be an improvement...
 57 2017-01-21 19:11:35	0|gmaxwell|well people aren't using charting interfaces that make sense. Megabytes of mempool is a worthless metric... should be 300 all the time, short of expiration.
 58 2017-01-21 19:11:47	0|petertodd|gmaxwell: any miner with a mature mempool significantly greater than 1MB will likely be mining replacement OTS transactions rather than the first ones broadcast, assuming propagation is reasonably reliable anyway
 59 2017-01-21 19:12:00	0|BlueMatt|gmaxwell: yea, its no where near that much of the time, however
 60 2017-01-21 19:12:21	0|BlueMatt|petertodd: i think the propagation assumption is very weak :/
 61 2017-01-21 19:13:16	0|petertodd|BlueMatt: last I checked the logs, usualy the OTS tx that actually gets mined is after multiple attempts, IIRC ~5 or so
 62 2017-01-21 19:13:36	0|BlueMatt|petertodd: do you have stats on the propagation prior to that point?
 63 2017-01-21 19:14:49	0|petertodd|BlueMatt: point is, after that many attempts, it's likely that at least one will have gotten through
 64 2017-01-21 19:15:38	0|gmaxwell|BlueMatt: well if you want to write a PR that renames the minrelayfee option... I'd test and ACK it.
 65 2017-01-21 19:16:26	0|BlueMatt|yea, ok, willdo
 66 2017-01-21 19:16:34	0|petertodd|BlueMatt: after # of replacements before a OTS tx gets mined is 11
 67 2017-01-21 19:18:48	0|petertodd|BlueMatt: *average #
 68 2017-01-21 19:19:18	0|BlueMatt|petertodd: yea, I'm curious about propagation
 69 2017-01-21 19:19:22	0|BlueMatt|ping me next time it runs
 70 2017-01-21 19:20:17	0|petertodd|BlueMatt: it's a fully automated thing that's constantly running...
 71 2017-01-21 19:20:41	0|BlueMatt|well whats the current txid/feerate?
 72 2017-01-21 19:20:59	0|petertodd|BlueMatt: in 44s I'll give you a txid...
 73 2017-01-21 19:21:48	0|petertodd|BlueMatt: oh, wait, no a bit longer than that... gotta wait for a block
 74 2017-01-21 19:22:59	0|petertodd|BlueMatt: after a timestamp is confirmed, it waits a minimum timeout interval, then sends a timestamp tx the next time a block is found; after that every block found results in the fees being bumped
 75 2017-01-21 19:24:33	0|BlueMatt|k
 76 2017-01-21 19:33:34	0|petertodd|BlueMatt: just checked my logs, looks like 50% of the time where multiple replacements happen, the last replacement to be sent is the one that gets mined
 77 2017-01-21 19:34:02	0|petertodd|BlueMatt: so I'd say that's pretty suggestive evidence that about 50% of the hashing power is mining opt-in RBF
 78 2017-01-21 19:34:56	0|petertodd|here's a recently sent txid, the start of a new replacement chain: 2adceebdd5149b9f6c27764f7a619e69487e55ac37bee2d6f492067585b23f0e
 79 2017-01-21 19:35:00	0|BlueMatt|hmm, or at least a big chunk (eg btcc and bitfury, i assume)
 80 2017-01-21 19:36:46	0|BlueMatt|in any case, regarding ignoring opt-in rbf for fee calculation, i suppose it might make sense if we only use it for high-confirmation requests (eg 3/10 confirmations)
 81 2017-01-21 19:36:47	0|gmaxwell|well it's really easy to see who does and doesn't... the ones that don't (e.g. viabtc) collect a lot less in fees.
 82 2017-01-21 19:36:58	0|BlueMatt|but it certainly will have an impact on our 95% threshold
 83 2017-01-21 19:38:22	0|petertodd|BlueMatt: https://0bin.net/paste/zzWuyYFWyZdkufmh#Y8Zwe0qb684VrruOQC8ZbQ2ovgqE3XLXoe-Ca+w4xn0 <- look for yourself; that's all the replacement chain ends since september
 84 2017-01-21 19:57:07	0|morcos|BlueMatt: take a deep breath!  this is actually one thing fee estimatation does pretty well
 85 2017-01-21 19:57:22	0|morcos|(oops, didn't catch up all the way on backlog)
 86 2017-01-21 19:57:54	0|morcos|the reason we don't get estimates that low is the threshold is set very high and we don't have estimates for greater than 25 blocks
 87 2017-01-21 19:58:15	0|BlueMatt|im aware
 88 2017-01-21 19:58:31	0|BlueMatt|but that doesnt explain why estimates are currently in the 50s, and not the 10s
 89 2017-01-21 19:59:25	0|morcos|it does explain, the estimate is answering the question of i want a 95% chance of being filled in my target
 90 2017-01-21 19:59:40	0|morcos|10s will not get you filled withing 25 blocks wiht 95% chance
 91 2017-01-21 19:59:55	0|BlueMatt|yes, see later discussion, i believe that is in part due to propagation issues
 92 2017-01-21 20:00:40	0|morcos|10 sat very recently has had a 75% of being confirmed in 16 blocks and over a longer time period a 75% chance of being confirmed in 32 blocks
 93 2017-01-21 20:00:47	0|morcos|for 95% chance you have to go up to a whole day
 94 2017-01-21 20:01:01	0|morcos|thats not due to propagation issues i doubt
 95 2017-01-21 20:01:12	0|morcos|i think pretty much everyone would propagate > 10s byte
 96 2017-01-21 20:01:29	0|morcos|thats due to blocks being mostly full of stuff that pay more than that
 97 2017-01-21 20:01:34	0|morcos|as mentioned, recently it has come down
 98 2017-01-21 20:01:57	0|BlueMatt|im not convinced...many blocks being generated now are not full because there arent txn to include if you make 8 outbound connections
 99 2017-01-21 20:02:26	0|BlueMatt|on some of my public non-listening nodes they do not have 1MB of txn in their mempool
100 2017-01-21 20:02:43	0|BlueMatt|(and havent all day)
101 2017-01-21 20:03:52	0|morcos|i'd bet you just happen to be looking at lull in transaction backlog
102 2017-01-21 20:04:11	0|BlueMatt|these nodes have been online for months
103 2017-01-21 20:04:13	0|morcos|if it stays like this for a week... estimates will come down
104 2017-01-21 20:04:29	0|BlueMatt|thats possible, but also concerning
105 2017-01-21 20:04:31	0|morcos|yes, i'm saying a few days ago.. your mempool had more expensive txs in it
106 2017-01-21 20:04:37	0|BlueMatt|10 sat very recently has had a 75% of being confirmed in 16 blocks
107 2017-01-21 20:04:43	0|BlueMatt|what is your "very recently"?
108 2017-01-21 20:05:41	0|morcos|checking..  normal fee esitmation has a half life of about 2.5 days
109 2017-01-21 20:06:18	0|morcos|i have a node with short term and long term estimates
110 2017-01-21 20:06:30	0|morcos|the short term estimate has halflife of 6 hours
111 2017-01-21 20:07:09	0|morcos|sigh.. just realized it crashed 2 days ago...  so that number was from 2 days ago
112 2017-01-21 20:07:18	0|BlueMatt|ahh, grrr
113 2017-01-21 20:09:30	0|morcos|it is true that the existing algorithm doesn't respond very quickly to changes in conditions
114 2017-01-21 20:09:37	0|morcos|anyway.. i have to run now.. i'll restart that node
115 2017-01-21 20:09:51	0|morcos|and we can discuss in more detail later..  but i have TONS of historical data on this...
116 2017-01-21 20:10:32	0|morcos|the most important thing to actually let people place lower fee txs would be to have a way to say something like i want a 75% change of being confirmed in the next 6 hours
117 2017-01-21 20:10:46	0|morcos|in otherwords be able to decrease confidence and increase target
118 2017-01-21 20:10:59	0|morcos|then you'll get much lower estimates
119 2017-01-21 20:11:49	0|BlueMatt|yes, I super think we should set lower confidence thresholds for people who ask about 20 blocks
120 2017-01-21 20:12:18	0|BlueMatt|i mean if you're asking about 20 blocks I'd hope you understand its +/- 100 at that point
121 2017-01-21 20:12:38	0|morcos|agree entirely
122 2017-01-21 20:12:55	0|morcos|and its easy to do combo stuff
123 2017-01-21 20:13:16	0|morcos|ok well they want 20 blocks lets give them a 50% chance its within 20, but a 90% chance its within 40
124 2017-01-21 20:13:41	0|morcos|i mean you just ask both questions and give the max
125 2017-01-21 20:13:47	0|BlueMatt|yea
126 2017-01-21 20:13:57	0|BlueMatt|i mean that sounds like a uselessly complicated api
127 2017-01-21 20:14:08	0|BlueMatt|(how many people will actually use that?) but maybe
128 2017-01-21 20:14:25	0|morcos|well yeah i'm assuming that the default will be that happens behind the scenes
129 2017-01-21 20:14:58	0|morcos|but that if for some reason you don't like whatever magic core is doing, then you can ask the direct questions
130 2017-01-21 20:14:59	0|BlueMatt|sure, ok
131 2017-01-21 20:15:04	0|morcos|anwyay, lets discuss later...
132 2017-01-21 20:15:10	0|morcos|got to run
133 2017-01-21 22:28:47	0|morcos|BlueMatt: I have another node that has been consistently running that keeps estimates out to 1000 blocks but only at the default decay...
134 2017-01-21 22:29:56	0|morcos|For now it has enough data to accurately report lower fee rates..  but you are right actually that there are very few txs down there.. and so if no one is asking about those less quick confirmations.. (b/c we don't give them a way to)
135 2017-01-21 22:30:23	0|morcos|then no one will places txs at those lower feerates and we may eventually not have any data points
136 2017-01-21 22:31:17	0|morcos|so probably we should provide that opportunity sooner rather than later..   i wrote the code over a year ago.. but it was a bit complicated and i didn't want to worry about tryng to get it reviewed
137 2017-01-21 22:32:31	0|morcos|anyway, here is some data..   # blocks target:   95% threshold (current default)   ,  75% target (maybe what we could give)
138 2017-01-21 22:33:57	0|morcos|16: 60000 , 4600
139 2017-01-21 22:35:10	0|morcos|32: 40000, 4600
140 2017-01-21 22:36:18	0|morcos|64: 10000, 2500 (can't get lower due to lack of data)
141 2017-01-21 22:37:17	0|morcos|128: 4600, 2500
142 2017-01-21 22:38:16	0|morcos|My guess is things less than 10000 (10 sat/byte) have a problem reaching the very high thresholds b/c there are probably some miners with minrelayfee set at 10
143 2017-01-21 22:39:48	0|morcos|for good measure
144 2017-01-21 22:40:06	0|morcos|8: 65000, 15000
145 2017-01-21 23:30:47	0|BlueMatt|morcos: units???
146 2017-01-21 23:32:05	0|gmaxwell|So I have a question-- would we ever want to have support for trusted feerate information? e.g. to be used when your own estimator doesn't have enough data, or such that it only decreases your estimates?
147 2017-01-21 23:33:33	0|BlueMatt|if we have bumpfee........maybe
148 2017-01-21 23:34:00	0|BlueMatt|i mean its not like all other wallets dont have to....
149 2017-01-21 23:47:08	0|luke-jr|if you mean alert-style signed-by-someone, I think it would need to be outside the reference codebase probably, unless we want to change the policy on such things altogether
150 2017-01-21 23:47:49	0|luke-jr|unless perhaps it'd be signed by the miners themselves
151 2017-01-21 23:59:28	0|gmaxwell|luke-jr: I mean something you could configure, which wouldn't have a default.  "Get fee estimates from key XYZ"
152 2017-01-21 23:59:44	0|gmaxwell|(or really, multisig withnessscripthash)