1 2016-12-09 00:07:30	0|CubicEarth|I had a repeat of my node stalling when trying to sync a couple of days ago. Recent version of Ubuntu, 13.1, and basically nothing else on the machine. It was about 6 days behind, and promptly caught up to about 40 hours remaining.  Then it just stopped. CPU was idle, only relaying traffic was seemingly being passed. After about 10 minutes I restarted the node, and syncing was completed within just a coup
  2 2016-12-09 00:07:30	0|CubicEarth|le minutes of he restart.
  3 2016-12-09 00:09:01	0|CubicEarth|I did try booting peers, but that didn't help anything.
  4 2016-12-09 00:09:02	0|sipa|this is a known issue
  5 2016-12-09 00:09:17	0|sipa|it is resolved automatically if you wait for the next block
  6 2016-12-09 00:09:37	0|CubicEarth|I'm glad it's know :)
  7 2016-12-09 01:10:05	0|bitcoin-git|[13bitcoin] 15sipa opened pull request #9303: Update comments in ctaes (06master...06ctaes) 02https://github.com/bitcoin/bitcoin/pull/9303
  8 2016-12-09 01:17:51	0|gmaxwell|CubicEarth: it's due to nodes (usually spy nodes) that break the protocol and don't respond to headers requests interacting poorly with the sync logic.
  9 2016-12-09 01:25:20	0|CubicEarth|I was wondering...  Seems like my node ought to be a little bit more selfish, a little bit more aggressive, in requesting the data it needs. At least I wish it was. I'm guessing what you are describing is due to the fact Core codes nodes to be good network citizens, and non-standard nodes can disrupt that?
 10 2016-12-09 01:29:49	0|CubicEarth|Once payment channels and LN become a reality, I foresee the P2P layer integrating lots of micro-fees, charging for serving block data, for relaying TX, etc.
 11 2016-12-09 01:36:38	0|gmaxwell|CubicEarth: no, it's not due to that, pretty much the opposite.
 12 2016-12-09 01:37:05	0|gmaxwell|if it requests redundant data, then it might knock itself off the network if otherwise it only has enough bandwidth available to only fetch one copy.
 13 2016-12-09 01:39:58	0|CubicEarth|So my node requests a piece of data, what happens?  A peer says "yes, I have it, I'll give it to you" and then never does? And my node sites there, waiting, because if it asked another peer for data they could both end up providing it?
 14 2016-12-09 01:42:10	0|gmaxwell|CubicEarth: right. It will eventually give up-- but for this particular request its triggered by a new block showing up on the network.
 15 2016-12-09 01:43:22	0|gmaxwell|Smarter would be dynamic timeouts, -- the tricky thing is that care has to be taken to avoid unstable algorithins that can suffer congestion collapse. E.g. you have a limited bandwith network with 5 nodes on it... and then they fall behind and start agressively rerequesting and never recover.
 16 2016-12-09 01:43:41	0|gmaxwell|It's not _that_ hard, but ... so many other things going on...
 17 2016-12-09 01:50:36	0|CubicEarth|Other priorities for sure. It's nice that it respects bandwidth limits currently.  It makes sense for it to be conservative by default, but perhaps there should / could be a setting where you tell how much bandwidth you would like it make use of, not just an "upper limit" for inbound connections, but "please use this much to make things happen faster"
 18 2016-12-09 01:51:30	0|CubicEarth|Onto the back burner...
 19 2016-12-09 01:52:44	0|gmaxwell|well it's not that it respects, it just has no idea what they are, so it assumes it's operating with very little, since thats consderative.
 20 2016-12-09 01:53:46	0|gmaxwell|RE manual setting, I think very few users will use that correctly--  we should have settings, but they're way less important than better default behavior.
 21 2016-12-09 02:00:01	0|CubicEarth|The funny thing is the software is already 'aware' in the sense that it can generate a graph of network activity, but I get that it's probably not 'hardened code'.
 22 2016-12-09 02:01:58	0|gmaxwell|Past performance doesn't indicate future results. Assuming that it does is how you get schemes that suffer congestion collapse. :P
 23 2016-12-09 02:03:02	0|CubicEarth|gmaxwell: so a node dosent DDoS itself?
 24 2016-12-09 02:08:12	0|gmaxwell|CubicEarth: e.g. you have 5 nodes on a 1 mbit connection. They each observe 1mbit available.. but then they all try to use it at once and there will be far less than 1mbit available.
 25 2016-12-09 02:09:11	0|CubicEarth|got it
 26 2016-12-09 03:39:58	0|bitcoin-git|[13bitcoin] 15droark opened pull request #9304: Allow linearization scripts to support little endian hashes (06master...06master) 02https://github.com/bitcoin/bitcoin/pull/9304
 27 2016-12-09 04:22:18	0|bitcoin-git|[13bitcoin] 15kallewoof opened pull request #9305: Refactor: Removed begin/end_ptr functions. (06master...06remove-begin-end_ptr-usage) 02https://github.com/bitcoin/bitcoin/pull/9305
 28 2016-12-09 09:23:22	0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 3 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/86017842d6ef...72bf1b3d0962
 29 2016-12-09 09:23:23	0|bitcoin-git|13bitcoin/06master 14760765d 15Pieter Wuille: Update ctaes
 30 2016-12-09 09:23:23	0|bitcoin-git|13bitcoin/06master 148501bed 15Pieter Wuille: Squashed 'src/crypto/ctaes/' changes from cd3c3ac..003a4ac...
 31 2016-12-09 09:23:24	0|bitcoin-git|13bitcoin/06master 1472bf1b3 15MarcoFalke: Merge #9303: Update comments in ctaes...
 32 2016-12-09 09:23:41	0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #9303: Update comments in ctaes (06master...06ctaes) 02https://github.com/bitcoin/bitcoin/pull/9303
 33 2016-12-09 13:23:24	0|BlueMatt|hmmmmmm....I'm pretty sure CheckForkWarningConditionsOnNewFork is completely useless atm...
 34 2016-12-09 13:24:12	0|BlueMatt|it looks like it was written assuming pindexNewForkTip would be a CBlockIndex* to the highest block on a new fork (which I think is the case in the original code, and I'm not just saying it because I originally wrote it)
 35 2016-12-09 13:24:44	0|BlueMatt|but in the current code its given the last block which ActivateBestChainStep wanted to connect (but failed because either it, or a previous block) was invalid
 36 2016-12-09 13:25:30	0|BlueMatt|and that last block is always either current chain + 1 unless its a reorg
 37 2016-12-09 13:37:17	0|BlueMatt|ehh, excuse me...CheckForkWarningConditionsOnNewFork is called with a larger vector which isnt exactly the things ABCS tries, but what it might've tried if it didnt want to give up cs_main earlier
 38 2016-12-09 13:37:31	0|BlueMatt|still, i think due to headers first this stuff is horribly broken
 39 2016-12-09 14:37:03	0|gmaxwell|nah it does work.
 40 2016-12-09 14:42:28	0|BlueMatt|you sure? that code doesnt look sane to me now
 41 2016-12-09 14:42:42	0|BlueMatt|(do we have any tests for it? couldnt find any)
 42 2016-12-09 14:43:01	0|BlueMatt|and jl2012 was saying he tried to trigger it by sending invalid blocks with valid headers and couldnt
 43 2016-12-09 15:10:15	0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #9306: Make CCoinsViewCache::Cursor() return latest data (06master...06pr/coins-cursor) 02https://github.com/bitcoin/bitcoin/pull/9306
 44 2016-12-09 15:33:49	0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #9307: [trivial] Remove undefined FetchCoins method declaration (06master...06pr/coins-delfetch) 02https://github.com/bitcoin/bitcoin/pull/9307
 45 2016-12-09 15:36:20	0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #9308: [test] Add CCoinsViewCache Access/Modify/Write tests (06master...06pr/coins-test) 02https://github.com/bitcoin/bitcoin/pull/9308
 46 2016-12-09 15:54:37	0|Teroxice|Hello
 47 2016-12-09 15:54:52	0|Teroxice|I build an ATM software and right now its sending money from a centralized bitcore server with just one wallet.  I will offer my software to the market but I would like that every client has their own independent wallet in my centralized bitcore server.  That is why I'm asking if it is posible to have more than one wallet on the same bitcore server.  Anyone knows?
 48 2016-12-09 15:55:19	0|achow101|Teroxice: bitcore or bitcoin core? There is a difference.
 49 2016-12-09 15:55:37	0|achow101|also, wrong channel
 50 2016-12-09 15:55:58	0|Teroxice|bitcoin core
 51 2016-12-09 15:56:20	0|achow101|bitcoin core does not have multiwallet support
 52 2016-12-09 15:56:20	0|instagibbs|Teroxice, try #bitcoin
 53 2016-12-09 16:46:59	0|bitcoin-git|[13bitcoin] 15morcos opened pull request #9309: Spurious RPC test failure (06master...06rarerpcfail) 02https://github.com/bitcoin/bitcoin/pull/9309
 54 2016-12-09 18:09:29	0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #9310: Assert FRESH validity in CCoinsViewCache::BatchWrite (on top of #9308) (06master...06pr/coins-batch-assert) 02https://github.com/bitcoin/bitcoin/pull/9310
 55 2016-12-09 18:53:48	0|bitcoin-git|[13bitcoin] 15morcos opened pull request #9311: Flush wallet after abandontransaction (06master...06flushwalletonabandon) 02https://github.com/bitcoin/bitcoin/pull/9311
 56 2016-12-09 19:03:43	0|morcos|gmaxwell: maybe easier to explain what's happening in #9167 here
 57 2016-12-09 19:03:44	0|gribble|https://github.com/bitcoin/bitcoin/issues/9167 | IsAllFromMe by morcos · Pull Request #9167 · bitcoin/bitcoin · GitHub
 58 2016-12-09 19:04:09	0|morcos|if you ran that same command on the old code your grep would find fee twice
 59 2016-12-09 19:04:35	0|morcos|it prints an overall fee for the transaction and it prints a fee for each accounting entry
 60 2016-12-09 19:05:24	0|morcos|it was already the case that if it didn't think the tx was from you (meaning none of it was from you) it would leave off the overall fee for the tx
 61 2016-12-09 19:05:51	0|gmaxwell|ah, I see. what the heck is the fee for the accounting entries for?
 62 2016-12-09 19:05:52	0|morcos|i changed that logic to be whether all of it was from you
 63 2016-12-09 19:06:01	0|morcos|don't get me started on that
 64 2016-12-09 19:06:32	0|morcos|getbalance("*") uses those random incorrect negative fees to offset other errors in tracking balances
 65 2016-12-09 19:06:38	0|morcos|so it was not possible to fix those
 66 2016-12-09 19:06:58	0|morcos|i suppose we could not print them, but that seems like an api change
 67 2016-12-09 19:07:34	0|gmaxwell|We should probably be telling the wallet about the actual values of all the inputs... we know them (we're a full node, after all!). then the fees it displays can be correct.
 68 2016-12-09 19:08:01	0|gmaxwell|but thats a bigger change.
 69 2016-12-09 19:08:18	0|morcos|yeah i don't think we should try to make any changes of that behavior until we remove accounts
 70 2016-12-09 19:08:21	0|morcos|then we can clean up a lot
 71 2016-12-09 19:08:56	0|morcos|which reminds me i need to be participating in wumpus's labels discussion
 72 2016-12-09 19:09:03	0|sdaftuar|gmaxwell: +1 on telling the wallet!
 73 2016-12-09 19:09:08	0|sdaftuar|er, about al the fees
 74 2016-12-09 19:09:12	0|sdaftuar|i think it's nuts the way that works now
 75 2016-12-09 19:09:50	0|gmaxwell|we've let the accounts stuff deadlock us for a long time, I believe that this was also the reason we didn't fix the absurd handling wrt fees when it was first noticed. :(
 76 2016-12-09 19:10:31	0|sdaftuar|i think my proposal would just be to pass fee information through SyncTransaction().  we have it during ATMP, and we could cache it while validating a block
 77 2016-12-09 19:11:37	0|sdaftuar|that doesn't go as far as all input values, but i think it'd be a simple improvement
 78 2016-12-09 19:12:37	0|gmaxwell|it would be.  full input values are needed if we want to create detailed corrective accounting entries for txn with inputs which aren't fromme.
 79 2016-12-09 19:13:32	0|gmaxwell|e.g. txid 1234  spend 1000 of our coins, spent 4001 coins from {these sources}, paid 5000 coins, and 1 coin fee.
 80 2016-12-09 19:13:52	0|gmaxwell|if {these sources} is just "from outside this wallet" then we don't need per input amounts.
 81 2016-12-09 19:14:03	0|gmaxwell|we just need the fee.
 82 2016-12-09 19:17:23	0|sdaftuar|well i definitely agree that with the goal of per-input amounts being passed through!  maybe that's not so hard to implement either, actually...
 83 2016-12-09 19:18:12	0|sdaftuar|we can probably come up with some reasonable data structure and pass that through to the wallet as well
 84 2016-12-09 19:19:08	0|gmaxwell|sipa: do we have a philosophical opposition to decoderawtransaction using the UTXO set, telling you if each of the inputs is unspent,  and if they all are-- displaying the fee?
 85 2016-12-09 19:19:55	0|sipa|gmaxwell: i think that should be a separate rpc call
 86 2016-12-09 19:20:17	0|gmaxwell|sipa: what would it be called?
 87 2016-12-09 19:20:18	0|sipa|gmaxwell: decoderawtransaction is purely a utility function now, and i think it should stay that way
 88 2016-12-09 19:20:26	0|sipa|analyserawtransaction ?
 89 2016-12-09 19:20:42	0|gmaxwell|please no more words with different en_gb/en_us spelling!
 90 2016-12-09 19:20:43	0|gmaxwell|:P
 91 2016-12-09 19:21:18	0|sipa|rawtransactionanalysis
 92 2016-12-09 19:21:20	0|sipa|:p
 93 2016-12-09 19:21:33	0|sdaftuar|could we add memory-only per-input values to CTransaction(), so that they get filled in and passed through to the wallet in SyncTransaction()?
 94 2016-12-09 19:22:05	0|sipa|evaluaterawtransaction
 95 2016-12-09 19:22:49	0|sipa|sdaftuar: bleh... what if they aren't known? the consensus logic (which uses CTransaction) shouldn't need such values
 96 2016-12-09 19:23:05	0|sdaftuar|right, consensus wouldn't use it.  but it would be convenient to fill in for downstream consumers
 97 2016-12-09 19:23:22	0|gmaxwell|well consensus certantly does eventually need to know the input values! :)
 98 2016-12-09 19:23:25	0|sdaftuar|we could do outside of CTransaction() of course
 99 2016-12-09 19:23:37	0|sipa|gmaxwell: but CTransaction is by design now immutable
100 2016-12-09 19:23:43	0|sdaftuar|the witness is not?
101 2016-12-09 19:23:56	0|sipa|sdaftuar: it will be
102 2016-12-09 19:24:03	0|sdaftuar|ah! didn't realize that.
103 2016-12-09 19:24:12	0|sipa|(also, CTransactionRef is a ref to a _const_ CTransaction, which includes the witness)
104 2016-12-09 19:24:45	0|sdaftuar|ok so i guess stuffing data into that just won't work
105 2016-12-09 19:24:51	0|sipa|but we could have a wrapper around CTransaction that adds some metadata, which is used by ATMP and wallet code
106 2016-12-09 19:25:04	0|sipa|or just pass along a separate object that contains that metadata
107 2016-12-09 19:25:21	0|morcos|but speakign of that idea... lets imagine inputs were part of CTxMempoolEntry, then maybe you don't need a UTXO cache anymore
108 2016-12-09 19:25:46	0|sipa|morcos: though it makes the mempool's correctness now consensus critical
109 2016-12-09 19:25:55	0|sdaftuar|nack
110 2016-12-09 19:25:58	0|morcos|thats what we're arguign about
111 2016-12-09 19:26:38	0|gmaxwell|eventually the fast that txn are already verified in the mempool will have to be exploited for performance reasons.  :(
112 2016-12-09 19:27:29	0|sdaftuar|i would like that to be the last change that goes in before i stop working on the codebase :)
113 2016-12-09 19:27:38	0|gmaxwell|hah. yea. :(
114 2016-12-09 19:27:44	0|gmaxwell|or we'll just end up with miners using Joe-Marketers-Recklessly-Optimized-Fork that "validates blocks 5x faster"...
115 2016-12-09 19:28:09	0|gmaxwell|and then all the care in not being reckless didn't matter because relevant parties aren't using it.
116 2016-12-09 19:31:10	0|sipa|morcos: even if we had that, we'd need to apply utxo changes to the chainstate... which is perhaps harder if we haven't previously looked up the entry (because it's missing from intermediate cache layers, we don't know if it's fresh...)
117 2016-12-09 19:33:06	0|sipa|i'd prefer something weak-block based to have pre-evaluated sets of transactions to apply to the chainstate
118 2016-12-09 19:34:51	0|sdaftuar|sipa: so assuming the set that gets mined is identical to something you were expecting, you can have very fast validation?
119 2016-12-09 19:35:00	0|sipa|sdaftuar: yup
120 2016-12-09 19:35:22	0|sipa|you can basically have the utxo set diff cached
121 2016-12-09 19:36:17	0|gmaxwell|(not just that but you can have the block template for the next block you'd mine after it cached)
122 2016-12-09 19:48:52	0|bitcoin-git|[13bitcoin] 15morcos opened pull request #9312: Increase mempool expiry time to 2 weeks (06master...06longerexpiry) 02https://github.com/bitcoin/bitcoin/pull/9312
123 2016-12-09 20:34:40	0|gmaxwell|#9295 is an obvious simply bugfix that really would like to be merged (and backported)
124 2016-12-09 20:34:42	0|gribble|https://github.com/bitcoin/bitcoin/issues/9295 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
125 2016-12-09 20:44:16	0|morcos|gmaxwell: the whole requesting parents of orphans functionality...  i didn't realize it basically doesn't work when your peer is a core node.  were you aware of that?
126 2016-12-09 20:44:45	0|morcos|b/c you won't let a peer getdata a tx that's not in your relay map
127 2016-12-09 20:44:50	0|gmaxwell|morcos: it works so long as the parents are still in the relay pool, or if they're an older version.
128 2016-12-09 20:45:03	0|gmaxwell|(which will answer out of the mempool)
129 2016-12-09 20:45:14	0|gmaxwell|Yes, I knew that when I did it.
130 2016-12-09 20:45:37	0|gmaxwell|In particular it's helpful for older versions that still do the trickling.. they're the source of most orphans I see, and they also answer out of the mempool.
131 2016-12-09 20:45:55	0|morcos|ok.. i just noticed it on a new node i started up
132 2016-12-09 20:46:38	0|gmaxwell|I'm glad someone else noticed.
133 2016-12-09 20:47:35	0|morcos|i'm about to PR a small fix to fee filter your minrelaytxfee if your limitfreerelay is 0..  will save some unnecessary free tx requesting/rejecting
134 2016-12-09 20:47:53	0|gmaxwell|thank god.
135 2016-12-09 20:48:53	0|gmaxwell|Re the minrelayfee decay,  did much thought or testing go into it?  ISTM it looks like it goes too low. e.g. continues dropping at a relatively high speed even once its past a level where your mempool will fill again, given enough time.
136 2016-12-09 20:52:02	0|bitcoin-git|[13bitcoin] 15morcos opened pull request #9313: If we don't allow free txs, always send a fee filter (06master...06minminfee) 02https://github.com/bitcoin/bitcoin/pull/9313
137 2016-12-09 20:53:15	0|morcos|gmaxwell: much thought went into it, there is a bit of a discussion on #6722 on my reasoning behind the half-life of 12 hours...   But its certainly possible that the tradeoff has changed a bit.
138 2016-12-09 20:53:17	0|gribble|https://github.com/bitcoin/bitcoin/issues/6722 | Limit mempool by throwing away the cheapest txn and setting min relay fee to it by TheBlueMatt · Pull Request #6722 · bitcoin/bitcoin · GitHub
139 2016-12-09 20:54:21	0|morcos|The idea behind the min fee was to protect the limited resource of your memory, so it wasn't meant to be smart enough to know that certain fees are really never going to be worth it..
140 2016-12-09 20:57:24	0|morcos|Actually rereading the justification now, I think if anything we'd move it the other way..   Packages are limited to much smaller than the 2.5MB envisioned in that argument...  So the amount of "free relay" is really small.
141 2016-12-09 20:57:56	0|gmaxwell|morcos: I guess what trips it up is that it decays at a constant rate but the supply of transactions is not uniform (or even 1/rate).
142 2016-12-09 20:58:22	0|morcos|what exactly is the behavior you are seeing that you think is not good
143 2016-12-09 20:59:56	0|gmaxwell|that it drops it back down to nothing and then one of these clowns that relays old transactions connects, fills me back to 300mb... then 72 hours later, these expire, and it slides back down....
144 2016-12-09 21:00:58	0|gmaxwell|At least in my mind what it should be trying to do is find a value that results in the mempool being close to full-- but it ends up far lower than that; maybe I'm thinking of the wrong goal.
145 2016-12-09 21:01:08	0|morcos|yeah but even if it didn't go down at all, he could do the exact same thing at 2 sat/byte.  They still wouldn't be mined within 82 hours
146 2016-12-09 21:01:15	0|morcos|72
147 2016-12-09 21:01:49	0|sdaftuar|i think modeling the arrival rate of transactions is hard
148 2016-12-09 21:02:47	0|gmaxwell|I think what I'm talking about is as much a question of the distribution of feerates in the supply of unconfirmed transactions, as it is arrival. (more so, because at the 2sat byte level, they're never getting mined.)
149 2016-12-09 21:03:03	0|morcos|i think if you look at tx supply..  there is a backlog of between 1MB - 100MB of txs that pay > 1 sat/byte   and there is a backlog of 100-1000MB additional of txs that pay 1 sat/byte   it just happens to be the distribution of txs now i think
150 2016-12-09 21:03:16	0|morcos|gmaxwell: ah, but thats wrong
151 2016-12-09 21:04:22	0|morcos|for txs that pay between 1.5-2 sat/byte  95% of them are mined within 1000 blocks and 75% of them are mined within 256 blocks
152 2016-12-09 21:04:25	0|morcos|crazy right
153 2016-12-09 21:05:05	0|gmaxwell|bleh. okay. Crazy.
154 2016-12-09 21:06:55	0|morcos|it's just there are sooo many in the 1-1.5 range that only about 10% of them get mined within 1000 blocks
155 2016-12-09 21:56:52	0|luke-jr|gmaxwell: hmm, I wonder if we should be rescanning for conflicts then (#9290)
156 2016-12-09 21:56:54	0|gribble|https://github.com/bitcoin/bitcoin/issues/9290 | Make RelayWalletTransaction attempt to AcceptToMemoryPool. by gmaxwell · Pull Request #9290 · bitcoin/bitcoin · GitHub
157 2016-12-09 21:57:15	0|gmaxwell|luke-jr: make rescan take less than N hours? :-/
158 2016-12-09 21:57:26	0|luke-jr|hours now? :o
159 2016-12-09 21:57:40	0|gmaxwell|I think it takes 5 on my laptop.
160 2016-12-09 21:57:54	0|gmaxwell|on a really fast machine it's not so terrible.
161 2016-12-09 21:58:10	0|luke-jr|I guess ideally we should be waiting until the blockchain is synced, checking if we have unconfirmed txns, checking a wallet flag, and rescanning at runtime
162 2016-12-09 21:58:19	0|luke-jr|sounds complex though :/
163 2016-12-09 21:59:06	0|luke-jr|(oh, and then we could simply not broadcast unconfirmed txns until it finishes the rescan)
164 2016-12-09 21:59:13	0|gmaxwell|the 'getting real fee info' will be another reason to rescan the chain for all wallets.
165 2016-12-09 21:59:49	0|gmaxwell|so it would be worth giving some thought to adding an extra kinda of versioning to the wallet (metadata-rescanned-since).. and making rescan faster...
166 2016-12-09 21:59:57	0|luke-jr|yeah
167 2016-12-09 22:00:21	0|luke-jr|"rescan depth" or something
168 2016-12-09 22:00:25	0|sdaftuar|luke-jr: it's in general not really possible to always know that a transaction is conflicted.  i think we should keep that in mind before doing anything expensive...
169 2016-12-09 22:00:29	0|gmaxwell|well they won't broadcast if conflicted-- they'll fail for mempool add. so the only harm is wasting a little time trying to look up utxos that aren't there.
170 2016-12-09 22:00:56	0|luke-jr|to be happy with downgrading+upgrading, we'd need a timestamp per each depth, but that seems unnecessarily over-compatible
171 2016-12-09 22:01:03	0|dcousens|gmaxwell: rescan is for private keys no?
172 2016-12-09 22:01:14	0|dcousens|or is it UTXO?
173 2016-12-09 22:01:15	0|luke-jr|gmaxwell: oh, right. that's not too bad compared to hours rescanning
174 2016-12-09 22:01:56	0|luke-jr|sdaftuar: ah, parent conflicts
175 2016-12-09 22:02:25	0|gmaxwell|Yes, though we won't spend non-wallet inputs (ignoring coinjoins because stupid) until they are six confirms old.
176 2016-12-09 22:02:40	0|gmaxwell|So it's really hard to end up in a case where there is an undetectable conflict in your wallet.
177 2016-12-09 22:02:52	0|luke-jr|gmaxwell: if someone else sends you a payment using coins later conflicted?
178 2016-12-09 22:03:07	0|dcousens|why is rescan so slow? is it because it tries to match all possible scripts or?
179 2016-12-09 22:03:22	0|gmaxwell|luke-jr: you won't spend that payment until it is 6 confirmed.
180 2016-12-09 22:03:35	0|gmaxwell|dcousens: I think much (most?) of the time is spent hashing the blocks.
181 2016-12-09 22:04:12	0|luke-jr|gmaxwell: but you'll try to mempool-add the receive, no?
182 2016-12-09 22:04:16	0|gmaxwell|luke-jr: okay, sure nevermind me.. that payment itself will be an undetectable conflict, I was only thinking about IsFromMe transactions.
183 2016-12-09 22:04:55	0|dcousens|gmaxwell: why is it hashing blocks? (i'll look into the code ooi)
184 2016-12-09 22:05:20	0|gmaxwell|dcousens: side-effect of deseralization.
185 2016-12-09 22:07:32	0|dcousens|gmaxwell: still weir,t I use similar enough deserialization code (lib/consensus) in my own parser and it does a full parse purely bottlenecked at IO, so 3-4 minutes on an SSD, script checking shouldn't be much more on that i'd of though...
186 2016-12-09 22:07:41	0|dcousens|but I guess I'd have to look into the code to find out why
187 2016-12-09 22:09:05	0|gmaxwell|At least my vague recollection of profiling it before was that the time was all in the heap allocator and sha256.
188 2016-12-09 22:12:46	0|dcousens|gmaxwell: hmmm, by hashing do you mean the merkle tree generation?
189 2016-12-09 22:13:02	0|dcousens|well, merkle root calculation***
190 2016-12-09 22:13:19	0|gmaxwell|I assume, I don't know what else sha256 would be used for while scanning blocks.
191 2016-12-09 22:14:42	0|sipa|the checksum?
192 2016-12-09 22:15:02	0|sipa|blocks on disk have a checksum, i think?
193 2016-12-09 22:15:17	0|sipa|or am i confusing it with peers.day
194 2016-12-09 22:15:25	0|dcousens|sipa: they have a magic byte
195 2016-12-09 22:15:30	0|dcousens|no checksum afaik
196 2016-12-09 22:15:51	0|gmaxwell|fun random graph https://people.xiph.org/~greg/temp/blk-sizes-windowed.png
197 2016-12-09 22:16:14	0|dcousens|a checksum would probably work wonders in bypassing that
198 2016-12-09 22:16:35	0|dcousens|esp. given the situation of all the zero padding in the files
199 2016-12-09 23:11:12	0|dcousens|RE: 9312, 2 week expiration time,  won't that further prevent parties from attempting broadcast a conflict of a stuck transaction 2.1 days after the first broadcast, and having a high chance of the other transaction being "mostly" out of the network by then?
200 2016-12-09 23:11:18	0|dcousens|of course, that is what RBF is for, but
201 2016-12-09 23:13:38	0|gmaxwell|dcousens: not really. (1) right now those conflicts are pretty successful immediately; due to nodes restarting, full-rbf miners, etc.  (2) there are people who go around connecting to everyone constantly rebroadcasting old transactions... so they're already defeating that timeout.
202 2016-12-09 23:13:55	0|dcousens|gmaxwell: in practice it works though?
203 2016-12-09 23:14:10	0|dcousens|oh wait, misread what you wrote
204 2016-12-09 23:14:26	0|gmaxwell|in practice it works even in 1 hour, so it works-- but not because of the 72 hour timeout.
205 2016-12-09 23:14:32	0|dcousens|yeah
206 2016-12-09 23:16:45	0|gmaxwell|we could also consider adding a new datastructure, a blacklist: if something hits the expire time, that txid becomes blacklisted for n-months.  That would actually make things much easier to replace with a double spend after two weeks.
207 2016-12-09 23:17:34	0|dcousens|gmaxwell: is there a way to "push out" a transaction from the mempool using the RPC? aka, "oops, forget the last one, use this instead"
208 2016-12-09 23:17:50	0|dcousens|aka, ignore conflicts, or drop conflicts
209 2016-12-09 23:18:13	0|dcousens|just thinking about you could by-pass that timeout without wiping your local mempool
210 2016-12-09 23:18:35	0|dcousens|obviously that wouldn't help others
211 2016-12-09 23:18:49	0|dcousens|but, as you say, others are quite the dynamic
212 2016-12-09 23:22:25	0|gmaxwell|dcousens: kinda useless to do something locally, it won't do it to anyone else...
213 2016-12-09 23:25:14	0|dcousens|gmaxwell: depending on their expiration timeout, mempool size & fee filter, full-RBF, etc, could go either way no?
214 2016-12-09 23:26:34	0|gmaxwell|dcousens: you can send things that aren't in your mempool, the whitelisting stuff does that already.
215 2016-12-09 23:27:21	0|dcousens|gmaxwell: true