1 2016-12-14 01:34:45 0|gmaxwell|jonasschnelli: want to go comment on http://bitcoin.stackexchange.com/questions/50125/failing-to-build-bitcoin-core-v0-13-1-on-debian-stretch and point out its fixed in master?
2 2016-12-14 01:51:44 0|bitcoin-git|[13bitcoin] 15gmaxwell opened pull request #9344: Do not run functions with necessary side-effects in assert() (06master...06assert_no_sideeffects) 02 https://github.com/bitcoin/bitcoin/pull/9344
3 2016-12-14 01:53:06 0|BlueMatt|again? :(
4 2016-12-14 01:53:21 0|gmaxwell|looks like we missed some.
5 2016-12-14 01:54:05 0|sipa|or 2.
6 2016-12-14 01:54:20 0|gmaxwell|or 4.
7 2016-12-14 01:54:38 0|BlueMatt|missed, or re-added?
8 2016-12-14 01:54:53 0|gmaxwell|the flush ones are from 2013, didn't check the others.
9 2016-12-14 01:54:55 0|gmaxwell|looking
10 2016-12-14 01:56:46 0|gmaxwell|BlueMatt: the other two you introduced in 9c837d54.
11 2016-12-14 01:57:03 0|gmaxwell|You have have the brownpaper bag when 2013-sipa is done with it.
12 2016-12-14 01:57:16 0|BlueMatt|lol, oops
13 2016-12-14 04:15:12 0|adam3us|djb has a database called cdb (used in djbdns) that guarantees key->value lookup in 2 disk hits. cost is it is slow to update (batch job)
14 2016-12-14 04:15:49 0|luke-jr|I just mean for block data. Instead of deleting, move it to a USB drive or smth
15 2016-12-14 04:15:59 0|luke-jr|maybe even a location that can be offline when you don't need it
16 2016-12-14 04:16:18 0|luke-jr|so when you want to restore an old backup or whatever, you just plug it in
17 2016-12-14 04:17:08 0|adam3us|yes. i am talking about something only semi-related. that utxo access latency (with sig checking disabled) is increasing due to less spam creating lower cache hit rates, and larger utxo hitting disk more.
18 2016-12-14 04:18:25 0|adam3us|so then maybe a three tier strategy could be old stuff in cdb, more recent (eg last 2months?) in current disk layout, plus memory cache. once per month do the batch to move another 1month chunk to the cdb compact/efficient lookup format.
19 2016-12-14 04:18:49 0|adam3us|cdb has constant time lookup, where general lookup is log n.
20 2016-12-14 04:36:38 0|Lightsword|are there any known issues getting master to compile on the latest osx?
21 2016-12-14 04:55:33 0|Lightsword|nvm was protobuf version issue
22 2016-12-14 05:08:28 0|luke-jr|why is secp256k1.c +x now?
23 2016-12-14 05:08:37 0|luke-jr|main_impl.h also
24 2016-12-14 06:38:08 0|phantomcircuit|instagibbs, are wallet transactions loaded into the mempool on restart?
25 2016-12-14 06:38:11 0|phantomcircuit|i didn't think they were
26 2016-12-14 06:38:26 0|phantomcircuit|wouldn't that break #9262
27 2016-12-14 06:38:28 0|gribble|https://github.com/bitcoin/bitcoin/issues/9262 | Prefer coins that have fewer ancestors, sanity check txn before ATMP by instagibbs ÷ Pull Request #9262 ÷ bitcoin/bitcoin ÷ GitHub
28 2016-12-14 06:38:46 0|gmaxwell|phantomcircuit: yes, they are.
29 2016-12-14 06:38:56 0|phantomcircuit|gmaxwell, hmm
30 2016-12-14 06:39:01 0|gmaxwell|(assuming the mempool would take them).
31 2016-12-14 06:39:03 0|gmaxwell|they do you say it would break 9262?
32 2016-12-14 06:39:08 0|phantomcircuit|they're not rebroadcast though right?
33 2016-12-14 06:39:20 0|gmaxwell|they're rebroadcast if they get into the mempool.
34 2016-12-14 06:39:29 0|phantomcircuit|if they're not in the mempool then checking their ancestor depth will always fail of course
35 2016-12-14 06:40:19 0|gmaxwell|Yea, I see what you're saying-- they're put into the mempool. (and IIRC the wallet will not try to spend them if they're unconfirmed and not in the mempool)
36 2016-12-14 06:40:45 0|phantomcircuit|hmm
37 2016-12-14 06:40:48 0|gmaxwell|(but I could be wrong there and instead remembering the behavior it should have instead of does have... :) )
38 2016-12-14 08:33:15 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9264: 0.13.2 backports (060.13...062016_12_backports_0_13) 02 https://github.com/bitcoin/bitcoin/pull/9264
39 2016-12-14 08:33:53 0|bitcoin-git|13bitcoin/06master 14da9cdd2 15Gregory Maxwell: Do not run functions with necessary side-effects in assert()
40 2016-12-14 08:33:53 0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/26fe5c98ab6a...82ccac739e2a
41 2016-12-14 08:33:54 0|bitcoin-git|13bitcoin/06master 1482ccac7 15Wladimir J. van der Laan: Merge #9344: Do not run functions with necessary side-effects in assert()...
42 2016-12-14 08:34:08 0|bitcoin-git|[13bitcoin] 15sipa opened pull request #9346: Batch construct batches (06master...06reusebatch) 02 https://github.com/bitcoin/bitcoin/pull/9346
43 2016-12-14 08:34:52 0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/82ccac739e2a...47e6a19e6b57
44 2016-12-14 08:34:53 0|bitcoin-git|13bitcoin/06master 14ed6b377 15Jonas Schnelli: [Qt] Console: add security warning
45 2016-12-14 08:34:54 0|bitcoin-git|13bitcoin/06master 1447e6a19 15Wladimir J. van der Laan: Merge #9330: [Qt] Console: add security warning...
46 2016-12-14 08:35:08 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9330: [Qt] Console: add security warning (06master...062016/12/qt_add_sec) 02 https://github.com/bitcoin/bitcoin/pull/9330
47 2016-12-14 08:35:28 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9323: [0.13] Add release notes for wallet/mempool rejections. (PR #9302 and #9290) (060.13...06mempool_relnote) 02 https://github.com/bitcoin/bitcoin/pull/9323
48 2016-12-14 09:00:12 0|phantomcircuit|gmaxwell, that is indeed the behaviour it has
49 2016-12-14 09:18:44 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9211: [0.12 branch] Backports (060.12...06Mf1611-back12) 02 https://github.com/bitcoin/bitcoin/pull/9211
50 2016-12-14 09:25:55 0|wumpus|what's the status on #9037? "Add test-before-evict discipline to addrman" looks like it's finished for a while but hasn't got any review yet
51 2016-12-14 09:25:56 0|gribble|https://github.com/bitcoin/bitcoin/issues/9037 | net: Add test-before-evict discipline to addrman by EthanHeilman ÷ Pull Request #9037 ÷ bitcoin/bitcoin ÷ GitHub
52 2016-12-14 09:28:31 0|bitcoin-git|13bitcoin/06master 14f692fce 15Gregory Maxwell: Make RelayWalletTransaction attempt to AcceptToMemoryPool....
53 2016-12-14 09:28:31 0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/47e6a19e6b57...57e337d40e94
54 2016-12-14 09:28:32 0|bitcoin-git|13bitcoin/06master 1457e337d 15Pieter Wuille: Merge #9290: Make RelayWalletTransaction attempt to AcceptToMemoryPool....
55 2016-12-14 09:28:46 0|bitcoin-git|[13bitcoin] 15sipa closed pull request #9290: Make RelayWalletTransaction attempt to AcceptToMemoryPool. (06master...06resend_retries_mempool) 02 https://github.com/bitcoin/bitcoin/pull/9290
56 2016-12-14 09:30:57 0|phantomcircuit|gmaxwell, iirc SelectMinCoins can be supppper expensive
57 2016-12-14 09:31:06 0|phantomcircuit|er SelectCoinsMinConf
58 2016-12-14 09:31:42 0|gmaxwell|phantomcircuit: it has linear cost in the number of spendable coins below the amount you're sending.
59 2016-12-14 09:32:16 0|phantomcircuit|ok
60 2016-12-14 09:32:25 0|phantomcircuit|was that fixed from something much worse in the past?
61 2016-12-14 09:32:54 0|gmaxwell|wumpus: fell off my radar before I finished review; I will review later today (your time).
62 2016-12-14 09:33:22 0|gmaxwell|phantomcircuit: it used to call the IsFromMe stuff that could be O(terrible).
63 2016-12-14 09:33:31 0|sipa|morcos: so, initial report (see https://github.com/sipa/bitcoin/commits/pertxoutcache)... gmaxwell benchmarked it, and it seems to make normal operation somewhat faster, but the cache memory usage larger resulting in more frequent flushes, and the flushes themselves are maybe 3x slower
64 2016-12-14 09:33:49 0|sipa|so for large dbcache it seems to be a win, while for low dbcache sizes it clearly is not
65 2016-12-14 09:38:52 0|gmaxwell|basically for largeish dbcache it looked ~15% faster than master until the moment where it flushed, then it took 2.5x longer... the cache also used about 20% more memory, so 20% less cache in a given size... so flushing more often and the further it went the performance change tended to a large slowdown. :( The cache using more memory is perhaps something of an illusion of an effect, since a m
66 2016-12-14 09:38:57 0|gmaxwell|ajor motivator of the change was to eliminate the existance of modified entries in the cache, which would make a good replacement policy possible instead of dumbly dumping the whole thing.
67 2016-12-14 09:39:15 0|gmaxwell|so if the issue were only the 20% reduction in cache size, that much could easily be recovered by smarter cache handling that it makes possible.
68 2016-12-14 09:39:25 0|gmaxwell|The flushing slowdown though... :(
69 2016-12-14 09:50:15 0|wumpus|bah :(
70 2016-12-14 09:57:20 0|bitcoin-git|13bitcoin/06master 14a13fa4c 15Matt Corallo: Remove unused CDiskBlockPos* argument from ProcessNewBlock
71 2016-12-14 09:57:20 0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/57e337d40e94...b68685a16a81
72 2016-12-14 09:57:21 0|bitcoin-git|13bitcoin/06master 14b68685a 15Wladimir J. van der Laan: Merge #9273: Remove unused CDiskBlockPos* argument from ProcessNewBlock...
73 2016-12-14 09:57:37 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9273: Remove unused CDiskBlockPos* argument from ProcessNewBlock (06master...062016-12-remove-unused-pnb-param) 02 https://github.com/bitcoin/bitcoin/pull/9273
74 2016-12-14 12:04:37 0|bitcoin-git|[13bitcoin] 15MarcoFalke opened pull request #9347: [0.13.2] wallet backports (060.13...06Mf1612-013back) 02 https://github.com/bitcoin/bitcoin/pull/9347
75 2016-12-14 12:12:37 0|MarcoFalke|I assume #9262 is no longer for backport?
76 2016-12-14 12:12:39 0|gribble|https://github.com/bitcoin/bitcoin/issues/9262 | Prefer coins that have fewer ancestors, sanity check txn before ATMP by instagibbs ÷ Pull Request #9262 ÷ bitcoin/bitcoin ÷ GitHub
77 2016-12-14 12:12:56 0|MarcoFalke|We already return the txid and try to reaccept to the mempool
78 2016-12-14 12:13:11 0|MarcoFalke|Also, the default of 9262 is off, and who changes defaults anyway?
79 2016-12-14 13:18:27 0|instagibbs|MarcoFalke: only the rejecting of transactions us off by default
80 2016-12-14 13:18:46 0|instagibbs|That said I don't think it's necessary for backport
81 2016-12-14 14:45:35 0|morcos|I don't think we should discuss 9262 for backport until after its merged... if it's too late then, it's too late.
82 2016-12-14 14:46:11 0|morcos|if it works right, its a less disruptive change for backport... but, we don't want to take chances on the "if"
83 2016-12-14 16:34:03 0|morcos|sigh... wumpus: i think i messed up with #9240 we might want to revert it
84 2016-12-14 16:34:05 0|gribble|https://github.com/bitcoin/bitcoin/issues/9240 | Remove txConflicted by morcos ÷ Pull Request #9240 ÷ bitcoin/bitcoin ÷ GitHub
85 2016-12-14 16:34:43 0|morcos|I didn't properly think through notifications that would no longer happen because SyncTransaction was no longer being called
86 2016-12-14 16:35:31 0|sipa|where else do we need notifications for conflicts?
87 2016-12-14 16:36:04 0|morcos|I think there are 3 things that won't happen 1)zmqnotifications (but we dont' do those for other things that leave the mempool, so maybe doesn't matter)
88 2016-12-14 16:36:29 0|morcos|2) UI notifications or whatever NotifyTransactionChanged does (haven't dived into this yet, but I also think this doesn't matter)
89 2016-12-14 16:36:36 0|morcos|3) -walletnotify notifications
90 2016-12-14 16:37:00 0|morcos|3) is tricky, b/c it might make more sense to call those from MarkConflicted in the wallet except for the case of someone paying you
91 2016-12-14 16:37:15 0|sipa|but what we need for all of those is not conflicts
92 2016-12-14 16:37:27 0|sipa|maybe we need a class of "reorged" transactions
93 2016-12-14 16:37:32 0|morcos|someone paying you is not something you would identify in wallet conflict code as now being conflicted, but something you might want to know about in walletnotify
94 2016-12-14 16:37:44 0|morcos|whats the definition of that if not conflict
95 2016-12-14 16:38:12 0|wumpus|morcos: well that can happen, at least it's not one of the backported ones :)
96 2016-12-14 16:38:28 0|sipa|reorged is previously confirmed transactions that are now no longer confirmed
97 2016-12-14 16:38:41 0|morcos|sipa: you still are notified on those
98 2016-12-14 16:38:44 0|sipa|conflicted is previously unconfirmed transactions that are now inconsistent with the tip
99 2016-12-14 16:39:04 0|morcos|this is just mempool transactions that are now conflicted (actually nothing to do with a reorg, just a new block that conflicts with it)
100 2016-12-14 16:39:11 0|morcos|yeah sorry
101 2016-12-14 16:39:28 0|sipa|yes, i'm questioning whether we need that
102 2016-12-14 16:39:37 0|sipa|their confirmation count is not changing
103 2016-12-14 16:40:46 0|sipa|oh, maybe we need it for wiping a cached balance?
104 2016-12-14 16:41:02 0|morcos|i think it might be nicer to just have -walletnotify for any wallet transactions that left the mempool... then it might not be necessary to distinguish if you can mark it conflicted or not
105 2016-12-14 16:41:07 0|morcos|but thats difficult
106 2016-12-14 16:42:38 0|morcos|sipa: i don't think so... it doesn't add to your credit if its not from you, and if it spends your money, then well... i suppose it could be a mixed debit tx that you don't identify... but imo thats in the class of things we could never fully get anyway
107 2016-12-14 16:42:58 0|morcos|i think my thinking was just about our own wallet code, which i think is still ok... its the external notifications i hadn't considered
108 2016-12-14 16:43:24 0|sipa|i didn't even know we sent notifications for left-mempool
109 2016-12-14 16:43:38 0|morcos|we don't we send them for SyncTransaction
110 2016-12-14 16:43:50 0|sipa|?
111 2016-12-14 16:43:55 0|morcos|which we were calling for things that left the mempool via conflict
112 2016-12-14 16:44:12 0|morcos|we send them from SyncTransaction for anything that is InvolvingMe
113 2016-12-14 16:44:34 0|morcos|SyncTransaction -> AddToWalletIfInvolvingMe -> AddToWallet
114 2016-12-14 16:44:43 0|morcos|AddToWallet calls -walletnotify
115 2016-12-14 16:47:18 0|morcos|I was saying if you had a generic way of calling SyncTransaction for everything that left the mempool (for reasons other than being included in a block which is already covered) then that might be the most logical thing...
116 2016-12-14 16:47:51 0|morcos|And I don't think it would be all that important to distinguish the reason it left, at least our own wallet isn't smart about that.
117 2016-12-14 16:48:06 0|morcos|But that is not easy to do I don't think.
118 2016-12-14 16:48:59 0|sipa|hmm, we could have a callback installed in the mempool
119 2016-12-14 16:49:16 0|morcos|So maybe its important to still call SyncTransaction for txConflicted (the mempool conflict detection i removed) just so we get the -walletnotify for payments to us that have now been double spent / replaced / conflicted, what have you
120 2016-12-14 16:50:24 0|sipa|ok
121 2016-12-14 16:51:37 0|morcos|believe me i don't like txConflicted.. but i'm worried about accidentally making a change that somebody might be relying on for zero-conf doublespend detection
122 2016-12-14 16:52:23 0|sipa|so you'll revert 9240 entirely, or keep txConflicts but remove its use in wallet conflicts checking
123 2016-12-14 16:53:21 0|morcos|sipa: i think thats the same thing. it wasn't doing anything in wallet conflicts checking. so reverting 9240 entirely will not affect our wallet.
124 2016-12-14 16:54:08 0|morcos|i think we should either revert it, or decide that there is a not that complicated better solution.
125 2016-12-14 16:55:40 0|sipa|i see
126 2016-12-14 16:56:10 0|morcos|perhaps its worth spending an hour thinking about whether SyncTransaction called from mempool.remove is the right thing to do regardless.. if not.. then revert 9240, if so, then do that before 0.14.
127 2016-12-14 16:56:31 0|sdaftuar|i'd be hesitant to make changes without thinking through a clear API for how all these callbacks ought to work, and documenting that
128 2016-12-14 18:16:24 0|bitcoin-git|[13bitcoin] 15sipa opened pull request #9349: Make CScript (and prevector) c++11 movable. (06master...06movescript) 02 https://github.com/bitcoin/bitcoin/pull/9349
129 2016-12-14 18:36:21 0|bitcoin-git|[13bitcoin] 15Christewart opened pull request #9350: Adding label for amount inside of tx_valid/tx_invalid.json (06master...06master) 02 https://github.com/bitcoin/bitcoin/pull/9350
130 2016-12-14 18:56:52 0|morcos|sipa: wumpus: It turned out to be quite easy to track all mempool removals and call SyncTransaction on them. However there are some tricky issues around edge cases involving reorgs where you might call the Sync for its removal after the Sync for it entering a block.
131 2016-12-14 18:57:09 0|morcos|I don't believe this actually causes any issues.. but it'll take some time to think through it carefully
132 2016-12-14 18:58:00 0|morcos|It would be my preference to do that, and try to document this code better.. As I think calling SyncTransaction (and thus walletnotify) on things that leave the mempool makes sense
133 2016-12-14 18:58:17 0|morcos|And I think reverting 9240 is a step backwards towards cleaning this stuff up
134 2016-12-14 18:58:43 0|morcos|But it depends on how risk averse you are for leaving it in its current state for a little while.
135 2016-12-14 18:59:24 0|morcos|To summarize, I think the only regression is that -walletnotify isn't fired on transactions that were in the mempool and then evicted due to a tx in a new block conflicting them
136 2016-12-14 18:59:35 0|morcos|i've got to run for the day shortly..
137 2016-12-14 21:33:41 0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #9351: Remove CWalletTx::fFromMe member. (06master...06pr/atw-fromme) 02 https://github.com/bitcoin/bitcoin/pull/9351
138 2016-12-14 22:12:06 0|bitcoin-git|[13bitcoin] 15sdaftuar opened pull request #9352: Attempt reconstruction from all compact block announcements (06master...06optimistic-cb-2) 02 https://github.com/bitcoin/bitcoin/pull/9352