1 2017-03-31 00:40:52 0|gmaxwell|the leveldb logging with debug=1 is obnoxious, now I see why jnewbery wanted the exclude.
2 2017-03-31 02:32:01 0|gmaxwell|wumpus: uh. so boolean operations don't appear to exist for scoped enums. so ... I don't think I can use them without defining a set of boolean operations for them.
3 2017-03-31 02:32:30 0|sipa|gmaxwell: eh?
4 2017-03-31 02:33:56 0|sipa|ah, scoped enums don't have an implicit conversion to int
5 2017-03-31 02:34:03 0|sipa|you can still explicitly convert them, though
6 2017-03-31 02:34:25 0|sipa|but enums are generally a bad approach for bitfields, as they're intended to be exhaustive
7 2017-03-31 02:35:00 0|gmaxwell|sipa: wumpus suggested that I change the log print thing to use a scoped enum. The original thing that inspired this change is that rebroad wanted to do some horrible hack to do the logical equivilent of LogPrint(LOG_NET | LOG_MEMPOOL, ...). (and of course the matching also uses it as a bitfield.)
8 2017-03-31 02:35:30 0|sipa|right, a normal enum (which behaves much more like an int) may be more appropriate
9 2017-03-31 02:35:38 0|gmaxwell|well thats what I implemented.
10 2017-03-31 02:37:11 0|sipa|awesöme
11 2017-03-31 02:37:49 0|gmaxwell|in any case. I'm happy to change it to whatever. someone who has an opinion needs to tell me what to do. I think I only need a couple casts to make the enum class stuff go, but it would be ugly to do "LogPrint(LOG_NET | LOG_MEMPOOL," later unless we provide | for the type.
12 2017-03-31 07:29:04 0|bitcoin-git|13bitcoin/06master 14c0651cc 15Kyle Honeycutt: Update bitcoin.conf with example for pruning...
13 2017-03-31 07:29:04 0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/edc62c959ab3...8e4f7e72410d
14 2017-03-31 07:29:05 0|bitcoin-git|13bitcoin/06master 148e4f7e7 15MarcoFalke: Merge #10090: Update bitcoin.conf with example for pruning...
15 2017-03-31 07:29:25 0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #10090: Update bitcoin.conf with example for pruning (06master...06patch-1) 02 https://github.com/bitcoin/bitcoin/pull/10090
16 2017-03-31 08:22:04 0|bitcoin-git|[13bitcoin] 15laanwj pushed 4 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/8e4f7e72410d...de301b048810
17 2017-03-31 08:22:05 0|bitcoin-git|13bitcoin/06master 1421704f6 15John Newbery: Check stderr when testing bitcoin-tx
18 2017-03-31 08:22:05 0|bitcoin-git|13bitcoin/06master 14eb66bf9 15Awemany: bitcoin-tx: Fix missing range check...
19 2017-03-31 08:22:06 0|bitcoin-git|13bitcoin/06master 1419ecd1e 15John Newbery: Add tests for bitcoin-tx input checking
20 2017-03-31 08:22:26 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10130: bitcoin-tx input verification (awemany, jnewbery) (06master...06bitcoin_tx_input_sanitiser) 02 https://github.com/bitcoin/bitcoin/pull/10130
21 2017-03-31 09:31:21 0|bitcoin-git|13bitcoin/06master 14625488a 15Wladimir J. van der Laan: util: Work around (virtual) memory exhaustion on 32-bit w/ glibc...
22 2017-03-31 09:31:21 0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/de301b048810...ba12b3a84469
23 2017-03-31 09:31:22 0|bitcoin-git|13bitcoin/06master 14ba12b3a 15Wladimir J. van der Laan: Merge #10120: util: Work around (virtual) memory exhaustion on 32-bit w/ glibc...
24 2017-03-31 09:31:45 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10120: util: Work around (virtual) memory exhaustion on 32-bit w/ glibc (06master...062017_03_address_space_exhaustion_workaround) 02 https://github.com/bitcoin/bitcoin/pull/10120
25 2017-03-31 09:33:46 0|sipa|wumpus: have we ever done benchmarks for the performamce impact of having only 1 (or 2) arenas?
26 2017-03-31 09:34:26 0|wumpus|sipa: afaik, no. I've only done that on systems that would crash otherwise
27 2017-03-31 09:34:31 0|wumpus|so no way to compare :)
28 2017-03-31 09:34:55 0|sipa|infinite speedup, on those systems!
29 2017-03-31 09:35:00 0|wumpus|exactly!
30 2017-03-31 09:35:06 0|gmaxwell|for 32bit I don't think we really have a choice, in any case.
31 2017-03-31 09:35:54 0|wumpus|yes. the question is do we also want to customize arenas on 64-bit
32 2017-03-31 09:37:03 0|gmaxwell|sipa pointed out today that we really need to test that with network syncs since a chainstate reindex isn't really multithreaded in terms of malloc activity.
33 2017-03-31 09:37:17 0|wumpus|though I don't think so, I think on the longer run we should address leveldb's mismanagement of memory instead, as it's the largest reason we can't get along with many arenas
34 2017-03-31 09:37:45 0|gmaxwell|Also I made the observation that the mempool on disk likely increased our memory usage in the presence of multiple areans, because instead of only having two threads with substantial allocations it resulted in three.
35 2017-03-31 09:38:06 0|wumpus|yes
36 2017-03-31 09:38:47 0|gmaxwell|earlier I proposed a new database durability strategy to pieter that, if it works, will let us radically reduce peak memory usage.
37 2017-03-31 09:39:03 0|wumpus|and yes it may very well be that arenas doesn't help at all for our use case, but yes, profiling would make sense
38 2017-03-31 09:39:05 0|gmaxwell|(by elimiating large batch writes)
39 2017-03-31 09:40:11 0|wumpus|the glibc allocator also has some other 'tunables' that may have an effect on performance
40 2017-03-31 09:40:35 0|wumpus|gmaxwell: ok great!
41 2017-03-31 10:04:16 0|bitcoin-git|[13bitcoin] 15laanwj pushed 4 new commits to 060.14: 02 https://github.com/bitcoin/bitcoin/compare/b768202695a0...e4c9a90fe9aa
42 2017-03-31 10:04:17 0|bitcoin-git|13bitcoin/060.14 1404c21af 15Awemany: bitcoin-tx: Fix missing range check...
43 2017-03-31 10:04:17 0|bitcoin-git|13bitcoin/060.14 14ecc5232 15John Newbery: Check stderr when testing bitcoin-tx...
44 2017-03-31 10:04:18 0|bitcoin-git|13bitcoin/060.14 14f59e3df 15John Newbery: Add tests for bitcoin-tx input checking...
45 2017-03-31 10:06:37 0|bitcoin-git|13bitcoin/06master 147228ce8 15Pieter Wuille: Compensate for memory peak at flush time
46 2017-03-31 10:06:37 0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/ba12b3a84469...9ab26bf7e428
47 2017-03-31 10:06:38 0|bitcoin-git|13bitcoin/06master 149ab26bf 15Wladimir J. van der Laan: Merge #10126: Compensate for memory peak at flush time...
48 2017-03-31 10:07:00 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10126: Compensate for memory peak at flush time (06master...06peak_compensation) 02 https://github.com/bitcoin/bitcoin/pull/10126
49 2017-03-31 10:07:17 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10127: [0.14 backport] Mining: Prevent slowdown in CreateNewBlock on large mempools (060.14...062017-03-backport-cnb-optimizations) 02 https://github.com/bitcoin/bitcoin/pull/10127
50 2017-03-31 10:08:09 0|bitcoin-git|[13bitcoin] 15laanwj pushed 1 new commit to 060.14: 02 https://github.com/bitcoin/bitcoin/commit/0b5e162b84ec95cb63903dd21e4003a3d4503421
51 2017-03-31 10:08:10 0|bitcoin-git|13bitcoin/060.14 140b5e162 15Pieter Wuille: Compensate for memory peak at flush time...
52 2017-03-31 10:12:01 0|wumpus|last call for 0.14.1
53 2017-03-31 10:58:03 0|bitcoin-git|[13bitcoin] 15laanwj pushed 4 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/9ab26bf7e428...4aa07fa73569
54 2017-03-31 10:58:04 0|bitcoin-git|13bitcoin/06master 148d8f28d 15Wladimir J. van der Laan: refactor: Move RPCNotifyBlockChange out of `rpc/server.h`
55 2017-03-31 10:58:04 0|bitcoin-git|13bitcoin/06master 14e6dcfee 15Wladimir J. van der Laan: refactor: Move GetDifficulty out of `rpc/server.h`...
56 2017-03-31 10:58:05 0|bitcoin-git|13bitcoin/06master 14f885b67 15Wladimir J. van der Laan: refactor: Make rest.cpp dependency on `*toJSON` in `blockchain.cpp` explicit
57 2017-03-31 10:58:25 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10095: refactor: Move GetDifficulty out of `rpc/server.h` (06master...062017_03_getdifficulty_header) 02 https://github.com/bitcoin/bitcoin/pull/10095
58 2017-03-31 11:26:05 0|wumpus|will tag 0.14.1rc1 in a bit
59 2017-03-31 12:49:35 0|wumpus|never mind, won't tag 0.14.1rc1 today
60 2017-03-31 12:57:22 0|Victorsueca|wumpus: a wild bug appeared?
61 2017-03-31 13:03:58 0|wumpus|no
62 2017-03-31 13:04:46 0|wumpus|apparently #10126 reduces the dbache too much
63 2017-03-31 13:04:47 0|gribble|https://github.com/bitcoin/bitcoin/issues/10126 | Compensate for memory peak at flush time by sipa ÷ Pull Request #10126 ÷ bitcoin/bitcoin ÷ GitHub
64 2017-03-31 13:05:48 0|wumpus|not really a bug, I guess, but also not desirable for a release
65 2017-03-31 14:33:41 0|bitcoin-git|[13bitcoin] 15morcos opened pull request #10133: Clean up calculations of pcoinsTip memory usage (06master...06cachememory) 02 https://github.com/bitcoin/bitcoin/pull/10133
66 2017-03-31 14:34:12 0|morcos|wumpus: ok take a look at that. This would be my recommendation, but I broke it up into commits so we can talk about what makes sense for 0.14.1 and what doesn't....
67 2017-03-31 14:35:55 0|morcos|I care primarily about the first commit
68 2017-03-31 17:02:01 0|morcos|sipa: i was trying to reason about whether we need the 2x factor in the rollback test
69 2017-03-31 17:02:17 0|morcos|i went back and forth.
70 2017-03-31 17:02:37 0|morcos|is it true that non-dirty entries in the pcoinstip aren't subject to a 2x usage on flush?
71 2017-03-31 17:02:42 0|morcos|if so then i guess i agree...
72 2017-03-31 17:02:50 0|sipa|morcos: right
73 2017-03-31 17:03:01 0|morcos|but certainly if pcoinstip grows which it will in the rollback, and then it will be flushed later...
74 2017-03-31 17:03:28 0|sipa|are you sure? i thought we changed that test to not use the main cache
75 2017-03-31 17:03:49 0|morcos|pcoinstip will grow, but will only grow with non-dirty entries
76 2017-03-31 17:05:04 0|morcos|for instance if your tip is block 100, you flushed after block 99 and you are rolling back block 99, won't pcoinstip get the outputs created in block 99 pulled into it
77 2017-03-31 17:05:28 0|morcos|which will be marked dirty in coins, but never flushed to pcoinstip
78 2017-03-31 17:07:46 0|sipa|morcos: pcoinstip is not involved in the rollback test at all
79 2017-03-31 17:07:53 0|sipa|it uses a private cache on top of pcoinsdb
80 2017-03-31 17:08:03 0|morcos|oh... i missed that
81 2017-03-31 17:08:06 0|sipa|if (!CVerifyDB().VerifyDB(chainparams, pcoinsdbview, GetArg("-checklevel", DEFAULT_CHECKLEVEL),
82 2017-03-31 17:08:14 0|morcos|i assumed coins was built on top of pcoinstip
83 2017-03-31 17:08:35 0|sipa|it used to be, but it caused a memory blowup that quickly caused a flush afterwards anyway
84 2017-03-31 17:09:29 0|morcos|ok, then i agree completely unnecessary there... will remove
85 2017-03-31 17:10:04 0|morcos|while we're at it.. are you ok with the other changes.. i might as well get it to mergeable state.. i agree this is all relatively minor
86 2017-03-31 17:21:55 0|morcos|ugh.. wait on #10133..
87 2017-03-31 17:21:56 0|gribble|https://github.com/bitcoin/bitcoin/issues/10133 | Clean up calculations of pcoinsTip memory usage by morcos ÷ Pull Request #10133 ÷ bitcoin/bitcoin ÷ GitHub
88 2017-03-31 17:22:55 0|morcos|oh nm, maybe its fine.. i was just confusing myself with whether the 90% is still the right number, but you are comparing the doubled calculation with the 90% number, so it is fine
89 2017-03-31 17:27:12 0|bitcoin-git|[13bitcoin] 15sdaftuar opened pull request #10134: [qa] Fixes segwit block relay test after inv-direct-fetch was disabled (06master...062017-03-fix-segwit-relay-test) 02 https://github.com/bitcoin/bitcoin/pull/10134
90 2017-03-31 18:23:23 0|bitcoin-git|[13bitcoin] 15jnewbery opened pull request #10135: Send the correct error code in reject messages (06master...06correctrejectmessages) 02 https://github.com/bitcoin/bitcoin/pull/10135
91 2017-03-31 20:48:47 0|bitcoin-git|[13bitcoin] 15jnewbery closed pull request #10123: Allow debug logs to be excluded from specified component (06master...06debugexclude) 02 https://github.com/bitcoin/bitcoin/pull/10123