1 2018-03-10 01:07:37	0|karelb|I am porting the autogenerated doc into bitcoincore.org website so I can make a PR and it looks fine :)
 2 2018-03-10 01:07:41	0|karelb|https://i.imgur.com/Qu42lTZ.png
 3 2018-03-10 01:08:06	0|karelb|well at least on my notebook
 4 2018-03-10 01:08:51	0|karelb|I hijacked the GUI you have for article TOC for also adding a menu
 5 2018-03-10 01:09:59	0|karelb|(that is where I found https://github.com/bitcoin-core/bitcoincore.org/issues/525 )
 6 2018-03-10 01:13:14	0|luke-jr|I wonder if the Doxygen stuff would be useful to publish
 7 2018-03-10 01:21:58	0|karelb|There is something here
 8 2018-03-10 01:21:59	0|karelb|https://dev.visucore.com/bitcoin/doxygen/struct_c_coin.html
 9 2018-03-10 01:22:35	0|karelb|and I don't think it's more useful than just looking at the source, to be frank
10 2018-03-10 01:32:17	0|bitcoin-git|[13bitcoin] 15sipa opened pull request #12658: Sanitize some wallet serialize (06master...06201803_crazywalletser) 02https://github.com/bitcoin/bitcoin/pull/12658
11 2018-03-10 02:40:07	0|bitcoin-git|[13bitcoin] 15eklitzke closed pull request #12491: Try to use posix_fadvise with CBufferedFile (06master...06fadvise) 02https://github.com/bitcoin/bitcoin/pull/12491
12 2018-03-10 03:54:05	0|pierre_rochard|karelb: I’ve found the doxygen call/caller graphs to be useful, but otherwise agree that source is more useful
13 2018-03-10 06:54:53	0|bitcoin-git|[13bitcoin] 15eklitzke opened pull request #12659: Improve Fatal LevelDB Log Messages (06master...06leveldb_errmsg) 02https://github.com/bitcoin/bitcoin/pull/12659
14 2018-03-10 06:55:22	0|armid|what plugin in the code make spacing in variable declaration ?
15 2018-03-10 06:55:25	0|armid|such that
16 2018-03-10 06:55:27	0|armid|SELECTCOINS = (1 << 10),
17 2018-03-10 06:55:32	0|armid|QT          = (1 << 19),
18 2018-03-10 06:57:37	0|sipa|what?
19 2018-03-10 06:57:55	0|sipa|bitcoin core doesn't have plugins
20 2018-03-10 06:58:56	0|armid|i mean IDE plugin
21 2018-03-10 06:58:58	0|armid|or something
22 2018-03-10 06:59:06	0|armid|vscode-plugin ?
23 2018-03-10 06:59:08	0|armid|that can format code
24 2018-03-10 06:59:32	0|sipa|you can use whatever you want
25 2018-03-10 06:59:42	0|sipa|i think most develooers just use text editors
26 2018-03-10 07:00:59	0|armid|^^
27 2018-03-10 07:16:26	0|mrannanay|ryanofsky: Thanks!
28 2018-03-10 16:15:29	0|bitcoin-git|13bitcoin/06master 14a7324bd 15practicalswift: Format timestamps using ISO 8601 formatting (e.g. "2018-02-28T12:34:56Z")...
29 2018-03-10 16:15:29	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/29fad97c320c...bb98aec6743e
30 2018-03-10 16:15:30	0|bitcoin-git|13bitcoin/06master 14bb98aec 15Wladimir J. van der Laan: Merge #12567: util: Print timestamp strings in logs using ISO 8601 formatting...
31 2018-03-10 16:16:26	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #12567: util: Print timestamp strings in logs using ISO 8601 formatting (06master...06iso-8601) 02https://github.com/bitcoin/bitcoin/pull/12567
32 2018-03-10 17:06:09	0|bitcoin-git|[13bitcoin] 15eklitzke opened pull request #12664: Rename LevelDB background compaction thread (06master...06bgcompact) 02https://github.com/bitcoin/bitcoin/pull/12664
33 2018-03-10 17:17:47	0|bitcoin-git|[13bitcoin] 15practicalswift opened pull request #12665: Add compile time checking for all run time locking assertions (06master...06compile-time-checking-of-runtime-assertions) 02https://github.com/bitcoin/bitcoin/pull/12665
34 2018-03-10 17:35:13	0|eklitzke|i was looking at malloc stats before/after flushing the dbcache, it looks like glibc malloc doesn't do so hot: https://gist.github.com/eklitzke/9cdcedaf5d15157f79e59128edd83234
35 2018-03-10 17:35:50	0|eklitzke|the memory use for heap 7 (where the loadblk thread is running) is basically the same before and after the 2GB flush
36 2018-03-10 17:38:50	0|sipa|eklitzke: how do i view that?
37 2018-03-10 17:39:46	0|eklitzke|bitcoin-cli getmemory mallocinfo
38 2018-03-10 17:42:31	0|sipa|o
39 2018-03-10 17:44:14	0|eklitzke|you can get global stats with mallinfo(3) but the xml thing seems to be the only way to get the stats broken down by arena (for bitcoin each thread effectively gets its own arena)
40 2018-03-10 17:46:03	0|gmaxwell|eklitzke: by do so hot, you're expecting usage to go down... I wouldn't expect it to due to fragmentation. the cache is kazillions of tiny sparse allocations that all happened at different times.
41 2018-03-10 17:46:55	0|eklitzke|it may not be fragmentation per se but it's not releasing the memory back to the os
42 2018-03-10 17:47:33	0|eklitzke|i was experimenting with trying to dynamically change the dbcache size and it was ineffecitve for this reason, even when using rehash etc.
43 2018-03-10 17:48:26	0|eklitzke|it would be nice if the dbcache could be given its own arena since it uses so much memory (on many configurations) and has a somewhat unique allocation pattern
44 2018-03-10 17:49:21	0|gmaxwell|Instead, I'd rather it didn't allocate at all.
45 2018-03-10 17:49:52	0|eklitzke|you mean like with a slab allocator?
46 2018-03-10 17:49:57	0|gmaxwell|what I'd previously propsosed is that we make it into an open hash table, with all data internal. (and an exception map for rare unusually large entries)
47 2018-03-10 17:50:14	0|eklitzke|i was thinking about that too
48 2018-03-10 17:50:18	0|gmaxwell|everything inline, so no pointer chasing in lookups.
49 2018-03-10 17:50:32	0|gmaxwell|(except for the exception entries)
50 2018-03-10 17:50:36	0|eklitzke|nearly everything in the cache is the same size, the data could fit very compactly
51 2018-03-10 17:50:43	0|gmaxwell|right.
52 2018-03-10 17:51:18	0|gmaxwell|just make every entry the same size, and the rare ones that don't fit, the storage is a union with a pointer, and a flag tells you to look in an map for the payload.
53 2018-03-10 17:51:47	0|gmaxwell|or something along those lines.
54 2018-03-10 17:51:53	0|eklitzke|makes sense
55 2018-03-10 17:52:21	0|sipa|but avoiding allocating inside the LevelDB batch creation would need serious refactoring
56 2018-03-10 17:52:52	0|eklitzke|leveldb isn't really optimized for the way bitcoin writes to it
57 2018-03-10 17:53:03	0|eklitzke|you could make it a lot more efficient
58 2018-03-10 17:53:32	0|gmaxwell|and entries could be never deleted from the open hash table, just flagged.. and make the insert routine understand that it can just write over the first non-dirty entry it encounters.
59 2018-03-10 17:54:35	0|eklitzke|it would require kind of serious surgery to the leveldb guts, but leveldb already represents the data in ordered ranges for the sstables, and the data to be flushed can be sorted, so i think you could redo the merge algorithm to zip through the on-disk data and the data being flushed and do the merge without doing a lot of allocatoins
60 2018-03-10 17:54:50	0|luke-jr|IIRC, at least at one point, Linux didn't *support* processes releasing memory back to the OS. But that may have changed (it's been years)
61 2018-03-10 17:55:14	0|eklitzke|linux has supported releasing memory back to the operating system for 20+ years if you munmap the data
62 2018-03-10 17:55:21	0|eklitzke|you can't do it with brk/sbrk though
63 2018-03-10 17:56:37	0|eklitzke|glibc mixes both which is kind of weird
64 2018-03-10 17:56:58	0|eklitzke|from what i can tell the main arena is allocated with sbrk and the other arenas are allocated with mmap
65 2018-03-10 17:57:31	0|sipa|iirc it's just a threshold
66 2018-03-10 17:57:45	0|sipa|memory ranges above a certain size are anonymous mmaps
67 2018-03-10 17:58:06	0|eklitzke|i think step 1 though would be to implement greg
68 2018-03-10 17:58:18	0|eklitzke|s idea, since that doesn't require messing with the allocator used by the rest of the code
69 2018-03-10 17:58:23	0|sipa|i believe greg is notoriously hard to implement
70 2018-03-10 17:58:30	0|sipa|ah.
71 2018-03-10 17:59:45	0|sipa|eklitzke: i have a flame graph!
72 2018-03-10 18:00:12	0|eklitzke|did you get it color coded?
73 2018-03-10 18:00:36	0|sipa|yes; it's inside libsecp256k1 though, in a microbenchmark
74 2018-03-10 18:00:47	0|eklitzke|awesome!
75 2018-03-10 18:01:52	0|sipa|http://bitcoin.sipa.be/flame.svg
76 2018-03-10 18:02:36	0|eklitzke|for micro benchmarks you should look at "perf stat" (man perf-stat)
77 2018-03-10 18:02:42	0|sipa|ah
78 2018-03-10 18:02:53	0|sipa|the same scripts apply?
79 2018-03-10 18:03:34	0|eklitzke|no it just gives you detailed information about things like cache hits/misses, cpu cycles, page faults, etc.
80 2018-03-10 18:04:38	0|eklitzke|so much gmpz realloc
81 2018-03-10 18:05:02	0|sipa|the problem i'm trying to investigate is why this call to GMP takes 600 cpu cycles when run inside the microbenchmark, but 6000 cpu cycles when called from a higher-level benchmark in core
82 2018-03-10 18:05:32	0|sipa|and at this point i have no better guess than cache effects... but i'm very surprised it would be that much difference
83 2018-03-10 18:08:09	0|eklitzke|one guess: in the benchmark there's one thread so everything is in the same core and numa memory zone, but in core the code has to access memory in another numa zone
84 2018-03-10 18:08:35	0|eklitzke|there's some stuff in numactl you could do try to force core to run all on one core to test that hypothesis
85 2018-03-10 18:08:56	0|sipa|the whole thing should be one thread even when called from bench_bitcoin
86 2018-03-10 18:09:51	0|eklitzke|try perf-stat -a --per-core
87 2018-03-10 21:48:15	0|jbnj|hi
88 2018-03-10 21:49:21	0|jbnj|hi everybody
89 2018-03-10 22:16:42	0|bitcoin-git|[13bitcoin] 15luke-jr opened pull request #12666: configure: UniValue 1.0.4 is required for pushKV(, bool) (06master...06univalue-1.0.4-required) 02https://github.com/bitcoin/bitcoin/pull/12666