1 2018-05-06 05:49:26 0|fanquake|Anyone want to throw an ACK onto #13093, pretty trivial backport to get 0.15 Qt depends building again on macOS
2 2018-05-06 05:49:28 0|gribble|https://github.com/bitcoin/bitcoin/issues/13093 | [0.15] backport: depends qt patches by fanquake ÷ Pull Request #13093 ÷ bitcoin/bitcoin ÷ GitHub
3 2018-05-06 09:36:02 0|murrayn|utACK, I'm not able to test it.
4 2018-05-06 09:53:32 0|promag|#13160 <- is this considered bug fix? also backport? needs release note?
5 2018-05-06 09:53:34 0|gribble|https://github.com/bitcoin/bitcoin/issues/13160 | Unlock spent outputs by promag ÷ Pull Request #13160 ÷ bitcoin/bitcoin ÷ GitHub
6 2018-05-06 12:15:27 0|bitcoin-git|[13bitcoin] 15martinus opened pull request #13176: Improve CRollingBloomFilter performance: replace modulus with FastMod (06master...06optimize-CRollingBloomFilter) 02 https://github.com/bitcoin/bitcoin/pull/13176
7 2018-05-06 14:30:27 0|booyah|-Copyright (C) 2009-2017 The Bitcoin developers
8 2018-05-06 14:30:28 0|booyah|-Copyright (C) 2009-2017 The Bitcoin Core developers
9 2018-05-06 14:30:30 0|booyah|+Copyright (C) 2009-2018 The Bitcoin developers
10 2018-05-06 14:30:39 0|booyah|just fyi. in other project. in https://github.com/Bitcoin-ABC/bitcoin-abc/commit/d29d325e97eeff3feac456d1014e00d5d697fe9f#diff-9e9f66b5b2833c022b0408b81a17cfd9R516
11 2018-05-06 15:15:14 0|jojeyh|am i correct in assuming that the "bestblock" field in the output from `bitcoin-cli gettxout` is the block hash of the block that includes the "txid" passed into the argument ?
12 2018-05-06 15:15:32 0|jojeyh|if i am, i think something is wrong
13 2018-05-06 15:18:17 0|jojeyh|the block hash printed out does not match the block where the txid is
14 2018-05-06 15:59:31 0|provoostenator|fanquake: done. You may return the favor by testing #12404 :-)
15 2018-05-06 15:59:33 0|gribble|https://github.com/bitcoin/bitcoin/issues/12404 | Prune more aggressively during IBD by Sjors ÷ Pull Request #12404 ÷ bitcoin/bitcoin ÷ GitHub
16 2018-05-06 16:00:41 0|provoostenator|I also got the bright idea of making my ccache larger (~/.ccache/ccache.conf) and moving it to my SSD drive...
17 2018-05-06 16:08:27 0|bitcoin-git|[13bitcoin] 15ken2812221 opened pull request #13177: GCC-7 and glibc-2.27 compat code (06master...06compat) 02 https://github.com/bitcoin/bitcoin/pull/13177
18 2018-05-06 19:14:07 0|bitcoin-git|[13bitcoin] 15donaloconnor opened pull request #13180: Fix re-declared scoped enum as unscoped (Causes issues with some compilers) (06master...0606052018_unscoped_enum) 02 https://github.com/bitcoin/bitcoin/pull/13180
19 2018-05-06 19:32:03 0|provoostenator|Am I correct that during IBD the bulk of random read/write disk activity is for /blocks/index and /chainstate? Whereas /blocks gets written in larger batches?
20 2018-05-06 19:34:15 0|sipa|eh
21 2018-05-06 19:34:31 0|provoostenator|Totally wrong?
22 2018-05-06 19:34:40 0|sipa|blocks are written to more frequently (every block), where index/chainstate only when the cache fills up
23 2018-05-06 19:35:02 0|sipa|blocks and index are generally never read
24 2018-05-06 19:35:26 0|sipa|(blocks are read for rescan and serving to peers; index is only read at startup)
25 2018-05-06 19:35:51 0|sipa|chainstate is read/write and sorta random access, due to leveldb most of the writes are sequential
26 2018-05-06 19:37:35 0|provoostenator|For context, I'm trying to figure what the ultimate IBD EC2 instance would look like, in order to produce a small pruned result, which I would then copy to a low-end machine. Particularly what needs a RAM drive.
27 2018-05-06 19:39:19 0|sipa|much RAM
28 2018-05-06 19:39:39 0|sipa|run with large dbcache
29 2018-05-06 19:40:09 0|sipa|if your dbcache is 6G or so, there are no ransom access reads or weites at all
30 2018-05-06 19:40:43 0|provoostenator|The x1e.2xlarge has 244 GB RAM, so I can store the whole chain. But if > 6 GB has extremely dimishing returns I'd rather get more CPU's probably.
31 2018-05-06 19:42:46 0|sipa|yes
32 2018-05-06 19:42:51 0|sipa|get 8G or so
33 2018-05-06 19:43:18 0|sipa|it's not accessed during sync
34 2018-05-06 19:43:18 0|sipa|storing the chain itself in RAM is useless
35 2018-05-06 19:47:29 0|provoostenator|Has anyone ever plotted IBD vs. number of CPU's? I assume that flattens off at some point. EC2 can do up 72 CPUs :-)
36 2018-05-06 19:48:02 0|sipa|bitcoin core won't even use more than 16
37 2018-05-06 19:48:11 0|sipa|lock contention because worse than the gains
38 2018-05-06 19:48:57 0|sipa|i doubt it matters much if you use more than 8
39 2018-05-06 19:56:17 0|provoostenator|And it doesn't use GPU, right?
40 2018-05-06 19:56:46 0|sipa|nope
41 2018-05-06 19:59:41 0|provoostenator|But a local SSD drive might? Alright, I picked four machine with 8 CPU's each and some differences otherwise: t2.2xlarge, m5.2xlarge, c5.2xlarge and i3.2xlarge. I'll let them race eachother tonight...
42 2018-05-06 19:59:59 0|sipa|if your dbcache is big enough, disk doesn't matter at all
43 2018-05-06 20:00:24 0|sipa|you're configuring dbcache as large as possible on each?
44 2018-05-06 20:00:43 0|provoostenator|Oh ok, I'll stick to the first three then (with massive dbcache).
45 2018-05-06 20:00:46 0|sipa|it won't automatically use more than 450M unless instructed
46 2018-05-06 20:01:08 0|sipa|numbers would be interesting, though
47 2018-05-06 20:01:28 0|provoostenator|Any branch other than master you're interested in?
48 2018-05-06 20:02:58 0|sipa|if dbcache is not sufficiently large to fit the entire sync in ram, random access read speed of disk does start mattering
49 2018-05-06 20:03:30 0|provoostenator|These machines all have about 60 GB of RAM, so I'll set -dbcache=50000
50 2018-05-06 20:09:13 0|provoostenator|When I did my benchmarking last year I never ticked the "allow burst above baseline" options, so I probably got nowhere near max CPU performance.
51 2018-05-06 20:09:29 0|gmaxwell|provoostenator: EC2 has terrible terrible terrible IO latency in general.
52 2018-05-06 20:09:51 0|gmaxwell|and it slaughters performance unless your dbcache is big enough for the whole UTXO set (perhaps around 6GB or so)
53 2018-05-06 20:09:59 0|gmaxwell|(similar for most other VPS providers)
54 2018-05-06 20:10:02 0|provoostenator|Amazon seems to assume that most applications only need short CPU burst and so they make you "save" credits for these incidental burst. But obviously IBD is bit more intense.
55 2018-05-06 20:11:25 0|provoostenator|I used -dbcache=8000 back then https://medium.com/provoost-on-crypto/historical-bitcoin-core-client-performance-c5f16e1f8ccb
56 2018-05-06 20:11:50 0|sipa|well verify that it isn't actually flushing
57 2018-05-06 20:12:13 0|provoostenator|How do I reduce the log output to only show flushes?
58 2018-05-06 20:12:21 0|sipa|(you'll see the cache size field in the UodateTip line drop to ~0)
59 2018-05-06 20:12:25 0|gmaxwell|you can tell from the updatetip lies.
60 2018-05-06 20:12:30 0|gmaxwell|er lines
61 2018-05-06 20:12:38 0|provoostenator|:-)
62 2018-05-06 20:15:02 0|sipa|(sorrry for typos, in a plane, bad lag)
63 2018-05-06 20:36:58 0|provoostenator|Clock started at 20:30 UTC. The t2.2xlarge just crossed the 200K block line and is slightly ahead of the other two.
64 2018-05-06 20:37:43 0|spudowiar|provoostenator: Do one with Bitcoin-ABC :P
65 2018-05-06 20:38:01 0|spudowiar|I would say do Ethereum but... :)
66 2018-05-06 20:40:13 0|provoostenator|Someone else is welcome to do that. Apparantly Etherehum is cloud resistant, according to a recent Noded podcast with Jameson Lopp. Due to relatively heavy I/O, if you fully validate.
67 2018-05-06 20:41:05 0|spudowiar|Apparently Ethereum is validation-resistant
68 2018-05-06 20:41:20 0|spudowiar|I wonder how Anduck's sync is getting on
69 2018-05-06 20:41:36 0|sipa|he gave up afaik, but this is getting pretty off topic
70 2018-05-06 20:42:10 0|provoostenator|Indeed, I'll post the result tomorrow morning.
71 2018-05-06 20:44:56 0|spudowiar|provoostenator: There's no networking difference between instances, is there?
72 2018-05-06 20:46:54 0|provoostenator|I don't think networking is a bottleneck with these machines, but it could explain small differences in the beginning. Things don't get interesting until ~2015 when the UTXO set grows.
73 2018-05-06 20:55:57 0|Anduck|spudowiar: my machine ran out of space for non-pruned eth sync, got frustrated and haven't since done it again. it synced at some pace (my estimation is 4-5 weeks for full sync) with 3x ssd in raid0, with hdd would be impossible.
74 2018-05-06 20:56:44 0|spudowiar|nice
75 2018-05-06 20:56:45 0|Anduck|this is the 2nd test. initial one i gave up as it was clearly impossible to never get it synced
76 2018-05-06 21:01:31 0|jojeyh|i had to stop running an eth node cuz it made my laptop completely unusable
77 2018-05-06 21:36:19 0|phantomcircuit|provoostenator, you cant actually fully validate ethereum anymore
78 2018-05-06 21:36:38 0|phantomcircuit|there's a stretch of blocks which are effectively impossible to validate on anything
79 2018-05-06 21:36:54 0|phantomcircuit|besides they just make up the rules randomly anyways
80 2018-05-06 21:39:04 0|provoostenator|phantomcircuit: let's not go too far off topic. I'll post the result here, but unless there's a PR that can improve it, this is probably not the right place.
81 2018-05-06 21:42:53 0|provoostenator|Just created #blockchain-sync, let's move there.
82 2018-05-06 21:49:14 0|bitcoin-git|[13bitcoin] 15donaloconnor closed pull request #13180: Fix re-declared scoped enum as unscoped (Causes issues with some compilers) (06master...0606052018_unscoped_enum) 02 https://github.com/bitcoin/bitcoin/pull/13180
83 2018-05-06 21:51:59 0|bitcoin-git|[13bitcoin] 15donaloconnor reopened pull request #13180: Fix re-declared scoped enum as unscoped (Causes issues with some compilers) (06master...0606052018_unscoped_enum) 02 https://github.com/bitcoin/bitcoin/pull/13180