1 2017-04-17 01:06:22 0|bitcoin-git|[13bitcoin] 15jimmysong opened pull request #10219: Tests: Order Python Tests By Duration (06master...06order_tests_by_duration) 02 https://github.com/bitcoin/bitcoin/pull/10219
2 2017-04-17 06:31:17 0|wumpus|rc2 executables up https://bitcoin.org/bin/bitcoin-core-0.14.1/test.rc2/
3 2017-04-17 06:44:53 0|wumpus|don't think this will need a long testing time, given the minor changes relative to rc1
4 2017-04-17 08:08:36 0|wumpus|is it just me or is test_runner.py really slow now on master?
5 2017-04-17 08:09:27 0|wumpus|huh only happens on one vm, must be me
6 2017-04-17 08:19:23 0|wumpus|what I don't understand is that this is a much faster machine just much slower at running the tests
7 2017-04-17 08:20:42 0|sipa|slow i/9?
8 2017-04-17 08:20:50 0|sipa|i/o?
9 2017-04-17 08:24:46 0|MarcoFalke|we should really collect statistics about the tests such as run time and then plot them over time.
10 2017-04-17 08:25:09 0|MarcoFalke|I think we had a regression in the wallet once that made the wallet test run really slow...
11 2017-04-17 08:25:21 0|wumpus|fast/old computer: 3.98user 1.26system 0:33.90elapsed 15%CPU (0avgtext+0avgdata 57976maxresident)k 8inputs+42776outputs (0major+107146minor)pagefaults 0swaps
12 2017-04-17 08:25:34 0|wumpus|slow/new computer: 7.29user 2.48system 1:42.07elapsed 9%CPU (0avgtext+0avgdata 57904maxresident)k 0inputs+45776outputs (0major+108208minor)pagefaults 0swaps
13 2017-04-17 08:25:58 0|wumpus|(both for fundrawtransaction test)
14 2017-04-17 08:26:23 0|wumpus|to me it looks like it's just spending more time waiting
15 2017-04-17 08:27:42 0|MarcoFalke|tearing down the nodes takes ages for tiny tests compared to the test's actual run time
16 2017-04-17 08:29:55 0|gmaxwell|unfortunately with the test running on shared vm infrastructure timings are probably not all that useful.
17 2017-04-17 08:41:20 0|wumpus|it probably doesn't help rpc performance from python that authproxy calls log.debug for all data that comes in and goes out, pretty-printing everything even though usually it's discarded
18 2017-04-17 08:41:35 0|wumpus|(not likely the cause of my slowdown, just noticed)
19 2017-04-17 08:55:34 0|MarcoFalke|Indeed, no wall clock, but maybe cpu_time, memory_peak and io could help.
20 2017-04-17 08:55:50 0|wumpus|I'm on to something maybe, a getnewaddress call takes 0.013565 on the one system, 0.168683 (more than ten times as much) on the other. Could be slow i/o, but that slow?
21 2017-04-17 08:56:11 0|wumpus|this does not seem to extend to most other RPC calls (though havne't looked at them all)
22 2017-04-17 08:56:33 0|gmaxwell|well getnewaddress is syncing the wallet... so fsync time?
23 2017-04-17 08:56:40 0|sipa|we do a db sync operatiom after every new address
24 2017-04-17 08:56:43 0|sipa|jinx
25 2017-04-17 08:57:32 0|wumpus|if so we need a flag to disable that for the tests
26 2017-04-17 08:58:29 0|wumpus|fsync slow makes sense, I've noticed that before, I think it tries to sync the entire partition image
27 2017-04-17 08:58:30 0|gmaxwell|there is that eatmydata thing that could be used with tests.
28 2017-04-17 08:59:15 0|wumpus|cool, didn't know about that one
29 2017-04-17 09:02:53 0|sipa|s/partition/filesystem
30 2017-04-17 09:03:16 0|sipa|i think? or is literally the disk block cache?
31 2017-04-17 09:03:36 0|wumpus|yes, filesystem
32 2017-04-17 09:04:36 0|wumpus|or not sure really
33 2017-04-17 09:04:56 0|wumpus|it might as well be trying to sync the entire virtual disk to disk
34 2017-04-17 09:06:17 0|sipa|but syncing of a filesystem needs dependency information between sectors, or you may end up with an inonsistent state
35 2017-04-17 09:06:37 0|wumpus|I suspect it's something like that at least: fsync() inside the VM has file granularity, but qemu calling fsync() has complete file system granularity
36 2017-04-17 09:06:39 0|sipa|so even if it's a disk level cache, it needs information from the filesystem to order the wrotes
37 2017-04-17 09:07:33 0|wumpus|so not only the wallet is fsynced, but also all the other things the tests do such as writing tons of log files
38 2017-04-17 09:07:57 0|wumpus|anyhow I'll try with the eatmydata and see if it resolves the slowdown
39 2017-04-17 09:10:56 0|wumpus|yes!
40 2017-04-17 09:11:32 0|wumpus|old computer: 3.73user 1.04system 0:39.09elapsed 12%CPU (0avgtext+0avgdata 61112maxresident)k 8inputs+9064outputs (0major+45873minor)pagefaults 0swaps
41 2017-04-17 09:11:44 0|wumpus|new computer: 3.24user 0.78system 0:33.70elapsed 11%CPU (0avgtext+0avgdata 57832maxresident)k 0inputs+10472outputs (0major+108842minor)pagefaults 0swaps
42 2017-04-17 09:12:14 0|wumpus|more than three times as fast as before
43 2017-04-17 09:12:39 0|wumpus|wonder if this will help travis too
44 2017-04-17 09:20:17 0|bitcoin-git|[13bitcoin] 15laanwj opened pull request #10220: Experiment: test: Disable fsync in travis tests (06master...062017_04_tests_eatmydata) 02 https://github.com/bitcoin/bitcoin/pull/10220
45 2017-04-17 11:18:46 0|wumpus|so apparently on travis, eatmydata gives a 2x gain (thanks for testing MarcoFalke), not as good as in my VM (I may have some misconfiguration) but still nice
46 2017-04-17 11:19:36 0|sipa|nice indeed!
47 2017-04-17 11:30:11 0|SopaXorzTaker|offtopic PSA
48 2017-04-17 11:30:24 0|SopaXorzTaker|wumpus, sipa, the large bitcoin collider client script is untrustworthy
49 2017-04-17 11:30:38 0|SopaXorzTaker|refrain from running it until the author gives explanations
50 2017-04-17 11:31:31 0|sipa|don't worry, i had no intention of running it
51 2017-04-17 11:35:51 0|SopaXorzTaker|PSA on reddit: https://www.reddit.com/r/Bitcoin/comments/65uoaq/do_not_run_the_large_bitcoin_collider_client_its/
52 2017-04-17 11:35:57 0|SopaXorzTaker|just FYI
53 2017-04-17 11:58:58 0|bitcoin-git|13bitcoin/06master 1445f0961 15Gregory Maxwell: Prevent integer overflow in ReadVarInt....
54 2017-04-17 11:58:58 0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/f4db00f9a548...c5e9e428a919
55 2017-04-17 11:58:59 0|bitcoin-git|13bitcoin/06master 14c5e9e42 15Pieter Wuille: Merge #9693: Prevent integer overflow in ReadVarInt....
56 2017-04-17 11:59:13 0|bitcoin-git|[13bitcoin] 15sipa closed pull request #9693: Prevent integer overflow in ReadVarInt. (06master...06varint_maxsize) 02 https://github.com/bitcoin/bitcoin/pull/9693
57 2017-04-17 12:16:00 0|bitcoin-git|13bitcoin/06master 14b2c9254 15Matt Corallo: Check interruptNet during dnsseed lookups
58 2017-04-17 12:16:00 0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/c5e9e428a919...a077a90da88f
59 2017-04-17 12:16:01 0|bitcoin-git|13bitcoin/06master 14a077a90 15Pieter Wuille: Merge #10215: Check interruptNet during dnsseed lookups...
60 2017-04-17 12:16:24 0|bitcoin-git|[13bitcoin] 15sipa closed pull request #10215: Check interruptNet during dnsseed lookups (06master...062017-04-dnsseed-break) 02 https://github.com/bitcoin/bitcoin/pull/10215
61 2017-04-17 12:46:44 0|bitcoin-git|[13bitcoin] 15laanwj pushed 3 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/a077a90da88f...2584925077f9
62 2017-04-17 12:46:45 0|bitcoin-git|13bitcoin/06master 149fececb 15Matt Corallo: Remove CValidationInterface::UpdatedTransaction...
63 2017-04-17 12:46:45 0|bitcoin-git|13bitcoin/06master 14d89f8ad 15Matt Corallo: Make DisconnectBlock and ConnectBlock static in validation.cpp
64 2017-04-17 12:46:46 0|bitcoin-git|13bitcoin/06master 142584925 15Wladimir J. van der Laan: Merge #10178: Remove CValidationInterface::UpdatedTransaction...
65 2017-04-17 12:47:02 0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #10178: Remove CValidationInterface::UpdatedTransaction (06master...062017-01-wallet-cache-inmempool-2) 02 https://github.com/bitcoin/bitcoin/pull/10178
66 2017-04-17 13:39:47 0|wumpus|SopaXorzTaker: good sleuthing, but no, you don't have to be afraid I run random scripts from the internet on anything important, let alone bitcoin-related ones
67 2017-04-17 13:50:06 0|SopaXorzTaker|wumpus, yeah
68 2017-04-17 13:50:18 0|SopaXorzTaker|but this script actually does remote code execution
69 2017-04-17 13:53:16 0|wumpus|the whole premise is a bit scammy; it reminds me of the trojans in the 90's whose control component had a trojan too. So everyone using the trojan to grief other people got owned themselves too...
70 2017-04-17 13:55:07 0|luke-jr|what is it even supposed to do?
71 2017-04-17 13:55:11 0|wumpus|people running this script try to steal coins by generating random private keys. This is incredibly unlikely, and if it worked it'd be wrong in various ways
72 2017-04-17 13:55:18 0|luke-jr|lol
73 2017-04-17 13:55:45 0|luke-jr|bruteforcing privkeys is just ridiculous to attempt, but to do it with *Perl code*? lololol
74 2017-04-17 13:56:35 0|luke-jr|at face value, it's obvious the only purpose is to be a backdoor
75 2017-04-17 13:56:41 0|SopaXorzTaker|luke-jr, well
76 2017-04-17 13:56:56 0|SopaXorzTaker|it's actually done with an inner C program which is natively compiled by the script
77 2017-04-17 13:57:09 0|SopaXorzTaker|actually, the bruteforcing has some results
78 2017-04-17 13:57:16 0|sipa|and it seems to have OpenCL code too
79 2017-04-17 13:57:21 0|SopaXorzTaker|there were some addresses deliberately generated with weak PRNGs
80 2017-04-17 14:01:53 0|wumpus|yes, if it mimimcs specific bad PRNGs (or bad brainwallets) instead of simply randomly generating keys it can certainly turn up something
81 2017-04-17 14:03:04 0|sipa|including things put there by the script's author :)
82 2017-04-17 14:03:42 0|SopaXorzTaker|wumpus, yes
83 2017-04-17 14:03:52 0|SopaXorzTaker|there is a so-called puzzle transaction
84 2017-04-17 14:03:55 0|SopaXorzTaker|with 32 BTC
85 2017-04-17 14:03:56 0|SopaXorzTaker|look it up
86 2017-04-17 14:04:11 0|SopaXorzTaker|each address uses a key one bit stronger than the previous one
87 2017-04-17 14:04:19 0|SopaXorzTaker|(there's 256 addresses)
88 2017-04-17 14:04:30 0|SopaXorzTaker|eg. 0000..0001
89 2017-04-17 14:04:37 0|SopaXorzTaker|0000..0011
90 2017-04-17 14:04:39 0|SopaXorzTaker|0000..0101
91 2017-04-17 14:04:44 0|SopaXorzTaker|0000..1011
92 2017-04-17 14:04:46 0|SopaXorzTaker|and so on
93 2017-04-17 14:06:46 0|wumpus|yes, for the author it could be very profitable, and no need to bruteforce at all, just stealing all the wallets of people running this
94 2017-04-17 14:08:16 0|wumpus|in a way it's the classical con, make people believe something that's too good to be true
95 2017-04-17 14:18:01 0|bitcoin-git|[13bitcoin] 15TheBlueMatt opened pull request #10221: Stop treating coinbase outputs differently in GUI: show them at 1conf (06master...062017-04-no-coinbase-display-lag) 02 https://github.com/bitcoin/bitcoin/pull/10221
96 2017-04-17 16:13:39 0|morcos|I'm happy enough to just exclude in all my bitcoin.conf files, but just want to see if everyone else is ok with the leveldb log spam... it prints a lot of useless messages right at startup (where everythign else you care about is printed)
97 2017-04-17 16:17:37 0|morcos|I'm not sure exactly what information we're expecting from the leveldb logging, so maybe there is a better solution that concentrates on that infomration or at least cleanly aggregating at startup.. it's not really clear to me how to make much use of what it does log
98 2017-04-17 16:20:02 0|sipa|only when you enable the relevant debug category?
99 2017-04-17 16:20:18 0|morcos|sipa: i assume, but my habit is to always enable all
100 2017-04-17 16:37:37 0|wumpus|the more debugging is added, the less useful it becomes to run with debug=all
101 2017-04-17 16:38:42 0|wumpus|though I'm fine with a debug=alllowvolume or such if you want to add that, which excludes at least leveldb and libevent
102 2017-04-17 16:40:35 0|wumpus|but usually the recommedation is to add debug categories only when troubleshooting a certain subsystem; this became even easier with the RPC call to turn on/off individual debug flags
103 2017-04-17 17:21:13 0|jtimon|is there any advantage to Q_FOREACH over c++11 foreach?
104 2017-04-17 17:25:31 0|wumpus|no
105 2017-04-17 17:26:38 0|wumpus|fairly sure c++11 foreach will work with qt objects too
106 2017-04-17 17:36:12 0|jtimon|wumpus: it seems performance can be worse in some cases: https://www.dvratil.cz/2015/06/qt-containers-and-c11-range-based-loops/
107 2017-04-17 17:37:08 0|jtimon|I mean, not in a way that it can't be solved
108 2017-04-17 17:43:35 0|sipa|hmm, seems to be about some specifics with Qt containers
109 2017-04-17 18:03:18 0|jtimon|the reason I ask is because I was trying to remove PAIRTYPE and it seems Q_FOREACH requires it too, I'm not completely sure though, I'm compiling removing Q_FOREACH first and then I'll try again without removing Q_FOREACH in case we prefer to keep it, but only -j4 since I'm on the laptop...
110 2017-04-17 18:17:46 0|bitcoin-git|[13bitcoin] 15jnewbery opened pull request #10222: [tests] test_runner - check unicode (06master...06test_runner_check_unicode) 02 https://github.com/bitcoin/bitcoin/pull/10222
111 2017-04-17 19:57:18 0|bitcoin-git|13bitcoin/06master 14a97ed80 15John Newbery: [tests] test_runner - check unicode
112 2017-04-17 19:57:18 0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/2584925077f9...6ce733747e16
113 2017-04-17 19:57:19 0|bitcoin-git|13bitcoin/06master 146ce7337 15MarcoFalke: Merge #10222: [tests] test_runner - check unicode...
114 2017-04-17 19:57:48 0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #10222: [tests] test_runner - check unicode (06master...06test_runner_check_unicode) 02 https://github.com/bitcoin/bitcoin/pull/10222
115 2017-04-17 20:07:10 0|BlueMatt|sipa: hey
116 2017-04-17 20:12:57 0|BlueMatt|re: #10148's Clear() in ApplyTxInUndo: hmm, you may be right, anyway, dear god this is not trivial to reason about :/
117 2017-04-17 20:12:59 0|gribble|https://github.com/bitcoin/bitcoin/issues/10148 | Use non-atomic flushing with block replay by sipa ÷ Pull Request #10148 ÷ bitcoin/bitcoin ÷ GitHub
118 2017-04-17 20:13:45 0|bitcoin-git|13bitcoin/06master 14c9e31c3 15Warren Togami: Clarify importprivkey help text with example of blank label without rescan...
119 2017-04-17 20:13:45 0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 2 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/6ce733747e16...50a1cc0f0aef
120 2017-04-17 20:13:46 0|bitcoin-git|13bitcoin/06master 1450a1cc0 15MarcoFalke: Merge #10207: Clarify importprivkey help text ... example of blank label without rescan...
121 2017-04-17 20:14:07 0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #10207: Clarify importprivkey help text ... example of blank label without rescan (06master...06importprivkey) 02 https://github.com/bitcoin/bitcoin/pull/10207
122 2017-04-17 20:20:28 0|bitcoin-git|[13bitcoin] 15MarcoFalke pushed 3 new commits to 06master: 02 https://github.com/bitcoin/bitcoin/compare/50a1cc0f0aef...d86bb075bf6d
123 2017-04-17 20:20:29 0|bitcoin-git|13bitcoin/06master 1408e51c1 15John Newbery: [tests] Remove cache directory by default when running test_runner
124 2017-04-17 20:20:29 0|bitcoin-git|13bitcoin/06master 14c85b080 15John Newbery: [test] add warnings to test_runner
125 2017-04-17 20:20:30 0|bitcoin-git|13bitcoin/06master 14d86bb07 15MarcoFalke: Merge #10197: [tests] Functional test warnings...
126 2017-04-17 20:20:48 0|bitcoin-git|[13bitcoin] 15MarcoFalke closed pull request #10197: [tests] Functional test warnings (06master...06functional_test_warnings) 02 https://github.com/bitcoin/bitcoin/pull/10197
127 2017-04-17 20:28:41 0|sipa|BlueMatt: i'm going to add a WIP to the title, i'm not confortable with merging until there are more substantial tests
128 2017-04-17 20:29:53 0|BlueMatt|yea, I was starting to feel the same way....I mean alternatively we could drop the multi-head support and only support single-action (ie a series of connects/disconnects) between full flushes, which would simplify things and go back to a bit more how it was until you rewrote a bunch
129 2017-04-17 20:29:57 0|BlueMatt|sipa: ^
130 2017-04-17 20:32:51 0|sipa|BlueMatt: i think it is fine to only test the single-head case for now
131 2017-04-17 20:33:11 0|sipa|at worst, the result is not backward compatible omce we need muktihead
132 2017-04-17 20:33:17 0|sipa|*multi once
133 2017-04-17 20:33:25 0|BlueMatt|sipa: well my point was just the implementation of the multi-head-handling case is complicated enough that it adds a ton of review burden
134 2017-04-17 20:33:33 0|BlueMatt|esp pre-utxo-db-format-change
135 2017-04-17 20:34:00 0|BlueMatt|may be easier to just do it single-head-only, then do utxo-db, then change to multi-head...if we break compat there its ok
136 2017-04-17 20:34:39 0|sipa|the multihead code and pertxout are orthogonal, i think
137 2017-04-17 20:35:38 0|BlueMatt|i havent dug too much into pertxout yet, but shouldnt it simplify things, or is there still a concept of per-tx CCoins everywhere above the db?
138 2017-04-17 20:36:07 0|BlueMatt|my assumption was the review for this would be much simpler if you dont have to think about making sure entire transaction objects are correct, instead of there just being add/remove-outputs
139 2017-04-17 20:36:38 0|BlueMatt|at least i found it much easier to review prior to the latest changes, even ignoring the handle-disconnect stuff
140 2017-04-17 20:36:40 0|gmaxwell|I think they turn out to be pretty much orthorgonal.
141 2017-04-17 20:37:23 0|gmaxwell|okay thats a point.
142 2017-04-17 20:43:59 0|sipa|BlueMatt: the Clean call is indeed a possible violation of that orthogonality...
143 2017-04-17 20:44:21 0|BlueMatt|that was my primary example, indeed
144 2017-04-17 20:46:10 0|sipa|i don't think there are any others, but it is a fair point that pertxout is breaking backward xcompatibility already, so perhaps attempting to already support multihead isn't actually worth it
145 2017-04-17 20:46:21 0|sipa|but i think the complexity is mostly in testing
146 2017-04-17 20:46:52 0|sipa|the difference in implementation between multihead and single head is just that loop and building of a set
147 2017-04-17 20:47:07 0|BlueMatt|ok, i found the building of a set hard to reason about :p
148 2017-04-17 20:47:36 0|sipa|fair enough, but you can reason about the cases that are relevant for single head?
149 2017-04-17 20:48:33 0|BlueMatt|probably? dunno, i was more tired today than previous days, so there may also be a skew there :p
150 2017-04-17 20:49:13 0|sipa|well, maybe it is best to explain the full algorithm and reasoning why it is correct in text in comments