1 2016-05-13 00:14:40	0|sipa|$ ./walletbackup.py
  2 2016-05-13 00:14:44	0|sipa|INFO: Restoring using dumped wallet
  3 2016-05-13 00:14:44	0|sipa|Unexpected exception caught during testing: BrokenPipeError(32, 'Broken pipe')
  4 2016-05-13 00:14:51	0|sipa|Stopping nodes
  5 2016-05-13 00:14:51	0|sipa|WARN: Unable to stop node: CannotSendRequest('Request-sent',)
  6 2016-05-13 00:15:32	0|gmaxwell|sipa: patrick got to you?
  7 2016-05-13 00:15:49	0|gmaxwell|the question is-- why doesn't travis see it as failing?
  8 2016-05-13 00:18:13	0|sipa|gmaxwell, wumpus, jonasschnelli:
  9 2016-05-13 00:19:07	0|gmaxwell|sipa, sipa, sipa
 10 2016-05-13 00:20:21	0|sipa|gmaxwell: are you wumpus and jonasschnelli?
 11 2016-05-13 00:20:44	0|sipa|i wanted to paste my rorx results, but they seem to have disappeared into my terminal's forgotten history
 12 2016-05-13 00:20:58	0|gmaxwell|No, are you?
 13 2016-05-13 00:21:10	0|sipa|both rorxes were similar, and fastest
 14 2016-05-13 00:21:19	0|gmaxwell|well, see.. I saved them seconds of disappointment by wisely preempting your summons.
 15 2016-05-13 00:23:55	0|gmaxwell|sipa: did you see the usenix paper I linked to you some indeterminable number of days ago?
 16 2016-05-13 00:24:18	0|sipa|i skimmed it, not enough to actually find the xor trick you were referring to
 17 2016-05-13 00:25:19	0|gmaxwell|are you familar with cuckoo hashing?
 18 2016-05-13 00:25:59	0|sipa|yes
 19 2016-05-13 00:27:02	0|gmaxwell|So how do you support cuckoo hashing if the whole value can't be stored in the table? -- normally you rehash an entry to find its alternative location when you evict it.
 20 2016-05-13 00:28:49	0|gmaxwell|The make the primary location   H(full value)->to offset and the secondary location H(short value) ^ location.  So you only need the current location and the short value to swap an entry between slots.
 21 2016-05-13 00:29:36	0|gmaxwell|(and you don't need to track which one it was in, since the same xor relates them)
 22 2016-05-13 00:32:22	0|sipa|gmaxwell, wumpus, jonasschnelli:
 23 2016-05-13 00:32:23	0|sipa|SHA256_avx,175,0.00575656,0.0058105,0.00575941
 24 2016-05-13 00:32:23	0|sipa|SHA256_basic,119,0.0088715,0.00893307,0.00887388
 25 2016-05-13 00:32:23	0|sipa|SHA256_rorx,223,0.00482899,0.004861,0.00483046
 26 2016-05-13 00:32:23	0|sipa|SHA256_rorx_x8ms,223,0.00475737,0.0047915,0.00476165
 27 2016-05-13 00:32:26	0|sipa|SHA256_sse4,175,0.00574069,0.00577295,0.00574323
 28 2016-05-13 00:33:10	0|sipa|i7-4800MQ, fixed at 2.6 GHz
 29 2016-05-13 00:34:13	0|phantomcircuit|is the first column cycle count?
 30 2016-05-13 00:34:32	0|sipa|iteration count
 31 2016-05-13 00:34:35	0|gmaxwell|phantomcircuit: it's number of times the freaky bench innerloop ran.
 32 2016-05-13 00:34:36	0|phantomcircuit|ah
 33 2016-05-13 00:34:38	0|sipa|how many tests were done
 34 2016-05-13 00:34:54	0|sipa|each doing 1 MB of data
 35 2016-05-13 00:35:01	0|phantomcircuit|ok
 36 2016-05-13 00:35:21	0|phantomcircuit|neat
 37 2016-05-13 00:35:31	0|gmaxwell|really for our usage running with 32 and 64 bytes of data is much more interesting.
 38 2016-05-13 00:35:56	0|gmaxwell|might be useful to see if that changes the numbers at all... perhaps gives AVX a purpose for existing? :)
 39 2016-05-13 00:36:07	0|sipa|sure, but it's just a proxy for the number of sha256 compression function runs
 40 2016-05-13 00:36:43	0|gmaxwell|ah duh right, it doesn't have a finialization, so it's not going to change.
 41 2016-05-13 00:37:23	0|sipa|note, it's a mobile CPU; if i don't fix the clock speed, cpufreq increases my cpu speed somewhere during the rorx_x8m run
 42 2016-05-13 00:37:32	0|phantomcircuit|gmaxwell, loading into the right registers takes some amount of work
 43 2016-05-13 00:37:37	0|sipa|making it look wildly better than rorx
 44 2016-05-13 00:38:05	0|phantomcircuit|sipa, test on a build box?
 45 2016-05-13 00:38:22	0|phantomcircuit|gmaxwell, any idea if those have turbo or not?
 46 2016-05-13 00:38:29	0|sipa|turbo is easy to disable
 47 2016-05-13 00:47:42	0|sipa|on our 56-core machine:
 48 2016-05-13 00:47:44	0|sipa|SHA256_avx,255,0.00397131,0.00401497,0.00397456
 49 2016-05-13 00:47:44	0|sipa|SHA256_basic,175,0.00599988,0.00604987,0.00600181
 50 2016-05-13 00:47:44	0|sipa|SHA256_rorx,319,0.00334556,0.003407,0.00334662
 51 2016-05-13 00:47:44	0|sipa|SHA256_rorx_x8ms,319,0.00328553,0.00332999,0.00328678
 52 2016-05-13 00:47:46	0|sipa|SHA256_sse4,255,0.00395919,0.00401807,0.00396108
 53 2016-05-13 00:47:53	0|sipa|gmaxwell: what cpu is it?
 54 2016-05-13 00:51:42	0|sipa|i'm surprised that our C code is only 45% slower than intel's optimized asm code
 55 2016-05-13 00:56:07	0|gmaxwell|the 56-core are broadwell-ep
 56 2016-05-13 00:56:29	0|gmaxwell|but 'only' at 2.2GHz.
 57 2016-05-13 00:58:19	0|phantomcircuit|sipa, none of those are parallel calculation right?
 58 2016-05-13 00:58:52	0|sipa|indeed, all single threaded
 59 2016-05-13 01:00:28	0|gmaxwell|sipa: sha2 that computes four at once is a considerable additional speedup, but harder to use without more software changes.
 60 2016-05-13 01:01:19	0|sipa|very doable inside merkle trees, though
 61 2016-05-13 01:02:11	0|gmaxwell|Yes. though thats pretty much the only place where it's very doable.
 62 2016-05-13 01:02:25	0|sipa|also the place where it probably matters most
 63 2016-05-13 01:02:55	0|gmaxwell|I'm sure if you want to write the merkle tree function wumpus will hapily do the work to give you a sha2x4 to call. :)
 64 2016-05-13 01:11:55	0|gmaxwell|my vague recollection was "the asm was 2x faster than the C, and the 4-way was 3x faster than the C" I dunno how that generalizes with the rorx.
 65 2016-05-13 02:53:00	0|GitHub130|[13bitcoin] 15sipa opened pull request #8051: Fix walletbackup.py failure (06master...06fixwalletbackup) 02https://github.com/bitcoin/bitcoin/pull/8051
 66 2016-05-13 03:02:09	0|gmaxwell|sipa: considering your PR comments might it be a bit strong to call that 'fix'?  rather than 'mysteriously stir'? :)
 67 2016-05-13 03:04:16	0|sipa|gmaxwell: well, it's reproducible :)
 68 2016-05-13 03:04:28	0|sipa|maybe i should call it "seemingly fix"
 69 2016-05-13 03:04:44	0|sipa|done
 70 2016-05-13 03:05:08	0|gmaxwell|presumably it fails for you because your computer is fast and travis isn't.
 71 2016-05-13 03:06:18	0|sipa|hmm, i started off by adding sleeps and that didn't help
 72 2016-05-13 03:06:30	0|sipa|but let me try again, by adding a sleep exactly there
 73 2016-05-13 03:19:03	0|sipa|gmaxwell: the sync blocks there causes a sleep of 1-2 seconds
 74 2016-05-13 03:19:11	0|sipa|gmaxwell: an explicit sleep of 60s does not fix it
 75 2016-05-13 03:23:44	0|gmaxwell|uh how does that make any sense?
 76 2016-05-13 03:24:00	0|sipa|sense, it makes none
 77 2016-05-13 06:53:02	0|jonasschnelli|sipa: I read something in the logs: is walletbackup.py fixed?
 78 2016-05-13 06:53:29	0|sipa|jonasschnelli: for me it is
 79 2016-05-13 07:02:32	0|jonasschnelli|sipa: I can't reproduce the issue... but you fix looks strange. :)
 80 2016-05-13 07:06:33	0|sipa|without it, i just can't run rpc_tests.py
 81 2016-05-13 07:06:38	0|sipa|it runs forever
 82 2016-05-13 07:08:24	0|jonasschnelli|But you can run walletbackup.py independent?
 83 2016-05-13 07:10:46	0|sipa|no
 84 2016-05-13 07:10:55	0|sipa|see the paste in the PR
 85 2016-05-13 07:10:57	0|jonasschnelli|hmm...
 86 2016-05-13 07:11:01	0|jonasschnelli|Yes. Saw it...
 87 2016-05-13 07:11:11	0|jonasschnelli|like to debug it locally.
 88 2016-05-13 07:18:55	0|sipa|maybe it depends on python version or something else
 89 2016-05-13 07:19:09	0|sipa|but something very fishy is going on, as it's not just a race condition
 90 2016-05-13 09:05:15	0|nub|would it be possible to shorten the block creation time from 10 minutes to say 1 minute with a hard fork obviously dividing the reward by a factor of 10 that way confrimations would be much faster i'd imagine
 91 2016-05-13 09:06:27	0|sipa|yes, it wod be possible with a hars fork
 92 2016-05-13 09:06:44	0|sipa|a hars fork could also turn bitcoin into a frontend for paypal
 93 2016-05-13 09:06:47	0|sipa|*hard
 94 2016-05-13 09:07:02	0|nub|could you explain that?
 95 2016-05-13 09:07:32	0|nub|is that bad?
 96 2016-05-13 09:07:37	0|sipa|a hard fork is replacing the system with another system, where all participants agrwe to switch to different software
 97 2016-05-13 09:07:44	0|sipa|it can change anything
 98 2016-05-13 09:07:57	0|nub|could a soft fork change block time?
 99 2016-05-13 09:08:08	0|sipa|the question "is x possible with a hard fork?" always has "yes" as answer
100 2016-05-13 09:08:15	0|sipa|no, a soft fork can't do that
101 2016-05-13 09:08:47	0|sipa|it would also be a bad idea, as it would result is 10 times higher mining centralization pressure
102 2016-05-13 09:08:58	0|sipa|and confirmations that have less meaning
103 2016-05-13 09:09:44	0|gmaxwell|sipa: actually not 10 times higher, but likely much higher, since the relationship is non-linear. :)
104 2016-05-13 09:09:57	0|nub|how can we speed up transactions?
105 2016-05-13 09:10:01	0|nub|or confirmations
106 2016-05-13 09:10:15	0|sipa|nub: why do you want to do that?
107 2016-05-13 09:10:35	0|sipa|if you want them to be fast enough to work at a point of sale, you'll need different technology
108 2016-05-13 09:10:56	0|nub|im a store wanting to sell stuff accepting bitcoin but i dont want them to leave until the bitcoin is in my account....
109 2016-05-13 09:10:57	0|sipa|there is simply no way to get global consensus within seconds
110 2016-05-13 09:11:03	0|nub|that currently takes far too long
111 2016-05-13 09:11:14	0|sipa|1 minute is also far too long
112 2016-05-13 09:11:34	0|sipa|you just can't use bitcoin blockchain transactions for that purpose
113 2016-05-13 09:11:46	0|nub|what if the purchaser uses the same hosted wallet platfrom as the seller
114 2016-05-13 09:12:03	0|nub|then it could be instant and no bitcoin would need to be sent\
115 2016-05-13 09:12:06	0|sipa|that wouls be one example of another technology
116 2016-05-13 09:12:30	0|sipa|now you're using the database of the wallet provider rather than the blockchain
117 2016-05-13 09:12:58	0|nub|something like cassandra replicated to datacenters all around the world
118 2016-05-13 09:13:25	0|sipa|if you have a centrally trusted party running those datacenters, it's easy :)
119 2016-05-13 09:13:43	0|sipa|but that's not a luxury we have for bitcoin the base technology
120 2016-05-13 09:14:12	0|nub|whats the aim of bitcoin now?
121 2016-05-13 09:14:18	0|sipa|this discussion probably belongs in #bitcoin
122 2016-05-13 09:14:27	0|nub|wanna move to there?
123 2016-05-13 09:14:41	0|sipa|i'm going to sleep, but feel free :)
124 2016-05-13 09:15:15	0|nub|is it ok to discuss how a hard fork could work here?
125 2016-05-13 09:15:47	0|sipa|it's easy: make a change to the code that does what you want, and convince the whole world to switch to that code
126 2016-05-13 09:16:09	0|sipa|and no, the interblock time is not going to change
127 2016-05-13 09:16:48	0|nub|what if it was a slow transition say a client which works on both and at a certain (ntp time) it switches everyone
128 2016-05-13 09:17:21	0|nub|could put that feature in a year beefore the planned fork so everyones updated to the client that supports that by then
129 2016-05-13 09:17:28	0|sipa|you'd still need to convince the entire world to switch before that fkag date
130 2016-05-13 09:17:39	0|nub|make the app do it automatically
131 2016-05-13 09:17:42	0|nub|bitcoin core
132 2016-05-13 09:17:54	0|sipa|bitcoin core does not decide what the rules of the network are
133 2016-05-13 09:18:02	0|nub|fair enough
134 2016-05-13 09:18:09	0|sipa|it very intentionally does not have an auto update function
135 2016-05-13 09:18:19	0|sipa|as developers shouldn't be in charge of the rules
136 2016-05-13 09:18:20	0|nub|it should
137 2016-05-13 09:18:26	0|sipa|it absolutely should not
138 2016-05-13 09:18:29	0|nub|and if not updated you can't connect
139 2016-05-13 09:18:37	0|sipa|it would make developers central bankers
140 2016-05-13 09:18:40	0|nub|devs could add whatever they want
141 2016-05-13 09:19:00	0|nub|what if i were to hire them all?
142 2016-05-13 09:19:07	0|sipa|try me
143 2016-05-13 09:19:17	0|sipa|i'd find another job
144 2016-05-13 09:19:47	0|nub|a surcharge could be added to mining which goes to devs instead
145 2016-05-13 09:20:07	0|sipa|but even if you could, hopefully the ecosystem would protest and stop using bitcoin core
146 2016-05-13 09:20:38	0|nub|wouldnt a 1 minute block time be welcomed for faster trasnactions?
147 2016-05-13 09:20:42	0|sipa|no
148 2016-05-13 09:21:02	0|sipa|it's a misconception that that would be beneficial
149 2016-05-13 09:21:12	0|nub|it works for litecoin
150 2016-05-13 09:21:37	0|sipa|"it works" does not mean it is better
151 2016-05-13 09:22:11	0|nub|would bitcoin devs be interested in incorperating and being paid to develop?
152 2016-05-13 09:22:34	0|nub|in america
153 2016-05-13 09:22:53	0|sipa|see https://bitcoincore.org/en/2016/04/04/announcing_sponsorship_programme/
154 2016-05-13 09:23:00	0|nub|banks want there software to run on blockchain technology
155 2016-05-13 09:23:08	0|nub|we could be running the banks
156 2016-05-13 09:23:10	0|sipa|they don't need proof of work or miners
157 2016-05-13 09:23:25	0|nub|they dont
158 2016-05-13 09:23:39	0|nub|they need a special client which can create and destroy coins
159 2016-05-13 09:23:46	0|sipa|this is getting off topic, as you're not talking about developing bitcoin
160 2016-05-13 09:23:52	0|sipa|#bitcoin please
161 2016-05-13 10:41:34	0|nub|could a soft fork increase block rewards
162 2016-05-13 10:42:20	0|gmaxwell|nub: no, and please take further discussion elsewhere.
163 2016-05-13 17:15:51	0|GitHub189|[13bitcoin] 15kazcw opened pull request #8052: rpc tests: increase http timeout (06master...06rpcwallet-test-timeout) 02https://github.com/bitcoin/bitcoin/pull/8052
164 2016-05-13 18:22:48	0|arubi|nub did remind me of a question I was meaning to ask. and I'm sorry if this is obvious.  suppose we have a way to send an input entirely as miners' fee, then "provably" receive it as an output from a generation transaction in a block that the input->fee transaction was mined on.  if everyone decides one day to use this feature, then could we discard any blockchain data up until the block that this happened, essentially making this new block
165 2016-05-13 18:22:48	0|arubi|the genesis block?  this might even be a soft in its "forkiness".  a new client might want to start syncing from the new genesis, and and old client won't care that it happened?  this is very theoretical, I'm just looking to validate my understanding of how this could play out
166 2016-05-13 18:23:53	0|sipa|arubi: we can always declare a new block as the genesis block; just snapshot its utxo set
167 2016-05-13 18:25:46	0|arubi|sipa, is that the same?  why not do it?
168 2016-05-13 18:27:09	0|arubi|well it's not the same.  I will still need to verify the utxo set to believe that it follows the history since genesis
169 2016-05-13 18:27:49	0|sipa|arubi: you can do it. copy your chainstatre directory from someone you trust
170 2016-05-13 18:28:17	0|arubi|but trust isn't needed if there's a mechanism like I suggested, right?  (maybe not)
171 2016-05-13 18:30:03	0|sipa|of course you need trust in your model; you're relying on the assumption that everyone accepts that the new genesis block is in fact derived from the old one
172 2016-05-13 18:30:08	0|sipa|and there is no way to verify that
173 2016-05-13 18:30:34	0|sipa|we have no technology that can verify the correctness of the blockchain without seeing it :)
174 2016-05-13 18:31:38	0|arubi|hm.  so you're saying that I'd still need to know the history up until that new genesis block, right?  I understand the issue if so
175 2016-05-13 18:32:06	0|sipa|you don't need to know it if you trust that it's there
176 2016-05-13 18:32:17	0|sipa|but that's no different from copying a chainstate from someone
177 2016-05-13 18:32:38	0|arubi|right. thanks sipa.
178 2016-05-13 18:32:45	0|sipa|there are ideas about committing the hash of the UTXO set to the blockchain
179 2016-05-13 18:33:08	0|arubi|oh, so the utxo set could be shared?
180 2016-05-13 18:33:17	0|instagibbs|arubi, miners would commit to utxo set
181 2016-05-13 18:33:26	0|sipa|which would allow you to for example download/verify headers up to 1000 blocks in the past, then download a snapshot of the utxo set at that point from anyone, and verify that it matches the hash in the blockchain
182 2016-05-13 18:33:33	0|instagibbs|spv trust for utxo set in other words
183 2016-05-13 18:33:53	0|sipa|HOWEVER that still involved trust: you've now switched from a model of no trust in history to trusting that miners would not commit to an invalid history
184 2016-05-13 18:34:06	0|sipa|which in practice may very well be sufficient, but it's a very different security model
185 2016-05-13 18:35:02	0|arubi|so I'd really be trusting the network to verify and relay the chain correctly to me?
186 2016-05-13 18:35:12	0|sipa|no
187 2016-05-13 18:35:24	0|sipa|you'd be trusting miners to not build a chain with invalid history
188 2016-05-13 18:35:56	0|arubi|but that's impossible now, if the network verifies
189 2016-05-13 18:36:14	0|sipa|that's true but irrelevant
190 2016-05-13 18:36:23	0|sipa|it's impossible because YOU verify, if you run a full node
191 2016-05-13 18:36:41	0|sipa|if you assume the network verifies for you, you're trusting them
192 2016-05-13 18:37:17	0|arubi|right, so how is an spv node different in verifying the blocks?  is it about verifying the transactions themselves?
193 2016-05-13 18:38:50	0|sipa|an spv node assumes that miners will not produce an invalid chain
194 2016-05-13 18:39:19	0|sipa|(or that they're only connected to honest full nodes)
195 2016-05-13 18:40:59	0|arubi|I think I understand, like you mentioned that they verify the headers (and not the block itself?), they trust that what they're verifying is the actual chain verifying nodes use
196 2016-05-13 18:41:23	0|sipa|no, they assume that miners would not make an invalid chain
197 2016-05-13 18:41:40	0|sipa|(or that full nodes would filter out such an invalid chain for them)
198 2016-05-13 18:43:33	0|arubi|that's what I meant by "the actual chain..".  sure assuming miners won't produce an invalid chain is understandable, but if you somehow only connect to honest nodes, then you will not get it, and I see where this requires trust
199 2016-05-13 18:44:50	0|sipa|well there is no "the actual chain", different nodes can have a different idea of what chain to accept
200 2016-05-13 18:45:58	0|arubi|true.  the best chain used by the reference client is probably more specific, but also impossible to expect from just connecting to random nodes
201 2016-05-13 18:46:56	0|sipa|no no, not by the reference client
202 2016-05-13 18:47:14	0|sipa|every individual node, even if they are running the same software, could have a different idea of what the best chain is
203 2016-05-13 18:47:32	0|arubi|I know, but there is physically only one best chain
204 2016-05-13 18:47:50	0|sipa|no
205 2016-05-13 18:47:51	0|arubi|or maybe multiple "same height" chains.. that's bad in itself
206 2016-05-13 18:48:07	0|sipa|every indidual node has an idea of what the best chain is
207 2016-05-13 18:48:14	0|sipa|there is no guarantee that other nodes have the same idea
208 2016-05-13 18:48:44	0|arubi|I get that, but really the chain with the most work will overtake the others quickly, no?
209 2016-05-13 18:48:54	0|sipa|that's an assumption :)
210 2016-05-13 18:49:11	0|arubi|it worked so far, even on the chaos that is testnet :)
211 2016-05-13 18:49:14	0|sipa|and the correctness of the system relies on it, but it's anot a given
212 2016-05-13 18:49:24	0|sipa|it's the result of economic and technical properties
213 2016-05-13 18:50:26	0|sipa|and you change those if nodes in the network don't validate fully
214 2016-05-13 18:51:59	0|arubi|or forks intentionally, or fails due to a bug, right.  correctness of the best chain that's advertised to my node is something I really took for granted until now
215 2016-05-13 18:52:13	0|arubi|well, not my fully verifying node
216 2016-05-13 19:03:41	0|Chris_Stewart_5|sipa: An example of a node not knowing what the longest chain could be is a sustained sybil attack correct?
217 2016-05-13 19:04:32	0|sipa|Chris_Stewart_5: or any normal fork
218 2016-05-13 19:04:57	0|sipa|Chris_Stewart_5: it's not so much "a node does not know what the best chain is", it is that there _is_ no best chain
219 2016-05-13 19:05:10	0|sipa|best chain is something local to your client
220 2016-05-13 19:06:10	0|sipa|and we build a system that aims to provide convergence: making sure that over time, blocks in the history of different nodes' best chains end up being the same over tim
221 2016-05-13 19:21:18	0|Chris_Stewart_5|interesting. Thanks - it's easy to forget we have 5k individual computers running on this network that need to reach convergence on what is right
222 2016-05-13 21:17:24	0|kanzure|those sleeps in the tests are architecturally unfortunate :( shouldn't this be stuff that gets pinged/notified instead of waiting forever aimlessly? e.g. crash could have happened seconds ago.
223 2016-05-13 21:17:59	0|kanzure|or is that intentional etc...