1 2016-08-27 00:44:42	0|GitHub4|[13bitcoin] 15nomnombtc opened pull request #8608: Install manpages via make install, also add some autogenerated manpages (06master...06man_automake2) 02https://github.com/bitcoin/bitcoin/pull/8608
  2 2016-08-27 01:49:44	0|btcdrak|is there a way to sign each commit during an interactive rebase?
  3 2016-08-27 03:42:54	0|roasbeef|btcdrak: signoff-rebase = "!GIT_SEQUENCE_EDITOR='sed -i -re s/^pick/e/' sh -c 'git rebase -i $1 && while git rebase --continue; do git commit --amend -S --no-edit; done' -"
  4 2016-08-27 03:43:06	0|roasbeef|btcdrak: stuff that into an alias in your .gitconfig
  5 2016-08-27 05:15:28	0|luke-jr|hm, I would have thought verifying blocks was a read-only operation, but it seems to be writing about as much as it's reading?
  6 2016-08-27 05:15:54	0|luke-jr|more actually
  7 2016-08-27 05:21:33	0|gmaxwell|should be much more,
  8 2016-08-27 05:21:39	0|gmaxwell|if it's reading your dbcache is too small.
  9 2016-08-27 05:43:28	0|luke-jr|what's it writing?
 10 2016-08-27 05:44:33	0|gmaxwell|the chainstate.
 11 2016-08-27 05:51:09	0|wumpus|right, it's not the verification that causes writing, but *accepting* the blocks and updating the state
 12 2016-08-27 05:51:29	0|wumpus|rejecting blocks should be a read-only operations
 13 2016-08-27 05:51:46	0|wumpus|as well as verifying transactions outside blocks
 14 2016-08-27 05:55:35	0|wumpus|gmaxwell: re: BU removing outward connection limit, sigh, it was to be expected somehow, you can't stop anti-social people by just telling them not to do a certain behavior, we'll need explicit anti-DoS measures for that
 15 2016-08-27 05:56:29	0|wumpus|such a system could block spy-nodes connecting to everyone in one go
 16 2016-08-27 05:57:06	0|wumpus|that there is no damn reason to connect to connect to more nodes also won't stop anyone
 17 2016-08-27 05:57:36	0|wumpus|'look mom we're real anarchists, we can misbehave on an open P2P network!'
 18 2016-08-27 06:02:23	0|midnightmagic|any such mechanism if successful would at least in the short term eradicate blockchain.info's spying, as well as the various chainalysis mechs.
 19 2016-08-27 06:02:26	0|wumpus|their name, their whole raison d'etre, is 'remove limits', so they removed another limit. It doesn't matter whether it had a purpose, they've remved a few lines of very limiting-feeling codes, and now they feel happy
 20 2016-08-27 06:04:30	0|wumpus|pre-emtively filling all connection slots would also prevent blockchain.info's spying :-)
 21 2016-08-27 06:05:02	0|midnightmagic|:-)
 22 2016-08-27 06:07:20	0|wumpus|if this escalates and there is no solution, in the longer run, it will turn the P2P network to a ghetto, and I'd expect it to transition it to more of a F2F network w/ authenticated connections
 23 2016-08-27 06:08:02	0|wumpus|but also a few more mass-connectors likely won't break the netwerk, it just depends on the scale
 24 2016-08-27 06:12:16	0|midnightmagic|Bitcoin Classic -- DDoS'ing the Bitcoin network since 2016.
 25 2016-08-27 06:12:36	0|midnightmagic|I wonder if someone could get gavin to scold them for that
 26 2016-08-27 06:12:47	0|midnightmagic|or. Unlimited. Same diff.
 27 2016-08-27 06:14:07	0|wumpus|it reminds me a bit of seeders versus leechers discussion for bittorrent, with the difference is that 'defecting' there is advantageous to the leecher, here it's just about being a jerk just because
 28 2016-08-27 06:15:53	0|wumpus|though connecting to lots of nodes can get you blocks fractially faster, which is useful for miners, you can accomplish the same thing by listening and advertising
 29 2016-08-27 06:26:46	0|wumpus|you can set a virtually unlimited number of incoming connection slots, then by having a node with a high uptime, it will drift up in the DNS seeder rankings, which means its address will be dealt out more often, resulting in more connections to other nodes. It's a slower process but the social thing to do.
 30 2016-08-27 06:34:34	0|luke-jr|shouldn't those blocks (and the chainstate) already be correct? O.o
 31 2016-08-27 06:35:49	0|wumpus|but blocks are a delta to the chainstate
 32 2016-08-27 06:36:18	0|wumpus|if you accept a block, by definition, you have to update the chainstate. And undo files are produced, too, a reverse-delta just in case.
 33 2016-08-27 06:36:35	0|luke-jr|so it's actually rewinding and re-accepting them? somehow I thought it just verified the current state was sane
 34 2016-08-27 06:37:04	0|gmaxwell|wumpus: what you say is true, but its much harder to deal with abusive behavior when you also have many more 'honest' people also being abusive out of ignorance. (in part, because the motivational structure is different; e.g. we can discourage spy nodes by reducing information leaks, moving more users into tor, etc.)
 35 2016-08-27 06:37:28	0|wumpus|luke-jr: no, just applying the block results in writes to the utxo state, to mark outputs as spent, add new outputs. it rewinds only on reorg
 36 2016-08-27 06:37:38	0|gmaxwell|(But ignorance, "I'm gonna help the network out by connecting to EVERYTHING!" ... is harder to resolve by rational measures)
 37 2016-08-27 06:38:06	0|luke-jr|wumpus: I'm talking about the verifying blocks at startup.. checkblocks/checklevel
 38 2016-08-27 06:38:23	0|wumpus|luke-jr: ohhh! that wasn't clear to me.
 39 2016-08-27 06:38:32	0|wumpus|luke-jr: no, that should be read-only
 40 2016-08-27 06:38:33	0|luke-jr|sorry
 41 2016-08-27 06:38:53	0|luke-jr|hmm
 42 2016-08-27 06:39:34	0|wumpus|gmaxwell: sure, at least ignorance can be improved by trying harder to inform people about things
 43 2016-08-27 06:42:35	0|wumpus|people may assume 'moaaarrrr outgoing connections is better' unless it's explained , .e.g in user interfaces, blog posts, that it's bad for yourself as well as others, with alternatives how to get more connections. Sure, some people will ignore it, or be jerks, but hopefully a miniority and most will heed.
 44 2016-08-27 06:43:39	0|wumpus|luke-jr: the rewinding with checklevel 3 completely happens in memory, if it is writing anything to disk that'd be very wrong
 45 2016-08-27 06:44:01	0|wumpus|(there could be a bug of course....)
 46 2016-08-27 06:57:37	0|luke-jr|having trouble reproducing now. flushed Linux's disk caches though, and even with just reading, it's going super-slow :|
 47 2016-08-27 06:59:58	0|luke-jr|(as in 1% every 2 minutes or so, ETA 3 hours at this rate)
 48 2016-08-27 07:09:22	0|wumpus|yes it reports swapping as writing, but wouldn't account swapping of *other* processes to disk to bitcoind
 49 2016-08-27 07:09:46	0|wumpus|IIRC there's a kswapd that gets all the blame for swapping
 50 2016-08-27 07:09:50	0|luke-jr|hmm
 51 2016-08-27 07:10:03	0|luke-jr|it did pick up speed and finished in 15 mins (just now) fwiw
 52 2016-08-27 07:10:18	0|luke-jr|didn't see any writing this time either
 53 2016-08-27 07:11:29	0|wumpus|phew
 54 2016-08-27 07:12:38	0|luke-jr|but I guess I did manage to reproduce https://www.reddit.com/r/Bitcoin/comments/4zrxs1/qtcore_client_taking_ages_to_start/ after all
 55 2016-08-27 07:12:46	0|luke-jr|just a matter of dropping caches
 56 2016-08-27 07:12:59	0|luke-jr|gmaxwell: can you confirm on your slow laptop? echo 3 > /proc/sys/vm/drop_caches
 57 2016-08-27 07:14:13	0|wumpus|usually if the client takes ages to start there's a backlog of blocks to verify
 58 2016-08-27 07:15:08	0|sipa|question: how many times has the initial verification at startup actually caught corruption
 59 2016-08-27 07:15:48	0|sipa|nobody knows, of course
 60 2016-08-27 07:15:50	0|wumpus|I don't know - but I think it would be just as effective to just check a few blocks
 61 2016-08-27 07:15:53	0|luke-jr|sipa: not sure if it's still a problem, but every time when we had those powerfail-corrupts-db problem? (unless that was caught by something else?)
 62 2016-08-27 07:16:06	0|wumpus|the latest blocks are the most likely to be corrupt and below that it drops off
 63 2016-08-27 07:16:10	0|sipa|luke-jr: those result in a leveldb checksum error
 64 2016-08-27 07:16:31	0|sipa|there are only 4 checklevels :)
 65 2016-08-27 07:16:41	0|wumpus|luke-jr: same here - I think the default checkblocks should be much lower
 66 2016-08-27 07:16:43	0|luke-jr|well, if there's ever more added, I'm ready! :P
 67 2016-08-27 07:16:58	0|sipa|and 4 blocks is not very much
 68 2016-08-27 07:16:58	0|wumpus|a as-thorough-as-possible check on just a few blocks
 69 2016-08-27 07:17:07	0|wumpus|well, take 10 then
 70 2016-08-27 07:17:16	0|luke-jr|sipa: my PC is on UPS ;)
 71 2016-08-27 07:17:25	0|sipa|wumpus: jonasschnelli has a patch to switch to txcount based limiting
 72 2016-08-27 07:18:07	0|luke-jr|hmm, that's an interesting idea.
 73 2016-08-27 07:18:08	0|wumpus|that's good, but I think the effective default check depth should also be lowered, don't know if it does that
 74 2016-08-27 07:18:47	0|wumpus|or maybe write a flag on 'clean' shutdown, and do a reduced check in that case?
 75 2016-08-27 07:18:53	0|luke-jr|set it for the equivalent checkblocks back when it was introduced? :p
 76 2016-08-27 07:19:18	0|sipa|wumpus: it sets the default to 100000 txn
 77 2016-08-27 07:19:34	0|sipa|i guess it can be lower even
 78 2016-08-27 07:19:36	0|wumpus|sipa: ok
 79 2016-08-27 07:19:51	0|sipa|but it also insists on at least 6 blocks
 80 2016-08-27 07:19:56	0|gmaxwell|we need to do something about the startup check.
 81 2016-08-27 07:20:23	0|gmaxwell|unfortunately just checking a couple blocks is not much of a test, but anything more takes too long for most people.
 82 2016-08-27 07:20:33	0|sipa|wumpus: interesting idea
 83 2016-08-27 07:20:40	0|luke-jr|I wonder how difficult it would be to background it
 84 2016-08-27 07:20:44	0|gmaxwell|wumpus: neat idea!
 85 2016-08-27 07:22:10	0|gmaxwell|when we were having windows corruption, what was needed to reliably detect that?
 86 2016-08-27 07:22:56	0|wumpus|just opening the leveldb IIRC
 87 2016-08-27 07:22:59	0|btcdrak|roasbeef: thanks!!
 88 2016-08-27 07:23:20	0|wumpus|or maybe the first access. No thorough checking was necessary
 89 2016-08-27 07:24:03	0|wumpus|sounds good to me
 90 2016-08-27 07:25:10	0|wumpus|it's very helpful that leveldb has its own corruption detection here
 91 2016-08-27 07:25:23	0|gmaxwell|why bother with the txn count?
 92 2016-08-27 07:25:26	0|gmaxwell|just set it to 6 blocks?
 93 2016-08-27 07:25:39	0|wumpus|because that auto-adapts
 94 2016-08-27 07:25:55	0|gmaxwell|so? other than segwit the txn count of blocks won't change much.
 95 2016-08-27 07:25:56	0|wumpus|if blocks grow again, it won't become  slower
 96 2016-08-27 07:26:20	0|wumpus|heh yes that's a completely different discission
 97 2016-08-27 07:26:23	0|sipa|gmaxwell: if you are early in IBD you probably want more blockss
 98 2016-08-27 07:26:38	0|gmaxwell|sipa: okay, I'll but that.. but kind of a corner case.
 99 2016-08-27 07:26:40	0|wumpus|but the fact is that transaction could is a better measure of run time
100 2016-08-27 07:26:43	0|wumpus|count*
101 2016-08-27 07:26:46	0|gmaxwell|I was at the verge of suggesting this just do a single block.
102 2016-08-27 07:26:57	0|wumpus|and amount of data checked
103 2016-08-27 07:27:06	0|sipa|heh, fine by me as well
104 2016-08-27 07:27:22	0|gmaxwell|It's not yielding a high payoff in detected errors, the ones I know it detected (chainstate version corruption, need checking back to your last restat to reliably report)
105 2016-08-27 07:27:50	0|wumpus|well the first priority is to make the insane wait time go away
106 2016-08-27 07:28:02	0|wumpus|whether the new number becomes low or ultra-low is less important :)
107 2016-08-27 07:28:16	0|gmaxwell|One or two block keeps the code live and working, I think this code is useful around refactorings and changes to the code.
108 2016-08-27 07:48:12	0|btcdrak|i guess the reply to Roger's testnet response is no, BU runs in production and is a menace to the network
109 2016-08-27 07:48:44	0|btcdrak|BU has no peer review and no chance in hell of being used for real
110 2016-08-27 08:06:04	0|btcdrak|wumpus: rebased #7562
111 2016-08-27 08:07:38	0|sipa|btcdrak: what was the problem with the tests?
112 2016-08-27 08:07:43	0|btcdrak|gmaxwell: can you explain a bit more about your suggestion regarding my attempt at extracting the MacOSX sdk on linux?
113 2016-08-27 08:09:53	0|btcdrak|sipa: changing the defaul tx version affected a couple of tests that were comparing hash values, or sorting txs by hash. obviously the hashes change when tx version is bumped. Those seem innocuous. There is one test however that I dont understand which I commented on. Not sure why it is affected by changing the version number.
114 2016-08-27 08:10:16	0|btcdrak|I dont know if it's the tests' fault and innocuous, or revealing an issue.
115 2016-08-27 08:30:54	0|gmaxwell|btcdrak: so, first someone extracts it via OSX.  Then we take the extracted binary and find all the offsets for its data in the decompressed file.
116 2016-08-27 08:31:01	0|gmaxwell|we can distribute the offset list.
117 2016-08-27 08:31:22	0|gmaxwell|actually, a took like xdelta or another binary diff tool might just handle it.
118 2016-08-27 08:40:31	0|sipa|but the dmg file is compressed
119 2016-08-27 08:40:48	0|sipa|can we decompress it?
120 2016-08-27 08:41:05	0|sipa|without "installing", that is
121 2016-08-27 08:41:53	0|gmaxwell|it's a 7z file according to drak
122 2016-08-27 08:43:10	0|gmaxwell|so I was assuming that.
123 2016-08-27 08:43:12	0|btcdrak|gmaxwell: one thing I am not sure about is if 7z is actually extracting it correctly. It seems to be, and the bug seems more like in the linux implementation of hfsplus; but it is feasible that the compression algo was tweaked and 7z is unaware
124 2016-08-27 08:43:24	0|gmaxwell|unlikely, 7z has checksums.
125 2016-08-27 08:47:05	0|sipa|so we'd xdelta the 7z-decompressed dmg with the resulting compressed .tar?
126 2016-08-27 08:47:21	0|gmaxwell|uncompressed tar.
127 2016-08-27 08:47:44	0|gmaxwell|oh what we want is a tgz.
128 2016-08-27 08:47:45	0|gmaxwell|yes.
129 2016-08-27 08:47:53	0|gmaxwell|so the file we normally get out of it.
130 2016-08-27 08:48:05	0|gmaxwell|and if the resulting delta is small, we call it done.
131 2016-08-27 08:48:29	0|luke-jr|btcdrak: what version of 7zip did you use?
132 2016-08-27 08:48:50	0|sipa|btcdrak: where is hfsplus used?
133 2016-08-27 08:49:34	0|btcdrak|luke-jr: version 9.20
134 2016-08-27 08:49:43	0|luke-jr|hmm, it tells me: Error: Can not open file as archive
135 2016-08-27 08:56:44	0|btcdrak|sipa: https://www.irccloud.com/pastebin/S3zIzO3D/
136 2016-08-27 08:58:29	0|luke-jr|oh, my DMG file is truncated
137 2016-08-27 08:59:05	0|btcdrak|so basically the files we want are in Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk, but using the method above you get a bunch of empty files. On a Mac you run "tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk"
138 2016-08-27 08:59:36	0|btcdrak|cfields_ documented it here https://github.com/bitcoin/bitcoin/blob/master/doc/README_osx.md
139 2016-08-27 09:00:08	0|sipa|but we don't need all of the contents of that dmg, right?
140 2016-08-27 09:00:27	0|luke-jr|not nearly
141 2016-08-27 09:00:39	0|luke-jr|but it's copyrighted and non-redistributable :<
142 2016-08-27 09:00:40	0|btcdrak|sipa right, we just need the files from Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/* which we turn into a tar.gz
143 2016-08-27 09:00:46	0|sipa|i see
144 2016-08-27 09:00:55	0|btcdrak|That gives us 47MB tar.gz :-p
145 2016-08-27 09:02:09	0|btcdrak|we drop that in the gitian-builder/inputs/ folder when performing the Gitian rituals.
146 2016-08-27 09:02:47	0|sipa|got it
147 2016-08-27 09:03:04	0|sipa|14m here
148 2016-08-27 09:03:19	0|luke-jr|you and your fast internet :x :p
149 2016-08-27 09:03:59	0|sipa|luke-jr: this is over t-mobile roaming data
150 2016-08-27 09:04:29	0|luke-jr|sadly, T-Mobile data here seems to only be about 5 Mbps
151 2016-08-27 09:10:22	0|gmaxwell|2hr 49m remaining here.
152 2016-08-27 09:14:20	0|btcdrak|RIP sipa's roaming data charges
153 2016-08-27 09:14:25	0|sipa|btcdrak: it's free
154 2016-08-27 09:14:41	0|sipa|it's effectively cheaper than my swiss wired or wireless internet
155 2016-08-27 09:15:04	0|sipa|s/free/fixed price/
156 2016-08-27 09:16:30	0|gmaxwell|btcdrak: a while back tmobile ceo put out some open letter about "data abusers" and for some reason it said, 'mining bitcoin' -- for a while we wondered if perhaps he was talking about blockstream. https://newsroom.t-mobile.com/news-and-blogs/stopping-network-abusers.htm
157 2016-08-27 09:16:47	0|gmaxwell|apparently blockstream was using several TB a month on tmobile.
158 2016-08-27 09:18:27	0|gmaxwell|doesn't everyone run a full node on their phone?
159 2016-08-27 09:18:45	0|luke-jr|lol, xdelta is 45 MB
160 2016-08-27 09:18:59	0|sipa|:(
161 2016-08-27 09:19:09	0|sipa|luke-jr: how did you decompress the dmg?
162 2016-08-27 09:19:12	0|luke-jr|7z
163 2016-08-27 09:19:28	0|gmaxwell|lol
164 2016-08-27 09:19:53	0|gmaxwell|might need more options for it to actually find the match
165 2016-08-27 09:20:03	0|luke-jr|maybe there's some tweaking to the tar we need to put it in a better order?
166 2016-08-27 09:20:17	0|gmaxwell|yes, that too.
167 2016-08-27 09:20:21	0|luke-jr|oh wait
168 2016-08-27 09:20:27	0|luke-jr|hmm, nm
169 2016-08-27 09:20:33	0|luke-jr|the uncompressed tar is 306 MB FWIW
170 2016-08-27 09:20:38	0|luke-jr|but xdelta's patch is compressed.
171 2016-08-27 09:20:47	0|gmaxwell|were you using the compressed tar as the input?
172 2016-08-27 09:20:50	0|luke-jr|no
173 2016-08-27 09:21:22	0|sipa|hpmount, mount -t hfsplus, 7zr... all fail to open the file
174 2016-08-27 09:21:37	0|gmaxwell|maybe we should just encrypt the damn tar with the whole dmg file as the key, and then call that sufficient for legal purposes. :P
175 2016-08-27 09:22:03	0|luke-jr|sipa: need 7z, not 7zr
176 2016-08-27 09:22:07	0|luke-jr|7zr has formats removed
177 2016-08-27 09:22:32	0|luke-jr|gmaxwell: hmm, I wonder if that'd be okay
178 2016-08-27 09:22:54	0|luke-jr|I turned off xdelta's patch compression and it's 261 MB :/
179 2016-08-27 09:23:10	0|gmaxwell|smaller than 300 so it found something.
180 2016-08-27 09:23:48	0|gmaxwell|luke-jr: I think doing that would be defensible.
181 2016-08-27 09:23:59	0|luke-jr|xdelta3: warning: input position 310378496 overflowed instruction buffer, needed 43091 (vs. 32768), consider changing -I
182 2016-08-27 09:24:14	0|gmaxwell|consider changing -I
183 2016-08-27 09:24:29	0|luke-jr|not doing much better :<
184 2016-08-27 09:24:47	0|luke-jr|-rw-r--r-- 1 luke-jr luke-jr 261M Aug 27 09:24 MacOSX.xdelta
185 2016-08-27 09:24:52	0|gmaxwell|there are some other binary patching tools, xdelta was the first that came to mind.
186 2016-08-27 09:25:10	0|sipa|xdelta and xdelta3 seem to be different things
187 2016-08-27 09:28:04	0|gmaxwell|open-vcdiff
188 2016-08-27 09:28:24	0|luke-jr|I'm writing something custom
189 2016-08-27 09:28:52	0|luke-jr|hmm
190 2016-08-27 09:28:57	0|luke-jr|I can't mmap this DMG :<
191 2016-08-27 09:29:08	0|btcdrak|well surely the better solution would be to fix the hfsplus driver?
192 2016-08-27 09:29:09	0|gmaxwell|luke-jr: upgrade to a darn 64 bit OS already luke!
193 2016-08-27 09:31:51	0|luke-jr|oh, if I build with -static I can
194 2016-08-27 09:38:53	0|sipa|i can't get it below 46 MB
195 2016-08-27 09:39:17	0|sipa|that's with
196 2016-08-27 09:39:19	0|sipa|xdelta3 encode -P $((2**30)) -I 0 -B 2147483648 -W 16777216
197 2016-08-27 09:39:31	0|sipa|(the highest values for each of the limits that work)
198 2016-08-27 09:41:46	0|sipa|it runs too fast, though
199 2016-08-27 09:42:13	0|sipa|this is an 8 GB source file, and it only takes a few seconds
200 2016-08-27 09:42:27	0|gmaxwell|"I demand my np-complete problems solving software be slow"
201 2016-08-27 09:42:44	0|sipa|i demand it actually uses the whole input
202 2016-08-27 09:43:50	0|gmaxwell|try open-vcdiff?
203 2016-08-27 09:45:29	0|sipa|42 MB
204 2016-08-27 09:45:43	0|sipa|gmaxwell: seems that uses the same algorithm as xdelta3
205 2016-08-27 09:46:11	0|gmaxwell|failure?
206 2016-08-27 09:47:28	0|luke-jr|http://codepad.org/Yi5FEWML <-- any obvious bugs or possible optimizations?
207 2016-08-27 09:49:07	0|btcdrak|also, I tried the dmg2img tool, but that doesnt appear to have been updated to handle the altered compression
208 2016-08-27 09:50:21	0|GitHub180|[13bitcoin] 15ajtowns closed pull request #8575: leveldb: generate lib independent of locale sort (06master...06leveldb-locale-reproducible) 02https://github.com/bitcoin/bitcoin/pull/8575
209 2016-08-27 09:50:21	0|luke-jr|welp, failed to find the first 82 byte file :/
210 2016-08-27 09:50:26	0|gmaxwell|luke-jr: well the needles may not be in the file in a contigious chunk.
211 2016-08-27 09:50:41	0|luke-jr|gmaxwell: surely 82 bytes would be :<
212 2016-08-27 09:50:58	0|gmaxwell|unless the file system stores the first n bytes of files in the inode table.
213 2016-08-27 09:51:03	0|luke-jr|hmm
214 2016-08-27 09:51:05	0|gmaxwell|(so that magic is fast)
215 2016-08-27 09:51:19	0|gmaxwell|or just to prevent the indirection
216 2016-08-27 09:51:39	0|luke-jr|would it make sense to target the 2nd 4096 bytes of the file then?
217 2016-08-27 09:53:55	0|gmaxwell|to test your tool grab the second 4096 bytes of the haystack.
218 2016-08-27 09:54:02	0|gmaxwell|but yes, you could try that.
219 2016-08-27 09:54:41	0|luke-jr|http://codepad.org/VvfwXbqo isn't getting anything so far either
220 2016-08-27 09:54:59	0|luke-jr|you mean of the needle, right?
221 2016-08-27 09:55:26	0|gmaxwell|I mean grab a 4096 byte chunk out of the data you are searching in, just to verify your tool works.
222 2016-08-27 09:56:12	0|luke-jr|oh
223 2016-08-27 09:56:15	0|gmaxwell|dd if=5.hfs of=junk bs=4096 skip=1234 count=1
224 2016-08-27 09:56:18	0|luke-jr|"In Mac OS X Snow Leopard 10.6, HFS+ compression was added. In open source and some other areas this is referred to as AppleFSCompression. Compressed data may be stored in either an extended attribute or the resource fork."
225 2016-08-27 09:56:25	0|luke-jr|I bet this is what we're dealing with :|
226 2016-08-27 09:57:26	0|sipa|seems likely, indeed
227 2016-08-27 09:59:12	0|gmaxwell|bleh
228 2016-08-27 10:00:20	0|luke-jr|http://www.spinics.net/lists/linux-fsdevel/msg55545.html anyone want to implement? :/
229 2016-08-27 10:10:17	0|sipa|leveldb 1.19 was released
230 2016-08-27 10:11:27	0|sipa|52 files changed, 1976 insertions(+), 429 deletions(-)
231 2016-08-27 10:11:33	0|sipa|(compared to 1.18)
232 2016-08-27 10:11:59	0|gmaxwell|doesn't sound too bad to review.
233 2016-08-27 10:12:47	0|sipa|they added a cache pruning
234 2016-08-27 10:13:28	0|gmaxwell|doesn't sound especially relevant then.
235 2016-08-27 10:13:36	0|sipa|it's been 2 years since their previous release
236 2016-08-27 10:13:46	0|sipa|glad to see activity :)
237 2016-08-27 10:15:11	0|sipa|ARM64 support for memory barriers seems relevant
238 2016-08-27 10:15:16	0|sipa|cache size estimation
239 2016-08-27 10:15:26	0|gmaxwell|oh the arm64 might fix performance on odroid c2
240 2016-08-27 10:28:07	0|sipa|this is interesting but not included: https://github.com/google/leveldb/pull/309
241 2016-08-27 10:29:30	0|gmaxwell|wumpus started benchmarking what that might look like.
242 2016-08-27 10:29:42	0|gmaxwell|odroid c2 has a crc32c accelerator that is sutiable too.
243 2016-08-27 11:03:35	0|luke-jr|found implementation of HFS+ compression
244 2016-08-27 11:03:44	0|sipa|where?
245 2016-08-27 11:03:49	0|luke-jr|SleuthKit
246 2016-08-27 11:06:53	0|luke-jr|xz -d <inodes.xz | while read inode filename; do filename="y/$filename"; p=$(dirname $filename); mkdir -vp $p; icat  ../5.hfs $inode >"$filename"; done
247 2016-08-27 11:08:18	0|luke-jr|the tarball is missing a bunch of stuff that looks like dummy files (but still 66 MB)
248 2016-08-27 11:08:36	0|sipa|luke-jr: nice!
249 2016-08-27 11:09:25	0|luke-jr|FWIW, I generated the inode list using mount and stat :p
250 2016-08-27 11:12:38	0|luke-jr|fls seems to be the tool
251 2016-08-27 11:15:49	0|sipa|sleuthkit added support for HFS+ in version 3.1
252 2016-08-27 11:16:14	0|luke-jr|I have 4.0.2
253 2016-08-27 11:16:57	0|sipa|ubuntu 12.04 has sleuthkit 3.2.3
254 2016-08-27 11:17:08	0|sipa|so i think we're good
255 2016-08-27 11:18:04	0|luke-jr|oh.
256 2016-08-27 11:18:08	0|luke-jr|except gitian doesn't like it :<
257 2016-08-27 11:18:28	0|sipa|due to the missing manpages...?
258 2016-08-27 11:18:33	0|luke-jr|hmm
259 2016-08-27 11:18:35	0|luke-jr|I think symlinks
260 2016-08-27 11:18:38	0|sipa|does it do a checksum check perhaps on the .tgz?
261 2016-08-27 11:18:41	0|luke-jr|can't find the lib for c++
262 2016-08-27 11:18:50	0|luke-jr|it can't, everyone's .tgz is different ;p
263 2016-08-27 11:19:41	0|luke-jr|yeah, I think it's missing symlinks
264 2016-08-27 11:29:41	0|luke-jr|fls ../5.hfs -rpF 154283 | perl -nle 'm/^(r|l)\S*\s(\d+)\:\s*(.*$)/ && print "$1 $2 $3"' | while read type inode filename; do filename="MacOSX10.11.sdk/$filename"; mkdir -p "$(dirname "$filename")"; if [ "$type" = "l" ]; then ln -s $(icat ../5.hfs $inode) "$filename"; else icat ../5.hfs $inode >"$filename"; fi; done
265 2016-08-27 11:32:18	0|luke-jr|tempting to figure out some kind of trick to download data on demand for this :P
266 2016-08-27 11:39:28	0|luke-jr|hm, surprised there's no single-file curl FUSE fs
267 2016-08-27 11:54:48	0|luke-jr|gitian build matches
268 2016-08-27 12:09:15	0|sipa|\o/
269 2016-08-27 12:22:54	0|btcdrak|wow
270 2016-08-27 12:26:19	0|luke-jr|?
271 2016-08-27 12:32:40	0|CodeShark|I think btcdrak just considers perl code to be inherently aesthetically pleasing :p
272 2016-08-27 12:33:34	0|luke-jr|lol
273 2016-08-27 12:33:40	0|luke-jr|latest draft has no perl sadly: http://codepad.org/1wMp5vse
274 2016-08-27 12:33:49	0|luke-jr|I'll wait until I'm actually awake to turn it into a PR
275 2016-08-27 12:34:19	0|luke-jr|-rw-r--r-- 1 luke-jr luke-jr  21M Aug 27 12:33 MacOSX10.11.sdk.tar.gz
276 2016-08-27 12:35:29	0|luke-jr|night
277 2016-08-27 12:42:04	0|phantomcircuit|luke-jr, the client can take ages to start if you closed it after you downloaded a bunch of blocks but before you processed them
278 2016-08-27 12:42:14	0|phantomcircuit|cause it processes all of them in AppInit2
279 2016-08-27 12:42:27	0|phantomcircuit|(actually it doesn't anymore so this shouldn't be an issue in 0.13.x)
280 2016-08-27 12:43:24	0|sipa|leveldb 1.19 should start up significantly faster
281 2016-08-27 12:44:19	0|phantomcircuit|oh and leveldb reads it's journal which is potentially very slow also
282 2016-08-27 13:24:25	0|GitHub3|[13bitcoin] 15sipa opened pull request #8610: Share unused mempool memory with coincache (06master...06sharemem) 02https://github.com/bitcoin/bitcoin/pull/8610
283 2016-08-27 13:29:19	0|GitHub62|[13bitcoin] 15sipa opened pull request #8611: Reduce default number of blocks to check at startup (06master...06fastcheck) 02https://github.com/bitcoin/bitcoin/pull/8611
284 2016-08-27 13:34:56	0|phantomcircuit|sipa, #8610 instead of doing that can you add a framework for limiting memory globally?
285 2016-08-27 13:35:25	0|GitHub51|[13bitcoin] 15sipa opened pull request #8612: Check for compatibility with download in FindNextBlocksToDownload (06master...06fixwitban) 02https://github.com/bitcoin/bitcoin/pull/8612
286 2016-08-27 13:35:32	0|sipa|phantomcircuit: i don't have 5 years
287 2016-08-27 13:35:36	0|phantomcircuit|it can probably even be as simple as a global memory limit goal and percentages for now
288 2016-08-27 13:36:06	0|sipa|percentages isn't good enough
289 2016-08-27 13:36:35	0|sipa|you need something that detects "oh, the mempool actually does not need its maximum usage... let's move some of its allocation elsewhere"
290 2016-08-27 13:36:49	0|sipa|but can change that back when the mempool grows
291 2016-08-27 13:37:00	0|phantomcircuit|percentages and an atomic of how much everything thinks it's using?
292 2016-08-27 13:37:09	0|sipa|what does that solve?
293 2016-08-27 13:37:18	0|phantomcircuit|"usage is below 90% of limit i can exceed my limit"
294 2016-08-27 13:37:38	0|phantomcircuit|i guess you need callbacks to apply memory pressure then
295 2016-08-27 13:37:42	0|sipa|yeah
296 2016-08-27 13:37:47	0|phantomcircuit|but you need that for sharing memory at all
297 2016-08-27 13:38:04	0|sipa|utxo set and mempool are things that are constantly checked anyway
298 2016-08-27 13:38:25	0|sipa|what this PR implements is mempool < maxmempool && coincache + mempool < maxtotal
299 2016-08-27 13:38:44	0|sipa|so arguably it already has something like a global limit (though it's just coincache + mempool)
300 2016-08-27 13:39:13	0|sipa|the mempool has complicated semantics wrt computing relay fee based on its size
301 2016-08-27 13:39:26	0|sipa|making its actual limit dynamic is harder, i think
302 2016-08-27 13:41:08	0|sipa|also, i don't think it's needed to for example give your mempool 10 GB of memory even if you have 64 GB available... that'll just make your memory a sewer for spam that nobody else on the network accepts
303 2016-08-27 14:01:47	0|GitHub42|[13bitcoin] 15sipa opened pull request #8613: [preview] LevelDB 1.19 (06master...06leveldb119) 02https://github.com/bitcoin/bitcoin/pull/8613
304 2016-08-27 19:53:36	0|gmaxwell|sipa: I think we should backport feeler connections. Beyond the security/robustness improvement, they'll help reduce the harm of network density loss w/ segwit.
305 2016-08-27 20:18:47	0|GitHub126|[13bitcoin] 15luke-jr opened pull request #8617: Include instructions to extract Mac OS X SDK on Linux using 7zip and SleuthKit (06master...06gitian_osx_extractor) 02https://github.com/bitcoin/bitcoin/pull/8617
306 2016-08-27 20:20:27	0|luke-jr|sipa: so have you confirmed already that LevelDB 1.19 is reasonably certain to not have forking changes? or are you assuming we'll do collectively that before merging/releasing?
307 2016-08-27 20:22:45	0|sipa|luke-jr: i'm reasonably certain, yes, but i encourage review
308 2016-08-27 20:23:12	0|sipa|i'm just pring it to bitcoin already to make testing easier