1 2017-06-16 00:47:23	0|bitcoin-git|[13bitcoin] 15gmaxwell opened pull request #10608: Add a comment explaining the use of MAX_BLOCK_BASE_SIZE. (06master...06size_comment) 02https://github.com/bitcoin/bitcoin/pull/10608
  2 2017-06-16 01:27:48	0|NicolasDorier|coding/review/idea sharing together. If you are interested, come in the channel ##tokyocore . My company Digital Garage can give us a nice place to go for coding. Note: I certaintly forgot some name, if you are contributor and wish to come, you are defintively welcome. The 25 June we freeze the list.
  3 2017-06-16 01:27:48	0|NicolasDorier|wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 instagibbs achow101 : Sorry to ping you all, but since some dev are in Tokyo around end of july for some conferences, I thought it can be a good idea to have such occasion to have 1 or 2 days of pure bitcoin core
  4 2017-06-16 01:31:38	0|cfields|NicolasDorier: thanks for arranging!
  5 2017-06-16 02:18:47	0|cfields|gitian builders: 0.14.2 detached sigs are up
  6 2017-06-16 04:33:41	0|cfields|NicolasDorier: I just rsvpd as I was going to be there already. Thanks again.
  7 2017-06-16 04:57:13	0|NicolasDorier|yay awesome
  8 2017-06-16 09:50:28	0|luke-jr|you know you're up too late when you don't remember what the bug you were trying to fix was. ._.
  9 2017-06-16 10:01:52	0|wumpus|cfields: thanks, pushed 0.14.2 signed sigs
 10 2017-06-16 11:51:06	0|jonasschnelli|cfields: with be49a294a240ec81a901af1aaabbba2172d38dc1 I get only a single lock report... haven't looked at your code thourgh
 11 2017-06-16 11:51:09	0|jonasschnelli|*though
 12 2017-06-16 12:02:42	0|jonasschnelli|wumpus: the pollBalanceThread is mainly responsible for UI freezes...
 13 2017-06-16 12:02:55	0|jonasschnelli|If I disable that poll thread,.. stuff runs much better
 14 2017-06-16 12:23:50	0|jonasschnelli|ryanofsky: I'll give it a test (polling but keeping the cache)
 15 2017-06-16 12:24:22	0|jonasschnelli|ryanofsky: But what I don't understand is why we would/should do a TRY_LOCK on cs_main every 250ms
 16 2017-06-16 12:25:09	0|jonasschnelli|cfields: nighly gitian build crashes on OSX (not tested on WIN/LINUX): https://0bin.net/paste/8-QKc7g9psBmYIGJ#msAwcpNd1CtU4KdEObG7tN4PK0XiIAbiELdwUtJLzui
 17 2017-06-16 13:29:47	0|jonasschnelli|sipa: you said CCoinsViewDB::Upgrade() can be cancelled (and continued) any time. Is that correct?
 18 2017-06-16 13:30:13	0|jonasschnelli|Adding a check for ShutdownRequested() and break the while (and write the possible batch) would make sense then?
 19 2017-06-16 13:33:23	0|michagogo|Will have 0.14.2 signed up shortly
 20 2017-06-16 13:33:30	0|michagogo|Doing a bit of VM maintenance
 21 2017-06-16 13:33:56	0|michagogo|Speaking of which: does anyone know what the status is of Gitian in Xenial?
 22 2017-06-16 13:35:00	0|michagogo|ISTR that for some reason we can't actually compile on Xenial and so the guest needs to remain Trusty, but can the Gitian (LXC) *host* be Xenial?
 23 2017-06-16 13:36:27	0|jonasschnelli|michagogo: Xenial should work fine IMO
 24 2017-06-16 13:41:54	0|sipa|jonasschnelli: i guess!
 25 2017-06-16 13:42:08	0|jonasschnelli|sipa: Seems to work here...
 26 2017-06-16 13:42:30	0|jonasschnelli|sipa: Can you tell me again how I can calculate the progress in that pcursor while loop?
 27 2017-06-16 13:43:49	0|sipa|i'll make a commit later
 28 2017-06-16 13:43:53	0|sipa|thanks for doing thid
 29 2017-06-16 13:45:55	0|jonasschnelli|okay. thanks
 30 2017-06-16 13:52:49	0|wumpus|jonasschnelli: interesting! I hadn't expected that, it should only update the balance if the TRY_ succeeds not freeze on the lock update
 31 2017-06-16 13:53:22	0|jonasschnelli|I don't know why currently but I know that its much faster with that PR
 32 2017-06-16 13:53:28	0|jonasschnelli|At least on OSX
 33 2017-06-16 13:53:41	0|jonasschnelli|Would be nice if someone could profile it on Linux / Win.
 34 2017-06-16 13:54:07	0|wumpus|but the try_lock on cs_main should not *itself* cause freezes
 35 2017-06-16 13:54:44	0|wumpus|if it runs the poll, and the lock is not available, the whole point would be that it doesn't spend time
 36 2017-06-16 13:54:57	0|wumpus|if that's not how it works it seems the whole TRY_ concept is broken
 37 2017-06-16 13:55:23	0|jonasschnelli|but what if the TRY_LOCK can acquire the lock every 250 ms and do the calculation...
 38 2017-06-16 13:55:45	0|wumpus|it should only do the computation if the balance is dirty, right?
 39 2017-06-16 13:56:31	0|wumpus|so in by far most cases if it gets the lock,  it immediately is supposed to notice the balance is not dirty, so doesn't recompute it
 40 2017-06-16 13:56:35	0|jonasschnelli|I tested three options, 1) master, 2) only atomic caches with polling, 3) like 2 but polling replaced with signal
 41 2017-06-16 13:56:40	0|wumpus|if only needs to be recomputed if something changed
 42 2017-06-16 13:56:47	0|jonasschnelli|wumpus: so, yes. It should only lock when the balances are dirty
 43 2017-06-16 13:57:09	0|jonasschnelli|But in my test, I sent 20 txes, so the balance was always dirty afterwards
 44 2017-06-16 13:57:26	0|jonasschnelli|I don't know why its much slower with the TRY_LOCK
 45 2017-06-16 13:57:29	0|wumpus|right
 46 2017-06-16 13:57:54	0|jonasschnelli|I guess it must be the QTimer / TryLock overhead?!
 47 2017-06-16 13:57:55	0|wumpus|well yes if it gets a transaction every 250ms, sure
 48 2017-06-16 13:58:06	0|ryanofsky|i'd think TRY_LOCK only when balances were dirty would give best of both worlds
 49 2017-06-16 13:58:09	0|jonasschnelli|I manually sent the tx... ~every 0.5s
 50 2017-06-16 13:58:10	0|wumpus|but GUI freezes happen anso to people that don't have anythign in their wallet
 51 2017-06-16 13:58:12	0|wumpus|on the first sync
 52 2017-06-16 13:58:37	0|jonasschnelli|ryanofsky: I though as well,.. but look at my profile results,.. they tell a different story
 53 2017-06-16 13:58:47	0|wumpus|so it can't be just something with transactions, though there's clearly something with transactions too...
 54 2017-06-16 13:58:52	0|ryanofsky|oh ok, i didn't see that listed as one your three tests
 55 2017-06-16 13:59:05	0|jonasschnelli|Look here: https://github.com/bitcoin/bitcoin/pull/10251#issuecomment-309021160
 56 2017-06-16 13:59:40	0|jonasschnelli|Master with only the atomic caches result in still 33% of the execution time in that poll function
 57 2017-06-16 13:59:50	0|wumpus|ok
 58 2017-06-16 14:00:05	0|jonasschnelli|While I can't find any call in my profiler running pure (all commits) of 10251
 59 2017-06-16 14:00:06	0|wumpus|33% of the time for something that gets called 4 times per second?
 60 2017-06-16 14:00:14	0|wumpus|is that a very large wallet?
 61 2017-06-16 14:00:15	0|jonasschnelli|*any call that related to the balance update
 62 2017-06-16 14:00:30	0|jonasschnelli|wumpus: not really large,.. I only did generate 1100
 63 2017-06-16 14:00:40	0|jonasschnelli|depends what "large" is
 64 2017-06-16 14:00:46	0|wumpus|it's interesting that the balance computation is so slow
 65 2017-06-16 14:00:59	0|wumpus|well if so, that's already large, it seems
 66 2017-06-16 14:01:00	0|jonasschnelli|I guess we loop 6 times over the complete mapWallet
 67 2017-06-16 14:01:06	0|wumpus|oh wow
 68 2017-06-16 14:01:17	0|jonasschnelli|for all balance types...
 69 2017-06-16 14:02:56	0|wumpus|ok, yes then I understand why things are so slow, though I still don't understand why it recomputes also the times the balance was not updated
 70 2017-06-16 14:03:14	0|wumpus|it's as if the wallet is always dirty, even though only once in 0.5s a transaction arrives
 71 2017-06-16 14:03:41	0|jonasschnelli|I'm not sure if it recomputes then...
 72 2017-06-16 14:03:46	0|wumpus|because in principle, this should result in the same load: with the 0.25s poll, it should compute every time after a transaction comes it
 73 2017-06-16 14:04:00	0|jonasschnelli|But when it does,.. it seems to take much longer.. don't know why. Maybe because of the QTimer internals
 74 2017-06-16 14:04:02	0|wumpus|with the per-transaction notification, it also does
 75 2017-06-16 14:04:21	0|wumpus|what if the transaction rate is higher though, e.g. with 10 transactions per second the polling shoudl be faster
 76 2017-06-16 14:04:33	0|jonasschnelli|Yes. Indeed.
 77 2017-06-16 14:04:47	0|jonasschnelli|The signal should filter that out... min 250ms delta or something
 78 2017-06-16 14:05:01	0|wumpus|the fixed polling is also to reduce the maximum amount of work done
 79 2017-06-16 14:05:12	0|jonasschnelli|yes.. indeed.
 80 2017-06-16 14:05:25	0|wumpus|yes, it could, though if done not carefully it means the data is always one update behind
 81 2017-06-16 14:05:40	0|wumpus|(e.g. if you would naively do "process this update only if the last was >250ms ago")
 82 2017-06-16 14:05:41	0|jonasschnelli|Could it be the QTimer overhead?
 83 2017-06-16 14:05:46	0|wumpus|no, I don't believe that
 84 2017-06-16 14:05:59	0|wumpus|not at a 4Hz frequency
 85 2017-06-16 14:06:13	0|wumpus|if you remove the code from the handler but keep the timer, it'd probably be the same
 86 2017-06-16 14:07:19	0|jonasschnelli|I guess we should then try to continue to measure the atomic cache commit on top of master (with the TRY_LOCK polling).
 87 2017-06-16 14:07:41	0|jonasschnelli|This is already much faster (I don't have numbers though)
 88 2017-06-16 14:07:46	0|jonasschnelli|s/is/feels/
 89 2017-06-16 14:08:07	0|wumpus|I'm really surprised by the result at least...
 90 2017-06-16 14:08:12	0|jonasschnelli|What could be that we falsely always set fForceCheckBalanceChanged
 91 2017-06-16 14:08:25	0|jonasschnelli|Could also be OSX only... who knows
 92 2017-06-16 14:08:33	0|jonasschnelli|But since we also have a windows issue...
 93 2017-06-16 14:08:39	0|jonasschnelli|I doubt it's OSX only
 94 2017-06-16 14:09:10	0|wumpus|btw: what is the advantage of having a dirty flag per kind of balance - won't every update to the wallet invalidate *all* of them?
 95 2017-06-16 14:09:48	0|wumpus|or are there changes that, say, only invalidate the immutable balance
 96 2017-06-16 14:10:29	0|ProfMac|I feel like this is the secret password to join a club:  a60d7c8dde9b77e7ff547976ce37db1fe98c71833003465befe650d6bc102b6b  bitcoin-0.14.1-aarch64-linux-gnu.tar.gz
 97 2017-06-16 14:10:37	0|wumpus|ProfMac: congrats!
 98 2017-06-16 14:10:44	0|wumpus|so you got it to work, awesome
 99 2017-06-16 14:11:04	0|jonasschnelli|ProfMac: nice one!
100 2017-06-16 14:11:04	0|ProfMac|I said bad things & stomped my foot a time or two.
101 2017-06-16 14:11:08	0|ryanofsky|i think dirty flag doesn't really make sense for qt, but it does make sense for rpc
102 2017-06-16 14:11:37	0|wumpus|so the balance computations loop over all transactions?
103 2017-06-16 14:11:45	0|wumpus|which is always done 6 times
104 2017-06-16 14:11:53	0|jonasschnelli|wumpus: because each balance type has its own routine to calculate, we probably should have caches/dirty-flags per routine
105 2017-06-16 14:12:03	0|jonasschnelli|wumpus: that's the root source!
106 2017-06-16 14:12:05	0|wumpus|I wonder if it'd make sense to roll it into one routine, that computes all 6 balalnces in one pass
107 2017-06-16 14:12:14	0|jonasschnelli|wumpus: +100
108 2017-06-16 14:12:26	0|wumpus|it would be more cache friendly at least
109 2017-06-16 14:12:34	0|jonasschnelli|I guess that was a "organic growing" issue... we added one type after another over tim,e
110 2017-06-16 14:12:56	0|wumpus|the added overhead of, in that loop over every transaction, adding up one more number is probably neglible
111 2017-06-16 14:13:14	0|jonasschnelli|indeed
112 2017-06-16 14:13:16	0|wumpus|while LOCKing the wallet six times and iterating over its contents six times is probably bad
113 2017-06-16 14:13:35	0|wumpus|in any case it's good that your PR already brings improvements
114 2017-06-16 14:13:40	0|wumpus|then we should probably merge it
115 2017-06-16 14:13:45	0|jonasschnelli|But now it comes...
116 2017-06-16 14:13:55	0|jonasschnelli|We call GetDepthInMainChain for each mapWallet tx
117 2017-06-16 14:14:03	0|jonasschnelli|Which does mapBlockIndex.find(hashBlock);
118 2017-06-16 14:14:27	0|jonasschnelli|I tried multiple times to cache the height... but seems to be relatively difficult to do it right
119 2017-06-16 14:14:46	0|wumpus|that would also be reduced with factor 6 if it were to be done in one go
120 2017-06-16 14:14:55	0|jonasschnelli|Yes.
121 2017-06-16 14:15:41	0|jonasschnelli|That would be related to 10251 but with a bigger ramification
122 2017-06-16 14:15:53	0|wumpus|I mean *ideally* the wallet would compute balance incremementally
123 2017-06-16 14:15:56	0|jonasschnelli|I think the pure atomic caches is a first step... even if we then can throw away all down to a sigle cache
124 2017-06-16 14:16:04	0|jonasschnelli|wumpus: Yes.
125 2017-06-16 14:16:14	0|jonasschnelli|It should keep basepoints at certain height/states
126 2017-06-16 14:16:21	0|wumpus|but if we had to do a pass over the entire wallet it's by far best to do it only once
127 2017-06-16 14:16:41	0|wumpus|right, it's extremely complex to get right, which is why it was never done
128 2017-06-16 14:16:54	0|jonasschnelli|Indeed, ... and we should probably do in – as most things – in a background thread (GUI wise)
129 2017-06-16 14:17:18	0|jonasschnelli|If the node coms where in a background thread, nobody would tackle this
130 2017-06-16 14:17:21	0|wumpus|e.g. there are so many small things that can change how a transaction is counted
131 2017-06-16 14:17:41	0|wumpus|right, if the computation was not in the GUI thread, it'd be much less bad
132 2017-06-16 14:17:49	0|wumpus|there would be some unnoticable lag
133 2017-06-16 14:17:53	0|ryanofsky|if you get rid of the dirty flag and compute balance with each transactions that effectively would move computation to a background thread. still might lock up gui because of holding cs_main though
134 2017-06-16 14:17:55	0|wumpus|instead of the thing hanging and complaining
135 2017-06-16 14:18:24	0|wumpus|ryanofsky: but that'd be crazy, it'd make receiving transactions quadratic
136 2017-06-16 14:18:33	0|wumpus|ryanofsky: if every transactions received causes a scan over the entire wallet
137 2017-06-16 14:18:40	0|wumpus|ryanofsky: that's why the dirty flag exists
138 2017-06-16 14:19:08	0|jonasschnelli|Balances should only be calculated if the user wants them
139 2017-06-16 14:19:11	0|ryanofsky|yes, what i meant above by dirty flag makes sense for rpc but not gui
140 2017-06-16 14:19:20	0|jonasschnelli|In the GUI, thats a bit different
141 2017-06-16 14:19:30	0|wumpus|ryanofsky: right now it makes sense for the GUI too, as it's only polled 4 times per second
142 2017-06-16 14:19:34	0|wumpus|ryanofsky: so it's naturally rate limited
143 2017-06-16 14:19:56	0|wumpus|ryanofsky: if you get 400 transactions per second, it still only computes 4 times per second
144 2017-06-16 14:19:58	0|ryanofsky|for gui users who are creating more than 4 transactions per second i guess
145 2017-06-16 14:20:07	0|jonasschnelli|Modern GUI frameworks often poll/call updates when the according element is visible
146 2017-06-16 14:20:13	0|wumpus|or "receiving"
147 2017-06-16 14:20:16	0|wumpus|this happens during initial sync
148 2017-06-16 14:20:19	0|wumpus|if you have a full wallet
149 2017-06-16 14:20:35	0|ryanofsky|oh, that's true. dirty flag makes sense there too
150 2017-06-16 14:20:57	0|wumpus|people always ignore initial sync for some reason, while that is the most common source of hangs, after initial sync the GUI is pretty ok
151 2017-06-16 14:20:58	0|ryanofsky|anyway to be clear i'm not suggesting getting rid of dirty flag
152 2017-06-16 14:21:24	0|wumpus|(initial sync or: more often, catching up after not running the node for a few days)
153 2017-06-16 14:21:33	0|ryanofsky|i'm just saying one way to move balance computation to "background thread" instead of gui thread is to compute it on tx notifications in cases where that makes sense
154 2017-06-16 14:22:10	0|wumpus|sure
155 2017-06-16 14:50:27	0|cfields|jonasschnelli: the lock report is printed when threads destruct. So, mostly at shutdown
156 2017-06-16 14:50:37	0|jonasschnelli|ah.. okay.. then my fault
157 2017-06-16 14:50:52	0|jonasschnelli|cfields: have you seen the crash on OSX in current master over gitian
158 2017-06-16 14:51:23	0|jonasschnelli|It must be one of the last 10 commits
159 2017-06-16 14:51:41	0|jonasschnelli|could it be: https://github.com/bitcoin/bitcoin/commit/cb24c8539d1098d1a61605b452ecfa11a693320d
160 2017-06-16 14:51:50	0|cfields|jonasschnelli: no, i haven't rebased my osx work in maybe ~1week, though
161 2017-06-16 14:51:58	0|cfields|I'll fire one up now
162 2017-06-16 14:52:17	0|jonasschnelli|The crash only happens when build via gitian
163 2017-06-16 14:52:24	0|cfields|jonasschnelli: could also be the crc32 thing
164 2017-06-16 14:52:24	0|jonasschnelli|local builds are fine...
165 2017-06-16 14:52:34	0|jonasschnelli|But nighly build from 15th works
166 2017-06-16 14:52:38	0|cfields|interesting
167 2017-06-16 14:52:40	0|jonasschnelli|Most be the last 10 commits
168 2017-06-16 14:52:45	0|cfields|jonasschnelli: you have binaries handy?
169 2017-06-16 14:52:54	0|jonasschnelli|yes...
170 2017-06-16 14:52:58	0|jonasschnelli|see: https://github.com/bitcoin/bitcoin/issues/10611
171 2017-06-16 14:53:08	0|jonasschnelli|use my builders build 176 vs 175
172 2017-06-16 14:53:23	0|cfields|perfect, thanks
173 2017-06-16 14:53:59	0|cfields|argh, i never added the osx debug symbols, did i?
174 2017-06-16 14:54:47	0|jonasschnelli|I guess no...
175 2017-06-16 14:55:39	0|cfields|:(
176 2017-06-16 15:07:49	0|cfields|jonasschnelli / sipa: https://0bin.net/paste/+O0wEWXmLDshuG97#oQTOyc4av4get3wvsrwRNwbKaTO7ODBrTotk5nbXcf3
177 2017-06-16 16:02:12	0|bitcoin-git|[13bitcoin] 15jnewbery opened pull request #10612: The young person's guide to the test_framework (06master...06templatefunctionaltest) 02https://github.com/bitcoin/bitcoin/pull/10612
178 2017-06-16 16:04:43	0|Victorsueca|anybody else having issues when cross-compiling 0.14.2 for windows in ubuntu 14?
179 2017-06-16 16:06:06	0|Victorsueca|I'm getting this thing on a machine that used to compile properly until now https://0bin.net/paste/q13F+WfsGqY4fkcW#4UXO33FPpzPvu7zb-Omj6zz8tHPGiAOH18/E5ti0xNO
180 2017-06-16 16:06:09	0|gribble|https://github.com/bitcoin/bitcoin/issues/4 | Export/Import wallet in a human readable, future-proof format · Issue #4 · bitcoin/bitcoin · GitHub
181 2017-06-16 16:07:17	0|sipa|Victorsueca: "issues" is pretty vague
182 2017-06-16 16:07:42	0|Victorsueca|sipa: yeah, more specifically with miniupnpc apparently
183 2017-06-16 16:07:47	0|Victorsueca|see the 0bin
184 2017-06-16 16:07:56	0|sipa|oh, sorry i missed the link
185 2017-06-16 16:30:39	0|spudowiar|o/ jnewbery
186 2017-06-16 17:02:35	0|cfields|jonasschnelli: ok, i think i have the crash worked out. I've taken the opportunity to learn more about asm
187 2017-06-16 17:09:35	0|sipa|cfields: my condolences
188 2017-06-16 17:10:12	0|cfields|sipa: heh. my mistake for not diving in decades ago
189 2017-06-16 17:15:08	0|cfields|sipa: clang is nice enough to use ebx for its stack canary, which cpuid clobbers. I'm trying to understand the PIC/ebx interaction fully before PRing something
190 2017-06-16 17:16:15	0|cfields|(context: rdrand detection)
191 2017-06-16 17:16:52	0|sipa|cfields: huh?
192 2017-06-16 17:16:58	0|sipa|my PR avoids clobbering ebx...
193 2017-06-16 17:17:16	0|cfields|more context: #10611
194 2017-06-16 17:17:17	0|gribble|https://github.com/bitcoin/bitcoin/issues/10611 | Gitian build (current master) crashes on OSX · Issue #10611 · bitcoin/bitcoin · GitHub
195 2017-06-16 17:17:40	0|cfields|sipa: my understanding is that the instruction still clobbers it, the compiler just isn't aware of that
196 2017-06-16 17:17:49	0|sipa|it does not clobber it
197 2017-06-16 17:18:01	0|sipa|the cpuid instruction clobbers it, but the whole asm block does not
198 2017-06-16 17:18:11	0|sipa|it moves the old value to a temp register first, and restores it
199 2017-06-16 17:19:22	0|cfields|grr, i was looking at my hacked up local source. I see that now, thanks.
200 2017-06-16 17:19:45	0|sipa|can you show me objdump -dC libbitcoin_util_a-random.o ?
201 2017-06-16 17:20:25	0|cfields|yea, sec, let me revert
202 2017-06-16 17:20:51	0|cfields|please don't tell the problem/solution though, I'd like to work it out
203 2017-06-16 17:21:42	0|sipa|i don't expect to learn anything interesting - just want to see if the clang compiler isn't doing something unexpected
204 2017-06-16 17:22:43	0|cfields|https://pastebin.com/raw/tS1TUPe0
205 2017-06-16 17:24:55	0|sipa|oooh!
206 2017-06-16 17:26:14	0|cfields|just tell me how deep i'm going to have to dive? :)
207 2017-06-16 17:27:54	0|sipa|hint: we'll need different asm code for 32bit and 64bit
208 2017-06-16 17:33:13	0|cfields|full canary address doesn't get restored via mov?
209 2017-06-16 17:34:45	0|sipa|indeed
210 2017-06-16 17:35:09	0|sipa|see why?
211 2017-06-16 17:36:33	0|cfields|vaguely. I'm so green I'm having trouble distinguishing values from addresses. Investigating.
212 2017-06-16 17:39:40	0|wumpus|oh, so the compiler isn't being told correctly what registers are being clobbered?
213 2017-06-16 17:40:11	0|sipa|wumpus: if you tell the compiler yoi clobber ebx, it will complain saying that it needs ebx for PIC
214 2017-06-16 17:40:16	0|sipa|and fail to compile
215 2017-06-16 17:40:29	0|sipa|so the code there manually saves and restores ebx
216 2017-06-16 17:41:01	0|sipa|i believe that is fixed in a later gcc
217 2017-06-16 17:41:07	0|cfields|gcc5
218 2017-06-16 17:41:28	0|cfields|i suppose it needs to be movq/rbx for 64bit instead?
219 2017-06-16 17:41:33	0|sipa|indeed
220 2017-06-16 17:41:35	0|wumpus|that seems kind of stupid, now you need to write the code in mind with the knowledge what every compiler might use for special things
221 2017-06-16 17:41:43	0|sipa|wumpus: indeed :(
222 2017-06-16 17:41:57	0|sipa|cfields: tmp needs to be a 64-bit int on 64-bit systems
223 2017-06-16 17:42:24	0|cfields|sipa: i've seen lots of rantings about gcc's x86 pic handling in the past, i'm beginning to understand the hatred now
224 2017-06-16 17:42:32	0|wumpus|(I suspect the cookie in ebx is not part of any official ABI convention, at least)
225 2017-06-16 17:43:59	0|cfields|well if you list ebx as a clobber, it sticks the cookie somewhere else. So my "fix" was totally accidental and wrong (and would ofc break on x86 pic).
226 2017-06-16 17:44:18	0|sipa|cfields: oh you're able to list ebx as a clobber?
227 2017-06-16 17:44:21	0|sipa|and it compiles?
228 2017-06-16 17:44:27	0|wumpus|in its defense, handling PIC on x86 32 bit is a nightmare
229 2017-06-16 17:44:40	0|sipa|in that case, we should probably do that on x86_64
230 2017-06-16 17:44:51	0|wumpus|modern architectures, and even less modern ones, have been designed with PIC in mind, but x86 never was
231 2017-06-16 17:45:32	0|cfields|sipa: yea
232 2017-06-16 17:46:06	0|sipa|you want to write a pr?
233 2017-06-16 17:47:26	0|cfields|sipa: sure, will be a while though. Still reading/experimenting/learning
234 2017-06-16 17:50:40	0|sipa|cfields: the issue is that the 32-bit ebx register and the 64-bit rbx register occupy the same space
235 2017-06-16 17:51:01	0|sipa|ebx is just a "view" of the lower 32 bits of rbx
236 2017-06-16 17:52:19	0|cfields|that makes sense
237 2017-06-16 17:52:57	0|sipa|and the "b" constraint can either mean ebx or rbx, depending on the size of the variable
238 2017-06-16 17:53:26	0|sipa|there are 8-bit and 16-bit views too
239 2017-06-16 17:55:21	0|cfields|so the compiler adapts the mov to the register size?
240 2017-06-16 17:55:28	0|sipa|yup
241 2017-06-16 17:55:34	0|sipa|the assembler, rather
242 2017-06-16 17:55:50	0|cfields|ok, got it
243 2017-06-16 17:55:56	0|cfields|thanks for the 101 :)
244 2017-06-16 18:07:32	0|cfields|sipa: https://pastebin.com/raw/68f8cKHd ?
245 2017-06-16 18:14:00	0|cfields|mm, apparently that's broken since cpuid overwrites eax
246 2017-06-16 18:15:49	0|sipa|you can list eax as an input/outpit register
247 2017-06-16 18:16:02	0|sipa|but i think just listing all 4 as outputs is easier
248 2017-06-16 18:16:28	0|cfields|right, so:https://pastebin.com/BhBAHeFK
249 2017-06-16 18:16:32	0|cfields|ok
250 2017-06-16 18:16:45	0|cfields|will just do that. PRing. thanks again.
251 2017-06-16 18:34:13	0|Victorsueca|sipa: had to go for a while, any news on my issue? need extra debug info?
252 2017-06-16 18:38:00	0|sipa|Victorsueca: i have no clue
253 2017-06-16 18:39:50	0|achow101|Victorsueca: are you using WSL?
254 2017-06-16 18:40:21	0|Victorsueca|achow101: yes, it has been working until now
255 2017-06-16 18:40:47	0|achow101|IIRC it wasn't working for the past few months
256 2017-06-16 18:40:55	0|achow101|(at least not on my machines)
257 2017-06-16 18:40:59	0|ProfMac|I successfully did a gitian build on 0.14.1.  It took days to grok the web pages, and about 7 hours to execute.  I have placed some shell scripts at https://github.com/a-mcintosh/gitian-bitcoin-shell-scripts and these scripts capture the workflow mentioned at https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md#create-a-new-virtualbox-vm
258 2017-06-16 18:42:00	0|jonasschnelli|7h?!
259 2017-06-16 18:42:07	0|jonasschnelli|DId you pass in -j2 or -j4?
260 2017-06-16 18:43:03	0|Victorsueca|the last version i've built succefully on my WSL was 0.14.1, now I'm on the same machine and it won't do the depends for 0.14.2
261 2017-06-16 18:43:10	0|achow101|Victorsueca: try https://github.com/bitcoin/bitcoin/issues/10269#issuecomment-307293086
262 2017-06-16 18:43:15	0|ProfMac|lol.  At this point I'm so fatigued, I don't even know my own name.  It does seem pretty slow.  Have a look at the scripts and see if you spot any trouble.
263 2017-06-16 18:43:34	0|sipa|ProfMac: machine without hardware virtualization?
264 2017-06-16 18:44:58	0|achow101|ProfMac: perhaps try using kvm instead of lxc? it will need to run on hardware and not in a vm though
265 2017-06-16 18:46:23	0|Victorsueca|achow101: that seems to work \o/
266 2017-06-16 18:46:27	0|Victorsueca|thanks
267 2017-06-16 18:46:29	0|ProfMac|Probably just not configured correctly.  One of the scripts creates the virtualbox, so that can be corrected.  5.7 GiB RAM, Intel Core 2 Duo CPU E8400 @ 3.00 GHz x 2 Ubuntu 14.04 LTS 2.0 TB online.
268 2017-06-16 18:47:21	0|ProfMac|what kind of build times to others have?  My impression was that it build the entire Trusty system from source during the process.
269 2017-06-16 18:48:27	0|ProfMac|I don't know how to peek inside gitian-builder/target-trusty-amd64
270 2017-06-16 18:50:13	0|jonasschnelli|ProfMac: my build times are at bitcoin.jonasschnelli.ch
271 2017-06-16 18:51:13	0|ProfMac|Could I just make a new Trusty VM and do the build there?  If I restore a "bare" snapshot before each build, what do I give up?
272 2017-06-16 18:54:09	0|ProfMac|jonasschnelli, thanks, but I get a 403 forbidden when I drop that into an address bar.
273 2017-06-16 18:54:32	0|jonasschnelli|https://bitcoin.jonasschnelli.ch
274 2017-06-16 18:54:41	0|jonasschnelli|Oh.. the SSL forwarder seems broken
275 2017-06-16 18:55:35	0|ProfMac|I'm at the page.  Nice homage to Matrix, unless like me you still see pink after a night at the CRT.
276 2017-06-16 18:59:12	0|ProfMac|Oh my.  21 minutes vs 7 hours.  LOL.
277 2017-06-16 19:02:03	0|bitcoin-git|[13bitcoin] 15theuni opened pull request #10614: random: fix crash on some 64bit platforms (06master...06fix-osx-crash) 02https://github.com/bitcoin/bitcoin/pull/10614
278 2017-06-16 19:02:22	0|cfields|jonasschnelli: ^^
279 2017-06-16 19:09:08	0|sipa|cfields: also, for reusable code (if we want to use hw sha instruction or something, you can use "xchg %1, %%ebx" instead of mov, and get the real ebx out
280 2017-06-16 19:09:37	0|cfields|sipa: heh, i meant to ask about that. that's what gcc does internally
281 2017-06-16 19:09:55	0|sipa|not suggesting to do that here, but it's hardly harder to write a generic "cpuid number -> give me a tuple of 4 uint32_t" function
282 2017-06-16 19:13:25	0|cfields|sipa: you mean just drop the special case and always stash b?
283 2017-06-16 19:13:55	0|sipa|cfields: i think your PR is fine
284 2017-06-16 19:14:13	0|sipa|but maybe later if we ever have multiple things to query cpuid for
285 2017-06-16 19:14:25	0|cfields|sipa: well i'm pretty sure I whined about it not being generic in your original PR :p
286 2017-06-16 19:15:20	0|jonasschnelli|cfields: nice. Just started a build: https://bitcoin.jonasschnelli.ch/build/181
287 2017-06-16 19:15:37	0|cfields|sipa: heh, yea: https://github.com/bitcoin/bitcoin/pull/10377#pullrequestreview-37172453
288 2017-06-16 19:35:45	0|bitcoin-git|13bitcoin/06master 14cc0ed26 15Pavel Janík: Supress struct/class mismatch warnings introduced in #10284.
289 2017-06-16 19:35:45	0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/c2ab38bdd57a...7a74f88a26cf
290 2017-06-16 19:35:46	0|bitcoin-git|13bitcoin/06master 147a74f88 15Pieter Wuille: Merge #10598: Supress struct/class mismatch warnings introduced in #10284...
291 2017-06-16 19:36:25	0|bitcoin-git|[13bitcoin] 15sipa closed pull request #10598: Supress struct/class mismatch warnings introduced in #10284 (06master...0620170615_FeeCalculation_structclass) 02https://github.com/bitcoin/bitcoin/pull/10598
292 2017-06-16 19:51:00	0|sipa|cfields: we're using gcc 4.8 for win and linux builds, which i think has a sufficiently mature LTO implementation; any opinion on using it in release builds?
293 2017-06-16 19:51:16	0|sipa|i don't know the status of LTO in clang
294 2017-06-16 19:52:16	0|cfields|sipa: my main concern is that we could see a discrepancy between release binaries and what devs run
295 2017-06-16 19:52:40	0|sipa|cfields: why?
296 2017-06-16 19:52:42	0|cfields|as even with gcc7, linking still takes a long time. I don't think we could enable it by default, it'd have to be in gitian
297 2017-06-16 19:53:08	0|sipa|it does take less memory too
298 2017-06-16 19:53:36	0|sipa|maybe a generic "optimized build" available from configure and one that isn't, and gitian uses optimized
299 2017-06-16 19:53:45	0|cfields|(not opposed, just considering the downsides)
300 2017-06-16 19:53:59	0|sipa|it's a good point to bring up
301 2017-06-16 19:54:23	0|cfields|sipa: yea, that sounds reasonable
302 2017-06-16 19:54:23	0|sipa|i like that it categorically removes the concern about whether code needs to go in a header or not :)
303 2017-06-16 19:54:32	0|cfields|heh
304 2017-06-16 19:55:13	0|cfields|speaking of which, i worked on pre-compiled headers a few days ago. Shaved roughly ~30% off of build time
305 2017-06-16 19:55:49	0|cfields|sipa: another one of the big benefits is that our deps can be lto'd as well. I suspect static lto'd qt would be a big win for filesize
306 2017-06-16 19:55:56	0|cfields|but again, that makes linking take forever
307 2017-06-16 19:59:36	0|TD-Linux|cfields, https://git.xiph.org/?p=opus.git;a=blob;f=celt/x86/x86cpu.c;h=080eb25e413d9e6587a419933d85ea9a6243b46e;hb=HEAD#l61
308 2017-06-16 20:00:42	0|jonasschnelli|TD-Linux: hah. Always research first how other did it. ;)
309 2017-06-16 20:02:11	0|jonasschnelli|*others
310 2017-06-16 20:03:14	0|cfields|heh, yep
311 2017-06-16 20:09:13	0|luke-jr|has anyone benchmarked LTO vs normal?
312 2017-06-16 20:24:33	0|luke-jr|jonasschnelli: please add sources for companies you add to https://en.bitcoin.it/wiki/Segwit_support
313 2017-06-16 20:53:05	0|bitcoin-git|[13bitcoin] 15luke-jr opened pull request #10615: RPC: Allow rpcauth configs to specify a 4th parameter naming a specific wallet (06master...06multiwallet_rpc) 02https://github.com/bitcoin/bitcoin/pull/10615
314 2017-06-16 22:03:34	0|bitcoin-git|13bitcoin/06master 14131a8ce 15practicalswift: Make clang-format use C++11 features (e.g. A<A<int>> instead of A<A<int> >)
315 2017-06-16 22:03:34	0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/7a74f88a26cf...d76e84a21416
316 2017-06-16 22:03:35	0|bitcoin-git|13bitcoin/06master 14d76e84a 15Pieter Wuille: Merge #10602: Make clang-format use C++11 features (e.g. A<A<int>> instead of A<A<int> >)...
317 2017-06-16 22:04:13	0|bitcoin-git|[13bitcoin] 15sipa closed pull request #10602: Make clang-format use C++11 features (e.g. A<A<int>> instead of A<A<int> >) (06master...06clang-format-cpp11) 02https://github.com/bitcoin/bitcoin/pull/10602
318 2017-06-16 22:42:59	0|midnightmagic|eek why is there only two of us signing the windows signed executables :-P
319 2017-06-16 23:04:47	0|cfields|sipa: I'm confused about https://github.com/bitcoin-core/leveldb/commit/196962ff01c39b4705d8117df5c3f8c205349950
320 2017-06-16 23:05:07	0|cfields|how does that manage to link? there's no definition that returns zero
321 2017-06-16 23:05:22	0|sipa|cfields: the implementation is in posix_sse.c
322 2017-06-16 23:05:31	0|sipa|which apparently gets linked in
323 2017-06-16 23:06:08	0|cfields|sipa: ah, that's our problem then. leveldb's build doesn't link that in
324 2017-06-16 23:06:17	0|cfields|...but that would mean windows is broken upstream
325 2017-06-16 23:06:25	0|sipa|windows does not _exist_ upstream
326 2017-06-16 23:06:34	0|cfields|oooooh :)
327 2017-06-16 23:07:03	0|cfields|hehe, got it. will fix.
328 2017-06-16 23:08:29	0|cfields|i see. so we patched up their build discovery stuff at one point, but stopped when we migrated it into our build
329 2017-06-16 23:08:48	0|sipa|right, and we don't run the tests for the windows build
330 2017-06-16 23:09:58	0|cfields|got it, thanks
331 2017-06-16 23:37:09	0|bitcoin-git|13bitcoin/06master 14f2fb132 15practicalswift: Net: Fix resource leak in ReadBinaryFile(...)...
332 2017-06-16 23:37:09	0|bitcoin-git|[13bitcoin] 15sipa pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/d76e84a21416...de8db47b7ff3
333 2017-06-16 23:37:10	0|bitcoin-git|13bitcoin/06master 14de8db47 15Pieter Wuille: Merge #10587: Net: Fix resource leak in ReadBinaryFile(...)...
334 2017-06-16 23:37:36	0|bitcoin-git|[13bitcoin] 15sipa closed pull request #10587: Net: Fix resource leak in ReadBinaryFile(...) (06master...06fopen-not-followed-by-fclose-in-all-states-of-the-universe) 02https://github.com/bitcoin/bitcoin/pull/10587