1 2018-02-13 01:19:48	0|ProfMac|was there ever a time when GetBlockWork used the actual hash instead of nBits to calculate work?
  2 2018-02-13 01:20:17	0|bitcoin-git|[13bitcoin] 15promag opened pull request #12419: wallet: Force distinct destinations in CWallet::CreateTransaction (06master...062018-02-distinct-destinations) 02https://github.com/bitcoin/bitcoin/pull/12419
  3 2018-02-13 01:20:17	0|gmaxwell|no, that wouldn't be correct.
  4 2018-02-13 01:20:30	0|ProfMac|tell me more.
  5 2018-02-13 01:22:13	0|gmaxwell|the hash reflects more than actual work, by a factor of two on average.  It also has very high variance. meaning that if you got a block with ten times that 'work' expected (which you'd get in 1/10) blocks you could just delay announcing it until a couple blocks had passed, then announce it and be very likely to reorg them out.
  6 2018-02-13 01:22:48	0|promag|gmaxwell: regarding duplicate outputs #12419
  7 2018-02-13 01:22:50	0|gribble|https://github.com/bitcoin/bitcoin/issues/12419 | Force distinct destinations in CWallet::CreateTransaction by promag · Pull Request #12419 · bitcoin/bitcoin · GitHub
  8 2018-02-13 01:23:41	0|ProfMac|Yes, gmaxwell, I agree with all that.
  9 2018-02-13 01:23:51	0|ProfMac|Thanks.
 10 2018-02-13 01:24:24	0|ProfMac|Not unlike "curling" or "corn hole"
 11 2018-02-13 02:34:26	0|gmaxwell|n1bor: conversation continued in #bitcoin he posted bench logs which indicated to me that he was IO bound very badly. Today he followed up with,
 12 2018-02-13 02:34:29	0|gmaxwell|n1bor: 18:28:24 < murrayn> gmaxwell, just to follow up from last night: 1) i shutdown gracefully and restarted, and the reindex started from the beginning.  From what you said, it shouldn't do that? and 2) it was definitely disk IO bound. doing it on an SSD this time. much better
 13 2018-02-13 02:34:49	0|gmaxwell|(1) was something sipa warned him about in here, that restarting with -reindex-chainstate would restart the reindex.
 14 2018-02-13 02:39:22	0|midnightmagic|doh
 15 2018-02-13 03:10:44	0|meshcollider|looks like contrib/bitcoin-cli.bash-completion needs an update, hasn't been updated for ages
 16 2018-02-13 03:24:11	0|esotericnonsense|k, so i'm looking at the JSON-RPC 2.0 spec stuff now. 'strict compliance' requires two main changes as far as i can see. one of them I've made gated behind a flag 'strictjsonrpcspec' (better names accepted)
 17 2018-02-13 03:24:41	0|esotericnonsense|that's the 'client must send jsonrpc=2.0, server must send jsonrpc=2.0, only one of 'error' or 'result' should be returned'
 18 2018-02-13 03:25:33	0|esotericnonsense|the other bit is that if the client does not send an id with a request, the server should send a blank response. within a batch it should just be skipped. so if you send [a id=1, b, c id=3] you should get back [aresult, cresult].
 19 2018-02-13 03:26:54	0|esotericnonsense|i think that's enough to make it fully spec compliant. the former one is enough to get it to work with libjsonrpccpp, could probably write some unit tests based on the jsonrpc spec as well, manual tests i've done look fine
 20 2018-02-13 03:54:28	0|bitcoin-git|[13bitcoin] 15MarcoFalke reopened pull request #12349: shutdown: fix crash on shutdown with reindex-chainstate (06master...06fix-qt-shutdown) 02https://github.com/bitcoin/bitcoin/pull/12349
 21 2018-02-13 04:54:50	0|kallewoof|What's going on here? Either that comment is wrong, or the check is wrong... https://github.com/bitcoin/bitcoin/blame/master/src/script/interpreter.cpp#L1233 (it says nOut out of range, but it checks nIn against txTo.vout.size()...)
 22 2018-02-13 04:57:12	0|kallewoof|I guess it's because single requires the in to sign the same index out. Just weird looking.
 23 2018-02-13 05:06:29	0|aj|kallewoof: yeah, "no vout corresponding to nIn" or something would make more sense to me as a comment
 24 2018-02-13 05:06:39	0|aj|kallewoof: code seems right at least
 25 2018-02-13 05:09:40	0|aj|kallewoof: wow, that comment used to be a log output saying nOut out of range
 26 2018-02-13 05:14:37	0|kallewoof|aj: yeah agreed!
 27 2018-02-13 09:32:08	0|bitcoin-git|[13bitcoin] 15laanwj pushed 5 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/c997f8808256...2dbc4a4740cd
 28 2018-02-13 09:32:09	0|bitcoin-git|13bitcoin/06master 14c32cf9f 15John Newbery: [tests] Add P2PDataStore class...
 29 2018-02-13 09:32:09	0|bitcoin-git|13bitcoin/06master 14cc046f6 15John Newbery: [tests] Reduce NodeConn connection logging from info to debug
 30 2018-02-13 09:32:10	0|bitcoin-git|13bitcoin/06master 14359d067 15John Newbery: [tests] Fix flake8 warnings in invalidtxrequest
 31 2018-02-13 09:32:38	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #11771:  [tests] Change invalidtxrequest to use BitcoinTestFramework (06master...06refactor_invalidtxrequest) 02https://github.com/bitcoin/bitcoin/pull/11771
 32 2018-02-13 09:59:26	0|bitcoin-git|13bitcoin/06master 14a71c56a 15Luke Dashjr: clientversion: Use full commit hash for commit-based version descriptions...
 33 2018-02-13 09:59:26	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/2dbc4a4740cd...f4f4f51f1a94
 34 2018-02-13 09:59:27	0|bitcoin-git|13bitcoin/06master 14f4f4f51 15Wladimir J. van der Laan: Merge #11966: clientversion: Use full commit hash for commit-based version descriptions...
 35 2018-02-13 10:00:05	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #11966: clientversion: Use full commit hash for commit-based version descriptions (06master...06ver_full_commit_hash) 02https://github.com/bitcoin/bitcoin/pull/11966
 36 2018-02-13 10:40:18	0|wumpus|hmmm
 37 2018-02-13 10:40:19	0|wumpus|boost::interprocess::file_lock& lock = locks.emplace(pathLockFile.string(), pathLockFile.string().c_str()).first->second;
 38 2018-02-13 10:40:19	0|wumpus|util.cpp:384:121: note: in instantiation of template class 'std::__1::pair<const std::__1::basic_string<char>, boost::interprocess::file_lock>' requested here
 39 2018-02-13 10:41:22	0|wumpus|full error here: https://github.com/bitcoin/bitcoin/issues/12413
 40 2018-02-13 10:41:37	0|wumpus|anyone have an idea what can cause this c++ error monstriosity?
 41 2018-02-13 11:03:25	0|wumpus|so it looks like "static std::map<std::string, boost::interprocess::file_lock> locks;" is illegal with openbsd's clang + boost version combo
 42 2018-02-13 11:03:41	0|bitcoin-git|[13bitcoin] 15Sjors opened pull request #12421: [qt] navigate to  transaction history page after send (06master...062018/02/qt-goto-transactions-after-send) 02https://github.com/bitcoin/bitcoin/pull/12421
 43 2018-02-13 11:04:40	0|Lauda|Will there be rc4?
 44 2018-02-13 11:11:29	0|provoostenator|Lauda: there's two open PR's which I suspect need to be fixed before rc4: https://github.com/bitcoin/bitcoin/milestone/30
 45 2018-02-13 11:12:30	0|Lauda|Alright, thanks. I'll wait for those.
 46 2018-02-13 11:17:32	0|bitcoin-git|[13bitcoin] 15laanwj opened pull request #12422: util: Use unique_ptr in locks map (06master...062018_01_openbsd_util_fix) 02https://github.com/bitcoin/bitcoin/pull/12422
 47 2018-02-13 11:24:45	0|gmaxwell|wumpus: can you explain why it works inside a unique ptr? (I am just curious)
 48 2018-02-13 11:26:04	0|wumpus|gmaxwell: no, I have no idea why it fails in the first place
 49 2018-02-13 11:26:28	0|wumpus|tbh this is really over my limit of understanding C++
 50 2018-02-13 11:27:33	0|wumpus|well, the point of having pointers in a map instead of whole objects makes it 'simpler' to work with layout in memory I suppose
 51 2018-02-13 11:28:03	0|wumpus|as for some reason the whole lock objects can't be moved
 52 2018-02-13 11:28:13	0|wumpus|and pointers, obviously, can
 53 2018-02-13 11:28:59	0|gmaxwell|yea, I was wondering if values in maps had to be relocatable, but quick googling didn't answer the question for me.
 54 2018-02-13 11:29:56	0|wumpus|it must be some edge case that was solved in either a newer version of clang, or in a different version of boost, as I've not seen the issue on other platforms
 55 2018-02-13 11:30:54	0|wumpus|the 'map' here is used to avoid locking a directory multiple times I guess? this structure is write-only, no lookups are ever done in this map, so there should be no performance difference whatsoever
 56 2018-02-13 11:31:00	0|wumpus|gah
 57 2018-02-13 11:31:20	0|goatpig|from the look of the error log
 58 2018-02-13 11:31:25	0|goatpig|this has to do with the ctor
 59 2018-02-13 11:31:27	0|goatpig|not the map itself
 60 2018-02-13 11:31:42	0|wumpus|so emplace returns the current record if one already exists, right?
 61 2018-02-13 11:32:15	0|wumpus|I dont' think this construction even accomplishes what it is meant to accomplish
 62 2018-02-13 11:32:59	0|goatpig|emplace is like insert afaik, returns a pair of <iter, bool>
 63 2018-02-13 11:33:20	0|goatpig|the iterator points at the inserted element or the already existing one, the bool tells you if it was added (true) or fetched (false)
 64 2018-02-13 11:34:18	0|wumpus|ok that depends on what does lock->try_lock does on a lock that is already locked
 65 2018-02-13 11:34:30	0|goatpig|you mean within the same thread?
 66 2018-02-13 11:34:40	0|goatpig|or another one?
 67 2018-02-13 11:34:41	0|wumpus|within the same process
 68 2018-02-13 11:34:46	0|goatpig|i think it throws
 69 2018-02-13 11:34:47	0|goatpig|not sure
 70 2018-02-13 11:34:48	0|wumpus|this is a per-process lock, not per-thread
 71 2018-02-13 11:35:15	0|wumpus|I wonder what the intent is for LockDirectory being called on the same directory multiple times
 72 2018-02-13 11:35:25	0|goatpig|but that wouldnt have anything to do compile, that would just be runtime error
 73 2018-02-13 11:35:31	0|wumpus|I know.
 74 2018-02-13 11:35:41	0|wumpus|I'm just doubting that this code does what it is meant to do, at all
 75 2018-02-13 11:35:53	0|goatpig|typically that's to make sure you got the lock
 76 2018-02-13 11:35:57	0|wumpus|separate from what it takes to compile it
 77 2018-02-13 11:36:05	0|goatpig|now if the same threads is trying to get a lock multiple times
 78 2018-02-13 11:36:26	0|goatpig|it presupposes the methods that are accessed can be used on their own
 79 2018-02-13 11:36:33	0|wumpus|it should probably at least check the boolean it gets back from emplace
 80 2018-02-13 11:36:40	0|wumpus|whether the lock already exists
 81 2018-02-13 11:36:54	0|wumpus|otherwise it will lock an old lock another time
 82 2018-02-13 11:36:56	0|goatpig|that part is like
 83 2018-02-13 11:37:00	0|goatpig|idk id have to look at the code
 84 2018-02-13 11:37:02	0|goatpig|but it seems pointless
 85 2018-02-13 11:37:14	0|goatpig|why hold a reference to the lock in the scope? that's usless, the map perpetuates it
 86 2018-02-13 11:37:17	0|wumpus|either that's a no-op, or it throws, but it's pointless
 87 2018-02-13 11:37:33	0|goatpig|and that could be your compile issue
 88 2018-02-13 11:37:37	0|goatpig|that these locks are not copiable
 89 2018-02-13 11:37:46	0|goatpig|ie just get rid of the referencing altgother
 90 2018-02-13 11:37:46	0|wumpus|defining the map itself causes the compile issue
 91 2018-02-13 11:37:53	0|wumpus|referencing is not the problem
 92 2018-02-13 11:37:57	0|goatpig|boost::interprocess::file_lock& lock = locks.emplace(pathLockFile.string(), pathLockFile.string().c_str()).first->second;
 93 2018-02-13 11:38:03	0|goatpig|get rid of the reference and try
 94 2018-02-13 11:38:06	0|gmaxwell|in general I wouldn't expect a lock to be copyable.
 95 2018-02-13 11:38:07	0|goatpig|if that doesnt bork other stuff
 96 2018-02-13 11:38:17	0|wumpus|goatpig: I tried, see discussion in https://github.com/bitcoin/bitcoin/issues/12413
 97 2018-02-13 11:38:20	0|goatpig|i.e. just do locks.emplace(pathLockFile.string(), pathLockFile.string().c_str());
 98 2018-02-13 11:38:31	0|gmaxwell|what the heck is this doing?
 99 2018-02-13 11:38:33	0|gmaxwell|anyways?
100 2018-02-13 11:38:43	0|wumpus|goatpig: as I said above, defining the map itself already causes the error, even if you don't access it at all!
101 2018-02-13 11:39:11	0|wumpus|gmaxwell: that was also my question, what is the semantics supposed to be
102 2018-02-13 11:39:27	0|goatpig|you could just ninja that my making it a map of shared_ptr<lock>
103 2018-02-13 11:39:33	0|goatpig|by*
104 2018-02-13 11:39:47	0|gmaxwell|maybe we can remove it all entirely? thats always the best fix when possible. :P
105 2018-02-13 11:39:51	0|wumpus|goatpig: that's what I did in #12322 (but with unique_lock)
106 2018-02-13 11:39:53	0|gribble|https://github.com/bitcoin/bitcoin/issues/12322 | Docs: Remove step making cloned repository world-writable for Windows build. by murrayn · Pull Request #12322 · bitcoin/bitcoin · GitHub
107 2018-02-13 11:40:00	0|wumpus|eh #12422
108 2018-02-13 11:40:02	0|gribble|https://github.com/bitcoin/bitcoin/issues/12422 | util: Use unique_ptr in locks map (fix OpenBSD 6.2 build) by laanwj · Pull Request #12422 · bitcoin/bitcoin · GitHub
109 2018-02-13 11:40:49	0|goatpig|wumpus: sure, dunno what the code really intents. i'd prefer unique_ptr when possible
110 2018-02-13 11:41:00	0|goatpig|but if it's trying to reference the lock, then it should be using shared_ptrs
111 2018-02-13 11:41:08	0|wumpus|no, shared_ptre is not necessary
112 2018-02-13 11:41:14	0|wumpus|the lock does not leave the scope at all
113 2018-02-13 11:41:26	0|goatpig|the lock exists within the map
114 2018-02-13 11:41:29	0|wumpus|yes
115 2018-02-13 11:41:47	0|wumpus|and after the change, it exists inside a unique pointer
116 2018-02-13 11:41:58	0|goatpig|if you kill the map, if there is a thread with the ptr/reference, it will blow up
117 2018-02-13 11:41:59	0|wumpus|which is unique, inside the map
118 2018-02-13 11:42:11	0|wumpus|no one has a reference to it
119 2018-02-13 11:42:15	0|wumpus|please read the code
120 2018-02-13 11:42:22	0|wumpus|no reference leaves the function
121 2018-02-13 11:43:01	0|goatpig|checking
122 2018-02-13 11:43:02	0|wumpus|the destructor will be called when the process has shut down, after the main function, by automatic cleanup
123 2018-02-13 11:43:12	0|wumpus|by that time no one will be using the data dir anymore
124 2018-02-13 11:43:49	0|wumpus|the map exists to hold on to the objects until the process shuts down
125 2018-02-13 11:45:58	0|goatpig|so the locks persist the process?
126 2018-02-13 11:46:01	0|goatpig|or rather
127 2018-02-13 11:46:12	0|goatpig|no method tries to clean them up
128 2018-02-13 11:46:37	0|wumpus|as I said, they'll be automatically cleaned up by the runtime framework
129 2018-02-13 11:47:35	0|wumpus|<gmaxwell> in general I wouldn't expect a lock to be copyable. <- they are; it's only moving them at most, not copying
130 2018-02-13 11:50:11	0|goatpig|wumpus: is it assumed that code gets accessed by only a single thread?
131 2018-02-13 11:50:22	0|gmaxwell|goatpig: this isn't an ordinary lock
132 2018-02-13 11:50:27	0|gmaxwell|it's a file locking thing
133 2018-02-13 11:50:36	0|wumpus|goatpig: yes, it's only used during initialization, by the init thread
134 2018-02-13 11:50:42	0|gmaxwell|used to keep seperate processes from muckign about with the same data directories.
135 2018-02-13 11:50:51	0|wumpus|goatpig: oh... I'm actually not sure anymore
136 2018-02-13 11:50:52	0|goatpig|i understand that much, but it's not thread safe, so im assuming this is every hit once
137 2018-02-13 11:51:01	0|goatpig|err
138 2018-02-13 11:51:03	0|goatpig|hit ever once
139 2018-02-13 11:51:03	0|wumpus|wallet directories complicated this
140 2018-02-13 11:51:24	0|goatpig|i mean the semantics of the method suggest this is accessed just to check the lock is held at all
141 2018-02-13 11:51:24	0|wumpus|as soon as dynamic opening of wallets is supported, this can be called from the RPC thread
142 2018-02-13 11:51:33	0|wumpus|but right now it's only used from init
143 2018-02-13 11:52:19	0|wumpus|it's a good question, the function as it is is certainly not thread safe
144 2018-02-13 11:52:22	0|goatpig|you are adding elements to a map. ideally it should check the map has the element, otherwise grab a mutex, add the element, then go back to regular execution
145 2018-02-13 11:53:09	0|wumpus|it should grab a mutex before accessing the map at all
146 2018-02-13 11:53:12	0|gmaxwell|^
147 2018-02-13 11:53:14	0|wumpus|*if* it's supposed to be thread safe
148 2018-02-13 11:53:20	0|goatpig|ok, sure
149 2018-02-13 11:53:39	0|gmaxwell|and its trivial here in any case.. since the purpose of the map is just lifetime management for the contained objects.
150 2018-02-13 11:53:46	0|wumpus|yes
151 2018-02-13 11:54:10	0|goatpig|that would be if the method was not meant to return state of the lock
152 2018-02-13 11:54:23	0|goatpig|the semantics just suggest to me that it could be accessed by another thread
153 2018-02-13 11:54:47	0|wumpus|I think the intent is to return true if the process has a lock on the directory, whether it's new or not. Not sure the code matches that intent, at the moment.
154 2018-02-13 11:54:53	0|gmaxwell|nah its just init only now. though wumpus is right that dynamic wallet opening/closing will cause accesses from rpc.
155 2018-02-13 11:55:38	0|wumpus|I've asked meshcollider (who wrote the code) in the issue just in case
156 2018-02-13 11:56:00	0|gmaxwell|for wallet opening we probably want it to return true if it takes a lock and false if it already has it or can't take it.
157 2018-02-13 11:56:37	0|meshcollider|wumpus: I think ryanofsky suggested the currently used approach as a comment on the PR
158 2018-02-13 11:56:41	0|wumpus|gmaxwell:  it's a lock on a directory, not on the wallet, bdb takes care of not opening the same wallet multiple times
159 2018-02-13 11:56:53	0|wumpus|gmaxwell: there can be multiple wallets in a directory
160 2018-02-13 11:57:13	0|gmaxwell|wumpus: ah point.
161 2018-02-13 11:57:21	0|gmaxwell|Carry on then.
162 2018-02-13 12:06:10	0|wumpus|we could certainly use unit tests for that function
163 2018-02-13 12:10:01	0|wumpus|meshcollider: ok adding jnewbery to reviewers
164 2018-02-13 12:12:55	0|meshcollider|jnewbery or ryanofsky?
165 2018-02-13 12:13:12	0|meshcollider|wumpus: your PR looks good to me
166 2018-02-13 12:13:41	0|wumpus|eh, sorry, yes ryanofsky, added the wrong one
167 2018-02-13 12:15:02	0|meshcollider|wumpus: didn't even think about the dynamic loading/unloading case / thread safety when I wrote it, apologies
168 2018-02-13 12:15:17	0|meshcollider|Was just trying to fix the issue for 0.16
169 2018-02-13 12:15:44	0|wumpus|meshcollider: no worries, we found that issue before it became a problem at least
170 2018-02-13 13:11:22	0|provoostenator|You have got to be kidding... anyone had to jump through these hoops before? http://htmlpreview.github.io/?https://github.com/JKSH/QtSdkRepoChooser/blob/master/doc/index.html
171 2018-02-13 13:15:12	0|wumpus|no, not really, though I've had to build qt for embedded devices, the source code is huge, it takes a long time and it's a pain to configure for cross compile
172 2018-02-13 13:15:35	0|wumpus|it pretty much regards itself as an operating system in itself
173 2018-02-13 13:22:29	0|wumpus|we can be really happy that linux distributions simply package that stuff pre-built
174 2018-02-13 14:18:18	0|midnightmagic|/w/w 59
175 2018-02-13 14:20:01	0|cfields|wumpus: seems it's moot now, but I think std::piecewise_construct might be what's missing for the emplace
176 2018-02-13 14:23:06	0|cfields|as usual, spoke too soon. Red herring. Nevermind.
177 2018-02-13 14:25:20	0|wumpus|I'll look at that one, never heard of it
178 2018-02-13 14:26:53	0|cfields|wumpus: see for example: https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L574
179 2018-02-13 15:02:21	0|hkjn0|a bit lazy q: am I reading depends/ docs right that there's currently no way to use other platforms than x86_64 to build e.g. bdb?
180 2018-02-13 15:03:15	0|hkjn0|so if I'm compiling code on armv7l and want wallet support, best thing to do is to get on x86_64 instead, and crosscompile towards arm as necessary?
181 2018-02-13 15:04:00	0|hkjn0|(I did get bdb4.8 compiled from source on armv7l outside of the depends system, but had trouble getting it to be noticed by the build system..)
182 2018-02-13 15:06:14	0|wumpus|hkjn0: the depends system assumes it's building on x86_64, however if you want to build db4 outside the depends system there's the script  contrib/install_db4.sh and the flags  BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include"
183 2018-02-13 15:07:41	0|wumpus|would be nice if the depends system also supported building *from* other architectures, though
184 2018-02-13 15:10:19	0|hkjn0|ah, great, I bet I was just misisng BDB_LIBS or somesuch.. will try out the contrib/ script
185 2018-02-13 15:10:33	0|wumpus|esp. at some point we'd like to do gitian builds from other architectures
186 2018-02-13 15:12:00	0|hkjn0|right. I've been setting up these weird build environments already, would be happy to try to contribute to gitian work or otherwise support other build platforms, just need to understand more of how current system hangs together first.. :)
187 2018-02-13 16:22:41	0|cfields|depends doesn't assume build is x86_64. It's the only arch that really gets tested, though
188 2018-02-13 16:23:09	0|cfields|but, all of the machinery is there to auto-detect the build platform and try to cope.
189 2018-02-13 16:24:46	0|cfields|hkjn0: if you try building native on arm and run into issues, feel free to ping me
190 2018-02-13 16:28:25	0|hkjn0|cfield, thanks! building with contrib/install_db4.sh + BDB_LIBS it suggested for ./configure as wumpus suggested seems to work fine so far
191 2018-02-13 16:29:44	0|hkjn0|(before that I tried a plain `make` in depends/ and it failed on armv7l, but it is possible I am missing some packages)
192 2018-02-13 16:34:02	0|cfields|hkjn0: well if you'd be willing, I'd be happy to step through the depends with you and try to get it patched to build
193 2018-02-13 16:35:11	0|cfields|like wumpus said, it'd be nice to have several working build arches
194 2018-02-13 16:44:42	0|hkjn0|cfields: sure, let me know how you'd like to proceed.. do we take it off the main channel to not clutter it up too much?
195 2018-02-13 16:45:08	0|cfields|hkjn0: sure, #bitcoin-build
196 2018-02-13 16:55:00	0|michagogo|wumpus/cfields: Do you remember (on a describe-in-a-couple-sentences-off-the-top-of-your-head level) what the actual issue is with Xenial and why it's broken?
197 2018-02-13 16:55:27	0|michagogo|And does Artful(/Bionic) just work?
198 2018-02-13 16:55:42	0|michagogo|(with cross-compilation for Windows, I mean)
199 2018-02-13 16:56:36	0|michagogo|I tried looking through the issues on GitHub and got confused by the various different threads and things people tried and experienced
200 2018-02-13 16:56:37	0|cfields|michagogo: by default, the toolchain is configured for win threads rather than posix, which don't exist (at least with that gcc version)
201 2018-02-13 16:56:52	0|cfields|further, the ssp is busted
202 2018-02-13 16:57:31	0|michagogo|And were those fixed in later versions, e.g. Artful?
203 2018-02-13 16:58:13	0|cfields|I believe so. But we should be switching to our own toolchains for 0.17 anyway. Or if not a full switch, probably at least for mingw.
204 2018-02-13 16:59:11	0|michagogo|That sounds like it would be a good thing
205 2018-02-13 17:00:24	0|michagogo|I ask because last night and today I spent a bunch of hours trying to see how hard it would be to just backport the version of mingw-w64 that's in artful/bionic to xenial
206 2018-02-13 17:01:17	0|michagogo|I tried the straightforward "`backportpackage` into a PPA", but ran into chains of missing dependencies
207 2018-02-13 17:02:01	0|michagogo|I won't bore you with everything I tried and looked at, but at this point I've come to the realization that I don't know enough about Ubuntu packaging and how all that works, and beyond that, more important stuff
208 2018-02-13 17:02:18	0|cfields|michagogo: I can upload a stand-alone mingw toolchain soon, if it would help
209 2018-02-13 17:02:21	0|michagogo|Like, what is mingw-w64, vs what is gcc, vs what is gcc-mingw-w64
210 2018-02-13 17:02:23	0|cfields|heh
211 2018-02-13 17:02:27	0|michagogo|and how do all those interact
212 2018-02-13 17:02:44	0|cfields|right
213 2018-02-13 17:02:57	0|cfields|well, in case it helps...
214 2018-02-13 17:03:26	0|michagogo|In #ubuntu-devel someone said: 18:34:22 <rbasak> michagogo: I would step back and consider your approach. I don't know what problem you're solving, but there might be an easier way.
215 2018-02-13 17:04:10	0|michagogo|I explained that the version in xenial was broken, and they said this: 18:39:39 <rbasak> Also, if the reason it doesn't work in Xenial is due to a bug, we're quite happy in general to cherry-pick the fix for all Xenial users to benefit.
216 2018-02-13 17:04:28	0|michagogo|But somehow I suspect if it were that simple it would have already happened...
217 2018-02-13 17:04:38	0|cfields|to build a mingw-w64 toolchain, you need: mingw-w64 headers (think kernel headers), mingw-64 runtime (think glibc), gcc, and binutils. And obviously they need to be built in a way that lets them play nicely.
218 2018-02-13 17:05:04	0|michagogo|Right. So the thing is, I don't actually know the difference between mingw-w64 and gcc
219 2018-02-13 17:05:09	0|michagogo|Well, that's not entirely true
220 2018-02-13 17:05:15	0|michagogo|I sort of know what gcc is
221 2018-02-13 17:05:35	0|michagogo|What I don't know is, what is mingw-w64, and how does it interact with/based on/use gcc
222 2018-02-13 17:07:09	0|cfields|mingw-w64 is a shim that allows for posix-y code to work on windows.
223 2018-02-13 17:08:50	0|michagogo|So what is gcc-mingw-w64?
224 2018-02-13 17:09:20	0|cfields|so you can build a compiler (gcc) that's able to produce win32 code, but it wouldn't do much good if you couldn't use standard unixy apis.
225 2018-02-13 17:10:21	0|cfields|so, for example, mingw-w64 can use either win or pthread threading models. Configured without pthread support, you'll have trouble with lots of applications. That's one of our issues.
226 2018-02-13 17:13:12	0|michagogo|So gcc-mingw-64, say, is a version of GCC that builds the mingw-w64 layer into the binaries it produces?
227 2018-02-13 17:13:40	0|cfields|it's probably easiest to just think of mingw-w64 as a libc for windows.
228 2018-02-13 17:14:18	0|michagogo|I guess at this point the knowledge that I'm missing is: 1) what _exactly_ the term "toolchain" refers to and how you build one
229 2018-02-13 17:14:38	0|michagogo|2) How Ubuntu packages work together to make up a toolchain
230 2018-02-13 17:15:09	0|michagogo|And all kinds of other things related to those, like how exactly Ubuntu packaging works
231 2018-02-13 17:15:22	0|cfields|well, the packaging is an entirely different beast
232 2018-02-13 17:16:02	0|michagogo|I'm thinking I'll drop the issue for now, since I don't really have a ton of time and I suspect it would take me at least a few days to sort all these things out
233 2018-02-13 17:16:46	0|michagogo|I mean, regarding what rbasak said over in #ubuntu-devel, are the problems we have simple bugs with focused fixes that can be cherry-picked, or more fundamental problems that can't really be fixed without a wholesale upgrade to a newer version of MinGW?
234 2018-02-13 17:17:28	0|cfields|a toolchain is the group of progs that are required to produce a working binary. Typically that's target kernel/os headers in some form, target libc, host compiler, other host apps (in linux, binutils provides ar/ranlib/etc.), and a linker (also provided by binutils on linux)
235 2018-02-13 17:17:37	0|jcorgan|michagogo: it's been a little while since i've used gcc-mingw-w64 but i recall in ubuntu that installing that package pulls in all the accessories (runtime, binutils, etc.) that cfields mentioned
236 2018-02-13 17:17:59	0|michagogo|And actually, is the issue just with the MinGW component? Meaning, say one were to rebuild the gcc-mingw-w64 packages with the same GCC version and just an upgraded MinGW, would that work?
237 2018-02-13 17:19:04	0|jcorgan|in late to the conversation here, so not sure you're exact goal
238 2018-02-13 17:19:27	0|cfields|michagogo: the thread thing is just a config option. There's also the stack issue, which I haven't looked into for a while, but afaik that's no longer an issue in newer ubuntu versions. So it was either an upstream bug that's been fixed, or a config problem that ubuntu has fixed
239 2018-02-13 17:19:53	0|michagogo|jcorgan: Well, at the end of the day, here's my big-picture goal
240 2018-02-13 17:20:02	0|michagogo|Make https://github.com/bitcoin/bitcoin/blob/master/doc/build-windows.md#building-for-64-bit-windows die in a fire
241 2018-02-13 17:20:30	0|michagogo|Specifically, the lines telling you to sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu zesty universe"; sudo apt update;sudo apt upgrade
242 2018-02-13 17:20:55	0|michagogo|Because besides that not working, because zesty is EOL, that's just going to hose your system
243 2018-02-13 17:21:28	0|jcorgan|ah, i only have experience going from LTS to LTS, never use the ones in between
244 2018-02-13 17:21:56	0|michagogo|Even doing that, you never ever just add the newer repository like that
245 2018-02-13 17:22:00	0|michagogo|There's an upgrader
246 2018-02-13 17:22:37	0|michagogo|So my thought was, well, can I get the packages from the newer version brought back to xenial?
247 2018-02-13 17:23:04	0|jcorgan|it might be possible, but that way lies madness
248 2018-02-13 17:23:25	0|michagogo|So it seems.
249 2018-02-13 17:23:51	0|michagogo|I tried just naively `backportpackage`ing into a PPA
250 2018-02-13 17:24:10	0|jcorgan|you're on zesty and need packages that only exist later than that?
251 2018-02-13 17:24:22	0|cfields|michagogo: yea, I'm not sure it's worth the trouble. We already have a plan for resolving this, and it means that we don't have to rely on Ubuntu anymore
252 2018-02-13 17:24:24	0|michagogo|I’m not on zesty
253 2018-02-13 17:25:11	0|michagogo|I’m actually not an active Ubuntu (or Linux) user day to day
254 2018-02-13 17:25:14	0|jcorgan|you could see if the packages you need are in the 'backports' repo
255 2018-02-13 17:25:19	0|michagogo|Nah
256 2018-02-13 17:25:36	0|michagogo|My idea was to *get* a newer mingw-w64 into -backports
257 2018-02-13 17:26:25	0|michagogo|I filed the “please backport” bugs
258 2018-02-13 17:26:29	0|cfields|michagogo: well, that certainly wouldn't be a bad thing. It's definitely not just us.
259 2018-02-13 17:26:45	0|jcorgan|i expect that would require specialized knowledge of how gcc/binutils/etc are packaged that a typical user wouldn't normally need to know
260 2018-02-13 17:27:07	0|jcorgan|and personally i'd look for an alternative way to solve whatever problem
261 2018-02-13 17:27:32	0|michagogo|I had the naive thought that, well, it should be easy to backport, because it’s a cross-compiler, not anything that the system uses
262 2018-02-13 17:28:26	0|michagogo|jcorgan: AIUI “whatever problem” is “mingw-w64 on Xenial is very broken”
263 2018-02-13 17:31:03	0|jcorgan|personally i'd see if the must-be-xenial constraint can be relaxed. but if you wanted a challenge you could learn enough about how it works to help fix the issue. that may not be worth your time, but it could also be fun and very educational.
264 2018-02-13 17:31:22	0|jcorgan|(depends on you)
265 2018-02-13 17:31:25	0|michagogo|A no-changes backport failed, firstly, because it build-depends on gcc-7
266 2018-02-13 17:32:09	0|michagogo|Which must-be-Xenial constraint?
267 2018-02-13 17:32:24	0|michagogo|The thing is, 1) that’s the latest LTS
268 2018-02-13 17:32:52	0|jcorgan|right, i use that extensively
269 2018-02-13 17:32:54	0|michagogo|And 2) that’s the version that you get with WSL, and as far as I know trying to upgrade just breaks it
270 2018-02-13 17:33:32	0|jcorgan|ah, another piece of the picture i was unaware of
271 2018-02-13 17:34:08	0|michagogo|So the “constraint” is that we really want to let users on Xenial cross-compile is
272 2018-02-13 17:34:12	0|jcorgan|you're in a maze of twisty passages, all alike
273 2018-02-13 17:34:16	0|michagogo|cross-compile us*
274 2018-02-13 17:36:02	0|michagogo|I mean, it’s possible that if I knew Ubuntu packaging and had enough time I could try just reverting the changes that made it build with gcc 7
275 2018-02-13 17:36:05	0|jcorgan|so, getting a backport made of mingw does seem like the right way to go, but the number of people who are likely to know how to do that is probably small
276 2018-02-13 17:36:14	0|michagogo|And bring it back to 5.whatever
277 2018-02-13 17:36:59	0|michagogo|For all I know it might be just changing the build-depends line back to point at the gcc version that’s in Xenial
278 2018-02-13 17:37:07	0|jcorgan|sure
279 2018-02-13 17:38:00	0|jcorgan|but there could also be some specific bug in gcc5 that caused gcc7 to be needed.
280 2018-02-13 17:38:09	0|michagogo|I doubt it
281 2018-02-13 17:38:24	0|michagogo|I mean, I shouldn’t t say that
282 2018-02-13 17:38:49	0|jcorgan|that does sound like a (relatively) easy first thing to try, and you might get lucky
283 2018-02-13 17:39:05	0|michagogo|But I’m pretty sure that the upgrade to gcc 6 in... yakkety or zesty, I think? And to 7 in bionic, was just part of a wholesale upgrade to those versions
284 2018-02-13 17:39:52	0|jcorgan|i'm going to be in pain when i start looking at migrating everything to 18.04, aren't I?
285 2018-02-13 17:40:40	0|michagogo|But besides not knowing even the basics of Ubuntu packaging, I’d need the ability to go over all the other changes to the gcc-mingw-w64 source package and figure out which ones are related to the gcc 5/7 stuff, vs which ones should be there anyway
286 2018-02-13 17:40:57	0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #12424: Fix rescan test failure due to unset g_address_type, g_change_type (06master...06pr/scang) 02https://github.com/bitcoin/bitcoin/pull/12424
287 2018-02-13 17:41:01	0|michagogo|jcorgan: define “everything”
288 2018-02-13 17:41:34	0|jcorgan|most of the systems i use/support are 16.04, some still 14.04
289 2018-02-13 17:41:48	0|jcorgan|it sounds like a lot of changes are coming in 18.04
290 2018-02-13 17:41:54	0|michagogo|No idea
291 2018-02-13 17:42:06	0|jcorgan|anyway, this is straying a bit from coredev
292 2018-02-13 17:42:11	0|michagogo|I mean, it’s 2 years of progress in the overall Linux ecosystem
293 2018-02-13 17:42:18	0|michagogo|Bringing everything up to date
294 2018-02-13 17:42:38	0|michagogo|So yeah, some things may break…
295 2018-02-13 17:42:48	0|jcorgan|well, the older LTS versions get new packages for everything that isn't a fundamental system change
296 2018-02-13 17:43:25	0|jcorgan|and the backports repo even lets you run newer kernels and video drivers
297 2018-02-13 17:44:29	0|jcorgan|it's usually the lower-level stuff like sysvinit/upstart/systemd that causes me pain
298 2018-02-13 17:47:44	0|michagogo|Get new packages? What do you mean?
299 2018-02-13 17:48:22	0|michagogo|My understanding is that -updates only has bug fixes, ~never new wholesale versions of software that have features
300 2018-02-13 17:49:51	0|jcorgan|maybe i'm thinking of PPAs, where authors make recent versions of their software for several different ubuntu releases
301 2018-02-13 17:49:59	0|michagogo|Yep
302 2018-02-13 17:50:17	0|michagogo|With a PPA you can give people who decide to use you whatever you want, whenever you want
303 2018-02-13 17:50:52	0|michagogo|(Including installing a backdoor on their systems. It’s important that you trust the owners of PPAs you use.)
304 2018-02-13 17:51:08	0|jcorgan|well, i hope you figure out how to get teh mingw/WSL stuff working
305 2018-02-13 17:51:32	0|michagogo|Yeah, I think I'm just going to drop the issue at this point
306 2018-02-13 17:52:09	0|michagogo|cfields said that we're going to be using our own toolchain at some point (soon? ish?)
307 2018-02-13 17:52:59	0|cfields|michagogo: yes. Ubuntu versions/packages won't matter for gitian/travis after that.
308 2018-02-13 17:53:27	0|michagogo|cfields: I'm thinking mostly about the case of someone wanting to cross-build binaries for themselves
309 2018-02-13 17:53:56	0|michagogo|Either just not wanting to go through the hassle of setting up gitian, or with something like WSL
310 2018-02-13 17:54:15	0|michagogo|I really don't like the fact that we're giving users broken instructions, especially ones that (were they to work) have them completely mess up their system
311 2018-02-13 17:54:16	0|cfields|michagogo: sure. None of the new stuff will be required, ofc.
312 2018-02-13 17:54:39	0|michagogo|(https://github.com/bitcoin/bitcoin/blob/master/doc/build-windows.md#building-for-64-bit-windows)
313 2018-02-13 17:55:29	0|michagogo|And of course, there's the fact that mingw is broken for everyone else
314 2018-02-13 18:45:36	0|bitcoin-git|[13bitcoin] 15richardkiss opened pull request #12425: Add some script tests (06master...06feature/bool_tests) 02https://github.com/bitcoin/bitcoin/pull/12425
315 2018-02-13 20:24:37	0|gmaxwell|Note: if anyone here uses bitmessage, it has a remote code execution bug. (deserializing using eval)
316 2018-02-13 20:25:01	0|sipa|what is it written in?
317 2018-02-13 20:26:06	0|gmaxwell|Python.
318 2018-02-13 20:26:33	0|contrapumpkin|is today national facepalm security day or something
319 2018-02-13 20:29:57	0|esotericnonsense|deserializing using eval ???
320 2018-02-13 20:30:04	0|esotericnonsense|-_-.
321 2018-02-13 20:35:10	0|instagibbs|the vulnerability is being exploited in the wild as well
322 2018-02-13 20:35:39	0|instagibbs|not sure about success, but trying to run windows executables, grab electrum wallets, the like
323 2018-02-13 20:37:27	0|gmaxwell|esotericnonsense: deserializing using eval is a thing I've seen many times w/ javascript, php, and python.
324 2018-02-13 20:37:49	0|contrapumpkin|java does its own version of that too, if you use their native serialization
325 2018-02-13 20:39:14	0|esotericnonsense|gmaxwell: oh sure it's a thing. i just feel like it's one of the most famous 'don't do this' footguns
326 2018-02-13 20:39:31	0|esotericnonsense|evidently not
327 2018-02-13 20:48:19	0|mmgen|wouldn't use of the json module protect against this?
328 2018-02-13 20:49:40	0|mmgen|first json.loads(), then deserialize?
329 2018-02-13 20:52:27	0|reardencode|mmgen: yes.
330 2018-02-13 20:52:41	0|mmgen|strange they didn't do that
331 2018-02-13 20:59:08	0|mmgen|the fix: https://github.com/Bitmessage/PyBitmessage/commit/3a8016d31f517775d226aa8b902480f4a3a148a9
332 2018-02-13 21:02:32	0|contrapumpkin|that still seems like an iffy fix, even though the scope for mischief is far more limited
333 2018-02-13 21:03:14	0|contrapumpkin|it'll basically call an arbitrary method specified by untrusted data on the specified class
334 2018-02-13 21:03:28	0|bitcoin-git|[13bitcoin] 15MarcoFalke opened pull request #12426: qt: Initialize members in WalletModel (06master...06Mf1802-qtInitializeMembersWalletModel) 02https://github.com/bitcoin/bitcoin/pull/12426
335 2018-02-13 21:07:10	0|mmgen|contrapumpkin: hopefully just an emergency fix. far better than nothing in any case
336 2018-02-13 21:07:41	0|contrapumpkin|yup :)
337 2018-02-13 21:17:34	0|mmgen|seems like try: assert data[""
338 2018-02-13 21:18:08	0|mmgen|seems like try: assert data[""] in ('message','vote') would have been safer
339 2018-02-13 21:19:10	0|contrapumpkin|the hoops people go through to avoid writing a proper parser... 😢
340 2018-02-13 21:19:42	0|mmgen|since 'message' and 'vote' are the only permissible values
341 2018-02-13 21:19:53	0|sipa|it's getting a bit off topic for this channel
342 2018-02-13 21:19:59	0|contrapumpkin|sorry :) I'll shut up
343 2018-02-13 21:21:08	0|mmgen|sipa: sorry
344 2018-02-13 21:21:47	0|sipa|np!
345 2018-02-13 21:22:37	0|mmgen|sipa: though it **was** gmaxwell who brought up the topic
346 2018-02-13 21:23:46	0|sipa|i'm aware, and i participated too; no blame here
347 2018-02-13 21:23:57	0|sipa|just noting that the discussion is getting too far removed from the topic here
348 2018-02-13 21:24:46	0|mmgen|mmgen: i know, i'll really shut up now ;)
349 2018-02-13 21:29:13	0|bitcoin-git|[13bitcoin] 15sipa opened pull request #12427: Make signrawtransaction accept P2SH-P2WSH redeemscripts (06master...06201802_signrawp2shp2wsh) 02https://github.com/bitcoin/bitcoin/pull/12427
350 2018-02-13 21:30:55	0|gmaxwell|doh, how'd we miss that
351 2018-02-13 21:32:19	0|grafcaps|hah
352 2018-02-13 21:38:33	0|sipa|gmaxwell: it was actually partially being addressed in #11708
353 2018-02-13 21:38:36	0|gribble|https://github.com/bitcoin/bitcoin/issues/11708 | Add P2SH-P2WSH support to signrawtransaction and listunspent RPC by MeshCollider · Pull Request #11708 · bitcoin/bitcoin · GitHub
354 2018-02-13 21:41:06	0|promag|sipa: is it possible to add a test for #12427?
355 2018-02-13 21:41:07	0|gribble|https://github.com/bitcoin/bitcoin/issues/12427 | Make signrawtransaction accept P2SH-P2WSH redeemscripts by sipa · Pull Request #12427 · bitcoin/bitcoin · GitHub
356 2018-02-13 21:41:22	0|sipa|promag: yes, will look into that
357 2018-02-13 21:41:29	0|sipa|just wanted to have the PR visible as soon as possible
358 2018-02-13 21:51:10	0|Chris_Stewart_5|sipa: Is the basic idea that if we can sign the p2sh redeem script, we should be able to sign P2SH(P2WSH())?
359 2018-02-13 22:51:41	0|eklitzke|this is kind of cool, i got a build of bitcoind working with systemtap markers (in this case I'm marking CCoinsViewCache flushes): https://gist.github.com/eklitzke/8bf6957fe886ddec36cde737d69ac6f5
360 2018-02-13 23:00:53	0|gmaxwell|eklitzke: whats the advantage of that?
361 2018-02-13 23:13:46	0|eklitzke|i'm going to add more probes and then use that to measure some work i'm trying to do to the ccoinscacheview, e.g. i want to make the cache into a real writeback cache and get cache hit statistics out of it
362 2018-02-13 23:14:42	0|eklitzke|today if you wanted to get cache hit statistics from CCoinCacheView::GetCoin you'd have to hack up some existing rpc to dump the information (or dump it to a log), with stap/dtrace probes that would be much easier and less invasive
363 2018-02-13 23:16:15	0|warren|Does anyone have working gitian with qemu-kvm instead of lxc? I want to compare notes.
364 2018-02-13 23:18:36	0|gmaxwell|eklitzke: you'll defeat its purpose entirely
365 2018-02-13 23:20:28	0|gmaxwell|eklitzke: the cache's purpose for existing is not to act as a cache, it's purpose is to act as a buffer to prevent writes from hitting the database entirely. And also to allow atomic updates to keep the database consistent with the chain at a block level (this is not as important since we now use a replay for consistency).  It also satisfies reads, but if you defeat that it doesn't make a huge p
366 2018-02-13 23:20:34	0|gmaxwell|erformance difference in IBD.
367 2018-02-13 23:21:46	0|gmaxwell|okay maybe on a non-SSD the read cache matters somewhat more, to be fair I haven't benchmarked it with defeated read caching on a spinning disk.
368 2018-02-13 23:21:48	0|eklitzke|i have thought through these issues, you can increase the amount of coins in memory during ibd in a way that's safe and still batches disk writes
369 2018-02-13 23:21:57	0|eklitzke|yes this is primarily for spinning disks
370 2018-02-13 23:22:05	0|eklitzke|or worse, people's cloud vm instances
371 2018-02-13 23:22:46	0|sipa|eklitzke: i've experimented with various cache flushing strategies before which try to predict which entries will be more useful in the near future, and keep them longer
372 2018-02-13 23:22:52	0|gmaxwell|eklitzke: the purpose of the replay stuff we did in 0.15.x was in part to enable incremental flushing, so yes, I also see advantages there.
373 2018-02-13 23:23:04	0|sipa|in practice, anything that reduced the ability to avoid writes slowed things down
374 2018-02-13 23:23:14	0|gmaxwell|but what we did find is that trying to achieve a higher read hit rate is mostly pointless.
375 2018-02-13 23:23:32	0|gmaxwell|(on a SSD at least)
376 2018-02-13 23:24:08	0|gmaxwell|I fear we've more or less given up on performance on non-SSDs :(  esp since many large spinning disks are shingled now and have horrifying performance.
377 2018-02-13 23:24:51	0|gmaxwell|Incremenal flushing would also give the advantage of getting rid of flushing latency spikes and perhaps better overlap IO and validation.
378 2018-02-13 23:25:00	0|eklitzke|the flushing code right now is pretty inefficient, anyway databases like mysql do this by having a dirty buffer pool watermark that causes them to flush (potentially a large amount of data), and then they retaing data in memory that was flushed to disk if it's not dirty
379 2018-02-13 23:25:54	0|eklitzke|if you kept exactly the same semantics wrt flushing a large amount of data but didn't automatically expire the old coins from memory that would be an improvement (in my opinion)
380 2018-02-13 23:26:21	0|eklitzke|would love to chat with you about it next week, as i'm participating in the chaincode labs hacker residency program
381 2018-02-13 23:26:25	0|gmaxwell|eklitzke: We've measured that, it doesn't help.
382 2018-02-13 23:26:47	0|eklitzke|i've also measured it and shown that it does
383 2018-02-13 23:26:49	0|gmaxwell|Thats the whole reason I brought the subject up-- so you don't assume read caching is important.
384 2018-02-13 23:27:18	0|gmaxwell|eklitzke: Interesting, I'd love to see those results.
385 2018-02-13 23:27:19	0|eklitzke|if there are any writeups or mailing list posts about experiments in this area i'd be interested to read them
386 2018-02-13 23:27:40	0|gmaxwell|eklitzke: probably in the PR introducing the replay on start there are some of them.
387 2018-02-13 23:27:50	0|eklitzke|ok, i'll take a look
388 2018-02-13 23:29:20	0|sipa|eklitzke: i'll be at the residency on 22nd and 23rd
389 2018-02-13 23:29:24	0|gmaxwell|I'd offer you the specific code I benchmarked but I don't have access to the system with it anymore.
390 2018-02-13 23:31:02	0|sipa|eklitzke: the bitcoin utxo set is pretty unique as a data set in that a very large number of newly written entries are almost immediately after deleted again
391 2018-02-13 23:31:36	0|gmaxwell|We tested many configurations.  For an example, one of the specific cases we tested was having flush leave the entries in but marking them as non-dirty and existing in the backing store, and seperately removing non-dirty entries when full.  What appeared to be the case is that avoiding the read hit seemed basically irrelevant, because if i got read, that means it is getting spent, which means th
392 2018-02-13 23:31:42	0|gmaxwell|ere is going to be an erasure which must hit the disk.
393 2018-02-13 23:32:04	0|TD-Linux|since many large spinning disks are shingled now and have horrifying performance. <- these are still really uncommon
394 2018-02-13 23:32:34	0|gmaxwell|TD-Linux: they seem to be 100% of people showing up and talking about their node performance on spinning disks, but now that I say that outloud...
395 2018-02-13 23:32:35	0|sipa|eklitzke: but we should talk more about this
396 2018-02-13 23:33:03	0|eklitzke|i would like to change some of the leveldb settings too, but that's another matter entirely; for instance leveldb is configured to only open 64 files at once, so during ibd or a reindex you see the process thrashing a lot in a pattern where it opens an lbd file, mmaps it, munmaps it, and closes it
397 2018-02-13 23:33:09	0|eklitzke|that's another matter entirely though
398 2018-02-13 23:33:20	0|gmaxwell|eklitzke: by all means if you have results, great.
399 2018-02-13 23:33:53	0|gmaxwell|eklitzke: we suffer from FD exhaustion issues currently. :(
400 2018-02-13 23:34:35	0|eklitzke|yeah I talked to BlueMatt about it a bit yesterday, the thing is you can mmap a file, close the fd, and the mmap mapping is still valid
401 2018-02-13 23:34:56	0|gmaxwell|ah, cute.
402 2018-02-13 23:34:58	0|eklitzke|so if you didn't mind the extra memory overhead from the PTEs, you could potentially mmap all of the ldb files at once and not actually have 1000+ ldb files open
403 2018-02-13 23:35:05	0|eklitzke|doing *all* of them is probably too agressive
404 2018-02-13 23:35:13	0|gmaxwell|leveldb doesn't use mmap on 32 bit hosts.
405 2018-02-13 23:35:28	0|gmaxwell|and on 64-bit the only harm is the PTE as you note, which is pretty minor.
406 2018-02-13 23:37:21	0|gmaxwell|eklitzke: if you want to get into leveldb internals there are probably a lot of other gains. E.g. our keys are almost uniformly random, and so bisections in leveldb can probably be replaced with a secant search.  (and if our keys aren't random enough, we could fix that)   leveldb block size is tunable too
407 2018-02-13 23:37:45	0|gmaxwell|I think I found larger blocks to be a loss, but that might not be as true with a secant search.
408 2018-02-13 23:38:19	0|eklitzke|yeah I was actually surprised that leveldb itself wasn't doing the mmap trick I just described (or at least have it as a tuneable parameter), I want to dig deeper into their code
409 2018-02-13 23:39:00	0|promag|ryanofsky: friendly ping https://github.com/bitcoin/bitcoin/pull/11687#pullrequestreview-94272991
410 2018-02-13 23:40:51	0|gmaxwell|eklitzke: I have a believe which I've not been able to validate yet, that a lot of the performance limitations of the current dbcache is all the malloc activity... that often doesn't show up too clearly in profiles. We'd talked about changing the main dbcache to an open hashtable with fixed size entries (and some special indirection to an exception map for unusually large entries).. but it's a l
411 2018-02-13 23:40:57	0|gmaxwell|ot of work to just try it and see if it helps.
412 2018-02-13 23:42:10	0|eklitzke|interesting because glibc malloc already has systemtap probes bulitin, since 2.24 i think, so that would go well with my systemtap branch because you could write stap scripts that correlated activity in malloc with events in bitcoind
413 2018-02-13 23:58:24	0|Madscotslad|Wow nice to see a full IRC room its been a while :) hey folks.
414 2018-02-13 23:58:51	0|luke-jr|if you think this is full, you should see #bitcoin