1 2017-03-15 00:01:58	0|gmaxwell|TD-Linux: the link below the comment luke adds.
  2 2017-03-15 00:04:25	0|TD-Linux|oh duh
  3 2017-03-15 01:29:13	0|bitcoin-git|[13bitcoin] 15askmike opened pull request #9994: [trivial/doc] clearify current blockchain size (06master...06update-doc) 02https://github.com/bitcoin/bitcoin/pull/9994
  4 2017-03-15 01:43:27	0|bitcoin-git|[13bitcoin] 15askmike closed pull request #9994: [trivial/doc] clarify current blockchain size (06master...06update-doc) 02https://github.com/bitcoin/bitcoin/pull/9994
  5 2017-03-15 01:47:56	0|bitcoin-git|[13bitcoin] 15askmike opened pull request #9995: [trvial/doc] clarify current blockchain size (06master...06update-doc-2) 02https://github.com/bitcoin/bitcoin/pull/9995
  6 2017-03-15 04:03:52	0|NicolasDorier|afk11_: next time you need a block you can use https://api.qbit.ninja/blocks/[block]?format=raw where [block] can be height, hash, or tip. They can be combined with an offset like https://api.qbit.ninja/blocks/tip-2?format=raw to the block at (tip-1) height
  7 2017-03-15 04:04:26	0|NicolasDorier|sorry tapi for testnet
  8 2017-03-15 05:20:24	0|NicolasDorier|does anyone successfully ran bitcoind on a network share (datadir on cifs) ? it think it always worked for me but somehow it does not anymore
  9 2017-03-15 05:21:22	0|NicolasDorier|I don't think it has to do with a recent commit either
 10 2017-03-15 06:49:41	0|warren|gmaxwell: oh damn.  You're right.  I somehow failed to look at it that way.  This is even more clever than I thought it was!
 11 2017-03-15 06:50:53	0|gmaxwell|warren: woot. another person enlightenend.
 12 2017-03-15 07:00:10	0|warren|Makes a lot more sense why 4x isn't arbitrary!
 13 2017-03-15 07:37:27	0|wumpus|NicolasDorier: what kind of network share?
 14 2017-03-15 07:37:48	0|NicolasDorier|Windows network share cifs I am trying to debug into leveldb right now
 15 2017-03-15 07:38:14	0|wumpus|NicolasDorier: some of the databases have problems with network shares of some kinds
 16 2017-03-15 07:38:29	0|NicolasDorier|yeah the strange thing is that up till now it was working fine
 17 2017-03-15 07:38:42	0|wumpus|I think I worked around that with symlinking just the blocks directory there a few times, but that's not something that will work on windows
 18 2017-03-15 07:39:06	0|NicolasDorier|I tried to do that
 19 2017-03-15 07:39:16	0|NicolasDorier|for some reason with rpc-test it does not work
 20 2017-03-15 07:39:39	0|NicolasDorier|if I run bitcoind with datadir in my sym link it works fine
 21 2017-03-15 07:39:45	0|NicolasDorier|somehow if I run the tests it does not
 22 2017-03-15 07:39:49	0|wumpus|anyhow, improving leveldb to work with those FSes would make sense
 23 2017-03-15 07:39:58	0|NicolasDorier|I am debugging it now
 24 2017-03-15 07:40:11	0|wumpus|the tests don't use your datadirectory they create their own
 25 2017-03-15 07:40:13	0|NicolasDorier|problem in PosixWriteableFile::SyncDirIfManifest
 26 2017-03-15 07:40:26	0|NicolasDorier|I know, in /qa/cache
 27 2017-03-15 07:40:31	0|wumpus|PosixWritableFile shouldn't be used on windows at all
 28 2017-03-15 07:40:47	0|NicolasDorier|I know
 29 2017-03-15 07:40:51	0|NicolasDorier|I am compiling on linux
 30 2017-03-15 07:40:55	0|NicolasDorier|and running tests on linux
 31 2017-03-15 07:41:00	0|wumpus|there's a Win32 environment that we use for windows
 32 2017-03-15 07:41:03	0|NicolasDorier|except the code is on a share network  drive
 33 2017-03-15 07:41:08	0|NicolasDorier|I do that to code with visual studio
 34 2017-03-15 07:41:11	0|wumpus|ohh
 35 2017-03-15 07:41:38	0|wumpus|don't out-of-tree builds help there?
 36 2017-03-15 07:42:02	0|wumpus|maybe the test framework needs a way where to put its cra^D^D^D cache
 37 2017-03-15 07:42:25	0|NicolasDorier|out-of-tree builds ?
 38 2017-03-15 07:42:49	0|wumpus|yes. calling configure from outside the source tree to put the build output there
 39 2017-03-15 07:42:52	0|wumpus|https://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html
 40 2017-03-15 07:43:36	0|wumpus|it's designed for cases where the source is on say, a network share, and the build artifacts should go to some big fast local drive
 41 2017-03-15 07:43:45	0|NicolasDorier|oh I did not know.
 42 2017-03-15 07:43:54	0|NicolasDorier|I will first try to pinpoint why it fails in leveldb
 43 2017-03-15 07:44:00	0|NicolasDorier|if it fails I will do that
 44 2017-03-15 07:44:20	0|wumpus|well up until a few versions ago it didn't work with bitcoin core due to how we integrated leveldb. It works for 0.13/0.14 at least though
 45 2017-03-15 07:48:05	0|NicolasDorier|wumpus: it is fsync() which fail
 46 2017-03-15 07:48:32	0|NicolasDorier|with errno == 22
 47 2017-03-15 07:49:06	0|NicolasDorier|will find out why oO
 48 2017-03-15 07:50:20	0|NicolasDorier|wumpus: http://man7.org/linux/man-pages/man2/fsync.2.html EINVAL error
 49 2017-03-15 07:50:22	0|NicolasDorier|it makes sense
 50 2017-03-15 07:50:23	0|wumpus|ah our good friend EINVAL
 51 2017-03-15 07:50:33	0|NicolasDorier|but this blow mymind that it worked before
 52 2017-03-15 07:51:23	0|wumpus|a "special file" which does not support synchronization
 53 2017-03-15 07:51:51	0|wumpus|I'd read that as "device" or "pipe", not so much any random file on a network OS. But yes it's clear, network filesystems cannot give that guarantee either.
 54 2017-03-15 07:52:59	0|NicolasDorier|wondering if it is not my docker which got sneaky upgraded and use a network share for file sharing with the container instead of something else... But even then, I remember running bitcoin core on a network share just fine in the passt
 55 2017-03-15 07:53:36	0|wumpus|maybe cifs mounts have some mount settings that specify how to handle syncs?
 56 2017-03-15 07:54:54	0|wumpus|could also be a server setting but I'd think that less likely
 57 2017-03-15 07:55:19	0|NicolasDorier|searching... when I do mount : //10.0.75.1/E on /home/bitcoin type cifs
 58 2017-03-15 07:55:20	0|NicolasDorier|(rw,relatime,vers=3.02,sec=ntlmssp,cache=strict,username=NicolasDorier,domain=DESKTOP-M7N69JH,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.75.1,file_mode=0755,dir_mode=0755,iocharset=utf8,nounix,serverino,mapposix,nobrl,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
 59 2017-03-15 07:55:33	0|wumpus|nfs did have some extra daemon that needed to be installed to handle that at some point, I vaguely remember or probably misremember
 60 2017-03-15 07:56:24	0|wumpus|in any case you can make a minimal reproduction now - a small program that just opens a file on the network FS and tries to fsync() it
 61 2017-03-15 07:56:53	0|wumpus|a lot easier for testing than launching the whole bitcoind monilith
 62 2017-03-15 07:57:14	0|NicolasDorier|yep
 63 2017-03-15 07:58:18	0|wumpus|googling fsync cifs gives a lot of results about similar issues with other database products
 64 2017-03-15 07:59:32	0|NicolasDorier|yes this one seems relevant https://github.com/docker/for-win/issues/445
 65 2017-03-15 07:59:42	0|NicolasDorier|but still wondering why it worked before
 66 2017-03-15 07:59:46	0|wumpus|most common recommendation is "ignore EINVAL for fsync". I guess it's something that should log a warning once, then just continue
 67 2017-03-15 08:00:16	0|NicolasDorier|I can do that at leveldb level yes
 68 2017-03-15 08:00:38	0|NicolasDorier|is there a way to print leveldb warning in the console ?
 69 2017-03-15 08:00:41	0|wumpus|it is a data coherency issue and in the case of crashes not being able to guarantee synced-ness is a bad thing, but it's not a reason to crash out
 70 2017-03-15 08:00:46	0|NicolasDorier|I saw leveldb has its own log stuff
 71 2017-03-15 08:00:48	0|wumpus|it'd have to go to the bitcoind log some way, not the console
 72 2017-03-15 08:00:58	0|NicolasDorier|yes
 73 2017-03-15 08:00:59	0|wumpus|maybe leveldb logging can be redirected there
 74 2017-03-15 08:01:15	0|NicolasDorier|this is not done currently right ?
 75 2017-03-15 08:01:28	0|wumpus|no, I don't think so. It probably should be done though
 76 2017-03-15 08:01:45	0|NicolasDorier|ok I will work on a PR trying to fix all this stuff
 77 2017-03-15 08:01:45	0|wumpus|for all we know leveldb has been spewing other useful warnings and errors and we've been ignoring them
 78 2017-03-15 08:02:26	0|wumpus|we do do that for libevent, for example, we register handlers for their log messages and redirect them to our log
 79 2017-03-15 08:04:12	0|wumpus|but yes it is weird that this just happened and worked before
 80 2017-03-15 08:04:23	0|wumpus|some version upgrade maybe changed the behavior
 81 2017-03-15 08:06:11	0|wumpus|Add new mount
 82 2017-03-15 08:06:12	0|wumpus|fsync if desired (fsync still flushes all cached write data to the server).
 83 2017-03-15 08:06:12	0|wumpus|parameter to allow user to disable sending the (slow) SMB flush on
 84 2017-03-15 08:06:57	0|wumpus|https://www.kernel.org/doc/Documentation/filesystems/cifs/CHANGES version 1.57
 85 2017-03-15 08:07:11	0|wumpus|it doesn't say what that parameter is, though
 86 2017-03-15 08:07:31	0|wumpus|but this implies that fsync should simply work?
 87 2017-03-15 08:07:53	0|wumpus|even with disabling it it still flushes the data to the server,which is what it should do
 88 2017-03-15 08:08:26	0|NicolasDorier|I have not updated my environment I think
 89 2017-03-15 08:08:31	0|wumpus|it doesn't guarantee that the server will have written the data to disk, but it does guarantee that it is written to the server
 90 2017-03-15 08:09:20	0|NicolasDorier|so it should support it in theory
 91 2017-03-15 08:09:20	0|wumpus|so maybe it is worth doing another pass over local settings and configuration before resorting to leveldb changes, I don't know
 92 2017-03-15 08:09:26	0|wumpus|yes
 93 2017-03-15 08:10:20	0|NicolasDorier|might it be a SMB error caused by a windows update
 94 2017-03-15 08:10:54	0|NicolasDorier|I am quite sure my linux environemnt have not moved, I always work on fresh container of always the same image
 95 2017-03-15 08:13:10	0|NicolasDorier|ok I will just fix it in leveldb for now
 96 2017-03-15 08:13:42	0|wumpus|all the results I find deal with fsync on directories, which is apparently not always supported
 97 2017-03-15 08:16:08	0|wumpus|I don't think leveldb does that, or does it?
 98 2017-03-15 08:16:38	0|wumpus|oh it does!
 99 2017-03-15 08:16:52	0|wumpus|that's what "SyncDirIfManifest" is about, it syncs the dir
100 2017-03-15 08:17:02	0|wumpus|NicolasDorier: ^^ okay yes patching leveldb is the way forward
101 2017-03-15 08:17:56	0|NicolasDorier|ok. I will do that and try to add log support
102 2017-03-15 08:34:37	0|bitcoin-git|[13bitcoin] 15kobake opened pull request #9998: tests: Remove unused function dumpKeyInfo (06master...06remove-unused-dump-function) 02https://github.com/bitcoin/bitcoin/pull/9998
103 2017-03-15 09:45:53	0|bitcoin-git|[13bitcoin] 15NicolasDorier opened pull request #9999: [LevelDB] Plug leveldb logs to bitcoin logs (06master...06leveldblog) 02https://github.com/bitcoin/bitcoin/pull/9999
104 2017-03-15 09:47:38	0|Victorsueca|quick! someone make another PR so you're the PR #10000!
105 2017-03-15 09:47:39	0|gribble|https://github.com/bitcoin/bitcoin/issues/10000 | HTTP Error 404: Not Found
106 2017-03-15 09:47:56	0|Victorsueca|:v
107 2017-03-15 10:02:00	0|bitcoin-git|[13bitcoin] 15NicolasDorier opened pull request #10000: [LevelDB] Do no crash if filesystem can't fsync (06master...06fsync) 02https://github.com/bitcoin/bitcoin/pull/10000
108 2017-03-15 10:02:23	0|Victorsueca|\o/
109 2017-03-15 10:03:01	0|Victorsueca|Happy 10000 PRs!
110 2017-03-15 10:04:14	0|NicolasDorier|OH YEAH
111 2017-03-15 10:12:31	0|wumpus|whoa PR 10000 party
112 2017-03-15 10:29:30	0|rabidus|\o/
113 2017-03-15 10:30:21	0|luke-jr|why are LevelDB stuff being submitted to Core's git?
114 2017-03-15 10:31:15	0|wumpus|for visibility
115 2017-03-15 10:32:19	0|luke-jr|makes sense I guess. (maybe should have a label "DO NOT MERGE"?)
116 2017-03-15 11:01:22	0|wumpus|NicolasDorier: any idea where leveldb uses logging, if the logger is not generally passed around?
117 2017-03-15 11:02:01	0|wumpus|NicolasDorier: uhm.
118 2017-03-15 11:02:46	0|wumpus|NicolasDorier: I never mind, I was confused, thought I saw a global log function in leveldb
119 2017-03-15 11:03:21	0|wumpus|looks like it is used for some low-level logging in db_impl.cc
120 2017-03-15 11:04:04	0|wumpus|so it's more like a debug log than an error/warning log
121 2017-03-15 11:04:42	0|wumpus|void DBImpl::MaybeIgnoreError(Status* s) const also uses it (but it's never triggered as with paranoid checks it doesn't ignore any errors)
122 2017-03-15 11:05:28	0|wumpus|I guess it can't hurt to bridge the leveldb logger to our debug log (as you did, as a separate logging category) in any case, but just wondering what messages we've been missing
123 2017-03-15 12:37:36	0|NicolasDorier|wumpus: it seems there use only options_.info_log
124 2017-03-15 12:37:47	0|NicolasDorier|I don't find the debug logs you are telling me
125 2017-03-15 12:40:40	0|NicolasDorier|wumpus: this is a joke.... I came back home from my coffee shop and now I do not have fsync error.... there is a phase-of-the-moon bug somewhere
126 2017-03-15 12:41:02	0|NicolasDorier|ha maybe I did not recompiled
127 2017-03-15 12:43:41	0|NicolasDorier|ha no it is crashing... I thought being crazy for a while
128 2017-03-15 13:43:54	0|bitcoin-git|[13bitcoin] 15greenaddress opened pull request #10002: fix gitian doc example script typo (06master...06gitian_typos) 02https://github.com/bitcoin/bitcoin/pull/10002
129 2017-03-15 14:42:42	0|bitcoin-git|[13bitcoin] 15kobake closed pull request #9998: tests: Remove unused function dumpKeyInfo (06master...06remove-unused-dump-function) 02https://github.com/bitcoin/bitcoin/pull/9998