1 2017-12-01 01:08:22	0|promag|sipa: > involve downgrading/upgrading between versions
  2 2017-12-01 01:08:43	0|sipa|yes?
  3 2017-12-01 01:09:02	0|promag|could have those wallets as fixtures'
  4 2017-12-01 01:09:05	0|promag|?
  5 2017-12-01 01:09:30	0|sipa|?
  6 2017-12-01 01:12:23	0|promag|you mean that to test you have to use different versions of the wallet or the binary?
  7 2017-12-01 01:12:34	0|sipa|yes
  8 2017-12-01 01:13:02	0|sipa|any automated testing that downgrading/upgrading is supported under the intended scenarios would involve having tests with access to different versions of binaries
  9 2017-12-01 01:13:15	0|sipa|which is just hard logistically
 10 2017-12-01 01:17:08	0|promag|why? could make use of docker for instance
 11 2017-12-01 01:17:46	0|sipa|i fail to understand what kind of universe you live in where that doesn't qualify as "logistically hard" :)
 12 2017-12-01 01:17:47	0|promag|but yes, a pain
 13 2017-12-01 01:18:05	0|sipa|i'm not saying it's impossible - but it requires very nontrivial changes to our setup
 14 2017-12-01 01:18:17	0|sipa|so i'm looking for opinions
 15 2017-12-01 01:18:32	0|sipa|is it enough to do manual testing for downgrade/upgrade scenarios, for example?
 16 2017-12-01 01:19:06	0|promag|how about having some wallets created offline with different versions commited to the source code, to use in the the tests?
 17 2017-12-01 01:19:24	0|sipa|that would be a good idea!
 18 2017-12-01 01:19:38	0|sipa|it can't test downgrade scenarios, though - only upgrade
 19 2017-12-01 01:20:09	0|promag|why? the downgrade should produce an expected wallet (also commited)
 20 2017-12-01 01:21:15	0|sipa|we don't have old binaries available
 21 2017-12-01 01:21:21	0|meshcollider|promag: what about testing a wallet generated by the new version, against old software
 22 2017-12-01 01:21:32	0|sipa|oh, by downgrade i mean opening a new file in an old version of the software
 23 2017-12-01 01:21:33	0|meshcollider|e.g. generate wallet with 0.16.0, test against 0.15.1
 24 2017-12-01 01:22:14	0|meshcollider|downgrading the software not the wallet :)
 25 2017-12-01 01:22:14	0|promag|no need to test agains 0.15.1
 26 2017-12-01 01:22:27	0|meshcollider|I mean any previous versin
 27 2017-12-01 01:23:00	0|promag|you just happen to know downgrade result, so just check the downgrade result, offline you know it works in that version
 28 2017-12-01 01:23:25	0|sipa|promag: practically the entire discussion about segwit wallet support is about downgrading and restoring backups across versions
 29 2017-12-01 01:23:48	0|sipa|so no, we absolutely do care about how 0.15.1 deals with 0.16 files
 30 2017-12-01 01:24:27	0|promag|I'm not saying otherwise
 31 2017-12-01 01:24:29	0|meshcollider|old binaries are all stored on bitcoincore.org so just write another testing framework to use those ;)
 32 2017-12-01 01:24:41	0|sipa|promag: then i don't understand what you're saying
 33 2017-12-01 01:25:44	0|meshcollider|oh actually there is nothing pre-0.9.5 there
 34 2017-12-01 01:26:38	0|promag|so offline with create and downgrade the wallet, test with older versions, if it's ok then commit a test that does the same and at the end verifies the downgraded wallet
 35 2017-12-01 01:27:05	0|sipa|what does 'verifies the downgraded wallet' mean?
 36 2017-12-01 01:27:29	0|sipa|there is no such thing as a downgraded wallet - it's just a wallet file that was modified by master, and we care about how old software deals with it
 37 2017-12-01 01:27:48	0|promag|for instance, the sha(downgraded wallet file) must be ....
 38 2017-12-01 01:28:01	0|sipa|bdb files aren't deterministic
 39 2017-12-01 01:28:13	0|meshcollider|in reality it'd require a whole new testing framework to ensure things like signrawtransaction, listunspent, etc. all work on the old versions aye
 40 2017-12-01 01:28:22	0|sipa|and testing exact equivalence would be painful to maintain, as it would need updating after pretty much any change
 41 2017-12-01 01:29:01	0|promag|2 downgrades of the same can be different?
 42 2017-12-01 01:30:09	0|sipa|promag: BDB files aren't deterministic, so you can't predict at all what the wallet file will look like after _any_ change
 43 2017-12-01 01:30:17	0|sipa|you could dump the key/values in it, and compare those
 44 2017-12-01 01:30:23	0|sipa|but my point is that you don't want to do that
 45 2017-12-01 01:30:41	0|sipa|we don't care about the exact contents - we care about whether it works in old versions :)
 46 2017-12-01 01:31:04	0|promag|my point is that those "exact contents" work in previous versions
 47 2017-12-01 01:31:23	0|sipa|yes, but we can't predict what exact contents will be in master
 48 2017-12-01 01:31:39	0|sipa|unless you want to modify that pre-committed exact contents after any change in master that affects the wallet file
 49 2017-12-01 01:31:54	0|promag|that should not change often?
 50 2017-12-01 01:31:59	0|sipa|maybe not
 51 2017-12-01 01:32:07	0|sipa|it'd still be a pain
 52 2017-12-01 01:32:20	0|promag|I mean, until a better testing framework exists it's better than nothing
 53 2017-12-01 01:32:26	0|sipa|maybe
 54 2017-12-01 01:32:34	0|meshcollider|just checking the hash of the wallet file is not sufficient though?
 55 2017-12-01 01:32:51	0|sipa|meshcollider: no, the hash of the wallet file is completely unpredictable
 56 2017-12-01 01:33:05	0|meshcollider|yeah, isn't that what promag is suggestion though
 57 2017-12-01 01:33:43	0|promag|hash, key/values, whatever
 58 2017-12-01 01:34:27	0|sipa|i generally dislike tests that test for exact equivalence
 59 2017-12-01 01:34:29	0|meshcollider|key/values dumped by a newer version won't necessarily match those of an older version also, because changes to e.g. dumpscript would change the output
 60 2017-12-01 01:34:35	0|promag|the test should just expect something. if master changes that then it might mean an old version can't use a downgraded wallet
 61 2017-12-01 01:34:41	0|sipa|especially if its unclear how those expected values should change
 62 2017-12-01 01:35:16	0|promag|but yes, if a wallet decides to add a new key/value for soemthing else then its' a pain
 63 2017-12-01 01:35:29	0|sipa|or a timestamp
 64 2017-12-01 01:35:33	0|sipa|or a version number
 65 2017-12-01 01:35:45	0|sipa|or a txid that's hard to make constant during the test
 66 2017-12-01 01:36:07	0|promag|and pick the relevant key/values?
 67 2017-12-01 01:37:10	0|promag|nothing better than actually use the old binaries
 68 2017-12-01 01:37:18	0|meshcollider|yeah
 69 2017-12-01 01:37:39	0|promag|what versions you think it should tests?
 70 2017-12-01 01:38:05	0|meshcollider|well even 0.11 is now EOL isn't it, so I think it would be sufficient to just test on 12+ ?
 71 2017-12-01 01:38:44	0|meshcollider|(automatically, of course we shouldn't deliberately break compatibility with earlier versions too just not test against them all automatically)?
 72 2017-12-01 01:38:48	0|promag|only final releases right?
 73 2017-12-01 01:39:43	0|meshcollider|you mean like, not rc? or just 0.14.2 not 0.14.1?
 74 2017-12-01 01:40:29	0|promag|not rc
 75 2017-12-01 01:40:56	0|promag|anyone knows who owns this https://hub.docker.com/u/bitcoincore/ ?
 76 2017-12-01 01:41:51	0|sipa|it's a good question what versions should be supportedd for downgrading
 77 2017-12-01 01:45:08	0|sipa|the use case is generally that someone upgrades, notices that the new version does something else/unexpected that is incompatible with their business, and downgrades until the issue is resolved
 78 2017-12-01 01:46:18	0|sipa|and obvious for the specific case of segwit wallet support - downgrading below 0.13 is impossible of course
 79 2017-12-01 01:59:40	0|promag|I guess you should add this testing as a follow up in #11403
 80 2017-12-01 01:59:44	0|gribble|https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
 81 2017-12-01 02:01:45	0|promag|a checklist for manual testing downgrade would be great also
 82 2017-12-01 02:02:15	0|sipa|good point
 83 2017-12-01 02:04:52	0|promag|bed time o/
 84 2017-12-01 02:08:50	0|meshcollider|IMO its safe enough to support downgrading to versions which havent reached EOL and not bother testing the others too
 85 2017-12-01 04:23:30	0|Varunram|Is it possible to change the difficulty on a regtest chain?
 86 2017-12-01 07:46:39	0|kallewoof|Trying to get gitian working, and am having issues with LXC. I am getting "sudo: unable to resolve host gitian / cannot set terminal process group (1): Inappropriate ioctl for device / no job control in this shell" for the line "sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -u $TUSER $ENV -i -- $*" in libexec/on-target, called from make-clean-vm (the bootstrap-fixup part). Any ideas?
 87 2017-12-01 08:06:18	0|midnightmagic|kallewoof: I solved that by modifying the gitian tqrget config to add the hostname to the internal /etc/hosts file; originally I solved it by manually modifying the base image.
 88 2017-12-01 08:21:33	0|kallewoof|midnightmagic: Nice, that got rid of the first line, but the other two lines remain (and I get thrown into a shell as root on the LXC side)..
 89 2017-12-01 08:22:10	0|kallewoof|If I ^D, I get a bunch of "unknown user: ubuntu" errors when it tries to continue doing stuff.
 90 2017-12-01 08:28:19	0|midnightmagic|does yout bootstrap-fixup not do an adduser for you?
 91 2017-12-01 08:30:06	0|kallewoof|It's not being executed, because of the above errors, I suspect.
 92 2017-12-01 08:33:01	0|kallewoof|I tried copying it into the base VM and then running it as a script from inside. That sorta worked, but "initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused" (not sure if that is bad news or can be ignored), then stuff, and then it fails again on the "Preparing build environment" part (basically, it fails whenever a "<" is used to feed a script
 93 2017-12-01 08:33:04	0|kallewoof|into it).
 94 2017-12-01 08:33:37	0|kallewoof|In this case, the "system! "on-target setarch #{@arches[arch]} bash < target-bin/init-build.sh"" part in gbuild
 95 2017-12-01 08:44:03	0|kallewoof|I think an easy fix for me is to just copy target-bin into the image after copying it. Not sure why <'ing doesn't work tho. :/
 96 2017-12-01 09:04:57	0|kallewoof|I got to the copy-to-target part, which hangs indefinitely on "tar -C `dirname "$1"` -cf - `basename "$1"` | sudo $LXC_EXECUTE -n gitian -f var/lxc.config -- sudo -i -u $TUSER tar -C "$2" -xf -". I suspect the pipe is not functional just like the <, but I could be wrong.
 97 2017-12-01 12:18:09	0|wumpus|don't think Vision is still here but: https://github.com/bitcoin/bitcoin/pull/11809  should fix his issue
 98 2017-12-01 14:19:31	0|bitcoin-git|13bitcoin/06master 14b4058ed 15Dan Raviv: Fix code constness in CBlockIndex::GetAncestor() overloads...
 99 2017-12-01 14:19:31	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/13e31dd6548d...0d7e0a3289ba
100 2017-12-01 14:19:32	0|bitcoin-git|13bitcoin/06master 140d7e0a3 15Wladimir J. van der Laan: Merge #11337: Fix code constness in CBlockIndex::GetAncestor() overloads...
101 2017-12-01 14:29:22	0|bitcoin-git|13bitcoin/06master 14bf20a7d 15Tim Shimmin: [docs] Fixed outdated link with archive.is...
102 2017-12-01 14:29:22	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/0d7e0a3289ba...00d25e90db06
103 2017-12-01 14:29:23	0|bitcoin-git|13bitcoin/06master 1400d25e9 15Wladimir J. van der Laan: Merge #11804: [docs] Fixed outdated link with archive.is...
104 2017-12-01 15:33:01	0|Provoostenator|Tangentally related to testing the segwit branch. It looks like miners are no longer mining SegWit transactions on testnet. See my example in #bitcoin.
105 2017-12-01 15:33:36	0|Provoostenator|(well, much less, they do get mined eventually)
106 2017-12-01 15:33:58	0|wumpus|testnet mining seems a continuing issue
107 2017-12-01 15:34:34	0|Provoostenator|I can't go in there with my laptop CPU and mine it myself I presume?
108 2017-12-01 15:34:54	0|instagibbs|people point ASICs at it, so no
109 2017-12-01 15:35:08	0|Provoostenator|Ethereum created a consortium testnet out of desperation, which seems like a pretty bad solution.
110 2017-12-01 15:35:42	0|Provoostenator|Would it be hard to spin up smaller testnets amongst "friends"?
111 2017-12-01 15:36:39	0|instagibbs|you may be interested in #8994
112 2017-12-01 15:36:41	0|gribble|https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
113 2017-12-01 15:37:17	0|instagibbs|for segwit itself something like 4 new testnets were created(segnet)
114 2017-12-01 15:37:20	0|Provoostenator|instagibbs: thanks
115 2017-12-01 15:47:12	0|Provoostenator|The Github issues and pull requests form an infinitely deep and complex rabbit hole; fascinating :-)
116 2017-12-01 15:47:59	0|Provoostenator|"There's a PR for that" might make a good meme.
117 2017-12-01 15:49:35	0|instagibbs|The best one is where there is a PR that's been revived by 2+ people.... thar be dragons
118 2017-12-01 16:00:23	0|Provoostenator|Would it make sense to tweak the PoW algorithm on testnet so it can't (easily) be ASIC mined? Maybe that would keep the difficulty at somewhat manageable levels, where more people can step in and mine SegWit and (experimental) non-standard transactions.
119 2017-12-01 16:02:46	0|Provoostenator|Maybe now is a bad time with heated emotions and such, it could be taken the wrong way. DIY testnets like in the above PR is safer.
120 2017-12-01 16:03:52	0|wumpus|there was some talk of a testnet with signed blocks instead of mining at some point, as mining is always a problem with regard to incentives, but it's pretty annoying to implement and touches consensus code
121 2017-12-01 16:03:57	0|Provoostenator|And I suppose any code supporting such PoW change on testnet would cause maintenance headaches.
122 2017-12-01 16:04:30	0|wumpus|there's no need for PoW at all on testnet, unless for testing miners I guess
123 2017-12-01 16:04:53	0|wumpus|and in that case using a different one is useless
124 2017-12-01 16:05:03	0|Provoostenator|Hardcode the difficulty at 0 and add a consensus rule for now more than 1 block per minute?
125 2017-12-01 16:05:55	0|Provoostenator|I suppose that would be a DDOS vector.
126 2017-12-01 16:06:54	0|wumpus|but yes I think DIY/private testnets are the best solution usually, most testing scenarios don't really want all the noise around
127 2017-12-01 16:07:09	0|Provoostenator|The noise helps you catch unexpected stuff though.
128 2017-12-01 16:08:26	0|Provoostenator|And having frequent reorgs on testnet would be nice too for testing robustness.
129 2017-12-01 16:08:57	0|wumpus|that's what we have already, most requests are for a more reliable testnet
130 2017-12-01 16:09:29	0|Provoostenator|Right, maybe we need a seperate calmnet and wildnet.
131 2017-12-01 16:12:04	0|Provoostenator|I used to just test everything on mainnet when fees were a bit lower (still works if you use the minimum relay fee and spend unconfirmed txs).
132 2017-12-01 16:22:40	0|mlz|devs, current testnet blockheight:  1251999, could we reset testnet or redesign it all together that LN testers don't have to wait for a segwit miner to come along?
133 2017-12-01 16:22:49	0|wumpus|indeed, testing on mainnet is possible, though that's bad practice to recommend to anyone, much foot-shooting potential
134 2017-12-01 16:28:26	0|wumpus|mlz: yes, testnet seems broken at the moment, and no one seems to be able to spare miners to point at it :/
135 2017-12-01 16:29:50	0|mlz|wumpus, is it possible to make it CPU mining? i can help and so can many others
136 2017-12-01 16:35:02	0|wumpus|one idea that has been proposed is to do a testnet with signed blocks, e.g. wumpus signs a block every 10 minutes. Mining, of whatever kind, will always introduce these kind of issues.
137 2017-12-01 16:35:53	0|wumpus|it's not that sha256 is not CPU minable, just that someone is mining and pushing up difficulty w/ inconvenient settings or version
138 2017-12-01 16:37:53	0|wumpus|just two year ago I CPU-mined quite a few blocks on testnet, while no one else was pointing miners at it
139 2017-12-01 16:38:50	0|wumpus|anyhow for a reliable testnet it's probably best to have a fixed set of rules, distributed mining competition doesn't really help
140 2017-12-01 16:51:17	0|wumpus|as mentioned before it'd be good to try to move #8994 forward
141 2017-12-01 16:51:20	0|gribble|https://github.com/bitcoin/bitcoin/issues/8994 | Testchains: Introduce custom chain whose constructor... by jtimon · Pull Request #8994 · bitcoin/bitcoin · GitHub
142 2017-12-01 17:01:40	0|Varunram|Provoostenator: The difficulty of testnet has been down recently, could give it a shot
143 2017-12-01 17:04:04	0|Provoostenator|Varunram: I suppose I can try it for a day. Which begs the question how: https://bitcoin.stackexchange.com/questions/51169/how-can-i-solo-mine-using-bitcoin-core
144 2017-12-01 17:04:28	0|Provoostenator|(feel free to answer in #bitcoin instead)
145 2017-12-01 17:05:14	0|Varunram|(yeah)
146 2017-12-01 17:27:27	0|Provoostenator|Is there a way to get `make cov` to produce an lcov file rather than HTML?
147 2017-12-01 17:28:10	0|Provoostenator|(happy to tweak Makefile.am, but no idea how)
148 2017-12-01 17:58:56	0|phantomcircuit|wumpus, ye but lots of miners use testnet to test
149 2017-12-01 17:59:04	0|phantomcircuit|about the only people who do really
150 2017-12-01 18:00:24	0|BlueMatt|nah, the ln folks are consistently pissed that testnet sucks
151 2017-12-01 18:00:29	0|BlueMatt|so they use it
152 2017-12-01 18:02:41	0|phantomcircuit|BlueMatt, heh
153 2017-12-01 18:04:02	0|Provoostenator|I'm getting a second hand Block Erupter USB on Sunday, we'll see if that helps.
154 2017-12-01 18:04:11	0|BlueMatt|lol
155 2017-12-01 18:04:37	0|Varunram|for mining segwit txs on testnet?
156 2017-12-01 18:05:10	0|Provoostenator|Yes, as the guy in Armageddon said: this is how we fix things in Russian space station :-)
157 2017-12-01 18:06:12	0|Varunram|lol
158 2017-12-01 18:10:20	0|adiabat|BlueMatt: I use testnet3 all the time, it's a little weird but oh well
159 2017-12-01 18:10:54	0|adiabat|maybe the biggest issue is that you'll have miners show up who don't run segwit and mine for days where no segwit txs can be confirmed
160 2017-12-01 18:11:55	0|gmaxwell|need some hashpower there that mines segwit invalid blocks.
161 2017-12-01 18:12:56	0|adiabat|the messiness of testnet3 is something of a feature though; found some reorg bug in my code the other day
162 2017-12-01 18:13:20	0|adiabat|though it's specific to the testnet difficulty reduction, so maybe actually not that great a feature :)
163 2017-12-01 18:13:30	0|Varunram|adaibat: further adding to BlueMatt's assertion that ln folks use testnet grudgingly
164 2017-12-01 18:13:35	0|Varunram|adiabat*
165 2017-12-01 18:14:39	0|adiabat|I mean, it's a mess, but at the same time it allows to test weird edge cases
166 2017-12-01 18:15:05	0|adiabat|for example, I have code to checkpoint SPV headers, which is only there because testnet has over 100MB of headers
167 2017-12-01 18:15:44	0|Varunram|nice e.g. lol
168 2017-12-01 18:16:48	0|BlueMatt|heh, there's been persistent discussion of resetting testnet to bump min diff to limit the insane header volume issue
169 2017-12-01 18:17:12	0|adiabat|if anything, making testnet3 more congested may help, as it would allow fee pressure and testing different fees, RBF, etc
170 2017-12-01 18:17:31	0|BlueMatt|heh, well someone can write a bot to do that.....
171 2017-12-01 18:17:37	0|BlueMatt|spam testnet!
172 2017-12-01 18:17:47	0|adiabat|yeah the min diff thing is weird, could definitely do without that
173 2017-12-01 18:18:02	0|adiabat|I don't know the etiquette on that
174 2017-12-01 18:18:14	0|adiabat|I spammed it a bit last year, and made those 3.7MB blocks
175 2017-12-01 18:18:42	0|adiabat|wouldn't be hard to spam more, but figured people might get annoyed at a 200GB testnet
176 2017-12-01 18:19:06	0|Varunram|people would quit it altogether
177 2017-12-01 18:19:26	0|phantomcircuit|BlueMatt, ye testnet probbaly needs a reset
178 2017-12-01 18:19:38	0|phantomcircuit|i mean
179 2017-12-01 18:19:47	0|phantomcircuit|pretty sure im still 90% of testnet by bytes
180 2017-12-01 18:20:14	0|adiabat|I may be a couple %
181 2017-12-01 19:31:25	0|Provoostenator|Rather than spamming, why not drastically decrease the block size? I kind of like that I do an IDB in less than an hour for a change.
182 2017-12-01 23:55:49	0|jimpo|In a local branch I'm getting linking errors along the lines of "multiple definition of `StartShutdown()'". Can someone explain why the definitions in test_bitcoin_main.cpp don't conflict with the ones in init.cpp?