1 2016-11-21 07:37:53	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9193: Update copyright year to 2016 (06master...06patch-1) 02https://github.com/bitcoin/bitcoin/pull/9193
  2 2016-11-21 09:52:11	0|bitcoin-git|[13bitcoin] 15laanwj pushed 5 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/44adf683ad23...7490ae8b699d
  3 2016-11-21 09:52:12	0|bitcoin-git|13bitcoin/06master 140e85204 15Pieter Wuille: Add serialization for unique_ptr and shared_ptr
  4 2016-11-21 09:52:12	0|bitcoin-git|13bitcoin/06master 14da60506 15Pieter Wuille: Add deserializing constructors to CTransaction and CMutableTransaction
  5 2016-11-21 09:52:13	0|bitcoin-git|13bitcoin/06master 141662b43 15Pieter Wuille: Make CBlock::vtx a vector of shared_ptr<CTransaction>
  6 2016-11-21 09:52:21	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9125: Make CBlock a vector of shared_ptr of CTransactions (06master...06sharedblock) 02https://github.com/bitcoin/bitcoin/pull/9125
  7 2016-11-21 09:52:50	0|gmaxwell|\O/
  8 2016-11-21 10:23:39	0|BlueMatt|sipa: did you ever benchmark how much of #9125's performance improvement was actually just the #9014 stuff? (I assume a lot, though the real benifit is compact block speedups here, which reindex cannot test)
  9 2016-11-21 10:23:41	0|gribble|https://github.com/bitcoin/bitcoin/issues/9125 | Make CBlock a vector of shared_ptr of CTransactions by sipa · Pull Request #9125 · bitcoin/bitcoin · GitHub
 10 2016-11-21 10:23:44	0|gribble|https://github.com/bitcoin/bitcoin/issues/9014 | Fix block-connection performance regression by TheBlueMatt · Pull Request #9014 · bitcoin/bitcoin · GitHub
 11 2016-11-21 10:51:29	0|bitcoin-git|13bitcoin/06master 144662553 15Cory Fields: net: don't send feefilter messages before the version handshake is complete
 12 2016-11-21 10:51:29	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/7490ae8b699d...c3906403c8ed
 13 2016-11-21 10:51:30	0|bitcoin-git|13bitcoin/06master 14c390640 15Wladimir J. van der Laan: Merge #9117: net: don't send feefilter messages before the version handshake is complete...
 14 2016-11-21 10:51:40	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9117: net: don't send feefilter messages before the version handshake is complete (06master...06feefilter-assert) 02https://github.com/bitcoin/bitcoin/pull/9117
 15 2016-11-21 12:49:19	0|jonasschnelli|wumpus: do you have plans to rebase #7729? Otherwise I can pick it up.
 16 2016-11-21 12:49:21	0|gribble|https://github.com/bitcoin/bitcoin/issues/7729 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
 17 2016-11-21 12:49:41	0|wumpus|yes, I'll get back to that at some point
 18 2016-11-21 12:50:30	0|wumpus|happy some people are finally commenting on the API
 19 2016-11-21 12:52:19	0|jonasschnelli|Yes. Thanks to last weeks meeting.
 20 2016-11-21 13:20:12	0|instagibbs|wumpus, I'm not an old timer, which is why I'm asking more basic account questions. It's always been deprecated in my Bitcoin lifetime :P
 21 2016-11-21 13:21:11	0|wumpus|right, it's always been deprecated, it's time to move forward there
 22 2016-11-21 13:21:21	0|wumpus|"always"
 23 2016-11-21 13:22:18	0|instagibbs|But my assumption is correct, yes? That accounts gave balances which were often screwy/inconsistent, people should be doing this at higher layers by getting lists of outputs under labels or whatever?
 24 2016-11-21 13:23:04	0|wumpus|yes, it's a mess, people have shot themselves in the foot using it many times, and also most expect it to be something else than it is (eg "subwallets" with their own utxos)
 25 2016-11-21 13:23:19	0|wumpus|it's indeed not something that belongs at the wallet level
 26 2016-11-21 13:23:38	0|instagibbs|Yes I recall finding the comment in code about using * was incorrect in retrieving same balances, but I couldn't even describe the replacement text and gave up
 27 2016-11-21 13:29:36	0|wumpus|so should the argument to importaddress and sendtoaddress etc be called "address" or "bitcoinaddress"? we use a mix of both, which isn't very consistent
 28 2016-11-21 13:30:30	0|wumpus|same with importprivkey which uses "bitcoinprivkey" as argument name, but "importpubkey" which just uses "pubkey". Should make these consistent for #8811
 29 2016-11-21 13:30:31	0|gribble|https://github.com/bitcoin/bitcoin/issues/8811 | rpc: Add support for JSON-RPC named arguments by laanwj · Pull Request #8811 · bitcoin/bitcoin · GitHub
 30 2016-11-21 13:30:53	0|wumpus|my preference would be the shorter name, it *is* bitcoin we don't have any other context where we use address/pubkey/privkey
 31 2016-11-21 13:35:30	0|instagibbs|shorter is better unless it results in overload in the context
 32 2016-11-21 13:36:25	0|wumpus|right
 33 2016-11-21 13:36:43	0|wumpus|changing to just "address" then
 34 2016-11-21 14:28:14	0|bitcoin-git|13bitcoin/06master 14fa7cc5a 15MarcoFalke: Set DEFAULT_LIMITFREERELAY = 0 kB/minute
 35 2016-11-21 14:28:14	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/c3906403c8ed...b9a87b459d25
 36 2016-11-21 14:28:15	0|bitcoin-git|13bitcoin/06master 14b9a87b4 15Wladimir J. van der Laan: Merge #9179: Set DEFAULT_LIMITFREERELAY = 0 kB/minute...
 37 2016-11-21 14:28:27	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #9179: Set DEFAULT_LIMITFREERELAY = 0 kB/minute (06master...06Mf1611-blockFreeTxs) 02https://github.com/bitcoin/bitcoin/pull/9179
 38 2016-11-21 14:33:47	0|bitcoin-git|13bitcoin/06master 147451cf5 15jnewbery: Allow bitcoin-tx to parse partial transactions...
 39 2016-11-21 14:33:47	0|bitcoin-git|[13bitcoin] 15laanwj pushed 2 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/b9a87b459d25...210891143ba0
 40 2016-11-21 14:33:48	0|bitcoin-git|13bitcoin/06master 142108911 15Wladimir J. van der Laan: Merge #8837: allow bitcoin-tx to parse partial transactions...
 41 2016-11-21 14:33:54	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #8837: allow bitcoin-tx to parse partial transactions (06master...06bitcoin-tx-partial-transactions) 02https://github.com/bitcoin/bitcoin/pull/8837
 42 2016-11-21 14:41:57	0|bitcoin-git|[13bitcoin] 15laanwj pushed 4 new commits to 06master: 02https://github.com/bitcoin/bitcoin/compare/210891143ba0...0c577f2638b7
 43 2016-11-21 14:41:58	0|bitcoin-git|13bitcoin/06master 143451203 15Matt Corallo: [qa] Respond to getheaders and do not assume a getdata on inv
 44 2016-11-21 14:41:58	0|bitcoin-git|13bitcoin/06master 14d768f15 15mrbandrews: [qa] Make comptool push blocks instead of relying on inv-fetch
 45 2016-11-21 14:41:59	0|bitcoin-git|13bitcoin/06master 14037159c 15Matt Corallo: Remove block-request logic from INV message processing
 46 2016-11-21 14:42:09	0|bitcoin-git|[13bitcoin] 15laanwj closed pull request #8872: Remove block-request logic from INV message processing (06master...06fetchflags2) 02https://github.com/bitcoin/bitcoin/pull/8872
 47 2016-11-21 16:03:03	0|bitcoin-git|[13bitcoin] 15ryanofsky opened pull request #9196: Send tip change notification from invalidateblock (06master...06pr/notify-tip) 02https://github.com/bitcoin/bitcoin/pull/9196
 48 2016-11-21 16:14:11	0|thermoman|hi. I've got a transaction that's somehow not relayed to the network. but when doing getrawtransaction and sendrawtransaction afterwards it's relayed to the network
 49 2016-11-21 16:14:19	0|thermoman|is this a known issue somehow?
 50 2016-11-21 16:15:54	0|sdaftuar|thermoman: is this for a brand new transaction you created, or an old wallet transaction?
 51 2016-11-21 16:16:06	0|thermoman|ok, it just was relayed to the network - but a hughe delay
 52 2016-11-21 16:16:13	0|thermoman|sdaftuar: it's a brand new tx
 53 2016-11-21 16:16:32	0|thermoman|we have noticed big delays since 2 or 3 days
 54 2016-11-21 16:16:50	0|sipa|thermoman: define huge?
 55 2016-11-21 16:17:02	0|thermoman|doing sendrawtransaction as a workaround after the initial TX the TX was then immediately visible
 56 2016-11-21 16:17:29	0|sdaftuar|thermoman: can you clarify how you're telling that the tx is "visible"?
 57 2016-11-21 16:17:58	0|sdaftuar|ie, visible on some 3rd party explorer?
 58 2016-11-21 16:18:12	0|sdaftuar|or are you looking at your node's relay behavior more directly?
 59 2016-11-21 16:18:13	0|thermoman|lemme get that info from our dev
 60 2016-11-21 16:18:20	0|thermoman|I guess 3rd party
 61 2016-11-21 16:18:24	0|thermoman|like blockchain.info
 62 2016-11-21 16:18:45	0|thermoman|i query the dev right now
 63 2016-11-21 16:20:33	0|sipa|also, what kind of delays are we talking about? second? minutes? hours?
 64 2016-11-21 16:21:44	0|thermoman|http://pastebin.com/LXT4GmnP
 65 2016-11-21 16:21:51	0|thermoman|sipa, sdaftuar ^
 66 2016-11-21 16:22:05	0|thermoman|delay ~20 minutes
 67 2016-11-21 16:24:55	0|sipa|thermoman: can you show a few lines that come after the first broadcast?
 68 2016-11-21 16:30:29	0|thermoman|sure, one moment
 69 2016-11-21 16:35:20	0|thermoman|sipa: http://pastebin.com/8TEFK0Ee
 70 2016-11-21 16:35:27	0|thermoman|this comes directly below it
 71 2016-11-21 16:35:55	0|thermoman|the node in question is connected to a proxy node via connect=<IP>:8333
 72 2016-11-21 16:36:05	0|thermoman|it seems like the connection to the proxy node got reset
 73 2016-11-21 16:36:18	0|thermoman|all nodes run 0.13.1
 74 2016-11-21 16:36:20	0|sdaftuar|right, so you had no peer to broadcast to initially
 75 2016-11-21 16:37:28	0|sdaftuar|which would explain the delays you're seeing, though it raises the question of why you're getting disconnected
 76 2016-11-21 16:37:41	0|thermoman|right. the proxy node is on the LAN
 77 2016-11-21 16:38:23	0|sdaftuar|if you can share the debug log info around the time of a disconnect, that might help indicate if it's a bitcoind issue
 78 2016-11-21 16:38:46	0|thermoman|you mean from the proxy node, right?
 79 2016-11-21 16:39:01	0|thermoman|the proxy node isn't running with debug=1 - will have to change that
 80 2016-11-21 16:39:03	0|sdaftuar|well it might be helpful to see it from both sides' point of view
 81 2016-11-21 16:39:23	0|sdaftuar|if you just have one debug log, we can look at that
 82 2016-11-21 16:40:17	0|thermoman|lemme activate debug=1 on the proxy node. and when the error occurs again tomorrow i'll put both debug logs on pastebin
 83 2016-11-21 16:40:21	0|thermoman|ok?
 84 2016-11-21 16:41:01	0|sipa|thermoman: i believe you were not even connected to your proxy node when the transaction was creates
 85 2016-11-21 16:42:16	0|sipa|oh, sdaftuar already told you thay
 86 2016-11-21 16:42:53	0|sdaftuar|thermoman: does your proxy node -whitelist your internal node?
 87 2016-11-21 16:43:40	0|thermoman|sdaftuar: yes
 88 2016-11-21 16:43:52	0|thermoman|# Whitelist internal networks
 89 2016-11-21 16:44:01	0|thermoman|whitelist=a.b.c.d/24
 90 2016-11-21 16:44:12	0|thermoman|from bitcoin.conf on proxy node
 91 2016-11-21 16:46:14	0|thermoman|we now have debug=1 on proxy node. we'll monitor the debug.log from the proxy for new connections from our nodes and when this happens again I will come back here
 92 2016-11-21 16:46:18	0|thermoman|thanks so far
 93 2016-11-21 16:47:24	0|sdaftuar|ok, sounds good.  if you have the debug.log from your internal node available where you saw the disconnect, we should be able to tell if your internal node decided to disconnect your proxy for some reason.
 94 2016-11-21 16:48:02	0|sdaftuar|also, if the proxy rejected something your node sent it, causing a disconnect, we would be able to see that as well, i thnk
 95 2016-11-21 16:57:17	0|thermoman|ok
 96 2016-11-21 16:57:35	0|thermoman|here is the interesting part from the node behind the proxy
 97 2016-11-21 17:05:24	0|thermoman|sdaftuar, sipa: http://pastebin.com/ZzLpLiLU
 98 2016-11-21 17:05:41	0|thermoman|this is the stripped down log from the node sending the TX (behind the proxy)
 99 2016-11-21 17:05:54	0|thermoman|the issue occured several times after importing a privkey
100 2016-11-21 17:07:05	0|sdaftuar|ah.  i wonder if your node was too busy to respond to ping messages, causing a disconnect?
101 2016-11-21 17:07:14	0|thermoman|that might be
102 2016-11-21 17:07:29	0|thermoman|it seems the import did block everything else
103 2016-11-21 17:07:59	0|thermoman|and directly after the import the TX was created (while being disconnected from the proxy)
104 2016-11-21 17:08:41	0|thermoman|should timeouts be upped on our nodes?
105 2016-11-21 17:08:44	0|sdaftuar|the second disconnect seems a bit different though
106 2016-11-21 17:08:55	0|thermoman|yeah
107 2016-11-21 17:09:00	0|thermoman|it's directly after
108 2016-11-21 17:09:01	0|thermoman|Erased 100 orphan tx from peer 1
109 2016-11-21 17:09:16	0|thermoman|should the nodes behind the proxy whitelist the proxy-ip, too?
110 2016-11-21 17:09:28	0|thermoman|at the moment only the proxy has whitelisted the client ips
111 2016-11-21 17:10:06	0|sdaftuar|thermoman: i don't think it would hurt, though i'm not sure yet what's going on
112 2016-11-21 17:14:01	0|sdaftuar|ok i guess the interesting thing about the second disconnect is that your node tried to connect to it at 15:28:07, and then didn't send the version message untl 15:51
113 2016-11-21 17:14:31	0|sdaftuar|so this looks like the wallet operation is contending with the p2p code, and network timeouts are being hit
114 2016-11-21 17:15:38	0|thermoman|ah ok, didn't miss the connection between "Added connection" and "sending version message"
115 2016-11-21 17:15:45	0|thermoman|s/didn't/did/
116 2016-11-21 17:16:29	0|thermoman|current workaround for us is to wait a minute after importing is done
117 2016-11-21 17:17:16	0|sdaftuar|thermoman: seems like a reasonable workaround for now.  thanks for reporting this, i think this is something we should look into and improve
118 2016-11-21 17:22:01	0|thermoman|do you add an issue in github or should I?
119 2016-11-21 17:23:41	0|thermoman|we've only seen this issue after upgrading the proxy node von 0.11.2 to 0.13.1
120 2016-11-21 17:24:15	0|thermoman|the node with the TX was running 0.11.2 until some hours ago and we noticed this behaviour with 0.11.2 and even after upgrading the node to 0.13.1
121 2016-11-21 17:24:42	0|thermoman|so it seems to have started after upgrading the proxy node
122 2016-11-21 17:25:04	0|thermoman|if I don't hear from you I'll open an issue on github tomorrow
123 2016-11-21 17:25:06	0|sdaftuar|hmm, this issue really seems like it's specific to the internal node.
124 2016-11-21 17:25:27	0|sdaftuar|please go ahead and open the issue, thanks!
125 2016-11-21 17:25:33	0|thermoman|we never had this with 0.11.2 on both nodes
126 2016-11-21 17:25:46	0|thermoman|(or didn't notice?)
127 2016-11-21 17:26:33	0|sdaftuar|my memory is failing me right now about when and whether we've made changes from 0.11 to 0.13 with locking/thread contention.  so in my view it's possible there's a performance regression, i'd need to investigate more.
128 2016-11-21 17:27:00	0|thermoman|ok, thanks so far. will open the issue tomorrow
129 2016-11-21 17:43:53	0|sdaftuar|thermoman: do you often call importprivkey in your setup, ie you were doing that on 0.11.2 as well as regularly on your 0.13.1 nodes?
130 2016-11-21 17:44:29	0|sdaftuar|it looks to me like that is always a slow operation, and will cause bitcoind to be unresponsive while rescanning
131 2016-11-21 17:47:37	0|sdaftuar|(and i think this would have been true in 0.11.2 as well)
132 2016-11-21 18:04:07	0|gmaxwell|sdaftuar: importing a private key with rescan caused timeouts and disconnects for me w/ 0.12 too. If thermoman saw a change it could have also just been a bit of chain growth taking it above threshold.
133 2016-11-21 18:23:31	0|sipa|gmaxwell: or rescans became a bit slower since
134 2016-11-21 19:04:41	0|sdaftuar|instagibbs: in #9194, wouldn't it make sense (if we're going down this road) to also support legacy serializations for zmq and rest?
135 2016-11-21 19:04:42	0|gribble|https://github.com/bitcoin/bitcoin/issues/9194 | Add flags to getrawtransaction and getblock to return non-segwit seri… by instagibbs · Pull Request #9194 · bitcoin/bitcoin · GitHub
136 2016-11-21 19:10:54	0|instagibbs|sdaftuar, yes, I have literally no experience with those interfaces, what would that entail
137 2016-11-21 19:13:40	0|instagibbs|fwiw I think the number of spots we'll need this for is basically two, the ones I've done. We support backwards compatbility when talking to non-segwit nodes(and maybe future serializations!), I'm just giving an option for that at rpc level.
138 2016-11-21 19:13:57	0|instagibbs|so whatever makes sense for that
139 2016-11-21 19:28:54	0|sdaftuar|instagibbs: see for instance rest.cpp:371
140 2016-11-21 19:29:12	0|sdaftuar|oh wait, i made a mistake
141 2016-11-21 19:31:23	0|sdaftuar|oh maybe not -- sorry i've never really looked at this code. looks like that function gets a transaction and serializes it for a user
142 2016-11-21 19:32:06	0|sdaftuar|similarly in the rest_block() function in that same file
143 2016-11-21 19:32:43	0|sdaftuar|and then i think there are two analogous places in zmq/zmqpublishnotifier.cpp
144 2016-11-21 19:33:10	0|sdaftuar|maybe that's it though... if your changes were comprehensive for the RPC stuff (sorry I need to review still), then perhaps this isn't as bad as I feared
145 2016-11-21 19:58:19	0|instagibbs|gettransaction also returns a hex field I'm not converting, but it feels weird messing with an api that is wallet-based for this purpose.
146 2016-11-21 19:59:47	0|instagibbs|I think "raw" apis are a better fit for these changes. If you're using wallet features, just use wallet and what the wallet understands
147 2016-11-21 20:02:24	0|gmaxwell|the overall goal should be "no one should need to hesitate to install 0.13.1+ simply because they haven't updated their applications yet".  This was part of the attractiveness of having the segwit code in a release ahead of activation and avoiding behavior changes at activation time.
148 2016-11-21 20:34:54	0|kanzure|should there be a nonce in rpc client requests? otherwise how does sendmany work if you had a socket timeout on the response? or other must-only-run-once critical rcp endpoints.
149 2016-11-21 20:35:14	0|kanzure|'scuse me, *rpc endpoints
150 2016-11-21 20:36:20	0|gmaxwell|kanzure: 'check listtransactions' :-/  you're right, it should have a nonce. Well the whole rpc thing was not well designed.
151 2016-11-21 20:36:55	0|kanzure|alternatively, you could say "well, just call an event log, and check before you retry" (someone posted a "retry all rpc requests" to petertodd's python-bitcoinlib repo and it raised my eyebrow)
152 2016-11-21 20:37:42	0|kanzure|if sending money is the only scenario where something bad happens, then listtransactions is probably sufficient...
153 2016-11-21 20:37:44	0|gmaxwell|(or, better, a nonce, sequence number pair, and a nonce can be reused but only with a strictly greater sequence number)
154 2016-11-21 20:39:52	0|kanzure|what about: in RPC request to sendmany (or whatever else is broken here), include expected state and also the actual request. and if the expected state doesn't match, then raise error or don't send.
155 2016-11-21 20:42:40	0|kanzure|someone outside the channel proposes, 'if you want to enforce balance checking, make the server require the client to make a balance call before making a send call.'
156 2016-11-21 20:51:59	0|kanzure|do these problems exist in the http api?
157 2016-11-21 20:56:26	0|sipa|you mean the REST api? that's read only
158 2016-11-21 20:56:48	0|sipa|and only for public data (nothing wallet related)
159 2016-11-21 21:03:13	0|sipa|gcc 7.0 has -Wmisleading-indentation
160 2016-11-21 21:32:31	0|Chris_Stewart_5|Just to be clear, there is no compact size uint to indicate how many CTxInWitness inside of the serialization of a tx right?
161 2016-11-21 21:32:37	0|Chris_Stewart_5|You need to infer it from the number of inputs?
162 2016-11-21 21:47:42	0|sipa|Chris_Stewart_5: indeed
163 2016-11-21 21:49:07	0|kanzure|okay i posted an issue about the abovementioned rpc replay protection concerns https://github.com/bitcoin/bitcoin/issues/9197
164 2016-11-21 21:56:56	0|Chris_Stewart_5|Thanks sipa
165 2016-11-21 22:05:15	0|bsm1175321|Are there any RPC requests that would NOT be idempotent?  I think kanzure's suggestion is that bitcoind always complete the action of an RPC call even in event of an I/O error while sending the response.  Is it possible that this wouldn't be the case?
166 2016-11-21 22:05:39	0|bsm1175321|Further, is it possible for a truncated RPC request to end up being valid and acted upon?
167 2016-11-21 22:06:35	0|bsm1175321|BTW I think this is probably RPC "atomicity" not "idempotency".
168 2016-11-21 22:11:27	0|kanzure|no, my suggestion is not "always complete the action of an RPC call even in event of an I/O error while sending the response" -- that's a related topic i guess.
169 2016-11-21 22:11:36	0|bsm1175321|Nah kanzure's right in using idempotent -- CS lingo.  Too much math over here.
170 2016-11-21 22:12:09	0|kanzure|my suggestion is "change from 'always execute any RPC request' to 'check whether the nonce is the same first, and fail if the nonce is the same'"
171 2016-11-21 22:13:23	0|bsm1175321|Sure.  From the client's perspective the two situations are the same...he didn't get the response and doesn't know if the RPC sendmany was executed.  So he retries with the same nonce...
172 2016-11-21 22:14:17	0|kanzure|yes, retrying with the same nonce would be OK
173 2016-11-21 22:14:18	0|sipa|an alternative is having meta support for any asynchronous command
174 2016-11-21 22:14:28	0|bsm1175321|I like that idea...
175 2016-11-21 22:14:51	0|bsm1175321|(I like kanzure's nonce idea).  What's meta support?
176 2016-11-21 22:14:59	0|sipa|you call an RPC "schedule" with 3 args: 1) command 2) command args 3) call id
177 2016-11-21 22:15:19	0|sipa|and then you have a separate RPC through which you can chrck the result
178 2016-11-21 22:15:38	0|kanzure|have you poked at database internals before, like transactional concept?
179 2016-11-21 22:15:58	0|sipa|define 'poked'
180 2016-11-21 22:16:06	0|gmaxwell|sipa: that kind of framework is _necessary_ for some commands.
181 2016-11-21 22:16:21	0|kanzure|eh, 'poked' could be reading i guess :)
182 2016-11-21 22:16:29	0|gmaxwell|For example automatic sendmany batching and replacement will not know the txid until later.
183 2016-11-21 22:17:33	0|kanzure|daisychaining and 'schedule' sure. although if you require some of the prior information before generating the next call, ... can't send all at once.
184 2016-11-21 22:18:03	0|bsm1175321|To be clear -- the issue we're having is that bitcoind closes its RPC connections on a timer.  These are connections on localhost and python-bitcoinlib doesn't reopen them (I consider that it's bug).  But this could also be improved on the bitcoind side if we could configure it to not close RPC connections...
185 2016-11-21 22:18:18	0|bsm1175321|That's separate from calling RPC's over a flaky connection...
186 2016-11-21 22:18:30	0|kanzure|rpc connection closing behavior is somewhat unrelated; your application still needs to know whether to retry. and currently it can't possibly know that.
187 2016-11-21 22:18:41	0|bsm1175321|Yep yep
188 2016-11-21 22:18:41	0|gmaxwell|"not close RPC connections." < guarentees file descriptor exhaustion.  And is unlike any http server ever.
189 2016-11-21 22:19:36	0|gmaxwell|bsm1175321: I think your concern is just a "fix the client" concern. (the python-bitcoinlib behavior has burned me many times too)
190 2016-11-21 22:19:45	0|bsm1175321|But there's only one RPC user and it's on localhost.  I think this is a common configuration.
191 2016-11-21 22:20:13	0|bsm1175321|gmaxwell: I agree.  I'm gonna fix it to re-open and not bother me with I/O exceptions.
192 2016-11-21 22:20:38	0|kanzure|"only one rpc user" what?
193 2016-11-21 22:20:42	0|sipa|didn't we fix that issue in our in-tree fork of python-bitcoinlib?
194 2016-11-21 22:20:50	0|gmaxwell|often still results in exhaustion, e.g. when you get long idle connection objects that aren't GCed yet and so they've left their connections open.
195 2016-11-21 22:22:35	0|bsm1175321|sipa: yes it looks like it is fixed there.
196 2016-11-21 22:24:25	0|kanzure|oh there is an "id" parameter sent https://github.com/petertodd/python-bitcoinlib/blob/2e4d51a482a4331d91de9bae2ab7be56229e664a/bitcoin/rpc.py#L188
197 2016-11-21 22:24:45	0|kanzure|.. which is auto-incremented in this particular client library..
198 2016-11-21 22:27:00	0|bsm1175321|sipa: the fix here: https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/test_framework/authproxy.py#L134 looks like it is effectively a retry, and would, e.g. sendmany again.
199 2016-11-21 22:29:35	0|skyraider|connection handling is distinct from client state management. i think kanzure's concern is that it seems possible for a bitcoind client to say "hey, i didn't receive a reply" and then resend the same request to bitcoind - perhaps resulting in the accidental execution of two identical requests. banks used to behave this way when you clicked "do wire transfer"
200 2016-11-21 22:29:35	0|skyraider|sent wire transfer twice" type behavior, due to no client state management facilities in bitcoind to code against.. is that right?
201 2016-11-21 22:29:35	0|skyraider|then pressed "back" in internet explorer - they'd do another POST to send the wire transfer again. now we have idempotency via request nonces for the web, so it's not a problem - your nonce is considered expired once the nonce's corresponding request is processed. with python-bitcoinlib in particular, i think the concern is that you can still get "oops i
202 2016-11-21 22:31:37	0|gmaxwell|skyraider: applications can check list transactions to determine if the order has been executed yet.
203 2016-11-21 22:31:48	0|sipa|kanzure: that's just for ordering of responses within one connection
204 2016-11-21 22:32:08	0|gmaxwell|A common model is to poll outstanding requests, check list transactions, anything not paid yet, pay... anything already paid remove from the list.
205 2016-11-21 22:32:36	0|kanzure|gmaxwell: no, that doesn't work.   listtransactions will  not matter because you might have anohter concurrent attempt. then both sendmanys execute one after another.
206 2016-11-21 22:33:10	0|gmaxwell|"Don't do that."
207 2016-11-21 22:33:10	0|kanzure|ok well at the very least we should insist that client librares implement that pattern, maybe by rejecting sendmany rpc requests if previous required rpc requests have not been detected
208 2016-11-21 22:33:34	0|gmaxwell|I'm just describing to you an architecture I've seen before, which has generally acceptable properties.
209 2016-11-21 22:34:08	0|gmaxwell|I don't disagree that an optional nonce/sequence with replay protection would be good-- though it has some scalability challenges.
210 2016-11-21 22:35:28	0|kanzure|sipa: oh like for ordering responses to the _batch call?
211 2016-11-21 22:36:24	0|sipa|kanzure: you can send two rpcs with the same id, and you could get two different answers with the same id back
212 2016-11-21 22:37:22	0|sipa|gmaxwell, kanzure: a more lightway idea, just for send* calls, is passing as an argument to the rpc the expected wallet balance
213 2016-11-21 22:37:46	0|sipa|anything that actually interferes would cause failure, while other rpcs could still happen concurrently
214 2016-11-21 22:37:47	0|kanzure|wallet balance isn't good either... many simultaneous deposits/debits could put you back to the same balance anyway.
215 2016-11-21 22:38:28	0|kanzure|excuse me, credits/debits
216 2016-11-21 22:38:35	0|skyraider|so 'read before you try to send lots of bitcoin' - alright. but if another RPC connection writes in the meantime, won't my read be stale, or is there transaction isolation somehow? is the recommended policy therefore 'only ever use a single RPC writer' ?
217 2016-11-21 22:38:42	0|gmaxwell|sipa: ugh. so if you send some and recieve the same amount ... the double payment still goes through?
218 2016-11-21 22:39:28	0|kanzure|'transaction isolation' probably means database-kind of transaction model, not transactions?
219 2016-11-21 22:39:49	0|skyraider|uh, yeah, not bitcoin transaction. isolation as in the I from ACID.
220 2016-11-21 22:39:51	0|gmaxwell|skyraider: yes, you just retry in that model.
221 2016-11-21 22:40:19	0|kanzure|isn't that a failure
222 2016-11-21 22:40:26	0|kanzure|the idea is to prevent stales from causing you to retry
223 2016-11-21 22:41:08	0|gmaxwell|why would that be a failure? The retry causes no harm. The system is concurrent and you want an atomic change, there will always be retries needed.
224 2016-11-21 22:41:35	0|kanzure|you mean existing system, or proposed system is concurrent?
225 2016-11-21 22:41:43	0|gmaxwell|kanzure: so going back to the nonce, how do you propose avoding having to store nonces forever?
226 2016-11-21 22:42:29	0|skyraider|gmaxwell: sorry, could you clarify please - is RPC safe for concurrent writes at the moment, or are you saying stale reads (if you have, say, two concurrent connections) are possible, and therefore one should use a single RPC writer?
227 2016-11-21 22:43:37	0|gmaxwell|skyraider: your question is underspecified. It's perfectly fine to make concurrent writes.  But kanzure is asking about what happens when you get a socket failure on one of your writes.
228 2016-11-21 22:43:49	0|gmaxwell|If you just retry you may double pay.
229 2016-11-21 22:43:52	0|midnightmagic|wtf man another earthquake near fukushima?
230 2016-11-21 22:47:59	0|skyraider|balance data in read 1).
231 2016-11-21 22:47:59	0|skyraider|right, i don't mean to ask whether the application will crash if you do concurrent writes. rather, whether a client can expect the situation of "1) read balance in client thread A, 2) write balance in client thread B that makes 1's read stale, 3) write balance in client thread A" can result in the client double paying, due to making a decision based upon the
232 2016-11-21 22:48:50	0|sipa|skyraider: there is no 'write balance'
233 2016-11-21 22:49:05	0|sipa|there is 'attempt to make a transaction'
234 2016-11-21 22:49:32	0|skyraider|yes, sorry, shorthand for "try to send some bitcoin"
235 2016-11-21 22:51:14	0|gmaxwell|skyraider: I think you're overthinking things now.  If you call send* you will probably send.  If you call send once per payment you want to make you will never double pay.
236 2016-11-21 22:51:42	0|gmaxwell|if you are doing things based on 'balance' then god knows what will happen, as balance is not monotone. :)
237 2016-11-21 22:51:48	0|sipa|skyraider: if you "attempt to make a transaction" twice, it will obviously send twice... anything else would be highly unexpected behaviour
238 2016-11-21 22:52:19	0|kanzure|i think bitcoind seems to really very strongly want all applications to have only one rpc connection
239 2016-11-21 22:52:31	0|gmaxwell|kanzure: not at all.
240 2016-11-21 22:52:39	0|kanzure|there is no protection against stale reads at all
241 2016-11-21 22:52:45	0|kanzure|there is nothing in here at all like ACID
242 2016-11-21 22:53:06	0|gmaxwell|which is orthorgonal to "have only one rpc connection"
243 2016-11-21 22:53:18	0|kanzure|look i'm fine with one rpc client in serial. that's fine with me.  but i'm not sure everyone else knows to do that.
244 2016-11-21 22:57:11	0|skyraider|change via concurrent writes). it seems a safe solution for b) is "only ever execute RPC writes serially" and for a), "because we are doing serial writes, in which another write can't occur after my safety-check-read, we can query to see whether the send already occurred (the safety-check-read) before retrying." tldr; serial writes enables "should i retry?"
245 2016-11-21 22:57:11	0|skyraider|it might help to put on a client's shoes here. context was https://github.com/petertodd/python-bitcoinlib/pull/128 in which a blind retry python decorator was considered. turns out this can a) result in actually sending bitcoin multiple times, b) result in decisions made on stale read data (balance was just an example - really any bitcoind state subject to
246 2016-11-21 22:57:11	0|skyraider|logic in a safe way.
247 2016-11-21 22:58:08	0|gmaxwell|I disagree.
248 2016-11-21 22:58:27	0|gmaxwell|What you want is a write which is ordered with respect to a read.
249 2016-11-21 22:58:39	0|gmaxwell|Have you already paid is not something which can become untrue.
250 2016-11-21 22:58:59	0|kanzure|previous answer to "have you already paid" can become untrue
251 2016-11-21 22:59:19	0|gmaxwell|yes but only by performing a write that makes that payment.
252 2016-11-21 22:59:19	0|skyraider|yes, a write ordered with respect to a particular read (i.e., transaction isolation in the database sense) is a more general and better solution than serial writes.
253 2016-11-21 23:01:03	0|gmaxwell|Both of you keep repeating serial, and that is just wrong and in a meaningful way compared to typical usage.  You can perform as many sends as you want. But if you want to retry a send you must read first to make sure the send wasn't already performed.   Running two sends for the same payment in parallel is insanity in any case, as you have no way of knowing if the other went through (and usuall
254 2016-11-21 23:01:09	0|gmaxwell|y it does).
255 2016-11-21 23:04:37	0|bsm1175321|Of course, there are probably many people performing this logic now, since the "commit" of a send doesn't actually occur until it's mined.  The need to fee-bump is probably a more common cause of retries than RPC connection drops...
256 2016-11-21 23:04:53	0|skyraider|if you are e.g. running a group of ec2 servers or something and experience a temporary DNS failure of the kind that's common on AWS, that python-bitcoinlib decorator could result in probably 1-50 double payments per month, assuming you are sending every 30 seconds or so. serial writes seems to be a quick, safe way to get don't-pay-twice safety while querying
257 2016-11-21 23:04:54	0|skyraider|the current version of bitcoind. as to running sends concurrently, yes, it seems unusual / nonrealistic use case to do this.
258 2016-11-21 23:05:50	0|kanzure|fwiw i don't actually use sendmany
259 2016-11-21 23:07:43	0|kanzure|or any spends for that matter... heh.
260 2016-11-21 23:07:53	0|sipa|you hodler
261 2016-11-21 23:08:42	0|kanzure|(yeah who would ever want to spend coins??)
262 2016-11-21 23:10:51	0|midnightmagic|gmaxwell: that's how MP (operating manually) managed to rip himself off.
263 2016-11-21 23:13:08	0|kanzure|midnightmagic: go on?
264 2016-11-21 23:13:27	0|skyraider|and yeah i understand the criticism about serial writes - it seems at first it should be more like 'read before you write'.  but, you have to be careful with that because your read could easily be stale. and if you are running concurrent bitcoind sends in different client-application-level thread of execution, well - you shouldn't do this. you should write
265 2016-11-21 23:13:27	0|skyraider|in one thread because the lack of transaction isolation (read/write ordering) means client implementors might not know to handle what happens when a balance they just checked is now gone.
266 2016-11-21 23:13:56	0|skyraider|but, yes, this is not an actual real-world use case over here at the moment. so admitted.
267 2016-11-21 23:14:01	0|skyraider|thanks
268 2016-11-21 23:14:20	0|midnightmagic|kanzure: not respending himself, instead sending multiple sends to the same destination more than once, and the receivers obviously refused to give anything back (which was fair by MP's own mercenrary rules.) If he had properly respent himself, updated his sends, or otherwise presumed the sends could be mined at any time, he wouldn't have lost anything.
269 2016-11-21 23:23:19	0|kanzure|is the assumption that there is always a single node worker and never more than one? if so, we should probably add this to docs.
270 2016-11-21 23:23:42	0|kanzure|*node worker (rpc client communicating to bitcoind)
271 2016-11-21 23:25:22	0|bsm1175321|Having more than one violates the Isolation requirement of ACID.  Would it be interesting to require isolation on open wallets WRT rpc clients?  That is, two RPC connections must have mutually exclusive wallets?
272 2016-11-21 23:30:09	0|gmaxwell|kanzure: I keep telling you that there is no such assumption.
273 2016-11-21 23:33:44	0|sipa|kanzure: that also doesn't in any way help with the double issue problem
274 2016-11-21 23:53:48	0|kanzure|ah, there is a lockunspent at least.