1 2012-06-02 00:17:25 graingert1 has joined
   2 2012-06-02 00:18:22 kronosvl has joined
   3 2012-06-02 00:18:33 kronosvl has left ()
   4 2012-06-02 00:19:02 graingert has quit (Ping timeout: 245 seconds)
   5 2012-06-02 00:19:20 spiborg has quit (Read error: Connection reset by peer)
   6 2012-06-02 00:22:22 rdponticelli has quit (Ping timeout: 245 seconds)
   7 2012-06-02 00:30:25 Graet has quit (Ping timeout: 244 seconds)
   8 2012-06-02 00:31:13 rdponticelli has joined
   9 2012-06-02 00:36:21 brwyatt is now known as Away!~brwyatt@pool-96-226-236-130.dllstx.fios.verizon.net|brwyatt
  10 2012-06-02 00:36:51 d4de has quit (Ping timeout: 246 seconds)
  11 2012-06-02 00:36:58 Z0rZ0rZ0r has quit (Quit: Wheeeee)
  12 2012-06-02 00:37:22 rdponticelli has quit (Ping timeout: 245 seconds)
  13 2012-06-02 00:45:24 <luke-jr> would it be evil to finney attack satoshidice?
  14 2012-06-02 00:45:30 <TuxBlackEdo> no
  15 2012-06-02 00:45:31 <TuxBlackEdo> do it
  16 2012-06-02 00:45:44 <TuxBlackEdo> i'll give you 5 btc on success
  17 2012-06-02 00:53:41 <Joric> whoa it's complicated, how about attacking 1-confirmation payments is it possible? heard MM intentionally forked the blockchain
  18 2012-06-02 00:54:12 <BlueMatt> how ok would it be to let wallet txes end up getting in mapOrphanTransactions?
  19 2012-06-02 00:54:52 <TuxBlackEdo> mm intentionally forked the blockchain?
  20 2012-06-02 00:56:26 d4de has joined
  21 2012-06-02 00:57:14 <Joric> TuxBlackEdo, yep, orphaned a few valid blocks
  22 2012-06-02 00:57:43 <sipa> you cannot orphan a block
  23 2012-06-02 00:57:49 <sipa> you can make it stale
  24 2012-06-02 00:58:09 <sipa> orphan means that it has no (known) parent
  25 2012-06-02 00:58:50 <Joric> well, or that
  26 2012-06-02 00:59:23 <gmaxwell> Deepbit has invalidated an enormous number of blocks, your point?
  27 2012-06-02 01:00:07 <sipa> not intentionally, i hope
  28 2012-06-02 01:00:08 <luke-jr> sipa: no
  29 2012-06-02 01:00:09 <Joric> i just was wondering is it possible to reverse 1-confirmation payments
  30 2012-06-02 01:00:13 <luke-jr> sipa: orphan means it's not in the main chain
  31 2012-06-02 01:00:26 <luke-jr> Joric: expensive, but yes
  32 2012-06-02 01:00:36 <sipa> luke-jr: where do you find that terminology?
  33 2012-06-02 01:00:47 <luke-jr> sipa: listtransactions
  34 2012-06-02 01:00:48 <gmaxwell> Joric: yes if it wasn't we wouldn't be recommending to wait at least six.
  35 2012-06-02 01:00:53 <BlueMatt> Joric: reliably? you have to have 51%, sometimes, yea, but expensive
  36 2012-06-02 01:00:57 <luke-jr> sipa: also the terminology almost all pools use
  37 2012-06-02 01:00:58 <gmaxwell> sipa: no, not intentionally as far as I believe or know.
  38 2012-06-02 01:01:12 <sipa> luke-jr: really? ok
  39 2012-06-02 01:01:34 <gmaxwell> BlueMatt: you don't have to have 51% .. you have to have large hash power and get luckier than the network. Not quite the same thing.
  40 2012-06-02 01:01:54 <sipa> luke-jr: that is the generation transaction, not the block
  41 2012-06-02 01:01:57 <gmaxwell> At >50% you will evenutally win if you can sustain it.. though at only 51% it might take you a long time.
  42 2012-06-02 01:02:00 <BlueMatt> s/reliably/in theory, always/
  43 2012-06-02 01:02:05 <sipa> the coinbase of a stale block is an orphan
  44 2012-06-02 01:02:34 <sipa> and the source code afaik uses orphan only in the meaning of without parent
  45 2012-06-02 01:03:14 <sipa> anyway, not important
  46 2012-06-02 01:09:07 D34TH has quit (Ping timeout: 276 seconds)
  47 2012-06-02 01:09:23 D34TH has joined
  48 2012-06-02 01:09:24 D34TH has quit (Changing host)
  49 2012-06-02 01:09:24 D34TH has joined
  50 2012-06-02 01:12:33 mxmxmx has quit (Ping timeout: 246 seconds)
  51 2012-06-02 01:13:28 mxmxmx has joined
  52 2012-06-02 01:24:45 cuqa has quit (Ping timeout: 260 seconds)
  53 2012-06-02 01:26:40 darkee has quit (!~darkee@gateway/tor-sasl/darkee|Ping timeout: 276 seconds)
  54 2012-06-02 01:30:57 setkeh` has quit (Read error: No route to host)
  55 2012-06-02 01:32:56 setkeh has joined
  56 2012-06-02 01:33:47 sirk390 has quit (Quit: Leaving.)
  57 2012-06-02 01:36:13 darkee has joined
  58 2012-06-02 01:37:39 cuqa has joined
  59 2012-06-02 01:42:13 Tykling has quit (Excess Flood)
  60 2012-06-02 01:47:53 Tykling has joined
  61 2012-06-02 01:50:35 salitron has quit (Ping timeout: 245 seconds)
  62 2012-06-02 01:51:56 maqr has quit (Quit: :o)
  63 2012-06-02 02:00:55 graingert1 has quit (Read error: Connection reset by peer)
  64 2012-06-02 02:01:31 osmosis has joined
  65 2012-06-02 02:01:57 da2ce713 has joined
  66 2012-06-02 02:03:49 da2ce7 has quit (Ping timeout: 244 seconds)
  67 2012-06-02 02:06:10 danbri has joined
  68 2012-06-02 02:10:00 minimoose has joined
  69 2012-06-02 02:16:31 danbri has quit (Remote host closed the connection)
  70 2012-06-02 02:19:26 knotwork has quit (Ping timeout: 244 seconds)
  71 2012-06-02 02:20:55 knotwork has joined
  72 2012-06-02 02:35:10 da2ce713 is now known as da2ce7
  73 2012-06-02 02:44:41 TheSeven has quit (Disconnected by services)
  74 2012-06-02 02:44:51 [7] has joined
  75 2012-06-02 02:48:10 darsk1ez has quit (Ping timeout: 260 seconds)
  76 2012-06-02 02:55:31 Graet has joined
  77 2012-06-02 02:57:34 <gribble> New news from bitcoinrss: luke-jr opened pull request 1409 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1409>
  78 2012-06-02 03:05:46 tyn has joined
  79 2012-06-02 03:06:33 one_zero has joined
  80 2012-06-02 03:07:26 darsk1ez has joined
  81 2012-06-02 03:19:53 Diablo-D3 has quit (Ping timeout: 244 seconds)
  82 2012-06-02 03:29:01 RainbowDashh has joined
  83 2012-06-02 03:34:04 eoss has joined
  84 2012-06-02 03:39:07 MobiusL has quit (Quit: Ex-Chat)
  85 2012-06-02 03:39:17 setkeh has quit (Read error: Connection reset by peer)
  86 2012-06-02 03:39:38 setkeh has joined
  87 2012-06-02 03:40:32 tyn has quit (Quit: Leaving)
  88 2012-06-02 03:41:07 minimoose has left ()
  89 2012-06-02 03:43:24 MobiusL has joined
  90 2012-06-02 03:43:50 darkee has quit (!~darkee@gateway/tor-sasl/darkee|Ping timeout: 276 seconds)
  91 2012-06-02 03:55:07 RainbowDashh has quit (Quit: RainbowDashh)
  92 2012-06-02 03:55:59 darkee has joined
  93 2012-06-02 04:27:30 RainbowDashh has joined
  94 2012-06-02 05:01:38 da2ce7 has quit (Read error: Connection reset by peer)
  95 2012-06-02 05:02:53 da2ce7 has joined
  96 2012-06-02 05:28:42 brwyatt is now known as brwyatt|Away
  97 2012-06-02 05:50:32 brwyatt is now known as Away!~brwyatt@pool-96-226-236-130.dllstx.fios.verizon.net|brwyatt
  98 2012-06-02 05:50:47 brwyatt is now known as brwyatt|Away
  99 2012-06-02 05:56:28 dvide has joined
 100 2012-06-02 05:57:36 Turingi has joined
 101 2012-06-02 05:57:36 Turingi has quit (Changing host)
 102 2012-06-02 05:57:36 Turingi has joined
 103 2012-06-02 06:05:48 MobiusL has quit (Quit: Ex-Chat)
 104 2012-06-02 06:06:54 MobiusL has joined
 105 2012-06-02 06:24:44 Joric_ has joined
 106 2012-06-02 06:24:44 Joric_ has quit (Changing host)
 107 2012-06-02 06:24:44 Joric_ has joined
 108 2012-06-02 06:25:17 Joric has quit (Ping timeout: 248 seconds)
 109 2012-06-02 06:27:27 Joric has joined
 110 2012-06-02 06:27:28 Joric has quit (Changing host)
 111 2012-06-02 06:27:28 Joric has joined
 112 2012-06-02 06:28:13 eoss has quit (Ping timeout: 245 seconds)
 113 2012-06-02 06:29:01 Joric_ has quit (Ping timeout: 248 seconds)
 114 2012-06-02 06:39:36 Joric_ has joined
 115 2012-06-02 06:39:36 Joric_ has quit (Changing host)
 116 2012-06-02 06:39:36 Joric_ has joined
 117 2012-06-02 06:41:55 Joric has quit (Ping timeout: 265 seconds)
 118 2012-06-02 06:43:02 Joric_ is now known as Joric
 119 2012-06-02 06:48:13 nanotube has quit (Ping timeout: 245 seconds)
 120 2012-06-02 06:48:35 gribble has quit (Read error: Connection reset by peer)
 121 2012-06-02 06:49:10 gribble has joined
 122 2012-06-02 06:49:37 darkskiez2 has joined
 123 2012-06-02 06:50:57 RazielZ has joined
 124 2012-06-02 06:51:46 Joric has quit (Ping timeout: 252 seconds)
 125 2012-06-02 06:52:48 darsk1ez has quit (Ping timeout: 245 seconds)
 126 2012-06-02 06:53:58 gribble has quit (Ping timeout: 252 seconds)
 127 2012-06-02 06:54:51 Skaag has quit (Read error: Connection reset by peer)
 128 2012-06-02 06:55:19 Skaag has joined
 129 2012-06-02 06:57:05 gribble has joined
 130 2012-06-02 06:58:07 bitcoinbulletin has quit (Ping timeout: 252 seconds)
 131 2012-06-02 06:59:36 D34TH has quit (Read error: Connection reset by peer)
 132 2012-06-02 07:00:29 gribble has quit (Disconnected by services)
 133 2012-06-02 07:08:43 gribble has joined
 134 2012-06-02 07:13:19 bitcoinbulletin has joined
 135 2012-06-02 07:13:27 osmosis has quit (Quit: Leaving)
 136 2012-06-02 07:15:02 nanotube has joined
 137 2012-06-02 07:16:25 molecular has quit (Ping timeout: 244 seconds)
 138 2012-06-02 07:19:06 Xunie has joined
 139 2012-06-02 07:19:42 m00p has joined
 140 2012-06-02 07:20:10 phantomcircuit has joined
 141 2012-06-02 07:28:49 molecular has joined
 142 2012-06-02 07:41:01 Zarutian has quit (Quit: Zarutian)
 143 2012-06-02 07:41:14 Turingi has quit (Read error: Connection reset by peer)
 144 2012-06-02 07:41:37 erle- has joined
 145 2012-06-02 08:09:52 ovidiusoft has joined
 146 2012-06-02 08:14:22 sirk390 has joined
 147 2012-06-02 08:18:59 phantomcircuit has quit (Remote host closed the connection)
 148 2012-06-02 08:23:17 Diablo-D3 has joined
 149 2012-06-02 08:25:12 sgornick has quit (Ping timeout: 240 seconds)
 150 2012-06-02 08:31:19 phantomcircuit has joined
 151 2012-06-02 08:34:28 sgornick has joined
 152 2012-06-02 09:06:58 TD has joined
 153 2012-06-02 09:12:20 silp has joined
 154 2012-06-02 09:12:48 Prattler has joined
 155 2012-06-02 09:14:13 silpee has quit (Ping timeout: 244 seconds)
 156 2012-06-02 09:20:17 graingert has joined
 157 2012-06-02 09:20:25 graingert has left ()
 158 2012-06-02 09:30:35 _Fireball has joined
 159 2012-06-02 09:34:34 vigilyn has joined
 160 2012-06-02 09:42:48 Ken` has joined
 161 2012-06-02 09:54:33 Xunie has quit (Quit: Can God microwave a taco so hot that not even *HE* can eat it without burns?)
 162 2012-06-02 10:00:38 Joric has joined
 163 2012-06-02 10:03:57 copumpkin has quit (Ping timeout: 245 seconds)
 164 2012-06-02 10:05:45 copumpkin has joined
 165 2012-06-02 10:13:42 m00p has quit (Ping timeout: 240 seconds)
 166 2012-06-02 10:20:31 Joric has quit ()
 167 2012-06-02 10:21:03 erle- has quit (Quit: erle-)
 168 2012-06-02 10:21:09 datagutt has joined
 169 2012-06-02 10:25:30 graingert has joined
 170 2012-06-02 10:26:09 TD has quit (Quit: TD)
 171 2012-06-02 10:34:09 Z0rZ0rZ0r has joined
 172 2012-06-02 10:34:14 da2ce7 has quit (Read error: Connection reset by peer)
 173 2012-06-02 10:35:31 da2ce7 has joined
 174 2012-06-02 10:36:12 nanotube has quit (Read error: Connection reset by peer)
 175 2012-06-02 10:36:12 gribble has quit (Read error: Connection reset by peer)
 176 2012-06-02 10:38:48 gribble has joined
 177 2012-06-02 10:39:36 nanotube has joined
 178 2012-06-02 10:42:41 skeledrew has joined
 179 2012-06-02 10:42:44 hnz has joined
 180 2012-06-02 10:45:59 tsche has joined
 181 2012-06-02 10:54:54 saieko has joined
 182 2012-06-02 10:58:53 toffoo has quit ()
 183 2012-06-02 11:04:10 da2ce7 has quit (Read error: Connection reset by peer)
 184 2012-06-02 11:05:30 da2ce7 has joined
 185 2012-06-02 11:12:03 rdponticelli has joined
 186 2012-06-02 11:16:14 Z0rZ0rZ0r has quit (Quit: Leaving)
 187 2012-06-02 11:16:38 Z0rZ0rZ0r has joined
 188 2012-06-02 11:22:27 rdponticelli_ has joined
 189 2012-06-02 11:23:05 rdponticelli has quit (Ping timeout: 256 seconds)
 190 2012-06-02 11:26:46 rdponticelli has joined
 191 2012-06-02 11:27:23 rdponticelli_ has quit (Ping timeout: 245 seconds)
 192 2012-06-02 11:28:09 TD has joined
 193 2012-06-02 11:29:21 mmoya has joined
 194 2012-06-02 11:39:20 Diapolo has joined
 195 2012-06-02 11:40:05 darkee has quit (!~darkee@gateway/tor-sasl/darkee|Remote host closed the connection)
 196 2012-06-02 11:41:03 darkee has joined
 197 2012-06-02 11:46:27 t7 has joined
 198 2012-06-02 11:50:19 yorick has quit (Ping timeout: 260 seconds)
 199 2012-06-02 11:51:42 copumpkin has quit (Ping timeout: 248 seconds)
 200 2012-06-02 11:52:14 copumpkin has joined
 201 2012-06-02 11:54:04 one_zero has quit ()
 202 2012-06-02 12:03:23 rdponticelli has quit (Read error: Connection reset by peer)
 203 2012-06-02 12:07:36 rdponticelli has joined
 204 2012-06-02 12:16:54 graingert has quit (Remote host closed the connection)
 205 2012-06-02 12:20:07 setkeh has quit (Read error: No route to host)
 206 2012-06-02 12:20:28 Diapolo has left ()
 207 2012-06-02 12:22:01 TD has quit (Quit: TD)
 208 2012-06-02 12:22:27 graingert has joined
 209 2012-06-02 12:24:31 erle- has joined
 210 2012-06-02 12:32:44 mmoya has quit (Ping timeout: 244 seconds)
 211 2012-06-02 12:52:47 pooolop has joined
 212 2012-06-02 12:54:38 <pooolop> hi, i have a question abount merged mining + pooshpol + bitcoind 0.6.2.  is there any path to bitcoin 0.6.2 ? to get it all together work ?
 213 2012-06-02 13:08:43 rdponticelli has quit (Read error: Connection reset by peer)
 214 2012-06-02 13:13:45 jamesstanley has quit (Quit: Leaving)
 215 2012-06-02 13:16:55 eoss has joined
 216 2012-06-02 13:22:11 setkeh has joined
 217 2012-06-02 13:36:29 <Eliel> pooolop: pooshpol? do you mean pushpoold?
 218 2012-06-02 13:50:12 rdponticelli has joined
 219 2012-06-02 13:56:35 <[7]> hm, doesn't seem like having bitcoin on an ssd is a good idea
 220 2012-06-02 13:56:59 <[7]> it makes bitcoin-qt on my laptop start up in minutes instead of seconds, but causes a damn lot of ssd wear
 221 2012-06-02 13:57:10 <[7]> seems like it's more than the rest of the system combined
 222 2012-06-02 13:57:30 <BlueMatt> that doesnt make sense
 223 2012-06-02 13:57:34 <Eliel> [7]: what kind of ssd?
 224 2012-06-02 13:57:47 <BlueMatt> nothing is gonna start slower on any kind of sane ssd than a rotation hdd
 225 2012-06-02 13:57:57 <BlueMatt> unless its one of those shitty ssd thats is unbearably slow
 226 2012-06-02 13:58:06 <[7]> er, i said that the wrong way round
 227 2012-06-02 13:58:18 <[7]> it starts in seconds instead of minutes, but causes awful lots of wear
 228 2012-06-02 14:08:11 <[7]> in the last 13h it caused a whopping 6GB of writes
 229 2012-06-02 14:08:35 <[7]> SSDs MTBFs are specified on 20GB/day IIUC
 230 2012-06-02 14:09:46 <[7]> my ssd has a lifetime write capacity of 1250TB
 231 2012-06-02 14:10:20 <sipa> 6Gb during chain syncup, or just stable usage?
 232 2012-06-02 14:10:42 <[7]> well, starting up (after a downtime of maybe 5 minutes) and then just idling
 233 2012-06-02 14:10:51 <[7]> so normal long-term behavior on a laptop
 234 2012-06-02 14:11:11 <gribble> New news from bitcoinrss: Diapolo opened issue 1410 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1410>
 235 2012-06-02 14:11:13 * Eliel wonders what bdb has been up to writing 6GB during 13 hours.
 236 2012-06-02 14:11:41 <[7]> 48MB during the last 6:45 minutes
 237 2012-06-02 14:11:56 <[7]> just idling in the background
 238 2012-06-02 14:12:20 <sipa> well, bitcoin dumps its address database (typically 1-2 megabytes) to a file every two minutes
 239 2012-06-02 14:12:25 <[7]> oddly enough it has only read 460MB while writing 6GB
 240 2012-06-02 14:12:35 <[7]> I suspect that this has something to do with massive fsyncs
 241 2012-06-02 14:12:42 <sipa> yes, certainly
 242 2012-06-02 14:13:56 <[7]> anything I could try to help nail down the cause?
 243 2012-06-02 14:14:00 <[7]> run it with eatmydata for a day?
 244 2012-06-02 14:15:05 <[7]> looks like this is one more reason to decouple blockchain storage from the wallet and GUI
 245 2012-06-02 14:15:18 <[7]> I do care both about disk space and writes on my laptop
 246 2012-06-02 14:15:27 <[7]> but I don't at all on my server, which is running a bitcoind anyway
 247 2012-06-02 14:15:37 <[7]> but I'd like to have bitcoin-qt on my laptop
 248 2012-06-02 14:16:02 <[7]> I don't care whether the wallet is on the server or on the laptop, but for some people the wallet would ideally be on a dedicated secure device
 249 2012-06-02 14:16:16 <[7]> so these three components should probably be decoupled somehow
 250 2012-06-02 14:16:54 <[7]> no way I'll let bitcoin cut my ssd life into half
 251 2012-06-02 14:18:07 <[7]> in case you're curious how much traffic it caused on your box, try this:
 252 2012-06-02 14:18:08 <[7]> $ cat /proc/$(pidof bitcoin-qt)/io
 253 2012-06-02 14:18:08 hnz has quit (Ping timeout: 244 seconds)
 254 2012-06-02 14:19:47 <[7]> so... any ideas how I could reduce that traffic, except for remote-X11ing it onto my server? :)
 255 2012-06-02 14:20:58 <[7]> my addr.dat is 6.5MB btw
 256 2012-06-02 14:21:02 <[7]> so that could be the main cause
 257 2012-06-02 14:21:14 <[7]> what exactly is stored in that file?
 258 2012-06-02 14:21:26 <[7]> what happens if I just delete it? will it regenerate it properly?
 259 2012-06-02 14:21:41 <[7]> i.e. could I move that to a ramdisk?
 260 2012-06-02 14:22:21 Prattler has quit (Quit: Ex-Chat)
 261 2012-06-02 14:24:36 <BlueMatt> [7]: is there a way for you to track io usage per file?
 262 2012-06-02 14:24:53 <[7]> I don't know of any, but I'd have some other use for that as well
 263 2012-06-02 14:25:07 <BlueMatt> I think lsof output has it in there somehow?
 264 2012-06-02 14:27:37 <BlueMatt> though I have nfc if it counts just data written, or data physically written which could be significantly different due to all the fsyncs
 265 2012-06-02 14:27:52 p0s has joined
 266 2012-06-02 14:28:34 <sipa> [7]: you can safely delete addr.dat
 267 2012-06-02 14:28:54 <sipa> [7]: 0.7.0 will stop using addr.dat and use peers.dat instead, with a more compact format
 268 2012-06-02 14:28:59 <BlueMatt> ln -s /dev/null addr.dat
 269 2012-06-02 14:29:15 <BlueMatt> how mad would bitcoind get with that?
 270 2012-06-02 14:29:36 <sipa> i guess for peers.dat it wouldn't really be a problem - bdb may complain :)
 271 2012-06-02 14:30:06 <BlueMatt> well, ln addr.dat to a tmpfs
 272 2012-06-02 14:31:06 <[7]> BlueMatt: lsof doesn't show traffic, just the current position in that file, right?
 273 2012-06-02 14:31:38 darkee is now known as !~darkee@gateway/tor-sasl/darkee|darkee
 274 2012-06-02 14:31:44 <BlueMatt> [7]: mmm, might not, sorry, I must be dreaming
 275 2012-06-02 14:32:41 <[7]> hm, bitcoin isn't happy *at all* with addr.dat symlinked to some nonexistant file on /tmp
 276 2012-06-02 14:33:09 <[7]> even linked to a zero-byte file:
 277 2012-06-02 14:33:13 <[7]> A fatal error occured. Bitcoin can no longer continue safely and will quit.
 278 2012-06-02 14:33:13 <[7]> EXCEPTION: 22DbRunRecoveryException
 279 2012-06-02 14:33:13 <[7]> DbEnv::open: DB_RUNRECOVERY: Fatal error, run database recovery
 280 2012-06-02 14:33:13 <[7]> bitcoin in Runaway exception
 281 2012-06-02 14:33:37 <BlueMatt> linked to a 0-byte file, probably not, linked to a working addr.dat on tmpfs, should work
 282 2012-06-02 14:35:20 <[7]> removing it altogether seems to work though
 283 2012-06-02 14:35:49 <[7]> after starting and stopping it, that file is like 500K
 284 2012-06-02 14:36:00 <BlueMatt> will it not get re-written all the time if you just remove it, though?
 285 2012-06-02 14:36:23 <[7]> I wouldn't care if it would rewrite it to a ramdisk all the time :P
 286 2012-06-02 14:36:34 <[7]> so i just need to find a way how to move that to /tmp
 287 2012-06-02 14:37:58 <[7]> will it go mad if i occasionally reset addr.dat to an older state?
 288 2012-06-02 14:38:15 <[7]> i.e. copy a template addr.dat to the ramdisk on every boot and symlink to that
 289 2012-06-02 14:38:29 <[7]> i.e. does it need to be in sync with the other files in .bitcoin?
 290 2012-06-02 14:38:56 <sipa> yes
 291 2012-06-02 14:39:06 <sipa> in particular, those in the database subdir
 292 2012-06-02 14:39:08 <BlueMatt> run with -dbdetach
 293 2012-06-02 14:39:17 <[7]> hm, damn
 294 2012-06-02 14:39:19 <BlueMatt> and remove database/* (assuming clean shutdown)
 295 2012-06-02 14:39:26 <[7]> BlueMatt: what does that flag do?
 296 2012-06-02 14:39:34 <BlueMatt> or...backup database and then remove
 297 2012-06-02 14:39:44 <[7]> i.e. rename database?
 298 2012-06-02 14:39:44 <BlueMatt> oh, thats only master, I think
 299 2012-06-02 14:39:53 <BlueMatt> yea
 300 2012-06-02 14:40:04 <BlueMatt> its default before master, or was it in 0.6.2, sipa?
 301 2012-06-02 14:40:31 <[7]> what does bitcoin store where btw?
 302 2012-06-02 14:40:39 <BlueMatt> its all in ~/.bitcoin
 303 2012-06-02 14:40:49 <[7]> how much effort is it to regenerate .bitcoin/database?
 304 2012-06-02 14:40:49 <BlueMatt> blkNNNN.dat are raw blocks
 305 2012-06-02 14:41:01 <[7]> what is blkindex.dat, and can it be regenerated from blkNNN.dat?
 306 2012-06-02 14:41:03 <sipa> -detachdb is since 0.6.0
 307 2012-06-02 14:41:14 <BlueMatt> its (usually) safe to remove .bitcoin/databse if you run with -detachdb and had a clean shutdown
 308 2012-06-02 14:41:34 <[7]> so what is stored in that directory?
 309 2012-06-02 14:41:35 <gribble> New news from bitcoinrss: TheBlueMatt opened pull request 1411 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1411>
 310 2012-06-02 14:41:36 <BlueMatt> but better to move it instead in case something goes wrong
 311 2012-06-02 14:41:47 <BlueMatt> bdb log files
 312 2012-06-02 14:41:55 <[7]> BlueMatt: well, worst case would be having to redownload the blockchain, right?
 313 2012-06-02 14:42:05 <BlueMatt> worse case would be corrupting your wallet
 314 2012-06-02 14:42:53 <[7]> yeah, but IIUC restoring an old wallet state (within 100 address generations) and removing the rest of .bitcoin should take care of pretty much any kind of failure, right?
 315 2012-06-02 14:43:11 <BlueMatt> yea, you may lose accounts, though
 316 2012-06-02 14:43:29 <[7]> accounts?
 317 2012-06-02 14:43:33 etotheipi_ has joined
 318 2012-06-02 14:43:34 <BlueMatt> s/may/will/
 319 2012-06-02 14:43:34 <BlueMatt> assuming that wallet.dat was run with -detachdb and cleanly shut down
 320 2012-06-02 14:43:51 <BlueMatt> eg address->human-readable name mappings
 321 2012-06-02 14:43:58 <[7]> where are those stored?
 322 2012-06-02 14:46:13 <sipa> you lose your address book, account balances, and transaction comments
 323 2012-06-02 14:46:16 <BlueMatt> wallet
 324 2012-06-02 14:46:24 <sipa> you will not lose money
 325 2012-06-02 14:46:26 <[7]> ah, so just everything since the last wallet backup
 326 2012-06-02 14:46:29 <BlueMatt> you'll lose everything but money
 327 2012-06-02 14:46:38 <BlueMatt> essentially
 328 2012-06-02 14:46:43 <[7]> ok, fine
 329 2012-06-02 14:46:46 Zarutian has joined
 330 2012-06-02 14:47:02 <[7]> i did remove addr.dat and database now, and it doesn't start anymore
 331 2012-06-02 14:47:06 RainbowDashh has quit (Quit: RainbowDashh)
 332 2012-06-02 14:47:15 <[7]> EXCEPTION: 11DbException
 333 2012-06-02 14:47:15 <[7]> Db::open: Das Argument ist ungültig
 334 2012-06-02 14:47:15 <[7]> bitcoin in Runaway exception
 335 2012-06-02 14:47:20 <[7]> i.e. invalid argument
 336 2012-06-02 14:50:25 <[7]> putting database back doesn't help either
 337 2012-06-02 14:50:34 <[7]> did it screw up by blksomething.dat files?
 338 2012-06-02 14:51:04 <sipa> don't mess with the separate database files unless you started with -detachdb the previous run
 339 2012-06-02 14:51:08 minimoose has joined
 340 2012-06-02 14:51:22 <sipa> copying files in/out or deleting them will corrupt the directory if you, otherwise
 341 2012-06-02 14:51:24 <[7]> ah, the *previous* run... that's valuable information
 342 2012-06-02 14:51:27 <sipa> (except for the wallet)
 343 2012-06-02 14:51:43 <gribble> New news from bitcoinrss: TheBlueMatt reopened pull request 1411 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1411>
 344 2012-06-02 14:51:44 <sipa> yes, that function disconnects the db files from their environment at shutdown
 345 2012-06-02 14:51:47 <[7]> what is blk0001.dat btw? some proprietary format or BDB?
 346 2012-06-02 14:51:59 <sipa> just a binary concatenation of all bitcoin blocks
 347 2012-06-02 14:52:01 RainbowDashh has joined
 348 2012-06-02 14:52:12 <[7]> i.e. it can't really have screwed that up?
 349 2012-06-02 14:52:21 <[7]> so blkindex.dat is the victim?
 350 2012-06-02 14:52:26 abracadabra has joined
 351 2012-06-02 14:52:29 <sipa> no idea what you did
 352 2012-06-02 14:52:38 <abracadabra> anybody have a problem adding addresses to bitcoin client?
 353 2012-06-02 14:52:46 <sipa> abracadabra: define adding?
 354 2012-06-02 14:52:47 <abracadabra> says i can't add because it's already there
 355 2012-06-02 14:52:54 <abracadabra> i'm adding a sending address
 356 2012-06-02 14:52:54 <[7]> sipa: basically removing the database dir and addr.dat and firing up bitcoin-qt
 357 2012-06-02 14:53:08 <abracadabra>  i check the address book, but it's not there
 358 2012-06-02 14:53:10 <sipa> [7]: yes, blkindex.dat is the problem then
 359 2012-06-02 14:53:14 <abracadabra> restarted the client.. same problem
 360 2012-06-02 14:53:17 <sipa> [7]: i'm afraid you'll have to resync
 361 2012-06-02 14:53:20 <abracadabra> i sorted address book on label and on address, it's not listed
 362 2012-06-02 14:53:33 <sipa> abracadabra: is it an address of your own?
 363 2012-06-02 14:53:40 <abracadabra> no
 364 2012-06-02 14:53:53 <abracadabra> it's an adddress for someone else that i want to send btc to
 365 2012-06-02 14:53:53 <[7]> sipa: well, there shouldn't be any need to re-download if the blk0001.dat file is there, right? i mean, it does have the blocks!
 366 2012-06-02 14:54:14 <sipa> [7]: 0.7.0 has a -loadblock function to import an existing blk0001.dat file
 367 2012-06-02 14:54:30 <sipa> well, will have
 368 2012-06-02 14:54:43 <[7]> should copying over a blkindex.dat from another bitcoin install help?
 369 2012-06-02 14:54:47 <sipa> no
 370 2012-06-02 14:54:52 <[7]> blk0001.dat should be identical, shouldn't it?
 371 2012-06-02 14:54:55 <sipa> no
 372 2012-06-02 14:55:08 <sipa> it contains the blocks in the order you downloaded them
 373 2012-06-02 14:55:15 <sipa> not in the order of the current best blockchain
 374 2012-06-02 14:55:22 <[7]> but copying everything except the wallet from the other instance should work i guess?
 375 2012-06-02 14:55:26 <sipa> yes
 376 2012-06-02 14:55:43 <sipa> if you include the database/ directory
 377 2012-06-02 14:56:00 <sipa> or first run the other instance with -detachdb
 378 2012-06-02 14:57:59 ThomasV has joined
 379 2012-06-02 14:59:19 <[7]> hm, what I'm actually looking for might be described as a "private e-wallet"
 380 2012-06-02 14:59:27 RainbowDashh has quit (Disconnected by services)
 381 2012-06-02 14:59:27 Rabbit67890 has joined
 382 2012-06-02 14:59:34 <[7]> I don't really care whether it's a bitcoin-qt gui or a web application
 383 2012-06-02 14:59:40 <[7]> is there some such thing?
 384 2012-06-02 14:59:52 <sipa> define 'private' ?
 385 2012-06-02 15:00:01 <[7]> hosted on my own box
 386 2012-06-02 15:00:23 <[7]> basically a web gui for bitcoind
 387 2012-06-02 15:00:36 <sipa> hmm, maybe
 388 2012-06-02 15:01:11 <BlueMatt> tcatm has one, not sure how long its been since its been updated
 389 2012-06-02 15:01:34 <BlueMatt> a year
 390 2012-06-02 15:01:35 <BlueMatt> https://github.com/tcatm/bitcoin-js-remote
 391 2012-06-02 15:01:51 rdponticelli has quit (Read error: Connection reset by peer)
 392 2012-06-02 15:01:54 <BlueMatt> would probably need updating, but I used to use it all the time
 393 2012-06-02 15:02:06 <BlueMatt> theres also Spesmilo, but that also hasnt been updated in probably about the same amount of time
 394 2012-06-02 15:02:26 <[7]> I'd prefer to stick with the official "satoshi" client
 395 2012-06-02 15:02:34 rdponticelli has joined
 396 2012-06-02 15:02:36 <[7]> so that tcatm thing looks great :)
 397 2012-06-02 15:02:45 <BlueMatt> spesmilo uses bitcoind as its backend
 398 2012-06-02 15:02:52 <BlueMatt> as does bitcoin-js-remote
 399 2012-06-02 15:03:08 <BlueMatt> major difference is just one is gui one is web
 400 2012-06-02 15:03:45 <BlueMatt> but Id think both would need updating to use the features in modern bitcoind's, but...maybe not
 401 2012-06-02 15:03:49 <[7]> so spesmilo can deal with a remote bitcoind i assume?
 402 2012-06-02 15:04:00 <BlueMatt> it has to
 403 2012-06-02 15:04:04 <BlueMatt> well, it can use local bitcoind
 404 2012-06-02 15:04:50 <[7]> so, from your experience, which one of those two do you think is better?
 405 2012-06-02 15:05:00 <[7]> i.e. more feature complete, more stable, more up to date, ...
 406 2012-06-02 15:05:02 <BlueMatt> its been a while since Ive used either
 407 2012-06-02 15:05:04 <[7]> easier to set up
 408 2012-06-02 15:05:14 <BlueMatt> though, I probably used the latest version of both
 409 2012-06-02 15:05:27 <BlueMatt> spesmilo wasnt perfectly stable, but was much easier to set up
 410 2012-06-02 15:05:31 Rabbit67890 has quit (Quit: Rabbit67890)
 411 2012-06-02 15:05:47 <sipa> any specific reason why you don't want bitcoin-qt ?
 412 2012-06-02 15:05:51 <BlueMatt> iirc
 413 2012-06-02 15:06:19 <[7]> sipa: because it requires the blockchain to be stored locally
 414 2012-06-02 15:06:29 <sipa> you can run it over X forwarding :p
 415 2012-06-02 15:06:35 <sipa> but ok, point taken
 416 2012-06-02 15:07:01 <[7]> i have a bitcoind running for p2pool anyway, so no point in storing the blockchain twice
 417 2012-06-02 15:07:26 <BlueMatt> give it a few months and there may be a blockchaind ;)
 418 2012-06-02 15:07:47 * BlueMatt always runs .bitcoin on a 3g tmpfs
 419 2012-06-02 15:07:53 danbri has joined
 420 2012-06-02 15:07:55 <BlueMatt> (and manually backs-up wallet)
 421 2012-06-02 15:08:06 <[7]> in a few months (which usually turns out to be actually a few years) this beast will have worn out my SSD :P
 422 2012-06-02 15:08:26 <BlueMatt> heh, ok, probably
 423 2012-06-02 15:10:52 da2ce7 has quit (Read error: Connection reset by peer)
 424 2012-06-02 15:12:11 da2ce7 has joined
 425 2012-06-02 15:18:16 Z0rZ0rZ0r has quit (Quit: Leaving)
 426 2012-06-02 15:19:03 rdponticelli_ has joined
 427 2012-06-02 15:20:05 rdponticelli has quit (Ping timeout: 260 seconds)
 428 2012-06-02 15:20:15 rdponticelli_ is now known as rdponticelli
 429 2012-06-02 15:28:35 RainbowDashh has joined
 430 2012-06-02 15:31:20 <[7]> can bitcoind deal with an encrypted wallet?
 431 2012-06-02 15:31:59 <BlueMatt> yea
 432 2012-06-02 15:32:15 <BlueMatt> neither spesmilo nor bitcoin-js-remote can, Im sure, but bitcoind can
 433 2012-06-02 15:32:22 <[7]> how would i specify the password? through the rpc protocol?
 434 2012-06-02 15:33:05 <[7]> and how can i make bitcoind decrypt the wallet for now?
 435 2012-06-02 15:34:33 <sipa> it only decrypts keys in it when it needs them
 436 2012-06-02 15:34:40 slush has joined
 437 2012-06-02 15:34:44 <sipa> permanently decrypting to a file isn't implemented
 438 2012-06-02 15:35:31 <slush> hi
 439 2012-06-02 15:35:55 <slush> I'm writing simple tool which connects to bitcoin p2p network, but I have one issue
 440 2012-06-02 15:36:31 <slush> some nodes, without any obvious reason, are sending me parts of their block chains, but I didn't requested it :-)
 441 2012-06-02 15:36:39 <slush> Is there any way how to prevent this?
 442 2012-06-02 15:37:18 <slush> I need to be bandwidth-conservative as much as possible...
 443 2012-06-02 15:37:20 <sipa> really?
 444 2012-06-02 15:37:28 <slush> yes, I'm sending only version packet
 445 2012-06-02 15:37:35 <sipa> they send you block packets?
 446 2012-06-02 15:37:39 <slush> yes
 447 2012-06-02 15:37:51 <sipa> what version do they report?
 448 2012-06-02 15:38:28 rdponticelli_ has joined
 449 2012-06-02 15:39:01 <slush> well, version and verack
 450 2012-06-02 15:39:20 rdponticelli has quit (Ping timeout: 260 seconds)
 451 2012-06-02 15:42:20 <slush> sipa: moment
 452 2012-06-02 15:44:12 chrisb__ has joined
 453 2012-06-02 15:46:53 rdponticelli_ is now known as rdponticelli
 454 2012-06-02 15:47:08 Skaag has quit (Read error: Connection reset by peer)
 455 2012-06-02 15:47:19 <gmaxwell> slush: I manually did that to some nodes that were reporting old chains, trying to unstick them.
 456 2012-06-02 15:47:31 <gmaxwell> but what you're describing there sounds like an attack.
 457 2012-06-02 15:47:33 Skaag has joined
 458 2012-06-02 15:49:25 <slush> well, it is probably my fault, I responded to "inv" packet with getdata, even on old hashes
 459 2012-06-02 15:49:54 <slush> but I'm wondering why some nodes are sending inv packets with block hashes 2000 block old...
 460 2012-06-02 15:50:06 <[7]> is there a way to make bitcoin-qt (or that JS gui) show which address some coins got generated to?
 461 2012-06-02 15:50:16 <[7]> I'd like to be able to distinguish e.g. p2pool from eligius
 462 2012-06-02 15:53:40 Turingi has joined
 463 2012-06-02 16:00:24 rdponticelli_ has joined
 464 2012-06-02 16:00:44 rdponticelli has quit (Ping timeout: 245 seconds)
 465 2012-06-02 16:01:17 danbri_ has joined
 466 2012-06-02 16:01:20 danbri has quit (Read error: Connection reset by peer)
 467 2012-06-02 16:07:42 ThomasV has quit (Ping timeout: 245 seconds)
 468 2012-06-02 16:07:49 sneak has quit (Ping timeout: 248 seconds)
 469 2012-06-02 16:13:31 <slush> as an example, 129.241.137.181 is sending me "inv" for all blocks since #179439
 470 2012-06-02 16:15:33 sneak has joined
 471 2012-06-02 16:15:34 sneak has quit (Changing host)
 472 2012-06-02 16:15:34 sneak has joined
 473 2012-06-02 16:15:54 Zarutian has quit (Quit: Zarutian)
 474 2012-06-02 16:31:38 <slush> gmaxwell: "reporting old chains" = starting height in version packet?
 475 2012-06-02 16:32:11 <gmaxwell> slush: right. But I'm not actively doing this, I went and tried it against a bunch a few weeks ago.
 476 2012-06-02 16:33:31 _Fireball has quit (Read error: No route to host)
 477 2012-06-02 16:33:54 _Fireball has joined
 478 2012-06-02 16:34:34 danbri_ has quit (Remote host closed the connection)
 479 2012-06-02 16:36:02 <slush> gmaxwell: it looks like some other nodes are doing this as well. I'm reporting "0" in my starting height, because my tool doesn't know current blockchain
 480 2012-06-02 16:36:08 <BlueMatt> anyone else notice that Alan Reiner's email "Full Clients in the future - Blockchain management" is essentially just describing how the satoshi client does it?
 481 2012-06-02 16:38:35 <Eliel> well, it's pretty obvious way to do it :)
 482 2012-06-02 16:41:41 danbri has joined
 483 2012-06-02 16:41:58 <BlueMatt> true...
 484 2012-06-02 16:42:33 silpee has joined
 485 2012-06-02 16:44:13 silp has quit (Ping timeout: 240 seconds)
 486 2012-06-02 16:44:59 <sipa> he seems to assume you want to have pointers to all transactions in ram
 487 2012-06-02 16:45:11 RainbowDashh has quit (Quit: RainbowDashh)
 488 2012-06-02 16:45:22 <sipa> etotheipi_: right?
 489 2012-06-02 16:46:23 <etotheipi_> okay... so how does the satoshi client do it?
 490 2012-06-02 16:46:38 <etotheipi_> when the satoshi client is presented with a tx hash
 491 2012-06-02 16:46:43 <etotheipi_> how does it find the tx?
 492 2012-06-02 16:47:38 <etotheipi_> how does it even know if it has the tx?
 493 2012-06-02 16:48:41 <BlueMatt> txindex.dat is the bdb db that stores CTxIndexes
 494 2012-06-02 16:48:50 RainbowDashh has joined
 495 2012-06-02 16:48:56 <BlueMatt> a CTxIndex is mostly info on where the tx was spent and where you can find it on disk
 496 2012-06-02 16:49:48 RainbowDashh has quit (Disconnected by services)
 497 2012-06-02 16:49:48 Rabbit67890 has joined
 498 2012-06-02 16:50:32 <etotheipi_> you mean blkindex.dat?
 499 2012-06-02 16:50:39 <BlueMatt> oh, yea sorry
 500 2012-06-02 16:51:02 danbri has quit (Remote host closed the connection)
 501 2012-06-02 16:51:09 <etotheipi_> okay, well it's mildly concerning to me that blkindex.dat is 1/3 the size of the blockchain itself
 502 2012-06-02 16:51:15 <etotheipi_> though I presume it has more info than just that
 503 2012-06-02 16:51:26 eps has quit (Ping timeout: 248 seconds)
 504 2012-06-02 16:51:29 <BlueMatt> its probably around 95% just that
 505 2012-06-02 16:51:42 <BlueMatt> but its bdb, there is a lot of free space in the file
 506 2012-06-02 16:51:59 <BlueMatt> also note that pruning it removes >1/2 of its size https://github.com/bitcoin/bitcoin/pull/1405
 507 2012-06-02 16:52:01 eps has joined
 508 2012-06-02 16:53:03 <etotheipi_> so what is actually held in RAM?  when the blockchain has 1e9 transactions in it...
 509 2012-06-02 16:53:17 RainbowDashh has joined
 510 2012-06-02 16:53:39 <BlueMatt> we only hold CBlockIndexes in ram, ie pointers to the block on disk and the block headers
 511 2012-06-02 16:54:29 RainbowDashh has quit (Disconnected by services)
 512 2012-06-02 16:54:29 Rabbit67890_ has joined
 513 2012-06-02 16:55:06 Rabbit67890 has quit (Disconnected by services)
 514 2012-06-02 16:55:06 Rabbit67890_ is now known as Rabbit67890
 515 2012-06-02 16:55:14 Rabbit67890 is now known as RainbowDashh
 516 2012-06-02 16:55:31 <etotheipi_> oh, so BDB is handling that nastiness for you
 517 2012-06-02 16:55:37 <BlueMatt> yep
 518 2012-06-02 16:55:44 <etotheipi_> (nastiness, as in maintaining the txindex on disk without consuming ram)
 519 2012-06-02 16:55:49 <BlueMatt> yep
 520 2012-06-02 16:56:21 <gmaxwell> if not for all the complicated things like that we'd surely not use it. :)
 521 2012-06-02 16:56:30 * Eliel wonders how SQLite would compare to BDB in this.
 522 2012-06-02 16:56:47 <Eliel> even more complex :D
 523 2012-06-02 16:56:58 <BlueMatt> probably fairly similar, though more dealing with sql instead of neat C++ apis
 524 2012-06-02 16:57:21 <BlueMatt> but...that would all be in CDB anyway, so probably almost identical in how the code works
 525 2012-06-02 16:58:36 <gmaxwell> Eliel: all this needs is a key=value store (a hash table). Arguably bdb is itself huge overkill, except for a number of details (like etotheipi_'s concern) that make it not overkill.
 526 2012-06-02 16:59:29 RainbowDashh has quit (Quit: RainbowDashh)
 527 2012-06-02 16:59:36 <Eliel> I see
 528 2012-06-02 17:00:16 <etotheipi_> well it does seem to be remarkably space inefficient
 529 2012-06-02 17:00:35 <etotheipi_> but otherwise handles this the way we want it to
 530 2012-06-02 17:01:15 <Eliel> what would be the drawbacks of using CDB instead of BDB?
 531 2012-06-02 17:01:26 <BlueMatt> CDB is just a wrapper around BDB
 532 2012-06-02 17:01:32 <gmaxwell> etotheipi_: well, it's carrying indexes which more than double the size of it.
 533 2012-06-02 17:01:55 <BlueMatt> It uses more space to get more efficiency in looking up and inserting records, and since block download is largely limited by the speed of txindex.dat, Id say its well worth any faster db we can find
 534 2012-06-02 17:03:09 <Eliel> BlueMatt: libcdb1 package doesn't depend on anything but libc
 535 2012-06-02 17:03:24 <BlueMatt> we arent using libcdb, we are using our own CDB class
 536 2012-06-02 17:03:36 <Eliel> BlueMatt: I meant to ask about libcdb :)
 537 2012-06-02 17:03:58 <Eliel> not the bitcoin CDB class
 538 2012-06-02 17:04:11 <Eliel> http://cr.yp.to/cdb.html
 539 2012-06-02 17:04:20 <BlueMatt> yea, there are thousands of dbs out there we could have chose, bdb is well known, well-tested and probably the most-used db in the world, so...satoshi picked that
 540 2012-06-02 17:04:47 <BlueMatt> s/chose/chosen/
 541 2012-06-02 17:07:11 <BlueMatt> Eliel: that said, I know jgarzik has been looking at replacements for bdb, mostly as a curiosity, I dont think its actually gonna happen just because there would need to be huge advantages to move everyone using bitcoind to create a whole new db, but...
 542 2012-06-02 17:08:50 <Eliel> BlueMatt: I know, but it's relevant for etotheipi_ :)
 543 2012-06-02 17:09:00 <Eliel> armory doesn't have to do that
 544 2012-06-02 17:09:36 <dw> bdb also has nitty details like multple writers worked out a long time ago
 545 2012-06-02 17:10:57 <Eliel> looks like cdb is a hashtable based file format.
 546 2012-06-02 17:11:10 <dw> yep, and crappy incremental update support
 547 2012-06-02 17:11:14 <sipa> etotheipi_: blkindex.dat contains for each txout which other tx spent it, where a single bit would suffice
 548 2012-06-02 17:11:33 <sipa> though it makes detecting corruption easier
 549 2012-06-02 17:11:44 <Eliel> ok... cdb has a 4GB size limit. No go.
 550 2012-06-02 17:12:53 <dw> you can't really incrementally update a cdb file without wasting tonnes of space, and risking a damaged file on power failure
 551 2012-06-02 17:13:05 <dw> so its a crappy format for anything that requires frequent mutation
 552 2012-06-02 17:13:20 <sipa> jeff was looking at kyotocabinet
 553 2012-06-02 17:13:56 <dw> kyoto's documentation sucks, its api doesn't appear to be thread safe, and the author only seems to care about performance :)
 554 2012-06-02 17:14:14 <dw> it has some GetLastError() type call that is process global
 555 2012-06-02 17:14:22 <BlueMatt> luckily for us, working around all of those are easy
 556 2012-06-02 17:14:39 <BlueMatt> though better multi-threaded perf is nice for us
 557 2012-06-02 17:15:18 rdponticelli_ has quit (Ping timeout: 245 seconds)
 558 2012-06-02 17:18:06 RainbowDashh has joined
 559 2012-06-02 17:21:35 <etotheipi_> well I admit, I have zero experience with database engines
 560 2012-06-02 17:21:37 p0s has quit (Remote host closed the connection)
 561 2012-06-02 17:22:11 <etotheipi_> in hindsight, it's kind of a silly question... of course bitcoin is not the first application requiring such a solution
 562 2012-06-02 17:22:20 <sipa> haha
 563 2012-06-02 17:22:54 <jrmithdobbs> etotheipi_: ;p
 564 2012-06-02 17:23:08 <sipa> by the way, by blockchain compression, do you mean pruning?
 565 2012-06-02 17:23:09 <jrmithdobbs> ps: see, i'm not always a complete dick
 566 2012-06-02 17:24:08 <jrmithdobbs> dw: you can update cdb without risk on power failure if you use it how it was designed to be used
 567 2012-06-02 17:24:27 <jrmithdobbs> dw: but yes it is hugely wasteful of disk and mem i/o to do so atomically
 568 2012-06-02 17:24:50 Zarutian has joined
 569 2012-06-02 17:25:07 <etotheipi_> sipa, I mean unspent-tx-out-list-trees
 570 2012-06-02 17:25:18 <dw> jrmithdobbs: yep, that's not update tho, thats more like rewrite :P
 571 2012-06-02 17:25:32 <sipa> etotheipi_: how does that reduce any storage?
 572 2012-06-02 17:25:35 <dw> also relies on filesystem to ensure atomic rename
 573 2012-06-02 17:25:57 <jrmithdobbs> dw: well what do you expect from something designed to be able to do lookups of account existence and ip blacklists ;p
 574 2012-06-02 17:26:03 <jrmithdobbs> dw: works pretty well for those things
 575 2012-06-02 17:26:38 <sipa> etotheipi_: or you just mean: how to distribute just the essential information to verifying nodes that do not want to be a full node?
 576 2012-06-02 17:26:40 <etotheipi_> if you have 10 million users sending 1 simple tx per day, your storage reqts increases by 10million tx per day... but with unspent txout-trees it should be constant
 577 2012-06-02 17:27:26 <etotheipi_> of course it's never that simple... but there's a lot of space savings to be had, but I haven't worked out the compute complexity of maintaining such a tree yet
 578 2012-06-02 17:27:32 <Eliel> etotheipi_: you're thinking more about the RAM footprint than disc foorprint here, right?
 579 2012-06-02 17:27:38 <etotheipi_> no, disk foot print
 580 2012-06-02 17:28:23 <etotheipi_> so that almost-full-nodes can store the essentials of the whole chain without storing the whole chain
 581 2012-06-02 17:28:25 <sipa> etotheipi_: patricia tree on the tx hash, that maps to a bitvector?
 582 2012-06-02 17:28:50 <gmaxwell> he's talking about open transaction trees, the flip chain stuff.
 583 2012-06-02 17:29:00 <sipa> yes, i'm aware now
 584 2012-06-02 17:29:02 Joric has joined
 585 2012-06-02 17:29:45 <etotheipi_> unfortunately, now is not a good time for me to discuss it, but the tree-structure idea is here:  https://bitcointalk.org/index.php?topic=52859.msg885838#msg885838
 586 2012-06-02 17:30:13 <etotheipi_> which would give lightweight nodes the ability to retrieve&verify their unspent-output-lists with only a few kB/addr
 587 2012-06-02 17:30:39 <etotheipi_> and then another post brought up the idea that the hashes of those trees could be maintained in a separate blockchain with merged mining
 588 2012-06-02 17:30:47 <etotheipi_> which actually makes this possible, as opposed to a pipe dream
 589 2012-06-02 17:31:05 <gmaxwell> I'm not fond of address ordering the tree because I don't see how to keep it balanced.
 590 2012-06-02 17:31:41 <sipa> also, for a normal usecase, a wallet has no keys in it from the past when it is generated
 591 2012-06-02 17:31:57 <sipa> sure it's extremely useful for rescanning
 592 2012-06-02 17:32:10 <gmaxwell> Though my fondness of that would increase once I was convinced it was possible to avoid an attacker creating linear update cost.
 593 2012-06-02 17:32:15 <sipa> but i think we should rather find ways to avoid rescanning altogether
 594 2012-06-02 17:32:28 <etotheipi_> gmaxwell: I don't see where there is a balance issue
 595 2012-06-02 17:32:49 <etotheipi_> sipa: well if such a tree is maintained... you can rescan your whole wallet extremely quickly
 596 2012-06-02 17:32:55 TD has joined
 597 2012-06-02 17:33:06 <sipa> etotheipi_: exactly what i said :)
 598 2012-06-02 17:33:07 <etotheipi_> as I said, a few kB per address... and it's verifiable against the headers
 599 2012-06-02 17:34:11 <sipa> i don't disagree that it is perfect for doing fast rescans, but i don't think we should be aiming for technologies that require rescanning at all (except maybe in recover-from-disaster scenarios)
 600 2012-06-02 17:34:31 D34TH has joined
 601 2012-06-02 17:34:46 <etotheipi_> sipa: I don't understand... the trees can be updated incrementally
 602 2012-06-02 17:34:57 <etotheipi_> someone, somewhere, at some time has to do a full scan to create the tree
 603 2012-06-02 17:35:07 <etotheipi_> well, multiple people...
 604 2012-06-02 17:35:39 <gmaxwell> I'm much more interested in things that enable cheap block validation. But as you point out, your address tree thing could actually just be mergemined.
 605 2012-06-02 17:35:40 Z0rZ0rZ0r has joined
 606 2012-06-02 17:36:25 <sipa> only the root of the address tree, i suppose?
 607 2012-06-02 17:36:35 <sipa> merkle root
 608 2012-06-02 17:36:36 <etotheipi_> then you acquire the tree once... and incrementally update it with new blocks/txs
 609 2012-06-02 17:36:47 <sipa> yes, of course
 610 2012-06-02 17:36:57 <etotheipi_> right, the second chain, block headers contain the merkle root of the tree
 611 2012-06-02 17:37:10 <sipa> much in the same way as an open-tx-set could be maintained
 612 2012-06-02 17:37:18 <etotheipi_> and someone can download from peers the latest "snapshot"
 613 2012-06-02 17:37:25 <etotheipi_> and verify it against the headers
 614 2012-06-02 17:37:34 <sipa> yes, yes, i understand :)
 615 2012-06-02 17:37:51 <etotheipi_> okay... as usual, I'm slow at figuring out where the discussion is diverging...
 616 2012-06-02 17:37:54 <sipa> no need to convince me what it does or why it is useful
 617 2012-06-02 17:38:20 <sipa> i'm just arguing that there may be a viable future where it isn't needed at all
 618 2012-06-02 17:38:32 <Eliel> this sounds similar somehow to where I heard realsolid is heading with his next solidcoin (or microcash as he's renaming it)
 619 2012-06-02 17:39:06 <etotheipi_> well honestly, I don't think it's worth it just for the compression aspect
 620 2012-06-02 17:39:17 <gmaxwell> sure, where do you think he got the ideas for it? (not etotheipi_'s comments as much as bytecoins older discussion)
 621 2012-06-02 17:39:28 <etotheipi_> but for lightweight nodes to be able to quickly verify the balance of any address...
 622 2012-06-02 17:39:41 <Eliel> gmaxwell: ah, of course :)
 623 2012-06-02 17:40:32 <etotheipi_> for instance, light nodes can verify that a particular incoming zero-conf transaction is actually valid and unspent
 624 2012-06-02 17:41:29 <gmaxwell> etotheipi_: you need to be indexed by txns not by 'address' for that.
 625 2012-06-02 17:42:10 <gmaxwell> Or at least that what I'd been thinking— Because there can be multiple addresses to satisify an output, but perhaps not.
 626 2012-06-02 17:43:10 <Eliel> etotheipi_: how would the tree-update algorithm react to an attacker that kept doing transactions that try to overload one small area of the hash space?
 627 2012-06-02 17:43:51 <gmaxwell> Eliel: thats my comment about balancing.
 628 2012-06-02 17:44:29 <gmaxwell> amiller was commenting some on some magic for this, but I haven't had a chance to read all the papers he directed me to you.
 629 2012-06-02 17:44:36 <gmaxwell> gah. s/you/yet/
 630 2012-06-02 17:45:20 <Eliel> although, the maximum depth would still be limited since the processing power needed for increasing it would double for each increment of one.
 631 2012-06-02 17:45:58 ThomasV has joined
 632 2012-06-02 17:46:45 <etotheipi_> gmaxwell: excellent point... I was still assuming that the treenodes were accessible by OutPoint, but that's not the key anymore in my scheme
 633 2012-06-02 17:47:28 <etotheipi_> though, it wouldn't be hard to maintain a second tree indexed by OutPoint that has iterators to the address-indexed tree
 634 2012-06-02 17:47:35 <etotheipi_> but that is getting contrived
 635 2012-06-02 17:47:56 <sipa> well, we need the outpoint trew no matter what
 636 2012-06-02 17:48:30 <sipa> tree
 637 2012-06-02 17:49:27 <sipa> and a bitvector of spentness per tx is probably smaller than your iterator
 638 2012-06-02 17:49:28 <etotheipi_> speaking of patricia trees... anyone familiar with a "de la brandia tree"?
 639 2012-06-02 17:49:33 <gmaxwell> etotheipi_: right, I think thats the more interesting thing— allowing for light fullnodes is important to me— keeps us decentralized.  Though I also think your fast-secure-address indexes are somewhat interesting too.
 640 2012-06-02 17:49:59 <gmaxwell> Never heard of that.
 641 2012-06-02 17:50:06 <sipa> neither have i
 642 2012-06-02 17:50:17 <etotheipi_> I learned about de la brandia trees in my data-structures class along with patricia trees
 643 2012-06-02 17:50:24 <etotheipi_> but now I see no evidence they actually exist
 644 2012-06-02 17:50:32 <etotheipi_> or rather, maybe there were called something else
 645 2012-06-02 17:50:59 <etotheipi_> they were essentially patricia tress but using linked lists instead of pointer arrays
 646 2012-06-02 17:51:09 <etotheipi_> reducing the memory cost of each node
 647 2012-06-02 17:51:10 ThomasV has quit (Ping timeout: 260 seconds)
 648 2012-06-02 17:52:29 ovidiusoft has quit (Read error: Operation timed out)
 649 2012-06-02 17:52:33 ovidiusoft has joined
 650 2012-06-02 17:53:31 <etotheipi_> well back to the original point:  yes, I like the way we can maintain some extra degree of decentralization in this manner
 651 2012-06-02 17:54:02 <etotheipi_> but there's so many details I can't even encapsulate what it would take and what the potential problems are with this kind of implementation
 652 2012-06-02 17:54:21 <etotheipi_> I understand the concept of merged mining, but don't know what the risks are...
 653 2012-06-02 17:55:44 Z0rZ0rZ0r has quit (Remote host closed the connection)
 654 2012-06-02 17:55:53 <Nesetalis> ... something is weird... i've had a transaction pending since 11:40pm last night... (its 2pm today) still 0 confirmations.
 655 2012-06-02 17:56:12 Z0rZ0rZ0r has joined
 656 2012-06-02 17:56:42 <TD> hmm
 657 2012-06-02 17:56:50 <TD> is there any actually complete implementation of lightweight nodes, yet?
 658 2012-06-02 17:56:58 <TD> i mean of the type that connects directly to the p2p network
 659 2012-06-02 17:57:02 <TD> bitcoinj isn't .....
 660 2012-06-02 17:58:38 <etotheipi_> well there will have to be...
 661 2012-06-02 17:58:48 <Nesetalis> oh and NOW it hits the network, as soon as I mention it in here
 662 2012-06-02 17:58:58 <etotheipi_> in fact, I'd like to make Armory lightweight in standard-user mode
 663 2012-06-02 17:59:03 <etotheipi_> though I"m a ways off from that goal
 664 2012-06-02 18:02:16 <sipa> TD: what do you mean by a lightweight node that connects to the P2P network?
 665 2012-06-02 18:02:21 <TD> spv
 666 2012-06-02 18:03:00 slush1 has joined
 667 2012-06-02 18:03:07 slush has quit (Read error: Operation timed out)
 668 2012-06-02 18:05:06 <sipa> ah, ok; i interpreted lightweight as something weaker, more like electrum
 669 2012-06-02 18:06:08 banshee12 has joined
 670 2012-06-02 18:06:55 rdponticelli has joined
 671 2012-06-02 18:08:55 <BlueMatt> TD: Im pretty sure there isnt
 672 2012-06-02 18:09:05 <TD> well
 673 2012-06-02 18:09:12 <BlueMatt> TD: but thats like #2 on my list todo in the next ~month
 674 2012-06-02 18:09:24 <TD> to me there's little difference between something like electrum and SPV+a protocol extension to allow server side filtering of blocks
 675 2012-06-02 18:09:57 <TD> once you have that, there's little benefit to putting all the logic on the server beyond "clients are simple", but you can't avoid dealing with the block chain
 676 2012-06-02 18:10:07 <TD> you can only move the complexity around. so it may as well be on the client, at that point
 677 2012-06-02 18:10:10 <BlueMatt> the way I always saw spv is you download full blocks, just dont store them
 678 2012-06-02 18:10:24 <TD> well that's how it works now
 679 2012-06-02 18:10:35 <TD> but in future it should be "you download blocks plus transactions that match a filter"
 680 2012-06-02 18:12:01 <BlueMatt> +optionally download full blocks, Imho thats an important options for those who want to be able to run trust-less-ish
 681 2012-06-02 18:12:03 da2ce7 has quit (Read error: Connection reset by peer)
 682 2012-06-02 18:13:20 da2ce7 has joined
 683 2012-06-02 18:14:54 <BlueMatt> but, yea, not downloading full blocks on mobile clients would be very important
 684 2012-06-02 18:20:49 slush1 has quit (Read error: Connection reset by peer)
 685 2012-06-02 18:27:09 slush has joined
 686 2012-06-02 18:31:19 <Eliel> gmaxwell: I don't think the balance problem in the address indexed tree is a big issue. As long as the addresses are all the output of a cryptographic hash-algorithm, it'll take very high amounts of processing power to even be able to make it noticeable.
 687 2012-06-02 18:31:48 <Eliel> it would be much more effective and economical to just spam the network with transactions.
 688 2012-06-02 18:33:21 <luke-jr> sipa: is it possible to construct a signature with a specific fixed prefix/middle/suffix?
 689 2012-06-02 18:37:06 <sipa> luke-jr: i doubt that
 690 2012-06-02 18:37:28 <luke-jr> hmm
 691 2012-06-02 18:37:40 <luke-jr> any way to make signatures cheaper?
 692 2012-06-02 18:38:14 <sipa> 64 bytes suffices, instead of the DER encoding bitcoin uses...
 693 2012-06-02 18:39:34 <luke-jr> coinbase… 100 - 5 for height - 36 for MM - up to 8 for extranonce = 51 left
 694 2012-06-02 18:39:53 rdponticelli_ has joined
 695 2012-06-02 18:40:00 <luke-jr> hmm
 696 2012-06-02 18:40:17 <luke-jr> I suppose I could stuff it in the pool output <.,
 697 2012-06-02 18:40:18 <luke-jr> <.<*
 698 2012-06-02 18:40:19 rdponticelli has quit (Ping timeout: 245 seconds)
 699 2012-06-02 18:43:11 <sipa> you want to put a signature in the block chain?
 700 2012-06-02 18:43:23 <sipa> in the coinbase, i mean
 701 2012-06-02 18:43:32 <luke-jr> that'd be useful, yes
 702 2012-06-02 18:43:46 <sipa> for? identifying pools?
 703 2012-06-02 18:49:37 slush has quit (Read error: Connection reset by peer)
 704 2012-06-02 18:51:08 <TD> BlueMatt: even with filtering the trust level is still pretty low. a remote peer can hide transactions from you, but that's about it. if you select a bunch of peers, hopefully you can spot the discrepancy. i think over time the use of things like Tor will become routine, exactly to make such attacks difficult - if you don't know who is connecting to you, you can't easily hide transactions
 705 2012-06-02 18:51:14 <TD> (without being trivially caught)
 706 2012-06-02 18:51:54 <BlueMatt> TD: absolutely, but, in the end, it is still trust
 707 2012-06-02 18:52:09 <BlueMatt> TD: and I know there are quite a few people who like the no-trust aspect of bitcoin
 708 2012-06-02 18:52:15 <TD> sure
 709 2012-06-02 18:52:19 <TD> it's just a bandwidth tradeoff
 710 2012-06-02 18:52:22 <BlueMatt> yea
 711 2012-06-02 18:52:41 <TD> it'd be nice to find some way of allowing you to do server-side filtering without the remote node being able to lie to you
 712 2012-06-02 18:53:00 <TD> i'm thinking if you use loose enough filters and random spot checks of blocks, or something ..... not sure
 713 2012-06-02 18:53:47 <BlueMatt> yea...that plus connect to 5 unrelated servers and Id think your % chance of missing a tx would be low
 714 2012-06-02 18:53:55 O2made has joined
 715 2012-06-02 18:54:00 <gmaxwell> TD: the committed tree setups allow provable naks.
 716 2012-06-02 18:54:03 <BlueMatt> and a "do I have txes?" "no" exchange is very low bandwidth
 717 2012-06-02 18:54:25 <gmaxwell> Which make it impossible to tell an undetectable lie.
 718 2012-06-02 18:55:06 <TD> committed tree setups? this is where you include a  merkle tree of unspent outputs in the coinbase or something else?
 719 2012-06-02 18:55:28 <sipa> that
 720 2012-06-02 18:57:01 <gmaxwell> Right. The server gives you the headers and the connecting branches.  So the only way they can lie is by mining a doomed fork, and even that is detectable if you know the current time, and some of the correct chain from the not too distant past, unless they have an awful lot of hash power.
 721 2012-06-02 18:59:23 <BlueMatt> sending a full merkle tree of unspent outputs could get expensive, though you could enforce ordering the leaves and send the branches surrounding your addresses...still could get bw expensive either way
 722 2012-06-02 18:59:55 <gmaxwell> BlueMatt: you've missed etotheipi's proposal.
 723 2012-06-02 19:00:05 <gmaxwell> BlueMatt: which is basically to use the output as the tree key.
 724 2012-06-02 19:00:12 slush has joined
 725 2012-06-02 19:01:00 <gmaxwell> so I can just give you a list of keys I'm interested in updates on. (plus information about the trees I currently have for them) and you can send me the difference and the connecting branch to show that it's POW-confirmed.
 726 2012-06-02 19:02:32 <BlueMatt> ah...yea ok
 727 2012-06-02 19:02:38 RazielZ has quit (Ping timeout: 248 seconds)
 728 2012-06-02 19:03:07 <BlueMatt> still, compared to a raw list of txes or "no" its a significant order of magnitude greater bw (but Id think even mobile clients could handle that)
 729 2012-06-02 19:03:32 toffoo has joined
 730 2012-06-02 19:04:37 phyburn has quit (Remote host closed the connection)
 731 2012-06-02 19:06:05 <gmaxwell> well, you can still combine it with things like spotchecks. e.g. "Do I have new txns?, give me a signed response." "No." "Okay, prove it." "Ummm errr You see."  "Hey everybody, node XYZ lied to me and here is the proof"    making the spotchecks stronger has value.
 732 2012-06-02 19:06:33 <gmaxwell> Withthout the tree proof you can only spotcheck by checking several nodes and they could be sybils.
 733 2012-06-02 19:06:47 <gmaxwell> Plus you then have to load up several nodes doing the check.
 734 2012-06-02 19:07:19 <BlueMatt> yea, no that would work pretty well...
 735 2012-06-02 19:09:27 MobiusL has quit (Ping timeout: 276 seconds)
 736 2012-06-02 19:09:54 rdponticelli_ has quit (Ping timeout: 245 seconds)
 737 2012-06-02 19:15:31 slush has quit (Ping timeout: 250 seconds)
 738 2012-06-02 19:34:06 RazielZ has joined
 739 2012-06-02 19:42:43 mcorlett is now known as emkolretuui
 740 2012-06-02 19:42:54 emkolretuui is now known as mcorlett
 741 2012-06-02 19:57:13 Z0rZ0rZ0r has quit (Ping timeout: 240 seconds)
 742 2012-06-02 19:57:37 rdponticelli has joined
 743 2012-06-02 20:04:19 eoss has quit (Remote host closed the connection)
 744 2012-06-02 20:09:02 Z0rZ0rZ0r has joined
 745 2012-06-02 20:09:37 slush has joined
 746 2012-06-02 20:11:42 spiborg has joined
 747 2012-06-02 20:13:47 ThomasV has joined
 748 2012-06-02 20:19:50 osmosis has joined
 749 2012-06-02 20:20:23 Xunie has joined
 750 2012-06-02 20:20:29 rdponticelli_ has joined
 751 2012-06-02 20:21:09 rdponticelli has quit (Ping timeout: 245 seconds)
 752 2012-06-02 20:21:30 ovidiusoft has quit (Ping timeout: 252 seconds)
 753 2012-06-02 20:21:35 ovidiusoft has joined
 754 2012-06-02 20:23:07 Cory has quit (Ping timeout: 265 seconds)
 755 2012-06-02 20:25:12 slush has quit (Read error: Connection reset by peer)
 756 2012-06-02 20:26:30 slush has joined
 757 2012-06-02 20:33:07 slush has quit (Ping timeout: 252 seconds)
 758 2012-06-02 20:34:42 minimoose has quit (Quit: minimoose)
 759 2012-06-02 20:34:55 slush has joined
 760 2012-06-02 20:35:11 brwyatt is now known as Away!~brwyatt@pool-96-226-236-130.dllstx.fios.verizon.net|brwyatt
 761 2012-06-02 20:35:46 ovidiusoft has quit (Quit: leaving)
 762 2012-06-02 20:35:47 RainbowDashh has quit (Quit: RainbowDashh)
 763 2012-06-02 20:38:45 slush has quit (Read error: Connection reset by peer)
 764 2012-06-02 20:38:50 rdponticelli_ is now known as rdponticelli
 765 2012-06-02 20:39:08 RainbowDashh has joined
 766 2012-06-02 20:39:22 RainbowDashh has quit (Client Quit)
 767 2012-06-02 20:43:37 ovidiusoft has joined
 768 2012-06-02 20:44:04 ovidiusoft has quit (Client Quit)
 769 2012-06-02 20:44:12 ovidiusoft has joined
 770 2012-06-02 20:46:24 danbri has joined
 771 2012-06-02 20:50:49 danbri has quit (Remote host closed the connection)
 772 2012-06-02 20:54:43 slush has joined
 773 2012-06-02 20:57:47 Cory has joined
 774 2012-06-02 20:57:49 phma has quit (Remote host closed the connection)
 775 2012-06-02 20:57:57 danbri has joined
 776 2012-06-02 21:02:16 pooolop has quit (Ping timeout: 245 seconds)
 777 2012-06-02 21:02:24 rdponticelli has quit (Ping timeout: 245 seconds)
 778 2012-06-02 21:02:56 RainbowDashh has joined
 779 2012-06-02 21:05:38 minimoose has joined
 780 2012-06-02 21:06:27 rdponticelli has joined
 781 2012-06-02 21:09:30 MobiusL has joined
 782 2012-06-02 21:11:36 danbri has quit (Remote host closed the connection)
 783 2012-06-02 21:11:38 slush has quit (Read error: Connection reset by peer)
 784 2012-06-02 21:14:07 _W_ has quit (Excess Flood)
 785 2012-06-02 21:14:28 _W_ has joined
 786 2012-06-02 21:16:21 danbri has joined
 787 2012-06-02 21:17:32 erle- has quit (Quit: erle-)
 788 2012-06-02 21:17:45 <jrmithdobbs> sipa: so how do you specify which address to advertise when using ipv6 on a dualt stack host and you want both types of connections?
 789 2012-06-02 21:18:22 <jrmithdobbs> because i have a several ip6 assignments on the host, most of which really shouldn't accept bitcoin connections
 790 2012-06-02 21:20:56 <sipa> jrmithdobbs: you select where to bind with -bind, you specify the address to advertize with using -externalip
 791 2012-06-02 21:21:00 <jrmithdobbs> sipa: can you specify multiple -bind= and -externalip= ?
 792 2012-06-02 21:21:02 <sipa> both can be specified multiple times
 793 2012-06-02 21:21:12 <jrmithdobbs> ok cool
 794 2012-06-02 21:21:31 <sipa> and externalip works +- smart, so it will favor giving an ipv6 address to ipv6 peers
 795 2012-06-02 21:21:51 <jrmithdobbs> so -bind=0.0.0.0 -bind=[::] -externalip=ip4pub -externalip=ip6pub should work?
 796 2012-06-02 21:22:00 <sipa> yup
 797 2012-06-02 21:22:13 <sipa> though those two binds are the default anyway
 798 2012-06-02 21:22:19 <sipa> so no need to specify them
 799 2012-06-02 21:22:20 <jrmithdobbs> right
 800 2012-06-02 21:22:32 <jrmithdobbs> just an example
 801 2012-06-02 21:23:18 <sipa> there's a few more pull requests open that change some things though
 802 2012-06-02 21:23:30 <sipa> i don't know exactly anymore what's already merged
 803 2012-06-02 21:23:38 <jrmithdobbs> i already filter the connections on other ips at network level but would rather not pollute people's addr.dat if i am
 804 2012-06-02 21:25:39 <jrmithdobbs> didn't several things get merged a while back to fix building on *bsd?
 805 2012-06-02 21:25:58 <sipa> yes
 806 2012-06-02 21:26:22 <jrmithdobbs> anyone actively testing?
 807 2012-06-02 21:27:01 <jrmithdobbs> this is host is about to get migrated so i guess i'll find out
 808 2012-06-02 21:31:36 <sipa> there's a few bsd users
 809 2012-06-02 21:32:49 agricocb has quit (Ping timeout: 245 seconds)
 810 2012-06-02 21:34:01 graingert has quit (Remote host closed the connection)
 811 2012-06-02 21:34:26 Z0rZ0rZ0r has quit (Quit: Leaving)
 812 2012-06-02 21:36:25 <etotheipi_> sipa: what is the status of the new wallet format?
 813 2012-06-02 21:36:45 <etotheipi_> I'm not in any rush, I'm just curious
 814 2012-06-02 21:38:03 dinox has joined
 815 2012-06-02 21:38:47 <sipa> etotheipi_: you mean the deterministic wallets, or the append-only format?
 816 2012-06-02 21:41:16 <etotheipi_> sorry, I meant deterministic wallets
 817 2012-06-02 21:41:44 <etotheipi_> I didn't know you were working on an append-only format
 818 2012-06-02 21:41:44 <sipa> regarding the seeding idea
 819 2012-06-02 21:42:05 <sipa> ah
 820 2012-06-02 21:42:32 <sipa> it's an append-only key=value store really, written in blocks to the file, each with a cryptographic checksum
 821 2012-06-02 21:43:00 <sipa> so it can detect corrupted only half-written blocks and ignore them
 822 2012-06-02 21:43:05 <etotheipi_> ahh
 823 2012-06-02 21:43:13 <sipa> which should always result in a consistent state, with just a single file
 824 2012-06-02 21:43:33 <etotheipi_> ahh, that's a different way to do it
 825 2012-06-02 21:43:48 <etotheipi_> I just detect corruption and restore... you decided to write it in a way that it can just be skipped
 826 2012-06-02 21:44:24 <sipa> but it will need rewrite-close-atomicmove cycles anyway, as for example last known best block changes too frequently to me negligable
 827 2012-06-02 21:44:29 <etotheipi_> so how do you know that a partially-written block will have enough info in it to be meaningful?
 828 2012-06-02 21:44:38 <etotheipi_> oh... constant block size?
 829 2012-06-02 21:44:41 <sipa> no
 830 2012-06-02 21:44:52 <sipa> if the checksum of the block doesn't compute, ignore the entire block
 831 2012-06-02 21:45:11 <sipa> that means you lost durability, but the former block represented a former consistent state
 832 2012-06-02 21:45:14 <etotheipi_> but how do you know that the blocksize field didn't get corrupted somehow?
 833 2012-06-02 21:45:52 <sipa> if the blocksize field got corrupted to a too small value, you'll notice when trying to decode or certainly when checking the checksum
 834 2012-06-02 21:46:08 <sipa> if the blocksize field is too large, it'll take you past the end of the file
 835 2012-06-02 21:46:33 <sipa> which is also easily detectable
 836 2012-06-02 21:46:39 <etotheipi_> that's the one thing that always concerned me... if that value gets corrupted, you don't even know where the next block starts
 837 2012-06-02 21:47:12 <sipa> well, there are 4 marker bytes before each block, so you can search for those
 838 2012-06-02 21:47:32 <etotheipi_> gotch
 839 2012-06-02 21:47:34 <etotheipi_> a
 840 2012-06-02 21:47:46 danbri has quit (Remote host closed the connection)
 841 2012-06-02 21:47:52 <etotheipi_> actually, I should consider something similar
 842 2012-06-02 21:48:15 <etotheipi_> I don't use marker bytes
 843 2012-06-02 21:48:36 <sipa> i'd like a way for doing in-place updates anyway, but it's probably not worth the extra complexity
 844 2012-06-02 21:48:38 <etotheipi_> ehh... I don't think it matters... if anything goes wrong with my wallet, it's restored from the backup
 845 2012-06-02 21:48:43 <sipa> indeed
 846 2012-06-02 21:49:04 <etotheipi_> how so?  why can't you do in-place updates?
 847 2012-06-02 21:49:15 <sipa> what if they get written only partially?
 848 2012-06-02 21:49:20 <sipa> and they'll corrupt the checksums
 849 2012-06-02 21:49:28 <sipa> sure that's all fixable, but it becomes hard
 850 2012-06-02 21:50:37 <etotheipi_> I don't understand, why is overwriting different than adding?  in both cases a partial write leads to a messed up block of data
 851 2012-06-02 21:51:03 <sipa> overwriting something means at least temporarily corrupting a block that was perfectly consistent before
 852 2012-06-02 21:51:06 <sipa> i don't like that
 853 2012-06-02 21:51:49 <sipa> and adding is different because it doesn't overwrite live data
 854 2012-06-02 21:51:54 <etotheipi_> true, you risk losing stuff
 855 2012-06-02 21:51:57 <sipa> indeed
 856 2012-06-02 21:52:12 <etotheipi_> oh, that's a good point, too
 857 2012-06-02 21:52:17 <etotheipi_> I've been avoiding multi-threading
 858 2012-06-02 21:52:40 <sipa> plus, i'm sure that if i can show that the code never overwrites live data, it's much easier to gain trust that it will work
 859 2012-06-02 21:53:16 <etotheipi_> I assume you'll be at least blanking the old data?
 860 2012-06-02 21:53:22 slush has joined
 861 2012-06-02 21:53:27 <etotheipi_> in my file format, I have a code for "deleted"
 862 2012-06-02 21:53:32 <sipa> no, that would corrupt the checksums
 863 2012-06-02 21:53:40 <sipa> it's strictly append only
 864 2012-06-02 21:53:54 <etotheipi_> what about previously-decrypted key data?
 865 2012-06-02 21:54:00 <sipa> ?
 866 2012-06-02 21:54:01 RainbowDashh has quit (Quit: RainbowDashh)
 867 2012-06-02 21:54:02 <gmaxwell> make a new file.
 868 2012-06-02 21:54:18 <sipa> yes, that's why there are rewrite + move atomically cycles
 869 2012-06-02 21:54:21 <gmaxwell> which we already have to do because bdb's deletion of data isn't trustworthy.
 870 2012-06-02 21:54:32 <etotheipi_> oh, fair enough
 871 2012-06-02 21:54:49 <sipa> that compacts the old space away, and rewrites all remainind old data in a single block
 872 2012-06-02 21:55:10 danbri has joined
 873 2012-06-02 21:55:52 <sipa> it could be rewrite to .bak file, close .bak file, close real file, rewrite real file too
 874 2012-06-02 21:56:08 <sipa> so you always have a backup, like you do
 875 2012-06-02 21:56:25 danbri has quit (Remote host closed the connection)
 876 2012-06-02 21:56:35 <sipa> and there are no problems with the wallet file being a symlink
 877 2012-06-02 21:56:52 <etotheipi_> btw, stupid story
 878 2012-06-02 21:57:13 <etotheipi_> some guy emailed me to tell me that he used armory for the first time, put 107 BTC into it, then restarted and all his coins were gone
 879 2012-06-02 21:57:35 <etotheipi_> I must've spent 20 emails with him figuring out what happened to the wallet file, and how it could get corrupted so badly
 880 2012-06-02 21:57:49 <etotheipi_> I would've thought it was a scam, except he never asked for me to replace them
 881 2012-06-02 21:58:41 <sipa> and he used the wrong password?
 882 2012-06-02 21:58:49 <etotheipi_> it turns out he had never actually opened Armory -- he had opened multibit, transferred 107 BTC, then closed it and reopened Armory
 883 2012-06-02 21:58:57 <sipa> hahaha
 884 2012-06-02 22:00:06 <sipa> anyway, gmaxwell, etotheipi_: about the master key seeding
 885 2012-06-02 22:00:12 <etotheipi_> he donated 10 BTC for that
 886 2012-06-02 22:00:33 <sipa> how acceptable is using a table of 16 iteration count / compare values, for the various strengths?
 887 2012-06-02 22:01:16 <etotheipi_> you're going to have to elaborate on that ...
 888 2012-06-02 22:01:29 <sipa> https://gist.github.com/2731997
 889 2012-06-02 22:02:17 <jrmithdobbs> sipa: so uh, is there a way to force it to try and get at least one ip6 peer from seeds?
 890 2012-06-02 22:02:24 <jrmithdobbs> without running ip6 only
 891 2012-06-02 22:02:25 <sipa> jrmithdobbs: -addnode ?
 892 2012-06-02 22:03:05 <etotheipi_> sipa: I'm going to need some time to absorb this
 893 2012-06-02 22:03:17 <jrmithdobbs> sipa: so to bootstrap a dualstack host you really need to -addnode to ensure you get at least one of each?
 894 2012-06-02 22:04:01 chrisb__ has quit (Quit: Leaving)
 895 2012-06-02 22:04:07 <sipa> jrmithdobbs: i hope that after a while there will be plenty of ipv6 addresses in the dns seeds
 896 2012-06-02 22:04:41 <etotheipi_> sipa: what is A and B?
 897 2012-06-02 22:04:47 <jrmithdobbs> sipa: well, i just started it up and it doesn't look like it even tried any ip6 hosts from the log
 898 2012-06-02 22:05:15 <jrmithdobbs> sipa: normal 8 max outbound, 512 inbound, but since no v6 nodes it has broadcasted that address so your crawler wont pick it up ;p
 899 2012-06-02 22:05:23 <jrmithdobbs> err hasn't
 900 2012-06-02 22:05:35 <sipa> etotheipi_: basically, it's replacing the "check whether N bits are zero after 2^N steps" with "check whether the first 32-bit word of the hash is less than 2^32*f(N) after i(N) iterations
 901 2012-06-02 22:05:45 <jrmithdobbs> i can addnode to force bootstrap
 902 2012-06-02 22:06:19 <jrmithdobbs> but that might slow ip6 integration for a while if people need to do that and don't
 903 2012-06-02 22:06:31 <sipa> etotheipi_: and an optimization process to find optimal values for f(N) and i(N) such that you get expected runtimes of A*B^N
 904 2012-06-02 22:06:43 <sipa> etotheipi_: plus some other constraint
 905 2012-06-02 22:07:05 <sipa> jrmithdobbs: but you're not running dual stack?
 906 2012-06-02 22:07:06 <jrmithdobbs> sipa: will especially be an issue on hosts that have aaaa records deprioritized in /etc/gai.conf
 907 2012-06-02 22:07:09 <sipa> no, you are
 908 2012-06-02 22:07:20 <jrmithdobbs> ya i am
 909 2012-06-02 22:07:49 <sipa> well, there is such an abundance for ipv4 nodes that there is just not much chance for picking an ipv6 one
 910 2012-06-02 22:08:18 <etotheipi_> so what is A and B?
 911 2012-06-02 22:08:36 <sipa> whatever we decide, but i propose 65536 and 2
 912 2012-06-02 22:08:52 <jrmithdobbs> ya, maybe at least temporarily it might be worthwile to add some logic to try and hold one slot for an ip6 seed if a routable v6 address is found or explicitly -externalip'ed
 913 2012-06-02 22:09:03 <jrmithdobbs> one outbound i mean
 914 2012-06-02 22:09:10 <sipa> jrmithdobbs: probably we can add a v6-only dns seed
 915 2012-06-02 22:09:29 <jrmithdobbs> sipa: or could add a v6-only outbound connect pool and give them both 8 outbound
 916 2012-06-02 22:10:01 <jrmithdobbs> that might make sense really
 917 2012-06-02 22:10:13 ThomasV has quit (Ping timeout: 245 seconds)
 918 2012-06-02 22:10:51 <sipa> etotheipi_: advantage: you can pretty well estimate the runtime for finding a key, and they are separately better than a factor 4 apart
 919 2012-06-02 22:10:59 <sipa> s/key/seed/
 920 2012-06-02 22:11:11 <sipa> s/separately/separated/
 921 2012-06-02 22:11:14 ThomasV has joined
 922 2012-06-02 22:12:32 <jrmithdobbs> sipa: tbqh, for bitcoind, i'm half tempted to add a #define to prevent building any of the seed logic and just pre-parse your seed stats csv and -addnode
 923 2012-06-02 22:12:35 <jrmithdobbs> ha
 924 2012-06-02 22:12:58 <jrmithdobbs> everyone providing a seed should provide same-format stats information ;p
 925 2012-06-02 22:12:58 slush has quit (Read error: Connection reset by peer)
 926 2012-06-02 22:13:13 <sipa> jrmithdobbs: you know my crawler that creates that file also exposes it via my dns seed?
 927 2012-06-02 22:13:36 minimoose has quit (Quit: minimoose)
 928 2012-06-02 22:15:20 slush has joined
 929 2012-06-02 22:15:24 <jrmithdobbs> sipa: txt records?
 930 2012-06-02 22:15:26 <etotheipi_> sipa: I'm getting there... I'm just getting caught up in the equations and their meaning, with only single-letter abstract names
 931 2012-06-02 22:15:57 <sipa> etotheipi_: it wasn't really intended as a real explanation, just notes of my own calculations so i can redo them :)
 932 2012-06-02 22:16:09 <Eliel> sipa: I don't think the in-place update is necessary. You can always just append a block that "replaces" the old data.
 933 2012-06-02 22:16:10 <sipa> jrmithdobbs: eh no, A and AAAA
 934 2012-06-02 22:16:16 <jrmithdobbs> sipa: because that'd be much better than me pulling the whole thing (and sorry btw, i'll put that behind squid until i find out by what you mean, ha)
 935 2012-06-02 22:16:44 ovidiusoft has quit (Read error: Operation timed out)
 936 2012-06-02 22:17:45 <sipa> Eliel: sure, that's what happens now
 937 2012-06-02 22:18:00 <sipa> Eliel: (there's a prototype in my logdb branch)
 938 2012-06-02 22:18:07 <jrmithdobbs> sipa: should setup a reverse ip.arpa and in-addr.arpa style zone to pull the stats for specific nodes too
 939 2012-06-02 22:18:14 <Eliel> sipa: the occasional rewrite will then clean things up nicely :)
 940 2012-06-02 22:18:20 ThomasV has quit (Read error: Operation timed out)
 941 2012-06-02 22:18:24 <sipa> Eliel: exactly, that's the plan
 942 2012-06-02 22:18:42 <jrmithdobbs> sipa: like 255.255.255.10.in-addr.sipa.be IN TXT "<csv row>"
 943 2012-06-02 22:18:56 <sipa> jrmithdobbs: i'd rather not
 944 2012-06-02 22:18:59 <jrmithdobbs> then to query for yourself you don't have to pull the whole thing
 945 2012-06-02 22:19:12 <jrmithdobbs> oh, ya, i forgot how you're doing the dns stuff ;p
 946 2012-06-02 22:19:22 <sipa> ?
 947 2012-06-02 22:19:49 <jrmithdobbs> it'd be a pretty simple change to the stats output code if it were just generating zone files
 948 2012-06-02 22:19:59 <sipa> oh implementing it wouldn't be hard
 949 2012-06-02 22:20:11 <sipa> but i prefer not to expose too much information
 950 2012-06-02 22:20:25 <jrmithdobbs> well the information is already exposed in the stats data
 951 2012-06-02 22:20:39 <sipa> i don't intend to keep having that public forever
 952 2012-06-02 22:21:01 <jrmithdobbs> why? it's great for people running public nodes to use as a sort of bitcoin looking glass
 953 2012-06-02 22:21:31 datagutt has quit (Quit: Computer has gone to sleep.)
 954 2012-06-02 22:21:36 <Eliel> sipa: do you think it would make sense to store block headers (and perhaps merkle trees too) in one file and transactions in another?
 955 2012-06-02 22:21:43 <jrmithdobbs> i've noticed problems on my node because of it, in fact
 956 2012-06-02 22:21:45 <sipa> Eliel: yes
 957 2012-06-02 22:22:14 <Eliel> is this being planned for 0.7?
 958 2012-06-02 22:22:24 <etotheipi_> sipa: I'm almost there, but I don't understand what H(n) is computing
 959 2012-06-02 22:22:36 Zarutian has quit (Quit: Zarutian)
 960 2012-06-02 22:23:11 T_SG has joined
 961 2012-06-02 22:23:18 T_SG has left ("Leaving...")
 962 2012-06-02 22:23:47 <sipa> etotheipi_: how many iterations you've done on average *before* reaching strength N
 963 2012-06-02 22:24:31 <sipa> I(n) extends that with the number of iterations at strength N itself
 964 2012-06-02 22:24:54 <sipa> iirc
 965 2012-06-02 22:25:04 <sipa> etotheipi_: it's implemented even by jeff, but there were some performance issues left
 966 2012-06-02 22:25:07 <sipa> eh
 967 2012-06-02 22:25:09 shurnormal has joined
 968 2012-06-02 22:25:13 <sipa> Eliel: it's implemented even by jeff, but there were some performance issues left
 969 2012-06-02 22:25:39 <sipa> Eliel: split blkindex.dat into blkhash.dat, txhash.dat and blkmeta.dat or something
 970 2012-06-02 22:26:29 <etotheipi_> sipa: what is the conops?  I thought we pick a value N, and tons of seeds until we find one that meets the criteria
 971 2012-06-02 22:26:32 <Eliel> sipa: oh, I actually meant splitting blk0001.dat into two :)
 972 2012-06-02 22:26:44 <etotheipi_> but are you solving for lower-strength seeds, first?
 973 2012-06-02 22:27:21 <Eliel> sipa: with one containing the block headers + merkle trees and the other only transactions.
 974 2012-06-02 22:28:27 <sipa> etotheipi_: yes, you pick a value for N in advance, and then generate random seeds in a loop, and for all k<N, test whether the predicate doesn't hold, and for N it does
 975 2012-06-02 22:29:04 <sipa> etotheipi_: sorry if the formulas are a bit dense - i found them by just working out a concrete example and then generalizing; it's probably a bit harder to verify its correctness now
 976 2012-06-02 22:29:52 <sipa> etotheipi_: so you've picked N; let's say we want on average 2^20 iterations, so we pick N=4
 977 2012-06-02 22:30:10 <sipa> that's the goal strength of the seed we are generating
 978 2012-06-02 22:30:46 guruvan has quit (Changing host)
 979 2012-06-02 22:30:46 guruvan has joined
 980 2012-06-02 22:30:46 guruvan has quit (Changing host)
 981 2012-06-02 22:30:46 guruvan has joined
 982 2012-06-02 22:30:51 <sipa> you randomly generate a seed S
 983 2012-06-02 22:31:01 <sipa> you do 257 HMAC-SHA512 iterations
 984 2012-06-02 22:31:19 <sipa> and after 257 iterations, you check whether interpreting the first 4 bytes of the hash results in a value lower than 0x1010000
 985 2012-06-02 22:31:36 <sipa> if it is, pick a new seed S and repeat
 986 2012-06-02 22:31:59 <sipa> if it is not, continue iterating, and after 363 iterations, check that the hash is below 0xB60183
 987 2012-06-02 22:32:05 <sipa> if it is, pick a new seed S and repeat
 988 2012-06-02 22:32:23 <sipa> if it is not, continue iterating, and after 513 iterations, check that the hash is below 0x80C1A2
 989 2012-06-02 22:32:30 <sipa> ...
 990 2012-06-02 22:33:03 <sipa> eventually, when after 1028 iterations the hash is below 0x4080DF, you found a strength=4 seed
 991 2012-06-02 22:33:42 <sipa> and if it is not, pick a new seed S and repeat
 992 2012-06-02 22:35:11 * Eliel can't quite figure out what kind of a strength this measures.
 993 2012-06-02 22:35:50 <sipa> Eliel: the idea is having a short seed that generates an entire deterministic wallet
 994 2012-06-02 22:36:28 <sipa> but the seed encodes its own "strength", which corresponds to a number of hashing iterations must be done on it
 995 2012-06-02 22:38:02 <sipa> Eliel: https://gist.github.com/2731997
 996 2012-06-02 22:38:40 <sipa> it's not a full explanation though, just the math for an idea that was known to the people who would read it :)
 997 2012-06-02 22:38:49 <jrmithdobbs> sipa: i don't understand how that's useful if your prng is already giving you good seed data
 998 2012-06-02 22:38:59 _Fireball has quit (Quit:  HydraIRC -> http://www.hydrairc.com <- The professional IRC Client :D)
 999 2012-06-02 22:39:04 <sipa> jrmithdobbs: because people will use it with bad PRNG's
1000 2012-06-02 22:39:07 <sipa> jrmithdobbs: aka their mind
1001 2012-06-02 22:39:32 <jrmithdobbs> you're saying derive the seed from the passphrase?
1002 2012-06-02 22:39:48 <Eliel> sipa: so, this is an attempt to measure the quality of the seed?
1003 2012-06-02 22:39:48 <sipa> that's not the intented use case, but an assumed side use case :)
1004 2012-06-02 22:40:11 <jrmithdobbs> why not randomly generate the seed and encrypt it, then you can use pbkdf2/scrypt/whateveryouwant for the key deriv
1005 2012-06-02 22:40:13 <etotheipi_> jrmithdobbs: yeah.. people will pick stupid passwords... but if the client requires these special-format seeds, they have to add a X-bit nonce to get their stupid password
1006 2012-06-02 22:40:35 <etotheipi_> *in order to make their stupid password work
1007 2012-06-02 22:43:30 <sipa> Eliel: it's probably better to start with the original idea: start from a seed, and check whether after 256 iterations the first 8 bits are 0; if so, the 255th iteration is you key; otherwise, check whether after 512 iterations there are 9 zero bits, ... and so on
1008 2012-06-02 22:43:39 <etotheipi_> sipa: so the advantage here is that you are essentially accomplishing the same thing, but with less search time during seed generation?
1009 2012-06-02 22:43:54 <sipa> etotheipi_: hmm?
1010 2012-06-02 22:44:32 <sipa> etotheipi_: you need on average 1048576 to generate a strength=4 seed; that's true for both honest users and attackers
1011 2012-06-02 22:44:37 <etotheipi_> sipa, I don't like the complexity of the multiple steps.... I like "pick a strength N, and search for seeds until you find it"
1012 2012-06-02 22:44:38 <sipa> *iterations
1013 2012-06-02 22:44:45 <jrmithdobbs> sipa: i don't understand how this gaurantees much of anything over pbkdf2? why the proof of work? all you're doing is possibly losing entropy inside sha if there's some issue we don't know about and because of the clamping
1014 2012-06-02 22:44:50 <sipa> etotheipi_: that's exactly it
1015 2012-06-02 22:45:29 <sipa> etotheipi_: you pick a strength N, and generate seeds until it matches the strength=N criteria
1016 2012-06-02 22:45:48 <gmaxwell> jrmithdobbs: because it restricts the source of key material to prevent certian pathological cases, and without storing a strength leaves an attacker doing more computation than the user was willing to do.
1017 2012-06-02 22:45:52 <etotheipi_> oh, so all the checks at 256, 512, etc are just verifying that it doesn't pass there
1018 2012-06-02 22:45:58 <sipa> etotheipi_: yes
1019 2012-06-02 22:46:18 <gmaxwell> jrmithdobbs: basically it moves "I'll use a totally not random password as my seed" out of the path of least resistance.
1020 2012-06-02 22:46:53 <sipa> jrmithdobbs: and yes, you lose entropy in the key, but it is compensated exactly by the iterations
1021 2012-06-02 22:47:09 <etotheipi_> sipa: I thought you were accepting seeds of a variable strength (i.e. if it passes for any value between N and N+i, then I"ll use it)
1022 2012-06-02 22:47:49 <sipa> etotheipi_: no no; there are just 16 predefined check points, each at a well-defined iteration count, and with a well-defined chance of success
1023 2012-06-02 22:47:59 <jrmithdobbs> gmaxwell: i'm saying don't directly derive the seed off the password at all
1024 2012-06-02 22:48:06 <Eliel> sipa: ok, so this is a clever scheme to make an attacker require more processing power than the user to brute force the seed?
1025 2012-06-02 22:48:14 rdponticelli has quit (Ping timeout: 245 seconds)
1026 2012-06-02 22:48:20 <sipa> jrmithdobbs: everybody smart says that, but not everyone will listen
1027 2012-06-02 22:48:39 TD has quit (Quit: TD)
1028 2012-06-02 22:48:49 graingert has joined
1029 2012-06-02 22:49:06 dinox has quit (Ping timeout: 265 seconds)
1030 2012-06-02 22:49:23 <sipa> Eliel: the nice thing is mainly that the attacker cannot know what the strength of a candidate seed it before spending work to derive a key from it
1031 2012-06-02 22:49:28 <sipa> *is
1032 2012-06-02 22:49:53 <sipa> and that you don't need to make the user remember his number of iterations
1033 2012-06-02 22:50:07 <graingert> sipa: how?
1034 2012-06-02 22:50:11 brwyatt is now known as brwyatt|Away
1035 2012-06-02 22:50:35 <sipa> graingert: scroll up
1036 2012-06-02 22:50:42 <graingert> I just joined
1037 2012-06-02 22:50:47 <jrmithdobbs> sipa: no i mean gen the seed off openssl's RAND, it's "good enough" on everything it runs on, then encrypt it, don't ever use the passphrase directly in the seeding material
1038 2012-06-02 22:50:47 <sipa> oh
1039 2012-06-02 22:50:51 <graingert> send me a link
1040 2012-06-02 22:50:59 <jrmithdobbs> sipa: *that* can be prevented in the software
1041 2012-06-02 22:51:22 <sipa> jrmithdobbs: and still people will build "vanity seed generators" or whatever
1042 2012-06-02 22:51:52 <jrmithdobbs> lol
1043 2012-06-02 22:52:35 <gmaxwell> Which isn't much of a problem (if you want to make a heroic effort to screw yourself, fine) except they distribute them and encourage other people to use them
1044 2012-06-02 22:52:56 <etotheipi_> sipa: I'm struggling to see what extra value we are getting out of this extra complexity... maybe if I start there, I will understand this better
1045 2012-06-02 22:53:20 <etotheipi_> instead of N bits after 2^N iterations... you want to do it your way because.... ?
1046 2012-06-02 22:54:22 <sipa> etotheipi_: the extra complexity of using that table instead of "N zero bits after 2^N iterations" gives you 1) generation difficulties that are a factor 2 apart and not 4 2) almost-exactly 2^N iterations in the average case, so you can predict how long generation will take
1047 2012-06-02 22:54:55 <Eliel> sipa: is the seed derived solely from the passphrase this way? That is, can the user regenerate it from scratch at some future time?
1048 2012-06-02 22:55:02 <etotheipi_> okay... I thought (1) was part of it, but apparently I didn't ask it in a way that made sense
1049 2012-06-02 22:55:41 <sipa> Eliel: there is no passphrase
1050 2012-06-02 22:56:05 <sipa> Eliel: the point is that you only have one thing to store/remember to regenerate the wallet keys: the seed
1051 2012-06-02 22:56:25 <sipa> the point it helps with is the case where people would try to use a passphrase AS seed
1052 2012-06-02 22:56:33 <sipa> (though it should just be randomly generated)
1053 2012-06-02 22:56:45 <Eliel> ah ok, now I get it.
1054 2012-06-02 22:57:36 <jrmithdobbs> sipa: you'll still be able to just use randomly generated seed completely and just zero out the top bits, yes?
1055 2012-06-02 22:57:36 <sipa> jrmithdobbs: ?
1056 2012-06-02 22:57:36 <etotheipi_> Eliel: even if the wallet is designed to only create wallets with a good PRNG, there's always a "wallet recovery" dialog where users can enter whatever they want
1057 2012-06-02 22:57:59 <sipa> that
1058 2012-06-02 22:59:14 <etotheipi_> and no matter how not at fault you are, when a user uses sha256("Bob") as their wallet seed, and an attacker finds it and steals all their funds... you still get blamed
1059 2012-06-02 23:01:27 <etotheipi_> they'll be complaining about how they lost all their money with your program, and yeah right that they'll admit they used a $#!+ passphrase
1060 2012-06-02 23:01:27 <Eliel> how cpu-intensive does this make the wallet regeneration?
1061 2012-06-02 23:01:27 <sipa> very quick, if you have the seed
1062 2012-06-02 23:01:27 <sipa> but only a fraction of seeds are valid, and an attacker needs to iterate the invalid ones to before he finds out there are invalid
1063 2012-06-02 23:01:27 <sipa> well, very quick unless you want to do it in javascript...
1064 2012-06-02 23:03:40 shurnormal has quit (Quit: http://driedleaves.no-ip.org)
1065 2012-06-02 23:03:40 <sipa> etotheipi_: the f and i values here jump with a factor of approximately sqrt(2) each level, so you get a *2 slowdown by combining them
1066 2012-06-02 23:03:40 <sipa> though the early drop-out for the cases where a lower strength is reached is compensated for as well
1067 2012-06-02 23:03:41 brwyatt is now known as Away!~brwyatt@pool-96-226-236-130.dllstx.fios.verizon.net|brwyatt
1068 2012-06-02 23:03:59 <sipa> graingert: it's not written down anywhere nicely and completely
1069 2012-06-02 23:05:48 <graingert> dayum
1070 2012-06-02 23:05:48 <gmaxwell> etotheipi_: and not just bob, — because this can't be salted the user gets complete multiple try and precomputation gain. So even apparently secure passwords like "8W3G7Pds9712++" are not.
1071 2012-06-02 23:05:48 <Eliel> does this make the initial generation of the seed very intensive?
1072 2012-06-02 23:05:48 <sipa> Eliel: as intensive as the user wants / as the application allows
1073 2012-06-02 23:05:48 brwyatt is now known as brwyatt|Away
1074 2012-06-02 23:05:48 <sipa> Eliel: with a variable (and a-priori selectable) number of iterations that is a power of two between 2^16 and 2^32
1075 2012-06-02 23:05:48 <sipa> (or maybe more, no need to stop there, technically)
1076 2012-06-02 23:06:41 <Eliel> this deterministic wallet thing is a whole can of worms :)
1077 2012-06-02 23:08:23 <sipa> i suppose one could use trellis optimization to get the iteration counts for future values closer to exactly those powers of two, but meh...
1078 2012-06-02 23:08:23 <sipa> it's within 99.9996%
1079 2012-06-02 23:08:23 <Eliel> can't you just do some kind of a entropy estimation on the supplied seed and reject ones with too little?
1080 2012-06-02 23:10:36 <gmaxwell> Eliel: No. because we need to worry about software the won't do that, and entropy estimation is a con,  what would you estimate the entropy of "8W3G7Pds9712++" is?
1081 2012-06-02 23:10:36 <Eliel> or alternatively have the seed always stored with a checksum that must be correct for the regeneration to work.
1082 2012-06-02 23:10:36 slush has quit (Read error: Connection reset by peer)
1083 2012-06-02 23:10:36 <gmaxwell> Eliel: incompetent or unethcial seed maker authors will just make tools to compute that for you.
1084 2012-06-02 23:10:36 <sipa> Eliel: these seeds *contain* a checksum, namely the fact that after that many iterations a hash lower than a particular number must appear
1085 2012-06-02 23:10:36 slush has joined
1086 2012-06-02 23:10:36 da2ce726 has joined
1087 2012-06-02 23:10:36 <sipa> furthermore, an attacker must try to recovery from a seed before noticing the checksum doesn't hold
1088 2012-06-02 23:12:54 <sipa> now, that checksum isn't particularly strong - 2-3 digits only
1089 2012-06-02 23:12:54 <sipa> but it's something
1090 2012-06-02 23:12:54 <jrmithdobbs> actually i think that's a bonus
1091 2012-06-02 23:12:54 <jrmithdobbs> as it acts as a checksum for legitimate users but is worthless to an attacker
1092 2012-06-02 23:12:54 <Eliel> yes, that is an interesting feature.
1093 2012-06-02 23:15:07 da2ce7 has quit (Ping timeout: 265 seconds)
1094 2012-06-02 23:15:07 <Eliel> sipa: how rare would valid seeds  be with this system?
1095 2012-06-02 23:15:07 <sipa> one in 256
1096 2012-06-02 23:15:07 <sipa> but the higher the strength, the rarer the seeds are
1097 2012-06-02 23:15:07 Turingi has quit (Read error: Connection reset by peer)
1098 2012-06-02 23:15:07 da2ce726 has quit (Read error: Connection reset by peer)
1099 2012-06-02 23:15:18 <Eliel> sipa: I think 1/256 chance for someone's dumb seed to get accepted is too much.
1100 2012-06-02 23:17:23 <Eliel> it'll happen on the first try to one person in 256
1101 2012-06-02 23:17:23 da2ce726 has joined
1102 2012-06-02 23:17:23 <gmaxwell> the tradeoff here is that slower makes it hard to make JS wallet code because sha512 is so astonishingly slow in JS. :(
1103 2012-06-02 23:17:23 <jrmithdobbs> sipa: how are you mixing the salt?
1104 2012-06-02 23:17:23 <etotheipi_> Eliel: that's only for minimum-strength seeds
1105 2012-06-02 23:17:23 <sipa> jrmithdobbs: salt?
1106 2012-06-02 23:17:23 <etotheipi_> using N=32 would be a 1-in-4billion chance
1107 2012-06-02 23:17:23 <sipa> jrmithdobbs: the iterations are HMAC-SHA512; the key is some fixed string
1108 2012-06-02 23:17:23 <jrmithdobbs> sipa: well, if used directly on a password/phrase, how would you combine a salt as you obviously don't want seeds to collide for people
1109 2012-06-02 23:17:23 <jrmithdobbs> ok, missed hmac part
1110 2012-06-02 23:18:00 <sipa> i'm tempted to allow an "advanced mode" where you can pick the key yourself (for example, wallet name, e-mail address, ...)
1111 2012-06-02 23:19:43 <Eliel> sipa: why not require people to use their email address as the key?
1112 2012-06-02 23:19:43 <jrmithdobbs> because i want to use random data not public information
1113 2012-06-02 23:19:43 <Eliel> would prevent people precalculating all allowed dumb passwords.
1114 2012-06-02 23:19:43 <sipa> Eliel: they'd complain that bitcoin is invaiding their privacy :p
1115 2012-06-02 23:19:43 <gmaxwell> Eliel: they couldn't get that the email wasn't actually going anywhere. Besides, these things can't be rotated, you really don't want to encourage people to use passwords for them. Humans can not pick secure passwords, not against attackers who can do hundreds of millions of tries per second.
1116 2012-06-02 23:21:57 <gmaxwell> That password I gave above was cracked from the mtgox database, and it was fully salted.. so the work applied to it was applied to it alone.
1117 2012-06-02 23:21:57 <jrmithdobbs> gmaxwell: source for that btw?
1118 2012-06-02 23:21:57 <Eliel> gmaxwell: I find it difficult to believe that was brute forced.
1119 2012-06-02 23:21:57 <jrmithdobbs> gmaxwell: do we know if it was generated by some kind of password vault software or something?
1120 2012-06-02 23:21:57 <gmaxwell> Eliel: It was.
1121 2012-06-02 23:24:14 <gmaxwell> Eliel: you likely have a limited conception of what cracking human generated passwords looks like.
1122 2012-06-02 23:24:14 <jrmithdobbs> gmaxwell: did someone publish a big list or something recently? i'd like to see if they broke mine
1123 2012-06-02 23:24:14 <jrmithdobbs> (mine wasn't human generated)
1124 2012-06-02 23:24:14 <jrmithdobbs> and was salted
1125 2012-06-02 23:24:14 <gmaxwell> Eliel: You see as random as that looks, it wasn't cracked by just iterating through every possible value.
1126 2012-06-02 23:24:14 <Eliel> gmaxwell: ok, in that case it's plausible.
1127 2012-06-02 23:24:14 <jrmithdobbs> what's the pattern?
1128 2012-06-02 23:24:14 <gmaxwell> rather, that string is really the result of a statistical model conditioned on hundreds of thousands of other passwords, and building up a sequence of likely passwords out of it.
1129 2012-06-02 23:26:28 <gmaxwell> You can see information about that kind of generator here: http://openwall.info/wiki/john/markov  (also links to the paper arguing that basically the same properties of passwords that make them memorable make them guessable)
1130 2012-06-02 23:26:28 Matt_von_Mises has joined
1131 2012-06-02 23:28:40 <Eliel> ... I see
1132 2012-06-02 23:30:56 <gmaxwell> jrmithdobbs: and no, yours isn't cracked as far as I know.
1133 2012-06-02 23:30:56 <Matt_von_Mises> Could the bug in OP_CHECKMULTISIG that pops too many items from the stack be fixed safely?
1134 2012-06-02 23:30:56 <gmaxwell> No.
1135 2012-06-02 23:30:56 <gmaxwell> but it's mostly harmless.
1136 2012-06-02 23:30:58 <Eliel> Isn't it fixed for P2SH transactions now?
1137 2012-06-02 23:31:15 <jrmithdobbs> ya, found the dump that's from
1138 2012-06-02 23:33:10 <jrmithdobbs> gmaxwell: i'm disappointed it hasn't been, honestly
1139 2012-06-02 23:33:10 <Matt_von_Mises> I guess it's also interesting. Imagine in 1000 years people look at the OP_CHECKMULTISIG bug in wonder.
1140 2012-06-02 23:33:10 <Eliel> I highly doubt Bitcoin will be anything but a historical relic by that time :)
1141 2012-06-02 23:33:11 <weex> with 22 million btc
1142 2012-06-02 23:34:47 <Matt_von_Mises> Eliel: Will we be trading dark matter or something by then?
1143 2012-06-02 23:35:53 <gmaxwell> well, bitcoin doesn't scale past the solar system at all, and not really much past the moon.
1144 2012-06-02 23:36:40 <Eliel> Matt_von_Mises: I'm not going to try to guess that far into the future :P
1145 2012-06-02 23:37:23 <gmaxwell> In any case, it can be fixed concurrently with a cryptosystem upgrade which already breaks old things.
1146 2012-06-02 23:39:29 hnz has joined
1147 2012-06-02 23:41:43 RazielZ has quit (Quit: Leaving)
1148 2012-06-02 23:43:43 dvide has quit ()
1149 2012-06-02 23:44:11 da2ce726 has quit (Read error: Connection reset by peer)
1150 2012-06-02 23:44:42 <sipa> gmaxwell, etotheipi_: anyway, is using such a table acceptable, or do you prefer sticking with the simpler N / 2^N formule, but have less fine-grained control over the generation process
1151 2012-06-02 23:45:26 da2ce726 has joined
1152 2012-06-02 23:46:41 <Eliel> sipa: I think it would be better to accept at most 1 in 65536 seeds as a valid seed even if that makes it impractical to do in javascript.
1153 2012-06-02 23:47:29 <sipa> that would mean moving to almost purely checksumming and not having much iteration left
1154 2012-06-02 23:50:39 <Eliel> I mostly think this because 1 in 256 is not enough to discourage the most persistent ones.
1155 2012-06-02 23:51:00 <gmaxwell> Eliel: thats not really the goal.
1156 2012-06-02 23:51:18 <gmaxwell> The goal is put the sane behavior on the path of least resistance.
1157 2012-06-02 23:51:23 Samuel has joined
1158 2012-06-02 23:51:35 <Eliel> well ok, 1 in 256 is enough for that.
1159 2012-06-02 23:51:44 <Samuel> Hello guys, it's been a long time since I've been to this chat
1160 2012-06-02 23:51:53 <gmaxwell> Eliel: so you can be stupid if you really want to, but not because you don't know better or are lazy.
1161 2012-06-02 23:52:11 <Eliel> Samuel: hello
1162 2012-06-02 23:52:14 <sipa> hi there
1163 2012-06-02 23:54:06 <etotheipi_> sipa: I would prefer the simpler option... generating seeds is rare, and I want to encourage people to use the concept, instead of being intimidated by the algorithm
1164 2012-06-02 23:54:30 <etotheipi_> I am enticed by the generation time reduction, but I don't feel like it's worth the extra complexity
1165 2012-06-02 23:55:11 <etotheipi_> (* encourage developers to use implement it)
1166 2012-06-02 23:55:43 <sipa> there's no reduction - just higher precision
1167 2012-06-02 23:56:23 <sipa> attacking always costs exactly as much during generation as during attack
1168 2012-06-02 23:56:29 <sipa> h
1169 2012-06-02 23:56:36 <Eliel> sipa: I think it's important to give users the option of selecting the HMAC key themselves. Just ask them to type in something that's personal and that they are sure to remember.
1170 2012-06-02 23:56:41 agricocb has joined
1171 2012-06-02 23:56:48 agricocb has quit (Changing host)
1172 2012-06-02 23:56:48 agricocb has joined
1173 2012-06-02 23:57:54 Samuel has quit (Quit: Page closed)
1174 2012-06-02 23:58:24 <etotheipi_> sipa, when you say that the gap between strength levels is 2 not 4, I thought you meant that generation cost was dropping
1175 2012-06-02 23:59:02 <etotheipi_> because in the simpler version, the verification cost of a seed is 2^N, but the serach time is 4^n
1176 2012-06-02 23:59:37 <sipa> yes, and in the more complex one the verification cost is 2^(N/2) and the search time is 2^N
1177 2012-06-02 23:59:57 <sipa> but obviously you'll choose a larger N