Blocks iterator

Iterates over Bitcoin blocks, decoding data inside Bitcoin Core's blocks directory.

Features: * Blocks are returned in height order, it avoids following reorgs (see max_reorg parameter) * Blocks come with metadata like all block's previous outputs, it allows computing transactions fee or verifying scripts in the blockchain.

Pipes

Other than inside Rust programs, ordered blocks with prevouts could be iterated using Unix pipes.

$ cargo build --release $ cargo build --release --examples $ ./target/release/blocks_iterator --blocks-dir /Volumes/Transcend/bitcoin-testnet/testnet3/blocks --network testnet --max-reorg 40 | ./target/release/examples/most_output_pipe | ./target/release/examples/heaviest_pipe >/dev/null ... [2021-10-21T10:10:24Z INFO most_output_pipe] most_output tx is d28305817238ee92e5d9ac0d81c3bf5ecf7399528e6d87226d726e4070c7e665 with #outputs: 30001 [2021-10-21T10:10:24Z INFO heaviest_pipe] heaviest tx is 73e64e38faea386c88a578fd1919bcdba3d0b3af7b6302bf6ee1b423dc4e4333 with weight: 3999608

Memory requirements and performance

Running main executable on threadripper 1950X, Testnet @ 2090k, Mainnet @ 705k. Spinning disk.

cargo run --release >/dev/null

| Network | --skip--prevout | --max-reorg | utxo-db | Memory | Time | |---------|-------------------|---------------|----------:|-------:|--------:| | Mainnet | true | 6 | no | 33MB | 1h:00m | | Mainnet | false | 6 | no | 5.3GB | 1h:29m | | Mainnet | false | 6 | 1 run | 201MB | 9h:42m | | Mainnet | false | 6 | 2 run | 113MB | 1h:05m | | Testnet | true | 40 | no | 123MB | 3m:03s | | Testnet | false | 40 | no | 1.4GB | 9m:07s | | Testnet | false | 40 | 1 run | 247MB | 16m:12s | | Testnet | false | 40 | 2 run | 221MB | 8m:32s |

Examples

Run examples with:

cargo run --release --example verify

• heaviest find the transaction with greatest weight
• most_output find the transaction with most output
• verify verify transactions

Similar projects

• bitcoin-iterate this project inspired blocksiterator, the differences are:
  • bitcoin-iterate is C-based
  • bitcoin-iterate it's more suited for shell piping, while blocksitearator is intended to use as a rust library
  • bitcoin-iterate doesn't give previous outputs information
  • bitcoin-iterate is making two passage over blocks*.dat while blocks_iterator is doing one pass keeping out-order-blocks in memory (the latter is faster at the expense of higher memory usage)
• rust-bitcoin-indexer this project requires longer setup times (about 9 hours indexing) and a postgre database, once the indexing is finished it allows fast queries on relational database.
• electrs specifically intended for the Electrum protocol