Blocks iterator

Iterates over Bitcoin blocks, decoding data inside Bitcoin Core's blocks directory.

Features: * Blocks are returned in height order, it avoids following reorgs (see max_reorg parameter) * Blocks come with metadata like all block's previous outputs, it allows computing transactions fee or verifying scripts in the blockchain.

Iteration modes

In rust programs

Used as a library blocks could be iterated:

• via the blocks_iterator::iter method like done in outputs_versions example
• via the blocks_iterator::par_iter method like done in verify example. par_iter is preferred when the task to be perforemd is computational costly.

Through Pipes

Other than inside Rust programs, ordered blocks with prevouts could be iterated using Unix pipes.

$ cargo build --release $ cargo build --release --examples $ ./target/release/blocks_iterator --blocks-dir /Volumes/Transcend/bitcoin-testnet/testnet3/blocks --network testnet --max-reorg 40 | ./target/release/examples/most_output_pipe | ./target/release/examples/heaviest_pipe >/dev/null ... [2021-10-21T10:10:24Z INFO most_output_pipe] most_output tx is d28305817238ee92e5d9ac0d81c3bf5ecf7399528e6d87226d726e4070c7e665 with #outputs: 30001 [2021-10-21T10:10:24Z INFO heaviest_pipe] heaviest tx is 73e64e38faea386c88a578fd1919bcdba3d0b3af7b6302bf6ee1b423dc4e4333 with weight: 3999608

Memory requirements and performance

Running (cargo run --release -- --network X --blocks-dir Y >/dev/null) on threadripper 1950X, Testnet @ 2130k, Mainnet @ 705k. Spinning disk.

| Network | --skip--prevout | --max-reorg | utxo-db | Memory | Time | |---------|-------------------|---------------|----------:|-------:|--------:| | Mainnet | true | 6 | no | 33MB | 1h:00m | | Mainnet | false | 6 | no | 5.3GB | 1h:29m | | Mainnet | false | 6 | 1 run | 201MB | 9h:42m | | Mainnet | false | 6 | 2 run | 113MB | 1h:05m | | Testnet | true | 40 | no | 123MB | 3m:03s | | Testnet | false | 40 | no | 1.4GB | 8m:02s | | Testnet | false | 40 | 1 run | 247MB | 16m:12s | | Testnet | false | 40 | 2 run | 221MB | 8m:32s |

Examples

Run examples with:

cargo run --release --example verify

• heaviest find the transaction with greatest weight
• most_output find the transaction with most output
• outputs_versions Count outputs witness version
• signaturesinwitness Count signatures in witness
• verify verify transactions in blocks using libbitcoin-consensus

Similar projects

• bitcoin-iterate this project inspired blocksiterator, the differences are:
  • bitcoin-iterate is C-based
  • bitcoin-iterate it's more suited for shell piping, while blocksitearator is intended to use as a rust library
  • bitcoin-iterate doesn't give previous outputs information
  • bitcoin-iterate is making two passage over blocks*.dat while blocks_iterator is doing one pass keeping out-order-blocks in memory (the latter is faster at the expense of higher memory usage)
• rust-bitcoin-indexer this project requires longer setup times (about 9 hours indexing) and a postgre database, once the indexing is finished it allows fast queries on relational database.
• electrs specifically intended for the Electrum protocol