Skip to main content

Reduce database size for Bonsai Tries

To decrease the database size using the Bonsai Trie data storage policy in Besu, enable the early access feature --Xbonsai-limit-trie-logs-enabled. When enabled, this feature can reduce database growth by more than 3GB each week on mainnet.

Limit Trie logs​

caution

The following commands are examples. Before executing these example commands on your node, modify them to apply to your node's configuration.

  1. Add the --Xbonsai-limit-trie-logs-enabled option to the Besu configuration file.
caution

If you are using a systemd service file, as recommended by CoinCashew and Somer, ensure you execute sudo systemctl daemon-reload.

  1. Stop Besu.
  2. (optional) Run the Besu trie log prune command. Specify the Bonsai Trie data storage format and the data directory for your Besu database:
    Run command to prune trie logs
    sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune
  3. Start Besu.
  4. Look for Limit trie logs enabled: retention: 512; prune window: 30000 in your Besu configuration printout at startup.

Prune outdated trie logs​

When you start Besu with --Xbonsai-limit-trie-logs-enabled, it will continuously prune the unnecessary trie log data, removing it one block at a time. This process begins after an initial reduction in the database size during startup.

Enabling --Xbonsai-limit-trie-logs-enabled on a long-running node does not immediately clear your backlog of trie logs in the same way resyncing does. Instead of resyncing, you can run an offline command to immediately prune old trie logs. To run the offline command, you must shutdown Besu for a minimal period. If the --Xbonsai-limit-trie-logs-enabled option is enabled, you do not need to run the offline command again after initially running it.

For minimal downtime, we recommend running the offline command before restarting Besu with --Xbonsai-limit-trie-logs-enabled.

If you are following the guides by Somer Esat or CoinCashew, you have set these options in your besu.service or execution.service systemd file:

Prune trie logs
...
ExecStart=/usr/local/bin/besu/bin/besu \
...
  --sync-mode=SNAP \
  --data-path="/var/lib/besu" \
  --data-storage-format=BONSAI \
...

To prune trie logs, the command should look similar to the following:

Prune trie logs
sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=SNAP storage x-trie-log prune

The logs should look similar to the following:

2024-02-02 05:45:41.162+00:00 | main | INFO  | KeyPairUtil | Attempting to load public key from /data/besu/key
 ...
2024-02-02 05:45:43.433+00:00 | main | INFO  | TrieLogSubCommand | Estimating trie logs size before pruning...
2024-02-02 05:45:43.837+00:00 | main | INFO  | TrieLogSubCommand | Estimated trie logs size before pruning: 9 GiB
2024-02-02 05:46:09.863+00:00 | main | INFO  | TrieLogHelper | Starting pruning: retain 512 trie logs, processing in 1 batches...
2024-02-02 05:46:09.918+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file /data/besu/database/trieLogsToRetain-1 (batch 1)...
2024-02-02 05:46:09.926+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
2024-02-02 05:46:10.100+00:00 | main | INFO  | TrieLogHelper | Clear trie logs...
2024-02-02 05:46:10.155+00:00 | main | INFO  | TrieLogHelper | Restoring trie logs retained from batch 1...
2024-02-02 05:46:10.222+00:00 | main | INFO  | TrieLogHelper | Key(0): 0xcd50706da7f6f2db7f9d54f3589122760900d9ab2508c20a4ca40b496d930368
... 
2024-02-02 05:46:10.336+00:00 | main | INFO  | TrieLogHelper | Key(511): 0x238f9649b59616430ad7e43b8f3cf65bc97cac4aa54a3eddf3ad6ee666ce733e
2024-02-02 05:46:10.441+00:00 | main | INFO  | TrieLogHelper | Deleting files...
2024-02-02 05:46:10.446+00:00 | main | INFO  | TrieLogSubCommand | Finished pruning. Re-estimating trie logs size...
2024-02-02 05:46:11.023+00:00 | main | INFO  | TrieLogSubCommand | Estimated trie logs size after pruning: 0 B (0 B estimate is normal when using default settings)
2024-02-02 05:46:11.023+00:00 | main | INFO  | TrieLogSubCommand | Prune ran successfully. We estimate you freed up 9 GiB!
Prune ran successfully. We estimate you freed up 9 GiB!

If you are using a TOML configuration file, you can run a command similar to the following:

TOML file prune trie logs
sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune

Troubleshoot​

Troubleshoot common errors that can occur when attempting to use the trie log prune command to reduce your database size. Ensure your command specifies the following:

  • --data-storage-format
  • --data-path
  • --sync-mode

Prune command for mainnet​

The prune command should look similar to the following:

Prune command for mainnet
sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune

Ensure you stop Besu before running the command.

Subcommand not working​

  • Error message: java.lang.IllegalArgumentException: Subcommand only works with data-storage-format=BONSAI

The --data-storage-format=BONSAI may be missing. To resolve, add the storage format. The command should look similar to the following:

Specify data storage format
sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=SNAP storage x-trie-log prune

Column handle not found for segment TRIE_BRANCH_STORAGE​

  • Error message: java.lang.RuntimeException: Column handle not found for segment TRIE_BRANCH_STORAGE

Ensure you specify the data-path. Your command should look similar to the following:

Specify data-path
sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune

Database not detected​

  • Error message: java.lang.IllegalArgumentException: Trying to retain more trie logs than chain length (0), skipping pruning

Ensure you specify the correct --data-path for your node. Your command should look similar to the following:

Correct data-path
sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune

Cannot store generated private key​

  • Error message: java.lang.IllegalArgumentException: Cannot store generated private key

Ensure you specify the correct --data-path for your node. Your command should look similar to the following:

Correct data-path
sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=SNAP storage x-trie-log prune

Valid keyPair not provided​

  • Error message: java.lang.IllegalArgumentException: Supplied file does not contain valid keyPair pair.

Check your file permissions and try running a sudo command to resolve:

Correct permissions
sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu storage --sync-mode=SNAP x-trie-log prune

Column handle not found for segment WORLD_STATE​

  • Error message: java.lang.RuntimeException: Column handle not found for segment WORLD_STATE

Ensure you are using data-storage-format=BONSAI instead of data-storage-format=FOREST on an existing Bonsai database.

Resource temporarily unavailable​

  • Error message: org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: While lock file: /data/besu/database/LOCK: Resource temporarily unavailable

Check if Besu is already running. You must shut down the Besu client before running the subcommand.

Resource temporarily unavailable​

  • Error message: java.lang.IllegalStateException: Unable to change the sync mode when snap sync is incomplete, please restart with snap sync mode

Check that you have specified the sync-mode. The default is sync-mode=FAST. Most mainnet users use X_SNAP or X_CHECKPOINT.

Cannot run trie log prune​

  • Error message: java.lang.RuntimeException: No finalized block present, can't safely run trie log prune

This message may appear if your node is relatively new or recently resynced. To resolve this error, ensure that your node is fully synced and properly configured to recognize finalized blocks.

Block does not match chain data stored​

  • Error message: org.hyperledger.besu.util.InvalidConfigurationException: Supplied genesis block does not match chain data stored in /data/besu.

Check if you are running the command for a network other than mainnet. To specify a network, run a command that looks similar to the following:

Specify network
`sudo /usr/local/bin/besu/bin/besu --network=holesky --sync-mode=SNAP --data-storage-format=BONSAI --data-path=/var/lib/besu storage x-trie-log prune`