ddiff

ddiff is a Linux tool for differential file and block device synchronization via SSH. It can also create differential files to quickly update offline backups.

Requirements

• Linux operating system on x86_64 or aarch64 (Raspberry Pi)
• Working SSH access for remote destinations
• dd available for block writes
• ddiff available on the remote destination for hash comparison via SSH

Warning

ddiff operates at a low level on files and devices. An incorrect path can overwrite data irreversibly. The author cannot be held responsible for any damage caused by the incorrect use of the command.

License

This software is distributed free of charge under the MIT license: https://mit-license.org/

Syntax

ddiff [OPTIONS] <input> <output>  
ddiff --hash [OPTIONS] <file> [start_block_number] [count]  
ddiff --map [OPTIONS] <input> <output_map>  
ddiff --makediff [OPTIONS] <map_file> <input> <output_diff_file> [output_map_file]  
ddiff --applydiff [OPTIONS] <diff_file> <output>

<input> can be a regular file or a block device. <output> can be a file or a block device, either local or reachable via SSH in the format user@host:/path.

Options

• -y, --yes: automatically answer "yes" to confirmation prompts
• -q, --quiet: hide progress bars

Direct Backup Examples

ddiff source.img backup.img  
ddiff /dev/sdb user@host:/srv/backup.img  
ddiff /dev/sdb root@host:/dev/sdb

How to Perform Offline Backups in Two Steps

To perform a two-step backup, you need to:

1) Create or have a map file of the current backup state

2) Create a diff file by comparing the source with the backup map file

3) Apply the diff file to the backup


Note: It is recommended to create and keep the map updated during the --makediff function to have it ready for the next backup.

Map File Creation

ddiff --map disk.img disk.map

The map file is binary and contains a sequence of BLAKE3 hashes for every 1 MiB block of the source file. The map file should be created for the backup and will be used to compare it with the source.

Diff File Creation

ddiff --makediff baseline.map disk.img patch.diff  
ddiff --makediff baseline.map disk.img patch.diff new.map

--makediff compares the current file with the base map_file and creates a ZIP archive containing only the modified blocks.

If output_map_file is specified, the new map file of the current source is also written. This is very useful for keeping backup references on the machine where the source resides.

Applying a Diff

ddiff --applydiff patch.diff restored.img

--applydiff writes the blocks present in the diff to an existing local file or block device. Before writing, it verifies that the current hash of the blocks to be overwritten matches the old field in the diff.

If any of these checks fail, the command exits with exit code 3.

If the output is a regular file, it is truncated to the final length recorded in the diff upon completion. If it is a block device, no truncation is performed.

Service Function

This function, although documented, is primarily used automatically during copy processes.

ddiff --hash disk.img  
ddiff --hash disk.img 120  
ddiff --hash disk.img 120 32

--hash returns the BLAKE3 hashes of one or more blocks:

• without start_block_number and count, it returns all hashes of the file in order;
• with only start_block_number, it returns a single block hash;
• with start_block_number and count, it returns count consecutive hashes.

Each hash is emitted to stdout in hexadecimal format, one per line. Progress bars, warnings, and service messages are sent to stderr.

File Formats

Map

The map file is a binary file consisting of consecutive 32-byte records, one for each BLAKE3 hash.

Diff

The diff file is a ZIP archive containing:

• index.json
• a binary file for each modified block, named after the block number

index.json contains:

• version
• size: expected final size of the output
• blocks: array of modified blocks

Exit Codes

• 0: execution completed successfully
• 1: generic error
• 2: ddiff missing on remote destination or incompatible version
• 3: old hash mismatch during --applydiff