Proxmox Backup Server Deduplication Mechanism and Backup Operations Design

Tue, 27 Oct 2026 10:50:02 +0900

As virtualization infrastructure scales, the proliferation of virtual machines (VMs) and data storage expansion lead to prolonged backup windows. Designs that routinely repeat conventional full image backups inevitably suffer from network bandwidth exhaustion and soaring storage costs, significantly impeding the reduction of RPO (Recovery Point Objective).

Proxmox Backup Server (PBS) is a backup solution designed to optimize data transfer and reduce storage consumption. This article outlines the internal architecture of PBS, deduplication mechanisms, encryption boundary constraints, and operational troubleshooting.

Differential Transfer and Chunk-Based Deduplication Architecture

PBS features a differential transfer mechanism leveraging QEMU dirty bitmaps. Because it detects and transfers only modified blocks, bandwidth consumption is significantly reduced.

+-----------------------------------------------------------------------+
| Incremental Backup Workflow |
+-----------------------------------------------------------------------+
| Proxmox VE (Client Side) |
| 1. Track modified blocks (QEMU dirty bitmaps) |
| 2. Split modified data into chunk payload |
| 3. Encrypt / Compress locally |
| |
| === Network Bandwidth Transit === |
| (Transmits ONLY new chunks) |
| || |
| \/ |
| Proxmox Backup Server (Datastore) |
| 4. Deduplicate against existing chunk catalog |
| 5. Write new chunks &amp; update snapshot manifest index |
+-----------------------------------------------------------------------+

Backup data is split into variable-sized chunks averaging approximately 4MB, and a SHA-256 hash is calculated for each chunk.

Deduplication Operating Principles

Reference Count Optimization: When a chunk hash that already exists in the same datastore is detected, only the reference count (index manifest) of the existing chunk is updated, avoiding duplicate writes of actual data.
Effectiveness in Multi-VM Environments: Deduplication benefits are particularly pronounced in multi-VM environments sharing the same base OS image.
Data Compression: The ZSTD algorithm is employed for data compression.

Deduplication Constraints Imposed by Encryption Boundaries

When client-side encryption (AES-256-GCM) is used in PBS, data is encrypted on the Proxmox VE host side prior to transfer. Ciphertexts processed with different encryption keys generate different hash values that cannot be decrypted, even if the source data is identical; therefore, deduplication does not function across different keys. When designing multi-tenant environments, the tradeoff between security isolation and deduplication efficiency must be evaluated.

Manifest Index and Live Restore

Unlike traditional incremental backups, PBS maintains an independent manifest file for each backup point.

+-----------------------------------------------------------------------+
| Manifest Indexing Architecture |
+-----------------------------------------------------------------------+
| Snapshot Index (Point-in-Time) |
| |-- Pointer -&gt; Chunk #00123 (Base OS data) |
| |-- Pointer -&gt; Chunk #04567 (Application binary) |
| |-- Pointer -&gt; Chunk #89012 (Modified DB block) |
+-----------------------------------------------------------------------+

During restoration, there is no need to sequentially apply a differential chain; any restore point can be accessed directly in a single pass by reading the chunks referenced by the manifest.

Additionally, the Live Restore feature for QEMU VMs enables immediate VM startup without waiting for full data restoration to complete. When the guest OS issues an access request to an untransferred region, the corresponding chunks are prioritized, fetched over the network, and allocated.

Data Integrity Verification

To address silent data corruption (bit rot) and storage failures, PBS executes periodic verification jobs (Verification Tasks).

Unencrypted Chunks: Verified by recalculating SHA-256 hashes.
Encrypted Chunks: Because the PBS server side does not hold decryption keys, integrity checks are performed using data structure checksums (CRC-32).

Troubleshooting

Operational failure patterns in PBS environments and their respective remediation workflows are structured as follows.

1. Drop in Deduplication Efficiency Due to Encryption Key Mismatch

Symptom: Despite multiple VMs using the identical OS image, storage consumption does not decrease and deduplication does not take effect.
Cause: Individual encryption keys are specified across different PVE nodes or jobs, resulting in mismatched chunk hashes.
Remediation: Configure shared usage of the same backup key for VM groups within the same tenant or security boundary.

proxmox-backup-client key create /etc/pve/priv/storage/pbs-shared-key.enc
pvesm set pbs-datastore --encryption-key /etc/pve/priv/storage/pbs-shared-key.enc

2. Disk I/O and Network Bottlenecks During Live Restore Execution

Symptom: Immediately after starting a VM via Live Restore, I/O wait spikes within the guest OS, leading to degraded service responsiveness.
Cause: Guest OS boot processes and runtime accesses concentrate, causing on-demand chunk fetching from the PBS datastore to saturate network bandwidth or IOPS limits.
Remediation: Ensure a network bandwidth of 10GbE or higher between PBS and PVE, and adopt NVMe/SSD arrays for the PBS datastore. Consider using standard background restoration except during emergencies.

3. Detection of Corrupt Chunks by Verification Jobs

Symptom: A verification failed: chunk does not match integrity check error is logged during periodic verification jobs.
Cause: Data degradation on storage media or packet corruption during network writes.
Remediation: Identify corrupted manifests and chunks, then perform replacement via remote sync (Sync Job) or by creating a fresh backup.

proxmox-backup-manager datastore verify datastore1
proxmox-backup-manager task log UPID:pbs-server:00001234:00005678:datastore1

Operational Verification Log Protocols

Verification logs capture datastore status and network port listening states on a PBS node.

# proxmox-backup-manager datastore status datastore1
┌────────────┬──────────────┬──────────────┬──────────────┬──────────────┐
│ Name │ Total │ Used │ Available │ History │
╞════════════╪══════════════╪══════════════╪══════════════╪══════════════╡
│ datastore1 │ 1.96 TiB │ 412.50 GiB │ 1.55 TiB │ Deduplication│
└────────────┴──────────────┴──────────────┴──────────────┴──────────────┘
Deduplication Factor: 3.42x

# ss -tulpn | grep proxmox-backup
tcp LISTEN 0 4096 *:8007 *:* users:(("proxmox-backup-",pid=1240,fd=6))

# proxmox-backup-client status --repository user@pbs!realm@pbs-server:datastore1
┌────────────────┬──────────────────────────────────────────┐
│ Field │ Value │
├────────────────┼──────────────────────────────────────────┤
│ datastore │ datastore1 │
│ total │ 2153922822144 (1.96 TiB) │
│ used │ 442918502400 (412.50 GiB) │
│ avail │ 1711004319744 (1.55 TiB) │
└────────────────┴──────────────────────────────────────────┘

Operational Notes

Proxmox Backup Server balances storage efficiency and restore speed through chunk-based deduplication and differential transfers. However, failing to establish client encryption policies, secure sufficient network bandwidth, and automate periodic verification jobs can hinder rapid recovery during failures. Designing with a thorough understanding of these architectural characteristics and conducting prior validation is recommended.

Proxmox-Backup on K-Life Hack | Systems Architecture & DevOps