Recovering a ZFS Pool Using the Resilvering Process in Proxmox

Introduction

When a disk in your ZFS pool fails, the pool enters a DEGRADED state. ZFS offers a robust recovery mechanism called resilvering, which allows you to replace the failed disk with a new one and rebuild the data. This article explains how to replace a failed disk and complete the resilvering process in your Proxmox ZFS pool.

---

Identifying the Problem

When a disk fails in the ZFS pool, the pool status will show as DEGRADED. Use the following command to check the pool status:

zpool status

Example output:

  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Replace the device using 'zpool replace'.

---

Steps to Replace the Failed Disk

1. Identify the Failed Disk

From the zpool status output, note the ID of the failed disk. For example:

  5446107257933431427                             UNAVAIL      0     0     0

2. Install the New Disk

Physically replace the failed disk with a new one. Verify that the new disk is recognized by the system:

ls -l /dev/disk/by-id/

Find the new disk’s ID (e.g., ata-WDC_WD30EFRX-68AX9N0_WD-WCC1T0809124).

3. Clear Any Existing Filesystem on the New Disk

If the new disk contains an existing filesystem, ZFS may reject it. Clear the filesystem using:

wipefs -a /dev/disk/by-id/<new_disk_id>

Replace <new_disk_id> with the appropriate disk ID.

4. Replace the Failed Disk

Use the zpool replace command to replace the failed disk with the new disk:

zpool replace rpool <failed_disk_id> /dev/disk/by-id/<new_disk_id>

Example:

zpool replace rpool 5446107257933431427 /dev/disk/by-id/ata-WDC_WD30EFRX-68AX9N0_WD-WCC1T0809124

---

Monitoring the Resilvering Process

After replacing the disk, ZFS will automatically start rebuilding the data (resilvering). Monitor the progress using:

zpool status

Example output:

  scan: resilver in progress since Sat Dec 21 21:32:03 2024
        10.9G / 10.9G scanned, 56.6M / 5.37G issued at 14.2M/s
        21.2M resilvered, 1.03% done, 00:06:24 to go

The process duration depends on the amount of data and system performance.

---

Post-Resilvering Verification

Once the resilvering process is complete, verify that the pool has returned to a healthy state:

zpool status

All disks should be listed as ONLINE. Example output:

  pool: rpool
 state: ONLINE
  scan: resilvered 5.37G in 0h7m with 0 errors on Sat Dec 21 21:39:01 2024
config:

        NAME                                                STATE     READ WRITE CKSUM
        rpool                                               ONLINE       0     0     0
          mirror-0                                          ONLINE       0     0     0
            ata-WDC_WD30EFRX-68AX9N0_WD-WCC1T1080266-part3  ONLINE       0     0     0
            ata-WDC_WD30EFRX-68AX9N0_WD-WCC1T1101796-part3  ONLINE       0     0     0
          mirror-1                                          ONLINE       0     0     0
            ata-WDC_WD30EFRX-68AX9N0_WD-WCC1T1401427-part3  ONLINE       0     0     0
            ata-WDC_WD30EFRX-68AX9N0_WD-WCC1T0809124        ONLINE       0     0     0

errors: No known data errors

---

Troubleshooting

If you encounter issues:

• Ensure the new disk is properly connected and recognized by the system.
• If zpool replace fails, check for existing partitions or data on the new disk and clear them using wipefs.
• Review system logs for any hardware errors: dmesg | grep ZFS

---

The resilvering feature in ZFS allows you to recover from disk failures without data loss. By following these steps, you can replace a failed disk and restore your ZFS pool to a healthy state. Regular monitoring and proper maintenance of your storage system ensure long-term reliability.