Hard Drive Failures

RustFS ensures read/write access can still be provided when some disks fail through mechanisms similar to erasure coding, and automatically heals data after replacing disks.

Table of ContentsCopy Anchor Link

1. Unmount Failed Disk
2. Replace Failed Disk
3. Update /etc/fstab or RustFS Configuration
4. Remount New Disk
5. Trigger and Monitor Data Healing
6. Follow-up Checks and Notes

Unmount Failed DiskCopy Anchor Link

Before replacing the physical hard drive, you need to safely unmount the failed disk from the operating system level to avoid I/O errors in the file system or RustFS during the replacement process.

sudo umount /data/rustfs0

sudo systemctl stop rustfs

Replace Failed DiskCopy Anchor Link

After physically replacing the failed disk, you need to partition and format the new disk, and apply the same label as the original disk.

sudo mkfs.xfs -f -i size=512 -n ftype=1 -L RUSTFS0 /dev/sdb

Requirements

• New disk capacity ≥ original disk capacity;
• File system type consistent with other disks (XFS, per the installation guide);
• Recommend using labels (LABEL) or UUID for mounting to ensure disk order is not affected by system restarts.

Update /etc/fstab or RustFS ConfigurationCopy Anchor Link

Confirm that the mount item labels or UUIDs in /etc/fstab point to the new disk. The mount point must stay identical to the one listed in RUSTFS_VOLUMES (in /etc/default/rustfs), so the healed disk rejoins the cluster at the same path.

UUID=<filesystem-uuid> /data/rustfs0 xfs defaults,noatime,nodiratime 0 0

sudo blkid -s UUID -o value /dev/sdb

sudo mount -a
findmnt /data/rustfs0

Remount New DiskCopy Anchor Link

Execute the following commands to batch mount all disks and start the RustFS service:

sudo systemctl start rustfs

Confirm all disks are mounted normally:

df -hT /data/rustfs0

Trigger and Monitor Data HealingCopy Anchor Link

Once RustFS detects a freshly formatted disk at a known mount point, its background scanner heals the missing data onto it automatically — no manual command is required. Confirm that recovery has started and track its progress through the service logs:

journalctl -u rustfs -f
tail -f /var/log/rustfs/rustfs.log

You can also open the RustFS Console and check the disk status of the affected node.

Follow-up Checks and NotesCopy Anchor Link

1. Performance Monitoring

• I/O may fluctuate slightly during healing, recommend monitoring disk and network load.

2. Batch Failures

• If multiple failures occur in the same batch of disks, consider more frequent hardware inspections.

3. Regular Drills

• Regularly simulate disk failure drills to ensure team familiarity with recovery processes.

4. Maintenance Windows

• When failure rates are high, arrange dedicated maintenance windows to speed up replacement and healing.