TR-20231225 // PUBLIC RELEASE

Being S.M.A.R.T. 🧠 About Used Storage 💾

Released by

Prayag Bhakar

Series

Hardware

Release date

2023-12-25

Revised

2026-07-29improve prose and fix grammer

Length

1,625 words · 9 min · grade 7

Subjects

#apollo#zfs

Machine Learning and Deep Learning datasets are MASSIVE. Just to put it into perspective, here is what a few of the popular public ones weigh.

Those numbers just cover the raw files. Turning them into tokens and keeping the result adds terabytes more. Cheap, dense storage was one of the first things Apollo needed.

A new Hard Disk Drive (HDD) costs more than $20 per terabyte, so my hopes of enterprise storage on a budget went out the window. Instead I returned to the depths of eBay and bought refurbished enterprise drives, the kind businesses call "too old" and "too risky" to run and the internet says have one leg in the grave. How much life does a used drive have left?

1 badblocks, finding bad sectors

badblocks writes a pattern to every sector and reads it back. A sector that hands back something else has stopped holding data. An HDD keeps bits on magnetic platters spinning under a head that flies nanometers over them, so it has many different points of failure and hardware degradation.

Start by installing the binaries and choose a drive, for example /dev/sdx.

$ yay -S e2fsprogs
$ fdisk -l

badblocks fails on a large drive unless the command names the drive's block size and each batch of reads and writes fits inside it.

$ blockdev --getbsz /dev/sdx

This drive reports 4096. Now scan it.

$ badblocks -t random -w -s -b 4096 /dev/sdx

$ nohup badblocks -t random -w -s -b 4096 /dev/sdx > dev.sdx.txt &

$ nohup badblocks -t random -w -s -b 4096 /dev/sdx $START_BLOCK > dev.sdx.2.txt &

2 smartctl, reading the drive's own log

While badblocks lets you test if the drive can store data, Self-Monitoring, Analysis, and Reporting Technology (SMART) helps log device metrics and errors.

Start by installing the binaries and check that the drive supports it.

$ yay -S smartmontools
$ fdisk -l

# check if the drive supports SMART
$ smartctl -i /dev/sdx

2.1 Manual tests

smartctl -t hands a self test background job to the drive's own firmware. The short test takes minutes and the long test lives up to its name.

# see how long each test takes on this drive
$ smartctl -c /dev/sdx

# start a test
$ smartctl -t short /dev/sdx
$ smartctl -t long /dev/sdx
# conveyance looks for damage from shipping
$ smartctl -t conveyance /dev/sdx

Read the log to see whether a test finished.

$ smartctl -l selftest /dev/sdx
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.7-arch1-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     43048         -
# 2  Short offline       Completed without error       00%     42997         -
# 3  Short offline       Completed without error       00%     42979         -

2.2 Reading the attribute table

Print the attribute table.

$ smartctl -A /dev/sdx
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.7-arch1-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       96
  3 Spin_Up_Time            0x0007   154   154   024    Pre-fail  Always       -       448 (Average 405)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       37
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   094   094   000    Old_age   Always       -       43051
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       37
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1603
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1603
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 22/49)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       2727

/dev/sdx's table reads Power_On_Hours at 43,051 hours, just short of five years, and other metrics. The attributes on offer differ by brand and drive connector. The reference tables in the wikis stay fresher than vendor documentation.

• Advanced Technology Attachment (ATA) attributes :: spinning and solid-state drives on a SATA cable.
• NVMe attributes :: drives in an M.2 slot.
• Small Computer System Interface (SCSI) attributes :: drives behind a USB bridge, which answers in SCSI.

[!EXAMPLE] /dev/sdx's UDMA_CRC_Error_Count reads 2727, which is high. The wiki names a loose cable as one root cause, and that turned out to be exactly it! Reseating the cable stopped the count climbing.

2.3 Automated tests and alerts

smartd runs the same tests on a schedule and mails you when an attribute moves. Once again, start by installing the binaries.

$ systemctl enable smartd
$ systemctl start smartd

Then configure it in /etc/smartd.conf.

# /etc/smartd.conf

# `DEVICESCAN` :: run tests on every S.M.A.R.T. enabled drive
# `-a` :: for all S.M.A.R.T. values
# `-o on` :: save offline data
# `-S on` :: save attribute data
# `-n standby,q` :: skips the test if the drive is not active (increases drive lifespan)
# `-s (S/../.././02|L/../../4/03)` :: runs a daily short test and a long test on the 4th of each month
# `-W 4,35,40` :: log changes and dangerous operating temps
# `-m email@gmail.com` :: send an email for alerts
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../4/03) -W 4,35,40 -m prayag.bhakar@gmail.com

3 zfs, the final frontier

badblocks and SMART tell you what shape a drive is in, and a drive with a few reallocated sectors is worth keeping. Software Redundant Array of Independent Disks (RAID) solutions such as Zettabyte File System (ZFS) protect data integrity with checksummed blocks that survive hardware failures, within the limits of the RAID layout you pick.

ZFS brings its own parity levels, named raidz1 through raidz3, but the tradeoff is needing to know the shape of your pool and drives beforehand.

Install the ZFS binaries and find the drives to use. zfs-dkms rebuilds ZFS against the running kernel, so the module keeps loading when a kernel upgrade lands ahead of a ZFS release. It also makes swapping driver versions easier.

# install and check zfs
$ yay -S zfs-dkms zfs-utils
$ modprobe zfs

# find the drives
$ fdisk -l

Enable the daemons that import and mount the pool at boot.

$ systemctl enable zfs-import-cache
$ systemctl enable zfs-import-scan
$ systemctl enable zfs-mount
$ systemctl enable zfs.target

For my setup, I chose a single raidz3 pool, which carries enough redundancy for three of the ten drives to give out without data loss. Swapping in a replacement resilvers the pool back to a healthy state. Luckily, I haven't needed to do that yet. Next, create the pool. Keeping with the space theme, it's named black_hole.

# create zfs pool
$ zpool create black_hole raidz3 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
$ zfs set mountpoint=/mnt/black_hole black_hole

# (optional) enable compression
$ zfs set compression=on black_hole

# (optional) create encrypted dataset
$ zfs create -o encryption=on -o keyformat=passphrase black_hole/enc

Check the pool.

$ zpool status black_hole
$ zfs get all black_hole

$ du -ah --max-depth 1
$ du -ah --max-depth 3 ~/downloads

4 What now?

A tested drive in a ZFS raidz3 pool with proper uptime brings us far more confidence that our data is secure and accessible even if we ended up using second hand HDDs. Watch the SMART attributes and zpool status from here to maintain observability of your data.

TR-20231225 // PUBLIC RELEASE

distribution per CC BY 4.0

Backlinks

INBOUND_REF01 LNK

🪐 Apollo 🛸 AI Compute Cluster for the GPU Poor

Hardware · 2023-12-24

Are you GPU Poor but want to spend your money wisely on that shiny brand new deep learning compute cluster?

/dev/null
[EOF]

/dev/null
[EOF]