TR-20231225 // PUBLIC RELEASE

Being S.M.A.R.T. 🧠 About Used Storage 💾

Released by

Prayag Bhakar

Series

Hardware

Release date

2023-12-25

Revised

2026-11-03add solid-state drives and correct the M.2 attribute guidance

2026-07-29improve prose and fix grammer

Length

2,154 words · 11 min · grade 6

Subjects

#apollo#zfs

Machine Learning and Deep Learning datasets are MASSIVE. Just to put it into perspective, here is what a few of the popular public ones weigh.

Those numbers just cover the raw files. Turning them into tokens and keeping the result adds terabytes more. Cheap, dense storage was one of the first things Apollo needed.

A new Hard Disk Drive (HDD) costs more than $20 per terabyte, so my hopes of enterprise storage on a budget went out the window. Instead I returned to the depths of eBay and bought refurbished enterprise drives, the kind businesses call "too old" and "too risky" to run and the internet says have one leg in the grave. How much life does a used drive have left?

Since then I've added M.2 Solid-State Drives (SSDs) to the pile. Same question, different answers.

1 badblocks, finding bad sectors

badblocks writes a pattern to every sector and reads it back. A sector that hands back something else has stopped holding data. An HDD keeps bits on magnetic platters spinning under a head that flies nanometers over them, so it has many different points of failure and hardware degradation.

An SSD trades that for flash. Nothing moves, but the flash wears out a little every time it's erased and rewritten. The firmware also hides its own bad blocks and shuffles where data lands, so tools like badblocks waste drive writes without showing any data. Use smartctl instead.

Start by installing the binaries and choose a drive, for example /dev/sdx.

$ yay -S e2fsprogs
$ fdisk -l

badblocks fails on a large drive unless the command names the drive's block size and each batch of reads and writes fits inside it.

$ blockdev --getbsz /dev/sdx

This drive reports 4096. Now scan it.

$ badblocks -t random -w -s -b 4096 /dev/sdx

$ nohup badblocks -t random -w -s -b 4096 /dev/sdx > dev.sdx.txt &

$ nohup badblocks -t random -w -s -b 4096 /dev/sdx $START_BLOCK > dev.sdx.2.txt &

$ yay -S f3
$ f3write /mnt
$ f3read /mnt

2 smartctl, reading the drive's own log

While badblocks lets you test if the drive can store data, Self-Monitoring, Analysis, and Reporting Technology (SMART) helps log device metrics and errors.

Start by installing the binaries and check that the drive supports it.

$ yay -S smartmontools
$ fdisk -l

# check if the drive supports SMART
$ smartctl -i /dev/sdx

M.2 tripped me up here. It only names the shape of the card, and the drive on it speaks either SATA or Non-Volatile Memory Express (NVMe). Mine speaks SATA and sits in a USB adapter, which smartctl mistook for an NVMe bridge.

$ smartctl -a /dev/sdb
Read NVMe Identify Controller failed: scsi error unsupported scsi opcode

-d sat tells it to talk SATA through the adapter, and -x prints everything the drive has, including a Device Statistics log that comes in handy later.

$ smartctl -x -d sat /dev/sdb
Device Model:     AngelTrax SSD
User Capacity:    1,920,383,410,176 bytes [1.92 TB]
Rotation Rate:    Solid State Device
Form Factor:      M.2
Device is:        Not in smartctl database 7.5/6061
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 1.5 Gb/s)

2.1 Manual tests

smartctl -t hands a self test background job to the drive's own firmware. The short test takes minutes and the long test lives up to its name.

# see how long each test takes on this drive
$ smartctl -c /dev/sdx

# start a test
$ smartctl -t short /dev/sdx
$ smartctl -t long /dev/sdx
# conveyance looks for damage from shipping
$ smartctl -t conveyance /dev/sdx

Read the log to see whether a test finished.

$ smartctl -l selftest /dev/sdx
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.7-arch1-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     43048         -
# 2  Short offline       Completed without error       00%     42997         -
# 3  Short offline       Completed without error       00%     42979         -

2.2 Reading the attribute table

Print the attribute table.

$ smartctl -A /dev/sdx
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.7-arch1-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       96
  3 Spin_Up_Time            0x0007   154   154   024    Pre-fail  Always       -       448 (Average 405)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       37
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   094   094   000    Old_age   Always       -       43051
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       37
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1603
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1603
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 22/49)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       2727

/dev/sdx's table reads Power_On_Hours at 43,051 hours, just short of five years, and other metrics. The attributes on offer differ by brand and drive connector. The reference tables in the wikis stay fresher than vendor documentation.

• Advanced Technology Attachment (ATA) attributes :: spinning and solid-state drives that speak SATA, including M.2 SATA cards.
• Non-Volatile Memory Express (NVMe) attributes :: M.2 cards that speak NVMe.
• Small Computer System Interface (SCSI) attributes :: drives behind a USB bridge, which answers in SCSI. Many bridges also pass ATA commands through when you run smartctl -d sat.

[!EXAMPLE] /dev/sdx's UDMA_CRC_Error_Count reads 2727, which is high. The wiki names a loose cable as one root cause, and that turned out to be exactly it! Reseating the cable stopped the count climbing.

An SSD's table reads differently. There's no Spin_Up_Time or helium to report, and Power_On_Hours is swapped for write counters. /dev/sdb isn't in the smartctl database, so half its table prints as Unknown_Attribute.

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  9 Power_On_Hours          0x0000   100   100   000    Old_age   Offline      -       22
161 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       148
168 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       3000
169 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      -       100
194 Temperature_Celsius     0x0000   100   100   000    Old_age   Offline      -       63
241 Total_LBAs_Written      0x0000   100   100   000    Old_age   Offline      -       55102

The IDs line up with the Silicon Motion presets in drivedb.h, which is a good hint at what they mean, but not proof.

• 161 :: spare blocks left to replace failed ones.
• 168 :: erase cycles the flash can take, by its rating.
• 169 :: percent of rated life left.
• 241 :: data the computer wrote, in 32 MiB chunks.

[!EXAMPLE] /dev/sdb's Total_LBAs_Written reads 55102, which looks like nothing until you scale it by 32 MiB. That comes to 1.85 TB!! Plus the Logical Sectors Written count from smartctl -x agrees.

For a used SSD, start with how much life the last owner spent.

0x07  0x008  1               0  ---  Percentage Used Endurance Indicator

Then check the spare blocks and the failed program and erase counts. NVMe drives make this easier, with a standard health log that reports Percentage Used and Available Spare under the same names on every brand.

2.3 Automated tests and alerts

smartd runs the same tests on a schedule and mails you when an attribute moves. Once again, start by installing the binaries.

$ systemctl enable smartd
$ systemctl start smartd

Then configure it in /etc/smartd.conf.

# /etc/smartd.conf

# `DEVICESCAN` :: run tests on every S.M.A.R.T. enabled drive
# `-a` :: for all S.M.A.R.T. values
# `-o on` :: save offline data
# `-S on` :: save attribute data
# `-n standby,q` :: skips the test if the drive is not active (increases drive lifespan)
# `-s (S/../.././02|L/../../4/03)` :: runs a daily short test and a long test on the 4th of each month
# `-W 4,35,40` :: log changes and dangerous operating temps
# `-m email@gmail.com` :: send an email for alerts
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../4/03) -W 4,35,40 -m email@gmail.com

3 zfs, the final frontier

badblocks and SMART tell you what shape a drive is in, and a drive with a few reallocated sectors is worth keeping. Software Redundant Array of Independent Disks (RAID) solutions such as Zettabyte File System (ZFS) protect data integrity with checksummed blocks that survive hardware failures, within the limits of the RAID layout you pick.

ZFS brings its own parity levels, named raidz1 through raidz3, but the tradeoff is needing to know the shape of your pool and drives beforehand.

Install the ZFS binaries and find the drives to use. zfs-dkms rebuilds ZFS against the running kernel, so the module keeps loading when a kernel upgrade lands ahead of a ZFS release. It also makes swapping driver versions easier.

# install and check zfs
$ yay -S zfs-dkms zfs-utils
$ modprobe zfs

# find the drives
$ fdisk -l

Enable the daemons that import and mount the pool at boot.

$ systemctl enable zfs-import-cache
$ systemctl enable zfs-import-scan
$ systemctl enable zfs-mount
$ systemctl enable zfs.target

For my setup, I chose a single raidz3 pool, which carries enough redundancy for three of the ten drives to give out without data loss. Swapping in a replacement resilvers the pool back to a healthy state. Luckily, I haven't needed to do that yet. Next, create the pool. Keeping with the space theme, it's named black_hole.

# create zfs pool
$ zpool create black_hole raidz3 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
$ zfs set mountpoint=/mnt/black_hole black_hole

# (optional) enable compression
$ zfs set compression=on black_hole

# (optional) create encrypted dataset
$ zfs create -o encryption=on -o keyformat=passphrase black_hole/enc

Check the pool.

$ zpool status black_hole
$ zfs get all black_hole

$ du -ah --max-depth 1
$ du -ah --max-depth 3 ~/downloads

4 What now?

A tested drive in a ZFS raidz3 pool with proper uptime brings us far more confidence that our data is secure and accessible even if we ended up using second hand drives. Watch the SMART attributes and zpool status from here to maintain observability of your data.

TR-20231225 // PUBLIC RELEASE

distribution per CC BY 4.0