Why lsblk is not reliable for repair.
Let’s walk through a simple 4 NVMe scenario to show why this happens and how to correctly identify the failed disk.
You have a server with 4 NVMe drives
[ PCIe Slot Mapping ]
+---------------------------+
| Slot A: NVMe 0 |
| Slot B: NVMe 1 |
| Slot C: NVMe 2 |
| Slot D: NVMe 3 |
+---------------------------+
On first boot, everything looks healthy:
$ lsblk
nvme0n1 931G
nvme1n1 931G
nvme2n1 931G
nvme3n1 931G
Now let’s say the drive in Slot B fails. After a reboot, the kernel renumbers the remaining devices
$ lsblk
nvme0n1 931G
nvme1n1 931G <-- used to be nvme2n1
nvme2n1 931G <-- used to be nvme3n1
At a glance, it looks like nvme3n1 is the “bad drive” because it’s missing compared to the original layout but that’s not actually true. Device names are not fixed to physical slots.
That’s the whole trap.
Linux assigns NVMe device names in the order they’re discovered at boot. If a drive disappears
The list becomes shorter
Higher numbered devices shift down to fill the gap
This is different from SATA (/dev/sdX) where naming can also shift, but NVMe changes are more common due to how PCIe enumeration works.
So if you rely only on lsblk, you may remove the wrong disk.
Here’s a practical way to confirm the correct failed NVMe drive without guesswork.
dmesg | grep -i nvme
This usually shows errors like timeouts, disconnects, or controller resets. You’ll often see which PCIe address the failed device belonged to.
Example,
nvme nvme1: I/O 8 QID 3 timeout
nvme nvme1: Failed status: 0x2002
pci0000:3c:00.0: unable to enumerate NVMe device
Note the PCI address (3c:00.0 in this example).
sudo nvme list
Healthy output:
Node SN Model Namespace Usage
/dev/nvme0n1 AB12A3CD4E Samsung PM9A3 1.92TB 1 1.92TB
After failure, the missing serial number confirms which unit is gone.
Serial numbers are your friend they don’t lie or reorder.
This is the most reliable mapping method on Linux:
ls -l /dev/disk/by-path/
Example output (abbreviated):
pci-0000:3b:00.0-nvme0 -> ../../nvme0n1
pci-0000:3c:00.0-nvme1 -> ../../nvme1n1
pci-0000:3d:00.0-nvme2 -> ../../nvme2n1
pci-0000:3e:00.0-nvme3 -> ../../nvme3n1
If the drive for pci-0000:3c:00.0 is missing, you now know exactly which slot to pull.
No guesswork. No renumbering confusion.
TL;DR for Troubleshooting NVMe Failures
lsblk is helpful, but NVMe names shift after failures and reboots
#linux #computer #computerscience
Always check:
dmesg for PCIe/NVMe errors
nvme list for serial numbers
/dev/disk/by-path/ to map device → PCIe slot
If you rely on /dev/nvmeXn1 naming alone, you’re one mistake away from pulling the wrong drive.
When troubleshooting NVMe drive failures, it's essential to understand that device naming in Linux is dynamic and can shift depending on hardware enumeration during boot. The lsblk command, while useful for listing block devices, depends on kernel-assigned device names such as /dev/nvme0n1, /dev/nvme1n1, etc. However, these names are assigned sequentially based on the detection order at boot time and are not permanently fixed to the physical PCIe slots or slots on the server. In scenarios with multiple NVMe drives, if one drive fails or is removed, subsequent devices can shift their names downward to fill the gap left by the missing device. For example, if the drive in Slot B (originally nvme1n1) fails, after reboot, nvme2n1 and nvme3n1 may be renamed to nvme1n1 and nvme2n1. This renumbering creates confusion when trying to identify which physical drive has actually failed, leading to the risk of replacing the wrong disk. To avoid this pitfall, administrators should supplement lsblk output with additional commands and checks: 1. **Check dmesg logs for NVMe errors:** Running `dmesg | grep -i nvme` shows kernel messages including I/O timeouts, device enumeration failures, and PCIe addresses associated with the failed device. These logs highlight which PCIe slot corresponds to the failed hardware. 2. **Use `nvme list` to view device serial numbers:** This command lists all detected NVMe devices alongside their serial numbers and metadata. After a failure, missing serial numbers can confirm which drive is absent, giving a more reliable identification than device names alone. 3. **Map devices with `/dev/disk/by-path/` links:** These symbolic links associate device nodes with their exact PCIe addresses, bridging the gap between logical device names and physical hardware locations. This is the most dependable method to confirm the exact slot of a failed NVMe drive. Together, these steps offer a comprehensive approach to safely and accurately troubleshoot NVMe drive issues in Linux environments. Relying solely on lsblk is risky due to the dynamic renaming caused by PCIe enumeration and drive failures. Instead, combining kernel messages, device serial numbers, and PCIe path mappings ensures administrators correctly identify and replace the faulty drive without guesswork or costly errors.
