[cfarm-users] cfarm103 maintenance

[Zach van Rijn]

On Thu, 2026-09-17 at 09:31 -0500, Zach van Rijn via cfarm-users
wrote:
> On Fri, 2026-09-11 at 13:08 -0500, Zach van Rijn via cfarm-
> users
> wrote:
> > ...
> 
> 
> It's going to be a few days before I can get to it. I don't
> think it's a data loss problem (it had been working fine but
> after 323 days of uptime it needed a reboot). I'll make a
> clone of the disk offline and go from there.

This took longer than expected. Full restoration of service is
still in progress. Here is the current status:

# dd if=/dev/nvme1n1 bs=1M | gzip -9 > nvme1n1.gz
953869+1 records in
953869+1 records out
1000204886016 bytes (1.0 TB, 932 GiB) copied, 145431 s, 6.9 MB/s

I don't know if there is data loss but the disk probably needs
to be replaced. There were 61 critical errors during backup:

...
[1760632.991777] nvme1n1: I/O Cmd(0x2) @ LBA 12644352, 1024
blocks, I/O Error (sct 0x2 / sc 0x81) MORE 
[1760632.991803] critical medium error, dev nvme1n1, sector
12644352 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 0
[1760697.776828] nvme1n1: I/O Cmd(0x2) @ LBA 13079552, 1024
blocks, I/O Error (sct 0x2 / sc 0x81) MORE 
[1760697.776851] critical medium error, dev nvme1n1, sector
13079552 op 0x0:(READ) flags 0x84700 phys_seg 128 prio class 0
[1760713.375805] nvme1n1: I/O Cmd(0x2) @ LBA 13436928, 1024
blocks, I/O Error (sct 0x2 / sc 0x81) MORE 
[1760713.375832] critical medium error, dev nvme1n1, sector
13436928 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 0
[1760768.109789] nvme nvme1: I/O tag 857 (f359) opcode 0x2 (I/O
Cmd) QID 12 timeout, aborting req_op:READ(0) size:524288
[1760768.109811] nvme nvme1: I/O tag 858 (f35a) opcode 0x2 (I/O
Cmd) QID 12 timeout, aborting req_op:READ(0) size:524288
[1760768.109818] nvme nvme1: I/O tag 859 (f35b) opcode 0x2 (I/O
Cmd) QID 12 timeout, aborting req_op:READ(0) size:524288
[1760768.112326] nvme nvme1: Abort status: 0x0
[1760768.112752] nvme nvme1: Abort status: 0x0
[1760768.113181] nvme nvme1: Abort status: 0x0
[1760795.150662] nvme1n1: I/O Cmd(0x2) @ LBA 14396416, 1024
blocks, I/O Error (sct 0x2 / sc 0x81) MORE 
[1760795.150685] critical medium error, dev nvme1n1, sector
14396416 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 0
...

For now, I will copy the data to another disk available on the
network and then reconfigure the NFS mount to restore access.

Expect another update in a few days. I don't think I will be
able to replace the NVMe for a few months at least, but being
online with a slower disk is better than being totally offline.


Z