[cfarm-users] cfarm103 maintenance

[Zach van Rijn]

On Fri, 2026-09-11 at 13:08 -0500, Zach van Rijn via cfarm-users
wrote:
> On Thu, 2026-09-10 at 05:33 -0500, Zach van Rijn via cfarm-
> users
> wrote:
> > The cfarm103 Mac Mini Linux machine has been updated and
> > rebooted but failed to come back online this time.
> > 
> > I will look at it in the next day or so.
> 
> It's online now. There is some kind of race condition with the
> startup scripts, so rebooting it twice (manually) is what was
> required. I tested it two more times for confirmation.

Unrelated to cfarm103 itself, but to the NFS /home, I rebooted
that host today for unrelated reasons and the NVMe that is
dedicated to cfarm103 and served via NFS is in a sad state:

...
[   66.564239] nvme nvme1: I/O tag 200 (00c8) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:122880
[   66.564276] nvme nvme1: I/O tag 201 (00c9) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:8192  
[   66.564283] nvme nvme1: I/O tag 203 (00cb) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:4096
[   66.564290] nvme nvme1: I/O tag 634 (127a) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:106496
[   66.564297] nvme nvme1: I/O tag 635 (127b) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:24576  
[   66.564303] nvme nvme1: I/O tag 638 (127e) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:4096
[   66.564310] nvme nvme1: I/O tag 835 (0343) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:12288
[   66.564316] nvme nvme1: I/O tag 836 (0344) opcode 0x2 (I/O
Cmd) QID 11 timeout, aborting req_op:READ(0) size:4096
[   97.287923] nvme nvme1: I/O tag 200 (00c8) opcode 0x2 (I/O
Cmd) QID 11 timeout, reset controller
[  103.948703] nvme nvme1: Abort status: 0x0
[  103.949417] nvme nvme1: Abort status: 0x0                   
[  103.952010] nvme nvme1: Abort status: 0x0                   
[  103.955039] nvme nvme1: Abort status: 0x0                   
[  103.955776] nvme nvme1: Abort status: 0x0                   
[  103.958381] nvme nvme1: Abort status: 0x0                   
[  103.960269] nvme nvme1: Abort status: 0x0                   
[  103.961442] nvme nvme1: Abort status: 0x0                   
[  104.264752] nvme nvme1: D3 entry latency set to 8 seconds
...


It's going to be a few days before I can get to it. I don't
think it's a data loss problem (it had been working fine but
after 323 days of uptime it needed a reboot). I'll make a clone
of the disk offline and go from there.

And the cost of NVMe disks these days... $200 USD? I think I
bought that one for $50. Wow.

Reminder that Compile Farm is not responsible for your data.


Zach