[cfarm-users] gcc112: disk errors

[Alexander Monakov]

(Bcc'ed to cfarm-admins@)

Hello,

on gcc112 there are disk errors and new logins are denied. In dmesg there's

kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): Metadata corruption detected at xfs_inode_buf_read_verify+0x84/0x110 [xfs], xfs_inode block 0xbd0a8540
kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): Unmount and run xfs_repair
kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): First 64 bytes of corrupted metadata buffer:
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e000: 46 69 6c 65 50 61 74 68 2e 6d 61 6b 65 64 69 72  FilePath.makedir
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e010: 73 7d 20 73 75 63 63 65 65 64 73 20 77 68 65 6e  s} succeeds when
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e020: 20 63 61 6c 6c 65 64 20 6f 6e 20 61 20 64 69 72   called on a dir
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e030: 65 63 74 6f 72 79 20 74 68 61 74 20 61 6c 72 65  ectory that alre

this "Metadata corruption" message is repeated a few times until

kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): Metadata corruption detected at xfs_inode_buf_read_verify+0x84/0x110 [xfs], xfs_inode block 0xbd0a8540
kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): Unmount and run xfs_repair
kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): First 64 bytes of corrupted metadata buffer:
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e000: 46 69 6c 65 50 61 74 68 2e 6d 61 6b 65 64 69 72  FilePath.makedir
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e010: 73 7d 20 73 75 63 63 65 65 64 73 20 77 68 65 6e  s} succeeds when
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e020: 20 63 61 6c 6c 65 64 20 6f 6e 20 61 20 64 69 72   called on a dir
kern  :alert : [Mon Apr 23 11:32:37 2018] c0000011fe95e030: 65 63 74 6f 72 79 20 74 68 61 74 20 61 6c 72 65  ectory that alre
kern  :alert : [Mon Apr 23 11:32:37 2018] XFS (dm-6): metadata I/O error: block 0xbd0a8540 ("xfs_trans_read_buf_map") error 117 numblks 16
kern  :notice: [Mon Apr 23 11:32:37 2018] XFS (dm-6): xfs_do_force_shutdown(0x1) called from line 370 of file fs/xfs/xfs_trans_buf.c.  Return address = 0xd00000001ce98e9c
kern  :alert : [Mon Apr 23 11:32:38 2018] XFS (dm-6): I/O Error Detected. Shutting down filesystem
kern  :alert : [Mon Apr 23 11:32:38 2018] XFS (dm-6): metadata I/O error: block 0x70271a2a ("xlog_iodone") error 5 numblks 64
kern  :notice: [Mon Apr 23 11:32:38 2018] XFS (dm-6): xfs_do_force_shutdown(0x2) called from line 1203 of file fs/xfs/xfs_log.c.  Return address = 0xd00000001ce83b8c
kern  :alert : [Mon Apr 23 11:32:38 2018] XFS (dm-6): Please umount the filesystem and rectify the problem(s)
kern  :warn  : [Mon Apr 23 11:32:38 2018] XFS (dm-6): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
kern  :warn  : [Mon Apr 23 11:32:38 2018] XFS (dm-6): xfs_log_force: error -5 returned.
kern  :warn  : [Mon Apr 23 11:32:38 2018] XFS (dm-6): xfs_log_force: error -5 returned.
kern  :alert : [Mon Apr 23 11:32:38 2018] XFS (dm-6): metadata I/O error: block 0x70252003 ("xfs_trans_read_buf_map") error 5 numblks 1
kern  :notice: [Mon Apr 23 11:32:38 2018] XFS (dm-6): xfs_do_force_shutdown(0x1) called from line 370 of file fs/xfs/xfs_trans_buf.c.  Return address = 0xd00000001ce98e9c
kern  :warn  : [Mon Apr 23 11:32:38 2018] XFS (dm-6): xfs_log_force: error -5 returned.
kern  :warn  : [Mon Apr 23 11:32:49 2018] XFS (dm-6): xfs_log_force: error -5 returned.
kern  :warn  : [Mon Apr 23 11:33:19 2018] XFS (dm-6): xfs_log_force: error -5 returned.


and after that the "xfs_log_force" error is repeated approx. every 30 seconds.

Alexander