[cfarm-users] cfarm104 maintenance

[Zach van Rijn]

On Mon, 2026-01-05 at 15:13 -0600, Zach van Rijn via cfarm-users
wrote:
> ...
> 
> It is online but not available to users yet. The updates
> changed some default directories so while the original /Users
> data and /Volumes/STORAGE data persist, some additional steps
> are required to get sshd to work with it again. I can do this
> remotely but I don't have time today to figure out the best
> solution and be sure that it won't happen with subsequent
> updates.
> 
> To cfarm admins: you won't have root access to this box
> temporarily; those keys also appear to have been deleted
> during the update so I need to debug that as well.

Turns out upgrading from 12.x to 13.x (or newer) is a giant mess
which causes all sorts of problems for all sorts of reasons.

For example, /root (or wherever the data originally live) was
deleted; this was first discovered when ssh root@ asked for a
password as the /root/.ssh directory (and various scripts) are
now gone. This data doesn't exist on the machine anymore. Sigh.

The lesson here is that /root is silently erased, a good start.

The annoying news is that Wi-Fi appears to be enabled by default
and cannot be disabled until two subsequent reboots after the
update, at which point the GUI toggle and the CLI command will
work again. This is clearly a bug, but presents a risk to anyone
who might be operating in sensitive environments who don't want
radios to be enabled. Prior to upgrade, radios were disabled.

I figured this out only by coincidence. But it's resolved.

The good news is that the default admin user (non-root, but with
sudo privileges) appears to remain untouched, with SSH access,
so it was trivial to restore the necessary keys to /root/.ssh
but I did not have a backup of some other important files.

Admins: the keys are (or should be) properly installed now. 

As far as I can tell, some data were deleted or overwritten,
while others were not. The /etc/motd was preserved, while the
/etc/pf.conf was overwritten and needed to be recreated. I am
not aware of a comprehensive list for such an "upgrade" but I am
documenting all of my observations here as a warning to others.

All Compile Farm user data were preserved but are not usable by
default (more on this later). The symptom is that SSH asks for
passwords, but it's worse. I immediately made a backup of the
/Users directory to an external machine using rsync due to the
limited disk space and to avoid any possible issues.

The external USB disk remains untouched and was not plugged in
during the updates. It continues to mount just fine.

The bad news is that user accounts are not correctly configured
and need to be fixed manually. Here is how that goes.

Note that if we look at /Users, we can see that some directories
are owned by users, and others have a UID only:

...
drwxr-xr-x   5 wolff     admin   160 Oct 18  2022 wolff
drwxr-xr-x   5 wschmidt  admin   160 Oct 18  2022 wschmidt
drwxr-xr-x   8 62073     admin   256 Dec 12 13:17 wwofoz
drwxr-xr-x   5 61886     admin   160 Oct 18  2022 wyp
drwxr-xr-x   9 xaeroxe   admin   288 Feb 23  2024 xaeroxe
drwxr-xr-x   5 xdje42    admin   160 Oct 18  2022 xdje42
...

I cannot find a pattern. Using 'dscl' it is clear that some
accounts do not have a shell set, among other issues, so this
indicates to me that the directory service is not in good shape.

The best course of action may be to delete and recreate all of
the accounts (which does not affect the data); some accounts
were able to be resurrected simply by setting UniqueID, while
others required a full delete+create. I do not know why.

# ls -l /Users/ | awk '{print $9,$3}' | grep -E '^[a-z]' | while
read name uid_or_name; do

    test $uid_or_name = $name && echo skip $name && continue;

    dscl . -delete /Users/${name}
    dscl . -create /Users/${name}
    dscl . -create /Users/${name} UniqueID ${uid_or_name};
    dscl . -create /Users/${name} UserShell /bin/bash
    dscl . -create /Users/${name} PrimaryGroupID 80
    dscl . -create /Users/${name} NFSHomeDirectory
/Users/${name}

done


Users: please try your accounts now. Let me know if there are
any issues in logging in or with permissions. Note that Homebrew
tools are not yet set up for use; I am doing that now.


Zach