[cfarm-users] cfarm427 and cfarm430 crashing

[Luke Yasuda]

On 2025-06-20 14:19, Pierre Muller via cfarm-users wrote:
> Hi Luke,
> 
> Le 18/06/2025 à 16:05, Luke Yasuda via cfarm-users a écrit :
>> Hi!
>> 
>> I've noticed that cfarm427 and cfarm430 (both FreeBSD) have crashed
>> (kernel panic) 2~3 times this month. Anyone has any idea why this
>> happens?
>> pid 56584 (tissurrogatepair2), jid 0, uid 61083, was killed: exceeded
>> maximum CPU limit
>> pid 68777 (theapthread), jid 0, uid 61083, was killed: exceeded 
>> maximum
>> CPU limit
>> pid 70078 (tissurrogatepair2), jid 0, uid 61083, was killed: exceeded
>> maximum CPU limit
>> -----------8<------cut-here------8<---------
> 
>   These d-messages are all generated by the nightly Free Pascal 
> testsuite that I am running on these two machines.
> 
> Some off these tests are really lengthy, and this is the reason why 
> they are launched with an explicit
> CPU limit limitation using 'ulimit -t 60' to limit
> each individual test for consume 60 seconds CPU time at most.
> 
>   Is it a problem that these interrupted tests generate a dmesg entry?

Oh, that's what it was. It's not a problem to me, but if a test takes 
too long and eventually will be killed, maybe it should've been disabled 
(not run at all) in the first place?

>   I didn't find any 'panic' containing lines on gcc427 inside 
> /var/log/messages file.

Me neither, it seems that cfarm427 could not log the panic to the file 
system. But the panic did happen. I saw it on the serial console.

>   I was not able to connect to gcc430 this morning,
> and couldn't check the recent content of that machine.

Yes...since I wrote the email, cfarm427 crashed once and cfarm430 
crashed twice.

>   Do you suspect a link between the CPU exceeded time messages and the 
> crashes you are reporting?
> Should I disable the testsuite on gcc430?
> 
>   I sincerely hope that the troubles are not due
> to the scripts that I run on these test machines.

I hope not too, I don't believe the free pascal test suites are the 
cause of those kernel crashes. You don't have to disable all of them 
(maybe just those very lengthy ones as you can tell).

> Thank you again for providing these configurations to the community!

It's my solemn duty to share my unused computing resources.


On 2025-06-19 11:08, Jacob Bachmeyer wrote:
> On 6/18/25 09:05, Luke Yasuda via cfarm-users wrote:
>> Hi!
>> 
>> I've noticed that cfarm427 and cfarm430 (both FreeBSD) have crashed 
>> (kernel panic) 2~3 times this month. Anyone has any idea why this 
>> happens?
> 
> I am less familiar with FreeBSD, but that *looks* like some kind of 
> rogue TCP/IPv6 packets are causing crashes in the network code.
> 
> It would be very interesting if you could catch the packets that are 
> causing this (you might need another box sniffing a mirrored switch 
> port with either an unconfigured NIC "raw" or a different (Linux?) 
> kernel) and identify what is going wrong.  This *looks* like a remotely 
> exploitable DoS in FreeBSD.  I hope it is happening by accident.
> 
> A search for 'FreeBSD tcp_do_segment "sent too much"' in search of 
> source code led to 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282605 where it seems 
> that there have been similar bugs.  I suggest opening a bug report at 
> FreeBSD.  It also looks like FreeBSD has features that can include 
> recently-received packets in crash dumps, which solves the need for 
> another box to catch the network traffic.

I have made 0 efforts to investigate this, unfortunately. Life is 
keeping me from doing so.

But fwiw, from the bug report you mentioned, a patch was committed, so I 
(re)built world and kernel from HEAD. It had not been upgraded for about 
6+ months. We'll see.


On 2025-06-19 11:27, Collin Funk wrote:
> Hi Luke,
> 
> Luke Yasuda via cfarm-users <cfarm-users at lists.tetaneutral.net> writes:
> 
>> I've noticed that cfarm427 and cfarm430 (both FreeBSD) have crashed
>> (kernel panic) 2~3 times this month. Anyone has any idea why this
>> happens?
>> [...]
>> Jun 18 13:13:34 cfarm430 kernel: panic: tcp_do_segment: sent too much
> 
> I agree with Jacob that this issue will probably require a FreeBSD bug
> report. I see that the condition that prints this message was changed 
> ~2
> months ago, perhaps that caused it [1]. Since FreeBSD 15.0 is an 
> upcoming
> release, I assume that change is in your kernel. But not sure how to
> confirm.

I had not upgraded it since maybe October 2024, but now the patch should 
be in there. We'll see.

>> Also a tiny personal news/announcement: effective Jun 6, my legal name
>> has been changed, because I'm now a naturalized Japanese citizen.
>> "Jing Luo", my birth name, is a compatibility symlink and is
>> deprecated and unsupported; please migrate to "Lucas Yasuda" or
>> shorten to "Luke" if you are in a hurry.
> 
> Congratulations on your newly acquired citizenship.

Thanks! I forgot to say that I also updated my gpg key.

> Collin
> 
> [1] 
> https://github.com/freebsd/freebsd-src/commit/934caaec3afc43638c2a1da8fbe3b672566db4fe


Thanks all!

-- 
Luke Yasuda
About me: https://jing.rocks/about/
GPG Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: OpenPGP digital signature
URL: <http://lists.tetaneutral.net/pipermail/cfarm-users/attachments/20250623/47f92a60/attachment.sig>