virtual machine – Dedicated server Ubuntu 20.04 freezing intermittently

We have 3 servers with a intermittent problem, they freeze every few days (sometimes 25, sometimes 2 (happened today)).

We use these dedicated ones for KVM virtualization, we have multiple machines running on them.

We still haven’t found out the reason, no error log is registered, they simply freeze, they don’t respond in the terminal (When we access via IPMI OVH), they stop responding to the ping, and they come back to work only with a hard rebot.

We have another 70 smaller dedicated ones, with 64GB RAM and Intel CPU, most of them using Ubuntu 20.04, and they never had this problem.

Has anyone here been through this, could anyone help us?

A curious fact was when we installed a new relic agent inside these dedicated ones. They started recording a lot of logs, and the problem intensified, freezing between 3 and 5 hours. We had to restart the dedicated several times a day because of this. We uninstalled the new relic agent the next day.

Dedicated Specs:
CPU: AMD Epyc 7371 – 16c/32t – 3.1GHz/3.8GHz
Memory: 256GB DDR4

name -a
Linux ns570494 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

The last syslog lines before freezes/hard reboot

Mar 16 14:48:48 ns571213 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-144-generic root=UUID=8c38902d-184f-426e-823b-4efd9a188577 ro nomodeset iommu=pt console=tty0 console=ttyS1,115200n8 nvme_core.default_ps_max_latency_us=0 crashkernel=512M-:192M
Mar 16 14:48:48 ns571213 kernel: [    0.000000] Linux version 5.4.0-144-generic (buildd@lcy02-amd64-089) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 (Ubuntu 5.4.0-144.161-generic 5.4.229)
Mar 16 14:39:01 ns571213 CRON[821929]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:38:01 ns571213 CRON[821876]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:37:01 ns571213 CRON[821834]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:36:01 ns571213 CRON[821778]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:35:02 ns571213 CRON[821647]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php >> /var/virtualizor/log/virt_check 2>&1)
Mar 16 14:35:02 ns571213 CRON[821645]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:35:02 ns571213 CRON[821644]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/firewalltest.php > /var/virtualizor/log/firewalltest.log 2>&1)
Mar 16 14:35:02 ns571213 CRON[821642]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 16 14:35:02 ns571213 CRON[821643]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/calculate_bandwidth.php >> /var/virtualizor/log/calculate_bandwidth 2>&1)
Mar 16 14:34:01 ns571213 CRON[821571]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:33:15 ns571213 libvirtd[3177]: End of file while reading data: Input/output error
Mar 16 14:33:01 ns571213 CRON[821503]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:32:11 ns571213 libvirtd[3177]: End of file while reading data: Input/output error
Mar 16 14:32:08 ns571213 libvirtd[3177]: End of file while reading data: Input/output error
Mar 16 14:32:01 ns571213 CRON[821213]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/cronm.php >> /var/virtualizor/log/cronm 2>&1)
Mar 16 14:32:01 ns571213 CRON[821211]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:31:02 ns571213 CRON[821134]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:30:01 ns571213 CRON[820967]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:30:01 ns571213 CRON[820969]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php >> /var/virtualizor/log/virt_check 2>&1)
Mar 16 14:30:01 ns571213 CRON[820968]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/firewalltest.php > /var/virtualizor/log/firewalltest.log 2>&1)
Mar 16 14:30:01 ns571213 CRON[820966]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/calculate_bandwidth.php >> /var/virtualizor/log/calculate_bandwidth 2>&1)
Mar 16 14:29:01 ns571213 CRON[820913]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:28:22 ns571213 kernel: [224068.305734] TCP: request_sock_TCP: Possible SYN flooding on port 5938. Sending cookies.  Check SNMP counters.
Mar 16 14:28:01 ns571213 CRON[820869]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:27:01 ns571213 CRON[820802]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:26:01 ns571213 CRON[820760]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:25:01 ns571213 CRON[820593]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/virt_check.php >> /var/virtualizor/log/virt_check 2>&1)
Mar 16 14:25:01 ns571213 CRON[820590]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 16 14:25:01 ns571213 CRON[820589]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/firewalltest.php > /var/virtualizor/log/firewalltest.log 2>&1)
Mar 16 14:25:01 ns571213 CRON[820588]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:25:01 ns571213 CRON[820587]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/calculate_bandwidth.php >> /var/virtualizor/log/calculate_bandwidth 2>&1)
Mar 16 14:24:11 ns571213 libvirtd[3177]: End of file while reading data: Input/output error
Mar 16 14:24:08 ns571213 libvirtd[3177]: End of file while reading data: Input/output error
Mar 16 14:24:01 ns571213 CRON[820263]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/cronm.php >> /var/virtualizor/log/cronm 2>&1)
Mar 16 14:24:01 ns571213 CRON[820262]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:23:01 ns571213 CRON[820206]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:22:01 ns571213 CRON[820130]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:21:01 ns571213 CRON[820061]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)
Mar 16 14:20:01 ns571213 CRON[819872]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/firewalltest.php > /var/virtualizor/log/firewalltest.log 2>&1)
Mar 16 14:20:01 ns571213 CRON[819876]: (root) CMD (/usr/local/emps/bin/php /usr/local/virtualizor/scripts/powercron.php >> /var/virtualizor/log/powercron 2>&1)