Full Root FS on ESXi due to iLOREST logfile

This really annoying issue was hunting me for several weeks until I discovered the root cause. One of my customers is running VMware ESXi on top of HPE ProLiant DX hardware, the customized Hardware from HPE for Nutanix. It’s simply a ProLiant DL with a specific set of available components, firmware, drivers and branding. Instead of running AHV, this customer chose to run VMware ESXi as hypervisor. Everything was running fine until the customer reported reocurring fails of a specific Nutanix Cluster Check, in this case the ‘host_disk_usage_check’. While investiagting the issue, I noticed that the root filesystem on all nodes of the clsuter was full.

nutanix@NTNX-CZJDXXXXX6-A:10.xxx.xxx.10:~$ hostssh "vdf -h | grep -i root"
============= 10.xxx.xxx.10 ============
root                       32M       32M        0B 100% --
============= 10.xxx.xxx.12 ============
root                       32M       32M        0B 100% --
============= 10.xxx.xxx.13 ============
root                       32M       32M        0B 100% --
============= 10.xxx.xxx.11 ============
root                       32M       32M       0B 100% --
============= 10.xxx.xxx.14 ============
root                       32M       32M        0B 100% --
nutanix@NTNX-CZJDXXXXX6-A:10.xxx.xxx.10:~$

Unable to identify any large file or directory on the fist spot, I restarted a single host and noticed that the issue was gone for the host. So I did a rolling restart of all hosts and called it a day. Several weeks later, the issue reappeared - and to be honest - the issue reappeared the next day, but the customer didn’t noticed it. Everything works normal.

After some more investigation, I found a large logfile called iLORest.log in the root filesystem, and I found it on the other nodes as well. So this was not a Nutanix or VMware issue, but an issue with the HPE iLO RESTful API.

Long story short: It’s an issue which is fixed with SPP 2025.09.00.00, but if you are running ProLiant DX, you should make sure that the update is supported. As a workaround, you can do the following:

  1. SSH to each ESXi host
  2. Execute the following command to log out of any active iLO sessions: esxcli ilorest cmd -q logout
  3. Delete the log file: rm /iLOREST.log
  4. If the space is not freed up, you may need to kill the ilorest.sh and corrosponding Python3 processes (lsof | grep '[l]ibilorestchif.so' | awk '{print $1}' | xargs kill and/ or lsof | grep '[i]lorest.sh' | awk '{print $1}' | xargs kill)
  5. Check if the space is freed up: vdf -h | grep -i root

As per the following HPE link, a new iLORest optional argument to omit the creation of an info log file is available in version 6.3.0.0.