I recently encountered an error on one of my ESXi 4.1 lab hosts trying to make a configuration change to the host. The error image is shown below and the text of the error stack pointed to full root file system:
Call "HostNetworkSystem.UpdateNetworkConfig" for object "networkSystem" on ESXi "esx01" failed.
Operation failed, diagnostics report: Error interacting with configuration file /etc/vmware/esx.conf:
Write failed during Unlock. This is likely due to a full or read-only filesystem.
Original Error was: Error interacting with configuration file /etc/vmware/esx.conf:
Unable to write to file /etc/vmware/esx.conf.vVPNWW while saving /etc/vmware/esx.conf operation aborted.
It is likely this was caused by a Full Disk.
I used the command vdf -h to confirm the error message and as shown in the below output.
~ # vdf -h
Tardisk Space Used
SYS1 201M 201M
SYS2 55M 55M
SYS3 1M 1M
SYS4 12K 12K
SYS5 12K 12K
Ramdisk Size Used Available Use% Mounted on
MAINSYS 32M 32M 0B 100% --
tmp 192M 112K 191M 0% --
updatestg 750M 64K 749M 0% --
hostdstats 78M 3M 74M 4% --
I used the find command to determine that the problem files were sel and sel.raw in the folder /var/log/ipmi/0. I deleted the files, but after a few minutes, the files filled the MAINSYS ramdisk again. I found that the file sel was completely made up of the repeating entry System Boot: and sel.raw was made up of ef ef …
These files are generated from System Event Log IPMI data from the host so I accessed the host’s management interface. Oddly I found that the SEL was empty. However on a whim a clicked the Clear Event Log link, deleted the sel and sel.raw files and the files did not reappear. As part of the troubleshooting for this I did reset the host’s ESXi configuration and reinstalled ESXi as well but these actions did not help.