AIX: Recovery from Reference Code 551, 555, or 557 in PowerVM VIOS

VIOS fails to boot up with reference code 551, 555, or 557

1. A corrupted filesystem in VIOS rootvg
2. A corrupted journaled file system (JFS) log device in rootvg
3. Hardware component failure (i.e. disk, adapter, etc) related to VIOS rootvg

PowerVM VIOS 2.2 or higher

Login to the VIOS management GUI (i.e. HMC, FSM, etc), select the VIOS partition and look at the Reference Code.

• 551 indicates VIOS rootvg activation (varyon)
• 555 means fsck error, fsck failure of / (root) filesystem, /dev/hd4
• 557 means failure to mount / (root) filesystem

To diagnose and attempt to fix any of these reference codes, you must boot the VIOS partition into maintenance mode to run a thorough file system check (fsck) and reformat the jfs2log, /dev/hd8.

1. Boot VIOS partition to maintenance mode and choose the following option:
Access this volume group and start a shell before mounting the filesystems

(option 2)

This option will attempt to import the VIOS rootvg in a limited shell (with no filesystems mounted). Pay close attention to the output as this is the point where I/O errors are commonly seen when the issue is due to an underlying hardware problem related to rootvg, i.e.
Fatal: I/O error

If you get errors at this point indicating that a physical volume is missing from the

rootvg

or errors suggesting a hardware component failure, such as the “I/O error”, do not continue with the rest of the steps in this document. Run diagnostics or contact your local Hardware Support Representative.

2. Format the default jfs2log for the VIOS rootvg if the volume group is successfully imported:
/usr/sbin/logform /dev/hd8Answer yes when asked if you want to destroy the log.
3. Next, run fsck to check and repair file system corruption. (The -y option gives fsck permission to repair file systems when necessary.)
fsck -y /dev/hd4           / (this is the root filesysem)
fsck -y /dev/hd2           /usr
fsck -y /dev/hd9var        /var
fsck -y /dev/hd3           /tmp
fsck -y /dev/hd1           /home
fsck -y /dev/hd10opt       /opt
fsck -y /dev/hd11admin     /admin
fsck -y /dev/livedump      /var/adm/ras/livedump
4. Type   exit .  The file systems will automatically mount after you type exit.

5. If the VIOS is hosting AIX clients’ rootvg on virtual SCSI disk backed by an hdisk, you can verify at this point that the volume group selected earlier (based on Volume Group ID) corresponds to the VIOS rather than a AIX client rootvg. A good way to do this, is by running:
/usr/ios/cli/ioscli ioslevelThe command should return the expected ioslevel installed on disk.
6. At this point, proceed to reboot the VIOS
shutdown -Fr