Not figured out anything automatic: If you have file system damage, you
really do have to go visit the switch, connect via console, and boot into
single user mode to manually run fsck -y until it comes up clean. In these
cases running "nand-mediack" is probably a good idea also. (This can run in
regular mode, not single user)
But:
We found that EX3300 and EX4500 have a known bug where they don't boot to
single user mode in ver 15.1R5 or 15.1R6; instead they just reboot. Need to
load, e.g., ver 14.x on the second slice, and then after the first attempt
at single user mode fails, it will boot to the other slice, and boot -s
works there. (I say "known bug", but I had a case open for about a month
before Juniper tech was able to tell me that this was so!)
Also:
I found that even on a live system, in most cases, running fsck will
complete without errors. (Depending how busy the file system is, sometimes
it reports a minor error, but if you re-run it, it will come up clean.)
I have found that systems that give zero fsck errors on all their members
can reliably run firmware updates remotely and reboot successfully.
However, for EX3300 or EX4200 stacks that do show persistent fsck errors in
"read only" mode, you really need to visit them and boot the members with
errors to single user mode and run fsck to clean up the errors before you
try doing a firmware update.
Finally:
Maybe common knowledge, but in some cases fsck will complain about . or ..
being bad, and say it can't fix the error. The error report is similar to
"UNEXPECTED SOFT UPDATE INCONSISTENCY
CANNOT FIX, FIRST ENTRY IN DIRECTORY CONTAINS lost+found" -- in this
specific case, the directory "lost+found" is in the way of ".", which needs
to be in the first spot.
Fix (in single user mode) is mount the disk, then "ls -fa" to see
everything in disk order, and then move the problem dir out of the way (mv
lost+found lost+foundX), then umount, and run fsck again. Without anything
in the way, it can re-create . or .. Then you can mv the problem dir back
to its proper name.
Hope this is some use, but it would be way better if firmware updates just
automatically nuked and repaved the /var and /var/tmp folders.
--
Steve Bohrer
IT Infrastructure, Emerson College
617-824-8523
really do have to go visit the switch, connect via console, and boot into
single user mode to manually run fsck -y until it comes up clean. In these
cases running "nand-mediack" is probably a good idea also. (This can run in
regular mode, not single user)
But:
We found that EX3300 and EX4500 have a known bug where they don't boot to
single user mode in ver 15.1R5 or 15.1R6; instead they just reboot. Need to
load, e.g., ver 14.x on the second slice, and then after the first attempt
at single user mode fails, it will boot to the other slice, and boot -s
works there. (I say "known bug", but I had a case open for about a month
before Juniper tech was able to tell me that this was so!)
Also:
I found that even on a live system, in most cases, running fsck will
complete without errors. (Depending how busy the file system is, sometimes
it reports a minor error, but if you re-run it, it will come up clean.)
I have found that systems that give zero fsck errors on all their members
can reliably run firmware updates remotely and reboot successfully.
However, for EX3300 or EX4200 stacks that do show persistent fsck errors in
"read only" mode, you really need to visit them and boot the members with
errors to single user mode and run fsck to clean up the errors before you
try doing a firmware update.
Finally:
Maybe common knowledge, but in some cases fsck will complain about . or ..
being bad, and say it can't fix the error. The error report is similar to
"UNEXPECTED SOFT UPDATE INCONSISTENCY
CANNOT FIX, FIRST ENTRY IN DIRECTORY CONTAINS lost+found" -- in this
specific case, the directory "lost+found" is in the way of ".", which needs
to be in the first spot.
Fix (in single user mode) is mount the disk, then "ls -fa" to see
everything in disk order, and then move the problem dir out of the way (mv
lost+found lost+foundX), then umount, and run fsck again. Without anything
in the way, it can re-create . or .. Then you can mv the problem dir back
to its proper name.
Hope this is some use, but it would be way better if firmware updates just
automatically nuked and repaved the /var and /var/tmp folders.
--
Steve Bohrer
IT Infrastructure, Emerson College
617-824-8523