Thank you so much for taking the time to respond!
We've been running nand-mediack a bunch but its been a weak predictor of which ones are going to fail so far. I found a suggestion elsewhere on J-net to run "dd if=/dev/da0 of=/dev/null bs=1m" to chek for errors. So far I've found it almost always errors out, but if you run nand again after it will find new blocks to fix. If I keep running dd followed by nand, eventually dd will be clean... but I still had boxes break in that scenario so it doesn't seem worth the time.
One ridiculous thing I tried was forcibly unmounting /var (not in single user) and cleaning it with fsck. Junos was hilariously confused of course:
me@device> show interfaces descriptions mgd: error: could not open database schema: /var/run/db/schema.db mgd: error: could not open database schema
...but I was able to reboot from shell and access remotely. The attempt to upgrade failed with:
panic: ufs_dirbad: /var: bad dir ino 12 at offset 0: mangled entry
On the reboot it successfully booted from the other slice (old version) and showed the new version on the primary. Rebooted again from the alternate ("request system reboot slice alternate media internal") and it was good! Which was great, and did not require console.. but not really a process I'm going to trust, especially at the scale in question.
Yesterday I tried setting some fsck options in rc.conf, but I don't really know what I'm doing in BSD at all and I honestly can't tell if it did anything or not.
Ultimately I think I have come to the same conclusions as you: fsck is the best indicator of trouble and I really wish it "nuked and repaved" /var and /var/tmp on upgrade.
Thanks so much again for the detailed response, I was assuming it was going to be one of these situations: https://xkcd.com/979/.
Bryon