Actually in your case it is a upgrade (from 12.3R3 to 12.3R4) as well as a downgrade (12.3R6 to 12.3R4). Now you can set-up the VC to automatically do this. See:
You could disconnect these switches from the VC upgrade a standalones, and then perform the package add. If possible I might suggest you upgrade the whole VC to 12.3R[latest] which right now I believe this is R12.
A reset/reboot of the VC is user affecting, as is the same for a standalone, if done why in standalone mode.