Seems the place to start is to verify the utilization on the ISP 100m uplink. If this really is maxing out, then this is likely the root cause of all the drops you are seeing. Are you collecting and graphing the bandwidth utilization?
If not, your ISP may be collecting that data for you in their portal.
If this proves to be the case, you can then look into putting downstream policers in place to limit bandwidth from particular VLANs in order to reserve space for those critical applications.