Actually I got it working now! I've reconfigured the reth interfaces and now the failover issue was gone. I can unplug any of the switch now without downtime. I can say that the issue is resolved now. Regarding the ISP, yes it is currently on 1 router only though I plan to add another switch on top of the routers to make it redundant.
Thank you very much for all your help!