I seem to have the type of luck that I encounter odd uncommon issues. I'm sure there is a logical explanation for the things I encounter but sometimes you just have to shrug it off and move on....
One of the things I did try was making EX side passive. That did not fix the issue in this case.
Oddly enough what I did was unplug the other side and plug it back in. So the pair of QFX there is an A and a B. I was unplugging B and loosing connectivety. I unplugged A and plugged it back in and then I was able to unplug B without loosing connectivety.
So for whatever reason this is now working as it should.....