Hello,
This firewall filter is stateless filter. So it will be applicable to not only traffic originating from source addresses specified but also to all reply packets from those IPs even if originator is anybody.
This could be the reason why it is dropping the unrelated connections as well.
Regards,
Rushi