Yes you are correct. After reading more I realied the configuration you have is how it is done on els. This note maybe what is causing the issue:
IRB Interface Limitation in a PVLAN
If your PVLAN includes multiple switches, an issue can occur if the Ethernet switching table is cleared on a switch that does not have an IRB interface. If a Layer 3 packet transits the switch before its destination MAC address is learned again, it is broadcast to all the Layer 3 hosts connected to the PVLAN. Note: Each host device that you want to connect at Layer 3 must be in the same subnet as the IRB interface and use the IP address of the IRB interface as its default gateway address.
Take a look at this artcile specifically the verification outputs and see if they compare to your system