You can try to apply per-vlan l2 filter to explicitly define allowed trafic.
It's an overkill... use it if you want to use common vlan for all users by any means.
https://www.juniper.net/documentation/en_US/junos12.3/topics/example/firewall-filter-ex-series-configuring.html
So, you can define filters based on src/dst mac/IP addresses etc (see you switch platform capability).
And you can apply input/output filters to a port or vlan.
If I remember port filters have higher priority than vlan based filters...
Basically L2 filter on user's vlan should be OK if you exactly know router and server mac addresses in user's vlan.
# Define filter
edit firewall family ethernet-switching filter myfilter
set term any-to-server from destination-mac-address 00:11:22:33:44:55/48
set term any-to-server then accept
set term server-to-any from source-mac-address 00:11:22:33:44:55/48
set term server-to-any then accept
set term router-to-any from source-mac-address 00:11:22:33:44:01/48
set term router-to-any then accept
set term any-to-router from destination-mac-address 00:11:22:33:44:01/48
set term any-to-router then accept
set term no-src-broadcast from source-mac-address FF:FF:FF:FF:FF:FF/48
set term no-src-broadcast then discard
set term any-to-broadcast from destination-mac-address FF:FF:FF:FF:FF:FF/48
set term any-to-broadcast then accept
# Apply filter to user's vlan
set vlan DATA filter input myfilter
# secure and bind router/server mac addresses to their interfaces if router and server are separate entities (your swich is not a router itself, no L3 vlan interface )
set ethernet-switching-options static vlan DATA mac 00:11:22:33:44:01 next-hop ge-0/0/0.0
set ethernet-switching-options static vlan DATA mac 00:11:22:33:44:55 next-hop ge-0/0/1.0
Alternatively you can try and plug IP phones to separate swich ports with voip vlan set.
And then you can use private vlan feature on ports with no ip phones attached.