Is there a protect firewall filter applied to the mgmt interface on the switches?
Look at the interface configuration that has the ip address you are doing the ssh to as the target. If there is an input filter applied on the interface this can be resricting what ip addresses are allowed to make connections and thus only allowing ssh from the core.