I get the fun in having random weapon specialization perks. What I don’t understand is why they have these bonuses (and penalties) attached. That’s pretty much saying: “Yeah, you have skill trees for specific weapons, but you can be even more efficient with those weapons if you happen to get a random perk. No need to even be of the correct class.”
What I’d personally prefer is removing the bonuses&penalties, then reducing the SP cost of an extra weapon specialization (to maybe 10 SP, feel free to balance it better). Now discard those penalties completely, and work the bonuses in the correct skill trees somewhere.
The result would be like saying: “Sure, you can learn how to handle a different weapon. If you want to truly excel in its use, however, you should dual-class into the correct skill tree.”
Edit: If this was a Star Wars game, you could always just leave it as is and call those perks dormant Force powers. I mean, a Jedi or a to-be-Jedi often is better at flying, shooting, or whatever, than those who have trained for it their whole life.