After 20 hours I still don't understand how manual aiming works

Yes, the idle animation impacts where the shot lands. Not only that, but the target continues to animate in slow motion after the shot is fired.

Anyway, the system is actually quite involved. If you are interested, it is covered extensively in the guide I was working on, with explanations from other players and the devs: