Ah yes, I forgot about fog of war, that should solve the problem.
I gave this one some some thought as well. My opinion is, both overwatch and return fire should be customizable in a way which takes your idea of effective fire and takes it a step further. Firstly, when setting overwatch, there should be a way to specify the zone to cover, i.e 160 deg forward, 90 deg to the left etc. I also think that to distinguish overwatch form return fire, it should have a limit on the zone a soldier can cover, say no more than 180 deg. Return fire will have no such limit(i.e. as long as a solder is being shot at, he can return fire).
Secondly, and here is where your idea comes in, an engagement range should be specified. By entering either a max engagement range or minimal chance to hit, players will be able to determine when to fire, avoiding the situation when your soldiers will fire at an enemy which just entered the field of vision but is so far away that taking pot shots at it will be nothing but a waste of bullets. Instead it should be possible to ignore distant targets entirely and wait for a closing enemy to get into a range for a more reliable shot rather than shooting it right as it appeared in the distance.