Indeed, in elite sport [well any level really] we coach to focus on the "controlables" - the things we/you can do to affect outcomes - what your opposition do is out of your control, so don't worry about them or give it any thought as it will only drain your limited mental energy - "control the controlables" ....
The act of "not worrying" is itself out of someone's control. So your logic is flawed.