You have to do the first yellow early enough, when he STARTS taking the piss.
As you observe, it's largely pointless doing it in the last few minutes because by then, that ship has sailed and they know they'll get away with it.
Whatever happened to that six second rule anyway ?