To be fair, if that is how it happened, sounds like he didn't do anything wrong. We've all made jovial comments like that. Those of us that are commuters will know bits of humour like that are needed to remain sane!
If a bus driver told me to get the back of a queue for no reason, I'd have...