Nope, that is absolutely correct.
However the referee is not an outside agent. Therefore, technically the ball can rebound off the ref and be a goal.
The makes me wonder; can a ball be played off of a linesman (as long as the whole ball doesn't cross the line) in a similar way to how the ball...