I'd go along with that. You weren't trying to diddle anyone, it sounds like an honest mistake. I think you need to resign yourself to having to pay back the money, perhaps ask them for advice in how to go about it during the interview given the stop start nature of your work.
Good luck, and try...