Whenever I am heading down early i won't risk it.
Having said that - I have never had my ticket inspected whilst travelling down at that time. i would assume the only problem would be getting through the barrier at Brighton. Perhaps the cheapest single to Brighton online and select to buy collect at any station. Then you will be fine
Alternatively, isn't it only about £6 from anywhere in that zone? So you could buy a ticket?
I imagine lunchtime would be taking the piss a bit, but I've never been questioned previously, and I often catch a train (from Haywards Heath) much earlier than officially permitted for 3pm kick offs.
Its not the price, I refuse to give Southern any more money than is necessary due to their awful service and their rude and somewhat dismissive staff.