For now apps will be fine, but as apps update, their requirements increase. You'd be hard pressed to find an app of worth that works on iOS4 in the app store now. I imagine it won't be long before iOS 5 is no longer supported by the majority of apps.
Several people in the ********official iOS 7******* thread have. A couple say it is noticeably slower on the 4, others say they notice no difference in speed.
Have you tried turning it off and on again. Not messing around with that, sometimes when you update ipods/ipads the battery drains fast until you turn it off and on again.
Is it related to power drain, i.e. have you turned on reduce motion (settings > general > accessibility > reduce motion)...