I have a lot of trouble saying no to people, too, so I know what you're talking about, reychop. I know it's tough, but probably the best thing would be to just say you've thought about it, but it isn't what you think is best for yourself. It's gentle but to the point and if they press you for an explanation, you don't have to give one.
In a way, my roommate last year put me in similar situations trying to hook me up with people for one night stands and I had to learn how to tell her no, fast. I knew I couldn't do that and keep my sanity, so after a while of backing out at the last minute it became easier to just tell her no up front. It was hard, but once I started doing it I found a power in myself and it's easier to stand up for myself now. Hopefully the same will be for you. The hardest time is the first, so if you manage to say no this time it'll be easier in the future.