If you want a real answer read on.
Your question is relevant to the individual asked, difficulty is relevant. No one else can give you a correct answer (or at least the odds are very slim). However I can tell you what I know from my education and experience.
I am sure many will agree with me and many others wont, but the higher levels of math are not required for "basic" programming (meaning pretty much anything that wouldn't be used by a corporation). That isn't to say you SHOULD NOT take the math courses. As I had to find out my own way, math is an abstraction for logic (just as words are an abstraction for ideas), pure logic is truth, therefore to understand the truth you have to understand the abstraction.
When I first entered college, I had never finished high school and never had gotten into higher math levels than Algebra 1. That isn't to say I don't understand the logic of higher math, programming typically helps that after you've been doing it a while.
Anyway, getting back to the language question. They weren't very hard for ME to learn, they could be different for you. I should tell you though that programming really doesn't stop and end at one specific language, and "true" programmers will never lock themselves into JUST one spefific language (there are exceptions, but few). If you truly want to succeed and become a programmer your goal should be to understand the logic behind programming; the "means to meaning" if you will. The languages will be nothing more than an abstraction to logic once you get past simple syntax (that is to say, the RULES of the language).
Lastly, let me say that the only real way your going to make a mistake (at least, in my own mind), is by NOT taking the course and seeing if you like it. Before I took my first programming course I thought I was stupid and would never amount to anything, and I "knew" I could never finish it. But because of my motivation (or arrogance, whatever), I stuck with it and found something true about myself and found something I truly enjoy doing.
Programming could be for you, or you could just be a codemonkey (that is a person that doesn't create algorithms, just writes and compiles code) or you could be as famous as one of my idols...Linus Torvolds or Richard Stallman. The only one that could possibly know is you.
"We are only bound to the restrictions that we set for ourselves"
-Me (or this may not be me, as I drink alot and it may be something that I heard while having a hangover)