Do you leave it running for extended periods of time? Firefox is notorious for memory leaks, so while it might not use a whole lot of memory right at start-up over time it will keep using more and more and more, even if you're not really doing anything that should be using that much.
I do mostly use Firefox for porn browsing, but that's because Chrome doesn't seem to do as well as FF when you have 13 million tabs open. For normal browsing though I prefer Chrome.