For literally gazillions of years pedants have complained that people use language "incorrectly". That is to say they don't use it in quite the same way as the pedants like to use it. And that's fine, because if we didn't have pedants we wouldn't have the tension between progress and conservatism that characterises so much of the history of English language, and our language would literally spiral out of control and become a garbled mishmash of slang, ghetto-speak and missing apostrophes. Or something...
Literally is one of those words that pedants have complained about. Critics argue that the word means "to the letter", so shouldn't be used as a figure of speech. It's a kind of anti-metaphor, if you like. When someone says "When she left me, she literally took a piece of my heart with her" a pedant might jump out of the bushes wagging a finger, responding "But she didn't literally remove part of your heart did she? She didn't actually take a bleeding chunk from your left ventricle and take it away". And you might respond, "Who are you and where did you come from? Put that wagging finger away; it's not even yours".
This piece on the BBC News Magazine's pages offers an enlightening insight into how literally has been used over the years and how it fits into a pattern of semantic shift that we see with many other words. If reading the article itself is too hard and literally going to ruin your day, then have a listen to the interview from Radio 4's Today programme, which is here.
Edited on 19.03.12 to add: Good link here to Polly Curtis in The Guardian discussing this.