This is such an important point. If you read the comics and interviews, we find out that Serenity takes place roughly 6 months after the end of Firefly. What a six months it has been. Remember all those botched jobs they had in the series? It's gotten worse. A lot worse. We can see this in the movie from the fact that Book and Inara have BOTH left, and not far into Serenity, River and the good Doc Simon are soon to follow.
This is not because Joss rewrote Mal's character. It's been a hard six months. Not a normal person's hard, but Serenity hard. I think we can all appreciate just how bad that must be. Mal is on the edge just trying to survive. "Don't push me and I won't push back," he tells Simon during the movie. This is not the good times of Firefly. Things are falling apart at the seams.
The signs are everywhere in the movie: as they are landing, when Mal says "this time I think it really will be it" (referring to crashing), when the twins up their cut from 25 to 40% and go into their speach about running when he should fight and fighting when he should settle. This isn't just happening with one contact, but MOST of them. Mal speaks of all the jobs, even HONEST ones, that they can't take because of the upped pressure from the Alliance to find River and Simon. To sum it up, Mal is on the edge, has lost sight of his beliefs because he's trying just to hang on. The entire movie is really about Mal finding his faith again, and getting back on the right side of that ledge he's frittering on.