As the author of a popular SPV wallet, I wanted to weigh in, in support of the Gavin's 20Mb block proposal.
The best argument I've heard against raising the limit is that we need fee pressure. I agree that fee pressure is the right way to economize on scarce resources. Placing hard limits on block size however is an incredibly disruptive way to go about this, and will severely negatively impact users' experience.