I think that is a misdirection on your part. The point of replace-by-fee is to make 0-confirms reliably unreliable. Currently people can "get away" with 0-confirms but it's only because most people arent actively double spending, and when they do it is for higher value targets. Double spend attacks are happening a lot more frequently than is being admitted here, according to Peter from work with various clients.
Like single address reuse, people have gotten used to something which is bad. Generally accepting 0-conf is also a bad idea(tm) and instant confirmation solutions should be sought elsewhere. There are already interesting solutions and concepts: greenaddress for example, and CHECKLOCKTIMEVERIFY micropayment channels for example. Rather than supporting and promoting risky 0-confirms, we need to spend time on better alternative solutions that will work for everyone and not during the honeymoon phase where attackers are fewer.
Here's value-free assessment of the issue here:
1. Zero-conf txs are unsafe.
2. We'd all want to have a safer instant payments solution if possible.
3. As a social artifact, today zeroconf txs happen to work for some people in some situations.
4. Replace-by-fee will break #3 and probably hasten development of #2.
The discussion boils down to whether we should make #2 happen sooner by breaking remnants of #3 sooner.
I personally would rather not break anything, but work as fast as possible on #2 so no matter when and how #3 becomes utterly broken, we have a better solution. This implies that I also don't want to waste time debating with Peter Todd and others. I want to be ready with a working tool when zeroconf completely fails (with that patch or for some other reasons).
TL;DR: those who are against the patch are better off building a decentralized clearing network rather than wasting time on debates. When we have such network, we might all want this patch to be used for all the reasons Peter has already outlined.