Gloria,

This is a brilliant post! Great job systematizing many of the issues. Quite a lot to chew on & I hope other readers of this list digest the post fully.

Three things come to mind as partial responses:

under:

I'd more simply put it:

Limiting the amount of work that must be done to consider the replacement

We don't particularly care about goal (1) or goal (2), we care about how much it costs to do (1) or (2). And there are scenarios where the (1) or (2) might not be particularly high, but the total work still might be. I can give you some examples to consider if needed. There are also scenarios where (1) and (2) might be high, but the cost is low overall. Therefore it makes sense to be a little more general with what the anti-DoS goal is.

An issue I'd like to toss into the mix is that of iterative / additive batching. E.g., https://bitcoinops.org/en/cardcoins-rbf-batching/

This is where an business puts a txn in the mempool that pays to N users, and as they see additional requests for payouts the update it to N+2, N+2... N+M payouts. This iterative batching can be highly efficient because the number of transactions per business per 10 minutes is 1 (with variable number of outputs).

One issue with this approach today is that because of the feerate rule, if you go from N to N+1 you need to pay 1 sat/byte over the whole txn. Applied M times, and you have to increase fees quadratically for this approach. Therefore the less efficient long-chain of batches model ends up being 'rational' with respect to mempool policy and irrational with respect to "optimally packing blocks with transactions". 

If the absolute fee rule is dropped, but feerate remains, one thing you might see is businesses doing iterative batches with N+2M outputs whereby they drop 2 outputs for every input they add, allowing the iterative batch to always increase the fee-rate but possibly not triggering the quadratic feerate issue since the transaction gets smaller over time.

Another possible solution to this would be to allow relaying "txdiffs" which only require re-relay of signatures + new/modified outputs, and not the entire tx.

I think this iterative batching is pretty desirable to support, and so I'd like to see a RBF model which doesn't make it "unfairly" expensive. 

(I'll spare everyone the details on how CTV batching also solves this, but feel free to ask elsewhere.)

A counterargument to additive batching is that if you instead do non iterative batches every minute, and you have 100 txns that arrive uniformly, you'd end up with 10 batches of size 10 on average. The bulk of the benefit under this model is in the non-batched to batched transition, and the iterative part only saves on space/fees marginally after that point.

A final point is that a verifiable delay function could be used over, e.g., each of the N COutpoints individually to rate-limit transaction replacement. The VDF period can be made shorter / eliminated depending on the feerate increase. E.g., always consider a much higher feerate txn whenever available, for things of equal feerate only consider 1 per minute. A VDF is like proof-of-work that doesn't parallelize, in case you are unfamiliar, so no matter how many computers you have it would take about the same amount of time (you could parallelize across N outputs, of course, but you're still bound minimally to the time it takes to replace 1 output, doing all outputs individually just is the most flexible option). 

Cheers,

Jeremy