@Russell: Appreciate the historical note, but as that op code was simultaneously disabled in that patch I don't think we can look back to how it was non-functionally changed (that number means nothing... maybe Satoshi was trying it out with 520 bytes but then just decided to all-out disable it and accidentally included that code change? Hard to say what the intent was.).
@Jorge:
That's one part of it that is worth hesitation and consideration. I'm not a fan of the 520 byte limit as well. My gut feeling is that the "right" answer is to compute the memory weight of the entire stack before/after each operation and reasonably bound it.
Below text is from the chain core documentation:
"""
Most instructions use only the data stack by removing some items and then placing some items back on the stack. For these operations, we define the standard memory cost applied as follows:
Instruction’s memory cost value is set to zero.
For each item removed from the data stack, instruction’s memory cost is decreased by 8+L where L is the length of the item in bytes.
For each item added to the data stack the cost is increased by 8+L where L is the length of the item in bytes.
----
Every instruction has a cost that affects VM run limit. Total instruction cost consists of execution costand memory cost. Execution cost always reduces remaining run limit, while memory usage cost can be refunded (increasing the run limit) when previously used memory is released during VM execution.
"""
Is there a reason to favor one approach over the other? I think one reason to favor a direct limit on op_cat is it favors what
I'll dub "context free" analysis, where the performance doesn't depend on what else is on the stack (perhaps by passing very large arguments to a script you can cause bad behavior with a general memory limit?). On the other hand, the reason I prefer the general memory limit is it solves the problem for all future memory-risky opcodes (or present day memory risks!). Further, OP_CAT is also a bit leaky, in that you could be catting onto a passed in large string. The chief argument I'm aware of against a general memory limit argument is that it is tricky to make a non-implementation dependent memory limit (e.g., can't just call DynamicMemoryUsage on the stack), but I don't think this is a strong argument for several (semi-obvious? I can go into them if need be) reasons.