This can be simplified if we do get a fee pool (less hardfork code, more softfork code), except that the effect will be partially reduced by the mining subsidy until it approximately reaches parity with average total fees.
We don't need to alter difficulty calculation.
Instead we alter the percentage of the fees that the miner gets to claim VS what he have to donate to the pool based on the size of the block he generated.
Larger block = smaller percentage of fees. This is another way to pay for blocksize. The effect of this is that on average, miners that generate smaller blocks takes a share of what otherwise would be part of the mining profits of those generating larger blocks.
We would need to keep pieces of the section from above on expected blocksize calculation. Because the closer you are to the expected blocksize, the more you keep. And thus we need to adjust it according to usage.