> The block size itself should be set based on the amount of fees being paid to miners to make a block.
There's a formula to this as well, though going from that to a blocksize number will be very difficult. Miner fees need to be sufficient to maintain economic protection against attackers. There is no reason that miner fees need to be any higher than "sufficient." I believe that "sufficient" value can be estimated by considering a potential attacker seeking to profit from short-selling Bitcoin after causing a panic crash. If they can earn more profit from shorting Bitcoin than it costs to buy, build/deploy, and perform a 51% attack to shut down the network, then we are clearly vulnerable. The equation for the profit side of the equation can be worked out as:
(bitcoin_price * num_coins_shortable * panic_price_drop_percentage)
The equation for the cost side of the equation depends on the total amount of miner hardware that the network is sustainably paying to operate, factoring in all costs of the entire bitcoin mining lifecycle(HW cost, deployment cost, maintenance cost, electricity, amortized facilities cost, business overheads, orphan losses, etc) except chip design, which the attacker may be able to take advantage of for free. For convenience I'm simplifying that complicated cost down to a single number I'm calling "hardware_lifespan" although the concept is slightly more involved than that.
(total_miner_payouts * bitcoin_price * hardware_lifespan)
Bitcoin_price is on boths ides of the equation and so can be divided out, giving:
Unsafe point = (num_coins_shortable * panic_price_drop_percentage) < (total_miner_payouts * hardware_lifespan)
Estimating the total number of shortable coins an attacker of nearly unlimited funds is tricky, especially when things like high leverage levels or naked short selling may be offered by exchanges. The percent of damage the resulting panic would cause is also tricky to estimate, but on both numbers we can make some rough guesses and see how they play out. With more conservative numbers like say, 2 year hardware lifespan, 10% short, 70% panic drop you get: 1,300k coins profit, 1800 BTC/day in fees minimum needed to make the attack cost more than it profits.
Using various inputs and erring on the side of caution, I get a minimum BTC/day fee range of 500-2000. Unfortunately if the blocksize isn't increased, a relatively small number of transactions/users have to bear the full cost of the minimum fees, over time increasing the minimum "safe" average fee paid to 0.008 BTC, 30x the fees people are complaining about today, and increasing in real-world terms as price increases. All that said, I believe the costs for node operation are the number that gets hit first as blocksizes are increased, at least past 2020. I don't think blocksizes could be increased to such a size that the insufficient-fee vulnerability would be a bigger concern than high node operational costs. The main thing I don't have a good grasp on at the moment is any math to estimate how many nodes we need to protect against the attacks that can come from having few nodes, or even a clear understanding of what those attacks are.
> A block so big that 100% of the transactions will always be mined in the
> next block will just cause a large section of people to no longer feel the
> need to pay fees.
This is also totally true. A system that tried to eliminate the fee markets would be flawed, and fortunately miners have significant reasons to oppose such a system.
The reverse is also a problem - If miners as a large group sought to lower blocksizes to force fee markets higher, that could be a problem. I don't have solutions for the issue at this time, but something I've turned over in my mind.