It is an argument against my admittedly vague definition of
"non-controversial change".

If it's an argument against something you said, it's not a straw man, right ;)

Consensus has to be defined as agreement between a group of people. Who are those people? If you don't know, it's impossible to decide when there is consensus or not.

Right now there is this nice warm fuzzy notion that decisions in Bitcoin Core are made by consensus. "Controversial" changes are avoided. I am trying to show you that this is just marketing. Nobody can define what these terms even mean. It would be more accurate to say decisions are vetoed by whoever shows up and complains enough, regardless of technical merit. After all, my own getutxo change was merged after a lot of technical debate (and trolling) ..... then unmerged a day later because "it's a shitstorm".

So if Gavin showed up and complained a lot about side chains or whatever, what you're saying is, oh that's different. We'd ignore him. But when someone else complains about a change they don't like, that's OK.

Heck, I could easily come up with a dozen reasons to object to almost any change, if I felt like it. Would I then be considered not a part of the consensus because that'd be convenient?
 
I'm sure that's not what the proponents of the size increase want, and
I'm not defending 1 MB as a sacred limit  or anything, but my question
is "where is the limit for them?"

20mb is an arbitrary number, just like 1mb. It's good enough to keep the Bitcoin ecosystem operating as it presently does: gentle growth in usage with the technology that exists and is implemented. Gavin has discussed in his blog why he chose 20mb, I think. It's the result of some estimates based on average network/hardware capabilities.

Perhaps one day 20mb will not be enough. Perhaps then the limit will be raised again, if there is sufficient demand.

You are correct that "no limit at all" is a possible answer. More precisely, in that case miners would choose. Gavin's original proposal was 20mb+X where X is decided by some incrementing formula over time, chosen to approximate expected improvements in hardware and software. That was cool too. The 20mb figure and the formula were an attempt to address the concerns of people who are worried about the block size increase:  a meet-in-the-middle compromise.

Unfortunately it's hard to know what other kinds of meet-in-the-middle compromise could be made here. I'm sure Gavin would consider them if he knew. But the concerns provided are too vague to address. There are no numbers in them, for example:
  • We need more research -> how much more?
  • I'm not against changing the size, just not now -> then when?
  • I'm not wedded to 1mb, but not sure 20mb is right -> then what?
  • Full node count is going down -> then what size do you think would fix that? 100kb?
  • It will make mining more centralised -> how do you measure that and how much centralisation would you accept?
and so on.