It would be nice if the detached signer and the normal wallet could both verify the correctness of generated addresses before you cause coins to be sent there.

e.g. the hardware wallet could give its master public key to Bitcoin Core and you can thereafter generate your receiving addresses on Core, with the option to have the HW wallet validate them.

One of my biggest fears about using any wallet is the "whoops, cosmic ray flipped a bit while producing receiving address; SFYL!" possibility. For high value cold storage, I always generate my addresses on two independent machines using two different pieces of software. Am I nuts for doing that?

With the above scheme, you are pretty well protected from losing money if your HW wallet is defective. You could still lose it if the HW wallet was evil of course, but that strikes me as much more likely to be discovered quickly.