It sounds like the main issue is this is a web wallet server of some kind. If the clients were SPV then they'd be checking their own balances and downloading their own tx history, which would mean the coordination tasks could be done by storing encrypted blobs on the server rather than the server itself having insight into what's going on (see: Subspace).

So whilst you might be able to use some scheme to avoid the server knowing the xpubkey, if the server still knows all addresses and all transactions because the clients are web wallets ..... is there any point? It seems like maybe going from server knows everything to server knows 95% of everything: maybe not worth the engineering cost.