So the question is, what's the precondition for std::any's swap and assignment operators? Should we really support those use cases, i,e. can we just compare the address directly? Even if we cannot ...

It's possible to assign between objects of a derived class via a trivial assignment operator of the base class. As a result, it's possible to synthesize a defaulted and trivial assignment operator ...