When someone's client says "my key is protected by a passphrase", nothing in the world can check that. The claim is signed, so you know who made it and that it has not been altered. But a determined person can modify their client, or simply strip the passphrase afterwards, and no signature or public key can reveal it.
What asserted custody genuinely gives you: it stops accidents and shortcuts, which is the overwhelming majority of real risk, and it gives an auditor a signed statement with a name on it. What it is not: a boundary that stops a person who has decided to lie. Only attested custody does that.