Everything else in this handbook assumes the secretsmgr on your machine is the one we published. Nothing in the design can prove that. The client holds your private key, sees every value you decrypt, and decides what gets encrypted to whom, so a modified build defeats every other protection here, and where you downloaded it from does not change that.
What genuinely helps: the release is signed, so an attacker needs the signing key rather than just a web server. Builds are reproducible, so anyone can rebuild from a tag and check that the published bytes match the published source, which turns "trust us" into "several unrelated people got the same result". And there are multiple ways to install (the install chapter), so if you would rather not run our binary, you can build one from source you have read.
What remains: the first time you obtain the release signing key, you trust the channel you got it from. There is no way around that, in this tool or any other of its kind. If your bar is high, build from source and compare checksums, which is the strongest position available.