"Trust" is a small world for a large field. You have to look at each variant individually:
Trust that the custom ROM does not brick your device or destroy smartphone components
The more people use a specific custom ROM, the more likely all serious problems are known and may be even already fixed. For that reason, I personally prefer LineageOS-based ROMs without any modifications (or the minimum to support the specific device).
Trust that the custom ROM does not contain malware
I assume that purposely infected ROMS are very unlikely. Most custom ROMS are created by experienced developers that are active for years. Publishing an infected ROM would result in a loss of reputation for such people. As a lot of users use their ROMs rooted, the chance is high that the infection would be detected. Therefore (as long as the developer account has not been hacked), infected ROMs are unlikely.
However, I would recommend to check the history of the author and only use popular ROMs from XDA-Developers or better those directly provided by Lineage OS.
Tust that there are no backdoors or similar
Custom ROMs contain a lot of compiled code. Especially the kernel and other sensitive software running with high privileges is usually compiled by the author. Even if the sources for the kernel used by the ROM are public (and is free of backdoors and so on), it is a common problem to prove that the source code is really the one used to build the kernel. Detecting something malicious in a kernel or a different compiled software is very problematic.
Again, you have to trust the author of the custom ROM...
Trust that the all security features work as intended
The main factor of security is that some actions are forbidden/denied for certain entities. The usual testing approach of custom ROMs is "it works for me". Therefore, I assume that most of the security features are not well-tested in custom ROMs. If something goes wrong in the build process, it may remain undetected for a long time.
One common problem for example is that custom ROMs are signed with AOSP private key (which is public). Therefore, it is easy for malware developers to sign an app with the system keys to gain special system permissions.