This page pointed me to the right direction.
Android 11 can only install user-provided root CA certificates to contain the X.509v3 CA:true flag, which I suspect wasn't necessary before for some reason, and kept on working after the upgrade until I tried to install a new one because, presumably, the flag is not necessary to validate a TLS trust chain.
The following command can check if a certificate contains that flag (replace input format and filename by the one you are using).
$ openssl x509 -inform der -in cacert.crt -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
....
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = **, O = ***
...
X509v3 extensions:
X509v3 Basic Constraints:
CA:TRUE
...
If CA:TRUE is not present under X509x3 Basic Constraints, your root certificate is likely not going to work on Android 11.
In order to generate a simple self-signed CA root certificate for Android 11, these minimal steps worked for me, and can be customized for your own certificate:
$ echo 'basicConstraints=CA:true' > android_options.txt
$ openssl genrsa -out priv_and_pub.key 2048
$ openssl req -new -days 3650 -key priv_and_pub.key -out CA.pem
$ openssl x509 -req -days 3650 -in CA.pem -signkey priv_and_pub.key -extfile ./android_options.txt -out CA.crt
$ openssl x509 -inform PEM -outform DER -in CA.crt -out CA.der.crt
The CA.der.crt or CA.crt files can be installed by going to Settings > Security > Encryption & credentials > Install a certficate.
Once installed, it appears proprely in the User certificates list, and all apps that try to connect to sites using that CA root succeed.
This answer appears to build on the same foundations but is much more complete and will probably work on more platforms, but the one above should be a good minimal working example.