Now, you should be clear on 2 things:
- All flashable packages are signed. They can be stock OTA zips, root packages or even custom recovery or ROM zips.
So what is the difference between the "signed" and allegedly "unsigned" packages?
It's in the cryptographic keys used to sign the deployment builds, or the zips in this case.
The "signed" zips, e.g. a stock OTA zip, use a private key specific to the OEM for signing the build. This is a "release" key, and the only type of a key a stock recovery will recognize using its corresponding public key when asked to "Install Update from SD Card".
The "unsigned" zips, e.g. , root packages or custom ROMs, use the publicly known AOSP (Android Open Source Project) private keys for signing the deployment build. Even if the build hasn't been explicitly signed, the test-signing is implicit. Any released device with a stock recovery from the OEM does not have or recognise the public key corresponding to the "test" private key. Hence, it will refuse to install such test-signed or "unsigned" zips.
This is to maintain device integrity and security, at least according to the OEMs.
This is the official source: The Sign Builds for Release Page on the AOSP website.
- A custom recovery like CWM or TWRP either disables or provides an option disable (which is usually enabled by default) "ZIP File Signature Verification".
This basically implies that it can flash any package, regardless of what keys, test or release, were used to sign the package. To the (stock) recovery, everything is an OTA zip.
So why can some stock recoveries flash a custom recovery zip or root package?
This is probably due to 2 reasons:
The stock recovery has a security vulnerability or exploit, which the flash package uses on being initially loaded to bypass or circumvent the signature verification procedure.
The OEMs private release keys are actually known to the developers of the flashable package, but not to the developers of the ROM. I have personally observed this with Mi and some low-end Samsung devices.
Also, there's another reason a custom recovery is necessary for flashing a custom ROM.
To install the ROM, you must do these steps before actually flashing the ROM zip:
Wipe /data and /cache. On a stock recovery, this amounts to a factory data reset.
Wipe /system. No stock recovery can do this, for obvious reasons (why would a recovery wipe the /system partition if it doesn't support getting it back from somewhere, such as a custom ROM zip?)
Wipe dalvik-cache, usually done after flashing the zip. Again, no stock recovery can do this, but in this case there's no issue with the device getting bricked if the stock recovery did this.
If your stock recovery does not have either of the above 2 "features", you'll need a flashing software which will flash the recovery or ROM images to your phone from your PC. These flashing software use disk image files of the partition to be flashed, and care nothing about whether that that is signed or not. This involves the bootloader of the device, and bypasses the recovery altogether.
Custom recoveries are usually distributed as flashable disk images, not as flashable ZIPs.
A few examples of such software are fastboot(most Android devices), Odin, Heimdall(both for Samsung devices), Smartphone Flash Tool(for Mediatek devices)
Even the bootloader might perform security verifications on the flashed files on some cases, but insofar as much as fastboot is concerned, the bootloader had to have this verification disabled, that is, it needs to be "OEM unlocked"before anything can be flashed to the device. Anyway, the disk images do not contain any signature related metadata in the first place.