Where does Android store the binaries of the apps themselves?

Just answering as the comments suggested,

 /data/app/package.name/base.apk

The APK File is placed inside the data partition.

This is different from Windows as you mentioned.

Windows uses a different protection for System Software called UAC (User Access Control)
whereas Android uses a separated Partition on the HDD ( SSD ) called the System Partition.

All Stock Software is stored under

system/app/*app name*/app name.apk

&

system/priv-app/*app name*/app name.apk

or

system/app/appname.apk

&

system/priv-app/appname.apk

Additional Software from OEM's is usually stored under

system/vendor/app/

However since the new partitions in Android 7.0+ i believe that Vendor is now designated its own partition, making the app directory

vendor/app

Or similar.

Some Vendors put APK's into the system tp be automata installed also under "custom" partition or root folder

custom/app

Furthermore, applications can be built by binaries under

system/bin/

&

system/xbin/

If you're the Administrator of your PC, you can hypothetically destroy the systems integrity on Windows - however due to any changes being placed in /data partition on Android, you can not delete system software or destroy the system integrity without root access.

The data partition & cache partition hold every single modification made to the device - this means a "Factory Reset" just Erases the Data & Cache partitions on Android - VS windows reinstalling the entire system from a separate backup partition.