First of all, nowadays, how does the process of rooting an android
phone work?
By using an exploit in the Android system, a binary file which is named su is added to /system/xbin partition. When an application wants to execute a command with root privileges, it executes the command with an su at the beginning of it. This sends a call to the binary file. Binary file reads a text document generated by the GUI application. If in this text file there aren't any specific information about the app that asks for root access, then the binary checks another text file (again, generated by the GUI) that shows how it should act by default in such a case. Normally in such a case, GUI app will prompt you about an app asking for root access and will ask you if you want to grant it. Depending on your decision, the GUI app will tell the binary to whether or not should it grant that app root access.
Whole process is pretty much that. However, methods of rooting are so many, that I can't fill them all in here. Because they mostly vary across manufacturers.
What would be the security risk of rooting an android phone ? If I am not mistaken, these could be grouped in at least two issues: the rooting process itself, and the aftermath.
Well, we can do so if you want.
For the rooting process itself:
First of all, it voids your warranty. Other than that, even though it is hard to completely brick the phone when only trying to root it, it is still possible when one is not careful enough. But the most common case is usually bootloops, which is not that much of a big problem in the hands of someone who knows what they are doing.
For the aftermath
A rooted phone is pretty much the same, only with more customization for a person who knows what they are doing. But every now and then there are some guys who corrupt their IMEIs, delete the Telephone app or sometimes completely brick their phone by messing up stuff in /system. Also, giving root access to an app with bad intentions, might not (and probably will not) end well.
a. Regarding the rooting process, is there any open source procedure
(or at least closely reviewed) to root a nexus phone that could
guarantee that there's no malware installed in the process ?
Well, I don't know about Nexus a lot but I don't think a rooting process might actually end up with malwares in your phone. Most su binary creators share their codes at gitHub already. And the popular ones (i.e. Chainfire from XDA) are trustworthy developers that have the trust of the community.
b. From what I understand, having a rooted android is no different
than having a linux OS with a root account. Are there any (free, open
source?) apps that can monitor (what commands have been launched, etc)
and prevent apps from getting access to the root account without my
agreement ? (so that it is linux OS where any account that requires
root privileges must go through 'sudo' and ask the user to enter their
password).
Know this: Apps can make use of the exploits on their own and gain access even on a non-rooted phone. However, the GUI apps like SuperSU or Superuser come with their own su binary that act only as ordered by the GUI app. So if you don't let them, they don't have root access.