There are two parts to this: the mechanism for the communication apps to say they can share things, and the mechanism for the apps with content to share. I'll deal with them one at a time, but first a quick note about intents.
Intents
The intent in Android is what's used to start an app. It can work like a shortcut, or like a Windows file association. The intent can specify the name of the app to start, but it doesn't have to. It can also specify the action to perform (e.g. "open your main activity", "view this URL", "create a calendar entry") and the MIME type of the object it will act on (e.g. that the thing you want to view has type image/jpeg). Some names of actions are specific to a particular app, but many are standardised by Android to allow apps to interact.
Intent filters and communication app
Each app specifies intent filters to say what kind of actions it can perform. e.g. an image viewer app would register an intent filter for "view this URL" where the MIME type is "image/*"; a web browser would register an intent filter for "view this URL" where the URL scheme is "http" or "https". To be able to share things, an app registers an intent filter for the "send" intent, either with a specific type (e.g. if it can only share photos), or for any type.
Content app
The app that wants to share some content creates an intent with the "share" action and a URL pointing to the content to share. (This is often a "temporary" URL that gives the receiving app temporary access to the file in the content app's private storage, or to a stream generated by the content app.) It doesn't specify the name of an app to handle the intent. Then, it gives that intent to the Android framework and asks Android to display the chooser dialog. Android compares the intent with the intent filters from all installed apps, and creates the list and shows it to the user. When the user picks a particular app, Android starts that app and passes it the intent. The app uses the URL to get the content from the original app.
New in Marshmallow: Direct Share
In Marshmallow, there's another mechanism called Direct Share. This supplements the intent mechanism so that you can choose people or groups to share to directly from the chooser dialog, skipping the extra step of "who would you like to send it to". This works slightly different: the communication app has to provide a service. When the app with the content asks for a chooser, Android starts the service for each relevant app. The service returns a list of relevant share targets, and Android assembles all those separate lists into the list that gets shown in the dialog.