Open links in corresponding app instead of browser

Open your Android settings and search for "default". This should open App Permissions; near the top, under the Default heading, tap on "Opening links." Near the top of the page should be a toggle for "Open links in apps" - set this to on/yes, and restart both the browser and YouTube to reset with the new setting (you may need to reboot your device, I'm not sure).

basically, the option in YouTube tells it to watch for those links so it can open them when it sees you click on one, but the Android system settings are what give it permission to actually open them. You can scroll down through the list of apps on that screen to the bottom and tap on YouTube to see exactly what links it will look for and open.

[Note: these steps are based on a tablet (Lenovo Tab 4 10 Plus) running Android 7, so they may be slightly different on a phone or a different tablet. You should be able to get to this setting via the Apps menu though in Settings, just it might be kinda buried and hard to find, which is why I use the search function.]