To have control, you need to configure the SD card as Storage and not Expansion. Not all phones offer these options automatically and it is possible you have to switch it. Changing modes, wipes data on the card since it gets reformatted.
Once you the SD card is configured as Storage, you can use it to store Data but generally not Apps, at least on most recent official versions of Android. However, many Apps that expect to produce or manage large data have the option somewhere to configure where that data is stored. This includes the Camera App, since it is expected that it can capture huge volumes of photos and videos, Netflix, Amazon Prime, Maps.ME and many more, but those are the ones I know and are configured on my current phone.
For applications that are media players, such as VLC, then generally you can place media yourself in any folder that could be on the SD card or in internal memory. You can place all your videos and audio in the SD card and then either navigate to where you want to open media or add folders on your SD card to their 'Media Folder' list. Technically, you can add the entire card but if it is huge, scanning it will take a long time, so it is preferable to create yourself a folder structure by media type. Note that you will have to grant permissions to modify the SD card for each application you want to use that way.
When you installed the card as memory extension, I am surprised that much data appeared but generally, the process involves creating some cache. Before you erase that, do take a look to see what those files are using the file manager. There should be mostly hidden files (starting with a dot) and folders called 'cache' or similar.
Note that modern Android phones mount SD cards by their Label which means that when you swap SD cards, most applications will no longer find their data and revert to internal storage or show an error. Ideally, there would be a way to mount the SD card in a fixed mount-point (but I have not figured out how to do this. It is likely to be possible since all phones so far have a single SD-slot, so they could give it a fixed path) but if you use a cloning tool (or just dd on Unix), then you can backup an SD card so that it has the same label. This is not so important but there are some apps that once they do not find their data, force you to redownload it, even if you put back the same SD card again later and point them to the same path (MAPS.ME does this).