I first started with Fraps, until the file sizes just became to large (avi) and then moved on to a trial of Bandicam which was really good, but rather expensive. That was until I found Mirillis Action! on sale at Steam a few years ago and I've been using it ever since. However, AMD, Nvidia and Microsoft have respectively improved ReLive, Shadowplay and Game DVR enormously over the years. There are advantage over Action!, in that you can record at a higher bitrate with ReLive, which makes a big difference in the quality, and the file size of course. It also helps that they're free, with the quality being as good, if not better, than paid software. I don't have much experience with Shadowplay since it's on my work PC and I don't game much on that one, but ReLive has an adjustable bitrate slider and I usually use around 50Kbps, encoding with HEVC. (I have to switch to AVC if I know I'm going to edit with Vegas Platinum 13, since HEVC isn't compatible) Game DVR also produces excellent quality with all setting at the highest level, with the bitrate at 42Kbps. Mirillis Action has issues with UWP games and very often churns out some really crap videos at 12kbps and I now use it less and less. For example, today I recorded Grid and it spat out a 30fps result, which looked awful. On the other hand, it records other games like DOOM perfectly well, but it doesn't have an adjustable bitrate. As for YouTube, generally speaking, videos uploaded tend to look awful at first and not so good in 1080p either with their avc encoding, which is why I now set the in-game resolution to 2560 x 1440, so that YouTube encodes with the VP9 codec. The difference in quality is night and day, quite frankly and usually takes effect after about ten minutes. If you right click on a YouTube video and hit Stats For Nerds, you'll see the encoding. I think I might give Bandicam another go and see what it's like nowadays. Edit I forgot to mention that Game DVR will only record at a max of 1080, regardless of resolution.
OBS is the go-to for pretty much everyone: https://obsproject.com/ you can set up scenes, mics, etc. for what you want to display, bit rate, audio/video codec, if GPU or CPU rendered, etc.
If you're just capturing "for fun", even the built-in windows one is good enough, hit windows+g, it also supports direct streaming to YouTube, plus it can auto-capture the last 30 seconds at all times, just hit the record last 30 seconds to save clips.
Compress the quality using Handbrake: https://handbrake.fr/ YouTube now supports H.265, use that if you have a bandwidth issue/cap, but it's better on your system to basically set it to ~12-20Mbps H.264 recording and then upload the giant file, that's about 2-4.5GB per half hour, YouTube will deal with automatically downgrading it for you/converting.
For recording, if you can always go CPU, it's better quality at lower bitrate, but if it's a game that stresses the CPU, go GPU and up the bitrate. Always make sure to pick a codec that your CPU or GPU supports for hardware encoding, all support H.264 encoding, H.265 (HEVC) depends on generation.