... To make it look like a thumbnail ...
It sounds like you aren't actually making a thumbnail, but rather loading the full size image and having Windows display it scaled down. The problem with this is exactly what you have seen: The object in memory is the full size image. If you are using a typical 10mp camera you are going to go out of memory really fast.
There are some built in GetThumbnail routines available to you from the .NET framework. Or you can read in the fullsize image, and scale it yourself into a new bitmap object, then dispose of the fullsize object and place the scaled image in your PictureBox. Personally this is what I do. My scaling methods have a few more arguments so I can choose to preserve the full color-depth or not, add a border or not, scale to max width or max height etc.
But for starters use the built in functions to see if that solves your issues and is of sufficient quality to save you the work in writting your own scaling.
TIP. The .NET method for GetImage.FromFile(string FilePath) has a glitch that keeps the file on hard drive locked up as long as you are using it. So if you use this method load to a TemporaryImage the copy that into a FinalImage then .Dispose() of TemporaryImage so your application is still tied to the harddrive file keeping any other program from having access to it.