Single Frame Release mode captures one picture each time you press the shutter button on your Nikon D5300. It's the default setting for all exposure modes except the Sports and Pet Portrait Scene modes.
Quiet Shutter mode works just like Single Frame mode but makes less noise as it goes about its business. First, the camera disables the beep that it emits by default when it achieves focus. (You can turn off the beep for other Release modes via the Beep option, found in the Shooting/Display section of the Custom Setting menu.)
Additionally, Quiet Shutter mode affects the operation of the internal mirror that causes the scene coming through the lens to be visible in the viewfinder. Normally, the mirror flips up when you press the shutter button and then flips back down after the shutter opens and closes. This mirror movement makes some noise.
In Quiet Shutter mode, you can prevent the mirror from flipping back down by keeping the shutter button fully pressed after the shot. This way, you can delay the sound made by the final mirror movement to a moment when the noise won't be objectionable.