The "Terrain Quality" setting is a graphics setting available in the F2 settings overlay in Experimental Rust. It is a slider ranging from 0 to 100, and determines the graphics quality of the terrain around you (such as the ground or mountains). On the slider, lower numbers would mean lower quality, and vice versa. If you are looking to enhance your FPS, is would probably be better to lower the Terrain Quality.
NOTE: Rocks, overhangings, and cliffs are NOT affected by this setting. They fall under the Object Quality slider.