I also mainly use earths, plus ultramarine blue. The kinds of scenes that I usually paint can be done entirely with those, but some other scenes would also need an option for a more chromatic green or some accents in a sunset.
Even then I find myself needing to neutralize yellow ochre with plenty of white and a little of some sort of neutralized earth or sometimes a little ultramarine violet because it’s still too chromatic for something like a rocky cliff, unless it’s in golden hour lighting.
Value and form matter much more for realism than hue and chroma.