Buying a portrait set to begin with is a great idea. Introduce yourself to colour gradually - if you start with a tonal rendition (shades of browns thro to pale oranges and creams) you will be less nervous about it. There is quite enough to tackle with a figure without breaking your head over colour!
Gradually, when you start to become familiar with the way to use pastel, you can begin to introduce more adventurous colour - blues and green and purples in the cool areas, and hotter colours where the light strikes. So long as you are true to the tones (values) you see on the body, you can use just about any colour you want.