Childhood malnourishment is a serious problem not only in the developing world, but also in the United States among some poor families. In extreme cases, it can negatively affect growth rates, not to mention making it very difficult for hungry children to concentrate on anything but their next meal. Height differences between older generations (who lived during scarcer times) and younger generations, are a good indicator that more calories leads to more growth. Some of the same differences can be seen between rich and poor countries.
Fortunately, the human body is very resilient, and millions of children grow up every year on less-than-ideal diets (not true deprivation) and they turn on out just fine, and extra height doesn’t necessarily indicate better health. People were shorter a hundred years ago; that doesn’t mean they were unhealthy compared to people today.
Most importantly, there’s no reason to worry about whether you ate the perfect diet as a child. You are who you are, and you can’t change the past. The only thing to do is to figure out how maximize your potential right now.