I think you’re onto something with the balance being best.
As for struggling to consume adequate calories I had a lot of trouble with that until I started drinking a lot more calorically dense liquids. Mostly whole milk in my case. My midwives expressed concern that I might be drinking too much milk so I cut back this last week and I’ve been significantly colder and more tired despite eating more or less the same. Abandoning that idea. Milkshakes have also been a delicious way to pack in some more calories between meals.