I'm pretty sure Granger has the emotional depth to be fearful about having his first kid, however, he's a blueblood as is his wife, they have a lot of money and are influential, and I'm quite sure Caroline would have the best care possible during her pregnancy and afterward.
After childbirth, there would be nannies, and other caregivers, mentors and instructors on the estate. It was not uncommon for upper crust parents - especially those in the peerage and above - to take a very standoffish role in raising their children, who spent the vast majority of their time in the company of their nannies and teachers. In fact, the children would have an audience with their parents at some point during the day, but adult concerns were usually more pressing than the emotional needs of the children.
It would be very surprising if a gentrified naval officer such as Granger expected there to be any other way to raise children - it was how he would have been raised, after all. Consider the prologue to The Gunroom - the relationship between George and his father is very defined and George's path is very carefully plotted to fit the Earl's plans, not George's, even though his sense of duty and obligation means he'll do what's necessary to further the family's goals.
In the middle and lower classes, children were much more important and families were much closer, but the upper classes were all about the political connections and proper upbringing than they were about familial love and tenderness.
I think it's really easy to see this story through the lens of our times and judge the characters by the the childrearing practices we know, and not through the times as they were lived by the real people in them. This is made a little more difficult by Mark's writing style, which tends to modern grammar and structure, thankfully - along with brevity and good editing - sailing off on a tangent to discuss how George and Caroline raise the little ones probably won't further the story much.
Just my two shillings, guvnor. Sorry if I came off as a bit technical here, now, be so kind as to pass the lard.