He couldn't go to Belfast because he was active duty in the British military.

But, he left the army to do his around the world walk, which means that he could have left the army and gone to Belfast instead.

He and his adult son have a good relationship now seemingly, but that doesn't mean his son didn't need a dad growing up. Having said that, it seems like this dude had a whole slew of mental issues, so perhaps it was for the best that he wasn't around his son much while he was growing up.