His mother and father lived in a council house making ends meet while he had millions. His brothers and sisters also had very little
Charity begins at home - period!
That was their choice to live in a council house, there's plenty of ways of getting out of that world if you actually try.
He doesn't (didn't) owe them anything.