I heard it was because the slave trade massively decreased the population. Even today when they go to university in the UK, etc, they just stay there.