To modern sensibilities it seems harsh, but history proves it to be correct. Keeping aside the black slave issue, look at North America versus South America. The Spanish enslaved and mixed with Amerindians. The Americans and Canadians displaced and/or killed the natives and largely kept homogenous. And which continent was more successful? At least until we abandoned that racial homogeneity. Even considering the blacks, more were sent to the Caribbean and Brazil than ever went to the US. It didn't turn them into empires, just mongrelized their population and lead to mediocrity.