Well, according to the visions of Blessed Anne-Catherine Emmerich (which actually predate similar beliefs later found in Mormonism as Blessed Anne had them a few decades before the founder of Mormonism was even born), non-whites are the cursed progeny of Noah's grandson Chanaan, who started out with fair complexions but became darker and fouler as they descended further into savagery. Since a lot of Blessed Anne's visions seem to match up with the general consensus of Christian thought at the time, one can assume that its a reasonable explanation.