In reality, by the dawn of the 19th century there was slavery pretty much everywhere in the United States. More in some places than others, but it was a thing all over. There were abolitionists as well – pretty much everywhere – carrying on about the evil of the peculiar institution and making everyone unhappy. It was an onerous institution, even for those not actually slaves. It was expensive and high maintenance and morally suspect, and after a bit the Northern states began realizing they just didn’t need it that badly. Gradually, the practice was phased out and eventually banned - and everyone seemed better off.
Besides, they already had an entirely different class of not-quite-people to exploit and dehumanize. In the elite world of historiography, we call them the "Irish.”