Madison seems to have had a different theory of American government than Lincoln.

"Each State, in ratifying the Constitution, is considered as
a sovereign body, independent of all others, and only to be
bound by its own voluntary act. In this relation, then, the
new Constitution will, if established, be a FEDERAL, and not a
NATIONAL constitution." --James Madison, Federalist No. 39

Now, I agree with Ayn Rand that a nation that breaches the rights of its citizens- or denizens, in the case of slaves and any other humans classified as less than citizens, restrained within its borders-has no basis on which to claim sovereignty, or immunity from invasion. But then, no nation has a duty to invade them on that basis alone. It would be well if people - politicians particularly - could remember that invasions are costly things for the invading country as a whole. Lives are lost, obviously (and tragically), and the time and labor spent rebuilding any assets seized would have been better spent in trading and building new things.