It is the tabernacle that remains, but there is a sense here in which it has indeed been defiled because of the people's uncleanness. Thus, the high priest must make atonement for the holy place - because it remains in the midst of an unholy people. Rashi and other Jewish authorities explain this in terms of ritual impurity:
i. e. for those who entered the Sanctuary in a state of uncleanness
without having finally become conscious of this fact (Shevuot 7b), for
it is said לכל חטאתם (their transgressions), and the word חטאת implies a sin committed
unconsciously.
This was a very serious matter, because the sanctuary itself could become defiled by the presence of people were ritually unclean.
Numbers 19:20
Those who become unclean and fail to purify themselves—those people
will be cut off from the assembly, because they defile the sanctuary
of the Lord.
Although there would be times in later years when the sanctuary would be defiled by housing forbidden objects (e.g. 2 Kings 23:4), here the issue was the ritual impurity of some of the people, often contracted unintentionally and even unconsciously. This was the reason why, in the time of Jesus, the rabbis had instituted the "tradition of the elders" that all Jews must wash their hands before meals (Matthew 15:2), so that they would cleanse themselves of any impurity contracted from accidentally touching any forbidden thing.
Conclusion: the "sin" was ritual impurity, which is a state often brought about through natural processes such as menstruation for women and nocturnal emissions for men, as well as accidentally or intentionally touching forbidden objects such as dead bodies, insects etc. (Leviticus 5:2) On Yom Kippur, the high priest had to make atonement for the sanctuary, which was likely to have become defiled, even accidentally, (Numbers 19:20)- because it "dwelt" or "remained" in the midst of the people.