Before answering this question, one might wish to consider another question:
Why would it be wrong to observe sabbaths and special days?
The answer to this will lead directly to the correct answer for the question posed in the OP.
Understanding the Ceremonies
When Christ died, his death fulfilled the ceremonial types and ordinances that had heretofore pointed forward to the Messiah. It was no longer required to sacrifice the blood of animals, for Jesus had shed his own blood for us.
In the Levitical system, three major feasts were observed, including the Passover (the feast of unleavened bread), the "feast of harvest" (also called the "feast of weeks"), and the feast of tabernacles (also called "the feast of ingathering").
Three times in a year shall all thy males appear before the LORD thy
God in the place which he shall choose; in the feast of unleavened
bread, and in the feast of weeks, and in the feast of tabernacles: and
they shall not appear before the LORD empty: (Deuteronomy 16:16, KJV)
Each of these feasts had special "sabbaths" attached to them. For instance, consider this passage:
29 And this shall be a statute for ever unto you: that in
the seventh month, on the tenth day of the month, ye shall afflict
your souls, and do no work at all, whether it be one of your own
country, or a stranger that sojourneth among you: 30 For
on that day shall the priest make an atonement for you, to cleanse
you, that ye may be clean from all your sins before the LORD.
31 It shall be a sabbath of rest unto you, and ye shall
afflict your souls, by a statute for ever. (Leviticus 16:29-31, KJV)
This "sabbath" was declared to be such, not by being the seventh day of the week, but because it was the annual day of atonement--a solemn day. In some years, the day of atonement landed on the weekly Sabbath, in which case the Jews considered it a "high" Sabbath--doubly "sabbath," if you will. Such was the case for the Sabbath in which Jesus rested in the tomb (see John 19:31).
But what happened to the "day of atonement" after Jesus' death? If the sacrificial system, along with the Levitical priesthood, was no longer in force, there was also no day of atonement to be observed, with its companion sacrifices.
Even after Jesus' death, his disciples observed the weekly Sabbath, the one addressed in the Ten Commandments, as did Paul himself.
Paul in the Synagogue on the Sabbath
And Paul, as his manner was, went in unto them, and three sabbath days
reasoned with them out of the scriptures, (Acts 17:2, KJV)
And he reasoned in the synagogue every sabbath, and persuaded the Jews
and the Greeks. (Acts 18:4, KJV)
Was Paul himself observing the Sabbath? Why, then, would he suggest to others that it was wrong?
Why Observing the Ceremonies is Wrong
To observe the weekly Sabbath, the moral Sabbath, was to keep the Ten Commandment law of God. But to observe the sabbaths of the annual feasts was to show lack of faith in Christ. To keep the ceremonial sabbaths declared to all that one did not accept Jesus as the Messiah; that Jesus' death could not save us. By keeping these feast days, therefore, one worked directly against Christ.
It was this sin against Christ which Paul addresses in teaching people not to observe those feast days.
10 Ye observe days, and months, and times, and years.
11 I am afraid of you, lest I have bestowed upon you labour
in vain. (Galatians 4:10-11, KJV)
In actual fact, Paul does not use the word "sabbath" (Greek: σάββατον/sabbaton) here. He is focused on the observance of these ceremonies in general.
The Ten Commandments were not Ceremonies
The Sabbath of the fourth commandment was not given as a ceremony. It was not tied to the promise of a Messiah. It was, instead, the central focus of the Ten Commandments, providing the identity of the Lawgiver as our Creator, and indicating His domain--inclusive of heaven, earth, and seas.
Remember the sabbath day, to keep it holy. (Exodus 20:8, KJV)
In the commandment, the definite article "the" is present. That article is absent from the text in Leviticus 16:31. Just as in English, the article makes a difference in meaning. If you give me something and later wonder whether I had appreciated your gift, you would be pleased to hear me say "I liked the gift" and would be rather unsatisfied to hear "I like gifts." In the latter case, you would be uncertain as to whether or not I had liked your gift. The article makes a big difference.
And the Bible puts a difference between the sabbath of the fourth commandment and those of the ceremonies. To observe the Ten Commandment law is to love Jesus, who said "If ye love me, keep my commandments"; whereas to observe the ceremonial sabbaths is to show lack of confidence in Christ.
Conclusion
The days Paul is addressing are those of the ceremonies which were abolished at the cross, and not the Sabbath of the Ten Commandments, all ten of which still stand today.