Scriptural context must be the primary driver for the answer, particularly because this section starts with "therefore". Paul's admonition to "let no one pass judgment" springs forth from verses 2:1-15. In 2:1-5, he desires to visit them "face-to-face" so that they may have a "...full assurance of understand and the knowledge of God's mystery, which is Christ..."...in whom are hidden all the treasures of wisdom and knowledge." Why? So "...that no one may delude [them] with plausible arguments." Paul was seeking to protect them from false teaching by helping them to understand, and have full knowledge of, Christ.
This then leads to V8, where he admonishes them to ensure they are not taken “…captive by philosophy and empty deceit, according to human tradition, according to the elemental spirits of the world, and not according to Christ.” This, I believe, is the key to this context: Paul juxtaposes Christ himself with philosophy and deceit that is according to “human tradition” and ungodly, worldly spirits. There are teachings being put forth that are fundamentally opposed to truth, to Christ.
Paul goes on to explain this by fleshing out who the Christ is and what the Colossians have in him. He is fully God in the flesh (V9) having all rule and authority (V10). They are filled in him (V9), not lacking in anything (from a righteousness perspective). Why? Because they were also circumcised in Jesus Christ (V11). To be as concise as possible, V11-15 are asserting that, because Jesus was cut off, or circumcised on their behalf, they are fully righteous, lacking in nothing in terms of their position / justification before God. By his becoming sin for them (2 Cor 5:21) and nailing their sin debt (taken into himself) on the cross (V14), taking on the curse of the Law for them (Gal 3:10-14), they have “...been made alive with [Jesus], having forgiven [them] all [their] trespasses.” This is the context of V16 and following.
Therefore, since they are filled up with Christ, being made once-for-all perfect (Hebrews 7:27), they need to add nothing to themselves in terms of righteousness, in law-keeping: their justification or standing before God. Why? Because in Jesus, they are already perfect as to the Law. In fact, Paul makes it clear in Galatians that those who are in Christ are in fact no longer under the law, it having been fulfilled in Christ’s obedience, and they are reckoned perfect in God’s sight having received Jesus’ perfect obedience as being credited to them. (See Romans 3-4 as well.)
So, being that this is the context, when Paul indicates that they should “…let no one pass judgment on [them] in questions of food and drink, or with regard to a festival or a new moon or a Sabbath”, he must be speaking to the fact that these things, regardless of their origin (Jewish holidays or official Sabbath days of rest), are no longer relevant to the Colossians in terms of their standing before God. These things no longer apply to them as needing to be done under the Law or as a command of God to please him. So, if they are not participating in these things, or if the elect to do so out of worship for God, they should no longer let others tell them they are judged before God for do or not doing so. Their standing before God does not depend upon these things, but upon what they already have in the Beloved.
In conclusion, to answer your question directly, I understand the historical context here to be that Jews, AKA “Judaizers”, were indicating to the church that at least some of the Old Covenant Law must be added onto Christ for them to be accepted by God, that they were still requirements for Christians that must be obeyed. This, however, in Paul’s (and therefore Christ’s) mind, completely antithetical to that which Christ had already accomplished in both his perfect life and substitutionary death. Here is a commentary that go into more depth.