"Is it a good interpretation of the phrase?"
In substance, yes--context shows it is the Jewish holidays being referred to. Martin Luther, in his Commentary on Galatians, agrees, commenting on "days and months and times and years" with:
The Apostle Paul knew what the false apostles were teaching the Galatians: The observance of days, and months, and times, and years. The Jews had been obliged to keep holy the Sabbath Day, the new moons, the feast of the passover, the feast of tabernacles, and other feasts. The false apostles constrained the Galatians to observe these Jewish feasts under threat of damnation. Paul hastens to tell the Galatians that they were exchanging their Christian liberty for the weak and beggarly elements of the world.
But in spirit, no--it is not so much the days themselves but that they show the Galatians' faith being weakened. The book of Galatians (seen as a whole) is a tirade against this church that had begun taking on Jewish custom in the belief that the Mosaic Law needed to be followed for salvation.
He speaks of encountering the behaviour previously when he was in Antioch (Galatians 2:14-16): (NIV quoted)
When I saw that they were not acting in line with the truth of the gospel, I said to Cephas in front of them all, “You are a Jew, yet you live like a Gentile and not like a Jew. How is it, then, that you force Gentiles to follow Jewish customs? We who are Jews by birth and not sinful Gentiles know that a person is not justified by the works of the law, but by faith in Jesus Christ. So we, too, have put our faith in Christ Jesus that we may be justified by faith in Christ and not by the works of the law, because by the works of the law no one will be justified."
Paul attempts in this book to separate the idea of salvation from the idea of following the law. The promise of salvation, given to Abraham, is separate from the Law, which was much later given to Moses, and for a different reason (cf. Galatians 3:15-19). Since Christ has come, the purpose of the Law is fulfilled, and we can rely instead on Christ alone (Galatians 3:23-25):
Before the coming of this faith, we were held in custody under the law, locked up until the faith that was to come would be revealed. So the law was our guardian until Christ came that we might be justified by faith. Now that this faith has come, we are no longer under a guardian.
The idea is that for those who have accepted Christ, no additional custom or law is necessary: you have already been saved by Christ, so what is putting yourself under additional restraint going to do for you?
So when we ask: "Does it mean he was suspicious of all holiday celebrations, or was this a particular sign of trouble for these particular churches?" we can only say that it is the latter. Because their faith in Christ's ability to save had weakened, they took to shoring it up with observances of Jewish Law--and we see from other parts of the book (e.g. Galatians 5:1-12) that it was not only holidays, but circumcision as well.
It is not likely that we can apply this to our holidays today. The Jewish days, we understand, were religious obligations--but our modern holidays are free celebrations, and do not promise us any justification. That we are under no obligation to keep them, and that they are not supposed to effect our salvation, puts them out of the way of at least this argument of Paul's.