No, baptism was not illegal. In fact baptism and similar washings were fairly widespread at the time. For instance, Josephus (Wars of the Jews 2.129) describes the practices of the Essenes:
After which they assemble themselves together again into one place; and when they have clothed themselves in white veils, they then bathe their bodies in cold water. And after this purification is over, they every one meet together in an apartment of their own, into which it is not permitted to any of another sect to enter.
In the Talmud (B. Yevamot 46a) there is debate over whether immersion is required for Gentile converts to Judaism or whether circumcision alone would suffice. Regardless of necessity, it was expected of converts that they would undergo immersion.
So why then do the Pharisees question John about his baptism?
While it is true that baptism and washings were not uncommon at the time, clearly John's baptism is different. Carson argues that in other baptisms, "baptism was self-administered. Candidates baptized themselves. One of the things that characterized the baptism of John the Baptist is that he himself administered it. It may even be that the authority implicit in such an innovative step triggered the assumption in the minds of at least some Pharisees that John’s baptism was an end-time rite administered by an end-time figure with great authority."
In other baptisms of repentence - like those of Gentile converts - the baptism is of those outside the community so that they may enter the community by repenting of their old life. Carson points out furthermore that John's baptism is of the Jewish people. And if that's the case, John must be inviting them in to a new kingdom. But what is his authority to do so if he is not "the Messiah, nor Elijah, nor the Prophet"?
Raymond Brown similarly agrees, "The objection posed by the Pharisees has its logic: if John the Baptist does not claim any recognizable eschatological role, why is he performing an eschatological action like baptizing?"
So it's not that John is doing anything illegal per se, but that the manner in which he is doing it suggests that he has an authority which he specifically denies.
Josephus, F., & Whiston, W. (1987). The works of Josephus: complete and unabridged (p. 605). Peabody: Hendrickson.
Carson, D. A. (1991). The Gospel according to John (p. 145). Leicester, England; Grand Rapids, MI: Inter-Varsity Press; W.B. Eerdmans.
Brown, R. E. (2008). The Gospel according to John (I–XII): Introduction, translation, and notes (Vol. 29, p. 51). New Haven; London: Yale University Press.