The word 'magi'
Nolland1 summarizes the initial difficulty as such:
The word 'Magi' was originally applied exclusively to members of a priestly caste of the Medes and Persian [sic] who had esoteric skills in interpreting dreams. However, the use of the word broadened to embrace various categories of persons who were marked out by their superior knowledge and ability, including astrologers, soothsayers, and even oriental sages. From here the term became debased first to functioning as a label for sorcerers and magicians in general, and then in the end to becoming a term for quacks, deceivers, and seducers. The difficulty with the term is that later developments do not displace earlier usages, but rather the various usages tend to coexist. . . .
This is also explained by Ogden2 as follows:
One of the words that the Greeks used to refer to a magician is magos (pl. magoi), a term originally used to denominate members of a caste of Iranian fire-priests. It has been suggested that members of the caste made their way to the Greek cities of Asia Minor in the second half of the sixth century BC and from there to the rest of the Greek world. The suspicion with which the rituals performed by magoi were met may have led to the word's acquiring a derogatory connotation.
Picking up on the obvious and subtle in Matthew's narrative, Nolland3 continues:
Matthew's Magi do not interpret dreams, but they do observe and interpret the stars (or at least one), and they are from the East. If Matthew has one eye on the role of Magi/astrologers in Moses' infancy haggadah (as seems likely), then this helps to bring the role of astrologer to the fore.
Astrologers following Jewish traditions
Because of the original association of magi with 'a priestly caste of the Medes and Persian[s]', and that Matthew mentions the magi come from 'the east', it has been commonly accepted that the magi are, indeed, astrologers who traveled from Chaldea, Media, or Persia.
Conservative Christian scholarship sometimes has the magi in Matthew following after some sort of prophetic tradition that traces back to from the Jews who were exiled to Babylon. Smillie4 calls the magi 'Chaldean seekers [of the Christ]', summarizing this view with:
The magi who come seeking Jesus in Matthew 2 are informed of the plan of God precisely because Israel had earlier been deported to and had taken up residence in Babylon, where their Scriptures became available to public scrutiny.
This is sometimes taken to a much more specific end, that the magi seen in Matthew came from a tradition that went back to the sage Daniel. MacArthur5 says:
The magi from the east . . . who came to see Jesus were of a completely different sort [than Simon Magus]. Not only were they true magi, but they surely had been strongly influenced by Judaism, quite possibly even by some of the prophetic writings, especially that of Daniel.
However, this entire theory would be undermined if messianic thought didn't emerge until after the Babylonian exile. The view that such a tradition was taught to Persian magi by Daniel is virtually non-existent in critical scholarship, since the stories found in Daniel 1-6 are considered to be legendary fiction.6
A connection to Balaam's 'star' prophecy
At least one alternate connection, from Goldberg,7 has been suggested as well:
Also of particular interest is the parallel between the non-Jewish magi who come to Jesus having seen 'his' star and that other Gentile magus, Balaam, who foresaw a star coming forth from Jacob (see Num. 24:17). For Matthew, therefore, Jesus' prophesied messiahship is once more unmistakable. See Brown, Birth of the Messiah, p. 188-96.
A text-critical approach
Within the narrative of Matthew, especially the opening chapters, it has long been recognized that the author sought to draw parallels between Jesus' career and that of Moses, but 'caution' is needed in drawing any connections to the 'star' prophecy of Balaam. Luz offers, I think, the best summary treatment of how the episode in Matthew should be approached. As Luz8 says:
Among kindred stories of the royal child, the Moses haggadah is closest to our story and to [Matthew] 2:13-23. Magi (TgJ on Exod. 1:15; ExR 1:18 on Exod. 1:22) or scribes (Josephus, Ant. 2.205) predict for Pharaoh the birth of Moses; he is perplexed (Josephus, Ant. 2.206) and conceives the plan of infanticide. The Moses traditions probably have fructified our story. At the same time, it proves itself independent of them—especially in the use of the motif of the Magi—that it can in no way be understood as a mere copy of the Moses haggadah.
This does not explain the motif of the star. A star occurs in the story of the Abraham child who is pursued by Nimrod. However, the examples are from a late period. In non-Jewish parallels there are reports of a comet at the birth of Mithridates and in the Nero episode in Suetonius. A "great sign in the sky" is mentioned in Rev. 12:1. Comets or other phenomena of light at the birth of great men are widespread in antiquity. The question is difficult whether there is reminiscence of Balaam's prophecy of the star out of Jacob (Num. 24:17). The messianic interpretation of this passage was widespread; the history of interpretation shows that Christians were aware of it. But the star is not identified with the Messiah, as in the interpretation of Num. 24:17. Literal reminiscences of the story of Balaam in Numbers 22-24 are almost completely missing in [Matthew] 2:1-12. In Jewish tradition, relationships are drawn between the Magi who appear in the Moses haggadah and Balaam, but the examples are late. Thus caution is indicated.
But this, of course, raises another question: If the author was deliberately arranging his birth narrative of Jesus to remind his readers of the birth traditions surrounding Moses, with the incorporation of a popular cosmic sign used in birth stories, should we understand the episode in Matthew as historical at all? Luz9 suggests not:
Our story is a legend told briefly and soberly which does not conform to the laws of historical probability. The desperate questions of the interpreters demonstrate this: Why did Herod not at least send a spy along with the Magi? How could the whole population of Jerusalem, the scribes, and the unpopular King Herod be perplexed by the coming of the Messiah? The star also is not described realistically, i.e., as astronomically possible.
. . .
Of all these attempts [to locate the 'star' in natural astronomical phenomena from the time period] it may be said that they rarely are a help for the explanation of our narrative. Matthew intended to describe a miraculous star which appeared in the East, preceded the Magi on the way from Jerusalem to Bethlehem (thus from north to south!), and then remained standing above the house where the child was to be found.
. . .
Finally, the fact that Luke does not report a similar event argues against a historical nucleas; besides, the episode of the Magi could not be integrated into the Lukan birth narrative. The parents of Jesus too seem not to know anything of the miraculous event at his birth (Mark 3:31-35)! In short, a historical nucleas is no longer comprehensible; however, the numerous parallel traditions in the history of religions makes the embellishment of the narrative more understandable.
In this case, there were no literal magi who visited Jesus. Any attempts to associate them with a particular group would be futile, because the author of Matthew (or perhaps an early Christian tradition he is relying on) didn't see their exact identity as necessary to the birth narrative's purpose.
Footnotes
1 John Nolland, The Gospel of Matthew: A Commentary on the Greek Text (2005), p.108.
2 Daniel Ogden, A Companion to Greek Religion (2010), p.358.
3 Nolland, Gospel of Matthew, p. 109.
4 Gene R. Smillie, '"Even the Dogs": Gentiles in the Gospel of Matthew', JETS 45.1 (2002), p. 73-97.
5 John F. MacArthur, The MacArthur New Testament Commentary: Matthew 1-7 (1985), p. 28. (Bold original.)
6 See the conclusion, with its citation, in my answer here.
7 Michael Goldberg, Jews and Christians: Getting Our Stories Straight (2001), p. 149, note 1.
8 Ulrich Luz, Matthew 1-7 (1992), p. 131. (Italic original.)
9 Ibid., p. 132-133.