Two Schools of Thought
While it is somewhat of a simplification because the field of textual criticism is very involved, there are basically two main schools of thought on determining the correct text form for manuscripts.
- Priority to oldest extant reading
- Priority to majority extant reading
Again, each school of thought has other factors it considers when examining texts, but one of the fundamental presuppositions each takes when approaching textual criticism is either of the two options above.
Since the 19th century, the #1 view has ascended to be the majority view of scholars, and the #2 view has become the minority view. Prior to that, the #2 view was in essence the de facto view because of the fact that the majority of manuscripts available to scholars had those readings (many would not even have been aware of the oldest readings, as they were later manuscript discoveries).
One main presupposition behind the two presuppositional views is whether having a very few older manuscripts with a minority text reading is warrant to assume that there were no older manuscripts behind the later manuscript copies that have the majority text reading. Another main presupposition behind the two views is the nature of God's role (if any) in insuring His words were kept accessible to later generations of believers (this, of course, assumes divine inspiration of the text if it is a factor considered at all).
It is important to understand the above in light of the fact that one will often read statements saying a reading is not supported by the "best" manuscripts, which is usually an indication of a #1 position asserting their presupposition of priority to oldest. However, knowing which group a textual critic belongs to will ultimately determine what he or she means by "best."
Disclaimer: Generally speaking, I am of the pre-19th century majority view, the current minority view on text critical matters.
Basis for the Decision
Most modern translations exclude the verse because they are following a text (such as NA28) that was constructed off of an oldest priority position on the text, which when variations occur is usually a minority for extant reading, and thus (as you have noted in your question), they do not follow the majority of manuscripts. The exceptions are those translations that tend to follow the majority textual witness (essentially KJV, NKJV for translations still in popular use).
So what is the "basis" given for this verse. We have the commentary from Bruce Manning Metzger, United Bible Societies, A Textual Commentary on the Greek New Testament, 2nd Edition (London; New York: United Bible Societies, 1994) to inform us. I have emphasized some points to discuss following:
17:21 omit verse {A}
Since there is no satisfactory reason why the passage, if originally
present in Matthew, should have been omitted in a wide variety of
witnesses, and since copyists frequently inserted material derived
from another Gospel, it appears that most manuscripts have been
assimilated to the parallel in Mk 9:29.
- "no satisfactory reason ... omitted in a wide variety of witnesses": This is based on the presupposition of the textual critics. Its omission in a wide variety of witnesses would be the same reasons for its inclusion in the even wider variety of witnesses, there was an early copy made that did not include the verse, and so that version got copied to a wide variety of witnesses. This in and of itself then tells us nothing about which reading is correct.
- "since copyists frequently inserted material derived from another Gospel": This is using an understood fact (there is evidence such occurs) to presume such must be so here. But it fails to acknowledge that copyists also omitted material for various reasons (one being simply accidentally skipping a line in the hand copying). It would only take one such "error" to be used as a master copy for other manuscripts to perpetuate the omission to "a wide variety" of manuscripts.
Conclusion
So the decision is largely presuppositional. Both views #1 and #2 have "reasons" they hold to the presuppositions that they do, but those presuppositions largely determine which reading one chooses.
A discussion of driving factors in views can be found here (note: that is from a majority text viewpoint). For some contra argument, see here.