ὀλιγόπιστος (oligopistos) occurs only six times in the NT, Matt 6:30, 8:26, 14:31, 16:8, 17:20, Luke 12:28 and always means:
- BDAG: of little faith/trust
- Thayer: of little faith, trusting too little
Now, NA28, NA27, UBS4, UBS5, W&H, etc, all have ὀλιγόπιστος in Matt 17:20; however, the Byzantine text, Majority, GOC, TR, all have ἀπιστίαν (= unbelief) here, and because the KJV follows TR, it has "unbelief". For a discussion of why oligopistos is preferable, see UBS5.
The sense of the verse is certainly "little faith" because Jesus goes on, in the same verse, discussing faith as small as grain of mustard seed, etc. We note that "unbelief" is not consistent with the flow of the text because Jesus' audience was not without belief!
Barnes observes about this dialogue:
The mustard-seed was the smallest of all seeds. It has been supposed
by some, therefore, that he meant to say, If you have the smallest or
feeblest faith that is genuine, you can do all things. The
mustard-seed produced the largest of all herbs. It has been supposed
by others, therefore, to mean, If you have increasing, expanding,
enlarged faith, growing and strengthening from small beginnings, you
can perform the most difficult undertaking. There is a principle of
vitality in the grain of seed stretching forward to great results,
which illustrates the nature of faith. Your faith should be like that.
This is probably the true meaning.
The Pulpit commentary adds more:
They had, indeed, shown some faith by making even the attempt at the
expulsion of the devil, and were not to be classed with the
unbelieving scribes; but they had acted in a half-hearted manner, and
had not displayed that perfect confidence and trust which alone can
win success and make all things possible ... If ye have faith as a
grain of mustard seed, which, as he says (Matthew 13:32), "is less
than all seeds." He means a faith real and trustful, though it be
small and weak. The phrase is proverbial, expressive of littleness and
insignificance. The mustard seed is quite little, but, grown in
favourable soil and under sunny skies, it becomes, as it were, a tree
among herbs, so that birds may nestle in its branches. To it faith is
compared, because, small at first, it contains within itself power of
large development and increase; from minute grains copious results are