The Hebrew text וְעִנִּיתֶם אֶת נַפְשֹׁתֵיכֶם (alternatively תְּעַנּוּ אֶת נַפְשֹׁתֵיכֶם) literally says "and ye shall afflict your souls," and it appears in two separate sections in Leviticus as well as once in Numbers. This text has always been understood to mean afflict your body through fasting. That being said, there is also a spiritual component to Yom Kippur in additional to the physical fast itself. A fairly substantial amount of data from Tanach supports this assertion. A good example is Isaiah 58:3 which reads:
לָמָּה צַּמְנוּ וְלֹא רָאִיתָ עִנִּינוּ נַפְשֵׁנוּ וְלֹא תֵדָע הֵן בְּיוֹם צֹמְכֶם תִּמְצְאוּ חֵפֶץ וְכָל עַצְּבֵיכֶם תִּנְגֹּשׂוּ:
Why have we fasted but You (G-d) have not seen; we have afflicted our souls but You do not know? Behold, on the day of your fast you pursue business, and [from] all your debtors you exact [payment].
In this example, the verb צמנ (fast) is clearly being equated with ענה (afflict).
Furthermore, the Hebrew word נֶפֶשׁ, in addition to meaning "soul," also has a physical meaning related to the human body. Psalms 107:9 is a good example of this:
כִּי הִשְׂבִּיעַ נֶפֶשׁ שׁוֹקֵקָה וְנֶפֶשׁ רְעֵבָה מִלֵּא טוֹב:
For He (G-d) has sated a longing soul, and filled a hungry soul with good.
Above we can see that the dual spiritual/physical meanings of נֶפֶשׁ are being used in the very same verse. So afflicting your soul on Yom Kippur means physically fasting, and being spiritually affected by that fast as well.
Here is an excellent article which discusses this topic in more detail.