There are three reasons which can account for the change from דָּ֣ג (and הַדָּ֔ג) in verse 1:17 to הַדָּגָֽה in verse 2:1 and back to לַדָּ֑ג in verse 2:10:
- Literary
- Natural
- Symbolic
Literary
As Abu Munir Ibn Ibrahim al Yahud explains, this section is poetic and dāḡâ "fits" better than dāḡ. Thus, the writer employed "poetic license." Obviously, from the description of events, "the fish" is the same fish which swallowed Jonah, something which is reinforced at the close of the passage when, the fish הַדָּ֔ג vomits him out on to dry land.
In addition, the male form is used to describe that which was prepared by YHVH and that which YHVH commanded to vomit Jonah onto dry land.
Now the LORD had prepared a great fish (דָּ֣ג) to swallow Jonah. And Jonah was in the belly of the fish (הַדָּ֔ג) three days and three nights. (Jonah 1:17 NKJV)
So the LORD spoke to the fish (לַדָּ֑ג), and it vomited Jonah onto dry land. (Jonah 2:10)
That is, the literary effect is not simply poetic; it also is used to separate the word of YHVH from the word of Jonah (his prayer).
Natural
There are 500 spices of fish which have the condition called hermaphroditism. That is, some fish may properly be described as male and later female, or vis versa. If the "great" fish was one such species, it would account for the use of both male and female term to describe the same fish.
While male and female is the norm in nature, the creation account states only humans were created "male "and "female;" all forms of life were created "according to their kinds."
Symbolic
After being swallowed, Jonah was transformed from a prophet who refused to do as YHVH instructed, to one who gave thanks and promised to pay his vows recognizing, salvation of YHVH.
But I will sacrifice to You With the voice of thanksgiving; I will pay what I have vowed. Salvation is of the LORD. (Jonah 2:9)
In this sense, Jonah was transformed while in "the belly" of the fish. "Belly" is מֵעֶה which can mean "womb." The specific form is מִמְּעֵ֖י, which is masculine, but is always otherwise used with a female:
By You I have been upheld from birth;
You are He who took me out of my mother’s womb.
My praise shall be continually of You. (Psalm 71:6)
עָלֶיךָ נִסְמַכְתִּי מִבֶּטֶן מִמְּעֵי אִמִּי אַתָּה גוֹזִי בְּךָ תְהִלָּתִי תָמִֽיד
“Listen, O coastlands, to Me,
And take heed, you peoples from afar!
The LORD has called Me from the womb;
From the matrix of My mother He has made mention of My name. (Isaiah 49:1)
שִׁמְעוּ אִיִּים אֵלַי וְהַקְשִׁיבוּ לְאֻמִּים מֵרָחוֹק יְהוָה מִבֶּטֶן קְרָאָנִי מִמְּעֵי אִמִּי הִזְכִּיר שְׁמִֽי
Additionally, the first use of מֵעֶה is a promise made by YHVH to Abram:
And behold, the word of the LORD came to him, saying, “This one shall not be your heir, but one who will come from your own body shall be your heir.” (Genesis 15:4)
וְהִנֵּה דְבַר־יְהוָה אֵלָיו לֵאמֹר לֹא יִֽירָשְׁךָ זֶה כִּי־אִם אֲשֶׁר יֵצֵא מִמֵּעֶיךָ הוּא יִֽירָשֶֽׁךָ
The heir, Isaac, not Ishmael, came from Abraham's body and Sarah's womb, not Hagar's. Thus, in the same way the heir can be promised to come from Abram's מֵעֶה which must also mean Sarah's מֵעֶה; Jonah's prayer, in which he asserts, יְשׁוּעָתָה לַיהוָֽה, salvation of YHVH, while in מִמְּעֵי הַדָּגָֽה the womb of the fish despite being in בִּמְעֵי הַדָּג, the belly of the fish for three days.
The language is symbolic but prophetic to the plan of salvation where the Son of God must physically come from the womb of a virgin woman who is betrothed to a man, in word only at the time of His birth.