Put simply, a symbol is a tangible representation of an intangible idea. Authors, both ancient and modern, use symbolic images to convey abstract concepts. The problem with symbols, however, is that they often appear as part of an implicit communication between author and audience. It's thus possible to misread an image as a symbol or a symbol as simply an image. Are there criteria to aid readers in determining when biblical authors use images symbolically?
Rabbi Samson Raphael Hirsch dedicated many essays to this subject. They may be found in English translation in his Collected Writings, Volume III; ISBN:0-87306-786-X.
The primary tools Rabbi Hirsch uses in his analysis are the following ground rules, which he develops in the introductory chapter (slightly abridged for convenience):
Taking the OP’s example question “[Might] the author of Jonah [have] intended his audience to see Jonah as symbol for the nation of Israel?”, we can submit it to Rabbi Hirsch’s criteria:
This analysis doesn’t rule out every possible interpretation of “Jonah as a metaphor for Israel” (though it leads me to doubt one can successfully be constructed), but it is a powerful tool for checking any such proposal.
Here are a few things that might help point the interpreter in the right direction. (NOTE: This answer is from a Christian perspective)
The referential nature of language
Language is referential. If I say "I own a house," any English-speaker will recognize that I am referring to a place of residence. However, if I said "I own a lamaroutous" that would be nonsensical, because that word does not currently refer to anything under the modern conventions of the English language. My audience would have no hope of understanding my speech. When we speak, we have to use words that are known to our intended audience.
Referential language works fine for day-to-day life in this world; I can "arise" from my "bed" and "eat" some "cereal" before I "read" the "newspaper"... and I can communicate that to virtually any English-speaker without a problem.
The purpose of a symbol
Communication is more difficult when you are attempting to describe something unfamiliar. For instance, when God attempts to describe future and/or spiritual realities to a local audience that is neither future nor spiritual, it becomes necessary to speak in familiar terms while describing the unfamiliar. This is where symbols come in. A symbol allows you to refer to something unfamiliar by way of a familiar term.
For example, God promised to send "Elijah" before the Day of the Lord. Jesus identified John the Baptist as the Elijah who was to come. If the prophecy had said that God would send "John the Baptist" that would be nonsensical, since no one would have had any idea what that term referred to! By saying "Elijah," God is able to call to mind something familiar which could give the audience a sense of what kind of person to expect.
So, start by examining the passage to determine whether the speaker is intending to communicate something unfamiliar (e.g. future or spiritual) in familiar terms.
In general, properly identifying the genre of Biblical literature is crucial to interpreting it correctly. Just as we must be careful not to confuse a comic strip with a news headline in the local newspaper, we must be cautious of confusing apocalyptic literature with historical narrative, or parable with discourse (etc.)
This is helpful for identifying symbols, because symbolism is much more common in some genres than others. For example, apocalyptic revelation typically comes in the form of dreams and visions, which tend to be highly symbolic, versus historical narrative, which is almost never communicated through symbols. (Although the events themselves may have been divinely intended as "symbolic"!)
God's focus is spiritual
This may perhaps make more sense to Christian interpreters than Jewish or secular, but God's ultimate concern is for the redemption of mankind, and that is a spiritual redemption. It was never really about moving a group of blood relatives to a new geographical location and helping them slay the other people around them. God's heart was never in slaughtering animals, or throwing rocks at sinners.
God's heart has always been for the redemption of mankind. His mission is to restore mankind to relationship with Himself. Ultimately, everything God says and does serves this purpose. Thus, anytime we see God speaking about something natural and temporal (i.e. non-eschatological) we should suspect that it is symbolic. And anytime we see God doing something natural and temporal, we should suspect that it is typological (which could also be called "symbolic.")
Jonah, for example
In order to confidently identify an image as a symbol, we must be able to show that the author (or Author) intended it as such. This can be as simple as showing from the text that the human author was using a metaphor. For example, in the opening chapter of John the narrator writes:
Here we can clearly see John using "Light" as a metaphor (a symbol) of the Son of God.
Since we are here wondering whether Jonah himself is symbolic, or whether the story as a whole is a spiritual commentary, we could also explore the possibility that God meant His sovereign actions there as symbolic -- just as God used the Exodus symbolically to teach us spiritual realities.
Again, given God's agenda, this is highly likely. The alternative -- that it is merely an ancient story of a repentant country and a prophet with a bad attitude -- is more difficult to sustain. It probably is "symbolic" in some sense.
This does not suggest that the story of Jonah is not true, or historically accurate -- it merely suggests that what God did there pointed to something future and spiritual. It pointed to God's ultimate concern: the redemption of mankind.
If an author intends to communicate through symbols he must either rely upon a community's preexisting symbolic language or make an effort to define the meaning of the image within the text itself. Ruben Zimmermann in his book Imagery in the Gospel of John thus offers two criteria for weighing a symbols plausibility: (1) conventional plausibility and (2) textual plausibility. He returns to these two criteria in Anatomies of Narrative Criticism
When an author appeals to convention by assuming his audience will recognize his meaning he or she draws a curtain between his group and those outside his community. The only way for present day outsiders to peek behind this curtain is to acquaint themselves with the common sources from which this community derived its symbols. For instance, John’s symbolic language like the Greek language he speaks arises in part from his social setting. Giving heed to the material that evidently played a part in his writing can supply ample information for symbolic investigation. Thankfully, with regards to the religious books of the Bible, their sacred texts are by and large still with us today. The plausibility of a proposed symbol is thus first weighed by its continuity with known scriptural convention.
But authors also define their symbols within the text. This occurs in at least two ways. First the author can use the narrator or characters within the story to make an explicit connection or comparison between two unlike things. A good modern example of how this shift takes place can be found in the movie The Shawshank Redemption. Notice how Andy links music to hope and Red more specifically to a harmonica.
Later in the film the mere image of a harmonica becomes a symbol by invoking the metaphor of the previous conversation. Andy gives Red a harmonica as a “parole rejection” present. When asked if he’s going to play it, Red responds, “no, not right now.” The gift has moved beyond a mere object and now points to the hope which Andy provides and Red doesn’t want to let in.
In John's gospel the symbols most easily recognizable are those found first in metaphor. “I am the light of the world” Jesus says. The incongruity of Jesus speaking of one thing in terms of another pushes the reader passed a literal meaning to reconcile meaning abstractly. Metaphors clearly denote John’s core symbols, images that occur frequently and contribute most to the gospel's message. For instance Christ’s claim to be “the light of the world” establishes light as a symbol. Its frequency and placement underscore its vital importance (1:9; 3:19; 8:12; 9:5; 12:46).
A second way an author can define a symbol is through narrative structure. For instance, Mark, the earliest of the four New Testament gospels, records the following scenes in this order.
The sandwiching of these stories indicates that the fig tree is a symbol of the temple. The cursing of the fig tree and its subsequent withering represents Jesus attack on the temple and its subsequent destruction. Jesus' later teaching on the mount of Olives (Matthew 24, Mark 13, Luke 21) has this meaning in mind. Here, Jesus appeals to the meaning of the fig tree.
One further thought. The symbols arising from the original metaphor or structural association need not be restricted to their specific language. For instance if Red had never made the explicit connection between music and a harmonica the harmonica still would have been a plausible symbol for hope in the proceeding scene. That's because a harmonica is a subset of the larger concept of music. For example, later in the Shawshank Redemption, we find Heywood listening to Hank William's records in the Library Andy has built. While Hank Williams has not been explicitly connected to the metaphor his music falls under the same category.
This occurs repeatedly in John's gospel. Beyond metaphors to light we find linked references to things like darkness, day, night, blindness, and sight. The conceptual link to a central symbolic image suggests that these images likewise are to be understood symbolically. Philip Wheelwright has observed that many symbols have “a bright focused center of meaning together with a penumbra of vagueness that is intrinsically ineradicable.” The core symbol established in metaphor and clearly defined in context acts as the “bright focused center” while the linked images appear to radiate out in more or less decreasing precision.
Some of these images are more transparently symbolic than others. For instance, when the statement “men loved darkness rather than light” appears at the end of Jesus’ dialogue with Nicodemus it suggests that Nicodemus’ approach by “night,” is more than simply setting. Likewise the regular reoccurrence of light in John indicates a similar symbolic sense for Judas’ final departure (13:30) though only “night” is mentioned in the immediate context.
Of course not all images related to a core symbol have this probability. Craig Koester states,
Frequency and or context are once again the clearest guides to establishing likelihood. For instance, John’s light motif may play an ironic role in the solder’s use of lanterns and torches to arrest Jesus, “the light of the world.” However, because lanterns and torches are not mentioned or connected elsewhere in John with the light motif, the probability of an intended symbol, though good, is not as great.
As is the nature of historical fields, such as the anthropological and linguistic fields which serve as our surest means of understanding such colloquial forms of language, at least some uncertainty or ambiguity will always haunt our attempts at determining the meaning of symbols, or most any other metaphor for that matter. Rationally speaking, it is a matter of continuing to uncover and learn the ancient contexts, psychologies, and uses of languages. But there is also a means of understanding some symbols by less rational (though not therefore irrational: merely arational) sources of knowledge, such as artistic perceptions like the sense of the aesthetic, of the narrative, of the intuitive and empathic, etc. And of course it is also a matter of spiritual discernment and prayer, which is unfortunately not so often emphasized in theology and hermeneutics as one would expect. Christian academic subjects they may be, but Christian they must remain!
In short, there is no easy answer. This, like so many other linguistic difficulties in Biblical hermeneutics, are a matter of diligence, of skill, of art, of happenstance (or providence, as it may be), and perhaps even of revelation.
NOTE: This answer was written in response to the original form of the Question, which appeared to me to be seeking a definitive rule by which to identify symbols with certainty. I have edited this Answer to align a little more closely with the now-edited Question, but in deference to the original purpose of this Answer and to avoid redundancy, I will leave to others to address the exegetical concern, as Jas 3.1 has done so well already.