There are a number of indicators:
Themes
In the texts in Chapter 11 and earlier, all of the stories are about God's punishment of mankind. While the theme of salvation is present in these texts, there is also a theme of the depravity of mankind and their continual fall from grace. This theme isn't really present in the texts after Chapter 11 - only the continued theme of salvation.
We also begin the theme of the Covenant ("I will make you a great nation") in Chapter 13 which is not present in prior chapters. In the preceding chapters, God is dismantling great nations - not making them.
Scope
The texts in Chapters 1-11 are stories which affect and involve all mankind, but in the stories beginning in chapters 12 and onward, the only people really involved or affected are the characters in the story (the patriarchs and their immediate family) and we are to view them as cautionary tales with morals and values we should learn from them. The stories in 1-11 tend to tell us how things came to be the way they are and make pronouncements about the whole of humanity - even when taken hyper-literally (Eg, with the fall of Adam and Eve, all mankind has fallen - including those members of mankind who did not exist yet). The stories also tend to shift from generalities to specific events - For example, the story of the Tower of Babel doesn't even have a named character, it's just a generic, unnamed group of people. The generic stories also have relationships towards other myths and artifacts in Mesopotamia (Eg, Eridu Genesis, Enuma Elishe, Epic of Gilgamesh, Epic of Atrahasis, Creation Myth of Ptah, Creation myth of Atum, Creation Myth of Amun, Creation Myth of Heliopolis, Ziggurats, etc) While the stories may be retelling of the Hebrew Oral Tradition or the other way around, the are at least notable for their points of similarity and dis-similarity (Eg, the theme of the mound, Primordial waters, A great flood, the fact that mankind was created to rule earth, not serve the gods, the monotheism of Genesis vs. The polytheism of other stories). Clearly, these other mesopotamian texts share elements with the Biblical texts, but this ends in Chapter 11. From Chapter 12 onward, we have no such relationships to other texts and very little other archaeological relationships. These stories are uniquely Hebrew and do not have any relationships with other cultures in the Mediterranean.
Toldoth Formula
The Toledot, or seven generations of patriarchs begins in Chapter 11. Each generation is divided by a genealogy and genealogies are used to logically divide genesis into sections. Being as we have a genealogy between the story of the Tower of Babel and the story of Abraham, we can at least conclude there is some division there. This division is unique however, in that we get two separate genealogical segments. So, we have "This is the account of Shem." followed by "This is the account of Terah." - a double genealogical helping that we have nowhere else in Genesis. This tends to indicate that there is some kind of thematic division in the materials.
Conclusion
For these reasons, most scholars conclude that the two sections of the text are at least thematically distinct even if they do not believe they are of a separate genre (Though most scholars do believe they differ in genre. Genre examples: historical narrative vs. allegory vs. psalm vs. wisdom literature vs. prologue vs. poetry, etc.). Some scholars go so far as to say that they are from separate sources and some even speculate that Genesis 1-11 could have been a later addition to the text.
