Asking "what came first" does not give an accurate portrayal of reality. Many things change in many small ways simultaneously. Over time, cultures and languages of relatively isolated people drift apart until we consider them to be different. There wasn't necessarily ever a cataclysmic event that suddenly changed everything overnight.
The Tower story was likely spuriously inserted among the geneologies as a post-hoc explanation for why people speak different languages. As with other Biblical interpolations, the redaction interrupts the flow of the surrounding text. Without the story, the geneologies read seamlessly as a continuous text.
Genesis has multiple cycles of construction and destruction. During the construction phase, an action of man is declared wicked, Yahweh destroys it, and the cycle repeats.
Gen 10 concludes the Flood story with the scattering of nations and a set of geneologies. Although Biblical geneologies may appear to be about individual people, they often really describe entire nations. (Recall 1Tim 1:3-4, ... so you may tell them not to teach other doctrines, nor myths and endless geneologies...) In the context of the Flood, it appears the initial division was by ancestry, from which different languages developed.
The next (short) cycle, in Gen 11, is the building of the Tower. Apologists often claim that the builders' crime is pride or disobedience, when they say, let us build a tower into the heavens, let us make a name for ourselves, let us not be scattered across the earth (Gen 11:4). But in modern vernacular, the builders are saying, let's build and name a city (make a name for ourselves) so that we can live together (not be scattered). "Tower into the heavens" is equivalent to the modern "skyscraper". Heavens is just another way to refer to the sky. "Into the heavens" is likely just and idiom meaning really high. (Recall Gen 1:20, Let birds fly above the earth across the dome of the heavens. Many translations say, across the sky.)
The crime, in Yahweh's eyes, is cooperation because (Gen 11:6) if as one people, speaking one language, they can accomplish this, nothing would be beyond them. As with the Flood, the story ends with the scattering of nations. In this case, division in speech appears to precede the development of nations. Another set of geneologies follows.