List of Firstborns? Not Necessarily
In the Sepher Ṯōl'ḏōṯ, or Generations Book, of Genesis 5 the very first son (Seth) listed is, as is evident from Genesis's immediately preceding chapter, actually not a firstborn son. Seth is Adam's third son, who, as the story apparently says in Ch. 4, is born when his brothers Cain and Abel are already full-grown (indeed when one of them has already died most likely in adulthood, as per vv. 8-15).
Down Seth's line, similarly, there shouldn't be any reason to conclude that necessarily any other son named in this sepher, prior to Noaḥ's offspring, is a firstborn anything. It could just as well mean, like in the premier instance [i.e. Seth], that every one of them has older brothers (who remain unmentioned in the sepher for whatever reason). The only thing the formula is explicitly saying about birth-timing is how old the son's father was when the son was born, not whether it was the first son or not. The father definitely has children after the named son, but this does not preclude him from having sired other—and perhaps a great many—children by that point already.
Sequencing the Offpsring
It would seem that the order in which the names of Noaḥ's sons are almost always listed (see Chs. 5-7, & 9), "Shem, Ḥām and Japheth", has more to do with their relationship with Israel, as ancestors, rather than their birth order.1 Shem and Ḥām are both Israel's progenitors (or Ḥām is the father of at least some parts of Israel) and among Ḥām's descendants are both friend and foe to Israel. Japheth has looser connections which become apparent only much later on in the chronology. (And incidentally the Ṯōl'ḏōṯ Benē Noaḥ, "Generations of Noaḥ's Sons" [in Genesis 10], are listed in reverse sequence from the trend thus far, starting with Japheth's sons, going forward to Ḥām's and then ending with Shem's offspring.)
Based on deductions which can be made from the Ṯōl'ḏōṯ Shem, "Shem's Generations" in Ch. 11, as well as the potentially problematic statements in Genesis 9.24 & 10.21, it is possible to at least safely conclude that Noaḥ's sons cannot be triplets and also that possibly this is the order of their birth, from first to last: Japheth -- Shem -- Ḥām.
TRIPLETS or No?
AmazingBibleTimeline.com's article "Are There Triplets In the Bible?" deals with a similar question regarding Abraham and his brothers but also has this to say about Noaḥ's more immediate family:
Shem begat Arpachshad when
he was 100, and Arpachshad was born two years after the flood (Genesis
11:10). We know that Noah was 600 years old at the start of the flood
(Genesis 7:6). So what do we get from this? If Shem were born when
Noah was 500 he would have been 100 the year of the flood but he isn’t
100 until two years later when Arpachshad is born. That means Shem was
born when Noah was 502.
Gen. 8.13, however, seems to imply that the flood-waters had dried up just when Noaḥ had turned 601, and if that's the case then Arpachshad may even have been born when Noaḥ was 603, and thus Shem was born when Noaḥ was 503. (Albert Barnes makes a similar calculation working out to this figure.)
Gen. 5.32 is usually interpreted to mean that at least one of Noaḥ's sons must have been born when Noaḥ was 500. By way of elimination, this could not be Shem (who is therefore at least 2 years younger than the firstborn), which leaves Ḥām and Japheth.
Canaan "The Little Son"?
The end of Genesis 9 relates an ambiguous narrative in which Noaḥ pronounces a curse (which per se is a huge kettle of other fish, as it were) against his own grandson Canaan, listed last among Ḥām's 4 sons in v. 6 of the succeeding chapter. Going by popular English translations like the KJV, & Douay & Webster's Bibles, Gen. 9.24 says that Noaḥ, before cursing Canaan, "knew/learned what his younger son had done to him." More recent, even more popular translations like the NIV, NLT & ESV say "youngest son". Haq-qāṭān, the adjective translated here, is used later in reference, in Ch. 27, to Jacob (the younger of Rebekah's two sons) and in Chs. 42 & 43, to Benjamin (the last of Jacob's 12 sons). And its Ancient Greek translation (from the Septuagint [LXX]), neóteros, does tend to mean "younger."
Notwithstanding that, exactly what is happening at the end of Gen. 9 is not all that obvious to determine. Contrary to much elaborate mythology especially over the last several hundred years, nowhere in the Bible does Noaḥ ever even mention Ḥām's name, much less curse him together with all his descendants. He does, however, curse Canaan, and quite bitterly at that. This has led many commentators including Charles Ellicott, Joseph Benson and Matthew Poole to the conclusion that Canaan, rather than Ḥām, must be Noaḥ's ben haq-qāṭān in this instance, noting, as does John Gill, that in Hebrew ben can mean "grandson" or "descendant" as much as "first-generation offspring." (A similar phenomenon occurs in other languages as well.)
In this view Gen. 10.6 is almost invariably taken to mean that Canaan is literally the lastborn of Ḥām's sons. And so then if Ḥām is Noaḥ's lastborn then this argument regarding the meaning of qāṭān / neóteros is bolstered by the interpretation "youngest son of [Noaḥ's] youngest son".
Japheth "The Big Brother"
Taking all that into account would make Japheth the one who was born when Noaḥ was 500. And thus Ḥām would've been born after Noaḥ was 502 or 503, following Shem's birth.2 In the Hebrew of Gen. 10.21 Shem is the "brother of Japheth hag-gāḏōl", for which the LXX has "Japheth meízonos", which in English translations, almost unanimously, makes Japheth "Shem's older/elder/eldest brother".
Meízonos is a superlative which can mean greater, larger, longer, taller or older. In its substantive form it can mean an elder, like the leader of a village. Hag-gāḏōl literally means "the great one" just as haq-qāṭān is "the little one." Cf. how the High Priest is hak-kōhēn hag-gāḏōl, "the Great/Big Kōhēn" (e.g. in Numbers 35.25), or the Mediterranean is hay-yām hag-gāḏōl, "the Great/Big Sea" (e.g. in Numbers 34.6-7). So it could be just as legitimately translated "brother of Japheth the greater"3 or "brother of Yaphet the magnificent".4
At any rate the passage does not explicitly indicate that Japheth is older than Ḥām, and the only evidence for this deduction is the cryptic reference in Gen. 9.24. Most of the aforementioned commentators note that Ḥām is always the second brother mentioned in the enumerations of Noaḥ's three sons, and thus have trouble seeing him as literally the lastborn. The only alternative to that option is that he is the firstborn. If that's the case, there is a potential narrative parallel later in Genesis5 for the conclusion that he, like Jacob's firstborn Reuben, could have lost his position of prominence (for whatever reason[s]).
Maybe Twins?
The term hag-gāḏōl occurs 5 other times in Genesis, the 1st time to refer to the sun as greater or bigger than the moon (Gen. 1.16), another time in reference to a river in the Near or Middle East (Gen. 15.18) and all the other times to indicate Esau as Jacob's older twin brother (at the beginning, in the middle & at the end of Ch. 27). I wonder how fair it would be infer from this that Shem and Japheth are twins. It certainly would expound on their closeness in Chs. 9 & 10.
At any rate, in my understanding of Gen. 7.6 & 11.10, what is certain is that Shem is younger (in the literal sense, and by years rather than minutes or hours) than at least one of his brothers.
1. In his Notes On the Whole Bible Albert Barnes asserts the same thing as "the historical relationship subsisting among the nations descended from them".
2. See, again, Barnes' Notes [as in prev. footnote].
3. The Apostolic Bible Polyglot (ABP)
4. The Ancient Hebrew Research Center's Revised Mechanical Translation (AHRC-RMT)
5. Ch. 35, vv. 21-22 to connect with Ch. 49, vv. 1-4
