Indeed, κόσμος may have different meanings in different places, for it can mean, for example, all peoples of the earth (John 12:19), or the entire earth (Mark 16:15), or the entire universe including the visible/perceptible and the invisible (i.e. angels, souls of humans, invisible and intelligible laws of physics according to which the visible universe operates etc.) like in Romans 1:20.
Now, in this immediate context Jesus links the πρὸ τοῦ τὸν κόσμον εἶναι with the notion of "eternal life" (John 17:3) which in turn is linked to God the Father inseparably from the Son (ibid.), thus this "eternal life" belongs likewise to the Father and the Son, whereas κόσμος in the next sentence stands in opposition to the αἰώνιον (eternal) and is, thus, linked to the notions of temporality and createdness. Now, if it is impossible that the Father and the Son were eternally living without this divine life being invested with glory, for divine life is eternally glorious life, thus "glory which I had, before the world was, with Thee" must imply that Jesus speaks about the creation of the entire universe, the visible and invisible, because the eternal life and eternal glory relates to the Father and the Son and not to anything else, for even the angelic hosts, qua created, did not share the eternal life and the eternal glory with the Father and the Son.
However, in the immediate sequel, until John 17:24, when Jesus speaks about the love which the Father had towards Him "before the beginning of the world", indicating the eternal relationship of divine Hypostases, Jesus uses the word κόσμος in a more specific meaning, as the earthly world that is under dominion of evil and uncertainty (John 17:6; 17:9; 17:11; 17:13; 17:14; 17:15; 17:16; 17:18; 17:21; 17:23 and 17:25), thus, I think, only in two instances in this chapter the κόσμος is used in the sense of the entire universe, the visible and invisible.
Moreover, if it is true that the Son was with the Father before the creation of the entire universe, that is to say, eternally or co-eternally (for there's nothing between eternal and temporal, and since the time came to existence together with the creation, then the Son who is before the creation and who is the Father's means in creation /John 1:3; Hebrews 1:2;/, is absolutely co-eternal and co-divine with the Father), then why should Jesus qualify this in His saying that "He had glory with Father before the creation of the world", or that "He was loved by the Father before the foundation of the world" by putting in the term "world" only an aspect of the entire creation and not the entire creation itself?
For instance, just fancy, Nazi Germany invading a Belgian town, say, in 1941, and after some five weeks closing all local supermarkets and building only one supermarket called "Germany"; and after that a Belgian man sighs and says prayfully: "Oh Lord, when shall we get back the freedom which we had before Germany!" - now, he can imply both Germany as the invading country and Germany as the title of the supermarket and both will be justified, for the freedom was enjoyed by Belgians both before the foundation of the supermarket (admitted, not in the immediate 5 weeks' before though) and before the invasion of Germany. However, the context would rather make us suggest that the latter option, that is to say, the country Germany rather than the supermarket "Germany" is more plausible.
Similarly, even if both the entire universe (κόσμος I) and, therefore, also the earth and humans living in it (κόσμος II) is created by Jesus, why should He qualify His saying that He had eternal life and eternal glory and eternal love with the Father before the κόσμος II and not rather the κόσμος I any more than this imaginary Belgian man would sigh unqualifiedly about general German invasion in saying "before Germany" and not about a more specific notion with reference to the supermarket "Germany"?