Katachthonios in Ancient Greek Literature
Literally speaking, καταχθόνιος just means 'under the earth', from the prefix κατά (down) and χθόνιος ([under] earth). But it seems to have been coined with a specific referent:1
Persephone, goddess of the underworld, is daughter of Demeter, goddess of corn, and Zeus Chthonios, the farmer's friend, can scarcely be dissociated from that Zeus Katachthonios whom Homer represents as ruling alongside Persephone.
This 'Zeus Katachthonios' was not the Zeus, but actually Hades, given a name that meant 'Zeus of the Underworld'.2 This immediately associates καταχθόνιος with the dead.
The Perseus Digital Library lists καταχθόνιος as appearing less than twenty times in ancient Greek literature, one of those occasions being Paul's use here in Philippians.
About three or four decades before Paul, Strabo used the word when referring to offerings made to καταχθονίους δαίμονας, 'infernal deities' (Geography 5.4.5) and καταχθονίοις θεοῖς, 'infernal divinities' (6.2.11).
Dionysius of Halicarnassus, a contemporary of Strabo's, uses it essentially the same way, but he also directly uses the word to describe the afterlife of humans:
And if there is in very truth a place which will receive men's souls when released from the body, it is not that subterranean [καταχθόνιος] and gloomy place where, men say, the unhappy dwell, that will receive mine, nor the region called the Plain of Lethe ... (Roman Antiquities 8.52.4)
According to the Theological Dictionary of the New Testament:3
katachthónios is common in secular Greek relative to the underworld. In the NT it occurs only in Phil. 2:10 in a phrase denoting, without further specification, the totality of beings (cf. Rev. 5:13). To attempt classification, e.g., by seeing in the katachthónioi the dead who rest in the earth, is to miss the poetical-liturgical nature of the passage.
Though the entry suggests Paul's usage is 'poetical-liturgical' and argues against a meaning referring to 'the dead who rest in the earth', it's hard to miss how he uses the word alongside ἐπουρανίων (in heaven) and ἐπιγείων (on earth) and the close parallels in Revelation 5.3 and 13. Verse 3 reads:
And no one in heaven [ἐν τῷ οὐρανῷ] or on earth [ἐπὶ τῆς γῆς] or under the earth [ὑποκάτω τῆς γῆς] was able to open the scroll or to look into it.
And Revelation 5.13:
Then I heard every creature in heaven [ἐν τῷ οὐρανῷ] and on earth [ἐπὶ τῆς γῆς] and under the earth [ὑποκάτω τῆς γῆς] and in the sea, and all that is in them, singing, ‘To the one seated on the throne and to the Lamb be blessing and honour and glory and might for ever and ever!’
John clearly intends to summarize all the tiers of God's creation, and groups them in a way similar to Paul. But John includes another realm, 'the sea'; he refers to creatures 'in' 'the sea' one other time, in Revelation 20.13: the dead are raised for judgment, and come out of 'the sea' as well as 'Death and Hades'. It appears that John uses 'under the earth' to describe Hades, the sphere of God's creation that encompasses the dead.
Based on all this, it is apparent that Paul's intention is to give a comprehensive summary of all creatures in all creation, where καταχθόνιος refers to the 'underworld' and all the dead dwelling there.
Regarding Tartarus
The term Τάρταρος (Tartarus), like καταχθόνιος, is used only once in the new testament, by the author of 2 Peter. This author, describing sinful angels imprisoned in Τάρταρος, wrote his letter circa AD 120, several decades after Paul had already died. Τάρταρος was sometimes associated with Hades, but it was also sometimes said to be even further 'under' the underworld.
While it is possible Paul intended καταχθόνιος to further include 'demons' or 'fallen angels' in addition to the dead, we have no way of knowing for sure if his conception of καταχθόνιος included or overlapped with Τάρταρος. Because 2 Peter was written quite a while after Paul was alive, it would be difficult to use it to retroactively interpret Paul's meaning in Philippians 2.10.
Footnotes
1 Robert Parker, On Greek Religion (2011) 81.
2 W.K.C. Guthrie, The Greeks and Their Gods (1950) 217.
3 Gerhard Kittel, Gerhard Friedrich, Theological Dictionary of the New Testament: Abridged in One Volume (1985) 421.