As you point out, ἄγγελος literally means messenger and could be translated either as such or angel (i.e. angelic being).
An explanation assuming the latter is that Jesus is contrasting the earthly witness of earthly beings (i.e. men) with the heavenly witness of heavenly beings (i.e. angels) that will be present at the eschaton. We also have Matthew's description of angels being present at the final judgment:
When the Son of man shall come in his glory, and all the holy angels with him, then shall he sit upon the throne of his glory: And before him shall be gathered all nations (Matt 25:31-32).
Confess (ὁμολογέω) here could also be translated as acknowledge (see, e.g. Matt 7:23 YLT). In Luke 12:9 ἀρνέομαι might also be translated as disown rather than deny (as is the YLT rendering later in Luke 22:57). The two in this context could be taken as antonyms: acknowledgement and disownment are the two possible outcomes of the judgment.
Basil the Great comments:
It is customary for the saints to deliver the commandments of God in the presence of witnesses, as also the apostle himself says to Timothy, The things which thou hast heard of me among many witnesses, the same commit thou to faithful men (2 Tim 2:2) and now he calls the angels to witness, for he knows that angels shall be present with the Lord when He shall come in the glory of His Father to judge the world in righteousness. For He says, Whoever shall confess me before men, him shall the Son of Man also confess before the angels of God, but he that denieth Me before men shall be denied before the angels of God and Paul in another place says, When the Lord Jesus shall be revealed from heaven with his angels (2 Thess 1:7). Thus he already testifies before the angels, preparing good proofs for himself at the great tribunal.1
Basil also calls attention to a number of Old Testament references that allude to the convocation that will take place at the heavenly judgment:
And not only Paul, but generally all those to whom is committed any ministry of the word, never cease from testifying, but call heaven and earth to witness on the ground that now every deed that is done is done within them, and that in the examination of all the actions of life they will be present with the judged. So it is said, He shall summon heaven above and the earth that He may judge His people (Ps 50:4). And so Moses when about to deliver his oracles to the people says, I call heaven and earth to witness this day (Deut 4:26) and again in his song he says, Give ear, O ye heavens, and I will speak, and hear, O earth, the words of my mouth (Deut 2:21) and Isaiah, Hear, O heavens, and give ear, O earth (Is 1:2).2
1. "Statement of the reason why in the writings of Paul the angels are associated with the Father and the Son", On the Spirit XIII; in Nicene and Post Nicene Fathers 2-08