There is no doubt that the commentators are in doubt!
Note the remarks of the Cambridge commentary:
20–24. Is it (a) Cyrus, as conqueror of Babylon, or (b) Babylon
herself, that is addressed? Jeremiah 51:24 seems to support (a), but
on the whole (b) is perhaps preferable. The future tenses can as well
be rendered as presents, denoting what Babylon has hitherto been
accustomed to do as the instrument of Jehovah. This view also
harmonizes with Jeremiah 51:14 (while we omit 15–19; see note there),
as well as with Jeremiah 51:25 ff., where Babylon is certainly the
subject.
Similarly, the Pulpit commentary:
The battle axe was much less frequently used. But who is addressed by
this terrible title? The commentators are divided, some inclining to
Babylon,
(1) because Babylon was the last person addressed (see ver. 14), and
(2) because a similar title was given to Babylon in Jeremiah 50:23:
others to Israel, on the ground that the tenses are the same
throughout the passage (vers. 20-24).
Let us examine the text very carefully. The key is actually in V19 of Jer 51.
The Portion [= the LORD] of Jacob is not like these, for He [= the
LORD] is the Maker of all things, and of the tribe of His [= the LORD]
inheritance— the LORD of Hosts is His name.
Recall that "LORD of hosts" could be translated "LORD of armies" (compare Josh 5:14, 6:1), that is, this title has a significant military connotation. This idea is taken up in places like Rev 19 where Jesus leads the armies of heaven. Now examine the next three verses with the clear understanding that it is the LORD of hosts that is the subject of V19 (the NKJV also capitalizes Portion, Maker, His)
20 “You are My war club, My weapon for battle.
With you I shatter nations;
with you I bring kingdoms to ruin.
21 With you I shatter the horse and rider;
with you I shatter the chariot and driver.
22 With you I shatter man and woman;
with you I shatter the old man and the youth;
with you I shatter the young man and the maiden.
23 With you I shatter the shepherd and his flock;
with you I shatter the farmer and his oxen;
with you I shatter the governors and officials.
Thus, grammatically, the "LORD of hosts" is the antecedent for all these pronouns, "you".
We have a similar situation in Zech 2:6-12 and Isa 48:11-16 where several times it is the the LORD who sends the LORD.