The Greek text of Heb. 9:4 is,
χρυσοῦν ἔχουσα θυμιατήριον καὶ τὴν κιβωτὸν τῆς διαθήκης περικεκαλυμμένην πάντοθεν χρυσίῳ ἐν ᾗ στάμνος χρυσῆ ἔχουσα τὸ μάννα καὶ ἡ ῥάβδος Ἀαρὼν ἡ βλαστήσασα καὶ αἱ πλάκες τῆς διαθήκης
The phrase in question is χρυσοῦν θυμιατήριον (chrysoun thymiatērion). Both the accusative and nominative declension are spelled θυμιατήριον.
However, does θυμιατήριον mean "altar"?
In the LXX, we find the phrase τὸ θυσιαστήριον τοῦ θυμιάματος (to thysiastērion tou thymiamatos) used to translate the Hebrew phrase מִזְבַּח הַקְּטֹֽרֶת, meaning "the altar of incense" (cp. Exo. 30:27).
Note the LXX of 2 Chr. 26:19:
καὶ ἐθυμώθη Οζιας καὶ ἐν τῇ χειρὶ αὐτοῦ τὸ θυμιατήριον τοῦ θυμιάσαι ἐν τῷ ναῷ καὶ ἐν τῷ θυμωθῆναι αὐτὸν πρὸς τοὺς ἱερεῖς καὶ ἡ λέπρα ἀνέτειλεν ἐν τῷ μετώπῳ αὐτοῦ ἐναντίον τῶν ἱερέων ἐν οἴκῳ κυρίου ἐπάνω τοῦ θυσιαστηρίου τῶν θυμιαμάτων
Here, we actually find both words: τὸ θυμιατήριον, which is used in reference to a "censer" used by King Uzziyahu to burn incense, and τοῦ θυσιαστηρίου (genitive declension of τὸ θυσιαστήριον) used in reference to the altar of incense itself.
We may conclude that the ESV erroneously translates θυμιατήριον as "altar" instead of "censer" (e.g., KJV).
Although the golden censer was not always located in the Holy of holies, it was brought within on Yom Kippur (cp. Lev. 16:12), which is the general context of that passage (cp. Heb. 9:7).