The Bible does often translate Elohim as gods (around 10% of the time), and indeed Isaiah more than most. For, example in Isaiah 21:9. However, the texts are normally ambiguous about the subject. In Isaiah those examples of "gods" are distinguished as such: 21:9, the images of the gods of Babylon, 36:20: the gods of these lands, 42:17, the gods of those who trust in Idols.
Isaiah 37:19 is worth quoting in full:
They have thrown their gods into the fire and destroyed them, for they
were not gods but only wood and stone, fashioned by human hands.
And chapter 41 deals with the same subject. Isaiah is clear: the gods of the Babylons are not real gods. They are powerless. For the prophet to suddenly go back on himself is too strange to be a sensible reading of the bare "elohim".
Remember the former things, those of long ago; I am God, and there is no other; I am God, and there is none like me.
Elsewhere, also in the Old Testament where elohim means "gods" there is a clear referent. In the Pentateuch, there are often references to "other gods" with "other" being used to distinguish them from God.
There is also a parallel description of the suffering servant, a few verses after 53:4 in 53:6
We all, like sheep, have gone astray, each of us has turned to our own way; and the LORD has laid on him the iniquity of us all.
This is the same he, and it describes the same suffering, however here it is using the divine name. Certainly, God is laying on the iniquity. In a different book, I think there could be a case to be made that 52:4 could be God using the powers and principalities to punish, but not in Isaiah that makes the powerlessness of idols such a central theme, not without some clear referent to resolve the ambiguity.
In conclusion, the grammar might permit it*. However, the context does not.
*I think that מֻכֵּ֥ה and מְעֻנֶּֽה take their singular forms from the singularity of the servant rather than Elohim, but I won't pretend to say with absolute confidence.