Unless they have the free will, and henceforth the real opportunity not to do iniquity, then it is unjust for God to punish them. Period.
In fact, for what misdeeds does God punish men?(I speak now from a profane perspective, for God does punish nobody, for all are punished already by their own sins for sins alienate us from God and what can be a greater punishment than alienation from God? If God will augment this punishment by some additional sadistic torment from outside, such a God, or rather, "god", would indeed have a very vulgar taste.);
but, again to return to the profanely asked question: for what does God punish men? Of course first of all for unmercifulness and iniquity, because we read that Jesus Christ will say to those who smugly think that paradise is guaranteed for them: "I was hungry, and you did not feed Me; I was thirsty, and you gave Me no water; go to hell now, you all, who behaved so unmercifully". But if there is no free will, then at those words of Jesus a committed Calvinist will just smile and say: "But, actually, dear Lord, it is you who has to go to hell, by the way!" And Jesus, bewildered, will ask: "Me? In hell? It is oxymoron!, How dare you?" And the Calvinist will chuckle: "Very neatly: for did not You yourself make me a vessel of wrath, which means that I could not not behave as I have behaved, that is to say, I could not do mercy to those people!" "So?" "And so, since You are the cause for my not doing those merciful deeds, then You are also the cause of my mercilessness, and therefore the primary culprit for that! Thus, as the primary culprit and the principal cause of my unmercifulness, it is You who are to go to hell, aren't You?" "In fact, your logic works, but then it turns that you are a victim of My whim and thus My guilt is not only the unmercifulness, but that I have doomed poor you to become an instrument of it!" "Yes, dear Lord, yes! That's my point!" "Ok, I go to hell then, but I do not know what to do with you, for you haven't done any good things and paradise thus is not for you!" Calvinist, beaming even brighter: "I will help You! In fact, those having done good deeds, who are now in paradise, they also should not be there!" "What? You mean, Abraham?" "Yes, him also!" "And Isaak?" "Isaak too!""But, why?" "Because, their good deeds is not their merit, but Your singular merit, You made them to commit them, so only You deserve paradise, whereas both they and me should stay out of paradise, but also out of hell!" "Thus, as I see, we arrive at a paradoxical situation, an impasse, in fact: I, as the principal cause of both mercifulness and the mercilessness should go both to hell and paradise simultaneously, whereas all of the people, both merciful and merciless, should go to somewhere that is neither hell nor paradise! Wow!" "Indeed, wow! But since You now see, o Lord, that You must construct for us all a neither-hell-nor-paradise place, make sure to make it comfortable, at least 5-star hotel level in earthly terms, with sauna and wellness centre" etc etc.
I don't know how far this absurd talk would go, you can phantasise at will, but just to cut it short, let us simply assert that there is free will and responsibility for both attaining paradise or falling short of it on the part of all humans.
Paul is not Augustine in the latter's sorry extremes against Pelagius, or Kalvin, who followed this extreme Augustine in such a callous logical consistency that sacrificed the very merciful God to this logic! Paul's mentioned passage, comparing God to a potter and men to pots of either wrath or mercy, with no possibility on their part to change anything, is heavily contextual (I do not go into explaining it, but of course I can, for I will not leave probably the most genial an influential apostle to the calumny of being as unsound as to introduce a capricious god hypnotising hapless humans volens nolens to be vessels of his wrath), for even in Romans Paul is emphatic that humans are responsible for their destiny, see Romans 2:6 ("[God] who will give to each according to his deeds") for just one example out of many, to say nothing about other epistles of Paul, in which this point is no less emphatic!