The context of James chapter 5 verses 7 to 12 has to do with patience in suffering. Also, the NIV translation speaks of Job's perseverance, not patience. Yes, Job persevered in the face of suffering - he endured, but no, he was not patient.
Job 16:1-3 is a prime example of Job's impatience with his so-called comforters. He railed against them and their long-winded speeches.
As for the context of Job 21:4, Job's complaint is against God because Job perceived that God was responsible for his suffering. Yet Job could not understand why he was being made to suffer.
As to the Christians to whom James wrote, he was asking them to be patient in the face of suffering (James 5:10). James was admonishing them to persevere, just like Job.
Job was patient in the face of all the suffering he endured, but there were moments when his impatience got the better of him. The ESV translation of James 5:11 puts it this way:
Behold, we consider those blessed who remained steadfast. You have heard of the steadfastness of Job, and you have seen the purpose of the Lord, how the Lord is compassionate and merciful.