It has been noted that the epistle of James has several small, but substantial, overlaps in thought and language with the 'sermon on the mount' (the Matthew version, not so much the Lukan 'sermon on the plain').
In the case of James 5.12, we see significant degree of overlap with this Jesus logia:
Again, you heard that it was said to the ancients, 'Don't swear
falsely,' but 'You shall give to Yahweh your oaths.' But I say to you,
don't swear at all, neither by heaven (for it is the throne of God),
nor by the earth (for it is the footstool for his feet), nor by
Jerusalem (for it is the city of the great King). Nor swear by your
head (for you cannot make one hair white or black). But let your word
be 'Yes, yes' or 'No, no'. And anything more than these comes from
evil. (Matthew 5.33-37, my translation)
The instruction in James and the Jesus logia have different contexts, and the Jesus logia is longer in form, but they are saying the same essential thing. In this case, it would be helpful to look at commentaries not just for James' epistle, but for the sermon on the mount.
On the passage in James, Hartin writes:
This command not to take oaths must be read against the background of
the Jewish understanding of the practice. On the one hand the Hebrew
tradition readily upheld the taking of oaths.
On the other hand, there is a tradition that shows a certain hesitancy
toward the taking of oaths in the Jewish tradition and an increasing
decline in the practice.
Concern for misusing the name of God in this way led to a variety of
ways of taking an oath that avoided the use of God's name: e.g., Philo
says: "However, if a man must swear and is so inclined, let him add,
if he pleases, not indeed the highest name of all, and the most
important cause of all things, but the earth, the sun, the stars, the
heaven, the universal world . . ." (Spec. 2:5; see also 2:2-38). See
also Philo in Decal.: "Next to not swearing at all, the second best
thing is to keep one's oath; for by the mere fact of swearing at all,
the swearer shows that there is some suspicion of his not being
trustworthy. Let a man, therefore, be dilatory, and slow if there is
any chance that by delay he may be able to avoid the necessity of
taking an oath at all; but if necessity compels him to swear, then, he
must consider with no superficial attention, everyone of the subjects,
or parts of the subjects, before him. . . . For an oath is the calling
of God to give his testimony concerning the matters which are in
doubt; and it is a most impious thing to invoke God to be witness to a
lie" (Decal. 84-86).1
The first Philo passage he cites above shows how, even among Jewish groups that preferred not to swear oaths, they would still sometimes take oaths 'by heaven' or 'by the earth', as found in both James 5.12 and the Jesus logia.
Hartin goes on to cite a passage where Josephus describes the Essenes as forbidding oaths, because they regard oaths 'as worse than perjury, for they say that one who is not believed without an appeal to God stands condemned already'.
It appears that James and the Jesus logia both stand in this tradition of thought, that oaths of all kinds should be avoided because they cast doubt on the swearer's capability to keep the oath, with the added implication that they are calling on God to stand behind the swearer's dubious capabilities.
1 Patrick J. Hartin. James (2003, 2009), p. 258-259.