וַיְהִ֗י אַחַ֨ר דִּבֶּ֧ר יְהוָ֛ה אֶת־הַדְּבָרִ֥ים הָאֵ֖לֶּה
אֶל־אִיֹּ֑וב וַיֹּ֨אמֶר יְהוָ֜ה אֶל־אֱלִיפַ֣ז הַתֵּֽימָנִ֗י חָרָ֨ה
אַפִּ֤י בְךָ֙ וּבִשְׁנֵ֣י רֵעֶ֔יךָ כִּ֠י לֹ֣א דִבַּרְתֶּ֥ם אֵלַ֛י
נְכֹונָ֖ה כְּעַבְדִּ֥י אִיֹּֽוב׃
And it was so, that after the LORD had spoken these words unto Job,
the LORD said to Eliphaz the Temanite, My wrath is kindled against
thee, and against thy two friends: for ye have not spoken of me the
thing that is right, as my servant Job hath. (KJV)
As the question points out, there is more than one way to translate this. In fact, in Hebrew, the preposition could mean: concerning, to, at, by, against, unto, within, in between, toward, in among, etc. There is a wide range of possibilities when it comes to prepositions, and in Hebrew, context is key.
Based on the context, we know that it applies to what these men have already said. Because we can see their words recorded in Job, we also know to whom they were speaking.
They were speaking to Job.
Therefore, we can eliminate "to me" (God) as a reasonable option.
We are left with two realistic possibilities: "of" or "about". Of these two, "about" is more specific. "Of" allows one to interpret in two ways: 1) it can mean "about"; or 2) it can mean "by". If, therefore, we are uncertain as to the original intent, our safest course is to translate it in a way that gives it the broadest sense.
In this case, "of" is the best option, as the KJV has it.