Is there a difference in historical records of a prostitute's veil and a common veil worn by women of that time? How did Judah know that Tamar was prostituting herself?
Answers to a more general question along these lines at Judaism.SE suggest that it wasn't the veil that identified Tamar as an harlot. It was only the means by which Judah failed to recognize her.
Her answer to his proposition suggests that her actions were one recognizable with prostitution (she didn't take offense to his proposition, and one must assume he wasn't propositioning every woman he saw, so something must have caught his eye). As for what those actions were, the Bible narrative does not say... but men and women have been acting suggestively toward one another since almost forever. This isn't a stretch of the imagination.
Judah was able to identify Tamar as a prostitute, not because a prostitute's veil was different from an ordinary one, but primarily because she was wearing one.
While there is variety of opinion regarding veiling and covering of the head in Biblical times, it seems likely that women would normally cover their hair in public, but not necessarily veil the face.
It was a shame for women to expose their hair in public (compare Num 5:18), yet women among the Jews ordinarily appeared in public with faces exposed (Genesis 12:14; Genesis 24:16; Genesis 24:65; Genesis 20:16; Genesis 29:10; 1 Samuel 1:12).
Notice, particularly, Gen 24:5-6:
64 And Rebekah lifted up her eyes, and when she saw Isaac, she dismounted from the camel 65 and said to the servant, “Who is that man, walking in the field to meet us?” The servant said, “It is my master.” So she took her veil and covered herself. Genesis 24:64-65 (ESV)
Rebekah was unveiled until she approached her future husband, Isaac.
Smith's Bible Dictionary comments:
In ancient times the veil was adopted only in exceptional cases, either as an article of ornamental dress, So [Song of Solomon] 4:1,3; 6:7 or by betrothed maidens in the presence of their future husbands, especially at the time of the wedding (Ge 24:65) or lastly, by women of loose character for purposes of concealment (Ge 38:14). Among the Jews of the New Testament age it appears to have been customary for the women to cover their heads (not necessarily their faces) when engaged in public worship.
The International Standard Bible Encyclopedia states:
In 24:65 Rebekah conformed to the etiquette which required the veiling of brides (see MARRIAGE). In Genesis 38 one motive for Tamar's use of the veil was certainly to avoid recognition, but it seems clear from the passage that veils were used by courtesans. Why is unknown, perhaps partly to conceal their identity, perhaps partly in parody of the marriage custom.
Thus it seems that the fact Tamar was veiled indicated that she was a prostitute. It was not that a prostitutes veil was in some way different from the common one.
Besides Tamar's veil, there may have been other indications that she was a prostitute, one being the time of year.
Victor Hamilton in the NICOT series comments:
But why does she make her move at sheepshearing time? Hos. 4:13-14 details the aberrant practice of sacred prostitution at feast times in Israel, and Hos. 9:1-2 comments on ritual fornication on threshing floors in the hopes of producing a bumper crop. If this is an accurate description of what accompanied the observance of feast days, then it might explain why Tamar chose this opportune moment to seduce Judah.
Hamilton also offers an interesting rendering of Gen 38:14:
She removed her widow’s garb, put on a veil, perfumed herself, and positioned herself seductively at Enaim, which is on the way to Timnah;
Tamar removes her widow’s garments, veils herself (to conceal her identity from Judah), perfumes herself (to attract Judah), and positions herself beeṯaḥ ‘ênayim (lit., “at [the] opening of [the] eyes”). The NEB and JB, following Targs., Pesh., and Vulg., understood the phrase as a parting of the road, a crossroads (“where the road forks in two directions”). Other versions (e.g., RSV, NIV, NAB, NJPS), following the LXX, take 'ênayim as a place name (“at the gate [or entrance] to Enaim”). We know of no Enaim from the OT, though a site in the Shephelah is known as Enam, ‘ênām (Josh. 15:34), and perhaps the two refer to the same place... Ira Robinson has suggested that the phrase beeṯaḥ ‘ênayim here ought to be compared with the phrase kesûṯ 'ênayim in 20:16, literally, “a covering of the eyes.” When the truth of Sarah’s identity is revealed to Abimelech after he had almost committed adultery with her, Abimelech gives to Abraham a thousand shekels of silver, which is to serve as “a covering of the eyes” to Sarah. That is, the money will vindicate Sarah publicly from any suspicion of irregular sexual behavior, and be a compensation for any embarrassment she has had to live with. If kesûṯ ‘ênayim signifies vindication from suspicion of harlotry, peṯaḥ 'ênayim may signify the opposite —to pose in such a way to cause one to stop, look, and open his eyes. Thus be eṯaḥ 'ênayim may be a double entendre: Enaim is not only the place where Tamar met Judah, but also her sexual invitation to Judah. In this setting the name beṯaḥ ʿênayim (lit., “opening of the eyes”) is particularly appropriate and ironic. At “Opening of the Eyes,” even though he has sexual congress with her, Judah’s eyes are closed as to the identity of his daughter-in-law, and thus he fails to recognize his partner.
From Shimon bM's answer at https://judaism.stackexchange.com/questions/26864/biblical-significance-of-face-veil there is archaeological evidence to support this conclusion.
Rabbi Elazar said, "She covered her face in the house of her father-in-law, as Rabbi Shmuel bar Nachmani said in the name of Rabbi Yonatan: every bride who is modest in the house of her father-in-law will merit to have kings and prophets descend from her."
In other words, when the Torah tells us that Yehudah [Judah] didn't recognize Tamar because she had covered her face, it means that she had always covered her face when she was in his house. Now that she was at the crossroads, with her face uncovered like that of a prostitute, he didn't recognize her. Stands to reason then that covering one's face with a veil is, in the opinion of Rabbi Shmuel bar Nachmani, a sign of virtuous modesty.
Neither wives of seigniors nor [widows] nor [Assyrian women], who go out on the street [may have] their heads [uncovered]. The daughters of a seignior … whether it is a shawl or a robe or [a mantle], must veil themselves; [they must not have] their heads [uncovered]. Whether … or … or … they must [not veil themselves, but] when they go out on the street alone, they must veil themselves. A concubine who goes out on the street with her mistress must veil herself. A sacred prostitute whom a man married must veil herself on the street, but one whom a man did not marry must have her head uncovered on the street; she must not veil herself. A harlot must not veil herself; her head must be uncovered; -- ("The Middle Assyrian Laws," Translator: Theophile J. Meek, Tablet A) Pritchard, J. B. (Ed.). (1969). The Ancient Near Eastern Texts Relating to the Old Testament (3rd ed. with Supplement, p. 183). Princeton: Princeton University Press.
The tablets themselves date from the time of Tiglath-pileser I in the 12th century B.C., but the laws on them may go back to the 15th century. -- Pritchard, J. B. (Ed.). (1969). The Ancient Near Eastern Texts Relating to the Old Testament (3rd ed. with Supplement, p. 180). Princeton: Princeton University Press.