While other commentators hinted at what Robertson wrote in his commentary, Robertson answered your question directly:
The old form of ἀθεμιτος [athemitos] was ἀθεμιστος [athemistos] from θεμιστο [themisto] (θεμιζω, θεμις [themizō, themis], law custom) and α [a] privative. In the N. T. only here and 1 Peter 4:3 (Peter both times). But there is no O. T. regulation forbidding such social contact with Gentiles, though the rabbis had added it and had made it binding by custom. There is nothing more binding on the average person than social custom. On coming from the market an orthodox Jew was expected to immerse to avoid defilement (Edersheim, Jewish Social Life, pp. 26–28; Taylor’s Sayings of the Jewish Fathers, pp. 15, 26, 137, second edition).
Robertson, A. T. (1933). Word Pictures in the New Testament (Ac 10:28). Nashville, TN: Broadman Press.
Note the word translated unlawful here in Acts isn't the normal word for unlawful used when related to Mosaic law. 1 Peter 4:3 uses the word associated with idolatry.
Devout Jews would not enter into idolaters’ homes lest they unwittingly participate in idolatry; they apparently extended this custom to not entering any Gentile’s home. It was considered unclean to eat Gentiles’ food or to drink their wine; although this purity regulation did not prohibit all social contact, it prevented dining together at banquets and made much of the Roman world feel that Jews were antisocial. Cornelius is undoubtedly accustomed to accepting reluctant (10:22) snubs, so Peter’s statement in 10:28 would mean much to him.
Keener, C. S. (1993). The IVP Bible background commentary: New Testament (Ac 10:27–29). Downers Grove, IL: InterVarsity Press.
Here are quotes with references:
Peter’s statement makes two moves toward interpreting his vision, one dramatic, one subtle. The dramatic move is that Peter perceives that his vision was about more than clean and unclean foods; it involves proper social interaction with persons—“no one should call a person impure or contaminated.” Gentile hostility toward Jews was rooted in part in Jewish adherence to dietary regulations and purity customs (Tyson 1987, 627). This view is born out by Jub. 22.16: “Keep yourself separate from the nations, and do not eat with them; and do not imitate their rites, nor associate yourself with them” (for a Gentile’s perspective, see Juvenal, Sat. 14.104–105; Tacitus, Hist. 5.5). Thus to move from the issue of food to persons would have seemed natural to the audience.
Parsons, M. C. (2008). Acts (p. 150). Grand Rapids, MI: Baker Academic.
Peter, in v. 28, striking what seems to be a stern note, advises the assembled group that it was taboo for a Jew to associate with or visit a foreigner, that is, if he or she wished to remain a clean Jew in good standing. The word αθεμιτον here could be translated “unlawful,” but it probably has its weaker sense of “taboo” or “strongly frowned upon.” There was no formal law that strictly forbade Jews from associating with Gentiles, it was just that they had to be prepared to pay a price for doing so, the price being becoming ritually unclean. Texts written by Roman authors such as Juvenal (Sat. 14.104ff.) and Tacitus (Hist. 5.5) show that Jews did regularly refuse to associate with Gentiles, and were objects of suspicion because of their “antisocial” behavior. Jubilees 22:16 expresses the sentiment of many early Jews: “Keep yourself separate from the nations, and do not eat with them; and do not imitate their rituals, nor associate with them.”
Having reminded his audience that they themselves are aware that a Jew shouldn’t be visiting or associating with Gentiles as he is now doing, Peter contrasts this approach with what God has now shown him. The second half of his opening remarks to the gathering in Cornelius’s house begins with καμοι, and the και probably has some adversative force—“but to me, God has shown.…” Here it becomes evident that Peter has now concluded that his vision was not just about food but also or perhaps primarily about persons. No person should be called common or unclean.
Footnote 106 references John 4:9 showing Jewish attitudes toward Samaritans.
Witherington, B., III. (1998). The Acts of the Apostles: a socio-rhetorical commentary (p. 353). Grand Rapids, MI: Wm. B. Eerdmans Publishing Co.