Short Answer: Yes, they would know what he meant
The longer answer is that the letter to the church in Rome (1:7) was to a mixed group of Gentiles (1:13) and Jews (2:17). Most believe the church started from some of the Jews present at Peter's preaching during Pentecost, the "visitors from Rome" (Act 2:10; NKJV/ESV/NASB).
Starting at 2:17, Paul begins more specifically addressing the Jewish audience of his letter, and the fact that they may still "rest" (NKJV) or "rely" (ESV/NASB) on the law, and how this is not what they need to rely in (2:25-29), for both Jews and Gentiles are sinners (3:9-18), and the law points that out (3:19).
So the context of v.20 speaking of the "deed" (NKJV) or "works" (ESV/NASB) of the law is within the section of the letter more specifically targeted to the Jews (v.9 and v.19 the "we" is Paul speaking in solidarity with the Jewish race, even though he was not in solidarity with those who were attempting to keep the law for purpose of being considered righteous—the "you" group of 2:17-24).
So the Jewish part of the church of Rome would certainly know what ἔργων νόμου referred to, and the Judaizing influences present in the church from the Jewish crowd probably made the Gentile believers aware as well (and if not, they had their Jewish fellow believers to educate them about it).1
Used Elsewhere? Yes, in some Qumran Literature
Robert Mounce notes in Romans, Vol. 27, The New American Commentary (Nashville: Broadman & Holman Publishers, 1995), 111 n.182:
The anarthrous phrase ἔργων νόμου refers to that which the [Mosaic] law requires. Fitzmyer points out that while the phrase has not been found in the OT, it has turned up in the Qumran literature, thus making Paul a “tributary to a genuine pre-Christian Palestinian Jewish tradition” (Romans, 338–39)
From the bibliography, Mounce is referring to the Anchor Bible commentary of J. A. Fitzmyer, Romans (New York: Doubleday, 1993).
The use in the Qumran literature confirms a more widespread understanding of the concept of "works of the law" in Jewish circles (from which the argument I make here is based).
Further research found this reference from James M. Hamilton Jr., "N. T. Wright And Saul’s Moral Bootstraps: Newer Light On 'The New Perspective,'" Trinity Journal 25 (2004), 152 n.70, which indicates at least one Qumran document the phrase is found in (and references a debate about the significance of that):
There has been some debate over the relevance of the text referred to
as “4QMMT” (4Q394-399). “MMT” abbreviates Miqsat Ma‘ase ha-Torah (מקצת
מעשׂי התורה), “Some Works of the Law.” Dunn thinks that this document
“preserves a vocabulary and manner of theologising which left its mark
on a wider spectrum of Jewish thought and practice, and that it was
just this sort of theologising and practice which confronted Paul in
Antioch and which he wrote Galatians to counter” (James D. G. Dunn,
“4QMMT and Galatians,” NTS 43 [1997]: 153). Wright outlines five
reasons why he thinks this claim will not stand (N. T. Wright, “Paul
and Qumran,” BRev [October 1998]: 18, 54).2
According to the Wikipedia article, the dating of 4QMMT is roughly contemporary to Christ's birth (the "early" part of 1st century BCE/CE, or as I prefer, BC/AD). That is before Paul's time.
NOTES
1 For some discussion of meaning of "works of the law" (not necessarily endorsing all conclusions, but for your information), see T. Schreiner, "Did Paul Believe in Justification by Works? Another Look at Romans 2," Bulletin for Biblical Research 3 (1993): 131-158.
2 It is worth noting that there is an article by Martin G. Abegg specifically discussing 4QMMT: "4QMMT C 27, 31 and 'Works Righteousness'", (Dead Sea Discoveries
6/2 [1999]: 139-147).
