It's probably αὐτὴν, as the modern critical editions have it.
The witnesses
The genitive pronoun αυτης is found (among consistently cited witnesses) only in the 4th-6th Century "correction" of Sinaiticus and the f1 group of miniscules ("Lake Group") from the 12th Century. The original (fourth C.) Sinaiticus and 𝔓64/67 omit the pronoun, a reading witnessed also by Clement (of Alexandria) and Tertullian. The accusative αυτην is found in essentially all other Greek witnesses including both Vaticanus and the Majority Text group.
This is not a variant that can be reliably differentiated in translation (see below), so the witness of the versions is not helpful.
What is the difference?
As you noted, the pronoun is likely the object of "looks with lustful intent" (ESV) regardless of its case. Whether it should be genitive or accusative depends on the semantics of the verb ἐπιθυμέω, "to desire". In ancient Greek, it consistently took a genitive describing the thing desired. The LXX of Exodus 20:17 1 (to which this verse may allude) deviated in using an accusative of the person.2 This usage is followed subsequently, particularly in dependent literature, where it tends to have sexual connotations that were obviously present in Exodus 20:17 (and Matt 5:28) but not necessarily part of ἐπιθυμέω previously.
The accusative is the harder reading (less consistent with normal Greek usage5 ), is present in the best Greek witnesses, and is accordance with the nearby allusion to Exodus 20. Given all of these factors (and the decision of the much-more-informed-than-I editors of the modern critical editions), the accuasative αὐτὴν is most likely.
1. All textual notes from the NA-28 apparatus.
2. οὐκ ἐπιθυμήσεις τὴν γυναῖκα τοῦ πλησίον σου - Rahlfs, with no textual variants listed in his apparatus. To what extent the choice of accusative reflects the fact that it translates the Hebrew transitive verb חמד is a question I'm not qualified to answer, but it may be a plausible explanation for the shift in Greek usage.
3. William Arndt, Frederick W. Danker and Walter Bauer, A Greek-English Lexicon of the New Testament and Other Early Christian Literature, 3rd ed. (Chicago: University of Chicago Press, 2000).
4. Apparently some have suggested that instead the pronoun is to represent the "subject" of the infinitive idea - i.e. "in order to cause her to lust". This seems to be an unlikely option in context (which would require ἐμοίχευσεν αὐτὴν to mean "to cause her to commit adultery" - an atypical (unattested?) use of that verb) and in light of Exodus 20:17.
5. This is consist with the "correction" of Sinaiticus to the genitive.