Twice elsewhere the author of the epistle to the Hebrews uses a genitive construct wherein he does not precede the proper name by a definite article:
- Heb. 9:4: ἡ ῥάβδος Ἀαρὼν ("the rod of Aaron")
- Heb. 11:30: τὰ τείχη Ἰεριχὼ ("the walls of Jericho")
Likewise, in Heb. 12:24, τὸν Ἅβελ could stand for τὸν αἷμα Ἅβελ, where Ἅβελ is an indeclinable proper name functioning as a genitive. But, you might say that τὸν αἷμα is impossible since αἷμα is a neuter gender noun, thus requiring the definite article τὸ, and you would be right. Fortunately, as noted by Constantin Tischendorf, there are several witnesses that support the readings τὸ and τὸ τοῦ.1

If indeed «τὸ Ἅβελ» or «τὸ τοῦ Ἅβελ» are the original readings, then they would stand for the expression «τὸ αἷμα τοῦ Ἅβελ», "the blood of Abel."
On the other hand, «τὸν Ἅβελ» could very likely be the legitimate reading (and one that I support), since: (1) it seem to be the more difficult reading; (2) it is supported by weightier witnesses; and, (3) it is supported contextually by Heb. 11:4. With this reading, the author could be using «Ἅβελ» to represent his blood via the figure of speech synecdoche (whole for part),2 for in Gen. 4:10, it was Abel's blood that cried out from the ground when Abel was dead. Furthermore, in Heb. 11:4, it is Abel who speaks while yet dead.
Georg Konrad Gottlieb Lünemann commented,3

Conclusion: "The Bloody Sacrifices of Abel," "Abel's Blood," or "Abel" (Himself)?
The original question asked if τὸν Ἅβελ, "Abel," referred to the bloody sacrifices of Abel, or Abel's blood which cried out from the ground after Cain murdered Abel.
In other words, is this verse speaking of the blood sacrifices of Abel (objective genitive) or is this verse actually talking about the blood of Abel, which cried from the ground in Genesis 4:10 (subjective genitive)?
Contextually, it would seem far-fetched to understand τὸν Ἅβελ as referring to Abel's bloody sacrifices, for in Heb. 11:4, the author refers to those same sacrifices of Abel as a (collective singular) θυσίαν, from the lemma θυσία, a feminine gender noun. However, neither τὸν (nor τὸ if we accept the variant reading) is declined in the feminine gender; rather, they are masculine and neuter, respectively. Therefore, the context seems to suggest that the author is referring to either Abel himself (τὸν Ἅβελ) or to Abel's blood (τὸ [αἷμα] Ἅβελ) which cried out from the ground.
Again, if we accept «τὸν Ἅβελ» as the reading, which seems most probable, then "Abel" could just as easily be understood as "Abel's blood" via synecdoche of the whole for part. Hence, as Lünemann commented,4

Footnotes
1 Vol 2, p. 833
2 Bullinger, p. 635-640
3 p. 411 (German text); p. 719 (English translation)
4 loc. cit.; The adverb "better" would be a translation of κρεῖττον, typically understood and translated as a comparative (adjective) declined in the accusative case, neuter gender, and singular number. However, here, κρεῖττον functions as an adverb with the same spelling. Note BDAG on κρεῖττον (=κρεῖσσον), p. 566:

Note: κρεῖττον (singular number) is the reading in NA28, while the TR has κρείττονα, which is declined in the plural number.
References
Bullinger, Ethelbert William. Figures of Speech Used in the Bible: Explained and Illustrated. London: Messrs; New York: Messrs, 1898.
Huther, Johann Eduard; Lünemann, Georg Konrad Gottlieb. Critical and Exegetical Handbook to the Epistles to Timothy and Titus, and to the Epistle to the Hebrews. Trans. Hunter, David; Evans, Maurice J. New York: Funk, 1885.
Lünemann, Georg Konrad Gottlieb. Kritisch exegetischer Kommentar über das Neue Testament, Dreizehnte Abtheilung, Kritisch exegetisches Handbuch über den Hebräerbrief. 3rd ed. Vol. 13. Göttingen: Vandenboeck and Ruprecht, 1867.
Tischendorf, Constantin. Novum Testamentum Graece. Vol. 2. Lipsiae: Giesecke, 1872.