Onan's sin was entirely related to his refusal to perform his levirate duty.
Quickly about the other three:
Coitus interruptus is not masturbation. It is a (very unreliable) method of birth control. Onan was attempting not to get Tamar pregnant because he did not want to provide an heir for his deceased older brother.
It was not "theft of Tamar's child." Rather, it was (once again) refusal to provide his deceased brother, Er, with an heir under the law of levirate marriage.
Though marrying a brother's wife was normally prohibited (see Leviticus 18:16; 20:21), the law of levirate marriage was an exception to that rule. Marrying a deceased brother's wife under this law was not considered adultery or inchastity. And the rules of "inchastity" for men were rather lax in those days anyway.
The law of levirate marriage is stated in Deuteronomy 25:5-10:
5 "If brothers dwell together, and one of them dies and has no son, the wife of the dead man shall not be married outside the family to a stranger. Her husband's brother shall go in to her and take her as his wife and perform the duty of a husband's brother to her. 6 And the first son whom she bears shall succeed to the name of his dead brother, that his name may not be blotted out of Israel. 7 And if the man does not wish to take his brother's wife, then his brother's wife shall go up to the gate to the elders and say, 'My husband's brother refuses to perpetuate his brother's name in Israel; he will not perform the duty of a husband's brother to me.' 8 Then the elders of his city shall call him and speak to him, and if he persists, saying, 'I do not wish to take her,' 9 then his brother's wife shall go up to him in the presence of the elders and pull his sandal off his foot and spit in his face. And she shall answer and say, 'So shall it be done to the man who does not build up his brother's house.’ 10 And the name of his house shall be called in Israel, ‘The house of him who had his sandal pulled off.'"
Although this law was recorded several centuries after the time of Onan, as with many statutes in the Mosaic Law it was a codification of a law that was already in effect long before the time of Moses, and that was common to many ancient cultures. For more on the law of levirate marriage, see: Jewish Encyclopedia -> Levirate Marriage (Hebr. "yibbum").
The law of levirate marriage was, however, complicated in its application, and has been the subject of much debate and varying interpretation since ancient times. Having said that, here is a summary of the key events in the passage, followed by the most likely scenario of precisely what Onan's sin consisted of.
At the time of the incident of Onan in Genesis 38:6-11, Judah had fathered three sons: Er, Onan, and Shelah (see Genesis 38:1-5). Er, the eldest, had received the punishment of death from the Lord for unknown offenses. Judah then instructed his second son, Onan, to fulfill the law of levirate marriage for Er, his deceased elder brother, and provide an heir for him. Onan went through the motions of doing this, but avoided actually getting Tamar pregnant by the practice of coitus interruptus, termed in the Biblical account "wasting his semen on the ground."
The text states Onan's purpose for doing this as follows:
But Onan knew that the offspring would not be his. So whenever he went in to his brother's wife he would waste the semen on the ground, so as not to give offspring to his brother. (Genesis 38:9)
This was why the Lord carried out a sentence of death on Onan:
And what he did was wicked in the sight of the Lord, and he put him to death also. (Genesis 38:10)
Why did Onan marry Tamar, but then avoid getting her pregnant, and why was this an offense punishable by death in the Lord's eyes?
By going through the motions of taking Tamar as his wife, and thus appearing to fulfill the law of levirate marriage, Onan would become the heir to his deceased brother's current possessions. If he refused to fulfill his duty under the law of levirate marriage, he would not receive his brother's property. So he had a financial incentive to at least appear to fulfill the levirate law.
However, if he got Tamar pregnant, and she bore a child, especially if it were a male child, that child would inherit:
- The primogeniture that would have gone to the first biological male child of his Levirate, or legal father Er, Judah's eldest son—which would cause the leadership of the clan to pass to the child, and
- A double inheritance from Judah, their father, which the firstborn son was entitled to by long custom (see, for example, Deuteronomy 21:15-17).
As mentioned above, Onan would have already received his deceased brother's property by taking Tamar as his wife. However, if he impregnated her and gave her a child, especially if it were a male child, he would lose both the primogeniture and the double portion of the inheritance from their father Judah.
A bit of simple math shows that assuming Judah fathered no more children, this would cut Onan's inheritance from two-thirds to one-fourth of his father's total wealth when his father died. And of course, it would deprive Onan of the primogeniture, or leadership of the clan, that would otherwise pass to him because his elder brother would have had no heir.
Additional source for the above: Jewish Encyclopedia -> Inheritance ("yerusbah," "naḥalah") -> Levirate Connections
The sin of Onan, then, for which the Lord punished him with death, was that of going through the motions of obeying the law of levirate marriage in order to obtain his deceased brother's property, but not actually performing his duty under that law—that of providing an heir for his brother—in order keep for himself both the primogeniture and the inheritance of a double portion of the wealth of their father Judah.
Short version: Onan made a pretense of following the levirate law, but in fact shirked his obligations to his family and clan under the law, all from motives of greed and desire for power.