The answer to your question is best examined by looking at Onan's sin in the context of the exchange between Judah and Tamar and requires a good understanding and background of the place of women in ancient middle eastern culture and the purpose of Leverite marriage practices.
We must remember that this culture had no medicare and no social security. Therefore, what was a widowed woman to do if she had no sons to provide for her financially and care for her in her old age after she was unable to tend a field or do any type of manual labor? As a woman, a widow would be unable to own property and had no standing in the courts and very little rights of any kind. In such a patriarchal society, being a woman was a losing proposition and a male offspring was necessary for the providence of the widowed.
Therefore, the humanitarian practice of Levirite marriage was developed to ensure that these women were provided and cared for in their old age.
Now, armed with this as context, we can see that by failing to give his youngest son Selah to Tamar as a father and husband, Judah was shortchanging a poor widowed woman - not a very classy move. Later in the text, Tamar tricks Judah in to fathering her child. As the patriarch of the family, it was Judah's moral duty to see that Tamar would be cared for and provided for and by withholding his son he was failing this moral obligation. This is why, in verse 26, upon the realization that he has been duped and is the father of Tamar's child, Judah goes from wanting to burn her to saying "[Tamar] is more upright than I am, because I wouldn’t give her to Shelah my son."*
He realized that it was his moral obligation to provide for Tamar and he had failed in his duty. Most people get hung up on the prostitution bit, but fail to realize that 1) the text doesn't condone this and 2) prostitution hadn't been explicitly forbidden yet because the law of Moses had not yet been given. Therefore, while frowned upon, the prostitution in the text isn't nearly as big of a deal as Judas flagrant disregard for the plight of his widowed daughter-in-law.
Understanding this, we can then see that Onan's sin wasn't spilling his seed per-se, but in failing to provide for his widowed sister-in-law just as his father fails to do later in the story. Furthermore, Onan does this out of greed and/or laziness which the text implies by saying "the seed would not count as his". In other words, he would only be a trustee of his brother's estate, but would not really get to benefit from the wealth. This activity had it's benefits for the guardian-redeemer, but it also had it's drawbacks. (See Ruth Chapter 4; in fact, just read the whole book - it is short and very relevant to this passage).
For taking this matter into her own hands and acting in such an admirable fashion, Tamar is provided as a role model and was revered and is one of the very few women who makes it into the genealogy of Jesus (a great honor) and truly was "more righteous". This passage is actually very progressive and feminist in nature in this regard once you understand all of the nuance - it provided female empowerment as a virtue and example.
Therefore, the text says that Onan's sin was #4, "Not performing his Levirate duty" (or better restated in context of our current culture, "screwing over a poor old widow")
*Note: as an aside, the motivation for Judah's failure to give his son to Tamar is superstitious. In this culture, they believed in curses, and Judah likely believed Tamar to be cursed and killing off his sons. It was typically believed that tragedies and maladies were caused by sins or curses (curses often caused also being caused sins or wrongdoings that angered some supernatural entity, which could be Yahweh himself.) This is why in John 9 the disciples inquire of Jesus regarding a blind man "Rabbi, who sinned, this man or his parents, that he was born blind?".