This answer is based on Karl Barth's commentary. This particular section may be found on Google Books, if you are interested.
Barth makes a distinction between the historical Adam and Adam as a type of all humanity. According to his theology, the Triune God decided before creation to elect some and reject others. So that Adam would reject God and that sin would enter into the world is a foregone conclusion. (Barth connects this passage with the theological idea of Supralapsarianism.)
Adam and his fall are not themselves primary things but logically derived from God's plan to come into the world in the form of a man, the Christ, and dispense the gift of grace. Since we would deserve to be saved if we had been righteous, it's necessary for us to be unrighteous so that God's rescue is undeserved (i.e. Grace). Jesus became the second Adam, but instead of bringing death to all humanity, Jesus brings the hope of life.
So under this scenario, even the historical Adam's death was caused by Adam's fall. In programming terms (ignore this if you aren't familiar) Adam and every other person since has been an instance of the generic type (which Paul calls Adam). But Adam is derived from the Christ type. The purpose of the Adam type is to be replaced with the Christ type. So, just as Adam was the first Adam, Christ is the first Christ.
When it comes to the historical Adam, he sinned and then/therefore died. And we see each person (male and female) after him inevitably sin and die. It's not really the case that we are held responsible for someone else's sin. Rather each one of us follows Adam's example in sinning and dying. (But our sin isn't causally connected to the historical Adam's sin. Rather we sin because we are of the Adam type.)