It's not a matter of either/or but both/and.
Spiritual idolatry, as you put it, and physical idolatry, as I put it, are very often--if not always--conjoined, in more ways than one.
In our post-modern world, it's rare for a person to bow down to an idol that is literally made of wood and stone and metal. The adherents to the Canaanite religions certainly did, and the NME culture, in general, was more prone to believe in talismans and the like than we are today, though an argument could be made that the New Age Movement and its belief in the power of crystals and the like is more talisman oriented than you might at first think.
Nevertheless, the Apostle John is alluding to any thing which comes between us and our unstinting devotion to our God and Savior. Idols of this sort could be any one of a number of things, including pleasure, fame, fortune, independence and self-reliance, grown-boys' toys, and in the Apostle's day, a false- and quasi religion, Gnosticism, which in part introduced the heresy that we can and should separate the spiritual from the material.
In conclusion, just as one cannot completely segregate spiritual- from physical idolatry, neither can you exclude the spiritual component from even physical things. In other words, human beings are an admixture, so to speak, of spiritual and physical (or material) aspects, and amazingly enough, so was Jesus! That's the way we--and He--were designed by God. Ultimately, however, aren't most, if not all, idolatries indicative of a spiritual problem?