Reading from the preceding section in the same chapter, the temple cited in verse 17 is referring to the collective body of believers (amplified in 1 Cor 3:10-14), and therefore not to our individual bodies as temples of God (which is mentioned later in 1 Cor. 6). The following words in Greek (highlighted in bold) are in the plural form, which substantiates that the "temple" is a collective in the immediate context, and not in reference to any one person.
1 Cor 3:16-17 (Greek)
16 οὐκ οἴδατε ὅτι ναὸς θεοῦ ἐστε καὶ τὸ πνεῦμα τοῦ θεοῦ οἰκεῖ ἐν ὑμῖν; 17 εἴ τις τὸν ναὸν τοῦ θεοῦ φθείρει, φθερεῖ τοῦτον ὁ θεός: ὁ γὰρ ναὸς τοῦ θεοῦ ἅγιός ἐστιν, οἵτινές ἐστε ὑμεῖς.
Paul therefore was speaking in this immediate about the divisions among the Corinthians within the body collective (plural), which is pointing to destroying the temple of God (collective body of believers). The problems in Corinth were literally "de-constructive."
As noted on other postings ont his website, the matter of God's building structure in the Bible (temple) is an important biblical topic. Thus to address the question on what constitutes "destroying" thus places emphasis on not building with gold, silver and precious stones. In other words, when you fail to build with gold, silver, and precious stones, you are building with "wood, hay, and stubble" (de-construction, or weakening God's temple).
According to 1 Cor 3:12, the proper building materials for the Temple of the Lord are gold, silver, and precious stones. Gold is incorruptible, and brings the reader to consider the gold used in the Hebrew Bible within the tabernacle and later the Temple. Silver is another precious metal that does not rust or corrode, and therefore is also incorruptible, and so again we return to the sanctity and purity of God's dwelling (temple). Precious stones are created through heat and pressure, and therefore echoes the transforming and constituting Spirit of Christ (2 Cor 3:18). We are all men of clay (Gen. 2:7; Romans 9:21). When we were saved, we became a stone in God's building (Matt. 16:17-18; 1 Pet. 2:5). Through pressures and refining fires in our environment, the Lord performs his transforming work on the stone transforming us "from glory to glory" (2 Cor 3:18); thus we become those precious stones in the collective temple of the Lord.