Reading from the preceding section in the 1 Cor. 3, the temple in verse 17 is referring to the building in verses 1 Cor. 3:10-14 and not our body as the temple of God (that is mentioned in 1 Cor. 6).
Paul was speaking about the divisions among the Corinthians, which is destroying the temple of God.
The matter of God's building in the Bible is a major topic. But to address your question on what constitute destroying is not building with gold, silver and precious stones.
According to 1 Cor 3:12, the proper building materials are gold, silver, and precious stones. There are three because these materials refer to the Three of the Trinity, to the Father, the Son, and the Holy Spirit. In other words, the church is a group of people constituted of the Triune God.
Gold refers to the life and nature of the Father God. It does not change or become corrupt in nature. Spiritually, it refers to His unchanging nature.
Silver signifies Christ's redemptive work. In Exodus 30, silver was used to redeem the life of God’s people. Well, the Lord was sold with 30 pieces of silver. This is to experience the cross of Christ to deal with every all problems in the universe (or in our daily life).
Precious stones is a transformed substance. It signifies the transforming and constituting Spirit (2 Cor 3:18). We were man of clay (Gen. 2:7; Romans 9:21). When we were saved, we became a stone in God's building (Matt. 16:17-18; 1 Pet. 2:5). Through pressures and refining fires in our environment :D, the Spirit does a transforming work in us until we are transformed into the same image from glory to glory; then we will be precious stones.