Well the glib answer is that it would be hard to build a temple with uncut stones.
The more serious answer is that you are right, the temple is a work of man and so must point to something else, namely that God dwells in man and the temples built by man are just shadows of this truth.
Acts 7:48–51 (KJV 1900)
48 Howbeit the most High dwelleth not in temples made with hands; as
saith the prophet, 49 Heaven is my throne, and earth is my footstool:
what house will ye build me? saith the Lord: or what is the place of
my rest? 50 Hath not my hand made all these things? 51 Ye stiffnecked
and uncircumcised in heart and ears, ye do always resist the Holy
Ghost: as your fathers did, so do ye.
So then why was the temple built, and why did God demand it?
Zechariah 8:9 (KJV 1900)
9 Thus saith the LORD of hosts; Let your hands be strong, ye that hear
in these days these words by the mouth of the prophets, which were in
the day that the foundation of the house of the LORD of hosts was
laid, that the temple might be built.
All things that can be seen - all of creation - is God's speech. It's symbolic, as speech is a representation of something else and is an attempt at communication.
The spirit is real, the speech of the spirit is what we can see and touch. So when we talk about spiritual things, we are using words, but when God talks, worlds are created, nations come alive, etc. But it's still just a representation of something else and is meant primarily to communicate something more real rather to enjoy as an end in and of itself.
Exodus 25:40 (KJV 1900)
40 And look that thou make them after their pattern, which was shewed
thee in the mount.
E.g. Moses designed the tabernacle according to a pattern of what he saw in the spirit, in his theophany. The temple itself is not the theophany, it is a pattern, a metaphor, of what he saw in the spirit. There was a pedagogical purpose behind it all.
Then you can ask, why didn't God build the temple by Himself? But God did, he made Adam's flesh before the fall, and Jesus' flesh as the replacement, - without human effort. But Israel also built two temples of stone with human effort as copies or imitations of God's temple, because the child copies what the father does.
Jesus answered and said unto them, Destroy this temple, and in three
days I will raise it up...But he spake of the temple of his body. John 2.19-21 KJV
The need of creation to echo back the words of the creator is inevitable and healthy, but you get into problems when the copies are elevated above the originals, which would be when the child prefers the work of their own hands to that of the father.
To this end, the uncut stone in the altar is God teaching us that the place of sacrifice must not be polluted by human effort - we must come to him in utter weakness with no efforts of our own - but the service around the altar requires great effort (courses of priests, and vessels, and garments, etc) using many instruments made by man. So this is again another picture of man, who labors constantly but at the center where he meets God, man must always be at rest with no effort and perfect meekness, like wind circling a place of perfect stillness (Ps 46.10).