Gesenius' 18th ed. (the leading dictionary of Biblical Hebrew) lists for the word שֵׁן three meanings: (1) tooth, either concrete or metaphorical (e.g. Ps 124:6); (2) ivory; (3) something geographical, often in combination with סלע to yield "rook tooth", i.e. "cliff" (German: "Felszahn", lit. "rock tooth"). As for (2), several examples are given (here all KJV):
- Ezek 27:15, "horns of ivory and ebony"
- 1 Kgs 10:18 | 2 Chr 9:17, "a great throne of ivory"
- 1 Kgs 22:39 / Am 3:15, "the ivory house" of Ahab in Samaria
- Ps 45:9, "the ivory palaces"
- Song 5:14, "his belly is as bright ivory"
- Am 6:4, "beds of ivory"
All these examples, with the exception of Song 5:14, seem to refer to the material, just like meaning (1) — for (3) both the material and the colour can be meant.
However, in Song 5:14 it is very clear that the colour is meant ("bright"). This is not strange; apart from Am 6:4 these are the only instances of metaphorical language. Based on the case in 5:14 I would also argue for understanding 7:4 as referring to the colour. This also provides for a nice contrast with "fishpools" in the remainder of 7:4 if we understand it to mean "dark".
Although nowadays it is fashionable in the western world to have a tanned skin, some centuries ago it was European fashion to have a white skin. In that time, workers would work outside a lot and get tanned, whereas the aristocracy had the privilege of staying inside, leading to the beauty ideal of a white skin — conversely, nowadays, people work a lot inside, and you show your wealth by showing you can get outside. This can perhaps help to understand the metaphor, although I do not know if there was a beauty ideal of a white skin in biblical times.
As for the "tower" metaphor, perhaps having a long neck was also part of the beauty ideal as for the Kayan people in Myanmar, but again this is conjecture.