Although Leviticus 11 https://www.biblegateway.com/passage/?search=Leviticus+11&version=NKJV does tell us what was forbidden to be eaten by the ancient people of Israel, Scripture seems to permit the usage of some of the by-products of these very things in the liturgy of the chosen people. That said, it has to be noted that the interpretation of various elements is still an open question. Perhaps God was making an exception here due to the fact the these items were not being taken internally?
We see in the Holy Incense, Moses included onycha in its composition. Now onycha can be interpreted as a form of mussel shell, but even this is not known for sure. https://www.biblegateway.com/passage/?search=Exodus%2030:34-38
It seems that, for anything relating to the liturgy in the temple, God wanted his chosen people to spare no expense to His honor. For the temple, gold, silver and bronze were incorporated into its construction, as well as other precocious materials. http://www.bible-history. com/tabernacle/TAB4Preparing_for_the_Tabernacle.htm
As for cloth used in the temple, it may even be possible that byssus cloth was employed in the temple. This is an extremely expensive sea silk fabric that was made from the Great Mediterranean Pen shell (Pinna nobilis). There are over 40 references of a biblical use of byssus. https://en.wikipedia.org/wiki/Sea_silk
"The threads of the fabric’s history are difficult to trace, beginning with the fact that the word byssus itself once referred to any precious textile. The Old Testament reportedly includes 45 mentions, but some of them, judging from the context, almost certainly refer to linen, cotton, or regular silk. The same is true for the cloth that Egyptians used to wrap mummies, which scholars have translated as “byssus." But it’s hard to be sure which byssus they’re talking about: sea silk, or another precious textile? The silken filaments that compose the pen shell’s beard are also called byssus, adding to the confusion." http://mentalfloss.com/article/69654/untangling-secrets-sea-silk-ancient-mediterraneans-elusive-luxury-textile
This same "precious" reason seems to apply in the procuring of dyes used in the vestments for the priests and for the temple. The dye for tyrian purple, which is made from mussels, was so expensive that it was reserved for royalty and the very wealthy. Thus I conclude that the reason why the ancient Hebrews used shells for dyeing cloth was that God wanted no expense to be omitted in His honor.