It may appear to be a 'mystical' creature to us, in this day and age, but the people of Isaiah's era would know what was meant. This can be ascertained from the three times the Hebrew words involved are used by Isaiah, with Jeremiah agreeing in his one use.
There are two Hebrew words; tsepha means cockatrice, basilisk, adder and viper.
The other Hebrew word is tsiphoni and simply means cockatrice.
Tsepha is only used once in the Hebrew scriptures, by Isaiah, in 14:29, where he says:
"Rejoice not thou, whole Palestina, because the rod of him that smote
thee is broken: for out of the serpent's root shall come forth a
cockatrice, and his fruit shall be a fiery flying serpent." (A.V.)
This is in the context of the coming desolation of Babylon, Israel's triumph in the fall of Babylon's king, and Palestina being threatened with complete dissolution.
Here are the three uses of the other word, tsiphoni, all from the A.V.
"And the sucking child shall play on the hole of the asp, and the
weaned child shall put his hand on the cockatrice' den." Isaiah 11:8
"They hatch cockatrice' eggs, and weave the spider's web: he that
eateth of their eggs dieth, and that which is crushed breaketh out
into a viper." Isaiah 59:5
"For, behold, I will send serpents, cockatrices, among you, which will
not be charmed, and they shall bite you, saith the Lord." Jeremiah
It seems clear that, whatever we think of the word 'cockatrice', the two prophets used it in tandem with speaking of snakes, almost as synonyms. Our lack of understand does not detract from the force of the prophetic statements. We surely all get the gist of their meaning? Thus, my answer is that some kind of snake is meant, known to them, if not to us. There's nothing mystical about it.