I think briars and thorns represent vain thoughts. In Hebrew, these would be called "imaginations" - yetser, יֵ֫צֶר. I recommend a word study of yetser.
Genesis 6:5–6 (KJV 1900)
And GOD saw that the wickedness of man was great in the earth, and
that every imagination of the thoughts of his heart was only evil
continually. And it repented the LORD that he had made man on the
earth, and it grieved him at his heart.
In the past, God destroyed these imaginations with water, but next he will destroy them with fire.
So why would briars and thorns represent vain imaginations? Allow me to present some evidence.
Linguistic
The root of yetser, יצר, or the three letter root y-ṣ-r can be pointed as yotser, which means "to form" or yetser, which means "to imagine". They are related by the fact the the yetser is a form that is not made real. It is a form that exists only in the mind. It is what man shapes, but not in the world, but just with the power of his mind, which is the only arena in which man can fight God - in his imagination; we cannot fight God by punching the air, we only fight God in our minds.
Imaginations
God does not like man's imaginations
Genesis 8:21 (KJV 1900)
And the LORD smelled a sweet savour; and the LORD said in his
heart, I will not again curse the ground any more for man’s sake; for
the imagination of man’s heart is evil from his youth; neither will I
again smite any more every thing living, as I have done.
The one who rejects God is the one who walks in the imaginations of his heart:
Deuteronomy 29:19–20 (KJV 1900)
19 And it come to pass, when he heareth the words of this curse, that
he bless himself in his heart, saying, I shall have peace, though I
walk in the imagination of mine heart, to add drunkenness to thirst:
20 The LORD will not spare him, but then the anger of the LORD and his
jealousy shall smoke against that man, and all the curses that are
written in this book shall lie upon him, and the LORD shall blot out
his name from under heaven.
When nations make war against God, they do so with vain imaginations:
Psalm 2:1 (KJV 1900)
Why do the heathen rage,
And the people imagine a vain thing?
Psalm 140:1–2 (KJV 1900)
Deliver me, O LORD, from the evil man:
Preserve me from the violent man;
Which imagine mischiefs in their heart;
Continually are they gathered together for war.
The redeemed man, who is at peace with God, will not walk in the imagination of his heart.
Jeremiah 3:17 (KJV 1900)
At that time they shall call Jerusalem the throne of the LORD; and
all the nations shall be gathered unto it, to the name of the LORD, to
Jerusalem: neither shall they walk any more after the imagination of
their evil heart.
God's rejection of Israel is based on the fact that they walk in the imaginations of their heart:
Jeremiah 13:10 (KJV 1900)
This evil people, which refuse to hear my words, which walk in the
imagination of their heart, and walk after other gods, to serve them,
and to worship them, shall even be as this girdle, which is good for
nothing.
We challenge God with our imagination:
Nahum 1:9 (KJV 1900)
What do ye imagine against the LORD?
He will make an utter end:
Affliction shall not rise up the second time.
Briars and thorns
Symbolically, the thorn is something that comes up out of the ground (out of man, who is the ground) yet it has no life and bears no fruit. Rather, it chokes that which does have life and does bear fruit. It's a constant source of pain.
As part of the curse, Adam was condemned to being tormented with thorns that would come out of the land, which he would have to struggle with:
Genesis 3:18 (KJV 1900)
18 Thorns also and thistles shall it bring forth to thee; and thou
shalt eat the herb of the field;
Thorns represent the portion of the land that is not conquered -- e.g. the portion of yourself that is not subject to God:
Numbers 33:55 (KJV 1900)
But if ye will not drive out the inhabitants of the land from
before you; then it shall come to pass, that those which ye let remain
of them shall be pricks in your eyes, and thorns in your sides, and
shall vex you in the land wherein ye dwell.
See also Joshua 23.13 and Judges 2.3
Thorns are unique in that they cannot be grasped with hands, they can only be burned with fire.
2 Samuel 23:6–7 (KJV 1900)
But the sons of Belial shall be all of them as thorns thrust away,
Because they cannot be taken with hands:
But the man that shall touch them must be fenced with iron and the staff of a spear;
And they shall be utterly burned with fire in the same place.
Thorns will be consumed with divine fire
And the light of Israel shall be for a fire,
And his Holy One for a flame:
And it shall burn and devour his thorns and his briers in one day;
So thorns represent the unconquered part of the land, that cannot be grasped with hands, but will be destroyed in one day by divine fire.
This suggests that the thorn is unrepentant, immaterial, and destined for destruction. Vain imaginations would satisfy this requirement, and would be the primary thing to satisfy it.
In the New Testament, we have the parable of the sower, where Jesus equates thorns with:
Matthew 13:22 (KJV 1900)
He also that received seed among the thorns is he that heareth the
word; and the care of this world, and the deceitfulness of
riches, choke the word, and he becometh unfruitful.
This also suggests vain thoughts, as both care of the world and care for riches is vanity.
Lastly, Jesus was given a crown of thorns on his head (Mark 15.17). Christians have long interpreted the crown of thorns as worldly thoughts that torment Christ. Here is Martin Luther[1]:
Do you behold his crown of thorns, believe the thorns are your wicked
thoughts
Final interpretation
Going back to the verse in question, let's look at the verse in context and give an interpretations with thorns as "vain thoughts"
Isaiah 27:2–5 (KJV 1900)
In that day sing ye unto her,
A vineyard of red wine.
I the LORD do keep it;
I will water it every moment:
Lest any hurt it, I will keep it night and day.
Fury is not in me:
Who would set the briers and thorns against me in battle?
I would go through them, I would burn them together.
Or let him take hold of my strength,
That he may make peace with me;
And he shall make peace with me.
He shall cause them that come of Jacob to take root:
Israel shall blossom and bud,
And fill the face of the world with fruit.
God is saying that he will tend to his vine (Jacob, representing the second born, or the promise) and keep and protect it, guarding it night and day. The enemy of the vine would be thorns (or vain thoughts), that would grow up from the land (from you) and seek to choke the vine (the inner man, that is the second born).
God does not have fury/rage, he is motivated solely by protection of the vine. But if any thorns appear, God will destroy them, so that the vine can continue to grow unimpeded.
Therefore, let the hearer - who is considering entertaining vain thoughts -- make peace with God, by holding onto his strength. God's strength will conquer vain thoughts, and then you will be at peace with God, and the vine will grow and bear fruit within you.
[1] See https://ccel.org/ccel/luther/sermons/sermons.iv.iii.html