No, vanity is not Adam's sin. It doesn't mean sin. Vanity is a reference to "vanishing" and it refers to things that pass away. One could also use the archaic "corruption" or the more modern "ephemeral". Basically it's the second law of thermodynamics -- "things fall apart".
Romans 8:20–21 (KJV 1900)
20 For the creature was made subject to vanity, not willingly, but by
reason of him who hath subjected the same in hope, 21 Because the
creature itself also shall be delivered from the bondage of corruption
into the glorious liberty of the children of God.
We see a parallel between "subject to vanity" and "bondage of corruption [φθορά]", e.g. a rewording of the same idea, namely that of dying, perishing, being destroyed.
In addition to corruption and vanity, παράγω - "passing away" - is also used to describe this idea:
1 John 2.17:
And the world passeth away, and the lust thereof: but he that
doeth the will of God abideth for ever.
1 Corinthians 7:29–31 (KJV 1900)
29 But this I say, brethren, the time is short: it remaineth, that
both they that have wives be as though they had none; And they that
weep, as though they wept not; and they that rejoice, as though they
rejoiced not; and they that buy, as though they possessed not; And
they that use this world, as not abusing it: for the fashion of this
world passeth away.
That is what is being talked about. Not sin. The relationship with sin is that Adam's sin caused the earth to become subject to vanity and thus pass away.
This idea that decay is the effect caused by sin can be seen in several places, e.g.
Galatians 6:8 (KJV 1900)
8 For he that soweth to his flesh shall of the flesh reap
corruption; but he that soweth to the Spirit shall of the Spirit reap life everlasting.
Paul uses corruption and vanity as synonyms, as can be seen in Ephesians:
Ephesians 4:17 (KJV 1900)
17 This I say therefore, and testify in the Lord, that ye henceforth
walk not as other Gentiles walk, in the vanity of their mind,
and then later in verses 22-23:
Ephesians 4:22–23 (KJV 1900)
22 That ye put off concerning the former conversation the old man,
which is corrupt according to the deceitful lusts; 23 And be renewed
in the spirit of your mind;
2 Corinthians 11:3 (KJV 1900)
3 But I fear, lest by any means, as the serpent beguiled Eve through
his subtilty, so your minds should be corrupted from the simplicity
that is in Christ.
And in both cases, the corruption of the mind is to be subject to vain thoughts or thoughts of things that perish or pass away.
Before Adam's sin, the life of God was in the earth. But after Adam's sin, the land was cursed. The effects of this curse are that the earth is dying, passing away. The whole of creation is perishing.
But God promises to renew it with a new creation in which there is a new heaven and a new earth, and the deposit of that new creation is the new life within us. This new life will swallow up the old life that is perishing, slowly withering like a plant removed from the window and put in a dark room after the light of God has been removed from it due to sin. The sin is the darkness and the withering of the plant is vanity, and our new life is the light that shines in the dark place, as a deposit on the future creation that will be swallowed up in light.
2 Corinthians 4:16–5:2 (KJV 1900)
16 For which cause we faint not; but though our outward man perish,
yet the inward man is renewed day by day. 17 For our light affliction,
which is but for a moment, worketh for us a far more exceeding and
eternal weight of glory; 18 While we look not at the things which are
seen, but at the things which are not seen: for the things which are
seen are temporal; but the things which are not seen are eternal. 5
For we know that if our earthly house of this tabernacle were
dissolved, we have a building of God, an house not made with hands,
eternal in the heavens. 2 For in this we groan, earnestly desiring to
be clothed upon with our house which is from heaven