"Vain repetition" refers to spells and prayers used in pagan religious practice. I will not re-hash the greek here as others have done an excellent job, but I will attempt to provide some context and background.
Repeating a divine name was thought to cause a god to better hear you and for the one offering up the incantation to better bend the ear of the subject of his prayer - typically in order to cause the spell or incantation to have more effect.
This principle is best illustrated in the form of a legend about the Egyptian god and goddess Ra and Isis. In this legend, Ra becomes injured, and Isis uses this fact as leverage to learn the divine name of Ra. Isis tells Ra that she could only heal him if she knew his secret name. Isis immediately cured Ra, but he could not take back the power that he had granted her by telling her his true name and from that point on Isis was equal even to the sun god in power.
It was believed in most mesopotamian cultures in antiquity that a god's true divine name contained power and that by learning that divine name an individual could control a god and gain power over that god. Therefore, most ancient spells spells and incantations involved some wording along the lines of "By the name of [divine name] I command [action]" - because it was believed that this lent the spell power. For example, on page 124 of Jewish Aramaic Curse texts from Late-Antique Mesopotamia by Dan Levene we see a spell in which the canter is instructed to use the name of Hadriel and Shakniel to silence "evil and violent people who stand gainst Berik-Yeheba son of Mama"
In the name of Hadriel, Shakniel, the well, the stone, and the pit, I adjure, I adjure you, in the name of he who is great and frightful, that you may silence from Berik-Yehaba son of Mama the mouth of all the people who write books, who sit in forts, who sit in market places and in streets, and who go out on the roads.
Another on page 46 seems to utilize as many names as possible as a power-enhancement tactic for the spell
I have adjured you by the holy angels, and by the name of Metatron the pure angel, Nidrel and Nuriel and Huriel and Sasgabiel and Hapkiel and Mehapkiel, shose seven angels that are going adn overturning the heavens and the earth and the stars and the zodiac signs and the moon and Plaedes. May you go and overturn evil sorceries and powerful magical acts...
This is also why Jews do not speak or write the name Yahweh to this day - it is a sign of respect, but few realize that this is the reason why. It would be disrespectful to speak this divine name in any attempt to control the one true God.
This then allows us to better understand why God does not use the actual Tetragramaton when responding to Moses in Exodus 3:14. Moses clearly knows whom he is speaking to, but is fishing for God's divine name. Instead of giving it, God answers with a name similar to his actual name in much the same way Ra answers Isis with his lesser names (notice God's answer is only one letter of difference from the spelling of Yahweh). Instead of giving his name, God responds by saying (as Dick Harfield noted in his answer) "I am who I am and I will be what I will be." With this one pithy response, God has both answered Moses and signaled to him that he will not be controlled by any mere mortal and that no use of his Divine Name will control him.
Jesus actually tells a parable about something similar to vain repetition in Luke 18:1-8:
Then Jesus told them a parable to show them they should always pray and not lose heart. He said, “In a certain city there was a judge who neither feared God nor respected people. There was also a widow in that city who kept coming to him and saying, ‘Give me justice against my adversary.’ For a while he refused, but later on he said to himself, ‘Though I neither fear God nor have regard for people, yet because this widow keeps on bothering me, I will give her justice, or in the end she will wear me out by her unending pleas.’” And the Lord said, “Listen to what the unrighteous judge says! Won’t God give justice to his chosen ones, who cry out to him day and night? Will he delay long to help them? I tell you, he will give them justice speedily. Nevertheless, when the Son of Man comes, will he find faith on earth?”
This parable is distinct from vain repetition however in that it is making the point that God is one who hears. When we pray to God, our prayer is never in vain. God is real true God, unlike other gods who do not hear the prayers of the supplicant. This also means that there is no need for repetitive supplication and therefore discourages it - because God is a God who hears his people.
In this way, God is distinguishing himself from other gods as both a God who hears his people and a God who shall not be controlled..
