The NET Bible has a helpful comment:
There is debate among commentators and scholars over the phrase one will be taken and one left about whether one is taken for judgment or for salvation. If the imagery is patterned after the rescue of Noah from the flood, as some suggest, the ones taken are the saved (as Noah was) and those left behind are judged. The imagery, however, is not directly tied to the identification of the two groups. Its primary purpose in context is to picture the sudden, surprising separation of the righteous and the judged (i.e., condemned) at the return of the Son of Man. [missing spaces inserted]
There's an excellent chance that Matthew had the text of Mark at hand when he composed this section. Mark 13:32 strongly parallels Matthew 24:36. But Mark doesn't include the sayings about Noah or the one taken and one left. On the other hand, the sayings are in Luke 17:26-37 (ESV):
Just as it was in the days of Noah, so will it be in the days of the Son of Man. They were eating and drinking and marrying and being given in marriage, until the day when Noah entered the ark, and the flood came and destroyed them all. Likewise, just as it was in the days of Lot—they were eating and drinking, buying and selling, planting and building, but on the day when Lot went out from Sodom, fire and sulfur rained from heaven and destroyed them all—so will it be on the day when the Son of Man is revealed. On that day, let the one who is on the housetop, with his goods in the house, not come down to take them away, and likewise let the one who is in the field not turn back. Remember Lot's wife. Whoever seeks to preserve his life will lose it, but whoever loses his life will keep it. I tell you, in that night there will be two in one bed. One will be taken and the other left. There will be two women grinding together. One will be taken and the other left.” And they said to him, “Where, Lord?” He said to them, “Where the corpse is, there the vultures will gather.”
By adding the story of Lot fleeing from Sodom, Luke emphasizes the concept of the one leaving the scene being righteous and the one who stays (or even looks back) being wicked. Further, Luke talks about the wicked being "destroyed" rather than being "took". (I don't know what to make of the vultures gathering, however. Perhaps that could be a separate question.)
The usual theory for passages included in both Luke and Matthew, but not Mark, is the Q hypothesis. The story of Noah and the one taken and one left are usually assigned to Q. (See New Testament History: A Narrative Account by Ben Witherington III for instance.) Therefore, these sayings were written down quite early and it's possible they were the basis for a passage in one of Paul's earliest letters, 1 Thessalonians 4:13-18 (ESV):
But we do not want you to be uninformed, brothers, about those who are asleep, that you may not grieve as others do who have no hope. For since we believe that Jesus died and rose again, even so, through Jesus, God will bring with him those who have fallen asleep. For this we declare to you by a word from the Lord, that we who are alive, who are left until the coming of the Lord, will not precede those who have fallen asleep. For the Lord himself will descend from heaven with a cry of command, with the voice of an archangel, and with the sound of the trumpet of God. And the dead in Christ will rise first. Then we who are alive, who are left, will be caught up together with them in the clouds to meet the Lord in the air, and so we will always be with the Lord. Therefore encourage one another with these words.
Unfortunately, this is a controversial text, but if Paul has in mind Matthew 24 his interpretation is clear. He is looking to a future coming of the Lord and notes that some of the the Thessalonians are concerned that the brothers and sisters who are dead will be among those who are left and thus counted as wicked. Paul counters, using language from the Jewish and early-Christian resurrection hope, that the dead and the living will be reunited in the coming of the Lord. Both will be taken if they are counted among the righteous. (I think Paul is subtly shifting terms by saying those who are alive are "left". The dead have already been taken to await the future resurrection. But it's not completely clear.)
Finally, while many translations use the "took" language in verse 39, some (ESV, NLT, MSG, and NRSV) use the verb "swept". When you think about it, the people who were "taken" were really the ones in the ark who were lifted away from the earth, while the ones who were pulled into the water were left behind to their fate. As you mention, the connection between the wicked and the taken turns mostly on the English translation rather than directly in the Greek.