The differences in the Greek from verse 39 to verse 40 are interesting. But it seems that the references and pictures Jesus gives to support what he is saying would make the ones who are taken the wicked.
Matthew 24:36-44 ESV “But concerning that day and hour no one knows, not even the angels of heaven, nor the Son, but the Father only. 37 For as were the days of Noah, so will be the coming of the Son of Man. 38 For as in those days before the flood they were eating and drinking, marrying and giving in marriage, until the day when Noah entered the ark, 39 and they were unaware until the flood came and swept them all away, so will be the coming of the Son of Man. 40 Then two men will be in the field; one will be taken and one left. 41 Two women will be grinding at the mill; one will be taken and one left. 42 Therefore, stay awake, for you do not know on what day your Lord is coming. 43 But know this, that if the master of the house had known in what part of the night the thief was coming, he would have stayed awake and would not have let his house be broken into. 44 Therefore you also must be ready, for the Son of Man is coming at an hour you do not expect
In verse 38 he says that "they were eating, and drinking, marrying and giving in marriage" speaking of the wicked. In verse 39 he says that "they were unaware until the flood came and swept them away." Again speaking of the wicked. The context is on those who were "swept" or "taken" away. He then says this is the same thing that will happen when he returns. The wicked will be taken and the righteous left. It wouldn't make sense to all of a sudden reverse the context from speaking about in the days of Noah and the wicked being swept or taken away and then say that when he returns the opposite would happen. If that were the case then he would have said the coming of the Son of Man will be the opposite of the days of Noah.
Also, in connection to what he is saying in verse 43 and 44 he refers to himself as a thief. A thief does not steal what is his. A thief steals what is not his. It seems he is telling his disciples to be ready and watchful over themselves so that they remain his and do not suffer loss in that day that he returns.
The passage in Luke 17:37 gives us the answer.
And they said to him, “Where, Lord?” He said to them, “Where the corpse is, there the vultures will gather.”
Again the context to the return of Jesus is in the days of Noah, where judgment swept over the earth. Also in the day of Lot, when Sodom and Gomorrah were judged. The context is clearly judgement and that many people lost their lives. When they asked for clarity on where the one was taken, he clarify's that it's not a place that they are taken too, it's that their lives are taken.
This also lines up with Old Testament prophecies of what will happen when the Messiah appears.
Isaiah 66:15-16 “For behold, the Lord will come in fire,
and his chariots like the whirlwind,
to render his anger in fury,
and his rebuke with flames of fire.
16For by fire will the Lord enter into judgment,
and by his sword, with all flesh;
and those slain by the Lord shall be many.
Also in Revelation, the return of Jesus is described the same way
Revelation 19:11-18 Then I saw heaven opened, and behold, a white horse! The one sitting on it is called Faithful and True, and in righteousness he judges and makes war. 12 His eyes are like a flame of fire, and on his head are many diadems, and he has a name written that no one knows but himself. 13 He is clothed in a robe dipped in blood, and the name by which he is called is The Word of God. 14And the armies of heaven, arrayed in fine linen, white and pure, were following him on white horses. 15 From his mouth comes a sharp sword with which to strike down the nations, and he will rule them with a rod of iron. He will tread the winepress of the fury of the wrath of God the Almighty. 16 On his robe and on his thigh he has a name written, King of kings and Lord of lords. 17 Then I saw an angel standing in the sun, and with a loud voice he called to all the birds that fly directly overhead, “Come, gather for the great supper of God, 18 to eat the flesh of kings, the flesh of captains, the flesh of mighty men, the flesh of horses and their riders, and the flesh of all men, both free and slave, both small and great.”
It's important to see that Jesus is not giving a detailed description of what the end of the age will look like. He is giving a summary of highlights. When taken with other "end of the age" scriptures from both the Old Testament and the New Testament we get a clearer picture. But I must say that we won't understand it all until after it's over. Much like the disciples during Jesus' first coming. They understood vaguely what Jesus was saying about his death, burial, and resurrection. It didn't come together for them until after it all happened. The reality is that it doesn't matter how much someone tells you about something you've never seen, it's not crystal clear until you actually witness it for yourself.