"Brothers" in Greek
The Greek word for "brothers" here is adelphoi (Strongs G80). This means literally "brothers". However, it can also mean "countryman" or "followers".
The NET Bible (which uses the most current translation, taking advantage of the latest in linguistic scholarship) translates this as "brothers and sisters". The footnote for this says of this word:
Grk “brothers,” but the Greek word may be used for “brothers and sisters” (cf. BDAG 18 s.v. ἀδελφός 1, where considerable nonbiblical evidence for the plural ἀδελφοί [adelfoi] meaning “brothers and sisters” is cited). In this context Jesus is ultimately speaking of his “followers” (whether men or women, adults or children), but the familial connotation of “brothers and sisters” is also important to retain here.
They show that Jesus is clearly speaking of his followers here even though they retain the familial connection.
Contextual Analysis
If we look at the entire context that you site (verses 31-46) we can see that there is strong support for Jesus meaning "followers" when he says brothers
The parable is talking about a King that separates out everyone on earth into two groups: the sheep and the goats. On his right will be the sheep that will enter heaven and on his left, the goats who get sent to eternal punishment.
In verses 34-36, Jesus says that the sheep helped him out when he was in need. Verses 37-39 are the sheep asking "When did we see you?"
Jesus replies with verse 40 (as you mention in the question) saying that if you did it to the "least of my adelphoi" (brothers or followers), then you did it to me.
Following the story down further, the opposite is true with the goats. However, when the goats ask "When did we see you?", the King (in the parable) replies:
Matthew 25:45b (NIV)
'Truly, I say to you, as you did not do it to one of the least of these, you did not do it to me.'
This response shows further support for the translation of "brothers" as followers because here the King is indicating that whoever did not help the least of the "these" (implying the "goats") did not help him.
Additional support
We can see this theme played out elsewhere, showing that these "brothers" are followers of Jesus:
Matthew 10:40-42 (NIV)
40 “Anyone who welcomes you welcomes me, and anyone who welcomes me welcomes the one who sent me. 41 Whoever welcomes a prophet as a prophet will receive a prophet’s reward, and whoever welcomes a righteous person as a righteous person will receive a righteous person’s reward. 42 And if anyone gives even a cup of cold water to one of these little ones who is my disciple, truly I tell you, that person will certainly not lose their reward.”
This is Jesus speaking to his disciples. Clearly, there's additional support for interpreting this passage later in Matthew (and the word adelphoi) as "followers".
"Least of"
It's interesting that Jesus uses this phrase. It seems to indicate that there is a "sorting order" or a way to organize his followers. If this is true, this means that while any one of his followers are still considered his followers, there are some that he considers "greater" than others and one in particular that he considers the "greatest". We can see this in opposition of the phrase "the least of".
Summary
Clearly, "the least of my brothers" is referring to the followers of God. This parable encourages us to help out other Christians since that would be directly helping out God.