Brother here is referring to a fellow Christian.
See 1 John 3:13 (ESV), "Do not be surprised, brothers, that the world hates you." If the word brothers there implied the whole world, then this statement would make no sense.
In 1 John 3:14 and 16, John uses the term "the brothers". You would never add the word "the" if it implied the whole world.
1 John 5:1b, "... and everyone who loves the Father loves whoever has been born of him." Here, it's explicitly limited to the those who are born of God. Christians are born of God. Those who are not Christians are not born of God.
1 John 5:2 is an exact mirror of 1 John 4:21. Compare them:
1 John 5:2 "By this we know that we love the children of God, when we love God and obey his commandments."
1 John 4:21 "And this commandment we have from him: whoever loves God must also love his brother."
Note that chapter 4 and chapter 5 flow together. The chapter divisions weren't in the original. The point is that if we love God, we obey his commandments, which is loving other Christians (i.e. brother, children of God).
Now, I am not advocating that we not love our neighbor who is not a Christian. Other texts make it obvious we must love everyone, but 1 John is crystal clear in focusing that we must love fellow Christians.
Paul also echoes this in Galatians 6:10. "So then, as we have opportunity, let us do good to everyone, and especially to those who are of the household of faith." We ought to do good (and love) everyone, but to especially love believers with a special love is appropriate.
Why is that appropriate? Because Jesus did the same thing. See 1 John 3:16, John 10:11. He laid down his life for the sheep, not for the wolves. So Jesus has a special love for his own. And just as Jesus laid down his life for the sheep (fellow Christians), so we ought to lay down our life for the brothers (fellow Christians).