Read in isolation the text does not tell us clearly whether those who came and said "He is beside himself" were friends or family. However, the context gives a possible answer, because Mark 3:20-21, Mark 3:22-30, Mark 3:31-35 form an intercalation or sandwich (A1-B-A2). This is listed by John Dominic Crossan in The Birth of Christianity, page 106, as among the six Markan intercalations most widely recognised by New Testament scholars.
A Markan intercalation is a literary device the author uses for mutual emphasis of two otherwise unrelated issues. The first issue begins with the claim that Jesus had lost his senses (A1), followed by the scribes saying that Jesus was in league with the devil (B), then Jesus rejecting his family, saying that, for him, his followers were now his mother and brethren (A2). Family ties were strong, so Mark needed a compelling reason to say that Jesus would reject his family, and their prior rejection of him provides that reason.
This sandwich matches the people who came and said "He is beside himself" (Mark 3:20-21, event A1) with his brethren and his mother (Mark 3:31-35, event A2), providing evidence that the people in Mark 3:21 were actually Jesus' family.