John 21:20,24 says that the beloved disciple "wrote these words," and with the almost universal assumption that the Gospel of John was written by John son of Zebedee, that should answer the question. However, the gospel was originally anonymous, and Burton L. Mack says in Who Wrote the New Testament, page 215, that the Gospel was then attributed to the Gnostic, Cerinthus, so the attribution to John the disciple must have taken place quite late in the second century. Raymond E. Brown says in An Introduction to the New Testament, pages 362,368-9, that it is doubted by most scholars that this Gospel was written by an eyewitness of the public ministry of Jesus, which would now seem to rule the disciple John out as the beloved disciple.
James, brother of Jesus, should be ruled out because of John 26-27, in which Jesus asks the beloved disciple to treat Mary as if she were his mother, and tells Mary to think of the disciple as her son. This dialogue would be too stilted and meaningless if it were Mary's son standing at the foot of the cross.
The other identified candidates in the list can be ruled out as either too speculative and improbable, or because Mary Magdalene's gender is wrong for the gender of the text.
Now, we either give up and say, as in list item #5 from the question, that the beloved disciple was an "unknown priest or disciple," or we can extend the search outside the list.
Brown says (ibid, page 369) that some scholars have evaluated the beloved disciple as a pure symbol, created to model the perfect disciple. That he is never given a name and that he appears alongside Peter in scenes known to us from the Synoptic Gospels where no such figure is mentioned have been invoked as a proof of nonhistoricity. This certainly gives us a credible answer, which a priest of Brown's standing would be comfortable with.
However, we can take this position a step further, with textual evidence that the beloved disciple was not only a symbol of the perfect disciple but had a quite specific role in John's Gospel. The New Testament scholar, Elaine Pagels, was the first to realise why this disciple was described in this way. In Beyond Belief, pages 61-63, she explains that she discerns in John’s Gospel a distinct bias against Peter. She noticed that the beloved disciple, in all but one case, was contrasted with the apostle Peter and always came out the more worthy apostle. Even the name 'disciple whom Jesus loved' suggests to us that Jesus saw him as more worthy than Peter (or any other disciple, but it is only Peter he is compared with). John Dominic Crossan, in The Birth of Christianity, page 566, also points to typically oblique examples of the exaltation of the Beloved Disciple over Peter in John's gospel. Perhaps some extracts from John chapter 21 will serve to illustrate this:
- 21:7 - the disciple whom Jesus loved realised that it was Jesus and told this to Peter, who had remained unaware that the miraculous catch of fishes was the work of the Lord.
- 21:15-17, Jesus asks Peter three times whether he loves him. John Carroll explains in The Existential Jesus, pages 144-5 that this is Peter's final humiliation. Jesus addressed him each time theatrically as “Simon, son of Jonah”, not as Peter, the name always previously used by Jesus, as if to humiliate Peter in front of the other disciples. The first time, Jesus asks about sacred love (Greek: agape) and the question is comparative: Do you love me more than these [the other disciples]? Peter answers that he loves him, but only using the Greek word for friendly or brotherly love (philia).
Not satisfied, Jesus again asks Peter, “Do you love me?” Again, Jesus asks about sacred love (agape) but this time does not ask whether Peter loves Jesus more than the others. Again, Peter replies with the Greek word for brotherly love (philia). In the third questioning, Jesus asks only whether Peter had brotherly love for him (philia). He accepts that this was the most that Peter would give. Peter is upset that it has been necessary to ask him three times, but Jesus knows Peter had denied him three times.
- In 21:21, Peter petulently asks Jesus about the beloved disciple, "And what shall this man do?" Jesus quite properly responds, "If I will that he tarry till I come, what is that to thee?"
From the above, it seems that the author of John sought to tone down what he saw as excessive veneration of St Peter early in the second century. I believe the most credible explanation for the beloved disciple is that he was a literary creation intended to achieve a theological purpose.