There are at least three explanations of what the rest refers to.
The Greek phrase here is τῶν δὲ λοιπῶν (and of the rest) οὐδεὶς ἐτόλμα (none dared) κολλᾶσθαι αὐτοῖς (to join [with] them).
It has been conjectured that λοιπῶν is a misprint and what was originally in the text at one time was Λευειτῶν, yielding the reading:
and of the Levites (Λευειτῶν) none dared to join with them
Another conjecture is that in some Aramaic text of Acts, the word שֵׂביוּתָא - meaning "elders" - was misread as שֵׁירִיתָא - meaning "the rest" - and was so translated in Greek. This seems rather far-fetched to me, since Luke was a Gentile.
Both of the above conjectures are described in Metzger's A Textual Commentary on the Greek New Testament (2nd ed.).
John Chrysostom's (c 349-407) Homily XII on Acts understands the them (αὐτοῖς) to mean the Apostles, so that the sense of Acts v.12-14 is that while many did, in fact, join themselves with the believers (πιστεύοντες), they did not dare to associate closely with the Apostles - out of fear of what had just happened to Ananias and Sapphira (v.1ff):
After this fear had come upon them, he wrought more miracles; both Peter and the rest; “And by the hands of the Apostles were many signs and wonders wrought among the people; and they were all with one accord in Solomon’s porch. And of the rest durst no man join himself to them,” i.e. to the Apostles.
Bede's comment here is, "The punishment of the two who had fraudulently tried to be joined with them gave an example to the rest."