There are three possible reasons to think it is God who is speaking to Moses, in Exodus 24:1:
1. In the context of Exodus, there is that great "sermon" beginning in Exodus 20:22. That is going on and on and there is no another "And the LORD said to Moses ..." or anything similar until Exodus 24:1.
This makes me think that the fact that it is God who is speaking is like self understood. So the fact that it is God who is speaking is kind of assumed, as there is nothing really to make the reader think that there is a different person who is speaking.
2. In Exodus 24:1 the text inverts the usual syntactical order:
... וְאֶל־מֹשֶׁ֨ה אָמַ֜ר (BHS)
Compare for instance with:
Exodus 20:22
... וַיֹּ֤אמֶר יְהוָה֙ אֶל־מֹשֶׁ֔ה (BHS)
And the LORD said unto Moses ... (JPST)
Returning to Exodus 24:1:
Literally
... וְאֶל־מֹשֶׁ֨ה אָמַ֜ר (BHS)
would be: "And to Moses He said..."
See the JPST version:
And unto Moses He said ... (JPST)
The Greek version (LXX) is doing more or less the same, by putting Μωϋσῆς in dative and the verb εἶπον in aorist, active, indicative, third person, singular:
Καὶ Μωυσῇ εἶπεν ... (LXX)
Indeed, it is possible that this inversion means that the word spoken is addressed to Moses&co only, and not to the whole of the Israelites. This is why I've put this as a number 2 hypothesis.
3. Ellipsis. We can look at Exodus 24:1 in connection with Genesis 32:22-33 (the Jacob wrestling with the angel narrative). In Genesis 32:24 we find a text talking about Jacob fighting with a "man" (אִיש), then this person is referred to just like in Exodus 24:1 (in the English text, all of the -he- pronouns are related to "the man" from 32:24), and in the end the name of Jacob is changed into Israel, according to the explanation from:
Genesis 32:28:
Then he said, “Your name shall no longer be called Jacob, but Israel,
for you have striven with God and with men, and have prevailed.”
(ESV)
28:29 כִּֽי־שָׂרִ֧יתָ עִם־אֱלֹהִ֛ים וְעִם־אֲנָשִׁ֖ים וַתּוּכָֽל׃ ... (BHS)
We know that Jacob fought with an angel from:
Hosea 12:4
He strove with the angel and prevailed ... (ESV)
But this is another story.
I think your question can be answered if we parallel Exodus 24:1 and Genesis 32:22-33. The two are similar in this respect of the way in which the Hebrew Bible is speaking about God without nominating directly, yet revealing through actions (see above Sola Gratia's helpful comment +1). However, ellipsis is quite common in the Hebrew, and it is not always related to God or implying a reference to God. This is why I have put this as a number 3 hypothesis.