You are correct in that the latter clause does not name either Yahveh or Samuel.
וַיִּגְדַּל שְׁמוּאֵל וַיהוָה הָיָה עִמּוֹ וְלֹא הִפִּיל מִכָּל דְּבָרָיו אָרְצָה
And Samuel grew up, and Yahveh was with him, and he did not let any of his words fall to the ground.
And Samuel grew up, and Yahveh was with him, and he did not cause any of his words to fall to the ground.
In his entry on the verb נָפָל, binyan Hif'il, p. 683, Heinrich Friedrich Wilhelm Gesenius noted the following:
Hence the expression הִפִּיל + דְּבָרִים + אָרְצָה means for a promise to be broken. Preceding the phrase with a negative particle expresses the idea of keeping one's promise (i.e., not allowing one's promise to be broken).
In the case of Samuel, it was Yahveh who made promises to Samuel (1 Sam. 3:11-14).
In 1 Sam. 3:12, Yahveh says,
In that day, I shall cause to stand (אָקִים) unto1 Eli, everything that I spoke unto1 his house. When I begin, I shall also make an end.
1 or "concerning"
בַּיּוֹם הַהוּא אָקִים אֶל עֵלִי אֵת כָּל אֲשֶׁר דִּבַּרְתִּי אֶל בֵּיתוֹ הָחֵל וְכַלֵּה
To cause words to stand (expressed by the binyin Hif'il conjugation of the verb קוּם) is the antithesis of causing words to fall (expressed by the Hif'il conjugation of the verb נָפָל).
words stand = promise fulfilled :: words fall = promise broken
Since Yahveh said He would cause the words He spoke to stand, then it is Yahveh who did not cause those same words to fall (and be broken). Hence, the phrase וְלֹא הִפִּיל מִכָּל דְּבָרָיו אָרְצָה refers to the fact that Yahveh did not break His promises to Samuel and everything occurred which Yahveh promised to and/ or by Samuel the prophet concerning Eli and his house (cp. 1 Kings 2:27).
Gesenius, Heinrich Friedrich Wilhelm. A Hebrew and English Lexicon of the Old Testament, including the Biblical Chaldee. Trans. Robinson, Edward. Boston: Crocker and Brewster, 1865. (source)