You are correct in that the latter clause does not name either Yahveh or Samuel.
Hebrew text:
וַיִּגְדַּל שְׁמוּאֵל וַיהוָה הָיָה עִמּוֹ וְלֹא הִפִּיל מִכָּל דְּבָרָיו אָרְצָה
English translation:
And Samuel grew up, and Yahveh was with him, and he did not let any of his words fall to the ground.
Or,
And Samuel grew up, and Yahveh was with him, and he did not cause any of his words to fall to the ground.
In his entry on the verb נָפָל, binyan Hifʿil, Heinrich Friedrich Wilhelm Gesenius wrote,1

Hence the expression הִפִּיל + דְּבָרִים + אָרְצָה means for a promise to be broken. Preceding the phrase with a negative particle expresses the idea of keeping one’s promise (i.e., not allowing one’s promise to be broken).2
In the case of Samuel, it was Yahveh who made promises to Samuel.3
In 1 Sam. 3:12,4 Yahveh says,
In that day, I shall cause to stand (אָקִים) unto5 Eli, everything that I spoke unto5 his house. When I begin, I shall also make an end.
בַּיּוֹם הַהוּא אָקִים אֶל עֵלִי אֵת כָּל אֲשֶׁר דִּבַּרְתִּי אֶל בֵּיתוֹ הָחֵל וְכַלֵּה
Causing words to stand (expressed by the binyin Hifʿil conjugation of the verb קוּם) is the antithesis of causing words to fall (expressed by the Hifʿil conjugation of the verb נָפָל).
words stand = promise fulfilled :: words fall = promise broken
words do not stand = promise broken :: words do not fall = promise fulfilled
Since Yahveh said He would cause the words He spoke to stand, then it is Yahveh who did not cause those same words to fall (and be broken). Hence, the phrase וְלֹא הִפִּיל מִכָּל דְּבָרָיו אָרְצָה refers to the fact that Yahveh did not break His promises to Samuel and everything occurred which Yahveh promised to and/or by Samuel the prophet concerning Eli and his house.6
References
Gesenius, Heinrich Friedrich Wilhelm. Gesenius’s Hebrew and Chaldee Lexicon to the Old Testament Scriptures. Trans. Tregelles, Samuel Prideaux. London: Bagster, 1860.
Footnotes
1 p. 557
2 cp. Jos. 21:45, 23:14; 1 Kings 8:56
3 cp. 1 Sam. 3:11–14
4 cp. 1 Sam. 1:23 and Gesenius, p. 728, קוּם, Hiphil, (2), (c)
5 or “concerning”
6 cp. 1 Kings 2:27