The NET Bible includes this translator's note:
Heb “the one coming out, who comes out from.” The text uses a masculine singular participle with prefixed article, followed by a relative pronoun and third masculine singular verb. The substantival masculine singular participle הַיּוֹצֵא (hayyotse’, “the one coming out”) is used elsewhere of inanimate objects (such as a desert [Num 21:13] or a word [Num 32:24]) or persons (Jer 5:6; 21:9; 38:2). In each case context must determine the referent. Jephthah may have envisioned an animal meeting him, since the construction of Iron Age houses would allow for an animal coming through the doors of a house (see R. G. Boling, Judges [AB], 208). But the fact that he actually does offer up his daughter indicates the language of the vow is fluid enough to encompass human beings, including women. He probably intended such an offering from the very beginning, but he obviously did not expect his daughter to meet him first.
Some translate “or,” suggesting that Jephthah makes a distinction between humans and animals. According to this view, if a human comes through the door, then Jephthah will commit him/her to the Lord’s service, but if an animal comes through the doors, he will offer it up as a sacrifice. However, it is far more likely that the Hebrew construction (vav [ו] + perfect) specifies how the subject will become the Lord’s, that is, by being offered up as a sacrifice. For similar constructions, where the apodosis of a conditional sentence has at least two perfects (each with vav) in sequence, see Gen 34:15-16; Exod 18:16.
An answer to a similar question at Jewish Life and Learning points out that Jephthah likely lived in a pillared house which was designed to shelter animals in side rooms on the first floor. It seems likely that prized animals would be allowed to move freely in and out of the house during the day while being stabled in the house at night. Presumably Jephthah assumed one of his valued animals would be wandering out of the house as he returned home.
There's also a clear indication of sacrificial animals being kept in the house in 1 Samuel 28:20-25 (ESV):
Then Saul fell at once full length on the ground, filled with fear because of the words of Samuel. And there was no strength in him, for he had eaten nothing all day and all night. And the woman came to Saul, and when she saw that he was terrified, she said to him, “Behold, your servant has obeyed you. I have taken my life in my hand and have listened to what you have said to me. Now therefore, you also obey your servant. Let me set a morsel of bread before you; and eat, that you may have strength when you go on your way.” He refused and said, “I will not eat.” But his servants, together with the woman, urged him, and he listened to their words. So he arose from the earth and sat on the bed. Now the woman had a fattened calf in the house, and she quickly killed it, and she took flour and kneaded it and baked unleavened bread of it, and she put it before Saul and his servants, and they ate. Then they rose and went away that night.
Another indication that people had close relationships with their animals at home is found in 2 Samuel 12:2-3 (ESV):
The rich man had very many flocks and herds, but the poor man had nothing but one little ewe lamb, which he had bought. And he brought it up, and it grew up with him and with his children. It used to eat of his morsel and drink from his cup and lie in his arms, and it was like a daughter to him.
Jephthah certainly sheltered animals in his house, but they probably weren't pets as we use the term.