Simply put, we don't know for sure, but we have clues!
We have three known facts about the Nephilim:
Their Name
The world "nephilim" comes from the Hebrew word nephiyl, which means "the fallen". What did they fall from? The Bible isn't clear about that.
It's possible that they fell from the sky (making them aliens) or from heaven (making them demons).
Regardless, it seems clear that the people who lived at that time called them "the fallen".
Lineage
Also, we know about their lineage: they are children of "sons of God" and "daughters of man":
Genesis 6:4 (NASB)
The Nephilim were on the earth in those days—and also afterward—when the sons of God went to the daughters of humans and had children by them. They were the heroes of old, men of renown.
So, the Nephilim themselves weren't actually the angels/aliens/kings, but the children of these beings.
Size
Another thing we know about Nephilim is that they were tall--huge, even:
Numbers 13:33 (NASB)
We saw the Nephilim there (the descendants of Anak come from the Nephilim). We seemed like grasshoppers in our own eyes, and we looked the same to them.
So, clearly, they are very large.
Aliens?!
One possible interpretation of this is the one you mentioned: that nephilim are a race of alien-human hybrids.
However, due to the Numbers reference, many translate the word "nephilim" to "giants". Of course, their size was only an aspect of who they were, but not necessarily the primary aspect. It could very well be that they had wings like angels or big bulbous heads like aliens.
Unfortunately, we simply do not know what nephilim were and many theories abound. One of which is, actually, that they were human-alien hybrids.