I go back and forth on my interpretation of Gen 6:1-4 as to if it is fallen angels or people. Fallen angels is certainly a view with history. That is the interpretation that the authors of 1 Enoch had (see especially Book of the Watchers). However, Walter Kaiser gives a good defense of the sons of God being human beings in The Old Testament Documents: Are They Reliable and Relevant?
However, regarding Daniel, the answer can be found looking at the Aramaic (this is from the Aramaic portion of Daniel).
...Mith'rbiyn lehon bizra' anasha' wla-lehon dabqin danah 'am danah...
The NET Bible translates this as:
2:43 ...so people will be mixed with one another without adhering to one another... [NET Bible. Emphasis added]
The word for people is anisha, "the people." This uses the Aramaic definite article (an aleph at the end instead of a he at the beginning). This word anish means "man, human being, mankind (as a collective).
One of the sites I found talking about this verse and giving credence to the nephilim interpretation relied heavily on the King James. "And whereas thou sawest iron mixed with miry clay, they shall mingle themselves with the seed of men: but they shall not cleave one to another, even as iron is not mixed with clay." It then goes on to draw a distinction between "they" and the "seed of men" saying that the they refers to the 10 toes of the empire and thus it is not the "seed of men" mixing together ("the fact that 'they' are differentiated from mankind hints that “they” may not be human"). This is not the case.
The verb driving the this portion of the sentence is Mith'rbiyn, the hithpa'el plural participle form of 'arab. Hithpa'el forms of verbs are reflexive. That is, the form indicates the subject is acting on itself (it both acts and receives the action). For example, two men wrestling could be expressed by a reflexive plural form (Exodus 2 does this using a reflexive niphal instead of a hithpael).
Thus it means "the seed of men will mix themselves" and does not indicate that any of the mixers are not the seed of men. In fact, the reflexive form tells us that the seed of men are the ones mixing together.
More information on the hithpael stem can be found here and here.