Modern scholars no longer expend much effort on whether Moses wrote the Book of Genesis, instead analysing the evidence for authors such as the Yahwist, the Elohist and the Priestly Source. We can go back to the nineteenth century, when commentators really began to investigate the possibility of Mosaic authorship and soon found evidence against it.
Samuel Davidson, D.D, in An Introduction to the Old Testament, Critical, Historical, and Theological, Containing a Discussion of the Most Important Questions Belonging to the Several Books (published 1862), identified several clear lines of evidence that Genesis can only have been written long after the time of Moses. Among these:
A. Words that obviously imply that when the writer lived, the Canaanites and Perizzites had been expelled from the land -
Genesis 12:6:And Abram passed through the land unto the place of Sichem, unto the plain of Moreh. And the Canaanite was then in the land.
Genesis 13:7: And there was a strife between the herdmen of Abram's cattle and the herdmen of Lot's cattle: and the Canaanite and the Perizzite dwelled then in the land..
B. Hebron is the name almost always used in Genesis, yet the city did not get that name until Caleb changed the name of the city from Kirjatharba to Hebron, meaning that this name is posterior to Moses:
Genesis 23:2: And Sarah died in Kirjatharba; the same is Hebron in the land of Canaan: and Abraham came to mourn for Sarah, and to weep for her.
Joshua 14:14-15: Hebron therefore became the inheritance of Caleb the son of Jephunneh the Kenezite unto this day, because that he wholly followed the LORD God of Israel. And the name of Hebron before was Kirjatharba; which Arba was a great man among the Anakims. And the land had rest from war.
C. In Genesis 14:14, Abraham is said to have pursued the kings who carried away Lot his nephew, as far as Dan. But we learn from Joshua 19:47 and Judges 18:29, that the name of the place was Laish, until the Danites possessed it and called it Dan, "after the name of their father."
D. Davidson says that, because of Genesis 36:"31, the book could hardly have been written before there reigned any king over the land of Israel:
Genesis 36:31: And these are the kings that reigned in the land of Edom, before there reigned any king over the land of Israel.
E. Here Jacob is referring to the "land of the Hebrews" at a time when it was supposedly the land of the Canaanites, a mistake Moses could scarcely have made:
Genesis 40:15: For indeed I was stolen away out of the land of the Hebrews: and here also have I done nothing that they should put me into the dungeon.
This is far from the totality of evidence that Moses did not write the Book of Genesis, but it is enough to be conclusive.