This is a very interesting question, because the 'Curse of Canaan' passes all the way from Genesis 9 to Zech. 14:21,"Yes, every pot shall be holiness unto the Lord of hosts: and all they that sacrifice shall come and take of them, and seethe therein: and in that day there shall be no more the Canaanite in the house of the Lord of hosts."
This clearly goes beyond the Num. 14:18 timeline of "...no means clearing the guilty, visiting the iniquity of the fathers upon the children unto the 3rd and 4th generation."
Ham, in spite of his transgression, was blessed becaused God had blessed him,"And God blessed Noah and his sons..."(Gen. 9:1) therefore, Noah could not undo what God had already done. Yet Canaan, though as of this time had committed no transgression, is made to 'pay' for the sin of his father-and it lasts for eternity!
I believe it goes beyond the scope of the text to 'assume sexual immorality', although Canaanites were abounding in this sin-the Ashtorath and Baal worship involved going into the prostitutes to obtain favors: Hos. 4:13-14 spells out this practice.
At the heart of issue was 'seeing his father's nakedness, and told his 2 brothers'(vs 22). The response of the brothers tells us of the severity of Ham's transgression, for even though there was no commandment "Thou shall honor thy father and thy mother", they with their action DID honor their father, and not repeat the sin of Ham.
I do believe we must search other references to Canaan to understand the truth: we find such a reference in the Book of Jubilees. Just as in the Book of Enoch, it is not 'canonical', yet certain passages are taken directly out of it in Scripture.
In Jubilees 10:30 it talks about sedition, in which Canaan is warned by Ham his father, and Mizraim and Cush his brothers,"Thou hast settled into a land which is not thine, and which did not fall to us by lot: do not do so(take the land by force): for if thou dost do so, thou and thy sons will fall in the land and be accursed through sedition; for by sedition ye have settled, and by sedition ye will fall, and be rooted out forever."
This of course explains why God rightly and properly gave the land of Canaan to Israel-it was never theirs in the 1st place; and also it explains the current struggle with the Palestinians, who through sedition, have attempted to overthrow the the King of Jordan, and were expelled, attempted to overthow Lebanon, until they were expelled in the '80's, attempted to take over Tunisia until 'asked to leave' in the '90's', and of course, no one remembers the Muslim Brotherhood which spawned Hamas, assassinating Sadat, and attempting to overthow the Egyptian government until put down by Mubarak in '81, in which Hamas was exiled from Egypt.
The sin of Canaan was that he dishonored his father, along with grandfather Noah, and took land not belonging to him, creating emnity and conflict which carries through to this very day.