Biblically speaking, a "man" was an adult, just as we would infer today; but the line between childhood and adulthood was not where in many countries it is drawn today. Many countries now consider 18 as the age of majority, though the ages for driving, voting, drinking, or even marrying, can vary somewhat from this rule. But in the Bible, there was a consistent mark: 20 years of age.
Consider the following passages in support of this figure:
Every one that passeth among them that are numbered, from twenty years
old and above, shall give an offering unto the LORD. (Exodus 30:14)
A bekah for every man, that is, half a shekel, after the shekel of the
sanctuary, for every one that went to be numbered, from twenty years
old and upward, for six hundred thousand and three thousand and five
hundred and fifty men. (Exodus 38:26)
A bekah for every man, that is, half a shekel, after the shekel of the
sanctuary, for every one that went to be numbered, from twenty years
old and upward, for six hundred thousand and three thousand and five
hundred and fifty men. (Numbers 1:20)
[Note: Numbers are given for those 20+ years of age for each of the other tribes as well.]
Importantly, the "generation" of those twenty years of age and older was that which perished in the wilderness during the 40 years of their wandering outside of Canaan.
Surely none of the men that came up out of Egypt, from twenty years
old and upward, shall see the land which I sware unto Abraham, unto
Isaac, and unto Jacob; because they have not wholly followed me:
(Numbers 32:11)
In terms of the usage of the word "man" in the text, this age of 20 is perhaps as much as one can legitimately infer. However, that Joseph was considerably older than this is evident from other passages and context. I touched on some of those evidences in my response to another question, found HERE.