Assuming a literal reading of the text (which is how the hermeneutic I hold takes Genesis), then in one 24 hour period, the 6th day of creation, Adam (and by extension on some of the points, Eve) was:1
- Made fully capable of understanding language, as God spoke to them (Gen 1:28; cf. Gen 2:15-17)
- Made fully capable of sexual reproduction to multiply on the earth (Gen 1:28)
- Made fully capable of immediately beginning to have dominion over the animals (Gen 1:28)
- Made fully capable of harvesting and eating from the plants (Gen 1:29)
- Made fully capable of tending to the garden in Eden (Gen 2:15)
- Made fully capable of being responsible for his actions (Gen 2:16-17)
- Made fully capable of cognitive thought in naming animals (Gen 2:19-20)
- Made fully capable of understanding the special relation he had to the woman (Gen 2:23)
These demonstrate Adam is not in a "babe" status during the first day of mankind's being on earth. Physical, mental, emotional, moral, and social aspects were fully in place. In God making everything "good" by the end of each day of creation, that part of creation was at a place of fully functioning perfection (only at the end of the day, for in the midst of day six, during the process of creation, not all was good and perfect yet; see for example Gen 2:18).
This need not conflict with your reading of Ps 139:15 (note: I am not necessarily agreeing with your connection of that to Adam's creation)2 since Adam was formed of the dust of the earth (Gen 2:7), in a figurative sense one could call that the "womb" he was made in/from as poetically displayed in the Psalm. But such a womb would not necessitate a "babe" form emerging, since it is obviously a different womb than that of a woman and different creation process than normal reproduction.
However, the points made above do not necessarily pinpoint Adam as a "middle aged man," but does offer much testimony that he was in the perfected state of man that God intended for all the duties He intended Adam for.
Whether this was at a physical state of what to us would seem to be 20, 30, or 40 years old, there is nothing by which to form such an exact "dogmatic conclusion." Strong, healthy (mind and body), and "fully developed" can be dogmatically asserted from the points above, but of his exact appearance in physique, Scripture is silent on.
1 For this answer, a literal six consecutive days of creation at the beginning of history is assumed, with Genesis 2 expanding upon details of the creation account in Genesis 1, as my hermeneutic holds. A defense of that is beyond the scope of this question, but foundational to understanding the answer as it is presented here.
2 I would tend to view the Ps 139:15 reference to be the opposite symbolism as you are taking it, that "lower parts of the earth" is a reference to the female womb, and David speaking of his own conception. This is because (1) Psalms is a poetic genre and far more prone to symbolism than Genesis (which is historical narrative), and (2) we know that David was born of a woman (Ruth 4:16).