Many reasons for this could be speculated, but four factors emerge.
1.) As you have stated, Mary was a woman and would not be afforded the same speaking rights today. In addition to this, Mary would have also been scoffed at as someone suspected of having a child out of marriage.
2.) Just as today, people may not have always referred to where they were from as where they were born. Jesus only spent a short amount of time as a baby before His family had to flee from Herod in Matt. 2:13. After this, he spent the next few decades in Nazareth.
3.) Keeping geography in mind, we must remember the sheer distance between Jerusalem and Nazareth. On top of this many Jews refused to travel through the Samaritan territory in between Judea and Galilee, opting to take a long water route instead. While Jewish leaders may have been willing to travel that distance to confront Jesus, it is unlikely that they would have gone that far to talk with Mary.
4.) Consistently throughout the Gospels, Jesus "deflects" claims about him being the Christ. Continually, Jesus rebuffs people calling him Christ in favor of the term Son of Man. The Christ that most people expected was not who Jesus came to be. While they desired a robust military ruler, Jesus combated misconceptions and instead talked about his death and resurrection. Jesus was never in the business of proving he was the Messiah, but rather being it whether people liked it or not.