It was a matter of wisdom, not trust. Joseph must obey God above all things.
It is not wise to base decisions on a supernatural revelation given to someone else. So, whether Joseph wants to believe Mary's account of the angel or not, he should not make decisions based on it. Though sometimes forgotten today, this is a teaching in Scripture. For instance ...
Legitimate prophets received true warnings from God about the dangers Paul would experience when he went back to Jerusalem. Paul considered these revelations to be from the Holy Spirit ("in every city the Holy Spirit warns me" - Acts 20:23) yet he did not consider them God's guidance to change his plans.
Agabus dramatically told the warning the Holy Spirit had given him about Paul's impending capture, being believed by Luke and the others who pleaded with Paul to change his plans (Acts 21:12). But Paul, wisely, did not let that revelation change the plans he had made based on his mission from God and his understanding of Scripture.
Note that Paul would readily have changed his plans if God had given the revelation to him directly. The Holy Spirit had changed Paul's plans several times on his second missionary journey. See Acts 16:6-10 which culminates with the vision of a man saying, "Come over to Macedonia and help us." Paul obeyed God's leading and went to Macedonia.
Of course there are occasions when someone claims to receive revelation from God but did not. How can one know the false from the true? To follow such "guidance" is dangerous and unwise.
The old prophet in Bethel lied to the man of God from Judah in 1 Kings 13:18. The man of God took this message to be God's guidance and it resulted in a lion killing him (1 Kings 13:24).
In effect, I believe Joseph, being righteous, chose to end the betrothal, based on his best understanding of Scripture. Then, being righteous, he decided to make the divorce as painless for Mary as possible. Her story of the angel, whether she was telling the truth or not, could not be considered in his plans.
But when God sent the angel directly to Joseph, he instantly submitted to God's command.