The OP question is put in a kind of a rhetorical way, suggesting obliquely yet quite clearly that neither the Virgin Mary, nor the Lord's brothers nor disciples would consider Him to be God, for He said that He was going to His God - His Father, and that the same God was their Father as well.
In fact, this is not even a question, for the author of the OP, from the outset, holds adamantly that Trinity is, as he says in one of his comments, “a bogus dogma” and that this site is for dismantling and debunking such dogmas. In short, the purpose of the question is not to really clarify for himself something that is sincerely curious or interesting for him, but to attack the Trinitarians in a form of this ersatz-question in order to push his, to use traditional theological term, heretical agendum. To conclude: it is a perfectly insincere and ideology-based or ideologically biased question.
In what follows I will show how misplaced such a smug anti-Trinitarian conviction is and that Trinitarian vision is the Theo-logical outcome - and a necessary one for that matter - of a sincere, grammatically sound consideration of the Scripture, the outcome of a 100% exegesis and a 0% of eisegesis.
We learn from the Scripture, that God is called "Father" by Christ with His own reference in a totally different sense than with reference to other humans: He calls the Solomon’s Temple exclusively His Father’s Temple (Luke 2:49) and He, Christ is the "only-begotten Son", who is the necessary mediator for others to get access to God the Father (John 14:6), whereas He, surely, has this access immediately, and if so, then He also knows the Father without mediation, being the only Principle through whom the Father can be known and Their - the Father’s and the Son’s - mutual knowledge of Each-Other is equal and infinite (Matthew 11:27), and if so, then the Son, Christ is equal to God, and only God can be equal to God.
The crucial theological question is whether Jesus is necessary only for humans to access God the Father, or Jesus is necessary also for God the Father to access humans? Since surely the second is the case, (and this does not even need any quote from Scriptures for it is analytically so that if Jesus' incarnation is necessary for that new degree of intimacy for men that that they can call God the "Father", then Jesus out of logical or even analytical necessity is necessary for the Father for entering into this new intimacy with men) then Jesus is co-God with the Father out of theo-logical necessity.
That’s why in many an instance in Gospels the Lord allows Himself to be worshipped and when He comes to them after resurrection He is explicitly worshipped by the disciples (Matthew 28:17), which is a sacrilege unless He holds the same worshipable dignity as God; in fact, does not one of them, Thomas address the resurrected Lord - “my Lord, my God” (John 20:28)? And how otherwise even to think, for Thomas saw a clear logic: the one who says that He has authority to give His life and take it back after three days (John 10:18), and shows through resurrection that these words were not some idle talk or theological metaphors, but actual reality, then how else one can take it? Who but God has authority to remove life and give it back again to a mortal biological organism, human body? Thomas' confession is logical and the only possible, for neither any men or any angel can possess such an authority as to do such things without any prayers to God, but sovereignly, as God.
Thus, surely brothers and apostles have the same God to whom (according to human nature) ascends His co-eternal Son, worshipped by the brothers and disciples alongside with the Father.
Eventually, it is He, the only-begotten Son who bestowed upon mankind such an intimacy as to allow us to call God “father”, but He has this intimacy naturally and eternally, while we derivatively, through grace bestowed by Him, the Lord Jesus Christ (cf. John 1:18). That is the sign of the bestowal, when He adds "and your God", for He wants them to be in the same intimate relationship with God the Father as He is, for neither Son nor the Father grudge Their divine life from humans created by Them, but want them to share in Their Life through the Life-creating Spirit that eternally comes out of the Father.
Thus, does He say that He ascends to God in sense that He ascends to the Holy Trinity? No! He says that He ascends to the Father according to His human nature, for according to His divine nature the Logos is always inseparably with the Father in Heavens, as He explicitly says that even while being on earth according to human nature, He simultaneously is with the Father in Heavens (John 3:13). But implicitly He of course, according to His human nature, ascends to the Holy Trinity, for God the Father is always there on the same level with the Son/Logos who is eternally born from Him and the Holy Spirit who eternally comes out of Him.
Moreover, see after all this drastic difference between God who "ascends amidst shouts of joy" (Psalm 47:5), that is to say Christ, and us, simple humans: He, Christ ascends with mediation of nobody, as the natural Son of the Father, whereas - just tell yourself sincerely! - can you or any man ascend to God the Father without mediation of Christ? Of course not (John 14:6)! And to Whom you shall hopefully ascend through Christ, that is to say to the Father, will not you see there on the same level also His Son, Christ, for He sits on the right hand of the Father (Mark 16:19), which is a metaphor standing for the "on the same level with". Thus, is it possible for you to ascend to the Father without Christ and without co-ascending to Christ God?
Therefore, to ascend to the Father without co-ascending to the Son - and to the H. Spirit for that matter - is a theological impossibility and who assert this impossibility are not theologians, but idle quibblers about the Bible.
And of course Christ and His brothers have the same God, but He as the unique man having, along with a fulness of the adopted human nature, the same uncreated divine Nature with the Father and uncreated divine Hypostasis distinct from that of the Father, whereas brothers and apostles have everything - both natures and hypostases/personalities - created in themselves.