God gave circumcision as an external sign in response to Abraham's faith. Nigel J points out to a very interesting verse highlighting that
A servant who is born in your house or who is bought with your money shall surely be circumcised; thus shall My covenant be in your flesh for an everlasting covenant.
Genesis 17:13 NASB https://bible.com/bible/100/gen.17.13.NASB
But at some point in history, circumcision became more important than the decision to believe and follow God as the following two passages show
Circumcise yourselves to the Lord And remove the foreskins of your heart, Men of Judah and inhabitants of Jerusalem, Or else My wrath will go forth like fire And burn with none to quench it, Because of the evil of your deeds.”
Jeremiah 4:4 NASB https://bible.com/bible/100/jer.4.4.NASB
and
Behold, the days are coming,” declares the LORD, “that I will punish all who are circumcised and yet uncircumcised— Egypt and Judah, and Edom and the sons of Ammon, and Moab and all those inhabiting the desert who clip the hair on their temples; for all the nations are uncircumcised, and all the house of Israel are uncircumcised of heart.
Jeremiah 9:25-26 https://www.bible.com/bible/100/jer.9.25-26.NASB
What matters to God is doing the things for the right motives.
Moreover the Lord your God will circumcise your heart and the heart of your descendants, to love the Lord your God with all your heart and with all your soul, so that you may live.
Deuteronomy 30:6 NASB https://www.bible.com/bible/100/DEU.30.6.NASB
That's what leads Paul to be consistent in his speech, who also does so in Romans 2:29 while quoting from Jeremiah 4:4 and Deuteronomy 30:6 when speaking of circumcision of the heart. He wanted to give emphasis to genuine loyalty to God. That's what God's good, pleasant and perfect will is.
In other words, Paul argues that circumcision has value only if one keeps the law and Christ is the only one who does this work (so we all need Him)
and in Him you were also circumcised with a circumcision made without hands, in the removal of the body of the flesh by the circumcision of Christ
Colossians 2:11 NASB https://www.bible.com/bible/100/COL.2.11.NASB
If by now you're still considering if God cares about our heart, then why circumcision... go ahead and read Romans 3 where that's addressed.
Are you so foolish? Having begun by the Spirit, are you now being perfected by the flesh?
Galatians 3:3 NASB https://bible.com/bible/100/gal.3.3.NASB