Consider the statement "let God be true and every man a liar" (Romans 3:4). This is not saying God should be true, but a fact. In the same sense, let him be accursed is not referring to the church cursing someone, but a statement of fact. Consider also Romans 9:3 "I could wish myself accursed." Accursed is to be under judgment of God and would not necessarily manifest a difference in this life. In Galatians 3:10, the latter half of the verse explains the curse he refers to:
Galatians 3:10 For as many as are of the works of the law are under the curse: for it is written, Cursed is every one that continueth not in all things which are written in the book of the law to do them.
It is a quote form the book of Jeremiah.
Jeremiah 11:3-5 And say thou unto them, Thus saith the Lord God of Israel; Cursed be the man that obeyeth not the words of this covenant,
4 Which I commanded your fathers in the day that I brought them forth out of the land of Egypt, from the iron furnace, saying, Obey my voice, and do them, according to all which I command you: so shall ye be my people, and I will be your God:
5 That I may perform the oath which I have sworn unto your fathers, to give them a land flowing with milk and honey, as it is this day. Then answered I, and said, So be it, O Lord.
In I cor. 16:22 the expression is used in conjunction with Maranatha indicating the judgement of Christ at His return.
1 Corinthians 16:22 If any man love not the Lord Jesus Christ, let him be Anathema Maranatha.