The answer to your question is that he will never cast you out, but you may be in a situation where you think you have come to him but you have not.
Matthew 8.12, from the context, describes jews that do not believe in him and so will be cast out, even if they are the sons of the kingdom (descended from Jacob). That is the plain meaning, and the spiritual meaning would be those who view themselves as members of his body after the flesh (after outward signs, such as attending Church or performing works they think are good works), but have not come to him.
Here another verse would be Matt 7.20-23 (KJV 1900):
21 Not every one that saith unto me, Lord, Lord, shall enter into the
kingdom of heaven; but he that doeth the will of my Father which is in
heaven. 22 Many will say to me in that day, Lord, Lord, have we not
prophesied in thy name? and in thy name have cast out devils? and in
thy name done many wonderful works? 23 And then will I profess unto
them, I never knew you: depart from me, ye that work iniquity.
So from these verses we see:
Jesus will push away those who never knew him but he will never push away those who came to him
Some people will fool themselves into thinking they knew him, but Christ will declare that he never knew them. These he will cast out.
Those who never knew Christ, but sincerely thought they did, believe they did many wonderful works, including casting out demons, prophesying, etc. Therefore outwardly they will look as if they are true believers, but inwardly they are not true believers, because they never came to him.
In Christ's eyes, what matters is "doing the will of my Father", not performing wonderful works such as prophesying, performing miracles, etc.
This creates a problem, because, for example, if you give some money to the poor, you may or may not be doing the will of the Father. You cannot tell from the outward sign of the act, as what matters is in the heart. Jesus described this case:
Matthew 6:2–4 (KJV 1900)
2 Therefore when thou doest thine alms, do not sound a trumpet before
thee, as the hypocrites do in the synagogues and in the streets, that
they may have glory of men. Verily I say unto you, They have their
reward. 3 But when thou doest alms, let not thy left hand know what
thy right hand doeth: 4 That thine alms may be in secret: and thy
Father which seeth in secret himself shall reward thee openly.
And again in Samuel 16.7, we see that God looks at the heart, at what is secret:
But the LORD said unto Samuel, Look not on his countenance, or on the
height of his stature; because I have refused him: for the LORD seeth
not as man seeth; for man looketh on the outward appearance, but the
LORD looketh on the heart.
So what matters is the heart, not what is visible, whether good deeds, or miracles, if it is something that you can see, then there is always a possibility that it is not according to God's will and is counterfeit, because God demands truth in the inward parts when we come to Him
John 4:24 (KJV 1900)
24 God is a Spirit: and they that worship him must worship him in
spirit and in truth.
This is why God needs to draw us to him, as we can force ourselves to declare "I believe in Jesus" and we can even sell all we have and give it to the poor, but we do not have control over the truth in our own hearts, and without actions imbued with integrity, there is no profit to us. Therefore he says in John 6.44:
No man can come to me, except the Father which hath sent me draw him: and I will raise him up at the last day.