The preceding verses in Matthew provide a little bit of context, but the parallel passage in Mark is even more detailed.
The Pharisees questioned Jesus because His disciples were not following the oral tradition of the elders, which proscribed various cleansing/washing rituals (above and beyond the commands in the Torah). Matthew, writing to Jews, did not need to provide as much cultural context; Mark, writing to Gentiles, gives greater background.
5 Then the Pharisees and scribes asked him, Why walk not thy disciples
according to the tradition of the elders, but eat bread with aunwashen
6 He answered and said unto them, Well hath Esaias prophesied of you
hypocrites, as it is written, This people honoureth me with their
lips, but their heart is far from me.
7 Howbeit in vain do they worship me, teaching for doctrines the
commandments of men.
8 For laying aside the commandment of God, ye hold the tradition of
men, as the washing of pots and cups: and many other such like things
9 And he said unto them, Full well ye reject the commandment of God,
that ye may keep your own tradition. (Mark 7:5-9, see also relevant context in vss. 1-4)
Jesus has just given a scathing rebuke of their traditions, suggesting that they are prioritizing the rituals of men above the commands of God. Among a group of people who prided themselves on their special status, telling them that the traditions (which they thought made them better than others) were in vain, was most displeasing to the local religious leaders.
This is the context of Jesus' summary pronouncement cited in the OP--people are not defiled because they fail to keep the impotent traditions of the elders, but they are defiled by the wicked things they say and do (some of which Jesus specifically criticizes in this pericope).