I would say that it was Jesus who was angry. I base this on the actions of the Pharisees at vs6. "And the Pharisees went out and immediately began taking counsel with the Herodians against Him."
I would also submit from Matthew 23 how Jesus reads them the riot act, ( I would say that He was pretty angry here), and then He concludes with Matthew 23:37-39, "O Jerusalem, Jerusalem, who kills the prophets and stones those who are sent to her How often I wanted to gather your children together, the way a hen gathers here chicks under her wings, and you were unwilling." Vs38, Behold your house is being left to you desolate.
Vs39, "For I say to you, from now on you shall not see Me until you say, Blessed is He who comes in the name of the Lord." Of course this is my opinion.
The following is what Greek Scholar A.T.Robertson has to say about the Greek.
When he had looked round on them with anger (περιβλεψαμενος αυτους μετ' οργης). Mark has a good deal to say about the looks of Jesus with this word (Mark 3:5; Mark 3:34; Mark 5:37; Mark 9:8; Mark 10:23; Mark 11:11) as here. So Luke only once, Luke 6:10. The eyes of Jesus swept the room all round and each rabbinical hypocrite felt the cut of that condemnatory glance. This indignant anger was not inconsistent with the love and pity of Jesus. Murder was in their hearts and Jesus knew it. Anger against wrong as wrong is a sign of moral health (Gould).
Being grieved at the hardness of their hearts (συνλυπουμενος επ τη πωρωσε της καρδιας αυτων). Mark alone gives this point. The anger was tempered by grief (Swete). Jesus is the Man of Sorrows and this present participle brings out the continuous state of grief whereas the momentary angry look is expressed by the aorist participle above. Their own heart or attitude was in a state of moral ossification (πωρωσις) like hardened hands or feet. Πωρος was used of a kind of marble and then of the callus on fractured bones. "They were hardened by previous conceptions against this new truth" (Gould). See also on Matthew 12:9-14.