From the IVP Bible Background Commentary: Old Testament:
8:17. "branch to the nose." There is an Akkadian expression (laban appi) that refers to a gesture of humility used to come contritely before deity with a petition. When this act is portrayed in art, the worshiper has his hand positioned in front of his nose and mouth, and is sometimes shown with a short cylindrical object in his hand. From the Sumerian tale called Gilgamesh in the Land of the Living there is some evidence that what is held is a small branch cut off a living tree. This would suggest that in Ezekiel the people are putting on a show of humility. It must be admitted, however, that these connections are very hazy and the significance may lie somewhere else entirely.
Matthews, V. H., Chavalas, M. W., & Walton, J. H. 2000. The IVP Bible background commentary : Old Testament (electronic ed.) . InterVarsity Press: Downers Grove, IL
As the verses before this talk about how the Hebrews are mixing neighboring religions with true worship (mounrning for Tammuz, idols in the temple courtyard, sun worship), I think the problem is two fold:
- This is an act of syncretism--the mixing of elements from several religions.
- They don't mean it.
The second is the heart of the problem. Crocodile tears for sins would be one way to understand the action. From the actions that follow the acts of contrition, it can be seen that they don't mean it. They aren't truly repenting. It would be like a catholic saying penance prayers and then turning around doing the same thing again.
The actions mean nothing unless the heart is in them.