Your shrewd observation completely surprised me; I had never thought about the possibility that those men had been standing all day in a place where labourers were hired, and that the landowner would have visited the spot four times already that day. Had the men been there all day long, it is reasonable to wonder why those men had not seen the hiring going on and sought the landowner's employ.
However, some points may be incorrectly assumed here. Was it exactly the same spot as early in the morning, as at the other three times, where the landowner returned to? Perhaps it was a large town where there were five locations where men gathered to seek daily employment. Or there may have been two locations, the landowner only going to the second one at the eleventh hour.
But the real point to bear in mind is that Jesus was not relating an actual event. He was giving a parable as to what the Kingdom of God was like. And this parable followed on the heels of Jesus telling his disciples that "Many who are first will be last, and the last first" (Mat. 19:30).
It is significant that the parable which followed ended with the labourers hired last receiving their wages first. And the labourers hired first, received their wages last.
Therefore, the hypothetical scenario does not require examination of such details as to why the men hired last, having been there all day, had not seen the landowner who had appeared four times previously searching for workers. There was no actual landowner, there were not various lots of workers engaged prior to the last batch at the eleventh hour, there was no handing out of wages at the end of the day.
But spiritually, the parable has profound implications for those who think that the longer and the harder they work for the Master (Christ) in his 'field' (the world - Mat. 13:38), the more they will deserve to 'earn' as a 'wage'.