At first glance, «Καὶ ἐγένετο ἐν τῷ εἶναι αὐτὸν» appears to be a Semiticism (which requires a paraphrase or reconstruction into English rather than a literal translation).
A. T. Robertson wrote,1
The Semitic influence is undoubted in the O. T. and seems clear in Luke, due probably to his reading the LXX or to his Aramaic sources.
The infinitive preceded by ἐν τῷ construction did occur in earlier Greek, but as A. T. Robertson wrote,2
Ἐν τῷ appears in the tragedies. It is found 6 times in Thucydides, 16 in Xenophon, 26 in Plato. But Blass observes that the classical writers did not use ἐν τῷ in the temporal sense of 'while' or 'during.' Moulton sought to minimize the fact that in the O. T. ἐν τῷ occurs 455 times (45 in the Apocrypha) and that it exactly translates the Hebrew בְּ and held that it did not in principle go beyond what we find in Attic writers. But he took that back in the second edition under the suggestion of Dr. E. A. Abbott that we must find Attic parallels for 'during.' So he now calls this "possible but unidiomatic Greek."
A. T. Robertson further remarked,3
In the N. T. we have ἐν τῷ and the inf. 55 times and 3/4 in Luke. In the Greek Bible as a whole it is nearly as frequent as all the other prepositions with the inf.
Regarding the syntax, George Benedikt Winer wrote,4
Hence the construction is correctly conceived in Greek, though the frequent use of ἐγένετο with the infinitive in the place of the historic tense of the main verb is in the first instance due to an imitation of the Hebrew ויהִי.
Example from the Septuagint
A search of the LXX for «καὶ ἐγένετο ἐν τῷ εἶναι» yields «καὶ ἐγένετο ἐν τῷ εἶναι αὐτοὺς ἐν τῷ πεδίῳ» in Gen. 4:8.5 The underlying Hebrew text states וַֽיְהִי בִּהְיֹותָם בַּשָּׂדֶה.
וַֽיְהִי is a very common verb in the Masoretic text, quite often translated by the Greek «καὶ ἐγένετο». It may be translated as, "and it came to pass," "and it occurred," "and it happened," and so forth.
The next word בִּהְיֹותָם consists of the preposition ב prefixed to the infinitive construct הֱיוֹת ("being"), which is suffixed with the pronominal suffix םָם (-am), meaning "their." Rather than translating this literally as "in their being," it is understood as "when they were." The past tense accords with the previous verb וַֽיְהִי, "and it happened." Altogether, the phrase would be understood as "and it happened, when they were in the field."
The Articular Phrase in Luke 11:1
We return to the phrase «Καὶ ἐγένετο ἐν τῷ εἶναι αὐτὸν ἐν τόπῳ τινὶ προσευχόμενον» in Luke 11:1.
The accusative «αὐτὸν», which would usually be translated as the direct object of a verb, is translated into English as the subject (as though it were αὐτός, i.e., in the nominative case).
The preposition «ἐν» is translated as "when" or "while."
The infinitive «τῷ εἶναι» is translated into English as a past tense verb, in this case, "was" ("was" agrees in number with the singular «αὐτὸν» which became the subject).
In addition, the participle «προσευχόμενον» ("praying") is joined to the verb "was."6 It is not merely that he "was in a certain place," but that he "was praying in a certain place."
1 p. 1073
2 p. 1072
4 p. 407
5 also cp. 2 Sam. 3:6: "When there was the war between the house of Sha'ul and the house of David..." The LXX has «καὶ ἐγένετο ἐν τῷ εἶναι τὸν πόλεμον ἀνὰ μέσον τοῦ οἴκου Σαουλ καὶ ἀνὰ μέσον τοῦ οἴκου Δαυιδ».
6 If we recall the aforementioned example in Gen. 4:8, the only verb was "were" because no participle occurred in the clause. Gen. 4:8 tells us that Cain and Abel were in the field; Luke 11:1 tells us that Jesus was praying while in a certain place.
Blass, Friedrich Wilhelm. Grammar of New Testament Greek. Trans. Thackeray, Henry St. John. 2nd ed. London: Macmillan, 1905. (p. 237, §71. 7)
Buttman, Alexander. A Grammar of the New Testament Greek. Andover: Draper, 1873. (p. 277)
Moulton, James Hope. A Grammar of New Testament Greek. Vol. 1. 2nd ed. Edinburgh: Clark, 1906. (p. 215-216)
Robertson, Archibald Thomas. A Grammar of the Greek New Testament in the Light of Historical Research. Vol. 1. New York: Hodder, 1914. (p. 1072-1073)
Winer, George Benedikt. A Treatise on the Grammar of New Testament Greek. 3rd ed. Trans. Moulton, William Fiddian. Edinburgh: Clark, 1882. (p. 413, §44. 6)