In this case the syntax is clear:

> ἀλλὰ τῷ **αὐτῶν** παραπτώματι ἡ σωτηρία τοῖς **ἔθνεσιν** εἰς τὸ παραζηλῶσαι **αὐτούς**.  

> but by **their** (*masc*) trespass the salvation [came to] the **gentiles** (*neut*) in order to provoke **them** (*masc*) to jealousy

As you can see, the masculine referent is the Jews ("Israel", from v. 7); thus, αὐτούς refers to the Jews. The distinct neuter masculine accusative pronoun αὐτά would be used if the antecedent were the neuter noun ἔθνη (gentiles).