In this case the syntax is clear:
ἀλλὰ τῷ αὐτῶν παραπτώματι ἡ σωτηρία τοῖς ἔθνεσιν εἰς τὸ παραζηλῶσαι αὐτούς.
but by their (masc) trespass the salvation [came to] the gentiles (neut) in order to provoke them (masc) to jealousy
As you can see, the masculine referent is the Jews ("Israel", from v. 7); thus, αὐτούς refers to the Jews. The distinct neuter masculine accusative pronoun αὐτά would be used if the antecedent were the neuter noun ἔθνη (gentiles).