In this case the syntax is clear:

ἀλλὰ τῷ αὐτῶν παραπτώματι ἡ σωτηρία τοῖς ἔθνεσιν εἰς τὸ παραζηλῶσαι αὐτούς.

 

but by their (masc) trespass the salvation [came to] the gentiles (neut) in order to provoke them (masc) to jealousy

As you can see, the masculine referent is the Jews ("Israel", from v. 7); thus, αὐτούς refers to the Jews. The distinct neuter masculine accusative pronoun αὐτά would be used if the antecedent were the neuter noun ἔθνη (gentiles).

In this case the syntax is clear:

ἀλλὰ τῷ αὐτῶν παραπτώματι ἡ σωτηρία τοῖς ἔθνεσιν εἰς τὸ παραζηλῶσαι αὐτούς.

but by their (masc) trespass the salvation [came to] the gentiles (neut) in order to provoke them (masc) to jealousy

As you can see, the masculine referent is the Jews ("Israel", from v. 7); thus, αὐτούς refers to the Jews. The distinct neuter masculine accusative pronoun αὐτά would be used if the antecedent were the neuter noun ἔθνη (gentiles).