The Septuagint is not giving a more literal translation of the Masoretic Hebrew text. The Masoretic text has וַיַּחְשְׁבֶהָ (which is qal, not niph'al), and the -ה suffix must be a direct object (Gesenius defines it under the third definition of חָשַב: "to impute something to someone"). "He counted it to him for righteousness" is a literal translation of the Hebrew.
The Septuagint could have had a different Hebrew text which used the passive verb וַיֵּחָשֵב instead of וַיַּחְשְׁבֶהָ (in which case it would be agreeing with the Peshitta and Vulgate; the passive verb is also used in this context in Psalms 106:31).
Alternatively, the Septuagint could be giving a loose translation: the Hebrew verb doesn't always take a subject, and if that were the case here, "he counted it" could mean "someone counted it" rather than "God counted it." A literal "someone counted it to him for righteousness" could be reasonably translated idiomatically as "it was counted to him for righteousness."
Another potential example of the latter possibility: In Isaiah 8:4, the qal verb יִשָּׁא + direct object is translated in the Septuagint by a middle voice (λήψεται) and the direct object is replaced by the same word as a subject (in accusative case, as normal for nouns with infinitives). In Isaiah, a different text would be less likely (it would apparently require the word אֶת to be deleted and יִשָּׂא to be changed to יִנָּשֵׂא). If the Septuagint used the same method here (i.e. translating an unnamed subject and active verb by a passive verb), it wouldn't require positing a different original text.
As for whether the Septuagint's reading ("it was counted to him") is more accurate than the other possible reading ("he [God] counted it to him"), the Septuagint's reading is certainly possible (and perhaps even supported by the similar wording in Psalms 106:31 in which both Hebrew and Greek use the passive/niph'al form). However, Psalms 32:2 also uses the same wording as the verse in Genesis, differing only in speaking of a sin rather than righteousness: "happy is the person to whom God doesn't count a sin." In this verse the subject is undoubtedly God, and therefore it could just as easily be God who counted Abraham's belief as righteousness in the Genesis verse. In any case, it seems that this is only a grammatical point: whether or not it was counted or he counted it, it was either God who counted or God by whom it was counted.