There's no real way to identify a proper noun (at least in Hebrew). In fact, even if Rahab is interpreted as a sea serpent, there's no real way to determine whether it's "Sea Serpent" or "a sea serpent" (the definite article never appears before names in Hebrew).
Furthermore, it's sometimes impossible to be totally certain in translating a word. The simple עֶרֶב ("evening," Habakkuk 1:8), for instance, is interpreted by the LXX as עֲרָב ("Arabia"). Even if a word isn't a proper noun, there are often many ways to interpret it.
That said, some ways to identify a proper noun are:
The Bible itself
The best way to know is from other places in the Bible. Obviously, a prose description of Rahab (like we have for many other figures in poetry, like Moses or David) would be the best starting point, but it doesn't exist (discounting the story of the Exodus, which doesn't mention Rahab). Traditions about Rahab have to be pieced together from the different poetic verses that speak about Rahab.
The notes you quoted give one good example of using prooftexts in this manner:
[89:11] Rahab: a mythological sea monster whose name is used in the Bible mainly as a personification of primeval chaos, cf. Jb 9:13; 26:12; Ps 74:13–14; Is 51:9.
Note that Isaiah 51:9, for instance, uses רהב in parallel to תנין (serpent), which makes it clear that the two are connected.
Psalms 40:5, on the other hand, clearly is not speaking of any sort of serpent when it uses רהבים to mean "arrogant people" (in parallel to שטי כזב).
Between the two possibilities of a sea serpent and arrogant people, the verse in question (Psalms 89:11) can be either, but the context of the sea and slaying foes seems to favor the first explanation.
The Talmud (Bava Batra 74b) explicitly describes Rahab as the "minister of the sea" (שרו של ים), and this is sometimes taken as an additional proof for the idea of Rahab being identified with the split sea. However, external sources are less reliable and can (and often do) use the same words for different concepts, so this should be used cautiously.
(Ugaritic texts are often invoked regarding Rahab, but since they, like the Exodus, don't actually mention Rahab, I am ignoring this source.)
Knowledge of mythology, geography, and history can all be put to use. They can be learned from contemporary sources (but of course also from reference works, and footnotes on Bibles).
As for Zaphon and Amanus (apparently these mountains), I am not quite sure what proofs can be brought for Amanus from the Bible itself. בעל צפון was also a place (Exodus 14:2), and could be referred to here, but I'm not aware of any sacred mountain called "ימין." The decision to translate it as Amanus is probably due to context (Tabor and Hermon appear many times as mountains). However, צפון can mean north and ימין can mean south, so I see no reason why not to translate "north and south."