The word נִדְגָּל is an adjective which means to be bannered, it comes from the same root as the word דֶּגֶל which means flag. In Hebrew almost every adjective can also be used as a noun. Thus, as this word is also an adjective, it can be conjugated to any grammatical case. The word כַּנִּדְגָּלוֹת just means like נִדְגָּלוֹת. Thus, the word means something roughly like "like bannered ones."
We have to remember that Song of Songs is written in poetry, so it is often missing some words. The word דגל (flag/banner) is often used in the context of war in the Bible, such as how the Israelites would go to war or camp such as in Numbers 1:52. The use of the female plural form is used to indicate a deeper meaning. One possible explanation is that it refers to some ancient practice of woman carrying banners. This is possible, as it would seem to fit someone like Deborah; however, it does not have much evidence to support it. Another, more plausible, explanation which is of Ibn Ezra, is that this word is conjugated in female plural to refer to מחנות which means camps. Thus this phrase would mean "the bannered camps." Another word which also could be used similarly is צבאות which means "armies" or "hosts." Both these words are grammatically female and plural. Almost every commentator agrees that in context, this word has something to do with armies and camps, which is also hinted at by the previous word אימה which means "awesome". Thus, the use of the feminine word is most likely due to these other words which it is based on.