I think their view on it is complicated. These B'nai Israel Jews were Jews at a time before either of the temples fell. Before we entered Rabbinical Judaism. They didn't have Talmud or any of the other writing or rulings that we have today. They were isolated from all that. They were kind of like the Russian citizens out in Siberia who didn't know(and may not still) that the Czars were dead.
That and they do have a different skin color from the Polish(which is where the Labvitchers started).
I also think it's the ultra-religious trying to impose their view on what Judaism is, was and is supposed to be. With...nevermind. I'll stop there