Is accent the only way to judge a person's fluency in any language?

for instance, obviously the Indians, or Latin people have accents much different from the Americans/British. But if their spoken English is correct grammatically and they are really confident in conversing in the language, is it fair to term them people who don't know English just because their accents are not American or British?