Oh, you mean with Iray. (My previous comments were about the OpenGL renderer, just to avoid any confusion.)
You can type in a bigger number to get more than 5 steps of tesselation.
I don't believe there is an option to invert normals. I dont work with maya but Fbx files should work correctly if exported correctly in my experience, though you may be using different settings.
It could be something else: Make sure you have a normal map plugged in to a normal map output (even if its just an empty normal node). An empty output is always considered as a black image, which of course looks very wrong if this happens to the normal output.
Cheers,
Esger