While the polygon fill create a mask base don the geometry, it will still produce a mask base don pixels and thus will be limited by the texel ratio / texture resolution.
The mask produced also don't generate anti-aliasing which can creates the jagged line you see. You can try to refine it by using a blur and a level effect together.
