In Substance Painter the way we compute the normal map from the height map channel is to preserve the details, which can produce some aliasing sometime on very hard edges. Other than blurring/smoothing the alpha that you used for your mask there is not much more you can do until we find a better algorithm.