There is indeed some situations were materials are kept in Inspire.
This is on our plans to optimize that part.
If you need to remove some variations generated with Inspire. You can use the "x" on each thumbnail or use the "Clear Variations buttons".
Best,