It is normal. A perfect UI Framework for your needs does not exists. As long as the overrides aren’t too large, it really worth it. It saves not only CSS, also Components features and some browsers compatibilities issues.
Try to keep it simple, maybe the changes are not always needed. They look pretty good by default. And match a whole design.
The other way is (having a professional designer design) from scratch, which is a lot of extra work. (If you don’t have such profesional, don’t even try it). But if it’s mandatory a fixed desig, very different from Material UI, go for it.
Finding another UI Framework, probably less wide/mature than Material UI for React, will requires changes too.
Of course try to reuse components.
Check the Theme for custom colors, ...
I hope it helps.
This is my opinion. I have used Material UI for a couple of projects, happy doing it.