"With the Harry Potter movies having a primarily white cast, it'd be nice to see a Black man step into such a prominent role."

Based on this alone I would argue there is a racist component to the reasoning. If it would be taken as unacceptable and racist to say that of a mostly black cast then I don't see why saying that of a primarily white cast is acceptable. It simply isn't.

Diversity for the sake of diversity just doesn't resonate well.

In expanded material there is a wizarding school in Africa called Uagadou. Seeing as the population statistics show that Africa has an over 80% black population I imagine the wizard school there would be overwhelmingly attended by black people. Hogwarts, in England, would be overwhelmingly attended by white people for the reason that the country's population is well over 80% white.

Put simply, hypothetically:

"With the Uaganda Origins movies having a primarily black cast, it'd be nice to see a White man step into such a prominent role."

I can't say I could see that being an acceptable thing to say. It's amazing how we've allowed it to be totally acceptable to treat the "white people" like a group of them is a problem, and the solution being to diversify them.