I believe it's important to precisely distinguish between good and evil. In this context, I would like to propose a thought experiment in an attempt to answer a question. So here goes...

I have a strong sexual preference for Emma Stone. In my mind she is fucking hot, but I will never act on that sexual preference because she has not given me consent.

Children cannot give consent - that's my opinion. So a person who finds children sexually preferable should never act on that preference. Never.

Preferences are not objective. They depend on who you ask. I like Emma Stone, and some rare few people like children.

Let's say Emma Stone will never give me consent to sleep with her. I know it's a stretch, but lets say she never ever does. If I raped Emma Stone this would be evil because she has not consented. I am using her body (her property) without her consent.

By logical extension: Given that a child cannot consent, an adult having sex with a child is always rape.

Question: Given that Emma Stone will never give me consent to have sex with her, am I evil for having a strong sexual preference for her?

If the answer to this is NO how would you answer this question?

Question: Given that a child cannot give consent, is a person evil for having a strong sexual preference for a child?

If your answer to this question is also no, then a person who regards the sexual preferences of pedophiles to be evil may have some other argument to make that assertion. I am aware of no such argument.

If having certain preferences does not make a person evil, then I believe it's important to distinguish in your mind between the people who have acted immorally and those who have not.