Contemplating sin is more serious than the sin itself (Yoma 29a).
Although actions generally have much greater impact than thoughts, thoughts may have a more serious effect in several areas.
The distance that our hands can reach is quite limited. The ears can hear from a much greater distance, and the reach of the eye is much farther yet. Thought, however, is virtually limitless in its reach. We can think of objects millions of light years away, and so we have a much greater selection of improper thoughts than of improper actions.
Thought also lacks the restraints that can deter actions. One may refrain from an improper act for fear of punishment or because of social disapproval, but the privacy of thought places it beyond these restraints.
Furthermore, thoughts create attitudes and mindsets. An improper action creates a certain amount of damage, but an improper mindset can create a multitude of improper actions. Finally, an improper mindset can numb our conscience and render us less sensitive to the effects of our actions. We therefore do not feel the guilt that would otherwise come from doing an improper act.
We may not be able to avoid the occurrence of improper impulses, but we should promptly reject them and not permit them to dwell in our mind.