Flying rust, asphalt dots, sap from trees and other aggressive dirt will stick to the surface of your paint and if you do not remove them with clay, it will damage your paint over time and it will not be possible to remove them other than by grinding and subsequent painting.