The short answer is NO. If it's buffing swarf/grit/compound it's next to impossible clean up without grinding/refinishing the handle material past the depth of the contamination. For future reference, especially if the handle material is light in color, or has an open grain, carefully tape off the handle material with painter's tape BEFORE you even go near the buffer, or try to "polish" the bolsters or guard.