Oh my Cow you wouldn't believe how many people can't wrap their head around this. Yes! This is very true. This is a real issue with a drilled hole. Drilled holes are not for precision applications.
In school you're taught to:
1 Spot drill (this gets the drill started accurately - the lip angle should be more obtuse than your drill to force it to start at the web)
2 Drill (you're just removing material)
3 Bore (this aligns the hole to the spindle and makes it round, two passes assures the tool didn't flex)
4 Ream (this makes a fairly round fairly smooth hole with diameter control to a few .0001)
In practice you:
1 Drill
2 Circular interpolate
3 Ream if you're feeling randy. You'll get better roundness reaming, but better positional and finish milling - size control can be a wash.
If it is important, you can clamp it to a face plate on a lathe at low speed and install a jig grinding or tool post grinding attachment and grind that sucker to a couple .0001 (if you got good stuff). My folder blades get circular interpolation - which is fine for most folders. Hell, drilled holes work for a lot of folks (though I'll pass).
Tracy, shouldn't this be under shoptalk?