You might see if you can put some epoxy in there and sand it off.  The problem is that the pin hole is too big.  Even if you go 1/64" over the size of the pin you are going to get a gap.  What you need to do is get some metric or wire guage (number) size bits that are just a few thousandths of an inch over the size of the pin stock.  A micrometer is very handy for this and I found one with a digital read out in metric, decimal, and fraction measurements.  For instance, I use a 6.4mm bit with 1/4" pin stock.  I use a 3.2mm for 1/8" stock but I could also use a #30 which would be just a little larger.  The reason it's good to mic you pin stock, and possibly your bits, is that they can vary slightly from the nominal size.  I have some slightly undersized 1/4" stock that will fit snugly into the pin hole with a 1/4" bit.
Doug