I agree with the thought of buying the biggest mill you can afford the first time. When I purchased my mill. I went through the same thing looking at the smaller Grizzly and LMS products, used Bridgeports, etc.
I ended up buying a mid size bench mill the PM932, which I have been happy with...