Much depends on the specific handle material. Some of the exotic woods work best if you DO NOT seal them. Some of them you simply can't because of the natural oils/resins in the wood. Woods such as rosewood, cocobolo, African Blackwood, desert ironwood etc, only receive a good coat of wax after they are finished.
Porous woods such as Koa, Maple, etc (non-stabilized) receive several coats of Tru-Oil or Lin-Speed in my shop.
Stabilized woods required no sealer, and will generally look nasty if you apply one. Finish these materials down finely (I got to 600-800) buff lightly with pink no-scratch, and then wax.
Ivories and other exotics can be tricky....my recommendation, especially for ivories is to sand them to a very fine finish (I got to at least 1200) then lightly buff and wax. IVORIES, PEARL, AND LIKE MATERIALS ARE VERY SUSCEPTIBLE TO HEAT! If these material get warm while grinding or finishing, it creates all kinds of problems. Be careful!
Finally, anything that is a natural material is going to "move"....even stabilized materials. Some folks might argue, but if you pay close attention, even stabilized material with shrink/expand a LITTLE. That is natural. Natural handle materials that are allowed to "breathe", in other words, not slathered in finishes or sealers, will have less problems, and look better for longer over the life span of a knife.