Abstract In this study, an optimization methodology is proposed to systematically define headcutter geometry and machine tool settings to introduce optimal tooth modifications in face-hobbed hypoid gears. The goal of the optimization is to simultaneously minimize tooth contact pressures and angular displacement error of the driven gear, while concurrently confining the loaded contact pattern within the tooth boundaries. The proposed optimization procedure relies heavily on a loaded tooth contact analysis for the prediction of tooth contact pressure distribution and transmission errors. The objective function and the constraints are not available analytically, but they are computable, i.e., they exist numerically through the loaded tooth contact analysis. The core algorithm of the proposed nonlinear programming procedure is based on a direct search method. Effectiveness of this optimization was demonstrated by using a face-hobbed hypoid gear example. Considerable reductions in the maximum tooth contact pressure and in the transmission errors were obtained.