We propose a new algorithm for estimating the absolute and relative pose of a camera system composed of general central projection cameras such as perspective and omnidirectional cameras. First, we derive a minimal solver for the minimal case of 3 line pairs per camera, which is used within a RANSAC algorithm for outlier filtering. Second, we also formulate a direct least squares solver which finds an optimal solution in case of noisy (but inlier) 2D-3D line pairs. Both solver relies on Grobner basis, hence they provide an accurate solution within a few milliseconds in Matlab. The algorithm has been validated on a large synthetic dataset as well as real data. Experimental results confirm the stable and real-time performance under realistic outlier ratio and noise on the line parameters. Comparative tests show that our method compares favorably to the latest state of the art algorithms.