The real world feasibility of a self-contained, purely vision based sense and avoid system, required for small unmanned aerial vehicles (UAV) is investigated in the present paper. No information is exchanged between aircrafts, only passive 2-D vision information is available to estimate the encountering traffic. The system is composed of three distinct components: intruder state estimation using Unscented Kalman filter (UKF); collision risk estimation based on probabilistic algorithms; and trajectory re-generation using motion primitives to minimize the expected probability of collision. Since the relative system dynamics are weakly observable special emphasis is made on persistent excitation. The system is tested on a high fidelity Hardware-in-the-Loop (HIL) simulation platform, where flight control algorithms, scene rendering, image processing and estimation algorithms are implemented individually over a network of computers. A high number of representative encounter scenarios are presented to provide performance measures including detection time and miss distance of distinctive approaches to assess the applicability of the proposed method.