With the deep integration of information technology and education, augmented reality (AR) has become a trend in science teaching, combining virtual reality and real-time interaction. This study designs and develops a science teaching system based on AR technology, integrated with educational game mechanisms, to overcome the problems of difficulty in understanding abstract content and low student participation in traditional science teaching. The system is guided by the theories of experiential learning and constructivism, and it integrates algorithms such as feature point tracking, image correction, and regression training. These algorithms improve the accuracy and stability of virtual models in real environments. Finally, a 12-month comparative experiment is conducted in a science and engineering major at a certain university. Experimental results showed that the average score for the science teaching method using AR technology improved significantly, by 20.11 points, compared with the traditional teaching method. The grade of Freshmen in the teaching system increased by 22.36 points, and the average score of seniors in the teaching system increased by 29.97 points. In the science teaching system, the indicators of educational games, playability, and applicability increased to 84, 87, and 86, respectively. Users' satisfaction with the system's usability, content accuracy, responsiveness, and clarity of indicators could reach 97%, 96%, 94%, and 98%, respectively. In summary, the science teaching system using an AR technology educational game has a significant improvement in students' performance and a strong practical effect in the science classroom.