In order to improve the effect of sports prediction, this article applies deep learning to the automatic collection and processing of video surveillance data. The established light field multi-viewpoint real-time 3D surface reconstruction system mainly includes multi-camera joint calibration, multi-view image dense matching and depth generation. In order to improve the efficiency and accuracy of multi-view image matching, a four-direction semi-global matching cost accumulation method is proposed. After the dense matching of the multi-view images is completed, the global optimization method is used to optimize the initial depth map, remove noise and smooth the homogeneous area, and improve the quality of the depth map. The experimental research results show that the deep learning method proposed in this article has a very good application effect in the automatic collection and processing of video surveillance data in sports prediction.