In the process of college students' gymnastics learning, the phenomena of weakened motivation and emotional instability have seriously affected the learning effect of current gymnastics motor skill learning. The change of learning motivation is an internal factor for continuous learning of sports skills, and the generation and weakening of individual frustration can lead to the change of motivation, thus prompting college students to undergo a reversal of motivation, so that they can maintain the motivation for continuous training and learning. Currently, emotion-driven training is an effective method. In this paper, we utilize emotion recognition and reinforcement learning to drive the training of college students in gymnastics programs, with a view to reducing college students' frustration and emotional fluctuations in the training process. Specifically, we propose an emotion semantic recognition algorithm based on reinforcement learning, and the reinforcement learning model is able to recognize the complex emotional features in college students' speech and extract the emotion semantics. Among them, the multimodal semantic extraction model can fully extract the multimodal features among college students' speech, the emotion attention enhancement mechanism can guide the optimization of the semantic feature extraction algorithm of the multimodal semantic extraction model by effectively tracking and feeding back the results of the emotion analysis, and the general extended dictionary can pay more attention to the content that affects the emotion category. With the collaborative training method, the reinforcement learning-based emotion semantic recognition model effectively comprehends the emotion information of college athletes, which provides a positive reference for the emotion-driven training method.