Scientists discover the moment AI truly understands language

A study in JSTAT describes the sharp shift in text comprehension strategies during neural network training.

Summary:Neural networks first treat sentences like puzzles solved by word order, but once they read enough, a tipping point sends them diving into word meaning instead—an abrupt “phase transition” reminiscent of water flashing into steam. By revealing this hidden switch, researchers open a window into how transformer models such as ChatGPT grow smarter and hint at new ways to make them leaner, safer, and more predictable.

Researchers found that transformer networks abruptly switch from tracking word positions to focusing on meaning once training data passes a critical threshold, mirroring a physical phase change and shedding light on AI’s inner workings. Credit: Shutterstock

The language capabilities of today's artificial intelligence systems are astonishing. We can now engage in natural conversations with systems like ChatGPT, Gemini, and many others, with a fluency nearly comparable to that of a human being. Yet we still know very little about the internal processes in these networks that lead to such remarkable results.

A new study published in the Journal of Statistical Mechanics: Theory and Experiment (JSTAT) reveals a piece of this mystery. It shows that when small amounts of data are used for training, neural networks initially rely on the position of words in a sentence. However, as the system is exposed to enough data, it transitions to a new strategy based on the meaning of the words. The study finds that this transition occurs abruptly, once a critical data threshold is crossed -- much like a phase transition in physical systems. The findings offer valuable insights for understanding the workings of these models.