Dr. Richard Sutton, a pioneer in the field of AI and an authority on reinforcement learning, has strongly criticized the current trend of overrelying on synthetic data (AI-generated data) for training AI models, calling it a "major mistake." He points out that to understand the infinitely complex real world, real-world data is essential rather than synthetic alternatives.
This statement was not released in conjunction with a specific product or new feature, but rather as a warning from a prominent researcher regarding prevailing AI training methodologies. Sutton discusses how current AI development trends may be underestimating "real-world complexity" in the pursuit of computational efficiency.
The core of Sutton's argument lies in the "complexity" and "unpredictability" inherent in real-world data. AI-generated data is merely a reconstruction of existing knowledge and patterns, presenting inherent limitations when learning unknown events or complex causal relationships. His perspective is that for models to represent the "infinite complexity" of the real world, an approach that goes beyond the limits of synthetic training and learns directly from raw, real-world data is necessary.
Sutton's proposal challenges the current trajectory of data curation and learning algorithms in foundational model development. As the adoption of generative AI accelerates the use of synthetic data, the challenge of how to efficiently and accurately incorporate real-world data is expected to become a critical focus in next-generation AI research.