Skip to content
Rabat, Morocco
Posts

The 99% That Remains in the Dataset

February 4, 2025
The 99% That Remains in the Dataset
When we talk about AI and deep learning, we usually focus on the 1% that makes it. You know, those polished models that actually work in the real world. But here's the thing: that successful model? It's built on top of countless iterations, discarded datasets, and experiments that didn't work out. What about the other 99%? All that unused data, failed training runs, and ideas that got rejected? They might seem like failures, but honestly, they're crucial. They're what shape the final breakthrough. It's easy to view discarded datasets and failed models as waste, but in reality, they're stepping stones. Every dataset that didn't work, every approach that overfitted, and every rejected architecture taught us something valuable. These iterations define the learning process, helping us understand what works, what doesn't, and why. In AI research, the first few models often seem promising. They fit the training data well, produce expected results, and appear to generalize. But as we test at scale, edge cases emerge, biases surface, and unexpected limitations arise. The process of refining AI models is not just about improving accuracy, but about ensuring robustness, fairness, and generalization. The 99% of discarded experiments are not failures but part of the iterative learning process. Every false start, unstable model, or impractical training strategy contributes to a better, more resilient system. For every model that didn’t make it, there’s a valuable insight gained. A dataset that seemed irrelevant today might become critical with new architectures or emerging AI techniques. A neural network that overfit in one scenario might be useful in another domain with slightly adjusted parameters. These shelved ideas form an archive of knowledge, helping AI researchers refine their understanding and build upon past work without repeating mistakes.
Alt text
Many AI breakthroughs from DeepSeek’s generative models to reinforcement learning advances, emerged from revisiting previously discarded approaches. A method that failed due to computational limits might now be viable with improved hardware. A rejected model architecture might outperform others in a new context. AI is an iterative process, where the best solutions often emerge from re-examining past work. No experiment is truly wasted, it simply hasn’t found its moment yet. The final, successful AI model is just the tip of the iceberg, the visible 1%. The other 99% of discarded data, failed tests, and abandoned approaches are just as important. They represent the trial and error, the persistence, and the deep exploration that fuel true innovation in AI. So next time an experiment fails or a dataset seems unusable, don't discard it completely. The answer you're searching for might already exist in the data, just waiting for the right model to unlock its potential.

Enjoyed this blog?

Seriously, thank you for spending your time here. If you made it to the end, you’re the kind of person who cares about craft, not just quick fixes.If you’d like to support my work, pick one (no pressure). It genuinely helps me keep writing and improving these blogs.
Subscribe to get a heads-up when I publish the next one. I keep it low-volume, and I only send what I’d personally want to read.

Subscribe to Josias's Newsletter

I occasionally write about software engineering, technology, and share thoughts on innovation and engineering.
One last thing: if you know someone who’d benefit from this, sharing it with them is one of the best compliments you can give. I appreciate you.