Why does overfitting matter?#
Understanding overfitting helps your technical team build more robust and reliable AI systems. It ensures the model has learned the general patterns in your data, not just the specific examples it was shown, so it can perform effectively and add value in a live business environment.
What causes overfitting?#
When an AI model is being built, it learns from a set of data known as "training data." The goal is for the model to identify the underlying patterns and relationships within that data.
Overfitting happens when the model becomes too complex and begins to memorise the training data, including its random fluctuations and irrelevant details—often called "noise." Because it has essentially memorised the specific training examples instead of learning the general principles, it becomes brittle. When presented with new data that doesn't perfectly match what it has memorised, its performance drops significantly.
How is overfitting different from underfitting?#
Overfitting is when a model is too complex and learns the training data's noise, making it perform poorly on new data. In contrast, underfitting is when a model is too simple to capture the underlying patterns in the data at all. A properly "fitted" model strikes the right balance, learning the general rules from the data without being overly influenced by its noise, allowing it to make useful predictions in a variety of situations.