Home/ Questions/Q 21626
Next
In Process

Technomantic Latest Questions

David
  • 4
  • 4
Begginer
Asked: 2025-05-29T21:36:40+00:00 In:

I trained my model, but it's performing too well on validation — could this be data leakage? How do I check for that?

  • 4
  • 4

I’m seeing 98–99% accuracy on my validation set, but when I test on truly unseen data, the performance drops significantly. Suspecting some leakage but not sure where it’s happening

ml

You must login to add an answer.

Need An Account,

2 Answers

  1. joseph1
    Begginer

    I once trained a model that was performing way too well on the validation set — like, suspiciously good. At first, I was excited… but something felt off. Turned out, it was data leakage.
    Here’s what I did to figure it out:

    • I rechecked my data splits and found that some similar entries had ended up in both training and validation.
    • I reviewed my features — one of them was indirectly revealing the target.
    • I even tested a basic model, and it still performed too well, which confirmed my suspicion.

    Lesson learned: if your model feels like it’s “too perfect,” always check for leakage. It’ll save you a ton of headaches later. Adopt this, It may solve this problem.

  2. Hassaan Arif
    Enlightened

    Absolutely, I’ve been in that spot, getting 98 to 99 percent accuracy on validation and feeling confident, only to see the performance drop a lot on truly unseen data. That’s usually a sign of data leakage. What helped me was carefully checking my data splits to make sure training and validation sets didn’t overlap. I also reviewed my features to find anything that might accidentally reveal the target. Sometimes a feature acts like a shortcut without you realizing it. I looked for very high correlations between features and the label because if something is almost perfectly correlated, that’s suspicious.

    Finally, I tried a simple model. If it also performed too well, it was another clue leakage was happening. Fixing these things usually made validation accuracy drop, but then the results matched real-world performance better, which is what really matters.