Let me ask you a question: Have you ever started a project feeling confident everything was under control, only to find out halfway through that nothing was going as planned? That’s exactly what happened to me when I was leading a data science project.

In my journey with data science projects, one truth has stood out: without the right mindset, even the best frameworks can fall flat. This truth came into sharp focus during a 2022 project I led for MasterCard’s Open Banking division. The task seemed straightforward: build a new API to expand into international markets. We had a clear business goal, a brilliant team of data scientists and analysts, and all the data we thought we needed.

But let me tell you a little secret about data science projects—they rarely go as planned.

As the project unfolded, Market demands changed. Regulations were updated. New challenges emerged. The team felt overwhelmed as our initial tools and plans fell short of the evolving needs. That’s when I realized: we didn’t just need a new process—we needed an agility mindset.

An agility mindset shapes how individuals and teams approach uncertainty, tackle challenges, and stay aligned on goals. It’s not just a concept—it’s the cultural backbone of successful projects. The AIP-DM framework, which I developed in response to these challenges, taught me the power of this mindset—a perspective that emphasizes collaboration, adaptability, and continuous learning. This mindset isn’t about rigidly following best practices; it’s about embracing a way of thinking that empowers you to pivot when necessary, collaborate openly, and learn from every step of the process.

You might wonder, “Why not use one of the existing data science frameworks?” It’s a fair question. These frameworks provide structure and guidance, but they often lack the flexibility required for real-world data science projects. Most are built on the idea of a linear process—first, define objectives; then, prepare the data; build models; and finally, deploy the results.

But here’s the reality: data science doesn’t follow a straight path. It’s a dynamic, iterative process where new insights can emerge unexpectedly, data issues can derail plans, and priorities can shift without warning. Traditional frameworks, while valuable in theory, often feel too rigid for the constant back-and-forth of real-world projects. To succeed, you need more than structure—you need a mindset that embraces flexibility, experimentation, and the ability to adapt as you go.

The team raised valid concerns. They told me:

1. We need space for deep exploration—strict time limits don’t feel like the right approach.
2. It’s hard to show progress early in data work when insights take time.
3. Every dataset is unique. We need the freedom to adapt, not follow rigid steps.

At that moment, I realized something important: agility isn’t just about the process; it’s about embracing a mindset that supports exploration, adaptability, and continuous learning. To address these challenges, I developed the Agile Iteration Process for Data Mining (AIP-DM). Unlike traditional frameworks, AIP-DM is not just a set of steps—it’s built on agility principles that embrace the iterative, dynamic nature of data science. Here’s how the mindset is integrated into each phase:

1. Exploration: Open collaboration is critical here. Business leaders, data scientists, and engineers align on goals and challenge assumptions together. This sets the stage for a shared vision and avoids siloed thinking.
2. Modeling: Adaptability becomes essential. As new insights emerge or priorities shift, the team is prepared to rethink approaches, test new hypotheses, and refine their models.
3. Deployment: Iterative feedback is at the core. Continuous learning—asking “What worked? What didn’t? What should we try next?”—ensures the project evolves and improves over time.

But here’s the part I’m most proud of: We focused less on rigid rules and more on fostering the right culture. Inspired by the 80-20 rule, we spent 20% of our effort creating a clear process and using tools like Jira to stay organized, and 80% on building a culture of collaboration, integrity, and creativity. This approach allowed the team to work together seamlessly, stay curious, and remain motivated to explore and innovate.

So, why is an agility mindset so important in data science? Because it transforms challenges into opportunities. Data science is rarely predictable—it’s a journey full of surprises. You might start with one question, only to realize halfway through that the real insight lies elsewhere. Or you might encounter messy, incomplete data that forces you to rethink your approach.

But here’s the thing: with agility, these moments aren’t roadblocks—they’re chances to adapt, try new ideas, and improve. An agility mindset helps you learn, relearn, and let go of old ways when needed. It’s about staying open to change, adjusting quickly, and getting better with every step.

In our journey with AIP-DM, this mindset became our foundation. We stopped seeing changes as disruptions and started treating them as chances to refine our work and deliver even greater value. Agility gave us the freedom to explore, the confidence to navigate uncertainty, and the resilience to overcome challenges.

The takeaway? Data science is about discovery, and discovery requires flexibility. With an agility mindset—and a commitment to learn, relearn, and unlearn—you can embrace the unknown, collaborate effectively, and grow at every step. So the next time your project takes an unexpected turn, remember this: Uncertainty isn’t the enemy. It’s an invitation. With an agility mindset, you can turn any challenge into a chance to discover, innovate, and grow. Because at the end of the day, data science isn’t about having all the answers. It’s about being ready to find them—no matter where the journey takes you.