Cloud Hosting

A Critical Perspective on Agile Methodologies in Data Science14 min read

Data Science is gaining prominence as a mainstream practice in various industries, leading companies to integrate it into their operations. However, there is a genuine concern that Data Science may be mistakenly categorized as just another software practice, akin to traditional web application development approaches. Over the past years, the Agile hype has spread throughout the technology industry, extending beyond its roots in web development.

Recalling an anecdote, I was once told about Agile being introduced into a legal practice, much to the surprise of the attorneys involved. They found themselves adopting techniques that were completely disconnected from their legal practice, day-to-day work, and actual needs. The resulting negative feedback and disengagement were so overwhelming that they cannot be ignored or understated. The impact was reported to be mentally distressing, almost as if it were an experiment conducted by Dr. Zimbardo[1] himself, the renowned social psychologist.

While coding does play a role in Data Science, it is not the primary activity of a Data Scientist. Unfortunately, this distinction is not widely recognized or understood by individuals outside the field. As organizations grapple with a misunderstanding of what Data Science truly entails, there is an increasing pressure to enforce alignment. In the dynamics of small groups, teams often expand, and IT views Data Science as a logical area for expansion, leading to a perfect storm of misalignment.

To illustrate this point, I once witnessed, with a sense of unease, an Agilist referring to a Data Scientist as a developer and a Notebook with a model as an application. Such remarks highlight the profound misunderstanding of Data Science within the IT industry. It seems that certain factions of the industry adopt a one-size-fits-all mentality, treating a data science study in the same manner as they would approach a web application development project. This approach feels archaic and reminiscent of a bygone era.

Before delving into a detailed analysis of why Agile does not align with Data Science, it is important to understand the origins of Agile and the motivations behind its creation.

The origins of Agile can be traced back to the proclamation of their Manifesto[2] by the Agile Alliance. In general terms, a manifesto is defined as a written statement publicly declaring the intentions, motives, or views of its issuer, as described by Merriam-Webster[3]. Traditionally, manifestos have been associated with movements and schools of thought in social, political, and artistic realms, representing or aspiring to bring about significant qualitative progress for humanity.

Historical examples of manifestos include The Declaration of the Rights of Man and of the Citizen (1789) by the National Assembly of France, The Abolitionist Manifesto (1829) by William Lloyd Garrison, and The Communist Manifesto (1848) by Karl Marx and Friedrich Engels. While individual opinions may differ on the content of these manifestos, they address matters of great importance to humanity. Manifestos are also commonly used to define artistic movements or schools, as exemplified by The Bauhaus Manifesto (1919) by Walter Gropius.

In light of these historical references, it is necessary to express some reservations about labeling a software development methodology, created by software practitioners, as a Manifesto. This usage could be seen as somewhat disrespectful to the likes of Walter Gropius, the National Assembly of France, William Lloyd Garrison, Karl Marx, and Friedrich Engels. It is essential to approach such grandiose associations with caution and scrutiny.

The Agile Manifesto was developed by a group of 17 software practitioners who founded the Agile Alliance. These individuals, including notable names like Kent Beck, Martin Fowler, and Ward Cunningham, are primarily recognized for their involvement in the creation of the manifesto. It is important to note that their expertise lies in coding and related activities, which has formed a consulting industry akin to other domains like coaching and training.

While this association is not inherently problematic, it is worth noting that these authors are not widely acclaimed for groundbreaking software advancements, with the exception of Ward Cunninghams involvement in the creation of the first wiki system. This observation highlights that the Agile Alliance lacks direct connections with industry leaders and innovators.

Recognizing their skill and competence is certainly commendable, and it is not fair or valid to diminish their contributions. However, it does raise questions about the significant impact asserted by the Agile Manifesto without substantial groundbreaking contributions to the field. It prompts us to ponder the underlying motivations and intentions behind the creation of the Agile Manifesto.

Considering who benefits from the Agile Manifesto could help shed light on why it was written and why an Alliance was established. The close association between the members and coaching/training/educational activities raises the question of whether the Agile practice is primarily driven by revenue generation.

While this is a common practice and not inherently wrong, illegal, or unethical, one can infer a conflict of interests that may prevent Agile from being solely focused on your best interests. The assessment of this potential conflict and the alignment of Agile with the broader Consulting industry will depend on your prior experiences with such value-added industries. However, it is widely known that these industries often face criticism due to the lack of accountability and difficulties in measuring performance improvements, especially when they move beyond mere rhetoric.

Recognizing that these reservations are subjective, I will now analyze each claim of the Agile Manifesto individually and evaluate its suitability for the field of Data Science.

These four values are advocated in the Agile Manifesto and form the core of Agile methodology. Lets analyze them in detail:

Data Science is generally considered a lightweight scientific activity. This is because many practitioners in the field primarily apply established methodologies to extract business benefits from data, rather than conducting groundbreaking scientific research. Therefore, the term scientist in data scientist can be seen as more of a vanity term that doesnt fully reflect the pragmatic nature of most practitioners.

However, it is important to note that the Data Science process aims to adhere to scientific methodology in terms of rigor, attention to detail, and employed procedures. It also involves significant mathematical aspects, which are inherently scientific. So, while the intensity of scientific method application may be lower compared to actual scientific research, the underlying aim is still present.

In the context of Data Science, it is difficult to prioritize individuals and interactions over processes and tools. In many cases, the focus is primarily on data, methodologies, and analysis rather than individual interactions. For example, when evaluating the value of a tumor marker, statistical rigor and manufacturing quality are typically more important than the level of individual interaction involved in its development.

Data Science is inherently complex, and in practice, it is often even more challenging to understand and verify compared to regular software development. Jupyter Notebooks have gained popularity because they provide a means of combining inline documentation, including mathematical explanations, with actual code. They resemble traditional scientific research notebooks where authors describe their analysis workflows.

In the context of Data Science, the principle of working software over comprehensive documentation does not align well. An undocumented notebook would be a nightmare scenario, as both the process and outcome of the analysis must be accurately described. In Data Science, comprehensive documentation is just as important as the software itself, if not more so.

In Data Science, the concept of customers is not typically present in the traditional sense. While there may be goals in certain projects, sometimes the work is purely exploratory without specific predefined objectives. Additionally, there are usually no formal contracts in Data Science, as it can be challenging to determine the potential outcomes or directions of a particular study or analysis.

However, it is crucial to clearly specify the specific analysis, outcomes, assumptions, data, and methodology involved in the activity itself. This documentation is typically included as part of the Data Science process, often within the notebook if notebooks are used for analysis. Its worth noting that some Agilists may argue that there are internal customers within a business or organization, as the data generated is intended to be valuable for the overall operation. However, this perspective does not align with the core principle of Data Science.

In summary, while customer collaboration may not be the central guiding principle in Data Science, the clarity and specification of analysis details are essential components of the practice.

In Data Science, traditional plans in the sense of predefined step-by-step procedures are not typically used. Instead, the process often involves formulating hypotheses and testing them or predicting and classifying future events based on past observations. The plan itself becomes an hypothesis to be validated.

However, its important to note that there is usually a script or plan outlining what needs to be done and how to do it. The exploratory nature of Data Science means that outcomes may change and redirect the course of analysis. While this principle of responding to change does not explicitly contradict Data Science, it is not entirely applicable to the field. This principle describes a contradiction that occurs in a different context, such as web application design, where lengthy requirement documents are commonly written and sometimes form part of contractual agreements.

In summary, while Data Science doesnt adhere to traditional plans, there is still a general script or plan in place that can be adjusted based on the evolving insights and outcomes of the analysis.

In addition to the values, the Agilists have principles:

Lets examine them one by one:

(1) Prioritize customer satisfaction by delivering valuable software frequently

In Data Science, the primary focus is on delivering actionable knowledge and insights from data, rather than software. Customer satisfaction is achieved through the quality and impact of the extracted knowledge, rather than the frequency of software deliverables. The value lies in the insights gained, not in the software itself.

(2) Welcome changing requirements, even if they occur late in the project

In the realm of Data Science, the requirements are often based on hypotheses to be tested or predictions to be made. While some flexibility may exist in refining the scope of a project, significant changes in requirements can have far-reaching implications. The iterative nature of Agile may not be as applicable to Data Science, where study cycles are often longer and altering the requirements late in the project can significantly disrupt the study methodology.

(3) Deliver working software frequently, with a preference for shorter timescales

Data Science is not focused on delivering software but rather on extracting meaningful insights from data. The notion of frequent software deliveries is not relevant or feasible in the context of Data Science. The emphasis lies more on the accuracy, validity, and impact of the knowledge extracted, rather than the frequency or timeliness of software releases.

(4) Collaborate with the customer and stakeholders throughout the project

While collaboration with customers and stakeholders is important in any project, it is worth noting that the nature of collaboration in Data Science differs significantly from that in software development. In the initial stages of a Data Science project, interactions with customers and stakeholders play a crucial role in understanding their requirements and objectives. However, once the project moves into the research and study phase, the focus shifts towards extensive data analysis, experimentation, and hypothesis validation, which often occur over longer periods with less frequent interaction.

In Data Science, the emphasis lies on delving deep into the data, applying statistical and mathematical techniques, and extracting valuable insights. This process requires time, careful analysis, and scientific rigor, which may not align with the iterative and rapid delivery approach commonly associated with software development. Therefore, while collaboration remains important, the dynamics of collaboration in Data Science projects differ significantly from those in software development, reflecting the unique nature of the field.

(5) Build projects around motivated individuals and give them the support they need

The idea of building projects around motivated individuals and providing necessary support seems like a self-evident concept applicable to any industry. In the context of Data Science, it is unlikely that professionals would deliberately choose unmotivated individuals or neglect to provide the support required to achieve project objectives.

(6) Measure progress through working software and adjust accordingly

In Data Science, progress is measured by the accuracy, reliability, and impact of the insights generated, rather than by working software. The focus is on refining and improving the analytical models and methodologies based on the data. Adjustments are made to enhance the quality and reliability of the insights, rather than solely based on the functionality of software.

(7) Maintain a sustainable pace of work

While maintaining a sustainable pace of work is important in any field, including Data Science, the nature of Data Science projects may involve extended periods of exploration, experimentation, and analysis. The pace of work may fluctuate depending on the complexity of the data, the methodologies employed, and the depth of insights sought. Striving for a sustainable pace must be balanced with the requirements of the specific project and the need for thorough analysis.

(8) Strive for technical excellence and good design

While technical competence is certainly important in Data Science, the goal is not to pursue technical excellence or intricate design for its own sake. Data Science is focused on utilizing appropriate mathematical tools and methodologies to extract meaningful insights from data. The emphasis lies on the accuracy, validity, and interpretability of the results, rather than striving for technical excellence in the traditional sense.

(9) Keep things simple and focus on what is necessary

The principle of keeping things simple and focusing on what is necessary applies universally to various fields and is not exclusive to Data Science. While simplicity and focus are important, the complexity of Data Science often necessitates specialized techniques and methodologies. The focus is more on deriving actionable knowledge from data rather than oversimplifying or neglecting important aspects of the analysis.

(10) Reflect on your work and continuously improve

The principle of reflection and continuous improvement is valuable in any professional endeavor, including Data Science. However, it is not unique to Data Science and is a widely accepted practice across industries. Professionals in any field are expected to reflect on their work, learn from their experiences, and strive for improvement. Therefore, this principle does not offer specific insights or considerations specific to Data Science.

Summary

In the context of Data Science, it becomes evident that Agile methodologies fall short and are largely irrelevant. The principles put forth by Agile proponents may be seen as nothing more than empty platitudes, failing to address the specific challenges and intricacies of the field. The notion of prioritizing frequent software delivery, embracing changing requirements, and collaborating with stakeholders throughout the project are not only obvious but also fail to recognize the distinct nature of Data Science. Agiles focus on technical excellence and good design disregards the fact that Data Science is more about using the right mathematical tools rather than achieving technical perfection. In truth, Agiles attempt to infiltrate the realm of Data Science can only be described as complete and utter nonsense.

The practical implementation of Agile, particularly in conjunction with the Scrum methodology, often falls short of its intended goals when applied to Data Science. The periodic meetings, known as stand-ups, where team members provide updates, lead to poor engagement and disruption in workflow. The presence of a non-technical or inadequately skilled Scrum master or project manager further compounds the issues, as they normally lack industry-specific knowledge and reduce complex workflows into simplistic task lists. This lack of understanding and accountability creates frustration and hinders the teams progress.

Additionally, the concept of user stories and the emphasis on user-centric requirements do not align well with Data Science, where the focus is more on data, hypotheses, and analysis rather than traditional user-driven needs.

Furthermore, when Agile consulting services are brought in, the emphasis often shifts to methodology and best practices, disconnecting them from the actual business needs and resulting in repetitive and irrelevant discussions. This disconnect and lack of understanding have detrimental effects on team morale and project outcomes, leading to project failures, low quality, massive hidden costs and other negative consecuences.

There is no one-size-fits-all solution for effective project management in Data Science, but based on my experience and observations, the following approaches seem to yield better results:

By embracing these principles, teams can foster a more focused and collaborative environment for Data Science projects.

The Agile Manifesto poses challenges due to its loose definition and sometimes feels akin another Conjoined Triangle of Success[5]. Its values and principles are not universally applicable across industries, and in the realm of Data Science, they often clash with the specific needs and workflows of projects in this field.

As a Data Scientist, it is not uncommon to find yourself pulled into the Agile methodology. However, I encourage you to consider alternatives. Agile is unlikely to serve the best interests of your employer or customers, and it may drain your energy, focus, and time, diverting you from the path of professional growth. Engaging in low-value activities that stray from your core skills can hinder your career.

On a personal and professional level, it is worth considering adjusting your compensation to reflect the challenges posed by following the Agile methodology. The Agile workflow often fosters an environment focused on justifying the methodology itself rather than addressing genuine business needs. Among the various negative aspects, the sense of wasted time can be particularly disheartening. As professionals and human beings, our time is limited, and how we allocate it directly impacts our learning curve and overall fulfillment.

Moreover, Agiles impact on creativity cannot be overlooked. The rigid planning, approvals, timeboxing, and administrative burdens disrupt the very essence of creativity crucial to excelling in Data Science. The prevalence of frequent meetings and administrative tasks stifles the creative process necessary for innovation.

Unfortunately, the prevailing trend indicates that Agile will continue to gain traction. As the world becomes more challenging, we can anticipate an increase in Agile practices.

In conclusion, as professionals, we recognize the importance of navigating the challenges of Agile with resilience as our armor and integrity as our compass. We shall always strive for impactful work, ensuring our actions align with our principles and exemplify professionalism.

I am a seasoned professional with over 20 years of experience in both technical and non-technical roles in technology. I provide contract services to small and medium sized Hedge Funds in AI/Quantitative and Financial Market Data areas.

I live with my family in Denmark in the countryside. If you would like to discuss industry trends, share insights, or explore potential collaborations, I am always happy to connect.

The opinions expressed in this article are solely my own and do not reflect the views or opinions of any past, present, or future employer or customer.

[1] https://en.wikipedia.org/wiki/Philip_Zimbardo

[2] https://agilemanifesto.org/

[3] https://www.merriam-webster.com/dictionary/manifesto

[4] https://de.wikipedia.org/wiki/Politoffizier

[5] https://www.bustle.com/articles/157415-are-the-conjoined-triangles-of-success-real-silicon-valley-mocks-a-famous-business-model

Read the original here:

Data Science: Debunking the Myth of Agile Compatibility - DataDrivenInvestor