Picture by Snapwire, on Pexels
This blog post is an updated version of part of a conference talk I gave on GOTO Amsterdam last year. The talk is also available to watch online.
As a Machine Learning Product Manager, I am fascinated by the intersection of Machine Learning and Product Management, particularly when it comes to creating solutions that provide value and positive impact on the product, company, and users. However, managing to provide this value and positive impact is not an easy job. One of the main reasons for this complexity is the fact that, in Machine Learning initiatives developed for digital products, two sources of uncertainty intersect.
From a Product Management perspective, the field is uncertain by definition. It is hard to know the impact a solution will have on the product, how users will react to it, and if it will improve product and business metrics or not Having to work with this uncertainty is what makes Product Managers potentially different from other roles like Project Managers or Product Owners. Product strategy, product discovery, sizing of opportunities, prioritization, agile, and fast experimentation, are some strategies to overcome this uncertainty.
The field of Machine Learning also has a strong link to uncertainty. I always like to say With predictive models, the goal is to predict things you dont know are predictable. This translates into projects that are hard to scope and manage, not being able to commit beforehand to a quality deliverable (good model performance), and many initiatives staying forever as offline POCs. Defining well the problem to solve, initial data analysis and exploration, starting small, and being close to the product and business, are actions that can help tackle the ML uncertainty in projects.
Mitigating this uncertainty risk from the beginning is key to developing initiatives that end up providing value to the product, company, and users. In this blog post, Ill deep-dive into my top 3 lessons learned when starting ML Product initiatives to manage this uncertainty from the beginning. These learnings are mainly based on my experience, first as a Data Scientist and now as an ML Product Manager, and are helpful to improve the likelihood that an ML solution will reach production and achieve a positive impact. Get ready to explore:
I have to admit, I have learned this the hard way. Ive been involved in projects where, once the model was developed and prediction performance was determined to be good enough, the models predictions werent really usable for any specific use case, or were not useful to help solve any problem.
There are many reasons this can happen, but the ones Ive found more frequently are:
To start an ML initiative on the right foot, it is key to start with the good problem to solve. This is foundational in Product Management, and recurrently reinforced product leaders like Marty Cagan and Melissa Perri. It includes product discovery (through user interviews, market research, data analysis), and sizing and prioritization of opportunities (by taking into account quantitative and qualitative data).
Once opportunities are identified, the second step is to explore potential solutions for the problem, which should include Machine Learning and GenAI techniques, if they can help solve the problem.
If it is decided to try out a solution that includes the use of predictive models, the third step would be to do an end-to-end definition and design of the solution or system. This way, we can ensure the requirements on how to use the predictions by the system, influence the design and implementation of the predictive piece (what to predict, data to be used, real-time vs batch, technical feasibility checks).
However, Id like to add there might be a notable exception in this topic. Starting from GenAI solutions, instead of from the problem, can make sense if this technology ends up truly revolutionizing your sector or the world as we know it. There are a lot of discussions about this, but Id say it is not clear yet whether that will happen or not. Up until now, we have seen this revolution in very specific sectors (customer support, marketing, design) and related to peoples efficiency when performing certain tasks (coding, writing, creating). For most companies though, unless its considered R&D work, delivering short/mid-term value still should mean focusing on problems, and considering GenAI just as any other potential solution to them.
Tough experiences lead to this learning as well. Those experiences had in common a big ML project defined in a waterfall manner. The kind of project that is set to take 6 months, and follow the ML lifecycle phase by phase.
What could go wrong, right? Let me remind you of my previous quote With predictive models, the goal is to predict things you dont know are predictable! In a situation like this, it can happen that you arrive at month 5 of the project, and during the model evaluation realize there is no way the model is able to predict whatever it needs to predict with good enough quality. Or worse, you arrive at month 6, with a super model deployed in production, and realize it is not bringing any value.
This risk combines with the uncertainties related to Product, and makes it mandatory to avoid big, waterfall initiatives if possible. This is not something new or related only to ML initiatives, so there is a lot we can learn from traditional software development, Agile, Lean, and other methodologies and mindsets. By starting small, validating assumptions soon and continuously, and iteratively experimenting and scaling, we can effectively mitigate this risk, adapt to insights and be more cost-efficient.
While these principles are well-established in traditional software and product development, their application to ML initiatives is a bit more complex, as it is not easy to define small for an ML model and deployment. There are some approaches, though, that can help start small in ML initiatives.
Rule-based approaches, simplifying a predictive model through a decision tree. This way, predictions can be easily implemented as if-else statements in production as part of the functionality or system, without the need to deploy a model.
Proofs of Concept (POCs), as a way to validate offline the predictive feasibility of the ML solution, and hint on the potential (or not) of the predictive step once in production.
Minimum Viable Products (MVPs), to first focus on essential features, functionalities, or user segments, and expand the solution only if the value has been proven. For an ML model this can mean, for example, only the most straightforward, priority input features, or predicting only for a segment of data points.
Buy instead of build, to leverage existing ML solutions or platforms to help reduce development time and initial costs. Only when proved valuable and costs increase too much, might be the right time to decide to develop the ML solution in-house.
Using GenAI as an MVP, for some use cases (especially if they involve text or images), genAI APIs can be used as a first approach to solve the prediction step of the system. Tasks like classifying text, sentiment analysis, or image detection, where GenAI models deliver impressive results. When the value is validated and if costs increase too much, the team can decide to build a specific traditional ML model in-house.
Note that using GenAI models for image or text classification, while possible and fast, means using a way too big an complex model (expensive, lack of control, hallucinations) for something that could be predicted with a much simpler and controllable one. A fun analogy would be the idea of delivering a pizza with a truck: it is feasible, but why not just use a bike?
Data is THE recurring problem Data Scientist and ML teams encounter when starting ML initiatives. How many times have you been surprised by data with duplicates, errors, missing batches, weird values And how different that is from the toy datasets you find in online courses!
It can also happen that the data you need is simply not there: the tracking of the specific event was never implemented, collection and proper ETLs where implemented recently I have experienced how this translates into having to wait some months to be able to start a project with enough historic and volume data.
All this relates to the adage Garbage in, garbage out: ML models are only as good as the data theyre trained on. Many times, solutions have a bigger potential to be improve by improving the data than by improving the models (Data Centric AI). Data needs to be sufficient in volume, historic (data generated during years can bring more value than the same volume generated in just a week), and quality. To achieve that, mature data governance, collection, cleaning, and preprocessing are critical.
From the ethical AI point of view, data is also a primary source of bias and discrimination, so acknowledging that and taking action to mitigate these risks is paramount. Considering data governance principles, privacy and regulatory compliance (e.g. EUs GDPR), is also key to ensure a responsible use of data (especially when dealing with personal data).
With GenAI models this is pivoting: huge volumes of data are already used to train them. When using these types of models, we might not need volume and quality data for training, but we might need it for fine-tuning (see Good Data = Good GenAI), or to construct the prompts (nurture the context, few-shot learning, Retrieval Augmented Generation I explained all these concepts in a previous post!).
It is important to note that by using these models we are losing control of the data used to train them, and we can suffer from the lack of quality or type of data used there: there are many known examples of bias and discrimination in GenAI outputs that can negatively impact our solution. A good example was Bloombergs article on how How ChatGPT is a recruiters dream tool tests show theres racial bias. LLM leaderboards testing for biases, or LLMs specifically trained to avoid these biases can be useful in this sense.
We started this blogpost discussing what makes ML Product initiatives especially tricky: the combination of the uncertainty related to developing solutions in digital products, with the uncertainty related to trying to predict things through the use of ML models.
It is comforting to know there are actionable steps and strategies available to mitigate these risks. Yet, perhaps the best ones, are related to starting the initiatives off on the right foot! To do so, it can really help to start with the right problem and an end-to-end design of the solution, reduce initial scope, and prioritize data quality, volume, and historical accuracy.
I hope this post was useful and that it will help you challenge how you start working in future new initiatives related to ML Products!
More:
Starting ML Product Initiatives on the Right Foot - Towards Data Science
- Global Data Science Platform Market Report 2020 Industry Trends, Share and Size, Complete Data Analysis across the Region and Globe, Opportunities and... [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Data Science and Machine-Learning Platforms Market Size, Drivers, Potential Growth Opportunities, Competitive Landscape, Trends And Forecast To 2027 -... [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Industrial Access Control Market 2020-28 use of data science in agriculture to maximize yields and efficiency with top key players - TechnoWeekly [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- IPG Unveils New-And-Improved Copy For Data: It's Not Your Father's 'Targeting' 11/11/2020 - MediaPost Communications [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Risks and benefits of an AI revolution in medicine - Harvard Gazette [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- UTSA to break ground on $90 million School of Data Science and National Security Collaboration Center - Construction Review [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Addressing the skills shortage in data science and analytics - IT-Online [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Data Science Platform Market Research Growth by Manufacturers, Regions, Type and Application, Forecast Analysis to 2026 - Eurowire [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- 2020 AI and Data Science in Retail Industry Ongoing Market Situation with Manufacturing Opportunities: Amazon Web Services, Baidu Inc., BloomReach... [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Endowed Chair of Data Science job with Baylor University | 299439 - The Chronicle of Higher Education [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- Data scientists gather 'chaos into something organized' - University of Miami [Last Updated On: November 11th, 2020] [Originally Added On: November 11th, 2020]
- AI Update: Provisions in the National Defense Authorization Act Signal the Importance of AI to American Competitiveness - Lexology [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Healthcare Innovations: Predictions for 2021 Based on the Viewpoints of Analytics Thought Leaders and Industry Experts | Quantzig - Business Wire [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Poor data flows hampered governments Covid-19 response, says the Science and Technology Committee - ComputerWeekly.com [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Ilia Dub and Jasper Yip join Oliver Wyman's Asia partnership - Consultancy.asia [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Save 98% off the Complete Excel, VBA, and Data Science Certification Training Bundle - Neowin [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Data Science for Social Good Programme helps Ofsted and World Bank - India Education Diary [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Associate Professor of Fisheries Oceanography named a Cooperative Institute for the North Atlantic Region (CINAR) Fellow - UMass Dartmouth [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Rapid Insight To Host Free Webinar, Building on Data: From Raw Piles to Data Science - PR Web [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- This Is the Best Place to Buy Groceries, New Data Finds | Eat This Not That - Eat This, Not That [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Which Technology Jobs Will Require AI and Machine Learning Skills? - Dice Insights [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Companies hiring data scientists in NYC and how much they pay - Business Insider [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Calling all rock stars: hire the right data scientist talent for your business - IDG Connect [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- How Professors Can Use AI to Improve Their Teaching In Real Time - EdSurge [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- BCG GAMMA, in Collaboration with Scikit-Learn, Launches FACET, Its New Open-Source Library for Human-Explainable Artificial Intelligence - PRNewswire [Last Updated On: January 12th, 2021] [Originally Added On: January 12th, 2021]
- Data Science Platform Market Insights, Industry Outlook, Growing Trends and Demands 2020 to 2025 The Courier - The Courier [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- UBIX and ORS GROUP announce partnership to democratize advanced analytics and AI for small and midmarket organizations - PR Web [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Praxis Business School is launching its Post Graduate Program in Data Engineering in association with Knowledge Partners - Genpact and LatentView... [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- What's So Trendy about Knowledge Management Solutions Market That Everyone Went Crazy over It? | Bloomfire, CSC (American Productivity & Quality... [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Want to work in data? Here are 6 skills you'll need Just now - Siliconrepublic.com [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Data, AI and babies - BusinessLine [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Here's how much Amazon pays its Boston-based employees - Business Insider [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Datavant and Kythera Increase the Value Of Healthcare Data Through Expanded Data Science Platform Partnership - GlobeNewswire [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- O'Reilly Analysis Unveils Python's Growing Demand as Searches for Data Science, Cloud, and ITOps Topics Accelerate - Business Wire [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Book Review: Hands-On Exploratory Data Analysis with Python - insideBIGDATA [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- The 12 Best R Courses and Online Training to Consider for 2021 - Solutions Review [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Software AG's TrendMiner 2021.R1 Release Puts Data Science in the Hands of Operational Experts - Yahoo Finance [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- The chief data scientist: Who they are and what they do - Siliconrepublic.com [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Berkeley's data science leader dedicated to advancing diversity in computing - UC Berkeley [Last Updated On: January 31st, 2021] [Originally Added On: January 31st, 2021]
- Awful Earnings Aside, the Dip in Alteryx Stock Is Worth Buying - InvestorPlace [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Why Artificial Intelligence May Not Offer The Business Value You Think - CMSWire [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Getting Prices Right in 2021 - Progressive Grocer [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Labelbox raises $40 million for its data labeling and annotation tools - VentureBeat [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- How researchers are using data science to map wage theft - SmartCompany.com.au [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Ready to start coding? What you need to know about Python - TechRepublic [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Women changing the face of science in the Middle East and North Africa - The Jerusalem Post [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Mapping wage theft with data science - The Mandarin [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Data Science Platform Market 2021 Analysis Report with Highest CAGR and Major Players like || Dataiku, Bridgei2i Analytics, Feature Labs and More KSU... [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Data Science Impacting the Pharmaceutical Industry, 2020 Report: Focus on Clinical Trials - Data Science-driven Patient Selection & FDA... [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- App Annie Sets New Bar for Mobile Analytics with Data Science Innovations - PRNewswire [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- Data Science and Analytics Market 2021 to Showing Impressive Growth by 2028 | Industry Trends, Share, Size, Top Key Players Analysis and Forecast... [Last Updated On: February 12th, 2021] [Originally Added On: February 12th, 2021]
- How Can We Fix the Data Science Talent Shortage? Machine Learning Times - The Predictive Analytics Times [Last Updated On: February 14th, 2021] [Originally Added On: February 14th, 2021]
- Opinion: How to secure the best tech talent | Human Capital - Business Chief [Last Updated On: February 14th, 2021] [Originally Added On: February 14th, 2021]
- Following the COVID science: what the data say about the vaccine, social gatherings and travel - Chicago Sun-Times [Last Updated On: February 14th, 2021] [Originally Added On: February 14th, 2021]
- Automated Data Science and Machine Learning Platforms Market Technological Growth and Precise Outlook 2021- Microsoft, MathWorks, SAS, Databricks,... [Last Updated On: February 14th, 2021] [Originally Added On: February 14th, 2021]
- 9 investors discuss hurdles, opportunities and the impact of cloud vendors in enterprise data lakes - TechCrunch [Last Updated On: February 14th, 2021] [Originally Added On: February 14th, 2021]
- Rapid Insight to Present at Data Science Salon's Healthcare, Finance, and Technology Virtual Event - PR Web [Last Updated On: February 14th, 2021] [Originally Added On: February 14th, 2021]
- Aunalytics Acquires Naveego to Expand Capabilities of its End-to-End Cloud-Native Data Platform to Enable True Digital Transformation for Customers -... [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Tech Careers: In-demand Courses to watch out for a Lucrative Future - Big Easy Magazine [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Willis Towers Watson enhances its human capital data science capabilities globally with the addition of the Jobable team - GlobeNewswire [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Global Data Science Platform Market 2021 Industry Insights, Drivers, Top Trends, Global Analysis And Forecast to 2027 KSU | The Sentinel Newspaper -... [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- A Comprehensive Guide to Scikit-Learn - Built In [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Industry VoicesBuilding ethical algorithms to confront biases: Lessons from Aotearoa New Zealand - FierceHealthcare [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- How Intel Employees Volunteered Their Data Science Expertise To Help Costa Rica Save Lives During the Pandemic - CSRwire.com [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Learn About Innovations in Data Science and Analytic Automation on an Upcoming Episode of the Advancements Series - Yahoo Finance [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Symposium aimed at leveraging the power of data science for promoting diversity - Penn State News [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Rochester to advance research in biological imaging through new grant - University of Rochester [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- SoftBank Joins Initiative to Train Diverse Talent in Data Science and AI - Entrepreneur [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Participating in SoftBank/ Correlation One Initiative - Miami - City of Miami [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Increasing Access to Care with the Help of Big Data | Research Blog - Duke Today [Last Updated On: February 22nd, 2021] [Originally Added On: February 22nd, 2021]
- Heres how Data Science & Business Analytics expertise can put you on the career expressway - Times of India [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- Yelp data shows almost half a million new businesses opened during the pandemic - CNBC [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- Postdoctoral Position in Transient and Multi-messenger Astronomy Data Science in Greenbelt, MD for University of MD Baltimore County/CRESST II -... [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- DefinedCrowd CEO Daniela Braga on the future of AI, training data, and women in tech - GeekWire [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- Gartner: AI and data science to drive investment decisions rather than "gut feel" by mid-decade - TechRepublic [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- Jupyter has revolutionized data science, and it started with a chance meeting between two students - TechRepublic [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- Working at the intersection of data science and public policy | Penn Today - Penn Today [Last Updated On: March 14th, 2021] [Originally Added On: March 14th, 2021]
- The Future of AI: Careers in Machine Learning - Southern New Hampshire University [Last Updated On: April 4th, 2021] [Originally Added On: April 4th, 2021]
- SMU meets the opportunities of the data-driven world with cutting-edge research and data science programs - The Dallas Morning News [Last Updated On: April 4th, 2021] [Originally Added On: April 4th, 2021]
- Data, Science, and Journalism in the Age of COVID - Pulitzer Center on Crisis Reporting [Last Updated On: April 4th, 2021] [Originally Added On: April 4th, 2021]