Product managers are increasingly drawn to AI product management as more and more companies are building their products using artificial intelligence techniques. 

At PM Exercises, we recognize the shift and are eager to discuss the best strategies for entering the AI PM job market. 

As a quick background about me, I’m Kai Wang, an AI product manager for Uber’s AI product called Michelangelo. I recently talked about the fundamentals of AI product management at one of PM Excercises’ AI PM community sessions, which I will also discuss in this blog.

So, what do aspiring PMs need to know about AI products and the technology that runs them? I will go over some of the most crucial concepts and share my personal experience at Uber to give you valuable insights about AI product management.

You can watch the community session video below or read the blog to learn more about this topic.

If you want to join our AI Product Managers community and be part of our weekly community sessions, sign up for free.

What is AI? 

AI, or artificial intelligence, is a machine’s simulation of human intelligence, especially by a computer system, aiming to automatically perform tasks that usually require human intelligence. 

AI is often equated with machine learning (ML). Are these two the same thing? 

The truth is that AI isn’t always powered by machine learning models. AI can be a rule-based system, where the hardcoded rules, if A, then do B, are programmed into the system. Depending on the complexity of the system, there can be tens of thousands of rules. By chaining all these rules together, such a system can make decisions or take action on new data without human intervention.

So why aren’t AI systems just built using rule-based programs? Sadly, hard-coded rules quickly fall short when the problem space becomes more and more complex.

Machine Learning

This is where machine learning (ML) comes into play. 

ML uses algorithms trained on large data sets, making the machine learn from such data and improve over time.

Why is machine learning a more efficient way to realize AI? ML models can adapt and learn from data via training, handle complexity, and scale much better. Unlike rule-based systems programming, ML doesn’t require manual coding of every condition. Hence, an ML system can generalize, adapt, and achieve better performance across diverse scenarios with less maintenance.

How Does Machine Learning Work?

Instead of hardcoding rules, we feed historical data—a lot of data—into the system and let it learn the rules by itself. We define how the system should learn by coding an algorithm. 

Then, the algorithm is run on data to learn the rules. The learning process is called training, and the output of this training is called the model.

When a model is deployed into production, it can make real-time predictions on input data, and that process is called inference.

That’s the brief definition of machine learning. So, what about deep learning? 

Deep learning is a subset of machine learning that can deal with more complex machine learning problems that usually involve unstructured data, such as text, images, and language. 

It leverages a type of architecture called a neural network in learning hierarchical representations of data and extracting intricate patterns to make better predictions. 

Generative AI leverages a specific type of neural network architecture called GPT (Generative Pre-trained Transformers) to achieve capabilities that generate content such as text, images, codes, videos, and music, while Large Language Models (LLMs) are a specific application of Generative AI that is focused on language-related tasks.

LLMs are trained on vast amounts of text data to understand and generate text based on human language. Today’s most commonly used LLMs include OpenAI’s GPT4, Google’s Gemini, and open-sourced ones such as Meta’s Llama series.

What is an AI Product? 

An AI product leverages AI—often ML—to perform a limited set of tasks with the goal of improving human productivity. 

There are two concepts in AI products: artificial narrow intelligence (ANI) and artificial general intelligence (AGI). So far, with today’s advancements, all AI products built are within the scope of ANI. This means that they can perform a limited set of tasks within a limited set of contexts.

AGI, if achieved, would possess broad cognitive abilities akin to humans. These may include abstract reasoning, adaptability to new situations, and the ability to learn and apply knowledge across diverse domains—capabilities beyond the scope of current ANI systems.

Although the recent breakthrough in generative AI, especially in large language models (LLMs), sparked the debate on how far away we are from AGI, most experts in the space still think we’re far away from that, although we’re making significant progress. 

In 2023, Elon Musk predicted that we would achieve “full” AGI by 2029. The path to AGI is still unclear, but the truth remains that we are advancing quickly. 

3 Major Categories of AI Products

1. AI Tooling 

AI tooling or machine learning toolings are platforms and frameworks that make ML developers’ lives easier. They cover a wide range of functions, such as data processing, model training, deployment, and monitoring. 

AI tooling simplifies and speeds up the development process. It lets both experienced teams and startups build and deploy AI systems in a more practical way.  

About 10% of all AI product managers work in AI tooling products. Some of the most popular examples of these products include: 

• Uber’s Michelangelo 
• Amazon SageMaker (AWS)
• Vertex AI (from GCP)
• PyTorch (from Meta)

2. AI Services

Using AI tooling products, many machine learning developers build niche-specific AI services that solve specific ML problems. 

About 20% of all AI product managers work in AI services, and some of the most common of these products include: 

• Amazon Rekognition: to recognize and classify images
• Azure Bot Service: to easily build chatbots 
• GCP Recommendations: to build any recommendation system (such as YouTube, TikTok, and Amazon shopping recommendations) 
• C3 AI Data Vision: to get insights and analytics for large-scale data sets
• IBM Watson Assistant: to build and deploy conversational AI solutions

3. Applied AI

Applied AI products take advantage of machine learning to create and enhance products that solve real-world problems. There are many examples of these products, but some of the most popular ones include: 

• Google’s search engine
• YouTube’s recommendation system
• Chase Fraud: a bank fraud detection product
• Google Maps functions (such as calculating ETA, producing the best route, etc.)
• OpenAI’s ChatGPT
• Face Recognition (powered by an AI technique called Computer Vision)

About 70% of all AI product managers work in applied AI products. 

How Uber Leverages AI 

As you know, I am an AI PM for Uber’s AI tooling product called Michelangelo. Michelangelo is a machine learning platform that Uber developed to build, iterate, and launch high-quality ML-based applications.

All of Uber’s machine-learning products and features are built with Michelangelo. 

So, how does Uber leverage AI? 

Background

Uber started its machine-learning journey in 2014 when a few teams, such as maps and risk management, started exploring the feasibility of replacing rule-based systems with machine learning. 

Fast forward 10 years, and Uber has fully embraced machine learning. 

Every single line of business at Uber uses ML in its daily operations. Virtually every single button click on the Uber app revolves around machine learning.

Uber Ride App

Let’s explore the Uber ride app. When a user logs in, the app uses ML to authenticate the identity. Then, when they pick a location, machine learning algorithms will suggest and rank the search results. 

Once the user chooses their destination, ML is used to calculate the ETA and map routing, which provide the price and ride recommendation. 

Then, Uber uses machine learning to match a rider with a driver and again, to calculate the routing and ETA. 

When the trip has been completed, fraud detection and payment management are still heavily dependent on ML. Uber’s customer service depends on ML too, using a generative AI-powered chatbot to talk to users. 

Uber Eats

Similar to the Uber Ride app, Uber Eats uses machine learning to power almost every single one of its functions, with a focus on recommendations, ranking, and delivery ETA. 

At Uber, we train about 20,000 models every month and have about 5,000 models in production. With all these models, we make 10 million real-time predictions per second. All this happens in Michelangelo.

Traditional Software Products vs. AI Products

To understand how AI product management differs from traditional software product management, it’s crucial to comprehend the differences between AI products and conventional software products.

Here is a quick comparison:

Success Criteria

A traditional software product is deemed successful if it fulfills all the specified functional requirements. In contrast, an AI product must not only meet functional requirements but also achieve the specified model performance or accuracy metrics. Even the best ML products still err, as is inherent to the nature of ML. It’s the model performance that differentiates good ML products from bad ML products.

Quality

Software product quality depends on your code quality; besides code quality, ML product quality also depends on the data and tuning parameters used. Even if you have a perfect code—which is never the case—it can still output garbage if you input garbage data.

Tools

Conventional products usually use one or two software stacks for tools. However, for AI products, you need to try various algorithms, libraries, and frameworks to achieve the desired results. 

Developers

Companies hire software engineers to develop software products. What about AI products? A team will be comprised of: 

• Data Engineers – to build the data pipelines, collect the data, and clean the data 
• Data Scientists – to build and evaluate the model
• Machine Learning Engineers – to deploy the model to production 
• Software Engineers – to build the whole infrastructure and frontend for users

As an AI product manager, you need to keep in mind that collaboration is going to be the key to successfully developing, deploying, and managing a product. 

Development Progress & Timeline

The development progress for traditional software products is commonly forward and incremental. 

For instance, at Sprint 1, the team is working on the database layer. Next is the API, and third is the UI. When something goes wrong in Sprint 3, it usually doesn’t affect the work done in previous sprints. 

For AI products, it’s a lot different and more complicated. An AI PM will need to try different strategies to get the final results. 

For example, data collection will be Sprint 1. In Sprint 2, you will train the model. In the next sprint, you will start evaluating the model and finding out something doesn’t work.

If things don’t work in Sprint 3, you will need to go back to the first step, asking questions like: 

• “Do we need to collect more data?”
• “Do we need to try a different model architecture?”
• “Do we need to use a different evaluation method?”

So, when this happens, you usually need to start from scratch. You need to repeat all the work you did in early Sprints using different methods. 

Because of that, the timeline for AI products is tricky to estimate. So, you need a sound project management system, as well as the ability to manage ambiguity. 

Interpretability

Lastly, interpretability. In traditional SW products, it’s usually easier to understand why certain actions triggered certain results, and issues are usually more trivial to debug. 

For AI products, issues that arise are going to be difficult to debug and interpret and even predict the results of the interpretation. 

Ensuring interpretability helps build trust and makes it easier to understand and fix issues in AI products.

Traditional SW PM Skills vs. AI PM Skills

Given the key differences between traditional SW and AI products, you can imagine that product management skills may also differ. So what are some of the additional skills required for AI PMs?

AI Technical Foundation

Develop a strong technical background in AI/ML concepts and best practices. This will enable effective collaboration with engineering teams, facilitate discussions on various approach trade-offs, and provide clarity on achievable ML outcomes.

Additionally, possessing the ability to communicate complex product and ML concepts to customers, discuss data strategies, and offer support as needed is essential. We’ll discuss this in more detail in the next section.

ML Problem Understanding

Product managers need to identify and define business problems. For AI product managers, you have to take one step further to map the business problem to a ML problem, to map the business metrics to ML performance metrics. Also, Not all problems warrant ML solutions. Deciding when to and when not to use ML is a critical responsibility for AI product managers.

Data Literacy

Data is the most crucial part of an ML product. Gain practical experience and expertise in handling data and models. Understand data generation and processing, identify data requirements for ML problems, effectively communicate with customers regarding data needs, and establish protocols for data collection and management.

Managing Uncertainty and Ambiguity

The dev timeline of a ML project is difficult to estimate. How do you handle a project timeline slip? Do you have backup plans? Did you set any buffer at the beginning. All these are worth thinking about even before the project starts.

Risk Management

Anticipate and prepare for potential model inaccuracies by implementing mitigation strategies and fallback plans in case your model makes inaccurate predictions, and it will. 

Explainability

Implement a robust strategy and utilize appropriate tools to explain model performance. For instance, if a bank’s fraud detection model rejects a credit card transaction, ensure mechanisms are in place to understand why this transaction was flagged as fraudulent.

Lastly, in AI product management—or in traditional SW product management at that—being user-centric is very crucial. While it’s easy to get so involved in data collection, model development, and model performance enhancement, an AI PM needs to keep in mind that the ultimate goal is solving user problems. 

Let’s get into further detail as to why that’s essential. 

Be User-Centric (1)

“Product goals should be focused on how users can benefit from using the products.”

What exactly does this mean? 

In product development, your key metric should be the success of the users, not the success of the business. 

Here’s a closer look into some popular companies and their product goals:

So, how do these companies make a profit if their product goals are not about making money? While the “But this” column contains their product goals, the items in the “Not this” column are actually their business goals. 

The theory is that if you meet your product goals well, you will reach your product market fit (PMF), meaning you have a growing user base. Once you have a large number of users who love using your product, you will naturally meet your business goals as well.

And what is the top reason for tech startups to fail? Some may say, “They ran out of money.” But why do they run out of money? Because they don’t have users. When you don’t have a product that solves users’ problems, you are likely to fail. 

Be User-Centric (2)

“AI is a means, not a goal, but it can be a very powerful one.”

AI is a powerful way to solve user problems. But, we need to look at it as a means and not the end goal. 

We do not use AI for the sake of using AI. We see so many companies or products that just put AI in their product marketing materials. “AI-powered this, AI-powered that.” But the question is: “Do you really need AI?“

Aside from being a powerful tool that shouldn’t be taken for granted, building an AI product is expensive. Collecting data, training a model, and maintaining that model is expensive. 

Let’s take OpenAI’s GPT-4 as an example. How much do you think the company spent in total to train the GPT4 model? 

The answer is $100 million. 

How about Meta? The company recently announced that it spent billions of dollars to purchase 350,000 H100 GPUs for its AI initiatives. And this is just the amount of money to buy hardware. The cost of building and running the products may cost even more. 

Additionally, developer resources are more expensive in AI products than in traditional SW products.

AI should be used when it’s actually applicable, putting users’ needs as the top priority. 

A Touching Story on AI Feature Development

This story is about a feature request that SenseTime, a leading AI company, received from one of its users. 

As a quick background, SenseTime provides AI technologies to various verticals, such as smart industrial appliances, smart TVs, and smart lights. They are also the creator of a fully-automated Go robot.

Go or Weiqi is a strategic board game played by two players. It involves placing black and white stones on a grid board, aiming to control the largest territory while strategically capturing opponent stones. 

While various mobile apps have been created to replicate the Go game, SenseTime has built a physical Go robot with arms that move around on the board to position the stones.

A co-founder of SenseTime shared a message from one of their users requesting a special feature for the Go robot. It read:

“Hello, I hope you can spare a few minutes to read this message. My grandfather loved playing Go, and we spent countless enjoyable moments together in this game. Last year, I gifted him this Go robot so that he could continue playing while I was not around. Sadly, he passed away this year and I will no longer be able to play with him. However, since every move of the Go games he played was recorded in the robot, I wish your company could develop a feature that generates an intelligent Go based on past data from my grandpa so that I can play a few more games with him again.”

This story exemplifies the importance of building AI products that are human-centric—using advancements in technology to make the world a better place. 

So, as someone deeply involved in AI development, I truly believe that we can make the world a much better place by using AI in the right way. And we should build it with heart.

Be User-Centric (3)

“Be the voice of your customers.”

Please refer to the image above.

If you were an AI PM, which of these answers would you choose? What is the most ideal, given the role of a product manager? 

In this case, D is the best way to respond to the engineer’s question

For C, we need to remember that it’s a PM’s job to talk to the users, not an engineer’s. 

And why should nobody choose B? It sounds like a legitimate answer to “how to solve this problem.” Remember, as AI PMs, we first define the “who.” Then, we define the “why,” “what,” and “when.” The only thing we don’t define is the “how.” However, we should still be heavily involved. 

ML Technical Depths of AI PMs

Based on my observation and experience as Uber’s AI PM, here are the different technical aspects that an AI PM needs to understand to build a successful career in the industry:

By no means can I cover all the topics here, but I hope this section provides a general direction if you want to get started on AI product management in terms of building technical foundations.

FINAL THOUGHTS 

AI is rapidly reshaping the development of products that solve the most complex problems that users have today. 

To summarize our discussion, if you’re aiming to enter or advance in the AI product management domain, consider the following:

• Build your AI technical foundations and be technical enough
• Learn how to relate business problems to ML problems, and when (not) to use ML
• Remain user-centric and ensure that your AI solutions address real user needs
• Keep sharpening your software PM skills as they remain relevant and applicable in AI product management.

PM Exercises Notes:

For aspiring and seasoned PMs alike, it’s a perfect time to break into AI product management and understand its key concepts and real-world applications. 

If you want to enhance your technical AI product management expertise, we at PM Exercises have created an AI Product Management Learning Program where seasoned PMs discuss all the technical aspects of artificial intelligence that you need to understand. 

Take advantage of this opportunity to not only back your resume up with an AI Product Management Course certificate but also enter a network of AI PMs who will share insights and experiences with you, as well as support your career growth.