# It's Time to Lead Your Team of AI Agents

Gopika Mohanan, Just Product 2026, 13:30. Automatic transcript of the livestream (Parakeet ASR, unedited, expect misheard names). Timestamps are positions in the talk video.

[0:02] Alright, so our next speaker.
[0:08] See, you know now I know what I forgot when I walked uh on that stage.
[0:13] That sometimes happens.
[0:14] So I have these, you see, I have these cards lying around, and I have a system.
[0:19] So I take them on stage whenever I go on stage.
[0:21] This time I didn't, and then afterwards I rip them so I don't take the old ones.
[0:26] Um that's a very manual, it's not not AI.
[0:29] Alright, so Gopika.
[0:32] So here's also like a bit of a personal story.
[0:34] So Gopika's name, so she applied to do a talk at the conference, and it popped up this year.
[0:40] Actually, she applied twice this year, like with two topics, and it popped up, and I thought, wow, I I think I know that name.
[0:47] And then I went through all the submissions like over the last years.
[0:51] Actually, I asked Claude to ask whether she did that before.
[0:54] And then I saw that she actually applied four times.
[0:58] So I think every edition of the conference, and I mean I would say that's persistent, and I really love that, like, especially in product field.
[1:05] I love persistence because you get so much shit, right?
[1:08] So, like persistence really pay uh uh pays off there.
[1:12] Um, she's currently a director of product at best secret group, and um she is managing a team of agents, and today she's showing you um how she is doing it.
[1:25] So please welcome her on stage.
[1:27] Go Pika.
[1:34] Have fun.
[1:39] I think I have a feeling my agents applied on my behalf.
[1:44] Anyway, okay.
[1:48] So let's get started with maybe a little exercise.
[1:53] So imagine, well, tomorrow is the weekend, but you go to the office on Monday, and your boss basically tells you that good news, you have five new people in your team.
[2:08] We've just hired these people for you.
[2:11] They are incredibly fast, they can research a market in a few minutes, they can pull out all your customer insights, they can code, they don't take bathroom breaks, thank God, and they had they have no working hours, but there are a few small problems.
[2:43] And sometimes they will give you something and be very confident about it, which most definitely might be wrong.
[2:53] So, how would you manage them?
[2:57] Would you look at every single thing that they do when they make a mistake, would you give them a 15-page long instruction?
[3:08] Would you go and correct and tweak every single output with a hope that someday they will get it right?
[3:18] Or would you start thinking about a few different things?
[3:22] Things like roles, context, decision rights, and essentially how to build a high-functioning team.
[3:31] I already lead a team of roughly eight product managers and across the one of the largest experiences across Best Secret.
[3:44] But as multi-agent orchestration systems came into play and started getting, you know, we've started integrating them into our workflows, and even the product teams have started integrating this in their day-to-day.
[4:01] I realized that by jumping on this new capability, it's we are not using it just for productivity.
[4:11] We're thinking about this capability as something that can truly take the team to the next level.
[4:33] What it's doing more than teaching us technical skills is forcing us to practice some very old timeless product leadership skills.
[4:42] And I think the true difference with working with agents and also working with your own teams of product managers is previously you would wait for your product manager to, you know, go through the hoops of hierarchy, and once they have a team and a team of people, you start, you know, you you teach them and uh coach them to be leaders.
[5:05] But I think that that has changed.
[5:08] Even your most junior product manager on your team needs to learn some of these very basic and timeless leadership skills.
[5:16] And I think the reason why we are talking about coaching your team with leadership is because if you look at the last years, interacting with AI has been more of a question and answer system, right?
[5:31] You ask it a question, it gives you an answer, but most of the work is still was still being done by you.
[5:38] And that has changed, especially I think sometime around May, June this year, where truly this experience of moving from a goal to result is what has emerged.
[5:52] So you're basically going from asking questions to delegating outcomes.
[5:58] When you ask or when you give a goal to an agent, what it's essentially doing, which most of you may already know, is it's observing the information it has, it's looking at the expectations from the person inputting the question or the goal.
[6:17] It's looking at what tools I have in my hand that I can reuse and connect to.
[6:23] It puts forward a bunch of solutions, thinks through it, inspects it, and then comes back with the result.
[6:31] And it keeps going and keeps looping around the system until the task is done.
[6:38] And what's happening here is you're no longer, if you look at the relationship with this tool, you're no longer just using a tool.
[6:46] You're directing work.
[6:48] And so the quality of output very much depends on what you're putting inside.
[6:53] And I think, yes, it's it's really exciting that you know, even my product teams are thinking about what tools to use, what's you know, the new latest models we can integrate in the features we are building, and what are other competitors doing?
[7:08] I think all those are very valid and right questions.
[7:11] But I think the larger question is when you can really delegate a huge part of your work, what are the core skills that truly makes you a great product manager?
[7:22] And I think the next four skills that is over the next few minutes that we will look at tries to answer this specific question.
[7:32] Let's move to the first one, which is delegation and context, right?
[7:38] Imagine this: you are a product manager for a fashion e-commerce platform, the senior leadership comes to you and tells you, you know, the size and fit, you know, returns caused by sizes are rock skyrocketing through the roof.
[7:56] Can you help develop a product strategy to help us reduce this and improve our order economics and the cost to serve?
[8:06] And usually this takes a huge amount of work.
[8:10] You need to collect all the data that's caused by this problem, whether that's inventory related or customer related, you need to look at the market, see what other competitors are doing, what external tools are there that help mitigate this problem that your company is facing.
[8:27] You need to explore your solutions based on feasibility, time to market, risk, impact, so on and so forth, and then come to a specific solution.
[8:38] But remember, you actually got these five new people in your team, and you're suddenly the manager.
[8:46] So how do you approach that?
[8:48] If you were to just go to your agent, let's say the data agent, and ask the agent, tell me what you find, look at our size returns and tell me what you find, a prompt like that will most definitely fail.
[9:04] Why is that?
[9:05] It's because the agent doesn't know what you mean by that.
[9:08] What markets, you know, we're a glow, we're a European-wide company, which markets should I look at?
[9:14] What customers should I look at?
[9:16] You know, what should I focus on returns?
[9:18] Rate or revenue impact?
[9:20] Should we be, do we have six months or 12 months?
[9:25] And you know all of these things as a product manager because you live inside of the problem, but the agent does not.
[9:33] So what you need to do is improve the brief or improve the context you're providing your agent, and this is just a very small part of it.
[9:42] We'll we'll go into a little deeper into that.
[9:45] Is instead of a very vague task, go deeper into your asks.
[9:50] Tell your agent what do you mean by analyze, what do you want it to analyze, what markets you wanted to analyze, what customer types.
[9:58] Should it focus on a specific type of problem or separate the problems?
[10:03] And the reason we do this is because the quality of your output, which you know, obviously depends on your input.
[10:12] We can also draw some parallels, right?
[10:14] I think I heard a previous talk where there was parallels made with a new employee.
[10:20] Let's imagine you have a new employee in your team.
[10:24] You of course give them a task to get them started on the work.
[10:27] And you know, you you structure the work in a or structure the first task to get them on boarded, and they do a pretty great, you know, they they try to get by with it, but you will still not get the full output from them because they have no clue about your company.
[10:48] They don't know that actually, you know, you're not a full price fashion uh e-commerce, you're an off-price.
[10:55] Your customers are quite different from the rest of the market.
[10:58] Six months ago, you actually tried and failed with an experiment, and that's not something you want to focus on.
[11:06] So your new employee has no context about what has happened in the past and what is the expectation now.
[11:13] And that is exactly the same with an agent and giving them context.
[11:18] So of course you can delegate a piece of your work by structuring the prompt and the brief well.
[11:24] But I think what's more important is to build an environment of context that the agent can grasp very quickly.
[11:32] So here, I mean, in the image you can see, right?
[11:35] The way we, and this is of course for internal product productivity.
[11:41] I think the structure looks a little different when you're doing an initiative that's that's customer-facing.
[11:49] But here what you're doing is organizing the different pieces of critical information in structured manner that your agent or a group of agents, depends on how you've set up the environment, can act on.
[12:48] And understands the context.
[12:51] I think a few days ago I came across this quote, which I think is not a quote by me, a quote that I saw, which actually pretty well describes the following skills as well, right?
[13:04] It says, as the cost of creating approaches to zero, creation itself becomes less valuable.
[13:11] And what becomes more valuable is knowing what to create, why it matters, and how to judge the results.
[13:17] I think the talks previously touched upon also some of these, some of these different inputs, but I think this flows into the next three skills.
[13:27] The second one, which is defining decision boundaries.
[13:31] Now let's imagine you've given your agent good context, you have created an environment where it knows how to behave and how to what information to pick from.
[13:43] What do you do when your agent actually starts to want to make a decision?
[13:48] What happens when they want to send that email or push that code or set up an A B test?
[13:55] I think that, and there's also a lot of questions that have been talked about recently, which is how autonomous should my agents be.
[14:06] I am not sure if that's the right question, because if you draw these parallels again with your human team that you have, you will never ask yourself how autonomous should my team be.
[14:17] I mean, of course you want your team to be autonomous and empowered and independent to make the decisions.
[14:24] But you also, when you work with your team, you also ensure that the team knows where they need to involve you, what critical decision points you need to be involved before they take that next step, whether that's due to risk or due to potential impact.
[14:42] I think that's that's different, but it's the same principles you apply to your agents.
[14:48] So you it's more you're defining the checklist on what they need to involve you on rather than how autonomous they should be.
[14:57] And this is a very simplified version of that, right?
[15:00] So you're basically creating a framework where you're telling the agent these are the different tasks that you can act on.
[15:08] So imagine the example that we just looked at, uh analyzing return data, right?
[15:14] If your agent notices something interesting and wants to trigger a second analysis, they should absolutely be able to do that.
[15:22] But if they want to now start a test that's running, they should absolutely not be able to do that to save your save your mind and your job.
[15:32] But these are some of the, and I think this this probably describes it a little better, right?
[15:37] So with agents, what's great is you can actually hard code it more than what you might do with your human team.
[15:44] So specific tools like your research agent would want to have a connection to the web.
[15:50] Um your uh analysis agent can have read access to certain things.
[15:55] So there's of course, you can define these decision boundaries much harder with an agent and probably more than with your with your human team.
[16:05] But what you're explicitly setting here is certain checkpoints where you want your agents to escalate to you before they take an action.
[16:14] And so to summarize, with decision boundaries, you're basically telling the agent this is the boundary.
[16:19] Within this, you can act, but outside of it, you can escalate.
[16:25] And again, I think I have said it once, I'll say it again, it's nothing new.
[16:30] It's just classic leadership.
[16:32] You're giving, you know, a good human manager does the same.
[16:35] They're incredibly clear about what they want their team to achieve, but with specific milestones that before the team goes and makes a decision, they want to be present to ensure that we are moving in the right path forward.
[16:50] And that's the same principles that you use here as well.
[16:55] Skill number three, elevating judgment and setting the quality bar.
[17:00] What does that mean?
[17:02] So now this is let's imagine this is in a general multi-agent orchestration.
[17:12] There's an orchestrator agent, and there are probably sub-agents that's working towards a goal, right?
[17:17] So the data agent is crafting something up, the research agent gets information from the customers, there's competitive analysis, but then there's this orchestrator whose main goal is to summarize or synthesize all of these findings and essentially present a proposal.
[17:38] I think as a product manager, the worst thing you can do in this situation is to tell your orchestrator, or in this case, a solution agent, to say, what should I do with all these findings?
[17:51] I think the great thing about having agents is they can truly help product managers or essentially every team to expand the breadth of the biggest advantages we have.
[18:07] So agents they don't get tired after reviewing the fifth idea or the sixth idea.
[18:12] They can go on in multiple loops.
[18:15] They can also look at the same problem using multiple angles, right?
[18:19] So you can, you know, uh one of the examples that we did was you can take a problem and essentially just fire up multiple agents looking at the same problem with different context.
[18:31] One agent is exploring those potentially potential solutions through uh time to market, the other with the risk, the other using how based on customer uh based on the user experience, what's the best path forward?
[18:46] And it's actually great if the agents don't agree, because that's also the beauty of evaluating trade-offs and evaluating options.
[18:55] Because as product managers, what this enables us to do is gives us the opportunity to spend more time exploring the alternatives before we commit.
[19:07] And previously, what we would do is have that one idea and defend it and then go to market and set up set up an experiment.
[19:15] And we don't we we, you know, the ch uh
[19:18] We always took a step back before we challenged our ideas from multiple angles.
[19:23] But that is what agents do with you.
[19:26] So I think a few people in my company have also set up these drill me agents, right?
[19:34] The whole job of this agent is to challenge on every input you give to expand your decision space or your opportunity space.
[19:43] But again, the division of labor is very critical here.
[19:48] So here, what you're doing is the agent gives you the possibility to expand the decision space and the product manager narrows it.
[19:56] The agent gives you a lot of load of possibilities, but it's your job to figure out which one is worth pursuing.
[20:03] And how that translates to hiring.
[20:06] So I mean, when we are now hiring product managers, I think the most critical thing we are looking at is critical thinking, right?
[20:13] Because previously, I mean, it has always been part of a great hiring, but that's where we're placing more and more of the effort in.
[20:21] How can we ensure that people with really great product taste and truly great critical thinking is is coming into our pipeline in because the cost of producing is getting to be so so little.
[20:36] Moving on to the fourth and the last skill, which is building organizational capability.
[20:42] Again, let's take an example.
[20:44] You have these agents running, and one of the agents constantly keeps making the same mistake.
[20:52] So, you know, hypothetically, you wanted to go into an analysis, a customer analysis, and every time it comes back, it's just giving you superficial information.
[21:06] If you again draw parallels to a human employee, right?
[21:09] If one of your team members is not able to produce the work that is of good standard, you don't go and simply correct it and move on.
[21:20] You actually invest in teaching the person what is what do we mean by good quality work?
[21:28] Can we question the assumptions behind the decision you made so that the next time you go and face the same challenge, you would keep this in mind.
[21:37] And it's the same principles with an agent.
[21:41] Because what we do when teaching an agent is we need to tell it how to think, what principles to follow, what good looks like.
[21:49] And again, the benefit with an agent is every piece of feedback that you give it, it is a constant permanent upgrade.
[21:57] So here, right, you you basically take a reusable framework, and this is a one-agent example.
[22:04] So you know this one agent is going off the rails in some use cases.
[22:10] You look, relook at what is the inputs behind the specific agent, tweak the learnings for this agent, and the next output that it produces, you can then compare if that has changed the output of this specific agent.
[22:27] But I think the great thing is it's it's once you've figured that out for one agent, you can actually build it for an entire workflow.
[22:35] So in production, we have systems where we have uh user-facing features where you know, if let's imagine a situation where you want to categorize certain products on the shop, right?
[22:50] We have an agent actually looking at the data and classifying it, but then there's another agent challenging that output.
[22:58] And the underlying principles and learnings are fed across the system.
[23:02] So, because as a leader, going back to the same principles, you are developing the judgment not just for one team member, but you're developing this judgment for the entire of your team so that they make good decisions consistently and time over time.
[23:20] And your job as a leader or as a leader of agents becomes the same.
[23:25] Yes, it's great to optimize one agent to produce one task, but how can you ensure that when these multi-agent systems collaborate, your outcome is, you know, the reproduction of the results is consistent with high quality.
[23:40] So let's go to the last question, which is what does this change for a product manager based on these based on these things that we talked about, and also from the outcome of this conference, right?
[23:55] We go back to this situation where you go to work on Monday, your boss tells you we have five new people.
[24:03] What do you actually do with these with these five new people agents?
[24:09] I think you do what great product leaders have always done.
[24:14] When you work or start, wherever you are in your journey, what you do is you define a great outcome, you set a great brief, you build an environment where the agent understands what are the limits they're working with.
[24:30] You set the boundaries so that you know exactly the key critical points in this journey of building on where you want to be involved and where you want to be escalated to.
[24:43] You use these agents or work with these agents to expand the decision space, get the right trade-offs, get the right constraints, truly work with them to flesh out the problem and solution, but at the end, you own the judgment while we still can.
[25:01] And lastly, you build a system that constantly improves over time.
[25:06] You don't optimize at uh, you know, at a at a local level.
[25:10] You look at the whole global system of how your your system is working well.
[25:17] So I think I will leave you with that.
[25:20] Your new team is waiting for you, and I think all it's looking for is you to go and lead them.
[25:26] So thank you so much.
[25:31] Thank you so much.
[25:33] Thank you so much.
[25:35] Wow.
[25:36] Okay, I like I I always find this like super interesting, especially like looking at it and how the way you are looking at it, because what it means is we all have to become product leaders now, right?
[25:47] That is correct.
[25:48] Okay.
[25:50] I heard I hear this is a very hard job.
[25:54] It's gonna get even harder, but that's what makes it exciting, right?
[25:58] Brilliant.
[25:59] Do you folks have questions for GopiCar?
[26:03] There is one already.
[26:11] Um I think it's um definitely meaningful to be here, but um, how are you struggling with the typical um problems regarding enablement?
[26:21] I mean, you cannot just install locally cloud and give it all your customer data, right?
[26:26] Um can you re-elaborate on your question?
[26:31] I mean, to be able to do all this agentic stuff, you need the capabilities within your company to be able to use them, right?
[26:41] So there's a lot of topics coming on legal stuff, right?
[26:45] Technical stuff and all this stuff.
[26:46] So how are you handling that in concrete?
[26:48] Uh is it just provided by some platform team that they say, hey, here's your or how yeah, I think that maybe it's also misleading that every single thing that a product person does or every feature you built is agentic.
[27:03] I don't think so.
[27:05] Um at least maybe it is for certain companies, but for us we look at very specific problems in isolation.
[27:12] So if there is, I mean, we just talked about size and fit, right?
[27:17] There are meetings happening on this, there are transcripts, there is Qualtrics data that we connect to, there is uh app outputs of customer reviews, we have uh supply data analytics that is connected to to the tooling.
[27:33] And uh I think the system is built specifically around that problem.
[27:38] We also have other initiatives that's customer facing, where you know we are a shop like e-commerce, and there's uh hundreds and thousands of products going into the shop.
[27:48] And there are also use cases where there are uh agentic systems built to classify the products on site, and for that you need a whole different tooling um that you work with.
[27:59] So I think it's never a case of enabling agents to do everything.
[28:04] It is we always define if it's internal, like productivity, which in the my example was that, or if it's external, which is customer facing, and we define the the environment based on that.
[28:16] So we never, I don't think I hope, because it also costs money the more you feed the agents a lot of things.
[28:18] I think the you know it's always like how much cost are we willing to spend to get the output?
[28:28] So that's why we also don't do this for every single uh area.
[28:32] We always weigh what's the true benefit of having a setup like this.
[28:38] Thank you.
[28:39] Perfect.
[28:39] We sadly don't have time for another question, but find Gopika, she's uh staying for a bit and ask her all the questions that we are not able to get you.
[28:49] Uh uh to I forgot what I wanted to say, never mind.
[28:53] All right, so uh again give her a roaring applause.
[28:57] Thank you so much.
[28:58] Thank you so much for the same thing.
[28:59] Thank you so much for being here.
[29:00] Yes, thank you.