Summary
Though language learning platform Duolingo had been using OKRs for some time, Project Manager Conor Walsh found them particularly beneficial when launching the platform’s Stories feature. Learn here how OKRs helped his team focus on and stay committed to their true priorities while they sought to better understand their audience.
When Conor Walsh joined language learning platform Duolingo as a brand-new product manager, he was immediately given a challenging problem to solve. The number of users for Duolingo Stories — a product that lets users put their newly acquired vocabularies in the context of an audio narrative — had been declining for a few months. The team had come off a series of unsuccessful experiments. As Walsh explained in his talk during the UXDX conference in Dublin, Ireland, “The results were terrible. The team was demoralized. We just couldn’t figure out what we needed to do to turn this product around.”
To make matters more challenging, resources were tight. Management was getting concerned and was skeptical of the product due to the failed experiments. Walsh needed to come up with a plan of attack, and rally his team and the company around that immediately.
The purpose of Duolingo Stories
As a product, Duolingo Stories existed to facilitate discourse-level learning. “We recognized pretty early on as a company that it isn’t enough to learn individual words,” says Walsh. “It’s not enough to learn just grammar. It’s not even though to learn sentences. If you think about it, the way language really works is through communication. It happens in person, it happens on the phone, it also pulls on a number of different reference points. It pulls on different styles and manners of speaking to speak the language, and also to practice the culture. We called this ‘discourse level’ — the idea of participating in the language that you’re using.”
So Duolingo decided to incorporate storytelling into its learning platform. Users can hear (and read) stories narrated by a native speaker, and they will then receive bite-sized questions to test their understanding of the story. One user, a concert cellist, puts it this way: “Duolingo is practicing scales, stories are like sight-reading.”
OKRs as a tool for setting ambitious goals
When Walsh joined Duolingo, Stories existed only on its web platform. It had originated from Duolingo Labs, an experimental part of the website where they tested new ideas. “We often test things on web because for one, we have dedicated web users, they’re some of the most passionate users of Duolingo, some of the ones who have been with us the longest. That makes it a really great testing ground to get lots of feedback.”
At the time, the number of users who engaged with Stories was relatively small. The product had 12,000 users at the most, compared to approximately the 300 million users the language app had amassed. Duolingo had conducted user research in the past, Walsh says, but didn’t have the bandwidth or resources to do a specific analysis on users who engaged with Stories. But Walsh knew that to improve the fate of Stories, that’s exactly what he needed to do. That means centering his team’s OKRs (Objectives and Key Results) on gaining a better understanding of what the Stories user’s experience was like.
As a company, Duolingo had implemented OKRs since its early days. Louis von Ahn, the company’s co-founder and CEO, worked at Google when he sold his previous company, re Captcha, to Google, in 2009. “I found OKRs to be an effective tool to set ambitious goals and track progress, and when I started Duolingo, I knew they could serve us well, too,” says von Ahn. The company sets its organizational OKR annually, while individual teams and functions sets them quarterly.
Walsh joined Duolingo just as the company was setting its OKRs for the next quarter. “It was a really interesting conversation,” says Walsh, “because I could look at OKRs from previous quarters. Along with the team lead, I really had to sit down and say, well, what are the things we’re trying to do, what are the issues we are trying to address? How do we convert them into Objectives so that our goals are concise and comprehensive?” A lot of effort went on to “wording them just right,” says Walsh, because he knew that revamping Stories would involve cross-functional collaboration. If another team were to read their OKRs, he wanted them to immediately understand what they meant.
Prioritizing objectives and tracking key results
When it came to setting the team’s OKRs, Walsh opted for a combination of qualitative and quantitative Objectives. Rather than starting with an Objective like “increasing Stories traffic by 50 percent,” Walsh decided to carve out space to understand Duolingo’s users. “We wanted to balance things that are very results-oriented and things that are process-oriented.”
“We treated our Objectives as the outcome that we want,” says Walsh. For qualitative goals, the Key Results would outline the process that they would undertake to get to that outcome. For example, one of the team’s Objectives was to:
This ended up being a time-consuming endeavor. Walsh placed an ad on Facebook and screened users based on their experience with the product, as well as their ability in the language they’re trying to learn. The process took almost an entire quarter.
Eventually, hours of interviewing led Walsh and his team to identify a set of pain points that the team categorized into high impact, medium-high impact, and medium impact. One of the highest-impact pain points, Walsh found, was that users found it challenging to use Stories on their phones. Enhancing the mobile experience eventually became another Objective for the team, but not before Walsh and his team tackled some medium-impact changes first. “They were a little bit faster to implement. We could see results. We were changing the content of the course.” As the team started to see progress, the team then turned its focus on bringing Stories to mobile. This Objective led to several significant changes, including a redesign of the product and the way that Stories are presented and narrated to the user.
But Walsh’s focus on understanding the Stories users’ pain points ended up paying off. After redesigning the entire user experience, growth among Stories user skyrocketed. When Walsh joined the company in June 2018, Stories had less than 20,000 users. By October 2019, more than 80,000 users were engaging with Duolingo Stories. Given that Duolingo was Walsh’s first experience with OKR, he admitted being surprised at just how effective they were in measuring and tracking qualitative goals. “They’re really powerful for targeting Objectives like understanding your user.”
Says von Ahn, “We have many ambitious goals that we want to achieve at Duolingo, and our OKR system has been important in keeping us focused and holding us accountable. We’re also a very data-driven company, so setting Objectives with highly specific and measurable Key Results has worked well for us.”
For Duolingo, OKRs are ultimately a guidepost to their destination. “It’s easy to say, ‘You have your OKRs that is set, and it can’t change,’” says Walsh. He believes in continually questioning, “Is this prioritization still correct? As you learn more, you can change the prioritization of your OKRs. There is nothing set about OKRs. It should grow, it should change, it’s not static. It’s important for people to understand that OKRs should be more of a guidepost, rather than a roadmap.”
If you’re interested in starting our OKRs 101 course, click here.
