As the leading student-first connected learning platform, Chegg's mission is to help every student achieve their best, in school and beyond. Chegg strives to improve the overall return on investment in education by helping students learn more in less time and at a lower cost. The foundation of Chegg's product is built on a substantial library of archives composed question and answers submitted by students across ample subjects, over the course of years. This library in turn was used by Chegg to build perhaps the most efficient LLMs in EdTech today.
My time at Chegg has been a rollercoaster. There are times in which it challenged my thinking, times that the product changed dramatically, times that the business approach shifted a complete 180, and even times of where a company reorg had consequential effects. All of these shaped me a Product Designer to not only master the skill of my trade but to do so with steadfastness and elegance.
At Chegg, I’ve owned everything from micro interactions and features within the product to the most used surfaces of Chegg. I’ve spent much of my time working on and owning the search experience, as well as the question and answer experience. Across design and cross functional teams, I was the point of contact and person of product knowledge on these surfaces. While these were the main surfaces that I worked on, I’ve also participated as the principle representative for the Design System as well as a new initiative for defining and implementing brand guidelines. I’ve also led many brainstorm sessions, most recently ad monetization focusing on utilizing data to surface relevant ads to users.
I hope to highlight the above in my case study on Chegg. I will share my work on the legacy product, the product today and the future of the product across the aforementioned experiences.
In UX research, students expressed that they don’t always find the exact matches they are looking for. To provide more transparency, it would help to provide them full visibility to the quality of the matches we display. Additionally, in the event that they don’t find the exact match, we give them the ability to post a question.
We have received feedback from our students that they are not confident in the accuracy of our search results. Exact Match is the feature that provides them a match percentage and corresponding match labels depending on the match percentage. Questions posted by students have been going down year on year. An integration with search, will also enable an additional touchpoint with PAQ and meet the student where they need it the most.
This is one of the biggest pain points that students have brought up in interaction with search. More than 50% of subscribers engage with search. This is a critical surface to serve students. It is also a critical moment of truth in the user journey, where a student might want to post a question, in the event that they don’t find an exact match.
Currently, 30% of searches have no results clicked. This is viewed as a negative search experience in which the CTR indicates that a user was able to find what they were looking for. Increasing the CTR percentage is taken as successful search attempt. Goal of this project was 7% CTR lift and a do no harm to cancel rates.
The proposal is to feature a badge name and percentage for all search results. There are three badges that will be featured based on the accuracy level. For great and partial matches, we will feature the post-a-question modal, which will provide students to post a question. The following is the search accuracy level login:
Additionally, we know form user research that when a user comes to Chegg, they are laser-focused on finding an exact match to their question. In this case the solution matters less and the question is of utmost relevance. In the solution we added bolden lettering and highlights to help students quickly and efficiently discern the similarity of their query to the question presented for our achieves. This addition is supplemental to the percent match label.
We also learned that Chegg has predominately two types of search queries, long-tail and short-tail. The former is the most common search query, usually composed of a homework help question, commanding over 90% of searches today whereas the latter is more geared towards conceptual definitions of topics. With this in mind, we attempted to capture the user’s intent when conducting a search and only showing relevant material, as opposed to the tabs experience in the previous search result page experience.
Finally, we changed the primary button on the search results page to match that of the Design System, increasing contrast and affordance.
The changes were a great success and overall improved the quality of the search experience with a 16% lift on CTRs across user types without doing no harm to cancel rates. There was a small lift in conversions as well.
Trying times came upon Chegg with the release of Open AI’s Chat GPT. Upon release, Chegg executives saw a dip in usage and subscribers as more and more students turned to artificial intelligence to get homework help. Investors and business experts questioned the existence of the Chegg product and business model that once turned to experts to derive answers for students to now instantaneous solutions. With a lot in question, reactivity to what unfolded took place. There was a company wide reorganization as teams and contributors were shifted around to counter the threat of Chat GPT. Chegg believe that the company still had an edge. The various tools acquired by Chegg over time and most importantly the multi-million data point archives that it possesses.
With survival on the line, Chegg took this opportunity to turn what as a legacy-based experience of search and ask, to a multi-turn chat experience leveraging the power of AI and the ample library of Chegg achieves.
At this point in time there were no product briefs, decisions were made on the fly and teams needed very much to adapt to one another. The goal was build a foundation for multi-turn chat and release it in stages.
The first milestone would be a duality between two worlds, where the product wasn’t exactly the legacy experience, but it also wasn’t exactly multi-turn chat either. It was meant to be an experience that is usable, scaleable and to serve as a foundation for what’s to come.
The goal in simple terms was build and input mechanism in which the search query would either be answered by an LLM answer automation, retrived from Chegg’s achieves or routed to an expert for help.
This is in fact a deeper question than one may presume as it questions the existential nature of Chegg as a company. This question is important because it begs for another question: why would students use Chegg, pay for it, ask a question and have to wait for answer when they can ask AI and get an answer instantaneously?
This is a fair and dire question to answer. However, once the hype of Chat GPT died down and the discovery phase was slowing down, we found that students did not entirely trust Chat GPT. It became known that Chat GPT can make mistakes and its output isn’t always correct which shifted tides a bit in Chegg’s favor.
If Chegg could find a way to leverage AI and couple it with the substantial library of achieves in its toolbox, there may be a bright future of an AI-enable EdTech product in the market.
The principle success metic of this milestone mainly pertains to the search experience. The goal was to maintain less than 5% of search queries that were returned to the user with no results. The secondary metric was a do no harm to subscriptions and cancelations, in which the experiment would meet or exceed plan for that same period.
To understand, what and how and why we’re doing what we’re doing, we started by creating a user journey map. This map illustrates the logic behind how the product will work.
When a user submits a search query, we first check our achieves to see if we have an exact match, adopted from the legacy search experience project. If we do not have an exact match to the search query, we would pass the query through a moderation check to ensure that the query does not violate Chegg’s Honor Code. If it passes, we then access if the query is of high-confidence or not. This is important because Chegg has been able to develop logic for subjects and topics in which the LLM is likely to return an erroneous response. If it doesn’t fall into the logic parameters identified, then we have high-confidence that the LLM will answer the question correctly. If we have low or medium confidence that the question would we answered correctly, then we route the query to return results from our achieves or send to an expert, a real person for help.
While this logic was being developed by our engineering teams, I was tasked with helping design what we coined as the Unified Input Component (UIC). A single input component that would handle any form entry the user might attempt. This includes text entry, image upload as well as math and science keyboards. The second requirement for the UIC is that it will be scaleable. For the first milestone, it will live on the homepage and accessible on other certain surfaces in certain scenarios, but for the next milestone it would be all of those things as well as a fixed component and the mechanism for enabling multi-turn chat.
Once the UIC was defined and there was alignment across teams, I started working on single content page that adjusts based on the search query. Depending on the user journey and the type of search query, this content page would either surface one or a combination of the following:
There for this search result page had to fend and bold to meet the needs of the content while adhering to a single design pattern, recognizable by students.
This search result page for a low-to-medium confidence search query in which the results returned to the user are AI-generated results, results from Chegg achieves and the ability to send the query to an expert. Since it was also a low-to-medium confidence search query, we ask the user to confirm the subject to increase confidence of relevant search results returned to the user.
When the user clicks on any of the search results we proceed to showcase the desired result to the user on the same page. The search results collapse and are hidden behind a chevron and only the result selected is in the user’s field of view. The user can expand the search result section to select another result if needed, but from the test ran on the legacy experience, we know that a click is positive indication that the user what they are looking for.
Furthermore, we attempt to collect feedback on the content. If they user is not happy with the content the user flow continues and we allow the user to submit their original search query to an expert for help.
Upon releasing to a cohort of users, we found that we exceeded expectations of our north star metric. The target was for searches without any results to maintain below 5% and our solution beat expectations to achieving lower than 1%. Additionally, the new release did not impact subscribers or have any harm on cancelations, maintaining the plan and status quo.
Upon a successful release of UAE and the impact being a success, we continued to simultaneously build on UAE to create a multi-turn chat experience. With MTC, Chegg leveraged Open AI’s LLM Chat GPT for some scenarios but also partnered with Scale AI to build Chegg its own LLM that leveraged multi-million point database and library of achieves to answer questions.
With MTC, not only would users submit a search query – or now what is referred to as a chat entry encompassing Chegg’s entire toolbox, but the user would be able to respond with follow up questions on their search query to continuously adjust and enhance their search parameters to expand on their original search query.
Still in startup mode, Chegg seeks to disrupt the marvel of Chat GPT and offer students something better, a study companion that they can reply on and trust. With products trust is a tricky thing and it means something different with the release of AI. How can you continuously satisfy the needs of users correctly so that they trust your product more than other rival products.
To do this, we had to introduce multi-turn chat. The next generation of a search engine. Traditionally, you can search the achieves or contents of any product and you can refine your search usually by filtering the results to narrow down your search. Search engines today need to be able to capture intent and do so at a high-confidence or at least acknowledge when confidence is not so high and put the power in the users’ hands.
How might we create a search engine that captures a user’s intent at a high-confidence, while acknowledging when it’s not to increase the search query relevance and the rate and accuracy of successful searches?
This matters because Chegg reached a point where it’s either sink or swim from a business perspective. In only revolutionizing the way a search query is performed and how the results returned to users can Chegg only beat the storm.
But for the user needs, it is an opportunity to finally achieve the long anticipated goal of forming what could be a true student companion. If the foundation of the solution is successful, the horizon has much more potential to accompany learners through their education journey.
In testing MTC through user cohorts, the release plan would be at 5% of users. At each 5% increment we will measure for do no harm to search volumes and consumption of exact match coverages.
To understand, what and how and why we’re doing what we’re doing, we started by creating by building on our existing user journey map. This map illustrates the logic behind how the product will work. and how we will incorporate MTC.
When a user submits a search query, we first check our achieves to see if we have an exact match, adopted from the legacy search experience project. If we do not have an exact match to the search query, we would pass the query through a moderation check to ensure that the query does not violate Chegg’s Honor Code. If it passes, we then access if the query is of high-confidence or not. This is important because Chegg has been able to develop logic for subjects and topics in which the LLM is likely to return an erroneous response. If it doesn’t fall into the logic parameters identified, then we have high-confidence that the LLM will answer the question correctly. If we have low or medium confidence that the question would we answered correctly, then we route the query to return results from our achieves or send to an expert, a real person for help.
Additionally now, certain search queries will be routed to Chat GPT whereas others will be answered by our own LLM; where non-academic queries, for example will be routed to OpenAI. Chegg’s LLM will handle additional queries as well, such as generating flashcards, generating practice questions, etc. The user flow map below highlights these cases.
Once again in the process we revisit the UIC to build upon original designs. Now, in addition to being the main component of the homepage, the UIC will be premaritally fixed across the MTC experience. The UIC itself as a component required heavy cross collaboration across broader teams and user research to ensure that designs faired nicely with users.
Designs for MTC were much more complicated than UAE. As the logic for the LLM advanced as did the design solutions. We were presented with some pretty complex problems to solve. In terms of performing a search query via chat input, these were the following use cases I was tasked with solving:
Since the MTC is a multi-layered interface the above use cases would surface to the users as modules within a holistic experience that can be summoned at any point in the MTC experience.
Here, search queries can surface to the user in various states depending on the use case. The user can request to view similar results to their original search query, browse search results, or take other actions depending on their needs. These modules were designed in a way that there is no endpoint for the user’s journey in conducting a search query. Based on the user’s search query, we surface what is of the highest confidence first, then allow the user through guided actions to drill down further to see additional relevant results.
Below is an illustrated flow of a user submitting a query with an exact match return. In this particular use case, it is demonstrated the different paths a user may take from there whether it be asking a follow-up question or something more.
This user flow is similar to the above user flow for AA except that we are pulling directly from the Chegg archives. In this user flow, we have high confidence that the result that surfaced to the user is an exact match. For this particular use case, however, we take a slightly different approach and allow the user to also browse other similar results right then and there using a carousel. The reasoning in a slight change for this use case is that we know users enjoy browsing other results to compare and contrast and sometimes practice. Also, from result to result, there is variation and sometimes added benefits may manifest through the variation.
In testing MTC through user cohorts, the release plan would be at 5% of users. At each 5% increment, we will measure for do no harm to search volumes and consumption of exact match coverages.
The table below is measure of MTC at 20%. Here we see stable and increasing search volumes and consumption of exact match coverages, a positive indication that users are taking well to the MTC implementation and utilizing it at it’s fullest potential.
