Case Study:
How to Test a Champion
TLDR;
I led the design, coordination, analysis, and presentation of a 2 part character design study that spanned 1 month and hosted over 25 participants. My mixed methods research, in combination with solution-focused reporting, gave the Champion’s team the information they needed to confirm their direction lock and continue to improve usability for players before release.
Context
Collaborated with the Champion’s team at Riot Games to stress test upcoming character releases, aiming to provide actionable feedback to meet our passionate player’s expectations.
Product: Testing a new character for League of Legends (League), a team-based strategy game where two teams of five powerful champions face off to destroy the other's base.
Background: Each player chooses a unique character they want to play, also known as their champion (champ). The Champion’s team designs between 2-4 new champions each year. This lab was testing two new champion designs for compatibility with League’s base game Summoner’s Rift (SR).
Users and Audience
Primary stakeholders
Champion’s team (design, art, narrative, production, publishing divisions)
General Audience
League of Legends players that play SR
Aurora Audience
Mage players comfortable with playing mid, top, or support roles
Skarner Audience
Players who play Jungle roles, players who previous chose Skarner as their main champion
Scope and Constraints
Time Frame was 1 month
2 weeks of discovery and goal alignment
Lab took place in a single day over 6 continuous hours
Analysis for both lab results took approximately 1 work week each (2 weeks total)
Results were presented in a staggered pace over two weeks, with Aurora presented first and Skarner presented second
My Responsibilities
Pulling data for recruiting, assigning positions for testers based on rank and lane, creation and deployment of 4 surveys, creation and presentation of onboarding PowerPoint, set up of lab computers (including OBS and testing environment), moderation of the lab (including technical troubleshooting), note taking during lab for future analysis, analysis of lab results (including survey qualitative and quantitative data, review of lab VODS, and study of previous notes taken), compilation of analysis into PowerPoint for stakeholders, presentation and sharing of analysis to stakeholders.
Team Details
Total team size was 5 members: 1 assistant researcher who helped in lab moderation, 2 lab assistants to help with check in, and 1 team lead to review work before approval.
Problem
League’s community often gives feedback that new champions negatively affect game balance and are not representative of diversity communities. Our job was to measure the sentiment of those metrics for the Champion’s team, and provide them with creative solutions to address any concerns expressed by players.
Business Impact: Previously Skarner had not tested well with players, and it was important to stakeholders that they won over the hearts of his dedicated fanbase with this update. Aurora was meant to represent a disenfranchised community, and her gameplay was an important part of reinforcing her powerful narrative so affected players felt seen and heard.
The two champs being tested in this lab were:
Aurora
User Objective: Aurora is a new champion meant to be flexible to multiple different playstyles. She is meant to feel powerful, relatable, and kindhearted.
Business Objectives:
Players are invested in Aurora’s quest to save corrupted spirits.
Observe whether players prefer aggressive or passive play.
Metrics of Success:
Players feel invested in Aurora’s storyline and gameplay, especially in relation to her saving corrupted spirits
Aurora feels very flexible in her playstyle, and is not too strong nor too weak
Skarner
User Objective: Skarner was an older champion who needed a gameplay and design update. Skarner is meant to feel like an ancient, intimidating monster with deep connections to the earth.
Business Objectives:
Players like the updates to Skarner’s story, design, and gameplay
Skarner’s gameplay goes well with his theme
Metrics of Success:
Updates to Skarner’s story, gameplay, and design feel like an improvement to his previous character
Gameplay resonates with the thematic of Skarner’s new character
Research
Methods
In order to collect data, the research team brings in 25+ players to test the new champions over 4 SR games (each 30-45 minutes long). Players are given 2 mixed method surveys, one thematic and one gameplay, and additional context is gathered through direct observation of gameplay. Our mixed method surveys and gameplay observation revealed unanticipated usability pain points, highlighting high risk development areas for the Champion’s team.
User Insights
Gameplay and thematic analysis showed that both champions had overall positive sentiment, however players did note some usability issues related to visual clarity with indicators and ability cast.
Impact on Project
Champion’s team applied research team’s recommendations based on player feedback, revamping visuals and redesigning abilities (such as Aurora’s ultimate) to better communicate their impact.
Impact on Skarner's design
Although earlier lab results had shown significant worry about Skarner’s new design not feeling like an upgrade, these results relieved those fears. These results reaffirmed the current direction of the team’s design, and provided further feedback on interactions needed for visual clarity.
Impact on Aurora's design
Aurora’s connection to the spirit realm was clear in her narrative, but not in her kit. This information, along with usability feedback related to her visual clarity, pinpointed iteration points for the gameplay to work on before her next testing round.
Lessons learned
Research reports can often feel like a judgment on a development team’s work as results are often given in front of their stakeholders. In an effort to reinforce the collaborative elements of labs, I aim to bring developers into the results process much earlier in future labs. In doing so, developers will have ample time to analyze potential issues to address, and will be prepared for stakeholder questions during the official presentation.
Areas for future improvement
Unfortunately there were some technical issues with the build we were using to test with players. Because of this, several players had far more limited time to try out the character abilities than anticipated. Going forward, we have prioritized a much closer connection with the Quality Assurance team to make sure builds are completely locked before playtesting.