Armoire Search and Discovery Study
Usability Study
Conducted a remote moderated usability study on the search and discovery functionality of Armoire, a women’s clothing rental company.
Duration: 10 weeks
Team: Anastasia Lucas, Carter Hearne, Emily Rowland
Context: HCDE 517 - Usability Studies
My Role: UX Researcher
I performed a heuristic analysis and co-developed the usability testing protocol. I also moderated usability testing sessions, analyzed qualitative and quantitative data, and presented design recommendations to stakeholders at Armoire.
CONTEXT
Armoire is a women’s clothing rental startup based in Seattle that gives its members access to a rotating closet of curated, contemporary styles.
A member of the design team at Armoire presented my team with the opportunity to evaluate the search and discovery function on Armoire’s desktop site. My team and I developed a usability testing plan and carried it out via remote moderated usability testing sessions with existing users.
RESEARCH OBJECTIVES + METRICS
Our study was guided by the following research questions:
How efficiently do participants find a desired clothing item from Armoire’s home page?
Can users find “curated for me”, “my favorites”, and “upcoming styles”?
Can users “favorite” an item? Do they understand the favoriting process?
Are participants satisfied with their search results?
We translated our initial research questions into more detailed objectives that could be analyzed using qualitative and quantitative data collected during our usability study.
USERS
Armoire’s primary target user group is established professional women between 30 and 50 years old, who are looking for great clothes to wear every day. We used Armoire’s user persona, Vanessa, to guide our research.
HEURISTIC EVALUATION
To begin the study, I completed a heuristic analysis of the website using Jakob Nielsen’s 10 Heuristics and 4-Step Severity Rating Scale. The other members of my team also performed individual heuristic evaluations, and our team compiled our individual analyses to determine the severity of existing usability issues.
Jakob Nielsen’s 10 Heuristics:
Visibility of system status
Match between system and real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from errors
Help and documentation
Jakob Nielsen’s 4-step Severity Scale:
0 = I don’t agree that this is a usability problem at all
1 = Cosmetic problem only: need not be fixed unless extra time is available on project
2 = Minor usability problem: fixing this should be given low priority
3 = Major usability problem: important to fix, so should be given high priority
4 = Usability catastrophe: imperative to fix this before product can be released
REMOTE MODERATED USABILITY TESTING
We then met with 8 existing Armoire users on Zoom to learn more about their experiences with Armoire’s item discovery navigation, including the site’s search functionality and browsing options.
Study Participants: Sample Size and Target Users
A screener survey was used to select usability study participants from a pool of existing Armoire users. I worked with my team to develop this survey to ensure that the participants we interviewed accurately reflected the experiences of Armoire’s users.
Our team chose to focus on new/slightly experienced subscribers to gain insight into the user experience of someone new to the product. We aimed to recruit current subscribers with accounts less than one year old.
Additionally, the participants we selected to interview held a variety of different subscription plans. The breakdown of our participants was reflective of the user statistics of Armoire’s current subscription plans: 4-item capsule, 7-item capsule, and unlimited.
We wanted to meet with between 5 and 8 participants. We were aiming for 8 participants, and we were successful in meeting this goal.
Participants were offered an extra item in their Armoire closet as compensation for their time and contribution to our research.
Usability Study Script
Our team used a detailed script to conduct our remote usability sessions. I co-wrote this script to include instructions to be read to the participant, instructions for the moderators, and open-ended and follow up questions.
Participants were asked to speak aloud their thoughts as they moved through three tasks: finding an item using search, browsing items and “favoriting” an item, and rating upcoming styles.
Usability Study Tasks
Participants were asked to work through the following tasks and subtasks:
1: Using the Search Function
Find and type in the search bar for an item by color or brand name (e.g. black cardigan)
Use filters to hone in on items
Add an item that meets the search criteria to add to the case
2: Browse Categories & Using “Favorites”
Using the “curated for me” section, add an item to your favorites.
Navigate to the “curated for me” area
Add an item from the “curated for me” carousel to user’s favorites
Add an item from your favorites to your case
Access favorites carousel or page
3: Rate Upcoming Styles
Navigate to the “Upcoming Styles” section.
Rate upcoming styles.
Data collection
For each usability study session, one team member acted as the moderator, and another team member took notes. I both moderated and took notes for multiple usability sessions.
After the completion of each task, users were asked to navigate to a google form, where they were asked Likert scale questions about the task they just completed.
Likert scale questions included ease / difficulty for each task, and satisfaction / dissatisfaction with the time it took to complete the task, as well as user satisfaction / dissatisfaction with the search results.
DATA ANALYSIS
Qualitative Data
Armoire users’ insights and opinions were collected through open-ended questions during the usability study sessions in a google spreadsheet.
After all usability sessions were completed, I worked with my team to compile and evaluate qualitative data using affinity diagramming in Miro .
Note-taking google spreadsheet
Affinity Diagramming User Insights in Miro
Quantitative Data
System Usability Scale (SUS) data and Net Promoter Scale (NPS) data were collected via a google survey that was provided throughout the usability testing sessions. SUS, NPS, and overall success/fail rates of specific tasks were compiled and analyzed using google sheets.
In this study, Armoire received a System Usability Scale (SUS) rating of 82/100, and a Net Promoter Score (NPS) Rating of +62.5.
Task Success and Fail Rates
SUS Data and Calculations
FINDINGS + DESIGN RECOMMENDATIONS
Final Report
I co-wrote a final report that compiled the methods, findings, and design recommendations used and found throughout the study. My team and I submitted the report in our class and presented it to Armoire stakeholders.
The full report can be viewed here.
Findings
Our team found overarching areas of success and areas of improvement, and provided overall recommendations summarized in the following slide:
We identified 9 specific areas for improvement…
Algorithm Logic Ambiguity
No “View All” Option
“Upcoming Styles” not Saving
Difficulty Finding Filters
Limited Filter Options
Difficulty Filtering Favorites
Search Usability
Locating “Upcoming Styles“
Disliking / Un-hearting Effect
…and tagged our findings into 5 categories:
Efficiently Completing Tasks
Confidence in Search
Learnability
Satisfaction
Usability
Each finding was published in our final report in the same format: each finding included categorization with tags, a heuristic evaluation severity rating, and the number of users impacted. Quotes or videos from user interviews were included, and videos of users navigating the website. Details for all findings are available in the final report here.
Examples of findings slides:
Design Recommendations
Each finding included design recommendations specific to that finding that were informed by user experiences and insights we gathered in the usability sessions.
For example, the “No ‘View All’ Option” finding included the suggestion to “indicate to users that they are viewing a selection of clothing, or all clothing.” This suggestion was justified by 4/8 of our usability study participants expressing that they would like to have access to that feature.
NEXT STEPS
Given more time on this project, I would work with the design team at Armoire to discuss the feasibility of our team’s design recommendations, then prototype and usability test the suggested feature changes within the site.
REFLECTION
This was a fun project that allowed me to flex my research skills! I also greatly enjoyed working with Armoire - it’s so exciting to support a company that empowers women and focuses on sustainability in the way that Armoire does. I felt lucky to have a great team to work with, the support of an incredibly helpful connection at Armoire (Thank you, Megan!) and excited users who truly cared about the product to learn from.