UX Designer, UX Researcher on a team of 5 members
Research-related: competitive analysis, domain research, user interviews, remote usability testing
Design-related: user journey mapping, persona, wireframing, prototyping
May 2020 - Aug 2020
Capstone Project @IBM
As part of the graduate curriculum from the Master's of Digital Experience Innovation program, we partnered with IBM Cognos Analytics in a team of 6 to complete a capstone project that was focused on improving the onboarding experience for B2B products in the support of AI technology. We followed the agile workflow and met with the IBM mentors on a weekly basis, updating sprint deliverables and improving the research and design process based on the mentors' feedback.
B2B products are normally complicated with a wide variety of capabilities. They often require technical or business knowledge in order for their customers to become active, productive users of the product. The journey from Discover, Try, and Buy to active usage is therefore complex, time consuming, and varies by user types. If the users are not able to easily use a given product or service during the onboarding process, they will be likely to give up early and cannot unlock the product's potential benefits.
How might we improve user onboarding journey with AI technology that allows customization to different skill levels and learning styles?
Our goal was to research the best practices and leverage Augmented Intelligence (AI) technology to help improve the onboarding experience and to attract and retain new users in a Business to Business (B2B) context.
Determine the Domain
In this project, we chose eCommerce as our focused domain to explore the problems as this project was not limited to IBM technologies/products or their competitors. After discussing with the team, we all agreed that eCommerce would be a good B2B fileld to focus on as due to Covid-19, many people are looking to digitalize their business and setting up an online store would be in a huge demand. With more people became interested in the field, the more likely we would be able to recruit them for research purposes.
- Led user research by defining the research questions and engaged the team with research activities such as literature review and domain research.
- Conducted 3 interviews as a note-taker and synthesized 12 other interview feedback to produce meaningful insights that helped identify user problems.
- Used Mural and facilitated design workshops and coded interview data into themes for deeper insights.
- Created wireframes and medium-fidelity prototypes to communicate design solutions and gain team feedback.
- Created usability testing protocol and assisted with qualitative data analysis for 10 usability testings.
Gamification that Drives User Incentives
Users can earn reward points during the onboarding process that could be used to extend trials or purchase apps upon activation of plan. This feature will help drive users' incentives and encourage users to complete essential onboarding steps.
Personalized Customer Service and Voice User Interface
Our AI assistant Shoppie will monitor users' movement based on onscreen activities to quickly understand users' needs and attempt to solve the problem in an automated way. Users can also switch to voice mode to directly talk with Shoppie and let it take control of store setup process.
Data Collection for Optimized Customization Result
Users answer a series of simple questions to obtain the most suitable template that works best for their goals and business needs.
Once we identified the domain, we focused on asking the below questions to help understand the industry landscape and formulate interview questions.
- Why is onboarding important to users?
- What are some of the successful onboarding case studies in eCommerce and why are they successful?
- How do we build contextual communication to effectively encourage onboarding actions?
In addition, we also used comparative analysis to learn about existing onboarding flows across eCommerce platforms. This practice allowed us to build interview questions that are specific and targeted to the industry, as well as learning from the current successful onboarding tactics.
In order to build empathy and stay in touch with users' needs, we interviewed 15 sponsored users to gain feedback about their onboarding journey using different eCommerce platforms.
- A quick and easy way to collect direct feedback from users.
- To understand user pain points in a contextual inquiry which helps define the problem statement.
- Prepare initial user data for synthesis and to inform Persona and Journey Map.
I first created a screener to help identify our targeted users. We aimed to look for participants who have experience of using at least one eCommerce platform. The participants were mainly recruited from the professional network such as LinkedIn and social media platforms such as Facebook and Slack groups. These participants represented a diverse demographics, from small business owners to individuals with side hustles. This was to make sure the user data we collected was less biased with high reliability.
The interview took 45 minutes to complete and covered questions from why and how users choose a specific eCommerce platform to what benefits and challenges they discovered during their onboarding process. Below were some example interview questions that we asked participants. For full details of the interview questions, please click here.
Affinity Diagram - Categorized Information in Themes
We used coding method to extract key information from interview transcripts and categorized users' problems under different themes.
We learned that the problems were mainly coming from the two themes: management & operations and customer experience. We focused on these two categories and delved deeper into the affinity diagram by gathering some quantitative data to understand the statistical distribution over the sample. Below are some of our key findings:
Based on the interview, we concluded the following insights:
- As users are different in skills and experience, the biggest challenge lies in the knowledge gap between users and the eCommerce platform.
- Onboarding should inspire interests of discovery. Users like to be empowered and involved in self-directed learning where they have flexibility and control to personalize features.
- Users rely on trusted and timely support yet this service wasn't well achieved with current resources.
- Onboarding process is essentially a learning journey for users.
Persona - Understanding the Targeted Users
To reflect the differences in skill and experience from the users, we made a Primary Persona to represent users with limited experience and a Secondary Persona for users with intermediate to advanced level of knowledge.
We decided to focus on Primary User Melody throughout our design process because the majority of our users were novice to intermediate users and if a design solution works for them, it would likely work for a more experienced user with similar needs as well.
Left: Primary Persona. Right: Secondary Persona
- To showcase a representation of targeted users with similar behavior patterns and goals.
- To make sure the assumptions and knowledge about user needs are aligned between team members and stakeholders.
A user journey map was also created to demonstrate the pain points through different stages of onboarding process. We identified six essential stages: Discover, Register, Setup, Trial, Evaluate and Buy.
This method helped us consolidate qualitative data thoroughly and find behavior patterns and common goals across different scenarios.
Why Journey Map?
- Provide a strong and visual narrative that outlines targeted user's behavior at each stage of onboarding process.
- Identify key areas where users encounter negative emotions to inform new desin opportunities.
- An efficient way to assign ownership for team members to work on specific touchpoints.
Facilitating Various Design Workshops
Partnered with IBM Cognos Analytics team, we ran 5 design workshops to brainstorm ideas and solutions that were tied to user needs.
Based on the research insights and brainstorming results, our team came up with a variety of ideas and wegrouped them into themes again. We then used dot-voting method to determine the best potential design ideas to move forward with.
In the end, we decided to focus on 4 major areas to improve the onboarding experience, which are customization capability, chatbot service, voice AI and gamification tactics.
I created the below sketches to visualize one of the user flows in registration and customization process. I also drew a landing page that showcases the entry-point of onboarding.
After discussion with the team, I iterated on my sketches and created medium-fidelity prototype that covers the steps from registering and choosing a theme during the onboarding process. To let users choose whether they would like to go over the personalization questions at their own pace, we let users build a sense of control of their own onboarding while providing tailored experience.
To validate our design, we conducted unmoderated usability testing sessions with 10 users
to validate our high fidelity prototype on InVision. We obtained participants' consent before conducting the testing and provided instructions for 5 activities
that they would perform. These activities allow the users to go through each stage that we mapped out in the user journey map.
The users completed a survey
after the test so that we collected qualitative feedback to identify general problems on the usability of our design. After the test, we used the System Usability Scale (SUS) to measure how usable our design solution is and obtained an average SUS score of 73, meaning that our product is usable by most users
. We realized that the score was also greatly affected by one outlier with an extremely low score due to some technical problems of running the prototype on his computer. Besides that, all the other users were positive about using the product.
Another constraint of our usability testing was that we were not able to testing the Voice User Interface (VUI) function due to the nature of technology. Therefore, we received a mixed reactions to the voice feature. 40% of users gave a positive feedback but 60% were average to negative. The feedback was that voice has to be accurate in order to be effective. However,
one participant stated that he has carpal tunnel and thought the voice feature is a huge benefit.
Below were some improvements that we made to improve our design based on the user feedback.
- We pinned the header at the store page because it was not noticed by a number of users when they scrolled down the page.
- We added a back and forth button for the tour guide as well as a "Skip" button so users can view previous content when navigating the platform.
- As product pages got complicated, some user were confused on where to click on the page. We were asked to animate some buttons by giving them a glow or arrow to draw attention to nudge users.
We attempted to make a compelling story by using a video to showcase the design solutions. The demo video below walks through all the main user flows that have been tested and improved.
Concept Video - Voice User Interface (VUI)
In addition, we also made a concept video for the Voice User Interface (VUI) solution. We believe that asking simple questions and receiving instant feedback are very valuable, especially for users with disability as well as limited skills and knowledge. Having intuitive and interactive support from VUI will help save users' time on simple tasks and increase overall accessibility.
Designing B2B product onboarding experience could be challenging because unlike B2C products, B2B products are normally more complicated in interface design with more functionalities. If time were permitted, we would like to run a second usability test to measure the changes in users' satisfaction and retention rates before and after we made iterations. In addition, since we only explored eCommerce as our focused research domain, some user problems identified in the study could be less relevant than that in another industry. We would also like to do more lateral research to explore onboarding best practices across different industries in the future.