My Asthma Pal

USABILITY TESTING & EVALUATION

Project Brief

My Asthma Pal is a mobile application that is designed to help children with asthma and their caregivers to track symptoms, manage treatment, and maintain a treatment plan. Since the application launch, there have been numerous complaints from users concerning usability. Our team was hired to evaluate and improve the application functionality.

Our team conducted a usability assessment using test objectives to evaluate the application thoroughly. These objectives allowed our team to collect data and identify problems by observing test participants work through tasks using the app. Each team member acted as both moderator and observer; we all took notes of the issues and performed our usability assessment. We then compiled all our test results and identified problems and subsequent findings for each test objective. Finally, we made a recommendation corresponding for each finding, to help improve the overall usability of the application.

Team objective

Our team aim is to provide better functionality than other asthma tracking applications. We evaluated its design to identify potential areas of improvement. Since its launch, there have been numerous complaints about its usability problems from its users in which we aimed to confirm and address in our usability tests. We tested the application and found several usability problems including account creation, navigational, and error reporting issues.

Heuristic Evaluation

To perform the heuristic evaluation, each member of the team first completed the process individually. We started by performing an initial “walkthrough” to get familiar with the application. Then we each navigated the app a second time, identifying usability problems based on Jakob Nielsen’s Ten Usability Heuristics for User Interface Design. For each problem we encountered, we identified which heuristic it violated, proposed a recommendation as to how to fix the problem, and rated the severity of the problem on a scale from 0 to 4 as explained below.

0: Do not agree that this is a usability problem

1: Cosmetic problem

2: Minor usability problem

3: Major usability problem; important to fix

4: Usability catastrophe; imperative to fix

We combined our evaluations into one comprehensive list by first combining redundant observations - problems more than one member of our team included in their preliminary evaluation. We eliminated several heuristic violations that we felt were not severe or would not be a problem for our target users. Of the remaining observations, we assigned priority levels based on the severity of the violation, listing them in descending order of most-critical to change to least critical. We found that none of the problems we identified were of Level 4 severity. However, over half of the issues we observed were rated Level 3, and we believe they would be essential to fix to prevent users from becoming frustrated and abandoning the application. More problems violated the heuristic of User Control and Freedom than any other heuristic. Below you will find our research results identifying several severe heuristic violations :

Cognitive Walkthrough

The purpose of this cognitive walkthrough is to focus on understanding the system’s learnability for new or infrequent users. The below combined cognitive walkthrough mainly focuses on four core tasks that are fundamental to the app and used across different tasks. The cognitive walkthrough emphasizes solutions for specific problems encountered in the action sequence of the tasks. The analyses draw attention to aspects of design, flexibility, and efficiency of use and ease of error recovery.

Problem Statement and Test Objectives

Based on our team heuristic evaluation and cognitive walkthrough, we identified specific usability problems that the users were having with the application. We plan to focus on these usability issues and evaluate how we can improve the users use of the app.

We used the data collected to identify usability problems such as does the user goals of the application match the mental model of its user types. Also, to find out if the interface had design inconsistencies and usability concerns within the content areas, such as, failure performing a common task and navigational issues. We also tested to find out how easy our users understood the application and their failures to locate the desired task and information.

Usability Test

Our primary objective for this usability test and subsequent report is to gather insightful and clean data using our evaluation measures and detailed note taking to determine the usability of the My Asthma Pal application. Our evaluation measures encompass both quantitative and qualitative data enabling us to test and evaluate the effectiveness and experience of using this application. We aim to create a concise report using tables to display our metrics and visual examples to illustrate the problem areas. We will record data findings through observation and interviews and then analyze our data. We will include the discussion and implications of the results that are relevant to specific tasks based on our evaluation measures. We will also include recommendations that will help reviewers improve the application.

Usability Test participants

This study included a total of five participants: one pilot test participant - recruited from our class, and four additional participants, recruited independently by each of our four team members, from outside our class. All participants were required to be over the age of 18 and voluntarily participated in this study. Two of our participants were in the 18-30 age range, one was in the 31-40 range, and one was 50+ years old. As this application is available only in written English, we made sure all participants were capable of reading the English language without a translator or text reader for the visually impaired. All of the participants in this study owned a smartphone that they used on a daily basis.

Because we were interested to see how practical this application is to a novice user, we did not require that participants have any previous experience any asthma-tracking applications. All of our participants indicated on the questionnaire that they had never used an asthma tracking application before. We were also interested to see if the asthma-specific language used within the app had any effect on usability. As such, we do not require that participants be diagnosed or familiar with asthma. Only one of our participants indicated that they had asthma or were responsible for the care of someone with asthma.

Test design

This study examines the usability of the My Asthma Pal mobile application, specifically the processes of setting up a new account, recording a new Asthma Control Test (ACT), and adding a medication. Each of these tasks is detailed in the following section entitled Task List. We collected quantitative data such as success and error rates of task completion, as well as qualitative data such as participants’ thoughts and feelings about the application and its functions.

Task List

Due to the multifunctionality of the My Asthma Pal application, and the time limit of approximately 15 minutes per task set, the three tasks we decided to evaluate are critical to effectively using the app for its intended purposes. The tasks were identical for all participants, who all participated from the “Parent/Caregiver” user perspective. However, tasks B and C were conducted in alternating order to prevent bias as a result of participant fatigue or familiarization with the application.

A. Create an account: the participant will open the application as a first-time user and attempt to set up a profile as a Parent/Caregiver

B. New asthma control test: the participant will attempt to locate and record a new asthma control test within the application

C. Add a new medication: the participant will attempt to locate their medication list and add medication and relevant information as provided by the moderator

Below is a breakdown of which test objectives each task addressed, and which data points were used to address our questions.

Evaluation measures

Our test is looking to answer the following questions about the usability of My Asthma Pal:

How easily and successfully can the user create an account?
How easily can the users navigate through the App successfully?
Do the users know to create a New Asthma Control Test?
How easily can the users add new medication?
How easy can the users read and understand their results?
How satisfied are the users with the task of adding a new medication?
How satisfied are the users with creating a new account?

Each of our quantitative measures helped us evaluate the overall success and learnability of each task our participants were asked to perform (Create New Login, Create a New Asthma Control Test, and Add New Medication). The measure of incorrect selections is used as a direct measure of the ability for the user to navigate through the app successfully.

Quantitative Measures

Number and percentage of tasks completed correctly without prompting
Number of prompts from the moderator (only if asked by the participant)
Number of errors encountered
Number or incorrect selections (times the user has to backtrack)

Our qualitative measures are aimed at evaluating the satisfaction and understanding the participants experienced while completing the tasks.

Qualitative Measures

Participant rated difficulty level for each task (in Log Sheet)
Participant response in expected versus actual task completion steps (in Log Sheet)
Observation Notes about participants think aloud process (in Log Sheet)
Usefulness of application for symptoms management
Ease of use overall

Results

After each evaluation assessment was conducted, the data was gathered from our notes taken in the moderator guide, evaluation recordings, and participant questionnaires. The quantitative data was tallied qualitative data inductively coded leading to the overall results of each test objective (as seen by the table below).

The task that caused the most confusion and errors was creating a new account leaving participant to be somewhat frustrated with the experience of just trying to log in. All of the participants needed to be prompted to complete the task, and all encountered multiple errors while working through the task. The problems that arose from this task include a lack of warning and error messages or requirements and a hard to find “Create New Account” link.

Half of the participants encountered errors while trying to add a new medication; this was again due to lack of error messages when missing input information. Navigating through the application also confused half of the users, this was caused by both not recognizing the menu icon and the menu opening unexpectedly while scrolling.

The two test objectives that did not present any significant usability problems was creating a new Asthma Control Test (ACT) and users reading and understanding their results after completing the tasks.

Findings and Recommendations

After consolidating our results as seen above, we then created findings that encompass both the quantitative and qualitative data and address the exposed problems we found through the usability evaluations. Each finding corresponds to one of our test objectives and is addressed by a recommendation to aid in solving these usability problems (see table below).

A majority of our findings address the violations of the heuristic; help users recognize, diagnose, and recover from errors. The tasks that do not follow this heuristic include “create a new account” and “add a new medication.” In each case, we found a lack of error messages and warnings left participants unable to figure out what input is missing or doesn't meet the requirements. To fulfill this heuristic, we recommend providing these appropriate error messages.

Another heuristic violation being addressed in our findings is the flexibility and efficiency of use. Upon logging in and adding a new medication, the participants had to tap of the keyboard with each consecutive input instead of being able to use an integrated keyboard to jump from input to input and submit within the keyboard. This problem causes the user many more clicks, wasting time and causing frustration. We recommend that the keyboard is integrated with the app allowing users to tab and submit without having to tap out of the keyboard.

The remainder of the findings has to do with the participants not being able to locate the necessary links/buttons and navigate the app successfully. We recommend creating more recognizable and larger buttons for a consistent experience and reduction of error across all of the tasks. There is also a bug present that unexpectedly exposes the menu when scrolling that we recommend fixing.