Milestone V: Evaluation


Of the three participants who volunteered, one test was held remotely from his home, the second was held in person at her home and the third one was conducted in a cafe where he frequents. Additionally, the participants interacted with the prototype using their own smartphones, giving them familiarity with the environment and the device. There are advantages and disadvantages to testing at the user’s location, however. One advantage was the participants felt comfortable in their own setting but the disadvantage was adjusting to the logistics there, such as noises and other distractions. Regardless of where the evaluation occurred I took confident in the 80% rule, which states that most usability issues are the same across locations, according to the “Handbook of Usability Testing” by Jeffrey Rubin and Dana Chisnell.

Before the evaluation began I started with a test plan, which served as a roadmap that included goals for each task, questions I planned to ask and how I planned to answer them. This ensured I collected data consistently among participants. Here are the documents I generated:

Participants, Tasks and Results

Each participant brought unique a perspective to the evaluation because of their distinct backgrounds and previous experiences.

I designed four tasks and objectives for the participants to complete. Using, each user was randomly assigned a participant number and completed the test in different orders for counter balance.

Task One: Sync Page

Objective: Do users prefer to auto or manually sync their apps with MultiManager? Can the user sync manually without difficulty?

Scenario: A classmate/co-worker recommends trying MultiManager. The user creates an account, goes through the tutorial, which opens by default for first-time users and lands them on the Sync page.

Highlights: All users clicked "skip" to bypass the tutorial, preferring to learn the UI as they go. Also, despite the recommendation for "auto sync" participants #2 and #3 preferred to sync their apps manually. A fix I made to the Manual Sync, per user feedback, was to allow users to select multiple apps rather than one at a time from the 1st iteration. Lastly, I implemented functionality suggested from expert opinion, professor Karen Doty, who suggested I enable the system to recommend apps to sync based on prior activity.

Task Two: Modify/Change a Card

Objective: Do users understand how to view detail information for a category? How to edit a card? Is the content relevant?

Scenario: You want to replace one of the existing cards with a category of your choice.

Highlights: Although users found the content useful, they STRUGGLED to complete this task. The first iteration led to sighs and frustation as users had to flip the card, open the additional menu to edit a card. The second iteration reflects user feedback with swiping gestures to remove categories, added "undo" as a security to user error, added "+" icon to build a category, replaced boxes with icons to prevent confusion as a UI element and enabled ability to add category-related apps manually.

Task Three: View Today's Schedule by the Hour

Objective: Determine if users know what is happening on Personal Assistant page? Do users find the content helpful? Can users successfully access content within their scheduled day?

Scenario: It's 7:00 a.m., the alarm on your phone sounds. Based on your settings, the phone says, "you have new messages in Outlook, Gmail and Snapchat." Snippets from each message are read aloud. The system moves down to the next category scheduled between 7:00 a.m. - 8:00 a.m. This cycle repeats for each category as the day progresses.

Highlights: The concept of this page was well recieved. A bit of confusion still persisted in how the UI completely works. I think more interactions need to be added for better understanding. 

Task Four: Overall Usability

Objective: Assess users ability to recall where content lives, impression of experience and whether the UI was intuitive enough for them to learn how to use the app on their own.

Scenario: Imagine you sign in and here is where you are within the app . . .

Highlights: On average, participants scored the MultiManager app a 75.8 (SD=13.77) out of 100 on the System Usabililty Score (SUS) survey, rating it a “C”. Meaning the app is sufficient, however, improvements are still needed.​​​​​​​

Once all the tasks were completed, the participants answered post-test survey questions​​​ on the project as a whole. Here are the results:

Future Development

This latest iteration of MultiManager would return to users for more testing but I would present a higher fidelity version this time. Since I updated the UI per user recommendations I would add more functionality and less assistance from me as the moderator. In addition, I would introduce color, typography and other design elements. I think a richer experience for the user will give me more valuable feedback at this stage as I work closer to a release.

Looking forward, if the third iterative tests well and the system can achieve a SUS rating of "B" or better I would consider a soft release. Depending on the app's success, I would scale future iterations of the Personal Assistant to an Executive Assistant version that includes voice recognition, remote control of external hardware and augmented reality. The goal will remain the same, make people's lives easier by bringing them relevant content when they typically consume it instead of them searching for it.

The MultiManager app still needs improvements and is not ready to go to market. However, one major question was answered through this process, would smart phone owners use a smart app that learns their behaviors and adjusts to their preferences. The answer is an astounding yes.