January 2023 - May 2023

Methodological Process

We began this investigation by asking:

Q1: How can we increase the usability and accessibility of the information provided on the Michigan Safer Opioid Prescribing Toolkit for medical providers and non-target users?

To conduct this investigation, we recruited target and non-target users for 2/5 methodologies employed in this study. Target users included primary care providers, patients undergoing treatment for pain management, and patients’ families. A combination of user-centered methods were employed to answer these research questions. These methods included:

• Designing an interaction map to develop a better understanding of the toolkit, primary user functions, and the overall navigation of this interface.

• Recruiting and interviewing users about their vision for optimal functionality, including their goals and personal satisfaction with the toolkit, as well as inhibitors of effective use, and recommendations for improvement.

• Stakeholder interviews to understand core elements and features of the toolkit, stakeholder concerns related to the usability and accessibility of the toolkit and intended target users and their needs.

• A thematic analysis was done for all interviews using a combination of in-person and virtual affinity walls to organize coded themes.

Phase 1: Recruitment

To evaluate the usability of the Michigan Safer Opioid Prescribing Toolkit, a total of 4 usability tests were conducted.

An initial email was sent to a list of users provided by the stakeholders of this project. In this initial email, we provided a brief description of this research, and explicitly stated that participation is voluntary.

A total of 2 target users and 3 non-target users responded to our email and were then asked to sign a consent form to participate in a virtual usability test.

*User self-identification on the homepage via any of the three options named above is a first step to ensure that users can focus on information that is relevant to them to alleviate time pressures, information overload, and feelings of fatigue, frustration, and confusion.

2. Future research is needed to evaluate the usability of the Michigan Safer Opioid Prescribing Toolkit using the survey we designed.

 3. To increase ongoing user feedback, a survey link should auto-populate on the homepage with an option to decline.

Community Resources and Add-Ons to improve DEI

1. Adding more interactive features such as tutorial videos (an updated “How To” video for navigating through the toolkit, for example), testimonials, online forums, and discussion boards are great ways to connect with users on emotional and collaborative levels. This also adds additional accessibility features by appealing to diverse learning styles and preferences for consuming large amounts of information. The “How-To” video should also be moved towards the top of the homepage to increase visibility and accessibility.

2.  More data visualizations and images should be added to balance paragraph text.

3. Community resources should be located in a more accessible location. “Linkage to Services” is also not as intuitive as 'Community Resources.’ Consider updating this heading.

4. A 24-hour hotline(s) link for addiction/crisis support should be added at the top of the homepage for anyone in need of immediate support.

Survey

We then designed a 25-item online survey that asked:

1. What social demographics are associated with toolkit use, and what are the most common uses?

2. How important to the user experience is it to access information quickly, and is it important for information to be available in a centralized location? To what extent does the toolkit fulfill these goals?

3. Do users consider this a “one-stop-shop” resource?

4. We also measured user satisfaction, attitudes about the usability and accessibility of the toolkit, and whether users rely on alternative resources to supplement their use and/or rely on as a replacement resource.

Each survey item was carefully worded and ordered based on stakeholder goals and overarching research questions.

Competitive Evaluation

With each pass, we documented heuristic violations and compliancies on a shared Google sheet, as well as the screen(s)/location of the issues observed, the heuristic used and how it explains the issue, and the severity of the issue. Screenshots of the most severe issues were additionally gathered. After individual evaluations were completed, we compared findings and a list of key priorities was compiled from the full list of issues documented. This was based on the severity of the issues found and the extent these issues would affect the user experiences of target and non-target users. 

Next we conducted a competitive evaluation to assess the usability of the Michigan Safer Opioid Prescribing Toolkit by comparing with five alternative resources. This evaluation measured on five criteria: functionality, target population, interaction style, usability, and aesthetics.

Based on this competitor analysis, the Michigan toolkit can improve its aesthetics, usability, and the functions available to users. We found that the Michigan toolkit does not include enough images, nor does it reach an expansive audience, relative to competitor resources. Additionally, most websites use a menu-based interaction style, but this may not be the most feasible, given the breadth of information available on the Michigan toolkit.

Heuristic Evaluation

Next we conducted a Heuristic evaluation that used Nielson’s 10 Usability Heuristics to evaluate the usability of the Michigan Safer Opioid Prescribing Toolkit.

As a team, we operationalized and documented criteria questions that would be used to guide individual evaluations across all 10 heuristics. Individual evaluations entailed multiple passes, in which we first conducted an unstructured evaluation of the full website and noted areas in need of improvement. The next pass focused on identifying and documenting areas of the website that either violated or supported heuristics 1-3 (visibility of system status, match between system and real world, and user control and freedom). The next pass followed the same procedure, except focused on heuristics 4-6 (consistency and standards, error prevention, and recognition rather than recall), and finally, heuristics 7-10 (flexibility and efficiency of use, aesthetic and minimalist design, help user recognize, diagnose, and recover from errors, and help and documentation).

Phase 3: Usability Test

The official usability test consisted of 5 tasks that participants were asked to complete while we observed and noted their activity. Each task was designed to assess whether participants were able to locate specific resources that we confirmed with stakeholders as “important resources users should be able to locate with ease.” Examples of these resources include the contact form on the website, the survey link to provide user feedback, and specific resources that would be important for different users (physicians versus educators, for example) to locate. As a team, we also came up with a list of success/failure criteria and an estimated time of completion per task that would be used to evaluate each user’s performance and the overall usability/accessibility of the toolkit.

Phase 4: Post-Test

At the end of each usability test, we asked users to answer a post-questionnaire that included the same series of questions they answered in the pre-questionnaire. This was done to assess and measure the stability of users’ confidence levels, attitudes, and impressions of the toolkit. At the end of completing the post-questionnaire, we debriefed users and thanked them for their participation.

Summary of Recommendations Based on Findings

Expanding Toolkit Use

1. We recommend expanding the target user base by including sections for non-prescribing users (social workers/therapists, friends/family members, paramedics, law enforcement, and others) who interact closely with patients for follow-up care. We have three options below for how to best achieve this:

• Option A: Dividing navigation bar sections of the homepage by user-type (not by topic). Topics pertinent to user-type can then be added within each user subpage

• Option B: Add a pop-up option on the homepage where users are asked to self-identify their user type. Upon selecting of the button that best identifies user type (patient, therapist/social worker, healthcare professional, friends/family, educators, policymakers, etc.), users would then be redirected to a new page that would contain menu bars with information specific to their needs.

• Option C: Minimal changes to navigation bar. Instead, replace “Topic” cards/buttons with “User Type” cards/buttons and add these to the homepage. Users can then have the option to navigate through the website based on “User-Type” or continue to use the navigation bar to navigate by “Topic.” 

Phase 2: Pre-Testing

Since an experimental element was added to this research, participants were asked to familiarize themselves with the toolkit prior to their scheduled usability test so that they would be prepared to answer a pre- and post-questionnaire. A pre-questionnaire was distributed to each participant prior to completing their usability test in order to gather a general sense of each participant’s confidence level for using the toolkit, as well as their attitudes and impressions of the toolkit. Users were not informed that pre-and-post questionnaires would be administered, given the minimal to no risk for participation in this research.

Web Design/Development

1.  A search bar should be added to the top of the homepage to help users quickly locate the information they are seeking.

2.  Ensure that the yellow banners at the top of each page are consistently and fully visible and not obstructed by the navigation bar. The banner should also remain visible when users scroll to the bottom of the page to indicate what page users are currently on.

3.  A thorough analysis of the website should be conducted to test for broken links. For example, we suggest revising the "Contact form" link to take users directly to the form (not having to click on “Contact Form” twice to get to this page). Additionally, the “Submit” link on the “Contact Us” page should allow users to provide minimal information (name, email address, and inquiry), without requiring additional personal information.