T12: Standardized Usability Questionnaires

Half Day Tutorial

James R. Lewis (short bio)
IBM Software Group, USA

Jeff Sauro (short bio)
measuringusability.com, USA

Objectives:

Standardized usability questionnaires are questionnaires designed for the assessment of perceived usability, typically with a specific set of questions presented in a specified order using a specified format with specific rules for producing scores based on the answers of respondents. For usability testing, standardized questionnaires are available for assessment of a product at the end of a study (post-study – for example, QUIS, SUMI, PSSUQ, SUS, and, most recently, UMUX and UMUX-LITE) and after each task in a study (post-task – for example, ASQ, Expectation Ratings, SEQ, SMEQ, and Usability Magnitude Estimation). Standardized questionnaires are also available for the general assessment of website usability (for example, WAMMI and SUPR-Q).

All of these questionnaires have undergone psychometric qualification, including assessment of reliability, validity, and sensitivity, making them valuable tools for usability practitioners. The purpose of this course is to provide an introduction to standardized usability measurement, an inventory of currently available instruments (including their psychometric properties) for after-task and after-study measurement, and discussion of recent research in the important characteristics of standardized usability questionnaires (including their interrelationships and statistical connections to other metrics).

At the end of the course, attendees will know about the key properties and uses for the currently available standardized usability questionnaires.

Content and Benefits:

The course starts with coverage of the basic psychometric properties of standardized measurement – reliability, validity, and sensitivity. The next topic is the inventory of post-study questionnaires (such as the QUIS, SUMI, PSSUQ, SUS, and, most recently, UMUX and UMUX-LITE) – descriptions of the instruments, summaries of their known psychometric properties, and key recent research findings. The SUS will receive special attention due to the relatively large amount of normative and usage data that has become available for it in the past five years. Following that we will cover the available after-task questions/questionnaires (such as ASQ, Expectation Ratings, SEQ, SMEQ, and Usability Magnitude Estimation), with descriptions, summaries, and recent research findings. We will end with a discussion of other standardized usability questionnaires of interest to the CHI community (e.g., some specifically for website evaluation and some key questionnaires from the marketing research literature, such as the TAM and Net Promoter Score).

Introduction to standardized usability measurement
1. What is a standardized questionnaire?
2. Advantages of standardized usability questionnaires
3. What standardized usability questionnaires are available?
4. Assessing the quality of standardized questionnaires
Post-study questionnaires
1. QUIS (Questionnaire for User Interaction Satisfaction)
2. SUMI (Software Usability Measurement Inventory)
3. PSSUQ (Post-Study System Usability Questionnaire)
4. SUS (System Usability Scale)
5. UMUX and UMUX-LITE (Usability Metric for User Experience)
6. Experimental comparisons of post-study questionnaires
Post-task questionnaires
1. ASQ (After-Scenario Questionnaire)
2. SEQ (Single Ease Question)
3. SMEQ (Subjective Mental Effort Questionnaire)
4. ER (Expectation Ratings)
5. UME (Usability Magnitude Estimation)
6. Experimental comparisons of post-task questionnaires
Questionnaires for assessing the perceived usability of websites
1. WAMMI (Website Analysis and Measurement Inventory)
2. SUPR-Q (Standardized Universal Percentile Rank Questionnaire)
3. Other questionnaires for assessing websites
Other questionnaires of interest
1. CSUQ (Computer System Usability Questionnaire)
2. USE (Usefulness, Satisfaction, and Ease of Use)
3. HQ (Hedonic Quality)
4. ACSI (American Customer Satisfaction Index)
5. NPS (Net Promoter Score)
6. CxPi (Forrester Customer Experience Index)
7. TAM (Technology Acceptance Model)
Wrapping up

Target Audience:

The course will likely be of interest to a wide variety of attendees, but will be especially useful to those usability practitioners and HCI researchers who currently use or plan to use standardized usability questionnaires.

Relevant links:

Bio Sketch of Presenters:

Jeff and Jim are co-authors of the 2012 book from Morgan-Kaufmann, "Quantifying the User Experience: Practical Statistics for User Research" and its companion book, "Excel and R Companion to ‘Quantifying the User Experience: Practical Statistics for User Research’: Rapid Answers to over 100 Examples and Exercises".

Dr. James R. (Jim) Lewis Ph.D., CHFP
Jim graduated with an M.A. in Engineering Psychology in 1982 from New Mexico State University, and received his Ph.D. in Experimental Psychology (Psycholinguistics) from Florida Atlantic University in 1996. He has worked as a human factors engineer and usability practitioner at IBM since 1981. He has published influential research on the measurement of usability satisfaction, use of confidence intervals, and sample size estimation for usability studies. He is on the editorial board of the International Journal of Human-Computer Interaction and the Journal of Usability Studies, and wrote the chapter on usability testing for the 3rd and 4th editions of the Handbook of Human Factors and Ergonomics (2006/2012). From 2004-2005 he chaired the Formative Usability Testing Metrics Workgroup for National Institute of Standards and Technology (NIST). He is a BCPE Certified Human Factors Professional, an IBM Master Inventor, author of "Practical Speech User Interface Design" (2011), was recently voted onto the board of the Association for Voice Interaction Design, and is a member of UPA and HFES.

Jeff Sauro
Jeff is a Usability Engineer and statistical analyst with over a decade of experience conducting quantitative usability and statistical analysis for Oracle, Intuit, PeopleSoft, PayPal, Sage Software and General Electric. Jeff has presented and taught courses at CHI, HCII, HFES and UPA. He was the guest editor for Interactions Magazine dedicated to Quantifying Usability. He holds a Masters from Stanford University from their school of Education specializing in teaching quantitative concepts. His teaching style is to work backwards from practical problems that the usability practitioner is likely to encounter when quantifying usability. From these problems he takes actual usability data and shows how to make better decisions with numbers.

Top of Page