Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study
November 30th, 2022
This study presents an experimental paradigm for a usability test combining subjective and objective evaluations. The objective evaluation adopted Neuro-Information-Systems (NeuroIS) methods, and the subjective evaluation adopted a usability questionnaire and a NASA-Task Load Index (NASA-TLX) scale.
This study introduces an experimental paradigm for a usability test of emerging technologies in management information systems. The usability test included both subjective and objective evaluations. Mobile fNIRS and eye tracking glasses were used for multimodal measurements solving the ecological validity problem of cognitive neuroscience tools used in real world behavior experiments.
Demonstrating the procedure will be Di Zhang, a master student from my laboratory. Yi Wang, a master student, will act as a participant. Begin by explaining the informed consent to the participants, and ask them to sign the consent form.
Next, perform a color vision test on the participants to confirm that they have normal color discrimination. Then present 30 mineral water brands to participants and instruct them to pick their familiar brands to ensure that they were unfamiliar with the brands of mineral water used in the experiment. Next, introduce the experimental procedure to the participants.
Conduct a pre experiment on the participants using mineral water brands unused in the study to make them familiar with augmented reality or AR and website operations. To wear the fNIRS probes, clean the forehead skin of the participants with skin preparation gel. Wrap the probes in plastic wrap to avoid sweaty circumstances, and fix them on the FP one and FP two position according to the international 10 to 20 system with a black headband.
Then cover the probes with a black bandana. Use eye tracking glasses to capture eye movement in real world environments. Fit prescription lenses on the head unit of the eye tracking glasses magnetically if required and ensure that the participants can walk around freely with a corrected sight.
Once all the participants are familiar with the experiment direct each participant from group A to perform the AR experiment first and then the website experiment. Similarly, ask the participants from group B to perform the website experiment first and then the AR experiment. Ensure the participants sit comfortably on a chair and rest for two minutes to collect baseline data before the experiment.
To measure the fNIRS data, open the recording software and connect the probes to a laptop via Bluetooth adapter. Then record the concentration changes of oxygenated hemoglobin in the participant's prefrontal cortex with a sampling frequency of 10 hertz. Check the received light intensity and tissue saturation index quality to control data quality.
Ensure that the received signal is located between one and 95%To measure the eye tracking, set up the hardware for the eye tracking glasses. Connect the plug of the eye tracking glasses to a USB port on the laptop. Then open the recording software and set the sampling frequency to 120 hertz.
Perform a one point calibration by asking the participant to focus on an identifiable object in their field of view At 0.6 meters. Move the cross hair cursor to the focused object in the scene video and click on the object. Then click the record button on the software interface to start recording.
Present the usability questionnaire and NASA TLX scale to the participants after they complete the tasks of the AR and website. After completing the experiment clean the transmitter and receiver of the probes using a 70%isopropyl alcohol pad. The median difference scores of the usability questionnaire showed that the participants had a better user experience in the AR condition than in the website condition.
The eye tracking data revealed that the participants had higher information search efficiency when using AR than the website. The median difference scores of the NASA TLX scale indicated that the AR technique led to a lower cognitive load than the website technique. The fNIRS data analyzed for the differences in the mean oxygenated, hemoglobin and website conditions showed dynamic cognitive load changes.
Participants familiar with the brands of mineral water used in the experiment were excluded because they would have performed the task based on their knowledge of the brand. This study proposes a experimental paradigm combining subjective and objective evaluations in the mix which can effectively evaluate the usability of emerging technologies.
View the full transcript and gain access to thousands of scientific videos
Overview
This study introduces an experimental paradigm for a usability test of emerging technologies in management information systems. The usability test included both subjective and objective evaluations using mobile fNIRS and eye tracking glasses.
Key Study Components
Area of Science
- Neuroscience
- Human-Computer Interaction
- Usability Testing
Background
- Usability tests are essential for evaluating emerging technologies.
- Combining subjective and objective evaluations enhances the validity of findings.
- Mobile fNIRS and eye tracking glasses address ecological validity in cognitive neuroscience.
- Real-world behavior experiments require reliable measurement tools.
Purpose of Study
- To develop a usability test paradigm for management information systems.
- To integrate multimodal measurements for comprehensive evaluation.
- To demonstrate the usability testing procedure with participants.
Methods Used
- Neuro-Information-Systems (NeuroIS) methods for objective evaluation.
- Usability questionnaire for subjective evaluation.
- NASA-Task Load Index (NASA-TLX) scale for workload assessment.
- Mobile fNIRS and eye tracking glasses for multimodal data collection.
Main Results
- The study successfully combines subjective and objective usability evaluations.
- Mobile fNIRS and eye tracking glasses provide valuable insights into user experience.
- Demonstration of the procedure highlights the practical application of the paradigm.
- Findings contribute to the understanding of usability in cognitive neuroscience tools.
Conclusions
- The experimental paradigm effectively assesses usability in real-world contexts.
- Combining different evaluation methods enhances the robustness of results.
- This approach can be applied to future studies in management information systems.
Frequently Asked Questions
Executive Industry Relevance
Integrating neuroimaging and behavioral analytics into usability testing enables biopharma R&D teams to objectively assess cognitive load and user experience for digital health and informatics platforms. This paradigm enhances predictive confidence in technology adoption decisions and supports risk-adjusted advancement of emerging digital tools within enterprise portfolios. The approach is directly relevant for evaluating AR, AI, and wearable interfaces in translational and operational settings.
Strategic Applications in Biopharma R&D
Early Discovery & Target Validation
- Supports hypothesis-driven evaluation of user interaction with digital health technologies.
- Enables functional validation of cognitive and behavioral endpoints in informatics tools.
- Provides objective data for de-risking technology integration into R&D workflows.
Screening & Assay Development
- Facilitates standardized usability assessment for digital platforms used in screening or data capture.
- Delivers reproducible, quantitative outputs on cognitive workload and user efficiency.
- Enables scalable evaluation of multiple interface prototypes or digital modalities.
Translational & Preclinical Research
- Aligns digital tool usability with real-world operational demands in translational research.
- Supports continuity from discovery-stage informatics to preclinical and clinical deployment.
- Provides mechanistic insight into user interaction, informing biomarker or endpoint selection.
Pipeline & Workflow Integration
This usability paradigm fits from early digital tool discovery through preclinical informatics validation, supporting technology triage and enterprise adoption decisions.
- Discovery Biology: Objectively quantifies cognitive and behavioral responses to digital interventions.
- Screening: Standardizes usability metrics for digital assay platforms or data collection tools.
- Analytics: Integrates neuroimaging and eye-tracking data for robust comparative analysis.
- Translational Research: Ensures digital tool readiness for operational deployment in preclinical and clinical settings.
- Enterprise Reuse: Establishes a reusable framework for evaluating diverse emerging technologies across R&D programs.
Operational & Enterprise Impact
- Scientific Value: Increases predictive confidence in digital tool selection and reduces mechanistic ambiguity.
- Operational Value: Delivers standardized, reproducible, and scalable usability assessments.
- Strategic Value: Informs go/no-go decisions for technology adoption and optimizes capital allocation.
- Portfolio Impact: Enables risk-adjusted prioritization of digital health and informatics investments.
Implementation Considerations
- Requires expertise in neuroimaging, behavioral analytics, and usability testing.
- Needs access to mobile fNIRS, eye tracking hardware, and analytical software infrastructure.
- Demands cross-team standardization of protocols and data interpretation.
- Adaptable to various digital modalities, including AR, VR, and wearable technologies.
- Ecological validity is enhanced but may be limited by participant familiarity and real-world constraints.
