Beyond P-Values: Bayesian Approaches for User Experience Research

• Mohsen Rafiei, PhD

  University of Arkansas at Little Rock

  Author

• Iman Tahamtan, PhD

  University of Tennessee, Knoxville

  Author

Null hypothesis significance testing (NHST), using p-values and confidence intervals, has long been the standard in user research, particularly in large-sample settings like A/B testing. However, user experience studies often rely on smaller samples, rapid iterations, and design-driven outcomes, in which p-values can be difficult to interpret, and confidence intervals may offer limited practical guidance. This paper introduces Bayesian statistics as a complementary framework better suited to these conditions. Unlike the frequentist view, which treats parameters (such as satisfaction score) as fixed but unknown quantities—meaning there is one true value in the population that doesn’t change—Bayesian methods treat parameters as uncertain and represent them through probability distributions, indicating which values are plausible given the data and any prior knowledge. Bayesian methods enable direct probability statements about parameters, integration of prior knowledge, and more interpretable results that align with iterative UX practices. In this paper, we introduce key Bayesian tools, such as Bayes factors and credible intervals, as more informative alternatives to p-values and confidence intervals that make it easier to compare different models and express uncertainty in a way that is more useful for iterative design decisions. Advantages include robustness with small samples (when using appropriately informative priors), flexibility in handling hierarchical models (for example, data in which tasks are nested within users or users are nested within groups), handling missing data (by estimating values from the posterior under assumed missingness), and decision-readiness in design contexts. By reframing statistical inference around probability, evidence, and prior knowledge, Bayesian methods provide UX researchers a more transparent and practical toolkit for guiding design decisions.

• Full Text Article PDF

1. Mohsen Rafiei, PhD, University of Arkansas at Little Rock

   

   Dr. Rafiei leads Quantitative UX Research at the Perceptual User Experience (PUX) Lab, working closely with industry partners to bring scientific rigor to real-world design challenges. He is also the director of the Experience Lab at the University of Arkansas at Little Rock, where he serves as an Assistant Professor of Psychological Science.

2. Iman Tahamtan, PhD, University of Tennessee, Knoxville

   

   Dr. Tahamtan is a UX lecturer at the University of Tennessee, Knoxville (UTK). His research focuses on analyzing user behavior, needs, and challenges in interactions with digital technologies to enhance their design, usability, and accessibility. He also serves on the editorial board of the Journal of User Experience (JUX).