Benchmarks are a great way to evaluate the effectiveness of drug management programs and are often divided by health plan line of businesses without further adjustment. However, subpopulations within line of businesses can have varying risk profiles associated with population characteristics (e.g., comorbid burden, demographics) that create biased key performance indicator (KPI) comparisons.  
  
Prescription drug utilization measures are limited by the lack of risk adjustment for population differences in demographics and health status. When comparing index and benchmark groups with varying risk profiles, KPIs may not be reflected appropriately among the index  but may instead reflect different  factors such as  disease burden. When risk adjustments are applied to KPIs, it can reduce bias and allow organizations to monitor change, make data-driven decisions and measure effectiveness with fewer influences.  
  
The purpose of this study, presented at the Academy of Managed Care Pharmacy (AMCP) Nexus, was to create a methodological framework for using propensity score weighting to adjust for varying risk profiles when comparing an index group to a benchmark group with a similar risk profile on pharmacy KPIs. The data suggests that accounting for confounding differences through propensity score weighting can control for population differences and identify a meaningful comparison. 

I (Alex Cook, part of Prime’s newsroom team) spent some time with Brooke Hunter, MS, senior health outcomes scientist at Prime Therapeutics (Prime), and lead author on this study, to learn more.

Alex Cook: This study certainly stands apart from the other Prime research posters because it takes a look under the hood to better understand how data is gathered and analyzed in managed care pharmacy research. Based on what you found in this study, what we some of the key factors researchers might take into account when they design a study to better account for KPIs?
Brooke Hunter: In general, standardization to ensure consistency of KPI measurement across the industry that allows for comparison to outside sources (e.g., per member per month [PMPM] cost calculations) is an important consideration. In this study specifically, we calculated our own benchmark. The key here was that we identified heterogeneity across the population on key influential factors related to PMPM costs, such as average age of membership and disease burden of high-cost conditions (e.g., oncology). Heterogeneity of these factors across the population introduces a threat of bias that can yield an unfair comparison of some index clients to the benchmark population.

Tell us more about propensity score weighting and how to accurately control population differences.
Typically, a propensity weight is employed in observational studies to simulate a randomized controlled trial by balancing covariates that threaten to introduce bias between a treatment and control group. In our study, the index client is analogous to the treatment group and the benchmark to the control group. The propensity weight controls for covariates that introduce bias by weighting the control/benchmark group to match the treatment/index group. This allows for a fairer comparison of KPIs that enables us to more accurately assess the performance of the index group.

One of Prime’s objectives is to deliver an easy, transparent experience to members/patients. While individuals may not usually review managed care pharmacy research, how does this study help accomplish this?  
Confounding covariates introduce latent bias that is not visible to the naked eye. By examining the data to see if population differences exist between the index group and the benchmark group, we can identify population differences that may influence KPI comparisons. For example, if an index group has a higher rate of oncology burden within their membership relative to the benchmark, then the raw PMPM calculations will likely be higher for the index group due to the high cost of oncology treatment. This is a valid comparison, but in the context of our medical pharmacy services, which are designed to help manage medical drug spend, it may be inferred that the index client isn’t benefiting from the program. By comparing the index group to a propensity matched benchmark, we better understand how well the medical pharmacy services are managing PMPM cost given the index group’s unique set of population characteristics.

What reactions have you received from your peers on your research?
The response related to our research has been overwhelmingly positive. When comparing client data to a benchmark, a common question we receive is, “How do we know the comparative group is like the reference client?” This work gives us talking points to be able to describe population differences and give confidence that we are making fair and accurate comparisons when evaluating KPIs. Additionally, we can discuss how we account for these population differences to get to more meaningful measurements to better assess the utilization and spend metrics.

Check out more AMCP Nexus 2024 in focus content at the Prime newsroom. For more information on this research, check out the research poster.