Study design and patient populations
The Rural Adolescent Vaccine Enterprise (RAVE) study [22] was a five-year cluster randomized trial that utilized a stepped-wedge study design to implement and evaluate a tailored multicomponent practice facilitation intervention. During the first phase of the project, the study team conducted a positive deviance analysis to identify workflows, organizational factors, and communication strategies among rural Oregon clinics with higher HPV vaccine up-to-date rates, these results are published elsewhere [18]. During the intervention phase of the project, clinics were recruited from the Oregon Rural Practice-based Research Network (ORPRN), a practice-based research network established in 2002 open to all Oregon primary care clinics [23]. Eligible clinics specialized in Family Medicine and/or Pediatrics and were located in rural areas, defined as Rural Urban Commuting Area (RUCA) code ≥ 4, or rural or frontier clinic designation from Oregon’s Office of Rural Health [24].
Clinics were eligible to participate if they were active members of the Vaccines for Children (VFC) program and served at least 20 adolescents aged 13–17 and at least 10 adolescents aged 11–12 in 2018. The federal VFC program provides free vaccines to clinics for patients who are uninsured, under-insured, participate in public programs such as Medicaid, or are members of special groups including American Indians & Alaskan Natives [25, 26]. Participation in VFC requires vaccination reporting to Oregon’s ALERT Immunization Information System (IIS), which was the primary data source for this project. Of the 151 Oregon clinics that met these criteria, we successfully enrolled 46 clinics (Fig. 1). A priori power calculations indicated that 40 clinics provided 80% power to detect a difference of 8.8% in HPV vaccination. Clinics were then randomized to one of five study groups with start times six months apart and intervention activities starting in June, 2019 (Fig. 2). Four groups of two clinics and one group of three clinics, each affiliated a unifying health system, were cluster-randomized together due to clinic request based on existing collaborative quality improvement activities.
Figure 2. CONSORT Diagram: Clinic participation in the Rural Adolescent Vaccine Enterprise (RAVE).
Fig. 1
CONSORT diagram: clinic participation in the rural adolescent vaccine enterprise (RAVE)
Fig. 2
Study timeline showing stepped-wedge, cluster-randomized design
Intervention
Each clinic participated in an 18-month intervention aimed at building quality improvement infrastructure and improving HPV vaccination. The intervention was led by practice facilitators from ORPRN. During months 1 and 2, clinics participated in baseline assessments of quality improvement and vaccination workflows, received basic instruction on the Model for Improvement [27], and received state registry data showing their clinic-level vaccination rates. A practice facilitator [28] worked with each clinic to identify areas for improvement and tailor strategies to their local environment. Strategies were collaboratively identified based on first phase positive deviance analysis of rural Oregon clinics, as well as clinic goals and perceived feasibility [18]. For example, some clinics emphasized systems for reminder and recall, while others focused on staff education, workflows, or patient materials. Most clinics used multiple strategies sequentially using Plan-Do-Study-Act methodology. While all clinics worked to improve HPV vaccination among 11–17-year-olds, some clinics chose to tailor individual strategies to prioritize vaccination initiation or completion alone, based on the perceived needs of their clinic and their patient population. Starting in month three, the facilitator planned monthly meetings with clinic staff to support continued progress using a Plan-Do-Study-Act improvement framework [29]. Facilitation visits were initially conducted with a mix of in-person and virtual formats, but were converted to all virtual starting in early 2020, due to the COVID-19 pandemic. During the final 12 months of the intervention, clinics conducted social marketing campaigns with a community partner of their choosing and were provided $2000 for these activities. The social marketing campaign was an exploratory component of the project as described elsewhere [30].
Data sources
The Oregon ALERT IIS registry was used to measure for all clinic-level vaccination outcomes. To factor out effects of mobility and time-based IIS record fragmentation, clinic-based counts for adolescent patients were calculated based on weighting of time since last report to the IIS. The weighting approach used an ogive weighting function, which has previously been shown to produce highly consistent matching between IIS and other patient count sources [31] and this methodology has been shown to have good concordance with vaccination data from electronic health records [32]. In Oregon, immunizations are reported to ALERT IIS via direct clinic reporting, payor reporting, and billing exchanges.
Additionally, data regarding clinic characteristics (e.g. number of providers and patients), and intervention characteristics (e.g., improvement strategies pursued) were collected directly from clinic staff and facilitators through validated survey assessment field notes (described below).
Study variables
Dependent variables included: (1) clinic-level proportion of adolescent patients (11–17) with receipt of ≥ 1 HPV vaccination, and (2) clinic-level proportion of adolescent patients with UTD HPV vaccination, defined as receipt of two vaccines if initiated before age 15 and three vaccines if initiated on or after age 15, per CDC guidelines [33, 34]. During the course of the RAVE study, the American Academy of Pediatrics endorsed starting HPV vaccination at age 9 [35], and while some clinics chose to include vaccination of these younger children in their clinic-tailored goals, expanding our data to examine outcomes in this age group was beyond the scope of this project. Since a secondary goal of the intervention was to enhance improvement infrastructure among rural primary care clinics [22], clinics were assessed with the validated Quality Improvement Change Assessment (QICA) tool [36] at the beginning and end of the intervention. A variable was created for change in QICA score for each of the survey domains, representing the difference in post- and pre-study QICA ratings provided by clinic staff.
Baseline clinic-level covariates were assessed at the beginning of the study using a practice structure survey and data from the ALERT IIS database from the first quarter of 2019. Covariates included: (1) number of active adolescent patients (ages 11–17), (2) practice specialty (family medicine vs. pediatrics or both), (3) clinic structure as community health center, health system (any clinic affiliated with a local hospital, payor, or regional chain), or private/independent, and (4) baseline quality improvement infrastructure (using QICA tool [36]).
Patient-level covariates were aggregated to the clinic-level and were selected based on previous literature [37,38,39] and data available within the ALERT IIS system and included: (1) % VFC (proportion of youth with most recent vaccination assigned this payor status), and (2) % non-white youth. More detailed racial/ethnic/language characterizations were not available within the ALERT IIS database at the time of this study. Patient-level covariates describe the population of youth (age < 18) attributed to each clinic that have ever received a vaccination in Oregon. Though this denominator may exclude some individuals (e.g., individuals who have never received vaccination), a comparison of vaccination data between electronic health records and Oregon’s ALERT IIS registry has shown good concordance [32]. Finally, a covariate for the month of intervention start (e.g. June or December) was included to account for known seasonal variation in vaccination rates. All covariates were described using means and standard deviations (SD) for continuous variables and frequencies and percentages for categorical variables.
Statistical analyses
First, we performed a descriptive analysis of quarterly HPV vaccination rates (initiation and completion) throughout the duration of the observation, intervention, and follow-up periods for each of the five study groups. We calculated means and standard deviations for each group at four timepoints: baseline, intervention beginning, intervention completion, and study completion. This descriptive analysis was intended to describe trends over time in each of the study groups.
We also conducted a descriptive analysis of change in QICA scores over the course of the study. For this analysis, we calculated mean pre- and post-study QICA score in each of the survey’s domains and overall, calculated ‘change in mean QICA score’ and conducted a two-tailed t-test to assess whether the observed changes were significantly different than the null with alpha > 0.05.
To assess the effect of the intervention, we used mixed-effects Poisson regression models to report incidence rate ratios for vaccination outcomes (i.e., HPV 1 + dose, and HPV UTD). The models account for temporal repeated measures over the study period within clinics and were adjusted to account for baseline differences in clinic population and structure (i.e., specialty, clinic type, number of adolescent patients, %VFC, %non-white, baseline QICA score, and baseline HPV vaccination rate). Utilizing an interrupted time series approach, we report the slopes and intercepts during two time periods (before the intervention and during the intervention and follow-up), as well as the difference in slope between the two time periods. An assumption of the Poisson model is that the mean of the outcome is equivalent to its variance, so we tested for overdispersion and found none. Clustering of clinics based on requested randomization was attempted but lead to model non-convergence so was not included. Statistical significance of p-values was assessed at alpha < 0.05.
As a sensitivity analysis, we repeated the above analysis limiting the sample to “as treated” clinics (i.e., clinics that did not withdraw from the intervention, n = 36 clinics) as opposed to the primary analysis which included all participating clinics (n = 45 clinics), including clinics that withdrew during the study.
Qualitative analysis
We conducted qualitative analyses to better understand and describe the implementation efforts at each clinic. Though a complete qualitative analysis is in progress, relevant contextual features of intervention uptake are included here. Qualitative data were collected by practice facilitators in semi-structured fieldnote logs maintained in REDCap. Fieldnotes included rich descriptions of intervention activities, PDSA cycles, and clinic context. Fieldnotes for all participating clinics were then exported from REDCap to ATLAS.ti for coding, using an a priori codebook. Next, summaries of coded data were populated into a matrix to identify patterns in implementation efforts. Final themes were solidified using a template analysis that classified types PDSA content and categories of facilitation activity. Matrix findings were then mapped to the template to produce final implementation themes for each clinic.
All study activities were approved by Oregon Health & Science University’s Institutional Review Board (IRB #18660).
