Skip Navigation LinksCIzQIDS

aaa print

Comparison of Immunization Quality Improvement Dissemination Strategies Project

Quality Improvement Technical Support (QITS)

QuIIN: Home to national pediatric improvement networks

Comparison of Immunization Quality Improvement Dissemination Strategies (CIzQIDS)

The Comparison of Immunization Quality Improvement Dissemination Strategies project partnered with the Practice Improvement Network (PIN), part of the Quality Improvement Innovation Networks (QuIIN) of the American Academy of Pediatrics (AAP).  The specific aims of the project were to compare the effects of two different  quality improvement dissemination models on the immunization coverage among children 3 to 18 months old.  This project used a cluster randomized trial design with physician practices (QuIIN PIN participants) serving as the unit of intervention and patients nested within practices (clusters) representing the unit of analysis.  Thirty-four practices were randomly assigned to receive either Quality Improvement Technical Support (QITS) or Pay for Performance (P4P). Both groups received a toolkit of resources to support use of quality improvement science to improve immunization delivery. The P4P practices participated in a tiered financial incentive program. The QITS practices participated in a virtual learning collaborative with on-line learning sessions and coaching, with an opportunity to earn MOC part 4 credit if specified criteria were met. The following provides details about the QITS intervention.

The AAP QuIIN co-managed the QITS intervention group with the Children’s National Health System CIzQIDS Expert Group, Community Quality Improvement Coach and Research Coordinator.


Acknowledgement of Funding

The CIzQIDS project is funded by a grant from Pfizer, Inc.

Vaccinator Toolkit – Coming Soon!

The Vaccinator Toolkit provides resources to support implementation of the National Vaccine Advisory Committee’s Standards for Pediatric Immunization Practice. Each section contains background information, a collection of useful print materials, links to additional electronic resources and a list of suggested quality improvement practices. Sections include

  1. Developing a quality improvement project
  2. Vaccine Basics: Who gets what, when, where and why
  3. Contraindications and precautions
  4. Administering multiple vaccines in a single visit
  5. National Vaccine Advisory Committee Standards
  6. Standards about availability of vaccines and vaccine services
  7. Assessment of vaccination status
  8. Effective communication about vaccine benefits and risks
  9. Staff training resources

Resources were compiled in March 2013. Because recommendations change frequently, readers are advised to check other sources for updates.

Project Goal

The goal was to increase the immunization coverage by improving adherence to recommendations of the Advisory Committee on Immunization Practices (ACIP) and the National Vaccine Advisory Committee.

Project Aims

The CIzQIDS-QITS project aimed to increase immunization coverage among 3- to 18-month olds in participating practices to 90% through a 12-month virtual learning collaborative.

Project Measures

  • Main outcome measure
    To determine pre- and post-intervention age-adjusted percent of 3- to 18-month-olds who were up-to-date for ACIP-recommended doses of DTaP, HepB, HIB, IPV PCV, MMR, and Varicella vaccines, for each practice, project staff reviewed 50 randomly-selected, age-stratified medical records.  Record reviews were performed either on-site or by remote access of electronic health records. No personally identifiable information was collected.
  • Process and Balancing Measures  
    Processes measured included (1) documentation of assessment of immunization status at every encounter; (2) giving each child all indicated vaccines at every encounter; (3) providing families with a reminder about, or appointment for the next immunization visit at each encounter; (4) perceived effect of immunization improvement activities on length of visits and on the clinical and operational work of the practice as a whole. The data for these measures were provided by local teams through monthly on-line surveys and a post-intervention survey using the REDCap survey platform.


To be eligible for participation, practices had to provide immunizations to more than 5 children ages 0-18 months per week and have less than 86% of 3- to 18-month-olds up-to-date for ACIP-recommended immunizations as determined by review of 50 randomly-selected medical records.

Sixteen primary care clinical teams participated in this quality improvement project. The selected practices represented a diversity of geographical locations, practice settings (urban, rural, and suburban), practice size, and type of organization (e.g., private practice, FQHC, hospital outpatient departments).

Practices were advised to form a local team consisting of a lead physician, clinical support staff (e.g. nurse), and an administrative or office staff person who could commit to participating in immunization-related improvement efforts during the year-long project. Click here to see a list of all of the clinical teams.

Project Design:

  • Baseline assessment:

    • Baseline survey: Local project leaders from each practice completed an on-line survey about practice demographics and immunization practices.

    • Medical record review: Project staff reviewed medical records of 50 randomly selected 3- to 18-month olds from each practice. Results were provided to practices.

  • Orientation Webinar:

    Introduction to the Quality Improvement Technical Support intervention.

  • 12-month action period:

    • Web-based learning sessions: Topics were chosen to provide support application of improvement science to problems with immunization delivery. After each session, participants were sent a link to provide feedback on the effectiveness of the session through a standard workshop evaluation questionnaire. The presentation slides below were prepared based on immunization available in 2013-2014. Readers should remember that immunization recommendations change frequently; more recent sources should be consulted for the latest recommendations.

    • Local team activities: Clinical teams initiated improvement strategies based on local needs. They used Plan-Do-Study-Act (PDSA) cycles to test whether their changes were improvements.  Each month they reported on their activities through an on-line survey about the following;
      • Immunization practices during the last month
      • Tests of change made during the last month
      • Review of 10-20 medical records of children seen during the last month
    • Monthly progress reports: To provide performance feedback, the QI coach used data from the monthly survey to create run charts for process measures and the main outcome measure to allow comparison of performance of individual team performance with the group mean and with benchmarks. Progress reports and suggestions from the coach were provided to each practice by email. The measures which were tracked in run charts are included.
      • Percent of records with an immunization assessment documented at the last visit
      • Percent of records indicating that the child received all indicated vaccines at the last visit.
      • Percent of records with documentation that the family received an appointment for, or reminder about the next vaccine visit.
      • Percent of records indicating that the child was up-to-date for all ACIP-recommended immunizations.
    • Monthly collaborative webinars: Each month the QI coach facilitated web-based conference calls where run charts were reviewed, participants shared their insights and experiences, questions were answered, resources were shared, and specific topics were discussed.

  • Post-Intervention Assessment:

    • Post-intervention survey: Local project leaders from each practice completed an on-line survey to provide data about changes initiated, local factors that might have affected immunization delivery, factors motivating immunization improvements, usefulness of various aspects of the QITS intervention, and overall perception of effects of participation in the project on the practice as whole.

    • Medical record review: Project staff reviewed medical records of 50 randomly selected 3- to 18-month olds from each practice.
    • Debriefing Session/Virtual Focus Groups: Fifteen practices participated in one of three web-based small-group debriefing sessions facilitated by an independent moderator using a semi-structured interview guide. Topics discussed included motivation for participation, team functioning and decision-making, levels of staff engagement, perceived barriers and facilitators, and effects of participation on individual practitioners, patients and the practice as a whole.

Project Data Collection included:

  • Medical record reviews
    • By project staff at baseline (50 per practice)
    • By local team each month through the 12-month action period (10-20 per practice)
    • By project staff post-intervention (50 per practice)
  • On-line surveys
    • Baseline survey
    • Monthly surveys
    • Post-intervention survey
  • Learning session evaluations
  • Debriefing session transcripts

Resources for Core Improvement Teams


Fu L, Zook K, Gingold J, Gillespie CW, Briccetti C, Cora-Bramble D, Joseph J, Moon R. Frequent Vaccinations Missed Opportunities at Primary Care Encounters Contribute to Underimmunization. J Pediatr 2015;166:412-7.

Zook K, Gingold J, Gillespie CW, Briccetti C, Moon R, Cora-Bramble D, Fu L. Missed opportunities to vaccinate children during pediatric office visits. Academic Pediatric Association Region IV Annual Meeting 2014 (presentation)

Zook K, Gingold J, Gillespie CW, Briccetti C, Moon R, Cora-Bramble D, Fu L. Missed opportunities to vaccinate children when they present for care. Pediatric Academic Society Annual Meeting 2014 (poster)

Gingold J, Zook K, Briccetti C, Gubernick R, Rice-Conboy E, Cora-Bramble D and Fu L. Using the Model for Improvement and a virtual learning collaborative to improve immunization delivery. 26th Annual National Forum on Quality Improvement in Health Care, 2014. Institute for Healthcare Improvement Storyboard presentation.

Special thanks to the Expert Group:

  • Linda Y Fu, MD, MS FAAP (Chair and Primary Investigator)
  • Christine Briccetti, MD, MPH FAAP
  • Janet Gingold, MD, MPH, Quality Improvement Coach
  • Kathleen Zook, RN, MPH, Research Coordinator
  • Rachel Y Moon, MD FAAP
  • Ruth Gubernick, MPH, Quality Improvement Consultant
Advertising Disclaimer