Systematic Review and Evidence Aggregation

Clinical Practice Guideline Systematic Review and Evidence Aggregation

Recommendations in clinical practice guidelines should be built on a foundation of the best available evidence and assembled in a nonbiased manner. A systematic review of the literature for each clinical question to be addressed by a guideline is the most effective way to create this foundation. This work should be completed before the guideline subcommittee meets to draft recommendations.

AHRQ Systematic Review

When an external agency (eg, AHRQ) is engaged to develop the systematic review for an AAP guideline, the AAP will nominate the appropriate content experts and a key methodologist to serve on the AHRQ technical expert panel as well as to serve as key informants for AHRQ. This is important to ensure that the clinical questions and scope of the project are understood adequately by the external agency group and then adequately addressed in the systematic review. Conference calls are recommended prior to the external group starting its work and then at intervals, with the external agency to review and discuss progress and findings.  

When the systematic review is conducted by AHRQ, these roles will not incur a fiscal note on behalf of the AAP. IOM/NAM standard 2.1 outlines 5 areas of expertise to be included: 

  • Pertinent clinical content 

  • Systematic review methods

  • Searching strategies for relevant evidence 

  • Quantitative methods (eg, statistical analysis and interpretation)

  • Other, as appropriate (eg, qualitative methods)

AAP Systematic Review

If AHRQ does not provide a systematic review, the AAP will decide whether or not to move forward with the guideline topic proposal. If it is determined that the AAP should develop the guideline without the aid of AHRQ, guidelines staff will contract a methodologist from the Section on Epidemiology, Public Health, and Evidence (SOEPHE), and the AAP will determine the funding source to support guideline development.  

In instances where the AAP receives external funding to conduct a systematic literature review for another entity, the COGD will collaborate with the systematic review panel to ensure that the scope of the work and the clinical questions rise to the level of rigor required for clinical practice guidelines should there be an opportunity to use the research as the foundation for AAP policy. 

IOM/NAM Standards for Systematic Review

In March 2011, the IOM (now National Academy of Medicine) released a set of national Standards for Systematic Reviews. These standards provide an aspirational roadmap for initiating systematic reviews, finding (selecting) and assessing individual studies relevant to the guideline, and synthesizing (aggregating) the evidence that provides the basis for each recommendation in the guideline. The following are utilized throughout the process:

  1. Developing Questions that Cover Desired Content and Clinical Issues to be Addressed (IOM/NAM Standard 2.5: Formulate the topic for the systematic review)

  2. Developing the Roadmap for the Systematic Review (IOM/NAM Standards 2.6 – 2.8) 

  3. Finding and Synthesizing the Relevant Data (IOM/NAM Standards 3 and 4) 

  4. Screening and Selection of Studies and Search Documentation (IOM/NAM Standards 3.3 and 3.4) 

  5. Data Collection and Study Appraisal (IOM/NAM Standards 3.5 and 3.6) 

  6. Synthesizing the Body of Evidence for Clinical Questions (IOM/NAM Standard 4) 

  7. Reporting the Results of the Systematic Review (IOM/NAM Standard 5)

Publication Biases and Grey Literature

The published literature at times is biased toward studies with positive results. Additionally, sometimes manuscripts for well conducted studies are simply never developed or submitted for publication. Some such studies are described in abstracts presented at research meetings or mentioned in either summaries of clinical conferences or so called “white papers” of various organizations. Collectively, these sources are termed the “grey literature,” as searches of bibliographic databases will not yield evidence of these studies. 

IOM/NAM stand 3.2 states that action should be taken to overcome potentially biased reporting of research results. Searching of grey literature databases, clinical trial registries, and other sources of unpublished information about studies are suggested approaches (IOM/NAM 3.2.1). If such studies are sought and found, the researchers who conducted them may be contacted to provide information on study eligibility and design (IOM/NAM 3.2.2) and to provide the subcommittee with unpublished data, including unreported outcomes (IOM/NAM 3.2.3). Manual searching of selected journals and conference abstracts (IOM/NAM 3.2.4), Web searches (IOM/NAM 3.2.5), and searches for studies in languages other than English (IOM/NAM 3.2.6) may be conducted.

General Outline for Literature Review Process

  1. Conduct meeting/call to determine preliminary search strategy 

    • Clinical practice guideline subcommittee Chair and Vice Chair in collaboration with the epidemiologist to determine initial search criteria and terms.

    • Librarian/epidemiologist documents proposed search strategy.

    • Search for meta-analyses and evidence-based clinical guidelines first, including Cochrane Reviews, NICE guidelines; only perform a primary literature search starting from the date that last evidence-based clinical guideline ended. If updating an existing AAP guideline, use Elsevier’s SCOPUS abstract and citation database of peer-reviewed literature to search/determine which manuscripts cited prior guideline.

  2. Conduct scout search. Initial search to include at a minimum: MEDLINE, CINAHL, EMBASE. Document search strategy and number of documents retrieved.

    NOTE: AHRQ has restricted search on occasion to randomized controlled trials (RCTs) and treatment studies. It is recommended that this restriction be followed only if the number of articles returned exceeds some predetermined maximum (e.g., 500 articles).

  3. Conduct meeting/call to finalize search strategy

    • Review any meta-analyses or evidence-based clinical guidelines that document their search strategy.

    • Review search results and refine search strategy and terms.

    • Librarian/epidemiologist documents final search strategy.

    • Librarian/epidemiologist conducts the search, updates number of documents retrieved, and sends the results to the Chair.

    • Guideline subcommittee may add hand-selected articles not found by literature search for consideration.

  4. Winnow articles by title/abstract

    • Prepare for article selection by determining at least 2 reviewers for each article.

    • From review of title/abstract, reviewers to determine relevance to clinical questions. Optionally can provide numerical rating for strength of evidence. Each reviewer decides if article is in or out. Articles are included if any reviewer opts to include.

    • Summary list of articles included is created and counts of excluded articles and reasons are documented. Full text articles are accessed, or interlibrary loan requests for inaccessible articles are made.

  5. Full text selection of articles

    • Make full text review assignments:

      • Determine which articles will answer each clinical question (ie, the article(s) with the highest evidence level relevant to the clinical question).

      • Assign 2 reviewers to each clinical question.

    • Reviewers can decide if an article is irrelevant or decide an article is relevant and summarize the article and add to evidence table.

    • If additional clinical questions arise during the review process, note clinical questions and return to full text review to refine search strategy.

    • For each clinical question, one reviewer will summarize relevant articles in evidence profile, and second reviewer will confirm.

    • Final reference list of articles is created and counts of excluded articles and reasons are documented.

Advertising Disclaimer