Evaluating the Evidence: Creation of Gold Standard Practices for Searching and Filtering the Biomedical Literature
Rebecca N. Jerome1, MLIS; Kimbra Wilder Gish1, MS; Taneya Y. Koonce2, MSLS; Nunzia B. Giuse1, MD, MLS.
1 Eskind Biomedical Library, Vanderbilt University Medical Center
2 Associate Fellow, National Library of Medicine
Presented at the Medical Library Association Annual Meeting, May 30, 2001.
Purpose: Describe a foundational philosophy for developing and sustaining a gold standard practices approach to searching and filtering the biomedical literature.
Setting: While incorporating evidence into practice and optimizing resource utilization are important concerns in today's healthcare climate, the proliferation of the medical literature presents a significant challenge to clinicians in assessing evidence for best practices. This large academic health sciences library facilitates the integration of information into the Medical Center's processes and practices through the librarians' provision of customized, filtered information packages for clinical and research teams.
Background: As we attempt to expand librarian expertise into settings beyond the library's walls, it is essential to guarantee a level of competency embraced by all librarians. To achieve this goal, the library has developed two professional conferences as a mechanism for the diffusion of searching and filtering skills among all librarians: SearchTalk and the Filtering Teaching Conference (FTC). In SearchTalk, librarians meet bimonthly to explore resources in response to actual clinical and research questions and develop gold standard search strategies for each question; three key articles are selected from the gold standard's results and distributed for discussion later in the month at the FTC meeting. The FTC adapts the SearchTalk model to the development of gold standard practices for evaluating study design, weighing evidence provided by articles, and summarizing information properly. In both conferences, the gold standard is not generated by an individual but by the whole team via consensus. Rather than two or three experts operating in isolation, these sessions have made it possible over the last two years to increase the competency of all librarians in these areas.
Methods: This study represents a preliminary examination of SearchTalk and FTC individual efforts as compared with the gold standard consensus of each meeting. For the 2000-2001 academic year, investigators collected search strategies, article summaries, and filtered articles from 14 participants; one participant did not participate in the March SearchTalk and FTC sessions, leaving 13 participants for this session. The investigators examined data for three selected sessions of the paired conferences: our first meeting of the year in September (clinical research question), our middle meeting in December (clinical cardiology question), and our final meeting of the academic year in March (genetics question). Each individual's search strategy was executed in PubMed and examined for retrieval of the three articles selected for the FTC from the gold standard results. To examine the FTC session data, two individuals reviewed the gold standard filtering of one article for each FTC and noted key article features. The two lists of key features were compared and discussed until consensus was reached. Each participant's filtering was then rated against the gold standard using a checklist of these key features.
SearchTalk results: In comparing searching performance between the September and March sessions, overall progress toward agreement with the gold standard consensus was clearly evident (Figure 1). The September data included 14 participants, three of whom retrieved all three of the gold standard articles, five of whom retrieved two of the gold standard items; the March data included 13 participants, five of whom retrieved all three gold standard articles, five of whom retrieved two of the gold standard items.
Filtering results: A definite trend in improvement was evident among the attendees in overall percentage agreement with the gold standard filtering for each selected session (Figure 2). Of the six individuals who had not previously participated in filtering exercises, the data indicated a steady improvement toward consistency with the gold standard; however, a drop in performance was noted for some with the genetics question in March, indicating a lack of comfort with this subject area. This subject knowledge base effect was even more striking among some of our more experienced participants. Several of these individuals maintained a fairly consistent level of agreement with the gold standard for the March question, but several participants demonstrated a clear drop in performance despite previous improvements, largely explained by a lack of comfort with the genetics subject area. There was a different subject knowledge base effect in two additional participants: these individuals rarely deal with clinical questions and demonstrated an obvious drop in comfort when dealing with the clinical question from the December meeting. In addition to the general improvement among the participants, the clear impact of the subject knowledge base on individuals' performance was striking.
Future directions: Beyond this preliminary analysis of the SearchTalk and FTC data, the investigators will carefully analyze all data collected during this academic year to better understand both individual and group trends. In addition, the authors hope to repeat this study with clinicians engaged in undertaking the same process.
Conclusions: By promoting gold standard practices for staff searching and filtering of the biomedical literature, participation in these sessions equips librarians to function as proactive, trusted members of clinical and research teams. While the librarians may initially have disparate levels of skills, these conferences reduce variance among individual proficiencies.
|
