APPD LEARN is the APPD's Longitudinal Educational Assessment Research Network.
APPD LEARN provides an infrastructure for multi-centered, collaborative research projects and a centralized data collection system. LEARN encompasses two research networks that share the APPD LEARN infrastructure and governance.
The APPD Pediatric Residency Investigators in Medical Education (APPD PRIME) network was APPD LEARN's first network, and focuses on research on Pediatric residents and residency programs.
The APPD Subspecialty Pediatrics Investigator Network (APPD SPIN) is a medical education research network for the Pediatric Subspecialties to evaluate the education and assessment of fellows. SPIN began as a collaborative effort of the American Board of Pediatrics, the APPD Fellowship Committee, APPD LEARN, the Council of Pediatric Subspecialties and each pediatric subspecialty's program director network. In 2020, SPIN became APPD SPIN, a network of APPD LEARN.
About APPD LEARN
The mission of APPD LEARN is to conduct meaningful educational research that advances the training of future Pediatricians by developing and promoting participation and collaboration in research by program directors for the purpose of improving the health and well-being of children. Our vision is to advance exemplary pediatric education through collaborative educational research by Pediatric program directors.
APPD LEARN pursues its mission through the following interconnected activities:
- Managing collaborative research networks of Pediatric Programs working together to conduct multi-site studies of educational methods and instruments
- Maintaining an online repository of educational research study materials, raw data, and findings for dissemination to APPD members and collaborators
- Promoting learning opportunities to enhance educational research participation and scholarship by Program Directors
- Providing expert consultation for research conducted within APPD LEARN
- Communicating regularly with the APPD membership and the larger medical education community about activities, opportunities, and outcomes
- Exploring, conducting, and coordinating research with other organizations and initiatives across a continuum of medical and non-medical education
The development of APPD LEARN began from APPD's experience with Share Warehouse, its site for sharing assessment tools and instructional materials among APPD member programs, and the R3P initiative. Planning for APPD LEARN began in 2006. A strategic plan was presented by Ann Burke, Patty Hicks, Susan Guralnick, and Rob McGregor in 2008. APPD and the Initiative for Innovation in Pediatric Education (IIPE) began allocating resources to support APPD LEARN and the initial Advisory Committee was formed in 2009.
APPD LEARN's first Director was Hilary Haftel. She was succeeded by Alan Schwartz in 2011
Alan Schwartz, PhD
APPD LEARN Director
Beth King, MPP
APPD LEARN Project Manager
Dennis West
APPD LEARN Research Assistant
APPD LEARN Advisory Committee, 2018-2020
- Teri Turner, MD (2018-2020, Chair)
- Sue Poynter, MD (2018-2020)
- Su-Ting Li, MD (2018-2019)
- Erika Abramson, MD (2018-2020)
- Javier Gonzalez del Rey, MD(2018-2019)
- Patty Hicks, MD, MHPE, PMAC (non-voting)
- Carol Carraccio, MD, ABP (non-voting)
- Richard Mink, MD, SPIN network (non-voting)
- Alan Schwartz, PhD, APPD LEARN Director
- Beth King, MPP, APPD LEARN Program Manager
- Laura Degnon, CAE, APPD Executive Director (ex officio)
APPD LEARN Proposal Review Committee (PRC), 2018-2020
- Maneesh Batra
- Michelle Barnes
- Sue Bostwick
- Kim Gifford
- Diane Stafford
APPD LEARN Educational Development Committee (EDC), 2018-2020
- Deb Hsu
- Heather McPhillips
- Angie Myers
- Dan Schumacher
- Pnina Weiss
APPD LEARN Director
The APPD LEARN Director is responsible for daily scientific and management oversight of all APPD LEARN activities, and serves as liaison to outside organizations. The Director reports to the APPD LEARN Advisory Committee and APPD Board. The APPD LEARN Director serves for a term of two years, renewable by the APPD Board indefinitely, at a time commitment of at least 0.4 FTE.
APPD LEARN Project Manager
The APPD LEARN Project Manager handles all administrative operations of APPD LEARN. Responsibilities of this full-time position include facilitating all communications among the participating institutions and funding agency, development of and adherence to project timelines, and observance of regulatory requirements. The APPD LEARN Project Manager is responsible for daily implementation of the project plan, and assisting PIs with study management. S/he serves as liaison with research administration at all participating institutions. In addition, the APPD LEARN Project Manager helps with additional network tasks and data collection efforts that arise. The APPD LEARN Project Manager works in conjunction with the APPD LEARN Director, and reports to the APPD Executive Director.
APPD LEARN Advisory Committee
The APPD LEARN Advisory Committee is be responsible for providing guidance to the APPD LEARN Director, setting policies for APPD LEARN activities and resources, and developing calls for proposals in specific research areas. The APPD LEARN Advisory Committee also conducts annual formative and summative evaluation of the performance of the APPD LEARN Director. The APPD LEARN Advisory Committee meetings monthly by conference call and annually in person.
The Committee consists of five voting members, including four APPD-designated members (one of whom serves as the Chair), and one IIPE-designated member. The APPD Board selects the Chair of the APPD LEARN Advisory Committee. In consultation with the Chair of the APPD LEARN Advisory Committee, the APPD Board selects the three additional APPD members. IIPE appoints one representative to the committee. APPD LEARN Advisory Committee terms are 2 years long, and renewable. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director serve ex officio on the Advisory Board.
APPD LEARN Educational Development Committee (EDC)
The APPD LEARN Educational Development Committee will advise the APPD LEARN Director in (1) determining the faculty development needs of APPD LEARN members in the area of educational research, (2) designing and interpreting the annual APPD LEARN needs assessment survey, and (3) identifying training opportunities. The committee chair and four members will serve two-year terms, which may be renewed. They will be appointed by the APPD LEARN Advisory Committee, in consultation with the APPD LEARN Director, after soliciting suggestions from the APPD Council of Task Force Chairs. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director will serve ex officio on the Committee. The committee meets as needed by conference call, at a frequency no more than quarterly, with online discussion as needed between calls.
APPD LEARN Proposal Review Committee (PRC)
The APPD LEARN Proposal Review Committee will assist the LEARN Director in the review of proposals to conduct research using the LEARN network and its member Programs. The committee chair and members will serve two-year terms, which may be renewed. They will be appointed by the APPD LEARN Advisory Committee, in consultation with the APPD LEARN Director, after soliciting suggestions from the APPD Council of Task Force Chairs. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director will serve ex officio on the Committee. The committee will meet as needed by conference call, at a frequency no more than monthly, with online discussion as needed between calls.
APPD LEARN Project Committees
Each study using the APPD LEARN network will have an ad hoc project oversight committee, composed of the project principal investigator, APPD LEARN Director, APPD LEARN Project Manager, and other members selected by the principal investigator and APPD LEARN Director. These committees will hold regular (e.g. biweekly) conference calls during the period of the study.
Member Programs
Advocate Lutheran General Hospital
Albany Medical Center
Albert Einstein College of Medicine (Jacobi)
Albert Einstein College of Medicine (Montefiore)
All Children's Hospital Johns Hopkins Medicine
American Board of Pediatrics
Ann & Robert H. Lurie Children's Hosp of Chicago
Atlantic Health System, Goryeb Children's Hospital
Baylor College of Medicine (Houston)
Baylor College of Medicine (San Antonio)
Baystate Medical Center/ Tufts University School of Medicine
Blank Children's Hospital
Boston Children's Hospital/Boston Medical Center
Brookdale University
Brooklyn Hospital Center
Brown University Medical School Hasbro Children's Hospital
Carilion Clinic-Virginia Tech Carilion School of Medicine
Carolinas Medical Center
Case Western Reserve University (MetroHealth)
Case Western Reserve University/University Hospital Case Medical Center/Rainbow Babies and Children's Hospital
Children's Hospital Medical Center of Akron/NEOUCOM
Children's Hospital of Los Angeles
Children's Hospital of Philadelphia
Children's Mercy Hospital-University of Missouri at Kansas City
Children's National Medical Center/George Washington University
Cincinnati Childrens Hospital Medical Center/University of Cincinnati College of Medicine
Cleveland Clinic Foundation
Connecticut Children's Medical Center
Cooper University Hospital/Department of Pediatrics
Crozer-Chester Medical Center
Dartmouth Hitchcock Medical Center
Driscoll Children's Hospital
Duke University Hospital Medical Center
East Tennessee State University College of Medicine
Eastern Virginia Medical School
Emory University School of Medicine
Florida Hospital Medical Center Pediatric Residency
Flushing Hospital Medical Center
Grand Rapids Medical Education Partners/ MSU/Helen DeVos Children's Hospital
Greenville Hospital System/University of South Carolina
Hospital Episcopal San Lucas-Ponce School of Medicine
Hurley Medical Center/Michigan State University
Icahn School of Medicine at Mount Sinai
Indiana University School of Medicine
Inova Fairfax Medical Campus/Inova Children's Hospital
Jefferson Medical College/duPont Hospital for Children
Jersey Shore University Medical Center
John H. Stroger Jr. Hospital of Cook County
Johns Hopkins University School of Medicine
Kaiser Permanente Medical Group (Northern California)
Kaiser Permanente Southern California (Los Angeles)
Lehigh Valley Health Network/ University of South Florida
Loma Linda University Medical Center
Los Angeles County-Harbor UCLA Medical Center
Louisiana State University (New Orleans)
Louisiana State University Health Sciences Center (Shreveport)
Maine Medical Center
Marshall University School of Medicine
Massachusetts General Hospital
Mayo Clinic College of Medicine (Rochester)
Medical College of Georgia at Augusta University
Medical College of Wisconsin
Medical University of South Carolina College of Medicine
Memorial Health-Mercer University Medical Center (Savannah)
Monmouth Medical Center
Morehouse School of Medicine
Nationwide Children's Hospital/Doctors Hospital
Nationwide Children's Hospital/Ohio State University
Naval Medical Center (San Diego)
Naval Medical Center Portsmouth
New York Medical College at St. Joseph's Regional Medical Center
New York Medical College at Westchester Medical Center
New York Presbyterian Hospital (Cornell Campus)
Newark Beth Israel Medical Center
NSLIJHS/Hofstra North Shore-LIJ School of Medicine at the Cohen Children's Medical Center
NYU Pediatric Residency
Oregon Health and Science University
Our Lady of the Lake RMC, Baton Rouge
Palmetto Health Richland Hospital
Penn State University/Milton S. Hershey Medical Center
Phoenix Children's Hospital/Maricopa Medical Center
Pitt County Memorial Hospital/ East Carolina University/Vidant Medical Center
Rush University Medical Center
Rutgers New Jersey Medical School
Shands Hospital at the University of Florida
Southern Illinois University School of Medicine
St. Christopher's Hospital for Children
St. Louis University School of Medicine
Stanford University
Staten Island University Hospital
SUNY at Stony Brook
SUNY Upstate Medical University
Texas Tech University (Lubbock)
Tripler Army Medical Center
Tufts Medical Center
Tulane University School of Medicine
University of Alabama Medical Center - Children's Hospital
University of Arizona Health Sciences Center
University of Arkansas for Medical Sciences
University of California (Davis) Health System
University of California (Los Angeles) Medical Center
University of California (San Diego)
University of California (San Francisco)
University of Chicago Corner Children's Hospital
University of Colorado Denver Children's Hospital
University of Florida College of Medicine-Jacksonville
University of Hawaii John A Burns School of Medicine
University of Illinois College of Medicine at Chicago
University of Illinois College of Medicine at Peoria
University of Iowa Hospitals and Clinics
University of Kansas (Wichita) School of Medicine
University of Kansas School of Medicine
University of Louisville School of Medicine
University of Maryland Medical System
University of Massachusetts
University of Michigan Medical Center
University of Minnesota Medical School
University of Nebraska Medical Center College of Medicine/Creighton University
University of Nevada School of Medicine (Las Vegas)
University of New Mexico Children's Hospital
University of North Carolina Hospitals
University of Oklahoma College of Medicine (Tulsa)
University of Oklahoma Health Sciences Center (Oklahoma City)
University of Pittsburgh Children's Hospital
University of Rochester
University of South Alabama Children's and Women's Hospital
University of South Dakota Sanford Pediatric Residency Program
University of South Florida Morsani College of Medicine
University of Tennessee College of Medicine
University of Tennessee College of Medicine at Chattanooga
University of Texas at Austin Dell Medical School
University of Texas at Houston
University of Texas Medical Branch (Galveston)
University of Texas Southwestern Medical School
University of Toledo
University of Utah Medical Center
University of Vermont Pediatric Residency Program
University of Virginia Health System
University of Washington-Seattle Children's Hospital
University of Wisconsin Hospitals and Clinics
Vanderbilt University
Virginia Commonwealth University Children's Hospital of Richmond
Wake Forest University Medical School
Walter Reed National Military Medical Center
Western Michigan
William Beaumont Hospital
Winthrop-University Hospital
Wright State University
Yale-New Haven Medical Center
PAPERS
Abramson EK, Weiss P, Naifeh M, Stevenson, Duncan J, Rama J, Mauer E, Gerber L, T. Li ST, MD. (In press) Scholarly Activity Training during Pediatric Fellowship: A National Assessment. (Pediatrics, accepted)
Staples BB, Burke AE, Batra M, Kemper KJ, Schwartz A, Wilson PM, Schubert CJ, Mahan JD, Serwint JR. Pediatric Resident Burnout-Resilience Study Consortium. (2020). Burnout and association with resident performance as assessed by Pediatric Milestones: An exploratory study. Academic Pediatrics. https://doi.org/10.1016/j.acap.2020.08.006
Reed S, Mink R, Li ST. Utility of Residency Milestones Reported to Fellowship Directors: A National Survey of Pediatric Fellowship Program Directors. (2020) Acad Pediatr DOI: 10.1016/j.acap.2020.01.004.
Mink R, Herman BE, Carraccio C, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DEJ, Weiss P, Curran ML, Dammann CEL, High PC, Hsu D, Kesselheim JC, Mahan JD, McGann KA, Myers AL, Pitts S, Turner DA, Schwartz A. J. (2020) Agreement of Program Directors With Clinical Competency Committees for Fellow Entrustment. Med Educ Curric Dev. 2020 Aug 6;7:2382120520936613. doi: 10.1177/2382120520936613. eCollection 2020 Jan-Dec.
Schwartz A, Balmer DF, Borman-Shoap E, Chin A, Henry D, Herman BE, Hobday P, Lee JH, Multerer S, Myers RE, Ponitz K, Rosenberg A, Soep JB, West DC, Englander R. (In press). Shared Mental Models among Clinical Competency Committees in the Context of Time-Variable-Competency-Based Advancement to Residency. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003638
Schwartz A, King B, Mink R, Hicks PJ. (In press). The emergence and spread of practice-based medical education research networks. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003641
Hu K, Hicks PJ, Margolis M, Carraccio C, Osta A, Winward ML, Schwartz, A. (In press). Reported Pediatrics Milestones (Mostly) Measure Program, Not Learner Performance. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003644. [RIME New Investigator Award winner 2020].
Mink R, Herman B, Carraccio C, Aye T, Baffa J, Chess P, Fussell J, Sauer C, Stafford D, Weiss P, Curran M, Dammann C, High P, Hsu D, Kesselheim J, Mahan J, McGann K, Myers A, Pitts S, Turner D, Schwartz A. (In press). Agreement of Program Directors with Clinical Competency Committees for Fellow Entrustment. Journal of Medical Education and Curricular Development.
Kemper KJ, Schwartz A, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2020). Bullying, Discrimination, Sexual Harassment, and Physical Violence: Common and Associated with Burnout in Pediatric Residents. Academic Pediatrics. Published ahead of print: https://www.academicpedsjnl.net/article/S1876-2859(20)30080-2/fulltext
Schumacher DJ, West DC, Schwartz A, Li S, Millstein L, Griego EC, Turner TL, Herman BE, Englander R, Hemond J, Hudson V, Newhall L, McNeal Trice K, Baughn J, Giudice E, Famiglietti H, Gifford K, Carraccio C, for the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network General Pediatrics EPAs Study Group. (2020). Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities: A Multi-Site Study. JAMA Network Open, 3(1):e1919316.
Schumacher DJ, Schwartz A, Zenel JA, Black NP, Ponitz K, Blair R, Traba CM, Poynter S, King B, Englander R, Rosenberg A, Patel D, Smith-King C, OConnor M, Gonzalez del Rey J, Lavoie S, Borman-Shoap E, Carraccio C, for the APPD LEARN General Pediatrics EPAs Study Group.. (In press). Vignette Assignments at Entrustment and Graduation: Integrating EPAs AND Milestones to Improve Learner Assessment. Academic Medicine.
Schumacher DJ, Poynter S, Burman N, Elliott SP, Barnes M, Gellin C, Gonzalez del Rey J, Sklansky D, Thoreson L, King B, Schwartz A, APPD LEARN CCC Study Group. (In press). Concordance of Clinical Competency Committee and Program Director Recommendations for Resident Supervisory Roles. Academic Pediatrics.
Kemper KJ, Wilson P, Schwartz A, Mahan JD, Batra M, Staples B, McClafferty H, Schubert CJ, Serwint JR, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (In press). Burnout in Pediatric Residents: Comparing Brief Screening Questions to the Maslach Burnout Inventory. Academic Pediatrics.
Reed SR, Kemper KJ, Schwartz A, Batra M, Staples BB, Serwint JR, McClafferty H, Schubert CJ, Wilson PM, Rakowsky A, Chase M, Mahan JD. (In press). Variability of Burnout and Stress Measures in Pediatric Residents: an Exploratory Single Center Study from the Pediatric Resident Burnout - Resilience Study Consortium. Journal of Evidence-Based Integrative Medicine.
Schumacher DJ, Parrigan A, Bartlett KW, King B, Calaman S, Garfunkel LC, Elliott SP, Frohna JG, Schwartz A, Michelson C, and the APPD CCC Study Group. (In press). Key Factors for Recommending Whether Residents May Serve as Supervisors: A National Study of Clinical Competency Committee Members. Academic Medicine.
Schumacher DJ, Bartlett KW, Elliott SP, Michelson C, Sharma T, Garfunkel LC, King B, Schwartz A, and the APPD LEARN CCC Study Group. (In press). Milestone Ratings and Supervisory Role Categorizations Swim Together, but is the Water Muddy? Academic Pediatrics.
Schumacher DJ, King B, Barnes MM, Elliott S, Gibbs K, McGreevy JF, Gonzalez del Rey J, Sharma T, Michelson C, Schwartz A, and the APPD LEARN CCC Study Group. (In press). The Influence of the Review Process Utilized by Clinical Competency Committee Members on Their Summative Assessment Decisions of Residents. Journal of Graduate Medical Education.
Hicks PJ, Margolis MJ, Carracio CL, Clauser BE, Donnelly K, Fromme HB, Gifford KA, Poynter SE, Schumacher DJ, Schwartz A, PMAC Module 1 Study Group. (2018). A novel workplace-based assessment for competency-based decisions and learner feedback. Medical Teacher, 40:11, 1143-1150.
Abramson EL, Naifeh M, Stevenson M, Mauer E, Hammad H, Gerber L, Li S. (In press). Scholarly Activity Training During Residency: Are we Hitting the Mark? A National Assessment of Pediatric Residents. Academic Pediatrics.
Kemper KJ, McClafferty H, Wilson P, Serwint JR, Batra M, Mahan JD, Schubert CJ, Staples BB, Schwartz A, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2018). Do Mindfulness and Self-Compassion Predict Burnout? Academic Medicine. Published ahead of print.
Park YS, Hicks PJ, Carraccio CL, Margolis MJ, Schwartz A, for the PMAC Module 2 Study Group. (2018). Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth from Ten Pediatrics Residency Programs. Academic Medicine, 93(11): S21-S29.
Schumacher DJ, Michelson C, Poynter S, Barnes MB, Li S, Burman N, Sklansky DJ, Thoreson L, Calaman S, King B, Schwartz A, and the APPD LEARN CCC Study Group. (2018). Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns. Medical Teacher, 40(1): 70-79.
Mink RB, Schwartz A, Herman BE, Turner DA, Curran ML, Myers A, Hsu DC, Kesselheim JC, Carraccio CL, for the Subspecialty Pediatrics Investigator Network (SPIN). (2018). Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities. Academic Medicine, 93:2, 283-291.
Mink R, Schwartz A, Carraccio C, High P, Dammann C, McGann K, Kesselheim J, Herman B and the Steering Committee of the Subspecialty Pediatrics Investigator Network. (2018). Creating the Subspecialty Pediatrics Investigator Network (SPIN). Pediatrics, 192:3-4.e2. doi: 10.1016/j.jpeds.2017.09.079.
Kesselheim JC, Schwartz A, Boyer D, for the APPD LEARN Study Group on Education and Service. (2017). Integrating education and service in Pediatric residency training: Results of a national survey. Academic Pediatrics, 17:8, 907-914
Li ST, Tancredi DJ, Schwartz A, Guillot A, Burke A, Trimm RF, Guralnick S, Mahan JD, Gifford KA, for the APPD LEARN Validity of Resident Self-Assessment Group. (In press). Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. Academic Medicine
Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A, for the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). Assessment of Pediatric interns and sub-interns on a subset of Pediatric Milestones. Academic Medicine, 92:6, 809-819.
Li ST, Tancredi DJ, Schwartz A, Guillot AP, Burke AE, Trimm F, Guralnick S, Mahan JD, Gifford KA, for the APPD LEARN Validity of Resident Self-Assessment Group.. (2017). Competent for unsupervised practice: Use of Pediatric residency training milestones to assess readiness. Academic Medicine, 92:3, 385-393.
Schwartz A, Margolis M, Multerer S, Haftel H, Schumacher DJ, and the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). A multi-source feedback tool for measuring a subset of Pediatrics Milestones. Medical Teacher, 38:10.
Kesselheim JC, Schwartz A, Belmonte F, Boland KA, Poynter S, Batra M, for the APPD LEARN Study Group on Social Media and Professionalism. (2016). A National Survey of Pediatric Residents' Professionalism and Social Networking: Implications for Curriculum Development. Academic Pediatrics, 16:2, 110-114
Hicks PJ, Margolis M, Poynter S, Chaffinch C, Tenney-Soeiro R, Turner T, Waggoner-Fountain L, Young R, Clyman SG, Schwartz A, for the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). The Pediatrics Milestones Assessment Pilot: Development of Workplace-based Assessment. Academic Medicine, 91:5, 701-709.
Tocco D, Jain AV, Baines H. (2014). The pediatrics milestone pilot project: perspectives of current pediatric residents. Acad Pediatr, 14(2 Suppl), S8-9.
Schwartz A, Young R, Hicks PJ; for APPD LEARN. (2014). Medical Education Practice-Based Research Networks: Facilitating collaborative research. Medical Teacher, 38:1, 64-74.
PRESENTATIONS
Mink R, Carraccio C, Herman B, Weiss P, Turner D, Pitts S, Moffat M, Kesselheim J, Czaja A, Schwartz A. Relationship Between Level of Supervision at the End of Residency Compared with Early Fellowship for the Entrustable Professional Activities (EPAs): A Pilot Study. Pediatric Academic Societies, 2020. (Accepted as platform presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202455.2.)
Park YS, Hicks PJ, Carraccio C, Margolis M, Schwartz A, for the PMAC Module 2 Study Group. (2018, November). Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth. RIME oral presentation. Annual Meeting of the Association of American Medical Colleges, Austin, TX.
Batra M, Mahan J, Schubert C, Wilson P, Staples B, Serwint JR, McClafferty H, Schwartz A, Kemper K.(2018, March). Burnout in Pediatric Residents: Findings From a National Longitudinal Survey. Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Lauden S, Wilson P, Fuast M, Mahan J, Batra M, Shubert C.(2018, March). Burnout & Who's Who in Pediatric Global Health: A Look at National Trends. Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Abramson E, Stevenson M, Naifeh M, Hammad H, Gerber L, LI ST. (2017, May). Scholarly Activity Training During Residency: A National Survey of Pediatric Residents and Program Directors. Pediatric Academic Societies, San Francisco, CA.
Gifford KA, Burke A, Li ST, Tancredi DJ, Trimm F, Schwartz A, Mahan J, Guralnick S, Guillot AP. (2017, May). Do Milestones Reported to the ACGME Better Describe Residents' Performance or their Programs' Assessment Practices? Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Bartlett K, Michelson C, Elliott S, Gibbs K, McGreevy J, Li ST, Sklansky DJ, Gonzalez del Rey J, Poynter S, Sharma T, Burman N, Gellin C, Thoreson L, Calaman S, Garfunkel L, Frohna J, Scott-Vernaglia S, Barnes M, Schwartz A. (2017, May). Association Between Milestone Assignments and Recommended Supervision Categorizations Among Clinical Competency Committee Members. Pediatric Academic Societies, San Francisco, CA.
Batra M, Mahan JD, Schubert CJ, Wilson PM, Staples BB, Serwint JR, McClafferty H, Schwartz A, Kemper KJ, for the Pediatric Residency Burnout and Resilience Consortium. (2017, May). Burnout in Pediatric Residents: A National Survey To Inform Future Interventions. Pediatric Academic Societies, San Francisco, CA.
Staples BB, Batra M, Mahan J, Schwartz A, Wilson P, McClafferty H, Schubert C, Kemper K, Serwint JR. (2017, May). Burnout Status and Milestone Performance in Pediatric Residents. Pediatric Academic Societies, San Francisco, CA.
Batra M, Mahan J, Shubert C, Wilson P, Staples BB, Serwint JR, McClafferty H, Schwartz A, Kemper K. (2017, April). Burnout in Pediatric Residents: A National Survey to Inform Future Interventions. Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA. Winner: 2017 Research Award.
Schwartz A. (2016, December). Milestones-based research at APPD LEARN. ACGME Milestones Summit, Chicago, IL.
Mahan JD, Serwint JR, Schubert CJ, Wilson PM, Staples BB, McClafferty, Schwartz A, Kemper KJ, Batra M. (2016, November) Building a national consortium (The Pediatric Resident Burnout-Resilience Study Consortium) to understand and promote pediatric wellness. The ACGME Second Symposium on Physician Well-Being: Commitment to Change. Chicago, IL.
Schwartz, A. (2016, August). APPD LEARN: Building a Pediatric Education Research Network. AMSPDC Webinar
Kesselheim JC, Boyer D. (2016, April). Balancing education and service in Pediatric residency training: Results of a national survey. Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA. Winner: 2016 Research Award
Mink RB, Schwartz A, Herman BE, Curran ML, Hsu DC, Kesselheim JC, Myers AL, Turner DA, Carraccio CL. (2016, April). Creation and validation of entrustment scales for the common pediatric subspecialty entrustable professional activities (EPAs). Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Li S, Burke AE, Guillot A, Schwartz A, Gifford KA, Trimm F, Mahan JD, Guralnick S, Tancredi DJ. (2016, April). Pediatric Resident Minimum Milestone Expectations by Level of Training. Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Li ST, Tancredi DJ, Schwartz A, Burke AE, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K. (2016, February). Pediatric Resident Minimum Milestone Expectations by Level of Training. Platform presentation at Pediatric Academic Societies (PAS) Annual National Conference, Baltimore, MD.
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DE, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Schwartz A, for the Subspecialty Pediatrics Investigators Network (SPIN). (2016, February). The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. ACGME Annual Meeting, National Harbor, MD.
Andriole D, Jeffe D, Reilly K, Schwartz A, Bostrom D. (2015, November). Bridging Data Across Divides: Challenges of Aggregating Data from Multiple Organizations. Learn, Serve, Lead: 2015 AAMC, Baltimore, MD.
Li S, Tancredi D, Schwartz A, Burke A, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K, for APPD LEARN validity of resident self-assessment using pediatric milestones group. (2015, April). Which Sub-Competencies Best Identify Overall Unsatisfactory/ Marginal Resident Performance? Pediatric Academic Societies, San Diego, CA.
Li S, Tancredi D, Schwartz A, Burke A, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K, for APPD LEARN validity of resident self-assessment using pediatric milestones group. (2015, April) How Well Do Resident Milestone Self-Assessments Correlate with Clinical Competency Committee Milestone Assessments? Pediatric Academic Societies, San Diego, CA.
Gifford K, Li S, Trimm RF, Schwartz A, Gurlanick S, Tancredi DJ, Burke A, Mahan J, for For APPD LEARN validity of resident self-assessment group. (2015, April). How Do Clinical Competency Committees Function in Pediatric Residency? Pediatric Academic Societies, San Diego, CA.
Li s, Tancredi DJ, Trimm RF, Guillot A, Mahan JD, Burke A, Gifford KA, Schwartz A, Guralnick S, for APPD LEARN validity of resident self-assessment. (2015, March). How well do resident milestone self-assessments correlate with clinical competency committee milestone assessments? APPD Annual Spring Meeting, Orlando, FL. Winner, APPD Research Award
Tenney-Soeiro R, Ronan JC, Weiss AK, Young D, Clyman SG, Schwartz A, Rose KM, Hicks PJ. (2013, April). Does the Quantity and Quality of Comments Provided on Intern and Sub-Intern Pediatrics Milestones Assessment Instrument Vary Based on Role of the Rater? Platform Presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Turner T, Hicks P, Rose K, Clyman S, Schwartz A, Poynter S, Haftel H, Ronan JC, Waggoner-Fountain L, Caputa G. (2013, April). Sub-Intern and Intern Performance Level on Pediatric Milestones. Platform Presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Chaffinch C, Poynter S, Turner T, Hicks P, for the APPD LEARN-NBME Milestones Assessment Group. (2013, February). Learner Feedback on the Pediatrics Milestones Assessment Project. Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Poynter S, Turner T, Chaffinch C, Hicks P, for the APPD LEARN-NBME Milestones Assessment Group. (2013, Februray). Learner Feedback on the Pediatrics Milestones Assessment Project. 2013 ACGME Annual Educational Conference, Orlando, FL.
Schwartz A. (2012, November). Assessment of Pediatrics Milestones and Development of a Collaborative Research Network for GME. Invited presentation, DeWitt Baldwin Seminar Series, ACGME, Chicago, IL.
POSTERS
Mink R, Carraccio C, Herman B, High P, Hsu D, Langhan M, Schwartz A. Factors Important in Determining Simple versus Complex Cases for the Pediatric Subspecialty Entrustable Professional Activity (EPA) Levels of Supervision, Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/ 2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.356.)
Mink R, Carraccio C, Herman B, Chess P, Dammann C, Hsu D, Kesselheim J, Moffat M, Myers A, Turner D, Czaja A, Schwartz A. Have Level of Supervision Ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) Changed Over Time? Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.369)
Smith S, Mensah-Bonsu N, Lopez M, Voigt R, Mink R, Carraccio C, Schwartz A, Herman B, Fussell J. (2018, September). Are Graduating Developmental-Behavioral Pediatrics Fellows Ready for Clinical Practice? A SPIN study. Poster Presentation at Society for Developmental and Behavioral Pediatrics 2018 Annual Meeting, Anaheim, CA.
Lopez M, Mensah-Bonsu N, Smith S, Voigt R, Mink R, Carraccio C, Schwartz A, Herman B, Fussell J. (2018, September). Using Entrustable Professional Activities to Assess Graduation Readiness in Developmental Behavioral Pediatrics: A SPIN Study. Poster Presentation at Society for Developmental and Behavioral Pediatrics 2018 Annual Meeting, Anaheim, CA.
Hu K, Hicks PJ, Margolis MJ, Carraccio CL, Osta AD, Schwartz A. (2018, August). Inter-program variance in ACGME Reportable Pediatrics Milestones. Can Residents' ACGME PM levels be compared among programs? Poster presentation at the 2018 Annual Meeting of the Association for Medical Education in Europe (AMEE 2018), Basel, Switzerland.
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carracio C.(2018, March). Expected Fellow Minimum Level of Supervision at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. Poser Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Mink R, Carracio C, Herman B, Baffa J, Fussell J, Kesselheim J, McGann K, Myers A, Sauer C, Stafford D, Weiss P, Schwartz A.(2018, March). Pediatric Fellows are Not Expected to Achieve "Entrustment" at Graduation from Fellowship. Poster Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Abramson E, Stevenson M, Naifeh M, Mauer E, Gerber L, Li ST. (2018, March). Scholarly Activity Training During Residency: How Well Aligned are Program Directors and Residents? Poster Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Hicks P, Margolis M, Carraccio C, Clauser B, Winward M, Schwartz A, and the PMAC Module 1 Study Group. (2017, May). Pediatrics Milestones Assessment Collaborative: Development and implementation of an authentic workplacebased assessment system (WBAS). Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Barnes M, Bartlett K, Burman N, Calaman S, Elliott S, Frohna J, Garfunkel L, Gellin C, Gibbs K, Gonzalez del Rey J, Michelson C, McGreevy J, Li ST, Poynter S, Scott-Vernaglia S, Sharma T, Sklansky DJ, Thoreson L, Schwartz A. (2017, May). Concordance of Clinical Competency Committee and Program Director Recommended Supervision Categorization. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Thoreson L, Sklansky DJ, Sharma T, Scott-Vernaglia S, Poynter S, Li ST, McGreevy J, Michelson C, Gonzalez del Rey J, Gibbs K, Gellin C, Garfunkel L, Frohna J, Elliott S, Calaman S, Burman N, Bartlett K, Barnes M, Schwartz A. (2017, May). Key Factors for Recommending a Resident May Serve as a Supervisor: A National Study of Clinical Competency Committee Members. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Reed S, Batra M, Kemper K, Staples BB, Serwint JR, McClafferty H, Schubert C, Wilson P, Schwartz A, Mahan J. (2017, May). How Variable are Burnout and Stress in Pediatric Residents? An Exploratory Single Center Study from the Pediatric Resident Burnout-Resilience Study Consortium. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Staples BB, Batra M, Mahan J, Schwartz A, Wilson P, McClafferty H, Schubert C, Kempter K, Serwint JR. (2017, April). Burnout Status and Milestone Performance in Pediatric Residents. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Schumacher DJ, King B, Barnes M, Bartlett K, Burman N, Calaman S, Elliott S, Frohna J, Garfunkel L, Gellin C, Gibbs K, Gonzalez del Rey J, Michelson C, McGreevy J, Li ST, Poynter S, Scott-Vernaglia S, Sharma T, Sklansky DJ, Thoreson L, Schwartz A. (2017, April). Concordance of Clinical Competency Committee and Program Director Recommended Supervision Categorization. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Gifford KA, Li ST, Schwartz A, Burke AE, Tancredi DJ, Guralnick S, Guillot AP, Trimm F, Mahan JD. (2017, April). Do Milestones Reported to the ACGME Describe Residents Performance or Their Programs Assessment Practices? Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Schumacher DJ, King B, Thoreson L, Sklansky DJ, Sharma T, Scott-Vernaglia, Poynter S, Li ST, McGreevy J, Michelson C, Gonzalez del Rey J, Gibbs K, Gellin C, Garfunkel L, Frohna J, Elliott S, Calaman S, Burman N, Bartlett K, Barnes M, Schwartz A. (2017, April). Key Factors for Recommending a Resident May Serve as a Supervisor: A National Study of Clinical Competency Committee Members. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Abramson E, Stevenson M, Naifeh M, Hammad H, Gerber L, Li ST. (2017, April). Scholarly Activity Training During Residency: A National Survey of Pediatric Residents and Program Directors. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Gifford KA, Burke AE, Trimm F, Mahan JD, Schwartz A, Li ST, Guillot AP, Guralnick S. (2016, April). Learner self-assessment as a contribution to competency assessment. Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Schwartz A, Dammann CE, High PC, McGann KA, Herman BE for SPIN. Creation of a pediatric subspecialty educational research network. (2016, April) Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DE, Weiss P, Schwartz A for SPIN. (2016, April). Do fellowship program directors (FPD) and clinical competency committees (CCC) agree in fellow entrustment decisions? Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Turner DA, Myers A, Kesselheim JC, Hsu DC, Curran ML, Schwartz A, for the Subspecialty Pediatrics Investigators Network (SPIN). (2016, February). Reliability and validity of a supervision scale for the common pediatric subspecialty entrustable professional activities. Poster presentation. ACGME Annual Meeting, National Harbor, MD.
Balmer DF, Hicks PJ, Carraccio C, Schwartz A. (2016 February). Bridging the Gap between Educational Research and Practice: The Normalization Process Theory Poster presentation. Pediatric Academic Societies, Baltimore, MD.
Schwartz A., Young, R., Hicks, P.J. (2014, November). Medical Education Research Networks: The case of APPD LEARN. Poster presentation at the 2014 American Association of Medical Colleges Medical Education Meeting, Chicago, IL.
Chaffinch CN, Poynter SE, Turner T, Schwartz A. (2013, April). Learner Feedback on the Pediatrics Milestones Assessment Project. Poster presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
McGregor R, Burke A, Carraccio C, Haftel H, Hicks P, Schwartz A. (2012, November). APPD LEARN: An Educational Research Network in Pediatric GME Digitized Poster Presentation at Group on Educational Affairs, AAMC Annual Meeting, San Francisco, CA.
APPD PRIME
Studies Now Recruiting
COMLEX Scores and Performance on ABP Certifying Examination
PI: Elizabeth Rodriguez-Lien
Synopsis: Retrospective data collection to investigate the relationship between COMLEX scores and performance on the ABP certifying examination.
Status: Recruiting
Using text messaging to help learners develop self-regulated learning skills
PI: Kimberly Gifford
Synopsis: Randomized controlled trial of text messaging to teach and assess self-regulated learning.
Status: Recruiting
Sustainable Pediatric Nephrology Workforce Project (SUPERPOWER)
PI: Susan Halbach
Synopsis: Further understand resilience and burnout as well as factors contributing to physician wellness in both pediatric nephrology fellows and faculty
Status: Recruiting
Identifying ERAS Filters that Predict Medical Student Success in Pediatric Residency
PI: Jessica DeBord
Synopsis: Identify medical school metrics reported via the Electronic Residency Application Service (ERAS) that predict success during pediatric residency. Involves retrospective data collection of medical school metrics.
Status: Recruiting
Promoting Med-ed Insight into Supportive Environments (PROMISE)
PI: Jyothi Marbin
Synopsis: Describe the experiences of Under-Represented Minority residents around feelings of belonging, inclusion and community. Compare the relationship between URM resident sense of belonging and wellness. Involves surveying residents.
Status: Recruiting
Studies Currently Ongoing
Do pediatric boot camps improve performance during early internship
PI: Molly Rideout
Collaborators: COMSEP
Synopsis: The specific aim of this project is to investigate whether participation in a pediatric-specific boot camp improves performance during initial inpatient Pediatrics or NICU rotations and after 6 months of internship, as reflected on intern performance ratings assessed with Pediatric Milestones.
Status: Collecting data.
Childbearing and parenthood decisions in Pediatrics
PI: Kelly Dundon
Synopsis: Characterize the variability in use and length of parental leave in Pediatrics residency programs and associations with educational and parenting outcomes.
Status: Collecting data.
Impact of Academic Half Days on Board Pass Rates
PI: Emily Borman-Shoap
Synopsis: Determine if there is an association between the use of academic half-days in pediatric residency programs and passage of ABP examination in the first attempt.
Status: Data analysis in progress.
Defining Key Factors in Burnout and Resilience in Pediatric Residents and the Relationship to Performance
PIs: John Mahan, MD and Maneesh Batra, MD for The Pediatric Resident Burnout-Resilience Study Consortium
Synopsis: Create and support a consortium to:
- Describe the epidemiology and relationships between burnout, resilience, empathy, and confidence in providing compassionate care in pediatric and medicine-pediatric (P/M-P) residents.
- Define the natural history of these parameters over time in P/M-P residents during training
- Identify modifiable factors that increase or decrease the risk of developing burnout and promoting positive wellness factors
- Develop and test different interventions (such as in-person seminars, on-line mind-body skills training, and individualized curricula such as global health electives) to affect burnout, resilience, empathy, compassion and wellness
Status: Data collection and analyses in progress.
Past Studies
How trainees perform on general pediatric EPAs using a rating scale of level of supervision leading up to entrustment
PIs: Dan Schumacher, MD, MEd; Carol Carraccio, MD
Collaborators: American Board of Pediatrics
Synopsis:
- Determine expectations for level of supervision needed by the majority of trainees over the course of and at the completion of residency training with the intention of using data to guide the value of the current list of EPAs in learner assessment.
- Assess the feasibility of using an educational intervention to meet Part 4 MOC requirements for a quality improvement project.
Status: Data collection complete
Assessing the Association between Entrustable Professional Activities (EPAs), Competencies, and Milestones in the Pediatric Subspecialties
Collaborators: APPD LEARN, Council of Pediatric Subspecialties, American Board of Pediatrics
Synopsis: The specific objectives are 1) to determine if there is a specific milestone level that must be reached for each of the subcompetencies within a specific EPA to judge that a fellow is deemed entrustable, 2) to compare the milestone level at which fellows are deemed entrustable across the pediatric subspecialties and to determine if any specific subcompetencies are more influential in the entrustment decision than others and 3) to compare the initial overall impression of fellow level of entrustment for a given EPA made by the Fellowship Program Director with that determined by the Clinical Competency Committee (CCC) after the milestone levels for that EPA are assigned.
Status: Data collection complete
Pediatrics Milestones Assessment Collaborative (PMAC)
Collaborators: APPD LEARN, American Board of Pediatrics, National Board of Medical Examiners
Project Director: Patricia Hicks, MD, MHPE
Synopsis: Develop a national Pediatrics Milestones-based Assessment System - an assessment system that will provide meaningful, feasible assessment evidence that can be used by the program leadership and clinical competency committee members to guide decisions for progression through increasing levels of responsibility and independence.
Status: Data collection complete. Analyses in progress.
Characteristics of Clinical Competency Committee (CCC) Member Milestone Level Assignments and Recommended Advancement Decisions
PI: Dan Schumacher, MD, MEd
Synopsis: Studies the relationship between individual CCC member judgments and and reported milestones and advancement decisions. Investigates whether program and CCC member factors moderate that relationship.
Status: Data collection complete, analysis in progress.
Validity of resident self-assessment using Pediatrics Milestones
PIs: Su-Ting Li (UC Davis) and Kimberly Gifford (Dartmouth-Hitchcock)
Synopsis: This study seeks to describe pediatric resident self-assessment of competence using the Pediatrics Milestones for 21 ACGME-selected sub-competencies, and characterize and explain variation in agreement between pediatric resident self-assessment, external assessments based on Pediatrics Milestones, and overall summative assessments of resident performance.
Status: Data collection complete.
Striving for the Optimal Balance Between Service and Education
PIs: Debra Boyer, MD (Harvard), Jennifer Kesselheim, MD (Harvard)
Synopsis: Survey residents and program directors to (1) establish each cohorts' assessment of the balance of service and education in their training programs, (2) measure program directors' and residents' perceptions of the amount of service and education inherent in common scenarios encountered during pediatric residency training, and (3) elucidate definitions of service and education, endorsed by both program directors and trainees, that can serve as a useful foundation for curriculum development during residency training
Synopsis: Data collected. Analyses in process.
New Professionalism Challenges in Medical Training: An Exploration of Social Networking
PI: Jennifer Kesselheim (Harvard)
Synopsis: This study surveys pediatric trainees to explore their perceptions of and experiences with social networking sites and training program policies around social media.
Status: Data collection is complete for this study.
Assessment of Pediatrics Milestones Pilot
PI: Patricia Hicks, MD (CHOP); Steve Clyman, PhD (NBME); Alan Schwartz, PhD (UIC)
Synopsis: In partnership with the NBME, this study seeks to test the feasibility of a set of assessment instruments constructed from nine Pediatrics Milestones. Instruments are designed to inform decisions about readiness to serve in the inpatient pediatric wards, based on ratings of direct observations recorded on handheld devices. Study instruments include: structured clinical observations for rounds and history taking; a multisource feedback instrument; and, Pediatrics Milestones Classification assessment form. Study learners include interns and sub-interns, with faculty, residents, and nurses serving as raters. Features include collection of de-identified learner data in APPD LEARN, individualized learner feedback, faculty development materials for rater and feedback provider training. Outcomes from this study will be used to inform further development of the Pediatrics Milestones and APPD LEARN.
Status: Data collection ended June 2013. Data analyses and manuscript writing in progress.
The New Duty Hour Regulations--Were Changes Necessary? A Survey of Pediatric Program Directors
PI: Hilary Haftel, MD (University of Michigan)
Synopsis: This study seeks to identify what, if any, changes in Pediatric Residency Program structure were necessary to achieve compliance with the new DHR by July 2011. This specifically includes any changes to the amount or periods of resident work hours, changes in supervision, and changes in external or internal moonlighting
Status: Data collection complete.
APPD SPIN
Studies Now Recruiting
National Needs Assessment of Telemedicine in Pediatric Fellowship Training
PI: Nicole Paradise Black, MD, MEd
Synopsis:
The study aims to ascertain 1) current fellow trainee exposure to telemedicine; 2) fellowship program use of formalized curriculum for telemedicine instruction and exposure; 3) barrier to implementation of telemedicine; 4) fellowship director interest in curriculum development and national collaboration
Studies Currently Ongoing
Utilization of EPAs in the Pediatric Subspecialties and Barriers to their Use
PI: Rich Mink, MD, MACM
Synopsis:
Primary Aim:
- To identify the facilitators and barriers to using EPAs to assess pediatric fellows
Secondary Aims:
- To determine how EPAs are currently being utilized in pediatric fellowships
- To examine the process used by CCCs in assigning the EPA level of supervision for their fellows, the time the CCC devotes to making the decision and the information used to make the decision
- To gauge FPD opinions about the value of using EPAs versus milestones to assess fellow progress in training
- To describe how a FPD’s assignment of level of supervision for the various EPAs factors into the determination of fellow readiness for graduation
- To compare and contrast responses among the subspecialties
Longitudinal Evaluation of the Required Level of Supervision for Pediatric Fellows
PI: Rich Mink, MD, MACM
Synopsis:
The specific aims of this project are:
- To obtain validity evidence for the subspecialty‐specific and scholarship EPA level of supervision scales
- To determine the developmental progression of levels of supervision for all EPAs (common and subspecialty‐specific) during the 3 years of fellowship and
- To investigate whether graduating pediatric fellows are meeting the previously defined minimum levels of supervision
- To determine if the time in training at which pediatric fellows meet the previously defined minimum levels of supervision differ among the subspecialties
- To determine if there is a level of supervision for any EPA below which remediation for poor performance is performed
- For learners in which remediation was initiated, to determine if the level of supervision rating guided the decision to do so.
- To compare level of supervision assessments made by the Clinical Competency Committee (CCC) with those of the pediatric fellow
- For the Scholarship EPA, to examine the association between the level of supervision rating and the milestone level of the competencies mapped to the EPA
- To investigate the thought process of the rater in deciding what constitutes a simple versus complex case
- For the 5 EPAs that cross the generalist to subspecialist roles, to compare the level of supervision assigned to a resident at graduation to the level assigned at first assessment as a fellow
Status: Collecting data.
Past Studies
Scholarly Activity Training During Fellowship
PI: Erika Abramson, MD, MS
Synopsis:
The specific aims for this project are:
- Understand resources available across fellowship programs to support fellow scholarly activity and identify perceived barriers to effective scholarly activity training
- Understand fellowship program director attitudes toward scholarly activity training during fellowship generally and satisfaction with current fellow experiences in their own program
- Identify factors associated with a) scholarly productivity during fellowship and b) fellowship director satisfaction with scholarly acclivity training in their program
Residency Milestones in Fellowship Programs
PI: Suzanne Reed, MD
Synopsis:
The purpose of this study is to:
- Determine the extent to which fellowship programs utilize residency milestones for their newly entering fellows.
- Determine the perceived value of residency milestones in fellowship programs for use with newly entering fellows.
Level of Supervision for Graduating Fellows
PI: Rich Mink, MD, MACM
Synopsis:
For each of the 7 EPAs, determine the opinion of fellowship program directors (FPD) about:
- The minimum level of supervision a fellow must achieve to complete fellowship
- Whether a fellow should be allowed to graduate if this level is not met
Assessing the Association Between EPAs, Competencies and Milestones in the Pediatric Subspecialties
PIs: Bruce Herman, MD and Rich Mink, MD, MACM
Synopsis:
- For six of the seven common EPAs, to determine if there is a specific milestone level at which a fellow is deemed entrustable.
- For six of the seven common EPAs, to compare the milestone level at which fellows are deemed entrustable across the pediatric subspecialties and to determine if any specific competencies are more influential in the entrustment decision than others.
- For six of the seven common EPAs, to compare the initial overall impression of fellow level of entrustment made by the Fellowship Program Director with that determined by the Clinical Competency Committee after the milestone levels are assigned
Status: Data collection complete
PRESENTATIONS
Mink R, Carraccio C, Herman B, Weiss P, Turner D, Pitts S, Moffat M, Kesselheim J, Czaja A, Schwartz A. Relationship Between Level of Supervision at the End of Residency Compared with Early Fellowship for the Entrustable Professional Activities (EPAs): A Pilot Study. Pediatric Academic Societies, 2020. (Accepted as platform presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202455.2.)
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carraccio C for the Subspecialty Pediatrics Investigator Network. Setting the Minimum Level of Supervision for Fellows at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. Platform presentation in the APA Presidential Plenary Session, Pediatric Academic Societies, Toronto, Canada, 2018.
Mink R, Carraccio C, Herman B, Aye T, Chess P, Curran M, Dammann C, High P, Hsu D, Mahan J, McGann K, Srivastava S, Turner D, Weiss P, Schwartz A for the Subspecialty Pediatrics Investigator Network. Is the Expected Minimum Level of Supervision for Graduating Pediatric Fellows for the Common Entrustable Professional Activities Similar Across the Subspecialties? Platform presentation in the Medical Education 1: Top Oral Abstracts Session, Pediatric Academic Societies, Toronto, Canada, 2018.
Mink RB, Schwartz A, Herman BE, Curran ML, Hsu DC, Kesselheim JC, Myers AL, Turner DA, Carraccio CL. Creation and validation of entrustment scales for the common pediatric subspecialty entrustable professional activities (EPAs). Oral presentation at the annual spring meeting of the Association of Pediatric Program Directors, April, 2016, New Orleans.
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DE, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Schwartz A for SPIN. The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. Oral and poster presentation at the annual education meeting of the Accreditation Council for Graduate Medical Education, February, 2016, National Harbor, Maryland.
POSTERS
Mink R, Carraccio C, Herman B, High P, Hsu D, Langhan M, Schwartz A. Factors Important in Determining Simple versus Complex Cases for the Pediatric Subspecialty Entrustable Professional Activity (EPA) Levels of Supervision, Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/ 2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.356.)
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carraccio C for the Subspecialty Pediatrics Investigator Network. Expected Fellow Minimum Level of Supervision at Graduation for The Common Pediatric Subspecialty Entrustable Professional Activities. Poster Presentation at the Association of Pediatric Program Directors, Atlanta, GA, 2018.
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carraccio C for the Subspecialty Pediatrics Investigator Network. Expected Fellow Minimum Level of Supervision at Graduation for The Common Pediatric Subspecialty Entrustable Professional Activities. Poster Presentation at the Association of Pediatric Program Directors, Atlanta, GA, 2018.
Mink RB, Carraccio CL, Schwartz A, Dammann CE, High P, Kesselheim JC, McGann K, Herman BE. Establishing a medical education research network for the pediatric subspecialties. Poster presentation at annual spring meeting of the Pediatric Academic Societies, May, 2016, Baltimore, Maryland.
Mink RB, Carraccio CL, Herman BE, Dammann C, Mahan J, Pitts S, Sauer CG, Schwartz A. Variability in fellow entrustment across the pediatric subspecialties for the common pediatric subspecialty entrustable professional activities (EPAs). Poster presentation at annual spring meeting of the Pediatric Academic Societies, May, 2016, Baltimore, Maryland.
Mink RB, Carraccio CL, Schwartz A, Dammann CE, High PC, McGann KA, Herman BE for SPIN. Creation of a pediatric subspecialty educational research network. Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, April, 2016, New Orleans.
Mink RB, Carraccio CL, Herman BE, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DE, Weiss P, Schwartz A for SPIN. Do fellowship program directors (FPD) and clinical competency committees (CCC) agree in fellow entrustment decisions? Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, April, 2016, New Orleans.
Mink RB, Schwartz A, Herman BE, Turner DA, Myers AL, Kesselheim JC, Hsu DC, Curran ML, Carraccio CL for SPIN. Reliability and validity of a supervision scale for the common pediatric subspecialty entrustable professional activities. Poster presentation at the annual education meeting of the Accreditation Council for Graduate Medical Education, February, 2016, National Harbor, Maryland.
Each subspecialty may designate up to two representatives who serve on the SPIN Steering Committee and are responsible for recruitment and communication within their subspecialty.
| Subspecialty | Representative 1 | Representative 2 |
| Adolescent Medicine | Sarah Pitts | |
| Cardiology | Brad Robinson | Shubhika Srivastava |
| Child Abuse | Bruce Herman | Mary Moffatt |
| Critical Care Medicine | Angela Czaia | Don Boyer |
| Developmental and Behavioral Pediatrics | Jill Fussell | Pam High |
| Emergency Medicine | Deborah Hsu | Melissa Langhan |
| Endocrinology | Diane Stafford | Tandy Aye |
| GI | Cary Sauer | |
| Hematology-Oncology | Jennifer Kesselheim | Mark Atlas |
| Infectious Diseases | Angie Myers | Kammy McGann |
| Neonatology | Christiane Dammann | Patricia Chess |
| Nephrology | John Mahan | Susan Halbach |
| Pulmonary Medicine | Pnina Weiss | Jennifer Rama |
| Rheumatology | Megan Curran | |
| APPD LEARN | Alan Schwartz | |
| Beth A. King | ||
| Dennis West | ||
| ABP | David Turner | |
| APPD Fellowship Chair | Bruce Herman | |
| APPD SPIN | Richard Mink | |
| Marzia Hazara |
Join APPD LEARN
All APPD member programs are eligible to be members of APPD LEARN. Membership may be acquired at any time by contacting the APPD LEARN Project Manager and providing the required information.
APPD LEARN members receive the following benefits:
- Opportunities to participate in research studies advancing Pediatrics Graduate Medical Education (GME)
- Eligibility to propose new research studies to be conducted by APPD LEARN, including assistance with:
- Designing studies and preparing proposals
- IRB processes
- Project Management
- Access to CME-designated APPD LEARN training opportunities
- Additional evidence in support of ACGME program requirement II.B.5 (The faculty must establish and maintain an environment of inquiry and scholarship with an active research component)
While an APPD LEARN member program is participating in a research study, it becomes an active member. APPD LEARN active members receive:
- Early access to new educational methods and tools developed through APPD LEARN
- Opportunities for group authorship or acknowledgment in study publications
- Public and professional recognition for participation in educational research
- Access to consultation by educational research specialists
APPD LEARN members have the following responsibilities:
- The program director must indicate knowledge and approval of the program’s membership in APPD LEARN.
- A liaison within the program (who may also be the program director) must be identified.
- The liaison must respond to and complete the annual APPD LEARN member survey and needs assessment.
- The program must be willing to participate in at least one research study every two years.
- A site investigator must be identified for each study in which the program is participating. The site investigator (who may also be the liaison or program director) is responsible for maintaining contact with the APPD LEARN staff and the study PI, and overseeing the implementation of the study protocol at the program.
Propose a Study
APPD LEARN member programs are eligible to propose studies to be conducted on the APPD LEARN network. See the links belowfor general proposal instructions.
Open calls for proposals
CFP 2012-1 (member-initiated proposals)
Proposals are reviewed three times per year, with submission due dates of April 30, August 31, and December 31 each year.
Sharing data through APPD LEARN
Projects involving any number of collaborating programs that are collecting resident-level data outside of APPD LEARN and that wish to archive and share their data through APPD LEARN should contact the APPD LEARN Director as early as possible in the project to discuss how data can be archived by APPD LEARN and APPD LEARN support for data de-identification and documentation.
Any APPD LEARN member program is eligible to submit a research proposal. APPD member programs that are not already APPD LEARN member programs must join APPD LEARN before or at the time of submission of a research proposal. Proposals may include collaborators who are not members of APPD LEARN, but an APPD LEARN member program must be the lead site for the proposal.
Proposals from APPD LEARN member programs may designate any affiliated faculty member(s) of any rank as project investigators, including principal investigator. However, all proposals must be approved by the Program Director and Department Head of the member program, and should be submitted by the program’s APPD LEARN liaison.
All studies that collect data through APPD LEARN are required to deposit their approved protocols and data into the APPD LEARN data repository in order to make them available to future investigators. Publications arising from APPD LEARN studies must acknowledge APPD LEARN and their manuscripts must be provided to APPD LEARN for archiving alongside the study data.
Proposals should include the following components, combined into a single PDF file in the order listed. All components should be formatted in 11 point Helvetica or Arial font, single-spaced, and with margins of at least 0.5 inches on all sides.
- Download the APPD LEARN proposal face page. This page should be saved, filled out, printed, signed, and scanned to PDF with signatures.
- Proposal form template. This document includes instructions and page limits for each section of the proposal.
- Appendices (no page limit):
- Copies of all study instruments
- Biosketch of principal investigator (NIH format encouraged: http://grants.nih.gov/grants/funding/phs398/biosketch.doc)
- If IRB approval has already been obtained at the member program (as the projects lead site), include a copy of the approval or exemption letter. IRB approval or exemption will be required before the project can begin, but need not be obtained before submitting the proposal; APPD LEARN staff will assist in the IRB application for successful proposals.
Investigators, particularly new investigators, seeking support for proposal preparation are encouraged to email the APPD LEARN Director, who can provide consultation on study design, analysis plan, and other aspects of the proposal.
Compile the components of the proposal into a single PDF file, named for the submitting investigator and the month and year submitted (e.g. Schwartz-10-2011.pdf) and email the complete proposal to: LearnProposals@appd.org
Proposals are reviewed three times per year by the APPD LEARN Proposal Review Committee (PRC) and APPD LEARN Director. Review cycles will group proposals submitted January 1-April 30, May 1-August 31, and September 1-December 31.
Criteria applied by the APPD LEARN PRC will include:
- Significance of the research question (including importance for Pediatric GME or medical education)
- Quality of the research plan (including study design, instrumentation, data analysis)
- Feasibility for APPD LEARN
Each proposal will be assigned a primary reviewer and at least two secondary reviewers. Reviewers will write narratives about each criterion and will assign each proposal an overall rating of excellent, very good, good, fair, and poor.
The Committee may make comments or suggest modifications to proposed studies to enhance any of these components. Committee recommendations are advisory. Final decisions about proposal approval or rejection will be made by the APPD LEARN Director.
Each study using the LEARN network will have an ad hoc project oversight committee, composed of the project principal investigator, APPD LEARN Director, LEARN Project Manager, and other members selected by the principal investigator and APPD LEARN Director. These committees will hold regular (biweekly or monthly) conference calls during the period of the study.
After formation of the oversight committee, the APPD LEARN Director and APPD LEARN Project Manager will assist in sampling and enrollment of APPD LEARN member sites into the study protocol and will work with participating sites to obtain IRB approval prior to data collection.
The Study Oversight Group is responsible for adopted processes for communicating with study memnbers and for scientific communications (e.g. manuscripts, presentations) that result from the study, consistent with the following APPD LEARN guidelines:
Request Data
Permitted users
- APPD LEARN members are permitted to obtain data from the repository at no charge, subject only to scientific and IRB approval discussed below.
- APPD members who are not members of LEARN may request access to repository data. These requests will be considered by the APPD LEARN Director and APPD LEARN Advisory Board, who will be responsible for determining any charges associated with providing repository data and any additional terms of use
- Investigators not associated with APPD may request access to repository data. These requests will be considered by the APPD LEARN Director and APPD LEARN Advisory Board, who will be responsible for determining any charges associated with providing repository data and any additional terms of use.
Scientific and budgetary approval
All data requests must include a summary of the data requested, research question to be answered by the data, and planned analyses. Data requests for projects that will involve additional data collection should explain whether and how newly collected data will be provided back to APPD LEARN; proposals that do not plan to share their data will be disadvantaged in review. APPD LEARN will provide a form for these data requests.
Requests received will be reviewed by the APPD LEARN Director for scientific merit and cost to APPD LEARN of providing data, and must be approved before data will be released. Requests that are denied may be appealed to the APPD LEARN Advisory Board, whose recommendations are binding and final.
IRB oversight
Because data maintained in APPD LEARN are not identifiable, use of APPD LEARN data by investigators who did not collect the data or interact with the subjects is likely to constitute research on “persons” rather than “human subjects”. In many institutions, however, investigators are required to obtain from their IRBs a determination that their proposed use of data does not constitute human subjects research. Investigators are responsible for satisfying their local requirements prior to obtaining and analyzing data.
To make a request for data, please complete and sign APPD LEARN's Data Request Form, and submit the form by emailing LEARN@appd.org. Procedures for obtaining data will be determined by the APPD LEARN Director for each request.
When several investigators are seeking to obtain data, priority will be given based on APPD LEARN membership status, scientific merit, and resource availability.
By downloading these Materials, I agree to the following:
- I will use the Materials only for academic or scientific purposes.
- I will not use the Materials to
- obtain information that could directly or indirectly identify subjects.
- produce links among the Distributor's datasets or among the Distributor's data and other datasets that could identify individuals or organizations.
- obtain information about, or further contact with, subjects known to me except where the use and/or release of such identifying information has no potential for constituting an unwarranted invasion of privacy and/or breach of confidentiality.
- I agree not to download any Materials where prohibited by applicable law.
- I agree not to use the Materials in any way prohibited by applicable law.
- I understand that although data sets may include documentation or codebooks that describe the instruments used to collect the data, these instruments are separately copyrighted, and access to the data set does not provide any license or rights to administer the instruments or reverse-engineer the instruments.
- I agree that any books, articles, conference papers, theses, dissertations, reports, or other publications that I create which employ data will reference the bibliographic citation accompanying this data and acknowledge the assistance of APPD LEARN in obtaining the data. These citations include the data authors, data identifier, and other information in accordance with the Recommended Standard (http://thedata.org/citation/standard) for social science data. I further agree to inform APPD LEARN about any such publications or presentations, and, subject to any copyright assignment or embargo agreements required by the publisher, to provide copies of publications or accepted-for-publication manuscripts to be archived alongside the data.
- The distributor makes no warranties, express or implied, by operation of Law or otherwise, regarding or relating to the dataset. I agree to indemnify and hold harmless APPD, APPD LEARN, and its officers, volunteers, agents, and employees for any claims arising out of my research using data provided through APPD LEARN.
Tools
The APPD LEARN Dataverse is a repository for all APPD LEARN study materials, including protocols and proposals, deidentified study data, and links to manuscripts reporting study results.
UIC IRB exemption for the APPD LEARN repository
Dr. Schwartz's HSPP/CITI education status for human subjects training, in case your IRB asks for evidence of CITI training.
The APPD LEARN de-identification system generates learner codes for use in reporting de-identified data to APPD LEARN in APPD LEARN studies. It is hosted at https://learn.cybermango.org/cgi-bin/xls-code.cgi. (This link only provides APPD LEARN ids - do not use it if your study has a study-specific link that also generates tailored survey urls.)
Details about this process and protecting learner privacy are available here
A description of APPD LEARN's survey platform and privacy protections is available here.
If your institution requires a data use agreement with APPD in order to provide data to APPD LEARN, this model agreement contains APPD LEARN's standard language and could be used.
APPD LEARN's authorship principles can be found here.
Individuals who submit data to a LEARN project (through either PRIME or SPIN) but who otherwise do not meet authorship criteria are considered collaborators in publications. The names of these individuals are Pub-Med searchable. SPIN strongly supports recognizing the academic contributions of collaborators and including information about participation in the collaborator's CV. One way to do this is to create a section in the CV entitled "Collaboration in National Studies" (or something similar) and listing the abstracts and publications under this heading. This will differentiate author vs. collaborator roles and avoid confusion.
Data Analysis
Best practices for working on data analyses with Alan
- Providing empty tables and mock-ups of desired figures is very useful to show what you'd like to see.
- Alan is supporting multiple projects at the same time, and often multiple papers/presentations within each project. This makes it challenging for him to quickly identify particular analyses he's run. You can make this much easier by:
- not renaming any analysis output files that he sends you (he can use the filename to quickly find your project/paper/etc on his computer), and
- attaching a copy of that analysis output file whenever asking questions about an analysis
