APPD LEARN is the APPD's Longitudinal Educational Assessment Research Network.
APPD LEARN provides an infrastructure for multi-centered, collaborative research projects and a centralized data collection system. LEARN encompasses two research networks that share the APPD LEARN infrastructure and governance.
The APPD Pediatric Residency Investigators in Medical Education (APPD PRIME) network was APPD LEARN's first network, and focuses on research on Pediatric residents and residency programs.
The APPD Subspecialty Pediatrics Investigator Network (APPD SPIN) is a medical education research network for the Pediatric Subspecialties to evaluate the education and assessment of fellows. SPIN began as a collaborative effort of the American Board of Pediatrics, the APPD Fellowship Committee, APPD LEARN, the Council of Pediatric Subspecialties and each pediatric subspecialty's program director network. In 2020, SPIN became APPD SPIN, a network of APPD LEARN.
APPD LEARN is celebrating its 10-year anniversary! Read the APPD LEARN 10-year Evaluation by Dr. Alan Schwartz.
About APPD LEARN
The mission of APPD LEARN is to conduct meaningful educational research that advances the training of future Pediatricians by developing and promoting participation and collaboration in research by program directors for the purpose of improving the health and well-being of children. Our vision is to advance exemplary pediatric education through collaborative educational research by Pediatric program directors.
APPD LEARN pursues its mission through the following interconnected activities:
- Managing collaborative research networks of Pediatric Programs working together to conduct multi-site studies of educational methods and instruments
- Maintaining an online repository of educational research study materials, raw data, and findings for dissemination to APPD members and collaborators
- Promoting learning opportunities to enhance educational research participation and scholarship by Program Directors
- Providing expert consultation for research conducted within APPD LEARN
- Communicating regularly with the APPD membership and the larger medical education community about activities, opportunities, and outcomes
- Exploring, conducting, and coordinating research with other organizations and initiatives across a continuum of medical and non-medical education
The development of APPD LEARN began from APPD's experience with Share Warehouse, its site for sharing assessment tools and instructional materials among APPD member programs, and the R3P initiative. Planning for APPD LEARN began in 2006. A strategic plan was presented by Ann Burke, Patty Hicks, Susan Guralnick, and Rob McGregor in 2008. APPD and the Initiative for Innovation in Pediatric Education (IIPE) began allocating resources to support APPD LEARN and the initial Advisory Committee was formed in 2009.
APPD LEARN's first Director was Hilary Haftel. She was succeeded by Alan Schwartz in 2011
APPD LEARN Staff
 |
 |
| Alan Schwartz, PhD APPD LEARN / PRIME Director |
Richard Mink, MD, MACM APPD SPIN Director |
 |
|
| Avery Ulrich, MA APPD LEARN Research Assistant |
|
APPD LEARN Advisory Committee, 2022-2024
- Su-Ting Li, MD (Chair, 2022-2023)
- Deborah Hsu, MD, MEd (2022-2023)
- Patricia Poitevien, MD, MSc, FAAP (2022-2024)
- Daniel Schumacher, MD, PhD, MEd (2022-2024)
- Pnina Weiss, MD (2022-2024)
- Lahia Yemane, MD (2022-2024)
- David Turner, MD (non-voting, ABP)
- Linda Waggoner-Fountain, MD (non-voting, APPD LESSON)
- Sabrina Ben-Zion, MD, FAAP (non-voting, APPD RSLC)
- Richard Mink, MD, SPIN network (non-voting)
- Alan Schwartz, PhD, APPD LEARN Director
- Laura Degnon, CAE, APPD Executive Director (ex officio)
APPD LEARN Proposal Review Committee (PRC), 2022-2024
- Maneesh Batra, MD, MPH (2022-2023)
- Sue Bostwick, MD, MBA (2022-2023)
- Jennifer Kesselheim, MD, MEd, MBE (2022-2024)
- Candice Lucas, MD, MPH, FAAP (2022-2024)
- Diane Stafford, MD (2022-2024)
APPD LEARN Director
The APPD LEARN Director is responsible for daily scientific and management oversight of all APPD LEARN activities, and serves as liaison to outside organizations. The Director reports to the APPD LEARN Advisory Committee and APPD Board. The APPD LEARN Director serves for a term of two years, renewable by the APPD Board indefinitely, at a time commitment of at least 0.4 FTE.
APPD SPIN and APPD PRIME Directors
The APPD SPIN and APPD PRIME Directors manage the operations of the APPD SPIN and APPD PRIME subnetworks, respectively.
APPD LEARN Research Manager
The APPD LEARN Research Manager handles all administrative operations of APPD LEARN. Responsibilities of this full-time position include facilitating all communications among the participating institutions and funding agency, development of and adherence to project timelines, and observance of regulatory requirements. The APPD LEARN Research Manager is responsible for daily implementation of the project plan, and assisting PIs with study management. They serve as liaison with research administration at all participating institutions. In addition, the APPD LEARN Research Manager helps with additional network tasks and data collection efforts that arise. The APPD LEARN Research Manager works in conjunction with the APPD LEARN Director, and reports to the APPD Executive Director.
APPD LEARN Advisory Committee
The APPD LEARN Advisory Committee is be responsible for providing guidance to the APPD LEARN Director, setting policies for APPD LEARN activities and resources, and developing calls for proposals in specific research areas. The APPD LEARN Advisory Committee also conducts annual formative and summative evaluation of the performance of the APPD LEARN Director. The APPD LEARN Advisory Committee meetings monthly by conference call and annually in person.
The Committee consists of five voting members, including four APPD-designated members (one of whom serves as the Chair), and one IIPE-designated member. The APPD Board selects the Chair of the APPD LEARN Advisory Committee. In consultation with the Chair of the APPD LEARN Advisory Committee, the APPD Board selects the three additional APPD members. IIPE appoints one representative to the committee. APPD LEARN Advisory Committee terms are 2 years long, and renewable. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director serve ex officio on the Advisory Board.
APPD LEARN Proposal Review Committee (PRC)
The APPD LEARN Proposal Review Committee will assist the LEARN Director in the review of proposals to conduct research using the LEARN network and its member Programs. The committee chair and members will serve two-year terms, which may be renewed. They will be appointed by the APPD LEARN Advisory Committee, in consultation with the APPD LEARN Director, after soliciting suggestions from the APPD Council of Task Force Chairs. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director will serve ex officio on the Committee. The committee will meet as needed by conference call, at a frequency no more than monthly, with online discussion as needed between calls.
APPD LEARN Project Committees
Each study using the APPD LEARN network will have an ad hoc project oversight committee, composed of the project principal investigator, APPD LEARN Director, APPD LEARN Project Manager, and other members selected by the principal investigator and APPD LEARN Director. These committees will hold regular (e.g. biweekly) conference calls during the period of the study.
Member Programs
AdventHealth Orlando
Advocate Health Care (Advocate Children’s Hospital/Oak Lawn)
Advocate Health Care (Advocate Children's Hospital/Park Ridge)
Albany Medical Center
Atlantic Health/Goryeb Children's Hospital
Baylor College of Medicine (Houston)
Baylor College of Medicine (San Antonio)
Brookdale University Hospital and Medical Center
Brooklyn Hospital Center
Brown University
CAMC/West Virginia University (Charleston Division)
Carilion Clinic-Virginia Tech Carilion School of Medicine
Carolinas Medical Center
Case Western Reserve Univ/Univ Hosps Cleveland Med Ctr/Rainbow Babies and Children’s Hospital
Case Western Reserve University (MetroHealth)
Children's Hospital Los Angeles
Children's Hospital Medical Center of Akron/NEOMED
Children's Hospital Oakland (UCSF Benioff Children's Hospital Oakland)
Children's Hospital of Michigan
Children's Hospital of Philadelphia
Children's Hospital/Boston Medical Center
Children's Mercy Hospital
Children's National Medical Center
Cincinnati Children's Hospital Medical Center
Cleveland Clinic Foundation
Cooper Medical School of Rowan University/Cooper University Hospital
Crozer-Chester Medical Center
Dartmouth-Hitchcock/Mary Hitchcock Memorial Hospital
Driscoll Children's Hospital
Duke University Hospital
East Tennessee State University
Eastern Virginia Medical School
Emory University
Flushing Hospital Medical Center
Hackensack University Medical Center
Hospital Episcopal San Lucas-Ponce School of Medicine
Hurley Medical Center/Michigan State University
Icahn School of Medicine at Mount Sinai (Kravis Children's)
Icahn School of Medicine at Mount Sinai (NYC Health and Hospitals/Elmhurst)
Indiana University School of Medicine
Inova Fairfax Medical Campus/Inova Children's Hospital
Jacobi Medical Center/Albert Einstein College of Medicine
Jersey Shore University Medical Center
John H. Stroger Jr. Hospital of Cook County
Johns Hopkins All Children's Hospital
Johns Hopkins University
Kaiser Permanente Medical Group (Northern California)
Kaiser Permanente Southern California (Los Angeles)
Kirk Kerkorian School of Medicine at UNLV Program
Lehigh Valley Health Network/University of South Florida College of Medicine
Loma Linda University Health Education Consortium
Los Angeles County-Harbor UCLA Medical Center
Louisiana State University
Louisiana State University (Shreveport)
Loyola University Medical Center
Maine Medical Center
Marshall University School of Medicine
Massachusetts General Hospital
Mayo Clinic College of Medicine and Science (Rochester)
McGaw Medical Center of Northwestern University
Medical Center of Central Georgia/Mercer University School of Medicine
Medical College of Georgia
Medical College of Wisconsin Affiliated Hospitals
Medical University of South Carolina
Memorial Health-University Medical Center/Mercer University School of Medicine (Savannah)
Mercy St. Vincent Medical Center
Monmouth Medical Center
Montefiore Medical Center/Albert Einstein College of Medicine
Morehouse School of Medicine
National Capital Consortium
Nationwide Children's Hospital/Ohio State University
Naval Medical Center (Portsmouth)
Naval Medical Center (San Diego)
Nemours Children's Hospital Program
New York Presbyterian Hospital (Cornell Campus)
New York University School of Medicine
Newark Beth Israel Medical Center
Nicklaus Children's Hospital
Oregon Health and Science University
Orlando Health/Arnold Palmer Hospital for Children
Our Lady of the Lake
Penn State Milton S. Hershey Medical Center
Phoenix Children's Hospital
Prisma Health-Midlands/University of South Carolina School of Medicine-Columbia, SC
Prisma HealthUpstate/University of South Carolina School of Medicine Greenville
Rush University Medical Center
Rutgers New Jersey Medical School
Sidney Kimmel Medical College at Thomas Jefferson University/duPont Hospital for Children
Southern Illinois University
Spectrum Health/Michigan State University/Helen DeVos Children's Hospital
St. Christopher's Hospital for Children
St. Joseph’s University Medical Center
St. Louis University School of Medicine
St. Vincent Hospital and Health Care Center
Stanford University
Stony Brook Medicine/University Hospital
SUNY Health Science Center at Brooklyn
SUNY Upstate Medical University
Texas A&M College of Medicine-Scott and White Medical Center (Temple)
Texas Tech University Health Sciences Center at Lubbock
Tripler Army Medical Center
Tucson Hospitals Medical Education Program
Tufts Medical Center
Tulane University
UCLA David Geffen School of Medicine/UCLA Medical Center
UMMS-Baystate
University at Buffalo
University of Alabama Medical Center
University of Arizona
University of Arizona College of Medicine-Tucson
University of Arkansas for Medical Sciences
University of California (Davis) Health System
University of California (San Diego)
University of California (San Francisco)
University of California (San Francisco)/Fresno
University of Chicago
University of Colorado
University of Connecticut
University of Florida
University of Florida (Orlando)
University of Florida College of Medicine-Jacksonville
University of Hawaii
University of Illinois College of Medicine at Chicago
University of Illinois College of Medicine at Peoria
University of Iowa Hospitals and Clinics
University of Kansas (Wichita)
University of Louisville
University of Maryland
University of Massachusetts
University of Michigan
University of Minnesota
University of Nebraska Medical Center College of Medicine
University of New Mexico
University of North Carolina Hospitals
University of Oklahoma Health Sciences Center
University of Oklahoma School of Community Medicine (Tulsa)
University of Rochester
University of South Alabama
University of South Dakota
University of South Florida Morsani
University of Southern California/LAC&USC Medical Center
University of Tennessee
University of Tennessee College of Medicine at Chattanooga
University of Texas at Austin Dell Medical School Pediatric Program
University of Texas Health Science Center at Houston
University of Texas Health Science Center School of Medicine at San Antonio
University of Texas Medical Branch Hospitals
University of Texas Southwestern Medical School
University of Toledo
University of Utah
University of Vermont Medical Center
University of Virginia
University of Washington
University of Wisconsin
UPMC Medical Education
Vanderbilt University
Vidant Medical Center/East Carolina University
Virginia Commonwealth University Health System
Wake Forest University School of Medicine
Washington University/St. Louis Children's
Westchester Medical Center
Western Michigan University Homer Stryker MD School of Medicine
Wright State University
Yale-New Haven Medical Center
Zucker School of Medicine at Hofstra/Northwell at Cohen Children's Medical Center
Zucker School of Medicine at Hofstra/Northwell at Staten Island University Hospital
PAPERS
2022
Halbach SM, Pillutla K, Seo-Mayer P, Schwartz A, Weidemann D, Mahan JD. (2022) Burnout in Pediatric Nephrology Fellows and Faculty: Lessons From the Sustainable Pediatric Nephrology Workforce Project. Frontiers in Pediatrics, 10. 10.3389/fped.2022.849370.
2021
Abramson EK, Weiss P, Naifeh M, Stevenson, Duncan J, Rama J, Mauer E, Gerber L, T. Li ST, MD. (2021) Scholarly Activity Training during Pediatric Fellowship: A National Assessment. Pediatrics, 147(1) e2020013953.
Fitzgerald M, Bhatt A, Thompson LA, Schwartz A, Thomas AO, Schinasi DA, Otero J, Carpenter P, Thomas JS, Paradise Black N. (2021) Telemedicine in Pediatric Training: A National Needs Assessment of the Current State of Telemedicine Education in Pediatric Training. Academic Pediatrics, (in press).
Pitts S, Schwartz A, Carraccio C, Herman B, Mahan J, Sauer C, Dammann C, Mink R; SPIN Steering Committee. Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activites Across Subspecialties . (2021) Academic Pediatrics, (pre-print published) https://doi.org/10.1016/j.acap.2021.12.019
Schwartz A, Borman-Shoap E, Carraccio C, Herman B, Hobday P, Kaul P, Long M, O'Connor M, Mink R, Schumacher D, Turner DA, West DC. (2021). Learner Levels of Supervision Across the Continuum of Pediatrics Training. Academic Medicine, 96(7S): S42-S49. http://dx.doi.org/10.1097/ACM.0000000000004095 (PRIME and SPIN)
Turner DA, Schwartz A, Carraccio C, Herman B, Weiss P, Baffa J, Chess P, Curran M, Dammann C, High P, Hsu D, Pitts S, Sauer C, Aye T, Fussell J, Kesselheim J, Mahan J, McGann K, Myers A, Mink R. (2021), Continued Supervision for the Common Pediatric Subspecialty
Entrustable Professional Activities May Be Needed Following Fellowship Graduation. Academic Medicine, 96(7S):S22-S28. doi: 10.1097/ACM.0000000000004091.
Weiss P, Schwartz A, Carraccio C, Herman B, Mink R. (2021), Minimum Supervision Levels Required by Program Directors for Pediatric Pulmonary Fellow Graduation. ATS Scholar, (open access). https://dx.doi.org/10.34197/ats-scholar.2021-0013OC
Weiss P, Schwartz A, Carraccio C, Herman B, Turner DA, Aye T, Fussell J, Kesselheim J, Mahan J, McGann K, Myers A, Stafford DEJ, Chess P, Curran M, Damman CEL, High P, Hsu DC, Pitts S, Sauer C, Srivastava S, Mink R. (2021), Achieving Entrustable Professional Activities During Fellowship. Pediatrics, 148(5):e2021050196. https://doi.org/10.1542/peds.2021-050196
2020
Hu K, Hicks PJ, Margolis M, Carraccio C, Osta A, Winward ML, Schwartz, A. (2020). Reported Pediatrics Milestones (Mostly) Measure Program, Not Learner Performance. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003644. [RIME New Investigator Award winner 2020].
Kemper KJ, Schwartz A, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2020). Bullying, Discrimination, Sexual Harassment, and Physical Violence: Common and Associated with Burnout in Pediatric Residents. Academic Pediatrics. Published ahead of print: https://www.academicpedsjnl.net/article/S1876-2859(20)30080-2/fulltext
Kemper KJ, Schwartz A, Wilson PM, Mahan JD, Schubert CJ, Staples BB, McClafferty H, Serwint JR, Batra M. (2020). Burnout in Pediatric Residents: Three Year of National Survey Data. Pediatrics, 145(1): e20191030. doi: 10.1542/peds.2019-1030
Mink R, Herman BE, Carraccio C, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DEJ, Weiss P, Curran ML, Dammann CEL, High PC, Hsu D, Kesselheim JC, Mahan JD, McGann KA, Myers AL, Pitts S, Turner DA, Schwartz A. J. (2020) Agreement of Program Directors With Clinical Competency Committees for Fellow Entrustment. Med Educ Curric Dev. 2020 Aug 6;7:2382120520936613. doi: 10.1177/2382120520936613. eCollection 2020 Jan-Dec.
Reed S, Mink R, Li ST. Utility of Residency Milestones Reported to Fellowship Directors: A National Survey of Pediatric Fellowship Program Directors. (2020) Acad Pediatr DOI: 10.1016/j.acap.2020.01.004.
Schumacher DJ, Poynter S, Burman N, Elliott SP, Barnes M, Gellin C, Gonzalez del Rey J, Sklansky D, Thoreson L, King B, Schwartz A, APPD LEARN CCC Study Group. (2020). Concordance of Clinical Competency Committee and Program Director Recommendations for Resident Supervisory Roles. Academic Pediatrics.
Schumacher DJ, Schwartz A, Zenel JA, Black NP, Ponitz K, Blair R, Traba CM, Poynter S, King B, Englander R, Rosenberg A, Patel D, Smith-King C, O'Connor M, Gonzalez del Rey J, Lavoie S, Borman-Shoap E, Carraccio C, for the APPD LEARN General Pediatrics EPAs Study Group. (2020). Narrative Performance Level Assignments at Initial Entrustment and Graduation: Integrating EPAs AND Milestones to Improve Learner Assessment. Academic Medicine. doi: 10.1097/ACM.0000000000003300.
Schumacher DJ, West DC, Schwartz A, Li S, Millstein L, Griego EC, Turner TL, Herman BE, Englander R, Hemond J, Hudson V, Newhall L, McNeal Trice K, Baughn J, Giudice E, Famiglietti H, Gifford K, Carraccio C, for the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network General Pediatrics EPAs Study Group. (2020). Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities: A Multi-Site Study. JAMA Network Open, 3(1):e1919316.
Schwartz A, Balmer DF, Borman-Shoap E, Chin A, Henry D, Herman BE, Hobday P, Lee JH, Multerer S, Myers RE, Ponitz K, Rosenberg A, Soep JB, West DC, Englander R. (2020). Shared Mental Models among Clinical Competency Committees in the Context of Time-Variable-Competency-Based Advancement to Residency. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003638
Schwartz A, King B, Mink R, Hicks PJ. (2020). The emergence and spread of practice-based medical education research networks. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003641
Staples BB, Burke AE, Batra M, Kemper KJ, Schwartz A, Wilson PM, Schubert CJ, Mahan JD, Serwint JR. Pediatric Resident Burnout-Resilience Study Consortium. (2020). Burnout and association with resident performance as assessed by Pediatric Milestones: An exploratory study. Academic Pediatrics. https://doi.org/10.1016/j.acap.2020.08.006
Webber S, Schwartz A, Kemper KJ, Batra M, Mahan JD, Babal J, Sklansky DJ, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium (PRB-RSC) (2020). Faculty and Peer Support during Pediatric Residency: Association with Performance Outcomes, Race and Gender. Academic Pediatrics. https://doi.org/10.1016/j.acap.2020.08.009
2019
Kemper KJ, Wilson P, Schwartz A, Mahan JD, Batra M, Staples B, McClafferty H, Schubert CJ, Serwint JR, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2019). Burnout in Pediatric Residents: Comparing Brief Screening Questions to the Maslach Burnout Inventory. Academic Pediatrics.
Schumacher DJ, Martini A, Bartlett KW, King B, Calaman S, Garfunkel LC, Elliott SP, Frohna JG, Schwartz A, Michelson CD, and the APPD LEARN CCC Study Group. (2019). Key Factors in Clinical Competency Committee Members' Decisions Regarding Residents' Readiness to Serve as Supervisors: A National Study. National Study of Clinical Competency Committee Members. Academic Medicine, 94(2): 251-258. doi: 10.1097/ACM.0000000000002469
561-565.
Schumacher DJ, Parrigan A, Bartlett KW, King B, Calaman S, Garfunkel LC, Elliott SP, Frohna JG, Schwartz A, Michelson C, and the APPD CCC Study Group. (2019). Key Factors for Recommending Whether Residents May Serve as Supervisors: A National Study of Clinical Competency Committee Members. Academic Medicine.
Schumacher DJ, Poynter S, Burman N, Elliott SP, Barnes M, Gellin C, Gonzalez del Rey J, Sklansky D, Thoreson L, King B, Schwartz A, APPD LEARN CCC Study Group. (2019). Justifications for Discrepancies Between Clinical Competency Committee and Program Director Recommendeed Resident Supervisory Roles. Academic Pediatrics, 19(5): 561-565.
2018
Abramson EL, Naifeh M, Stevenson M, Mauer E, Hammad H, Gerber L, Li S. (2018). Scholarly Activity Training During Residency: Are we Hitting the Mark? A National Assessment of Pediatric Residents. Academic Pediatrics.
Hicks PJ, Margolis MJ, Carracio CL, Clauser BE, Donnelly K, Fromme HB, Gifford KA, Poynter SE, Schumacher DJ, Schwartz A, PMAC Module 1 Study Group. (2018). A novel workplace-based assessment for competency-based decisions and learner feedback. Medical Teacher, 40:11, 1143-1150.
Kemper KJ, McClafferty H, Wilson P, Serwint JR, Batra M, Mahan JD, Schubert CJ, Staples BB, Schwartz A, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2018). Do Mindfulness and Self-Compassion Predict Burnout? Academic Medicine. Published ahead of print.
Li ST, Schwartz A, Burke AE, Guralnick S, Trimm RF, Guillois A, Mahan JD, Gifford K, Tancredi DJ. (2018). Pediatric Program Director Minimum Milestone Expectations Before Allowing Supervision of Others and Unsupervised Practice. Academic Pediatrics. doi: 10.1016/j.acap.2020.05.011
Li ST, Tancredi DJ, Schwartz A, Guillot A, Burke A, Trimm RF, Guralnick S, Mahan JD, Gifford KA, for the APPD LEARN Validity of Resident Self-Assessment Group. (2018). Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. Academic Medicine
Mink R, Schwartz A, Carraccio C, High P, Dammann C, McGann K, Kesselheim J, Herman B and the Steering Committee of the Subspecialty Pediatrics Investigator Network. (2018). Creating the Subspecialty Pediatrics Investigator Network (SPIN). Pediatrics, 192:3-4.e2. doi: 10.1016/j.jpeds.2017.09.079.
Mink RB, Schwartz A, Herman BE, Turner DA, Curran ML, Myers A, Hsu DC, Kesselheim JC, Carraccio CL, for the Subspecialty Pediatrics Investigator Network (SPIN). (2018). Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities. Academic Medicine, 93:2, 283-291.
Park YS, Hicks PJ, Carraccio CL, Margolis MJ, Schwartz A, for the PMAC Module 2 Study Group. (2018). Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth from Ten Pediatrics Residency Programs. Academic Medicine, 93(11): S21-S29.
Reed SR, Kemper KJ, Schwartz A, Batra M, Staples BB, Serwint JR, McClafferty H, Schubert CJ, Wilson PM, Rakowsky A, Chase M, Mahan JD. (2018). Variability of Burnout and Stress Measures in Pediatric Residents: an Exploratory Single Center Study from the Pediatric Resident Burnout - Resilience Study Consortium. Journal of Evidence-Based Integrative Medicine.
Schumacher DJ, Bartlett KW, Elliott SP, Michelson C, Sharma T, Garfunkel LC, King B, Schwartz A, and the APPD LEARN CCC Study Group. (2018). Milestone Ratings and Supervisory Role Categorizations Swim Together, but is the Water Muddy? Academic Pediatrics.
Schumacher DJ, King B, Barnes MM, Elliott S, Gibbs K, McGreevy JF, Gonzalez del Rey J, Sharma T, Michelson C, Schwartz A, and the APPD LEARN CCC Study Group. (2018). The Influence of the Review Process Utilized by Clinical Competency Committee Members on Their Summative Assessment Decisions of Residents. Journal of Graduate Medical Education.
Schumacher DJ, Michelson C, Poynter S, Barnes MB, Li S, Burman N, Sklansky DJ, Thoreson L, Calaman S, King B, Schwartz A, and the APPD LEARN CCC Study Group. (2018). Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns. Medical Teacher, 40(1): 70-79.
2017
Kesselheim JC, Schwartz A, Boyer D, for the APPD LEARN Study Group on Education and Service. (2017). Integrating education and service in Pediatric residency training: Results of a national survey. Academic Pediatrics, 17:8, 907-914
Li ST, Tancredi DJ, Schwartz A, Guillot AP, Burke AE, Trimm F, Guralnick S, Mahan JD, Gifford KA, for the APPD LEARN Validity of Resident Self-Assessment Group.. (2017). Competent for unsupervised practice: Use of Pediatric residency training milestones to assess readiness. Academic Medicine, 92:3, 385-393.
Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A; the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network–National Board of Medical Examiners Pediatrics Milestones Assessment Group. (2017). Validity Evidence From Ratings of Pediatric Interns and Subinterns on a Subset of Pediatric Milestones. Academic Medicine, 92(6):809-819. doi: 10.1097/ACM.0000000000001622. PMID: 28557947.
2016
Hicks PJ, Margolis M, Poynter S, Chaffinch C, Tenney-Soeiro R, Turner T, Waggoner-Fountain L, Young R, Clyman SG, Schwartz A, for the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). The Pediatrics Milestones Assessment Pilot: Development of Workplace-based Assessment. Academic Medicine, 91:5, 701-709.
Kesselheim JC, Schwartz A, Belmonte F, Boland KA, Poynter S, Batra M, for the APPD LEARN Study Group on Social Media and Professionalism. (2016). A National Survey of Pediatric Residents' Professionalism and Social Networking: Implications for Curriculum Development. Academic Pediatrics, 16:2, 110-114
Schwartz A, Margolis M, Multerer S, Haftel H, Schumacher DJ, and the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). A multi-source feedback tool for measuring a subset of Pediatrics Milestones. Medical Teacher, 38:10.
Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A, for the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). Assessment of Pediatric interns and sub-interns on a subset of Pediatric Milestones. Academic Medicine, 92:6, 809-819.
2014
Schwartz A, Young R, Hicks PJ; for APPD LEARN. (2014). Medical Education Practice-Based Research Networks: Facilitating collaborative research. Medical Teacher, 38:1, 64-74.
Tocco D, Jain AV, Baines H. (2014). The pediatrics milestone pilot project: perspectives of current pediatric residents. Acad Pediatr, 14(2 Suppl), S8-9.
PRESENTATIONS
Zuniga L, Schwartz A, Batra M, Mahan JD, on behalf of the PRB-RSC. (2021,May). Burnout in the time of COVID: An unexpected story of stress and resilience. Pediatric Academic Societies, 2021 Virtual Meeting.
Zuniga L, Schuh A, Seo-Mayer P, Kapklein M, Cramton R, Sieplinga K, Kaushik R, Schwartz A, Batra M, Mahan JD on behalf of the PRB-RSC. (2021, March). Burnout in the time of COVID: You don't know until you measure it! APPD Spring Annual Meeting, Virtual.
Mink R, Carraccio C, Herman B, Hsu D, Chess P, High P, Langhan M, McGann K, Curran M, Stafford D, Czaja A, Dammann C, Mahan J, Aye T, Schwartz A. (2019, May). Are Graduating Pediatric Fellows Meeting the Expected Level of Supervision for the Common Subspecialty Entrustable Professional Activities (EPAs)? Medical Education: Assessment: Pediatric Academic Societies, Baltimore, MD.
Reed S, Mink R, Li ST. (2019, May). Utility of Residency Milestones Reported to Fellowship Directors: A national survey of pediatric fellowship program directors. Medical Education: Assessment: Pediatric Academic Societies, Baltimore, MD.
Park YS, Hicks PJ, Carraccio C, Margolis M, Schwartz A, for the PMAC Module 2 Study Group. (2018, November). Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth. RIME oral presentation. Annual Meeting of the Association of American Medical Colleges, Austin, TX.
Minimum Level of Supervision for Fellows at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. (May 2018). APA Presidential Plenary, Pediatric Academic Societies, Toronto, Canada.
Is the Expected Minimum Level of Supervision for Graduating Pediatric Fellows for the Common Entrustable Professional Activities Similar Across the Subspecialties? (May 2018). Medical Education 1: Top Oral Abstracts. Pediatric Academic Societies, Toronto, Canada.
Batra M, Mahan J, Schubert C, Wilson P, Staples B, Serwint JR, McClafferty H, Schwartz A, Kemper K.(2018, March). Burnout in Pediatric Residents: Findings From a National Longitudinal Survey. Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Lauden S, Wilson P, Fuast M, Mahan J, Batra M, Shubert C.(2018, March). Burnout & Who's Who in Pediatric Global Health: A Look at National Trends. Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Abramson E, Stevenson M, Naifeh M, Hammad H, Gerber L, LI ST. (2017, May). Scholarly Activity Training During Residency: A National Survey of Pediatric Residents and Program Directors. Pediatric Academic Societies, San Francisco, CA.
Gifford KA, Burke A, Li ST, Tancredi DJ, Trimm F, Schwartz A, Mahan J, Guralnick S, Guillot AP. (2017, May). Do Milestones Reported to the ACGME Better Describe Residents' Performance or their Programs' Assessment Practices? Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Bartlett K, Michelson C, Elliott S, Gibbs K, McGreevy J, Li ST, Sklansky DJ, Gonzalez del Rey J, Poynter S, Sharma T, Burman N, Gellin C, Thoreson L, Calaman S, Garfunkel L, Frohna J, Scott-Vernaglia S, Barnes M, Schwartz A. (2017, May). Association Between Milestone Assignments and Recommended Supervision Categorizations Among Clinical Competency Committee Members. Pediatric Academic Societies, San Francisco, CA.
Batra M, Mahan JD, Schubert CJ, Wilson PM, Staples BB, Serwint JR, McClafferty H, Schwartz A, Kemper KJ, for the Pediatric Residency Burnout and Resilience Consortium. (2017, May). Burnout in Pediatric Residents: A National Survey To Inform Future Interventions. Pediatric Academic Societies, San Francisco, CA.
Staples BB, Batra M, Mahan J, Schwartz A, Wilson P, McClafferty H, Schubert C, Kemper K, Serwint JR. (2017, May). Burnout Status and Milestone Performance in Pediatric Residents. Pediatric Academic Societies, San Francisco, CA.
Batra M, Mahan J, Shubert C, Wilson P, Staples BB, Serwint JR, McClafferty H, Schwartz A, Kemper K. (2017, April). Burnout in Pediatric Residents: A National Survey to Inform Future Interventions. Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA. Winner: 2017 Research Award.
Schwartz A. (2016, December). Milestones-based research at APPD LEARN. ACGME Milestones Summit, Chicago, IL.
Mahan JD, Serwint JR, Schubert CJ, Wilson PM, Staples BB, McClafferty, Schwartz A, Kemper KJ, Batra M. (2016, November) Building a national consortium (The Pediatric Resident Burnout-Resilience Study Consortium) to understand and promote pediatric wellness. The ACGME Second Symposium on Physician Well-Being: Commitment to Change. Chicago, IL.
Schwartz, A. (2016, August). APPD LEARN: Building a Pediatric Education Research Network. AMSPDC Webinar
Kesselheim JC, Boyer D. (2016, April). Balancing education and service in Pediatric residency training: Results of a national survey. Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA. Winner: 2016 Research Award
Li S, Burke AE, Guillot A, Schwartz A, Gifford KA, Trimm F, Mahan JD, Guralnick S, Tancredi DJ. (2016, April). Pediatric Resident Minimum Milestone Expectations by Level of Training. Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Li ST, Tancredi DJ, Schwartz A, Burke AE, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K. (2016, February). Pediatric Resident Minimum Milestone Expectations by Level of Training. Platform presentation at Pediatric Academic Societies (PAS) Annual National Conference, Baltimore, MD.
The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. (2016, February). Annual education meeting of the Accreditation Council for Graduate Medical Education, National Harbor, MD.
Creation and validation of entrustment scales for the common pediatric subspecialty entrustable professional activities (EPAs). (2016). Acad Pediatr 16:e1–e2.
Andriole D, Jeffe D, Reilly K, Schwartz A, Bostrom D. (2015, November). Bridging Data Across Divides: Challenges of Aggregating Data from Multiple Organizations. Learn, Serve, Lead: 2015 AAMC, Baltimore, MD.
Li S, Tancredi D, Schwartz A, Burke A, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K, for APPD LEARN validity of resident self-assessment using pediatric milestones group. (2015, April). Which Sub-Competencies Best Identify Overall Unsatisfactory/ Marginal Resident Performance? Pediatric Academic Societies, San Diego, CA.
Li S, Tancredi D, Schwartz A, Burke A, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K, for APPD LEARN validity of resident self-assessment using pediatric milestones group. (2015, April) How Well Do Resident Milestone Self-Assessments Correlate with Clinical Competency Committee Milestone Assessments? Pediatric Academic Societies, San Diego, CA.
Gifford K, Li S, Trimm RF, Schwartz A, Gurlanick S, Tancredi DJ, Burke A, Mahan J, for For APPD LEARN validity of resident self-assessment group. (2015, April). How Do Clinical Competency Committees Function in Pediatric Residency? Pediatric Academic Societies, San Diego, CA.
Li S, Tancredi DJ, Trimm RF, Guillot A, Mahan JD, Burke A, Gifford KA, Schwartz A, Guralnick S, for APPD LEARN validity of resident self-assessment. (2015, March). How well do resident milestone self-assessments correlate with clinical competency committee milestone assessments? APPD Annual Spring Meeting, Orlando, FL. Winner, APPD Research Award
Tenney-Soeiro R, Ronan JC, Weiss AK, Young D, Clyman SG, Schwartz A, Rose KM, Hicks PJ. (2013, April). Does the Quantity and Quality of Comments Provided on Intern and Sub-Intern Pediatrics Milestones Assessment Instrument Vary Based on Role of the Rater? Platform Presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Turner T, Hicks P, Rose K, Clyman S, Schwartz A, Poynter S, Haftel H, Ronan JC, Waggoner-Fountain L, Caputa G. (2013, April). Sub-Intern and Intern Performance Level on Pediatric Milestones. Platform Presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Chaffinch C, Poynter S, Turner T, Hicks P, for the APPD LEARN-NBME Milestones Assessment Group. (2013, February). Learner Feedback on the Pediatrics Milestones Assessment Project. Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Poynter S, Turner T, Chaffinch C, Hicks P, for the APPD LEARN-NBME Milestones Assessment Group. (2013, Februray). Learner Feedback on the Pediatrics Milestones Assessment Project. 2013 ACGME Annual Educational Conference, Orlando, FL.
Schwartz A. (2012, November). Assessment of Pediatrics Milestones and Development of a Collaborative Research Network for GME. Invited presentation, DeWitt Baldwin Seminar Series, ACGME, Chicago, IL.
POSTERS
Mink R, Carraccio C, Herman B, High P, Hsu D, Langhan M, Schwartz A. Factors Important in Determining Simple versus Complex Cases for the Pediatric Subspecialty Entrustable Professional Activity (EPA) Levels of Supervision, Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/ 2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.356.)
Mink R, Carraccio C, Herman B, Chess P, Dammann C, Hsu D, Kesselheim J, Moffat M, Myers A, Turner D, Czaja A, Schwartz A. Have Level of Supervision Ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) Changed Over Time? Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.369)
Mink R, Carraccio C, Herman B, Weiss P, Turner D, Pitts S, Moffat M, Kesselheim J, Czaja A, Schwartz A. Relationship Between Level of Supervision at the End of Residency Compared with Early Fellowship for the Entrustable Professional Activities (EPAs): A Pilot Study. Pediatric Academic Societies, 2020. (Accepted as platform presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202455.2.)
Mink R, Carraccio C, Herman B, Hsu D, Chess P, High P, Langhan M, McGann K, Curran M, Stafford D, Czaja A, Dammann C, Mahan J, Aye T, Schwartz A. (2019, May). Are Graduating Pediatric Fellows Meeting the Expected Level of Supervision for the Common Subspecialty Entrustable Professional Activities (EPAs)? Pediatric Academic Societies, Baltimore, MD.
Mink R, Carraccio C, Herman B, Myers A, Fussell J, Turner D, Pitts S, Kesselheim J, Baffa J, Weiss P, Sauer C, Srivastava S, Moffatt M, Halbach S, Schwartz A. (2019, March). Do Pediatric Fellows Meet the Expected Level of Supervision at Graduation? Association of Pediatric Program Directors, New Orleans, LA.
Reed S, Mink R, Li ST. (2019, March). Utility of Residency Milestones Reported to Fellowship Directors: A national survey of pediatric fellowship program directors. Association of Pediatric Program Directors, New Orleans, LA.
Smith S, Mensah-Bonsu N, Lopez M, Voigt R, Mink R, Carraccio C, Schwartz A, Herman B, Fussell J. (2018, September). Are Graduating Developmental-Behavioral Pediatrics Fellows Ready for Clinical Practice? A SPIN study. Poster Presentation at Society for Developmental and Behavioral Pediatrics 2018 Annual Meeting, Anaheim, CA.
Lopez M, Mensah-Bonsu N, Smith S, Voigt R, Mink R, Carraccio C, Schwartz A, Herman B, Fussell J. (2018, September). Using Entrustable Professional Activities to Assess Graduation Readiness in Developmental Behavioral Pediatrics: A SPIN Study. Poster Presentation at Society for Developmental and Behavioral Pediatrics 2018 Annual Meeting, Anaheim, CA.
Hu K, Hicks PJ, Margolis MJ, Carraccio CL, Osta AD, Schwartz A. (2018, August). Inter-program variance in ACGME Reportable Pediatrics Milestones. Can Residents' ACGME PM levels be compared among programs? Poster presentation at the 2018 Annual Meeting of the Association for Medical Education in Europe (AMEE 2018), Basel, Switzerland.
Mink R, Carraccio C, Herman B, Aye T, Chess P, Curran M, Dammann C, High P, Hsu D, Mahan J, McGann K, Srivastava S, Turner D, Weiss P, Schwartz A for the Subspecialty Pediatrics Investigator Network. (2018, May). Is the Expected Minimum Level of Supervision for Graduating Pediatric Fellows for the Common Entrustable Professional Activities Similar Across the Subspecialties? Pediatric Academic Societies, Toronto, Canada.
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carraccio C for the Subspecialty Pediatrics Investigator Network. (2018, May). Setting the Minimum Level of Supervision for Fellows at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. Pediatric Academic Societies, Toronto, Canada.
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carracio C.(2018, March). Expected Fellow Minimum Level of Supervision at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. Poser Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Mink R, Carracio C, Herman B, Baffa J, Fussell J, Kesselheim J, McGann K, Myers A, Sauer C, Stafford D, Weiss P, Schwartz A.(2018, March). Pediatric Fellows are Not Expected to Achieve "Entrustment" at Graduation from Fellowship. Poster Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Abramson E, Stevenson M, Naifeh M, Mauer E, Gerber L, Li ST. (2018, March). Scholarly Activity Training During Residency: How Well Aligned are Program Directors and Residents? Poster Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Hicks P, Margolis M, Carraccio C, Clauser B, Winward M, Schwartz A, and the PMAC Module 1 Study Group. (2017, May). Pediatrics Milestones Assessment Collaborative: Development and implementation of an authentic workplacebased assessment system (WBAS). Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Barnes M, Bartlett K, Burman N, Calaman S, Elliott S, Frohna J, Garfunkel L, Gellin C, Gibbs K, Gonzalez del Rey J, Michelson C, McGreevy J, Li ST, Poynter S, Scott-Vernaglia S, Sharma T, Sklansky DJ, Thoreson L, Schwartz A. (2017, May). Concordance of Clinical Competency Committee and Program Director Recommended Supervision Categorization. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Thoreson L, Sklansky DJ, Sharma T, Scott-Vernaglia S, Poynter S, Li ST, McGreevy J, Michelson C, Gonzalez del Rey J, Gibbs K, Gellin C, Garfunkel L, Frohna J, Elliott S, Calaman S, Burman N, Bartlett K, Barnes M, Schwartz A. (2017, May). Key Factors for Recommending a Resident May Serve as a Supervisor: A National Study of Clinical Competency Committee Members. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Reed S, Batra M, Kemper K, Staples BB, Serwint JR, McClafferty H, Schubert C, Wilson P, Schwartz A, Mahan J. (2017, May). How Variable are Burnout and Stress in Pediatric Residents? An Exploratory Single Center Study from the Pediatric Resident Burnout-Resilience Study Consortium. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Staples BB, Batra M, Mahan J, Schwartz A, Wilson P, McClafferty H, Schubert C, Kempter K, Serwint JR. (2017, April). Burnout Status and Milestone Performance in Pediatric Residents. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Schumacher DJ, King B, Barnes M, Bartlett K, Burman N, Calaman S, Elliott S, Frohna J, Garfunkel L, Gellin C, Gibbs K, Gonzalez del Rey J, Michelson C, McGreevy J, Li ST, Poynter S, Scott-Vernaglia S, Sharma T, Sklansky DJ, Thoreson L, Schwartz A. (2017, April). Concordance of Clinical Competency Committee and Program Director Recommended Supervision Categorization. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Gifford KA, Li ST, Schwartz A, Burke AE, Tancredi DJ, Guralnick S, Guillot AP, Trimm F, Mahan JD. (2017, April). Do Milestones Reported to the ACGME Describe Residents Performance or Their Programs Assessment Practices? Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Schumacher DJ, King B, Thoreson L, Sklansky DJ, Sharma T, Scott-Vernaglia, Poynter S, Li ST, McGreevy J, Michelson C, Gonzalez del Rey J, Gibbs K, Gellin C, Garfunkel L, Frohna J, Elliott S, Calaman S, Burman N, Bartlett K, Barnes M, Schwartz A. (2017, April). Key Factors for Recommending a Resident May Serve as a Supervisor: A National Study of Clinical Competency Committee Members. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Abramson E, Stevenson M, Naifeh M, Hammad H, Gerber L, Li ST. (2017, April). Scholarly Activity Training During Residency: A National Survey of Pediatric Residents and Program Directors. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Mink R, Carraccio C, Schwartz A, Dammann C, High P, Kesselheim J, McGann K, Herman B. (2016, May). Establishing a Medical Education Research Network for the Pediatric Subspecialties. Pediatric Academic Societies, Baltimore, MD.
Mink R, Carraccio C, Herman B, Dammann C, Mahan J, Pitts S, Sauer C, Schwartz A. (2016, May). Variability in Fellow Entrustment Across the Pediatric Subspecialties for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs). Pediatric Academic Societies, Baltimore, MD.
Gifford KA, Burke AE, Trimm F, Mahan JD, Schwartz A, Li ST, Guillot AP, Guralnick S. (2016, April). Learner self-assessment as a contribution to competency assessment. Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Schwartz A, Dammann CE, High PC, McGann KA, Herman BE for SPIN. (2016, April). Creation of a pediatric subspecialty educational research network. Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DE, Weiss P, Schwartz A for SPIN. (2016, April). Do fellowship program directors (FPD) and clinical competency committees (CCC) agree in fellow entrustment decisions? Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DE, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Schwartz A for SPIN. (2016, April). The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. Annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DE, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Schwartz A for SPIN. (2016, February). The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. Annual Education Meeting of the Accreditation Council for Graduate Medical Education, National Harbor, MD.
Mink RB, Carraccio CL, Herman BE, Turner DA, Myers A, Kesselheim JC, Hsu DC, Curran ML, Schwartz A, for the Subspecialty Pediatrics Investigators Network (SPIN). (2016, February). Reliability and validity of a supervision scale for the common pediatric subspecialty entrustable professional activities. Poster presentation. ACGME Annual Meeting, National Harbor, MD.
Balmer DF, Hicks PJ, Carraccio C, Schwartz A. (2016 February). Bridging the Gap between Educational Research and Practice: The Normalization Process Theory Poster presentation. Pediatric Academic Societies, Baltimore, MD.
Schwartz A., Young, R., Hicks, P.J. (2014, November). Medical Education Research Networks: The case of APPD LEARN. Poster presentation at the 2014 American Association of Medical Colleges Medical Education Meeting, Chicago, IL.
Chaffinch CN, Poynter SE, Turner T, Schwartz A. (2013, April). Learner Feedback on the Pediatrics Milestones Assessment Project. Poster presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
McGregor R, Burke A, Carraccio C, Haftel H, Hicks P, Schwartz A. (2012, November). APPD LEARN: An Educational Research Network in Pediatric GME Digitized Poster Presentation at Group on Educational Affairs, AAMC Annual Meeting, San Francisco, CA.
APPD PRIME
Understanding the Experience and Impact of Parenting Electives on Pediatric Residents and Their Return to Work
PI: Sabrina Ben-Zion
Synopsis: Determine if there are associations between utilization of a parenting elective with satisfaction with transition back to work, post-partum depression, post-partum burnout, breastfeeding duration, and perception of support
Status: Planning
Training Opportunities for Sexual and Reproductive Health: A National Survey of Pediatric Residents
PI: Sarah Golub
Synopsis: Understand the extent of education and training in sexual and reproductive health (SRH) care for adolescent assigned females at birth (from here on referred to as “females”) that pediatric residents in the United States receive during their residency.
Status: Planning
How Well Do Milestones Correlate with ITE Scores?
PI: Su-Ting Li
Synopsis: Determine correlation between Milestone 2.0 score for MK1 (Clinical Knowledge) with overall ITE score (total scaled score); determine factors associated with improved correlation between Milestone 2.0 score for MK1 and overall ITE score (total scaled score)
Status: Planning
Studies Now Recruiting
Does a Gap Year in Medical Training Lead Trainees to Have an Overall Better Sense of Wellness?
PI: Masrur Khan
Synopsis: Assessing the impact of taking time off, including anticipated gap years and unanticipated time off, during training on wellness.
Status: Recruiting
Pediatric Resident Insulin Management Education: A National Needs Assessment
PI: Ellen Laves
Synopsis: Characterize the experience of pediatric residents with diabetes management tasks, including caring for patients with diabetes requiring insulin and ordering insulin in the inpatient and outpatient settings.
Status: Recruiting for Round 2
Studies Currently Ongoing
Defining Pediatric Resident Wellbeing: A Group Concept Mapping Study
PI: Jessica Babal
Synopsis: Create a conceptual model of pediatric resident wellbeing and identify differences in how stakeholder subgroups prioritize elements of pediatric resident physician wellbeing.
Status: Data collection
Defining Key Factors in Burnout and Resilience in Pediatric Residents and the Relationship to Performance
PIs: John Mahan and Maneesh Batra for The Pediatric Resident Burnout-Resilience Study Consortium
Synopsis: Create and support a consortium to:
- Describe the epidemiology and relationships between burnout, resilience, empathy, and confidence in providing compassionate care in pediatric and medicine-pediatric (P/M-P) residents.
- Define the natural history of these parameters over time in P/M-P residents during training
- Identify modifiable factors that increase or decrease the risk of developing burnout and promoting positive wellness factors
- Develop and test different interventions (such as in-person seminars, on-line mind-body skills training, and individualized curricula such as global health electives) to affect burnout, resilience, empathy, compassion and wellness
Status: Data collection and analyses in progress
Enhancing Learning with Daily Text Message PREP Questions
PI: Adin Nelson & Erika Abramson
Synopsis: The primary aim of this study is to test a program of daily spaced retrieval practice with exam style multiple choice questions. This study will consist of daily text messages sent to enrolled residents prompting them to log in to PREP and answer one specific practice question. Enrolled residents' responses will be linked to their ITE scores, which will be provided by ABP.
Status: Data collection
Are Final Residency Milestones Predictive of Early Fellowship Performance in Pediatrics?
PI: Suzanne Reed
Synopsis: Assess the correlation between Milestone levels achieved by fellowship-bound residents at the end of residency with Milestone levels achieved as first-year fellows. Determine factors associated with improved correlation between final residency Milestones and early fellowship Milestones.
Status: Data collection and analyses in progress
Crosswalk Study and Prospective Entrustment Decision Making Study
PI: Daniel Schumacher
Synopsis: Understand the association between ACGME Milestone assignments and EPA prospective entrustment decisions made by CCCs; explore factors that influence and determine prospective entrustment decisions.
Status: Data collection
Scaling the Harmonized Milestones 2.0
PI: Alan Schwartz
Synopsis: Develop response process validity evidence for the harmonized (cross-specialty) ACGME milestones 2.0. Investigate subjective differences in competence between milestones levels and tests the hypothesis that they can be considered equal intervals.
Status: Data collection and analyses in progress
Assessing the Association between Entrustable Professional Activities (EPAs), Competencies, and Milestones in the Pediatric Subspecialties
Collaborators: APPD LEARN, Council of Pediatric Subspecialties, American Board of Pediatrics
Synopsis: The specific objectives are 1) to determine if there is a specific milestone level that must be reached for each of the subcompetencies within a specific EPA to judge that a fellow is deemed entrustable, 2) to compare the milestone level at which fellows are deemed entrustable across the pediatric subspecialties and to determine if any specific subcompetencies are more influential in the entrustment decision than others and 3) to compare the initial overall impression of fellow level of entrustment for a given EPA made by the Fellowship Program Director with that determined by the Clinical Competency Committee (CCC) after the milestone levels for that EPA are assigned.
Status: Data collection complete
Pediatrics Milestones Assessment Collaborative (PMAC)
Collaborators: APPD LEARN, American Board of Pediatrics, and National Board of Medical Examiners
Project Director: Patricia Hicks
Synopsis: Develop a national Pediatrics Milestones-based Assessment System - an assessment system that will provide meaningful, feasible assessment evidence that can be used by the program leadership and clinical competency committee members to guide decisions for progression through increasing levels of responsibility and independence.
Status: Data collection complete; analyses in progress
Impact of Academic Half Days on Board Pass Rates
PI: Emily Borman-Shoap
Synopsis: Determine if there is an association between the use of academic half-days in pediatric residency programs and passage of ABP examination in the first attempt.
Status: Data collection complete; analyses in progress
Striving for the Optimal Balance Between Service and Education
PIs: Debra Boyer and Jennifer Kesselheim
Synopsis: Survey residents and program directors to (1) establish each cohorts' assessment of the balance of service and education in their training programs, (2) measure program directors' and residents' perceptions of the amount of service and education inherent in common scenarios encountered during pediatric residency training, and (3) elucidate definitions of service and education, endorsed by both program directors and trainees, that can serve as a useful foundation for curriculum development during residency training
Synopsis: Data collection complete; analyses in progress
ABS Pilot Study of Entrustable Professional Activities (EPAs)
PI: Jo Buyske
Synopsis: Collect entrustable professional activity assessment data on general surgery resident in approximately 25 residency program
Status: Data collection complete; analyses in progress
Understanding Pediatric Trainee Attitudes Towards Incorporating a Climate Change Curriculum Into Medical Training
PI: Jonathan Cogen & Mollie Grow
Synopsis: Assess pediatric knowledge on the impact of climate change and its effects on health; evaluate pediatric trainee perceived interest in the importance of incorporating a climate change curriculum into their pediatric training program.
Status: Data collection complete; analyses in progress
Identifying ERAS Filters that Predict Medical Student Success in Pediatric Residency
PI: Jessica DeBord
Synopsis: Identify medical school metrics reported via the Electronic Residency Application Service (ERAS) that predict success during pediatric residency. Involves retrospective data collection of medical school metrics.
Status: Data collection complete; analyses in progress
Childbearing and parenthood decisions in Pediatrics
PI: Kelly Dundon
Synopsis: Characterize the variability in use and length of parental leave in Pediatrics residency programs and associations with educational and parenting outcomes.
Status: Data collection and analyses complete; manuscript published
Using text messaging to help learners develop self-regulated learning skills
PI: Kimberly Gifford
Synopsis: Randomized controlled trial of text messaging to teach and assess self-regulated learning.
Status: Data collection complete; analyses in progress
The New Duty Hour Regulations--Were Changes Necessary? A Survey of Pediatric Program Directors
PI: Hilary Haftel, MD (University of Michigan)
Synopsis: This study seeks to identify what, if any, changes in Pediatric Residency Program structure were necessary to achieve compliance with the new DHR by July 2011. This specifically includes any changes to the amount or periods of resident work hours, changes in supervision, and changes in external or internal moonlighting
Status: Data collection complete
Sustainable Pediatric Nephrology Workforce Project (SUPERPOWER)
PI: Susan Halbach
Synopsis: Further understand resilience and burnout as well as factors contributing to physician wellness in both pediatric nephrology fellows and faculty
Status: Data collection complete
Assessment of Pediatrics Milestones Pilot
PI: Patricia Hicks, Steve Clyman, and Alan Schwartz
Synopsis: In partnership with the NBME, this study seeks to test the feasibility of a set of assessment instruments constructed from nine Pediatrics Milestones. Instruments are designed to inform decisions about readiness to serve in the inpatient pediatric wards, based on ratings of direct observations recorded on handheld devices. Study instruments include: structured clinical observations for rounds and history taking; a multisource feedback instrument; and, Pediatrics Milestones Classification assessment form. Study learners include interns and sub-interns, with faculty, residents, and nurses serving as raters. Features include collection of de-identified learner data in APPD LEARN, individualized learner feedback, faculty development materials for rater and feedback provider training. Outcomes from this study will be used to inform further development of the Pediatrics Milestones and APPD LEARN.
Status: Data collection completed June 2013; data analyses and manuscript writing in progress
New Professionalism Challenges in Medical Training: An Exploration of Social Networking
PI: Jennifer Kesselheim (Harvard)
Synopsis: This study surveys pediatric trainees to explore their perceptions of and experiences with social networking sites and training program policies around social media.
Status: Data collection complete
Validity of resident self-assessment using Pediatrics Milestones
PIs: Su-Ting Li and Kimberly Gifford
Synopsis: This study seeks to describe pediatric resident self-assessment of competence using the Pediatrics Milestones for 21 ACGME-selected sub-competencies, and characterize and explain variation in agreement between pediatric resident self-assessment, external assessments based on Pediatrics Milestones, and overall summative assessments of resident performance.
Status: Data collection complete
Developing Evidence for a Model for Equity in Assessment Based on the Anti-Deficit Framework Using Discrete Choice Experiments
PI: Margaret McNamara & Arianne Teherani
Synopsis: Test model of equitable assessment that avoids peer comparison, values narrative assessment, focuses on contributions to patient care beyond medical knowledge, captures learner growth, trains clinical supervisors to mitigate bias, and appreciates learner identity among self-identified underrepresented in medicine learners across institutions, clinical settings, and levels of clinical training.
Status: Data collection complete; analyses in progress
COMLEX Scores and Performance on ABP Certifying Examination
PI: Elizabeth Rodriguez-Lien
Synopsis: Retrospective data collection to investigate the relationship between COMLEX scores and performance on the ABP certifying examination.
Status: Data collection complete, data analysis in progress
Promoting Med-ed Insight into Supportive Environments (PROMISE)
PI: Jyothi Marbin
Synopsis: Describe the experiences of Under-Represented Minority residents around feelings of belonging, inclusion and community. Compare the relationship between URM resident sense of belonging and wellness. Involves surveying residents.
Status: Data collection complete; analyses in progress
Do pediatric boot camps improve performance during early internship?
PI: Molly Rideout
Collaborators: COMSEP
Synopsis: The specific aim of this project is to investigate whether participation in a pediatric-specific boot camp improves performance during initial inpatient Pediatrics or NICU rotations and after 6 months of internship, as reflected on intern performance ratings assessed with Pediatric Milestones.
Status: Data collection and analysis complete; manuscript complete
Characteristics of Clinical Competency Committee (CCC) Member Milestone Level Assignments and Recommended Advancement Decisions
PI: Dan Schumacher
Synopsis: Studies the relationship between individual CCC member judgments and and reported milestones and advancement decisions. Investigates whether program and CCC member factors moderate that relationship.
Status: Data collection complete; analyses in progress
How trainees perform on general pediatric EPAs using a rating scale of level of supervision leading up to entrustment
PIs: Dan Schumacher and Carol Carraccio
Collaborators: American Board of Pediatrics
Synopsis:
- Determine expectations for level of supervision needed by the majority of trainees over the course of and at the completion of residency training with the intention of using data to guide the value of the current list of EPAs in learner assessment.
- Assess the feasibility of using an educational intervention to meet Part 4 MOC requirements for a quality improvement project.
Status: Data collection complete
Assessment of Pediatric Emergency Medicine Fellow Milestones Benchmarks and Association with Passing of Certification Exams
PI: Paria Wilson
Synopsis: Determine if benchmark levels of competence can be defined for the 6 ACGME competency domains and 23 individual ACGME Milestones for each year of pediatric emergency medicine (PEM) fellowship; identify if Milestones ratings at graduation from PEM fellowship are a reliable proxy for passing the PEM ABP exam.
Status: Data collection complete; analyses in progress
Virtual Recruitment during the COVID-19 Pandemic: Temporary Solution or Necessary Evolution?
PI: Zachary Winthrop, Kelly McCullagh, & Debra Boyer
Synopsis: Assesses the use of virtual recruitment versus in-person recruitment modalities during interview days in residency and fellowship programs
Status: Data collection complete; analyses in progress
APPD SPIN
Validation of a Novel Task Specific Checklist for Transport Medical Control Performance by Pediatric Subspecialty Fellows
PI: Ryan Good
Synopsis: Assess the internal structure of the newly developed transport medical control (TMC) task-specific checklist for pediatric acute care subspecialty fellows; measure faculty reaction to the TMC task-specific checklist; measure the association between scores on the TMC task specific checklist and faculty level of supervision
Career Development for Fellowship Program Directors
PI: Jennifer Kesselheim
Synopsis: Determine how fellowship program directors (FPDs) prepare for and are supported in the role; determine the influence that being a FPD has on individual professional development more broadly
Relationship between Mechanical Ventilator Test Performance with Pediatric Critical Care Fellow Entrustment for Ventilatory Management
PI: Richard Mink
Synopsis: Examine whether a test of mechanical ventilation (for which validity evidence was previously obtained) can accurately predict fellow entrustment for managing the ventilator of seriously ill infants and children; determine if the percent of correct answers on a test of knowledge of mechanical ventilation can predict entrustment of pediatric critical care fellows to manage the ventilation of seriously ill infants and children and is correlated with the fellow’s assigned LOS for managing a mechanical ventilator
Status: Planning
Studies Currently Ongoing
Understanding Program Efforts to Improve Recruitment of Underrepresented Medicine Residents to Pediatric Subspecialities
PI: Jennifer Kesselheim
Synopsis: Understand how pediatric subspecialty fellowship program directors attend to data about applicants’ diversity in the fellowship recruitment process; describe what efforts pediatric subspecialty fellowship program directors take towards recruiting a diverse fellowship class; Identify the barriers pediatric subspecialty fellowship program directors face in recruiting a diverse fellowship class.
Status: Data collection
Longitudinal Evaluation of the Required Level of Supervision for Pediatric Fellows
PI: Rich Mink, MD, MACM
Synopsis:
The specific aims of this project are:
- To obtain validity evidence for the subspecialty‐specific and scholarship EPA level of supervision scales
- To determine the developmental progression of levels of supervision for all EPAs (common and subspecialty‐specific) during the 3 years of fellowship and
- To investigate whether graduating pediatric fellows are meeting the previously defined minimum levels of supervision
- To determine if the time in training at which pediatric fellows meet the previously defined minimum levels of supervision differ among the subspecialties
- To determine if there is a level of supervision for any EPA below which remediation for poor performance is performed
- For learners in which remediation was initiated, to determine if the level of supervision rating guided the decision to do so.
- To compare level of supervision assessments made by the Clinical Competency Committee (CCC) with those of the pediatric fellow
- For the Scholarship EPA, to examine the association between the level of supervision rating and the milestone level of the competencies mapped to the EPA
- To investigate the thought process of the rater in deciding what constitutes a simple versus complex case
- For the 5 EPAs that cross the generalist to subspecialist roles, to compare the level of supervision assigned to a resident at graduation to the level assigned at first assessment as a fellow
Status: Data collection
Scholarly Activity Training During Fellowship
PI: Erika Abramson, MD, MS
Synopsis:
The specific aims for this project are:
- Understand resources available across fellowship programs to support fellow scholarly activity and identify perceived barriers to effective scholarly activity training
- Understand fellowship program director attitudes toward scholarly activity training during fellowship generally and satisfaction with current fellow experiences in their own program
- Identify factors associated with a) scholarly productivity during fellowship and b) fellowship director satisfaction with scholarly activity training in their program
Status: Data collection and analyses complete; manuscript published
Central Venous Catheter Insertion Competency in Pediatric Critical Care Medicine Fellows
PI: Donald Boyer
Synopsis: Determine how a central venous catheter insertion (CVC) competency checklist performs compared to other standards of CVC insertion competency assessment in clinical practice.
Status: Data collection complete; analyses in progress
A Novel Leadership Assessment Tool for Pediatric Critical Care Fellows
PI: Angela Czaja
Synopsis: Assess the internal structure of the newly developed leadership assessment tool for pediatric critical care fellows; measure the association between scores on the leadership tool and other measures of leadership competency such as Milestones or EPAs.
Status: Data collection complete; analyses in progress
A Characterization of ECMO Training Practices in Pediatric Fellowships: Are We Doing Enough?
PI: Loren Farley
Synopsis: Characterize the currently used methods of teaching ECOM to fellows; compare training methods across specialties; investigate potential gaps
Status: Data collection complete; analyses in progress
Assessing the Association Between EPAs, Competencies and Milestones in the Pediatric Subspecialties
PIs: Bruce Herman, MD and Rich Mink, MD, MACM
Synopsis:
- For six of the seven common EPAs, to determine if there is a specific milestone level at which a fellow is deemed entrustable.
- For six of the seven common EPAs, to compare the milestone level at which fellows are deemed entrustable across the pediatric subspecialties and to determine if any specific competencies are more influential in the entrustment decision than others.
- For six of the seven common EPAs, to compare the initial overall impression of fellow level of entrustment made by the Fellowship Program Director with that determined by the Clinical Competency Committee after the milestone levels are assigned
Status: Data collection complete; analyses and manuscripts in progress
Level of Supervision for Graduating Fellows
PI: Rich Mink, MD, MACM
Synopsis:
For each of the 7 EPAs, determine the opinion of fellowship program directors (FPD) about:
- The minimum level of supervision a fellow must achieve to complete fellowship
- Whether a fellow should be allowed to graduate if this level is not met
Status: Data collection and analyses complete; manuscripts published
Utilization of EPAs in the Pediatric Subspecialties and Barriers to their Use
PI: Rich Mink, MD, MACM
Synopsis:
Primary Aim:
- To identify the facilitators and barriers to using EPAs to assess pediatric fellows
Secondary Aims:
- To determine how EPAs are currently being utilized in pediatric fellowships
- To examine the process used by CCCs in assigning the EPA level of supervision for their fellows, the time the CCC devotes to making the decision and the information used to make the decision
- To gauge FPD opinions about the value of using EPAs versus milestones to assess fellow progress in training
- To describe how a FPD’s assignment of level of supervision for the various EPAs factors into the determination of fellow readiness for graduation
- To compare and contrast responses among the subspecialties
Status: Data collection complete; analyses in progress
The Effect of the COVID-19 Pandemic on Pediatric Fellowship Training
PI: Monique Naifeh
Synopsis: Examine the impact of the COVID-19 pandemic on the education of pediatric fellows with a specific focus on clinical care, including redeployment of fellows and impact on procedural experience; educational conference structures and adaptations; and scholarly activity. Examine the impact of the COVID-19 pandemic on fellow wellness/fellowship program morale.
Status: Data collection complete; analyses in progress
National Needs Assessment of Telemedicine in Pediatric Fellowship Training
PI: Nicole Paradise Black
Synopsis: The study aims to ascertain 1) current fellow trainee exposure to telemedicine; 2) fellowship program use of formalized curriculum for telemedicine instruction and exposure; 3) barrier to implementation of telemedicine; 4) fellowship director interest in curriculum development and national collaboration.
Status: Data collection complete; analyses in progress
Residency Milestones in Fellowship Programs
PI: Suzanne Reed, MD
Synopsis:
The purpose of this study is to:
- Determine the extent to which fellowship programs utilize residency milestones for their newly entering fellows.
- Determine the perceived value of residency milestones in fellowship programs for use with newly entering fellows.
Status: Data completion and analyses complete; manuscripts published
Each subspecialty may designate up to two representatives who serve on the SPIN Steering Committee and are responsible for recruitment and communication within their subspecialty.
| Subspecialty | Representative 1 | Representative 2 |
| Adolescent Medicine | Sarah Pitts | Shellie Yussman |
| Cardiology | Brad Robinson | Ruchika Karnik |
| Child Abuse | Bruce Herman | Mary Moffatt |
| Critical Care Medicine | Angela Czaja | Don Boyer |
| Developmental and Behavioral Pediatrics | Jill Fussell | Pam High |
| Emergency Medicine | Deborah Hsu | Melissa Langhan |
| Endocrinology | Diane Stafford | Tandy Aye |
| GI | Cary Sauer | Daniel Kamin |
| Hematology-Oncology | Jennifer Kesselheim | Mark Atlas |
| Hospital Medicine | Vanessa McFadden | Michelle Lopez |
| Infectious Diseases | Scott James | Kammy McGann |
| Neonatology | Christiane Dammann | Patricia Chess |
| Nephrology | John Mahan | George Roshan |
| Pulmonary Medicine | Pnina Weiss | Jennifer Rama |
| Rheumatology | Megan Curran | Jay Mehta |
| ABP | David Turner | |
| APPD LEARN | Alan Schwartz | |
| Anjana Radhakrishnan | ||
| Avery Ulrich | ||
| APPD Fellowship Chair | Bruce Herman | |
| CoPS | Richard Mink |
Join APPD LEARN
All APPD member programs are eligible to be members of APPD LEARN. Membership may be acquired at any time by contacting the APPD LEARN Project Manager and providing the required information.
APPD LEARN members receive the following benefits:
- Opportunities to participate in research studies advancing Pediatrics Graduate Medical Education (GME)
- Eligibility to propose new research studies to be conducted by APPD LEARN, including assistance with:
- Designing studies and preparing proposals
- IRB processes
- Project Management
- Access to CME-designated APPD LEARN training opportunities
- Additional evidence in support of ACGME program requirement II.B.5 (The faculty must establish and maintain an environment of inquiry and scholarship with an active research component)
While an APPD LEARN member program is participating in a research study, it becomes an active member. APPD LEARN active members receive:
- Early access to new educational methods and tools developed through APPD LEARN
- Opportunities for group authorship or acknowledgment in study publications
- Public and professional recognition for participation in educational research
- Access to consultation by educational research specialists
APPD LEARN members have the following responsibilities:
- The program director must indicate knowledge and approval of the program’s membership in APPD LEARN.
- A liaison within the program (who may also be the program director) must be identified.
- The liaison must respond to and complete the annual APPD LEARN member survey and needs assessment.
- The program must be willing to participate in at least one research study every two years.
- A site investigator must be identified for each study in which the program is participating. The site investigator (who may also be the liaison or program director) is responsible for maintaining contact with the APPD LEARN staff and the study PI, and overseeing the implementation of the study protocol at the program.
If you are an active APPD affiliated residency program interested in becoming a part of APPD LEARN, please email learn@appd.org with the following information:
1. Institution Name
2. Program Director Name
3. Email Address
4. Phone Number
Propose a Study
APPD LEARN member programs are eligible to propose studies to be conducted on the APPD LEARN network. See the links below for general proposal instructions.
Open calls for proposals
CFP 2022-1 (member-initiated proposals)
Proposals are reviewed three times per year, with submission due dates of January 31, May 31, and September 30 each year.
Sharing data through APPD LEARN
Projects involving any number of collaborating programs that are collecting resident-level data outside of APPD LEARN and that wish to archive and share their data through APPD LEARN should contact the APPD LEARN Director as early as possible in the project to discuss how data can be archived by APPD LEARN and APPD LEARN support for data de-identification and documentation.
Any APPD LEARN member program is eligible to submit a research proposal. APPD member programs that are not already APPD LEARN member programs must join APPD LEARN before or at the time of submission of a research proposal. Proposals may include collaborators who are not members of APPD LEARN, but an APPD LEARN member program must be the lead site for the proposal.
Proposals from APPD LEARN member programs may designate any affiliated faculty member(s) of any rank as project investigators, including principal investigator. However, all proposals must be approved by the Program Director and Department Head of the member program, and should be submitted by the program’s APPD LEARN liaison.
All studies that collect data through APPD LEARN are required to deposit their approved protocols and data into the APPD LEARN data repository in order to make them available to future investigators. Publications arising from APPD LEARN studies must acknowledge APPD LEARN and their manuscripts must be provided to APPD LEARN for archiving alongside the study data.
Please submit your proposal using the APPD Research Submission portal: https://apps.appd.org/survey_submission/.
Proposals are reviewed three times per year by the APPD LEARN Proposal Review Committee (PRC) and APPD LEARN Director. Review cycles will group proposals submitted February 1-May 31, June 1-September 30, and October 1-January 31.
Criteria applied by the APPD LEARN PRC will include:
- Significance of the research question (including importance for Pediatric GME or medical education)
- Quality of the research plan (including study design, instrumentation, data analysis)
- Feasibility for APPD LEARN
Each proposal will be assigned a primary reviewer and at least two secondary reviewers. Reviewers will write narratives about each criterion and will assign each proposal an overall rating of excellent, very good, good, fair, and poor.
The Committee may make comments or suggest modifications to proposed studies to enhance any of these components. Committee recommendations are advisory. Final decisions about proposal approval or rejection will be made by the APPD LEARN Director.
Each study using the LEARN network will have an ad hoc project oversight committee, composed of the project principal investigator, APPD LEARN Director, LEARN Project Manager, and other members selected by the principal investigator and APPD LEARN Director. These committees will hold regular (biweekly or monthly) conference calls during the period of the study.
After formation of the oversight committee, the APPD LEARN Director and APPD LEARN Project Manager will assist in sampling and enrollment of APPD LEARN member sites into the study protocol and will work with participating sites to obtain IRB approval prior to data collection.
The Study Oversight Group is responsible for adopted processes for communicating with study memnbers and for scientific communications (e.g. manuscripts, presentations) that result from the study, consistent with the following APPD LEARN guidelines:
Request Data
Permitted users
- APPD LEARN members are permitted to obtain data from the repository at no charge, subject only to scientific and IRB approval discussed below.
- APPD members who are not members of LEARN may request access to repository data. These requests will be considered by the APPD LEARN Director and APPD LEARN Advisory Board, who will be responsible for determining any charges associated with providing repository data and any additional terms of use
- Investigators not associated with APPD may request access to repository data. These requests will be considered by the APPD LEARN Director and APPD LEARN Advisory Board, who will be responsible for determining any charges associated with providing repository data and any additional terms of use.
Scientific and budgetary approval
All data requests must include a summary of the data requested, research question to be answered by the data, and planned analyses. Data requests for projects that will involve additional data collection should explain whether and how newly collected data will be provided back to APPD LEARN; proposals that do not plan to share their data will be disadvantaged in review. APPD LEARN will provide a form for these data requests.
Requests received will be reviewed by the APPD LEARN Director for scientific merit and cost to APPD LEARN of providing data, and must be approved before data will be released. Requests that are denied may be appealed to the APPD LEARN Advisory Board, whose recommendations are binding and final.
IRB oversight
Because data maintained in APPD LEARN are not identifiable, use of APPD LEARN data by investigators who did not collect the data or interact with the subjects is likely to constitute research on “persons” rather than “human subjects”. In many institutions, however, investigators are required to obtain from their IRBs a determination that their proposed use of data does not constitute human subjects research. Investigators are responsible for satisfying their local requirements prior to obtaining and analyzing data.
To make a request for data, please complete and sign APPD LEARN's Data Request Form, and submit the form by emailing LEARN@appd.org. Procedures for obtaining data will be determined by the APPD LEARN Director for each request.
When several investigators are seeking to obtain data, priority will be given based on APPD LEARN membership status, scientific merit, and resource availability.
By downloading these Materials, I agree to the following:
- I will use the Materials only for academic or scientific purposes.
- I will not use the Materials to
- obtain information that could directly or indirectly identify subjects.
- produce links among the Distributor's datasets or among the Distributor's data and other datasets that could identify individuals or organizations.
- obtain information about, or further contact with, subjects known to me except where the use and/or release of such identifying information has no potential for constituting an unwarranted invasion of privacy and/or breach of confidentiality.
- I agree not to download any Materials where prohibited by applicable law.
- I agree not to use the Materials in any way prohibited by applicable law.
- I understand that although data sets may include documentation or codebooks that describe the instruments used to collect the data, these instruments are separately copyrighted, and access to the data set does not provide any license or rights to administer the instruments or reverse-engineer the instruments.
- I agree that any books, articles, conference papers, theses, dissertations, reports, or other publications that I create which employ data will reference the bibliographic citation accompanying this data and acknowledge the assistance of APPD LEARN in obtaining the data. These citations include the data authors, data identifier, and other information in accordance with the Recommended Standard (http://thedata.org/citation/standard) for social science data. I further agree to inform APPD LEARN about any such publications or presentations, and, subject to any copyright assignment or embargo agreements required by the publisher, to provide copies of publications or accepted-for-publication manuscripts to be archived alongside the data.
- The distributor makes no warranties, express or implied, by operation of Law or otherwise, regarding or relating to the dataset. I agree to indemnify and hold harmless APPD, APPD LEARN, and its officers, volunteers, agents, and employees for any claims arising out of my research using data provided through APPD LEARN.
Tools
The APPD LEARN Dataverse is a repository for all APPD LEARN study materials, including protocols and proposals, deidentified study data, and links to manuscripts reporting study results.
UIC IRB exemption for the APPD LEARN repository
Dr. Schwartz's HSPP/CITI education status for human subjects training, in case your IRB asks for evidence of CITI training.
The APPD LEARN de-identification system generates learner codes for use in reporting de-identified data to APPD LEARN in APPD LEARN studies. It is hosted at https://learn.cybermango.org/cgi-bin/xls-code.cgi. (This link only provides APPD LEARN ids - do not use it if your study has a study-specific link that also generates tailored survey urls.)
Details about this process and protecting learner privacy are available here
A description of APPD LEARN's survey platform and privacy protections is available here.
If your institution requires a data use agreement with APPD in order to provide data to APPD LEARN, this model agreement contains APPD LEARN's standard language and could be used.
APPD LEARN's authorship principles can be found here.
Individuals who submit data to a LEARN project (through either PRIME or SPIN) but who otherwise do not meet authorship criteria are considered collaborators in publications. The names of these individuals are PubMed searchable. We strongly support recognizing the academic contributions of collaborators and including information about participation in the collaborator's CV. One way to do this is to create a section in the CV entitled "Collaboration in National Studies" (or something similar) and listing the abstracts and publications under this heading. This will differentiate author vs. collaborator roles and avoid confusion.
Data Analysis
Best practices for working on data analyses with Alan
- Providing empty tables and mock-ups of desired figures is very useful to show what you'd like to see.
- Alan is supporting multiple projects at the same time, and often multiple papers/presentations within each project. This makes it challenging for him to quickly identify particular analyses he's run. You can make this much easier by:
- not renaming any analysis output files that he sends you (he can use the filename to quickly find your project/paper/etc on his computer), and
- attaching a copy of that analysis output file whenever asking questions about an analysis
