APPD LEARN is the APPD's Longitudinal Educational Assessment Research Network.
APPD LEARN provides an infrastructure for multi-centered, collaborative research projects and a centralized data collection system. LEARN encompasses three research networks that share the APPD LEARN infrastructure and governance.
The APPD Pediatric Residency Investigators in Medical Education (APPD PRIME) network was APPD LEARN's first network and focuses on research on Pediatric residents and residency programs.
The APPD Subspecialty Pediatrics Investigator Network (APPD SPIN) is a medical education research network for the Pediatric Subspecialties to evaluate the education and assessment of fellows.
The APPD Surveys Enhancing Educator Knowledge (SEEK), formerly the survey component of the Research and Scholarship Learning Community, is a research network that supports projects on program operations and projects designed to study novel curricular or evaluation instruments that will allow programs to address and measure ACGME competencies.
About APPD LEARN
The mission of APPD LEARN is to conduct meaningful educational research that advances the training of future Pediatricians by developing and promoting participation and collaboration in research by program directors for the purpose of improving the health and well-being of children. Our vision is to advance exemplary pediatric education through collaborative educational research by Pediatric program directors.
APPD LEARN pursues its mission through the following interconnected activities:
- Managing collaborative research networks of Pediatric Programs working together to conduct multi-site studies of educational methods and instruments
- Maintaining an online repository of educational research study materials, raw data, and findings for dissemination to APPD members and collaborators
- Promoting learning opportunities to enhance educational research participation and scholarship by Program Directors
- Providing expert consultation for research conducted within APPD LEARN
- Communicating regularly with the APPD membership and the larger medical education community about activities, opportunities, and outcomes
- Exploring, conducting, and coordinating research with other organizations and initiatives across a continuum of medical and non-medical education
The development of APPD LEARN began from APPD's experience with Share Warehouse, its site for sharing assessment tools and instructional materials among APPD member programs, and the R3P initiative. Planning for APPD LEARN began in 2006. A strategic plan was presented by Ann Burke, Patty Hicks, Susan Guralnick, and Rob McGregor in 2008. APPD and the Initiative for Innovation in Pediatric Education (IIPE) began allocating resources to support APPD LEARN and the initial Advisory Committee was formed in 2009.
APPD LEARN's first Director was Hilary Haftel. She was succeeded by Alan Schwartz in 2011
APPD LEARN Staff
 |
 |
| Alan Schwartz, PhD APPD LEARN / PRIME Director |
Richard Mink, MD, MACM APPD SPIN Director |
 |
 |
| Avery Ulrich, MS APPD LEARN Research Assistant |
Anna Sombrio, MS APPD LEARN Research Manager |
APPD LEARN Advisory Committee, 2023-2026
- Maneesh Batra, MD, MPH (2023-2025)
- Debra Boyer, MD, MHPE (2023-2025)
- Megan Aylor, MD (2024-2026)
- Nicola Orlov, MD (2024-2026)
- Diane Stafford, MD (2024-2026)
- Joanna Lewis, MD (2024-2026)
- David Turner, MD (non-voting, ABP)
- Daniel Schumacher, MD, PhD, MEd (non-voting, Crosswalk/CBME Data Hub)
- Linda Waggoner-Fountain, MD (non-voting, APPD LESSON)
- Audrea Burns, PhD (non-voting, APPD RSLC)
- Richard Mink, MD, SPIN network (non-voting)
- Alan Schwartz, PhD, APPD LEARN Director
- Laura Degnon, CAE, APPD Executive Director (ex officio)
APPD LEARN Proposal Review Committee (PRC), 2023-2025
- Alan Chin, MD (2023-2025)
- Cary Sauer, MD, MSc (2023-2025)
- Shoshana Friedman, MD (2024-2026)
- Barbara Gohre, BS (2024-2026)
- Ryan Good, MD (2024-2026)
- Sara Kane, DO, MSEd (2024-2026)
- Adin Nelson, MD, MHPE (2024-2026)
- Priti Tewari, MD (2024-2026)
- Eric Zwemer, MD (2024-2026)
APPD LEARN Director
The APPD LEARN Director is responsible for daily scientific and management oversight of all APPD LEARN activities, and serves as liaison to outside organizations. The Director reports to the APPD LEARN Advisory Committee and APPD Board. The APPD LEARN Director serves for a term of two years, renewable by the APPD Board indefinitely, at a time commitment of at least 0.4 FTE.
APPD SPIN and APPD PRIME Directors
The APPD SPIN and APPD PRIME Directors manage the operations of the APPD SPIN and APPD PRIME subnetworks, respectively.
APPD LEARN Research Manager
The APPD LEARN Research Manager handles all administrative operations of APPD LEARN. Responsibilities of this full-time position include facilitating all communications among the participating institutions and funding agency, development of and adherence to project timelines, and observance of regulatory requirements. The APPD LEARN Research Manager is responsible for daily implementation of the project plan, and assisting PIs with study management. They serve as liaison with research administration at all participating institutions. In addition, the APPD LEARN Research Manager helps with additional network tasks and data collection efforts that arise. The APPD LEARN Research Manager works in conjunction with the APPD LEARN Director, and reports to the APPD Executive Director.
APPD LEARN Advisory Committee
The APPD LEARN Advisory Committee is be responsible for providing guidance to the APPD LEARN Director, setting policies for APPD LEARN activities and resources, and developing calls for proposals in specific research areas. The APPD LEARN Advisory Committee also conducts annual formative and summative evaluation of the performance of the APPD LEARN Director. The APPD LEARN Advisory Committee meetings monthly by conference call and annually in person.
The Committee consists of five voting members, including four APPD-designated members (one of whom serves as the Chair), and one IIPE-designated member. The APPD Board selects the Chair of the APPD LEARN Advisory Committee. In consultation with the Chair of the APPD LEARN Advisory Committee, the APPD Board selects the three additional APPD members. IIPE appoints one representative to the committee. APPD LEARN Advisory Committee terms are 2 years long, and renewable. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director serve ex officio on the Advisory Board.
APPD LEARN Proposal Review Committee (PRC)
The APPD LEARN Proposal Review Committee will assist the LEARN Director in the review of proposals to conduct research using the LEARN network and its member Programs. The committee chair and members will serve two-year terms, which may be renewed. They will be appointed by the APPD LEARN Advisory Committee, in consultation with the APPD LEARN Director, after soliciting suggestions from the APPD Council of Task Force Chairs. The APPD LEARN Director, APPD LEARN Project Manager, and APPD Executive Director will serve ex officio on the Committee. The committee will meet as needed by conference call, at a frequency no more than monthly, with online discussion as needed between calls.
APPD LEARN Project Committees
Each study using the APPD LEARN network will have an ad hoc project oversight committee, composed of the project principal investigator, APPD LEARN Director, APPD LEARN Project Manager, and other members selected by the principal investigator and APPD LEARN Director. These committees will hold regular (e.g. biweekly) conference calls during the period of the study.
Member Programs
AdventHealth Orlando
Advocate Health Care (Advocate Children’s Hospital/Oak Lawn)
Advocate Health Care (Advocate Children's Hospital/Park Ridge)
Albany Medical Center
Atlantic Health/Goryeb Children's Hospital
Baylor College of Medicine (Houston)
Baylor College of Medicine (San Antonio)
Brookdale University Hospital and Medical Center
Brooklyn Hospital Center
Brown University
CAMC/West Virginia University (Charleston Division)
Carilion Clinic-Virginia Tech Carilion School of Medicine
Carolinas Medical Center
Case Western Reserve Univ/Univ Hosps Cleveland Med Ctr/Rainbow Babies and Children’s Hospital
Case Western Reserve University (MetroHealth)
Children's Hospital Los Angeles
Children's Hospital Medical Center of Akron/NEOMED
Children's Hospital Oakland (UCSF Benioff Children's Hospital Oakland)
Children's Hospital of Michigan
Children's Hospital of Philadelphia
Children's Hospital/Boston Medical Center
Children's Mercy Hospital
Children's National Medical Center
Cincinnati Children's Hospital Medical Center
Cleveland Clinic Foundation
Cooper Medical School of Rowan University/Cooper University Hospital
Crozer-Chester Medical Center
Dartmouth-Hitchcock/Mary Hitchcock Memorial Hospital
Driscoll Children's Hospital
Duke University Hospital
East Tennessee State University
Eastern Virginia Medical School
Emory University
Flushing Hospital Medical Center
Hackensack University Medical Center
Hospital Episcopal San Lucas-Ponce School of Medicine
Hurley Medical Center/Michigan State University
Icahn School of Medicine at Mount Sinai (Kravis Children's)
Icahn School of Medicine at Mount Sinai (NYC Health and Hospitals/Elmhurst)
Indiana University School of Medicine
Inova Fairfax Medical Campus/Inova Children's Hospital
Jacobi Medical Center/Albert Einstein College of Medicine
Jersey Shore University Medical Center
John H. Stroger Jr. Hospital of Cook County
Johns Hopkins All Children's Hospital
Johns Hopkins University
Kaiser Permanente Medical Group (Northern California)
Kaiser Permanente Southern California (Los Angeles)
Kirk Kerkorian School of Medicine at UNLV Program
Lehigh Valley Health Network/University of South Florida College of Medicine
Loma Linda University Health Education Consortium
Los Angeles County-Harbor UCLA Medical Center
Louisiana State University
Louisiana State University (Shreveport)
Loyola University Medical Center
Maine Medical Center
Marshall University School of Medicine
Massachusetts General Hospital
Mayo Clinic College of Medicine and Science (Rochester)
McGaw Medical Center of Northwestern University
Medical Center of Central Georgia/Mercer University School of Medicine
Medical College of Georgia
Medical College of Wisconsin Affiliated Hospitals
Medical University of South Carolina
Memorial Health-University Medical Center/Mercer University School of Medicine (Savannah)
Mercy St. Vincent Medical Center
Monmouth Medical Center
Montefiore Medical Center/Albert Einstein College of Medicine
Morehouse School of Medicine
National Capital Consortium
Nationwide Children's Hospital/Ohio State University
Naval Medical Center (Portsmouth)
Naval Medical Center (San Diego)
Nemours Children's Hospital Program
New York Presbyterian Hospital (Cornell Campus)
New York University School of Medicine
Newark Beth Israel Medical Center
Nicklaus Children's Hospital
Oregon Health and Science University
Orlando Health/Arnold Palmer Hospital for Children
Our Lady of the Lake
Penn State Milton S. Hershey Medical Center
Phoenix Children's Hospital
Prisma Health-Midlands/University of South Carolina School of Medicine-Columbia, SC
Prisma HealthUpstate/University of South Carolina School of Medicine Greenville
Rush University Medical Center
Rutgers New Jersey Medical School
Sidney Kimmel Medical College at Thomas Jefferson University/duPont Hospital for Children
Southern Illinois University
Spectrum Health/Michigan State University/Helen DeVos Children's Hospital
St. Christopher's Hospital for Children
St. Joseph’s University Medical Center
St. Louis University School of Medicine
St. Vincent Hospital and Health Care Center
Stanford University
Stony Brook Medicine/University Hospital
SUNY Health Science Center at Brooklyn
SUNY Upstate Medical University
Texas A&M College of Medicine-Scott and White Medical Center (Temple)
Texas Tech University Health Sciences Center at Lubbock
Tripler Army Medical Center
Tucson Hospitals Medical Education Program
Tufts Medical Center
Tulane University
UCLA David Geffen School of Medicine/UCLA Medical Center
UMMS-Baystate
University at Buffalo
University of Alabama Medical Center
University of Arizona
University of Arizona College of Medicine-Tucson
University of Arkansas for Medical Sciences
University of California (Davis) Health System
University of California (San Diego)
University of California (San Francisco)
University of California (San Francisco)/Fresno
University of Chicago
University of Colorado
University of Connecticut
University of Florida
University of Florida (Orlando)
University of Florida College of Medicine-Jacksonville
University of Hawaii
University of Illinois College of Medicine at Chicago
University of Illinois College of Medicine at Peoria
University of Iowa Hospitals and Clinics
University of Kansas (Wichita)
University of Louisville
University of Maryland
University of Massachusetts
University of Michigan
University of Minnesota
University of Nebraska Medical Center College of Medicine
University of New Mexico
University of North Carolina Hospitals
University of Oklahoma Health Sciences Center
University of Oklahoma School of Community Medicine (Tulsa)
University of Rochester
University of South Alabama
University of South Dakota
University of South Florida Morsani
University of Southern California/LAC&USC Medical Center
University of Tennessee
University of Tennessee College of Medicine at Chattanooga
University of Texas at Austin Dell Medical School Pediatric Program
University of Texas Health Science Center at Houston
University of Texas Health Science Center School of Medicine at San Antonio
University of Texas Medical Branch Hospitals
University of Texas Southwestern Medical School
University of Toledo
University of Utah
University of Vermont Medical Center
University of Virginia
University of Washington
University of Wisconsin
UPMC Medical Education
Vanderbilt University
Vidant Medical Center/East Carolina University
Virginia Commonwealth University Health System
Wake Forest University School of Medicine
Washington University/St. Louis Children's
Westchester Medical Center
Western Michigan University Homer Stryker MD School of Medicine
Wright State University
Yale-New Haven Medical Center
Zucker School of Medicine at Hofstra/Northwell at Cohen Children's Medical Center
Zucker School of Medicine at Hofstra/Northwell at Staten Island University Hospital
PAPERS
2024
Mink R, Schwartz A, Mahan J, Fussell J, George R, Schumacher D, McFadden V, Turner D, Atlas M and APPD SPIN. Level of Supervision for the Entrustable Professional Activities Common to General Pediatrics and the Subspecialties Decreases from Residency to Fellowship." Acad Pediatr 2024 doi: 10.1016/j.acap.2024.04.004.
Melissa L. Langhan, Donald L. Boyer, Deborah Hsu, Mary E. Moffatt, Sarah A. Pitts, Mark P. Atlas, Tandy Aye, Patricia Chess, Megan L. Curran, Angela S. Czaja, Christiane E.L. Dammann, Jill Fussell, Roshan P. George, Bruce E. Herman, Pamela High, Scott H. James, Daniel S. Kamin, Ruchika Karnik, Jennifer Kesselheim, Michelle A. Lopez, John D. Mahan, Vanessa McFadden, Kathleen A. McGann, Jay J. Mehta, Jennifer Rama, Bradley W. Robinson, Cary Sauer, Diane E.J. Stafford, David A. Turner, Pnina Weiss, Susan M. Yussman, Alan Schwartz, Richard Mink; Implementing Entrustable Professional Activities in Pediatric Fellowships: Facilitating the Process. Pediatrics 2024; e2023065024. 10.1542/peds.2023-065024
Chiel L, Gans HA, Bales M, et al. A Pilot Survey on the Perceptions of Pediatric Subspecialty Fellows and Program Directors on Virtual Compared with In-person Interview Experiences. J Pediatr. Published online March 19, 2024. doi:10.1016/j.jpeds.2024.114019
2023
Cogen JD, Perkins A, Mockler B, Barton KS, Schwartz A, Boos M, Radhakrishnan A, Rai P, Tandon P, Philipsborn R, Grow HM, on behalf of the APPD LEARN Climate Change Study Group. Pediatric Resident and Program Director Views on Climate Change and Health Curricula: A Multi-Institution Study. Academic Medicine ():10.1097/ACM.0000000000005633, January 12, 2024. | DOI: 10.1097/ACM.0000000000005633
Liu A, Ben-Zion S, Schwartz A, Mahan JD, Reed S. Well-being Factors Associated with Confidence in Providing Calm, Compassionate Care in Pediatric Residents. Patient Educ Couns. 2023 Oct;115:107906. doi: 10.1016/j.pec.2023.107906. Epub 2023 Jul 18. PMID: 37478547.
Yemane L, Kas-Osoka O, Burns A, Blankenburg R, Kester Prakash L, Poitevien P, Schwartz A, Taylor Lucas, C, Marbin J. Upholding our PROMISE: Underrepresented in Medicine Pediatric Residents’ Perspectives on Programmatic Interventions to Promote Belonging. Acad Med. 2023 Dec 1;98(12):1434-1442. doi: 10.1097/ACM.0000000000005443. Epub 2023 Aug 30. PMID: 37643583.
Zwemer E, Serwint JR, Sieplinga K, Wilson PM, Webber S, Lien ER, Babal JC, Donnelly KM, Nichols MH, Batra M, Schwartz A, Reese JW. Implementation and maturity of clinical learning environment components across pediatric residency programs. Acad Pediatr. 2023 Apr 22:S1876-2859(23)00142-0. doi: 10.1016/j.acap.2023.04.008. Epub ahead of print. PMID: 37094643.
Reed, Suzanne MD, MAEd; Mink, Richard MD, MACM; Stanek, Joseph MS; Tyrrell, Laura MD, MAEd; Li, Su-Ting T. MD, MPH. Are Final Residency Milestones Correlated with Early Fellowship Performance in Pediatrics?. Academic Medicine ():10.1097/ACM.0000000000005215, March 22, 2023. |DOI: 10.1097/ACM.0000000000005215
Alan Schwartz, Beth King, Richard Mink, Teri Turner, Erika Abramson, Rebecca Blankenburg, Laura Degnon; The APPD Longitudinal Educational Assessment Research Network’s First Decade. Pediatrics 2023; e2022059113. 10.1542/peds.2022-059113
2022
Halbach SM, Pillutla K, Seo-Mayer P, Schwartz A, Weidemann D, Mahan JD. (2022) Burnout in Pediatric Nephrology Fellows and Faculty: Lessons From the Sustainable Pediatric Nephrology Workforce Project. Frontiers in Pediatrics, 10. 10.3389/fped.2022.849370.
2021
Abramson EK, Weiss P, Naifeh M, Stevenson, Duncan J, Rama J, Mauer E, Gerber L, T. Li ST, MD. (2021) Scholarly Activity Training during Pediatric Fellowship: A National Assessment. Pediatrics, 147(1) e2020013953.
Fitzgerald M, Bhatt A, Thompson LA, Schwartz A, Thomas AO, Schinasi DA, Otero J, Carpenter P, Thomas JS, Paradise Black N. (2021) Telemedicine in Pediatric Training: A National Needs Assessment of the Current State of Telemedicine Education in Pediatric Training. Academic Pediatrics, (in press).
Pitts S, Schwartz A, Carraccio C, Herman B, Mahan J, Sauer C, Dammann C, Mink R; SPIN Steering Committee. Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activites Across Subspecialties . (2021) Academic Pediatrics, (pre-print published) https://doi.org/10.1016/j.acap.2021.12.019
Schwartz A, Borman-Shoap E, Carraccio C, Herman B, Hobday P, Kaul P, Long M, O'Connor M, Mink R, Schumacher D, Turner DA, West DC. (2021). Learner Levels of Supervision Across the Continuum of Pediatrics Training. Academic Medicine, 96(7S): S42-S49. http://dx.doi.org/10.1097/ACM.0000000000004095 (PRIME and SPIN)
Turner DA, Schwartz A, Carraccio C, Herman B, Weiss P, Baffa J, Chess P, Curran M, Dammann C, High P, Hsu D, Pitts S, Sauer C, Aye T, Fussell J, Kesselheim J, Mahan J, McGann K, Myers A, Mink R. (2021), Continued Supervision for the Common Pediatric Subspecialty
Entrustable Professional Activities May Be Needed Following Fellowship Graduation. Academic Medicine, 96(7S):S22-S28. doi: 10.1097/ACM.0000000000004091.
Weiss P, Schwartz A, Carraccio C, Herman B, Mink R. (2021), Minimum Supervision Levels Required by Program Directors for Pediatric Pulmonary Fellow Graduation. ATS Scholar, (open access). https://dx.doi.org/10.34197/ats-scholar.2021-0013OC
Weiss P, Schwartz A, Carraccio C, Herman B, Turner DA, Aye T, Fussell J, Kesselheim J, Mahan J, McGann K, Myers A, Stafford DEJ, Chess P, Curran M, Damman CEL, High P, Hsu DC, Pitts S, Sauer C, Srivastava S, Mink R. (2021), Achieving Entrustable Professional Activities During Fellowship. Pediatrics, 148(5):e2021050196. https://doi.org/10.1542/peds.2021-050196
2020
Hu K, Hicks PJ, Margolis M, Carraccio C, Osta A, Winward ML, Schwartz, A. (2020). Reported Pediatrics Milestones (Mostly) Measure Program, Not Learner Performance. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003644. [RIME New Investigator Award winner 2020].
Kemper KJ, Schwartz A, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2020). Bullying, Discrimination, Sexual Harassment, and Physical Violence: Common and Associated with Burnout in Pediatric Residents. Academic Pediatrics. Published ahead of print: https://www.academicpedsjnl.net/article/S1876-2859(20)30080-2/fulltext
Kemper KJ, Schwartz A, Wilson PM, Mahan JD, Schubert CJ, Staples BB, McClafferty H, Serwint JR, Batra M. (2020). Burnout in Pediatric Residents: Three Year of National Survey Data. Pediatrics, 145(1): e20191030. doi: 10.1542/peds.2019-1030
Mink R, Herman BE, Carraccio C, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DEJ, Weiss P, Curran ML, Dammann CEL, High PC, Hsu D, Kesselheim JC, Mahan JD, McGann KA, Myers AL, Pitts S, Turner DA, Schwartz A. J. (2020) Agreement of Program Directors With Clinical Competency Committees for Fellow Entrustment. Med Educ Curric Dev. 2020 Aug 6;7:2382120520936613. doi: 10.1177/2382120520936613. eCollection 2020 Jan-Dec.
Reed S, Mink R, Li ST. Utility of Residency Milestones Reported to Fellowship Directors: A National Survey of Pediatric Fellowship Program Directors. (2020) Acad Pediatr DOI: 10.1016/j.acap.2020.01.004.
Schumacher DJ, Poynter S, Burman N, Elliott SP, Barnes M, Gellin C, Gonzalez del Rey J, Sklansky D, Thoreson L, King B, Schwartz A, APPD LEARN CCC Study Group. (2020). Concordance of Clinical Competency Committee and Program Director Recommendations for Resident Supervisory Roles. Academic Pediatrics.
Schumacher DJ, Schwartz A, Zenel JA, Black NP, Ponitz K, Blair R, Traba CM, Poynter S, King B, Englander R, Rosenberg A, Patel D, Smith-King C, O'Connor M, Gonzalez del Rey J, Lavoie S, Borman-Shoap E, Carraccio C, for the APPD LEARN General Pediatrics EPAs Study Group. (2020). Narrative Performance Level Assignments at Initial Entrustment and Graduation: Integrating EPAs AND Milestones to Improve Learner Assessment. Academic Medicine. doi: 10.1097/ACM.0000000000003300.
Schumacher DJ, West DC, Schwartz A, Li S, Millstein L, Griego EC, Turner TL, Herman BE, Englander R, Hemond J, Hudson V, Newhall L, McNeal Trice K, Baughn J, Giudice E, Famiglietti H, Gifford K, Carraccio C, for the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network General Pediatrics EPAs Study Group. (2020). Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities: A Multi-Site Study. JAMA Network Open, 3(1):e1919316.
Schwartz A, Balmer DF, Borman-Shoap E, Chin A, Henry D, Herman BE, Hobday P, Lee JH, Multerer S, Myers RE, Ponitz K, Rosenberg A, Soep JB, West DC, Englander R. (2020). Shared Mental Models among Clinical Competency Committees in the Context of Time-Variable-Competency-Based Advancement to Residency. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003638
Schwartz A, King B, Mink R, Hicks PJ. (2020). The emergence and spread of practice-based medical education research networks. Academic Medicine (RIME Supplement). doi: 10.1097/ACM.0000000000003641
Staples BB, Burke AE, Batra M, Kemper KJ, Schwartz A, Wilson PM, Schubert CJ, Mahan JD, Serwint JR. Pediatric Resident Burnout-Resilience Study Consortium. (2020). Burnout and association with resident performance as assessed by Pediatric Milestones: An exploratory study. Academic Pediatrics. https://doi.org/10.1016/j.acap.2020.08.006
Webber S, Schwartz A, Kemper KJ, Batra M, Mahan JD, Babal J, Sklansky DJ, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium (PRB-RSC) (2020). Faculty and Peer Support during Pediatric Residency: Association with Performance Outcomes, Race and Gender. Academic Pediatrics. https://doi.org/10.1016/j.acap.2020.08.009
2019
Kemper KJ, Wilson P, Schwartz A, Mahan JD, Batra M, Staples B, McClafferty H, Schubert CJ, Serwint JR, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2019). Burnout in Pediatric Residents: Comparing Brief Screening Questions to the Maslach Burnout Inventory. Academic Pediatrics.
Schumacher DJ, Martini A, Bartlett KW, King B, Calaman S, Garfunkel LC, Elliott SP, Frohna JG, Schwartz A, Michelson CD, and the APPD LEARN CCC Study Group. (2019). Key Factors in Clinical Competency Committee Members' Decisions Regarding Residents' Readiness to Serve as Supervisors: A National Study. National Study of Clinical Competency Committee Members. Academic Medicine, 94(2): 251-258. doi: 10.1097/ACM.0000000000002469
561-565.
Schumacher DJ, Parrigan A, Bartlett KW, King B, Calaman S, Garfunkel LC, Elliott SP, Frohna JG, Schwartz A, Michelson C, and the APPD CCC Study Group. (2019). Key Factors for Recommending Whether Residents May Serve as Supervisors: A National Study of Clinical Competency Committee Members. Academic Medicine.
Schumacher DJ, Poynter S, Burman N, Elliott SP, Barnes M, Gellin C, Gonzalez del Rey J, Sklansky D, Thoreson L, King B, Schwartz A, APPD LEARN CCC Study Group. (2019). Justifications for Discrepancies Between Clinical Competency Committee and Program Director Recommendeed Resident Supervisory Roles. Academic Pediatrics, 19(5): 561-565.
2018
Abramson EL, Naifeh M, Stevenson M, Mauer E, Hammad H, Gerber L, Li S. (2018). Scholarly Activity Training During Residency: Are we Hitting the Mark? A National Assessment of Pediatric Residents. Academic Pediatrics.
Hicks PJ, Margolis MJ, Carracio CL, Clauser BE, Donnelly K, Fromme HB, Gifford KA, Poynter SE, Schumacher DJ, Schwartz A, PMAC Module 1 Study Group. (2018). A novel workplace-based assessment for competency-based decisions and learner feedback. Medical Teacher, 40:11, 1143-1150.
Kemper KJ, McClafferty H, Wilson P, Serwint JR, Batra M, Mahan JD, Schubert CJ, Staples BB, Schwartz A, on behalf of the Pediatric Resident Burnout-Resilience Study Consortium. (2018). Do Mindfulness and Self-Compassion Predict Burnout? Academic Medicine. Published ahead of print.
Li ST, Schwartz A, Burke AE, Guralnick S, Trimm RF, Guillois A, Mahan JD, Gifford K, Tancredi DJ. (2018). Pediatric Program Director Minimum Milestone Expectations Before Allowing Supervision of Others and Unsupervised Practice. Academic Pediatrics. doi: 10.1016/j.acap.2020.05.011
Li ST, Tancredi DJ, Schwartz A, Guillot A, Burke A, Trimm RF, Guralnick S, Mahan JD, Gifford KA, for the APPD LEARN Validity of Resident Self-Assessment Group. (2018). Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory Ratings. Academic Medicine
Mink R, Schwartz A, Carraccio C, High P, Dammann C, McGann K, Kesselheim J, Herman B and the Steering Committee of the Subspecialty Pediatrics Investigator Network. (2018). Creating the Subspecialty Pediatrics Investigator Network (SPIN). Pediatrics, 192:3-4.e2. doi: 10.1016/j.jpeds.2017.09.079.
Mink RB, Schwartz A, Herman BE, Turner DA, Curran ML, Myers A, Hsu DC, Kesselheim JC, Carraccio CL, for the Subspecialty Pediatrics Investigator Network (SPIN). (2018). Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities. Academic Medicine, 93:2, 283-291.
Park YS, Hicks PJ, Carraccio CL, Margolis MJ, Schwartz A, for the PMAC Module 2 Study Group. (2018). Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth from Ten Pediatrics Residency Programs. Academic Medicine, 93(11): S21-S29.
Reed SR, Kemper KJ, Schwartz A, Batra M, Staples BB, Serwint JR, McClafferty H, Schubert CJ, Wilson PM, Rakowsky A, Chase M, Mahan JD. (2018). Variability of Burnout and Stress Measures in Pediatric Residents: an Exploratory Single Center Study from the Pediatric Resident Burnout - Resilience Study Consortium. Journal of Evidence-Based Integrative Medicine.
Schumacher DJ, Bartlett KW, Elliott SP, Michelson C, Sharma T, Garfunkel LC, King B, Schwartz A, and the APPD LEARN CCC Study Group. (2018). Milestone Ratings and Supervisory Role Categorizations Swim Together, but is the Water Muddy? Academic Pediatrics.
Schumacher DJ, King B, Barnes MM, Elliott S, Gibbs K, McGreevy JF, Gonzalez del Rey J, Sharma T, Michelson C, Schwartz A, and the APPD LEARN CCC Study Group. (2018). The Influence of the Review Process Utilized by Clinical Competency Committee Members on Their Summative Assessment Decisions of Residents. Journal of Graduate Medical Education.
Schumacher DJ, Michelson C, Poynter S, Barnes MB, Li S, Burman N, Sklansky DJ, Thoreson L, Calaman S, King B, Schwartz A, and the APPD LEARN CCC Study Group. (2018). Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns. Medical Teacher, 40(1): 70-79.
2017
Kesselheim JC, Schwartz A, Boyer D, for the APPD LEARN Study Group on Education and Service. (2017). Integrating education and service in Pediatric residency training: Results of a national survey. Academic Pediatrics, 17:8, 907-914
Li ST, Tancredi DJ, Schwartz A, Guillot AP, Burke AE, Trimm F, Guralnick S, Mahan JD, Gifford KA, for the APPD LEARN Validity of Resident Self-Assessment Group.. (2017). Competent for unsupervised practice: Use of Pediatric residency training milestones to assess readiness. Academic Medicine, 92:3, 385-393.
Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A; the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network–National Board of Medical Examiners Pediatrics Milestones Assessment Group. (2017). Validity Evidence From Ratings of Pediatric Interns and Subinterns on a Subset of Pediatric Milestones. Academic Medicine, 92(6):809-819. doi: 10.1097/ACM.0000000000001622. PMID: 28557947.
2016
Hicks PJ, Margolis M, Poynter S, Chaffinch C, Tenney-Soeiro R, Turner T, Waggoner-Fountain L, Young R, Clyman SG, Schwartz A, for the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). The Pediatrics Milestones Assessment Pilot: Development of Workplace-based Assessment. Academic Medicine, 91:5, 701-709.
Kesselheim JC, Schwartz A, Belmonte F, Boland KA, Poynter S, Batra M, for the APPD LEARN Study Group on Social Media and Professionalism. (2016). A National Survey of Pediatric Residents' Professionalism and Social Networking: Implications for Curriculum Development. Academic Pediatrics, 16:2, 110-114
Schwartz A, Margolis M, Multerer S, Haftel H, Schumacher DJ, and the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). A multi-source feedback tool for measuring a subset of Pediatrics Milestones. Medical Teacher, 38:10.
Turner TL, Bhavaraju VL, Luciw-Dubas UA, Hicks PJ, Multerer S, Osta A, McDonnell J, Poynter S, Schumacher DJ, Tenney-Soeiro R, Waggoner-Fountain L, Schwartz A, for the APPD LEARN - NBME Pediatrics Milestones Assessment Group. (2016). Assessment of Pediatric interns and sub-interns on a subset of Pediatric Milestones. Academic Medicine, 92:6, 809-819.
2014
Schwartz A, Young R, Hicks PJ; for APPD LEARN. (2014). Medical Education Practice-Based Research Networks: Facilitating collaborative research. Medical Teacher, 38:1, 64-74.
Tocco D, Jain AV, Baines H. (2014). The pediatrics milestone pilot project: perspectives of current pediatric residents. Acad Pediatr, 14(2 Suppl), S8-9.
PRESENTATIONS
2024
Schumacher DJ, Kinnear B, Michelson C, Winn A, Turner DA, Martini A, Schwartz A for the APPD LEARN Crosswalk Study Group. (2024, September). “Readiness for Unsupervised Practice for the 17 ABP General Pediatrics EPAs: Are Graduating Residents Ready to Provide the Care Patients Need?” Paper Session Presentation, International Conference on Residency Education, Ottawa, Ontario, Canada.
Schumacher DJ, Kinnear B, Michelson C, Winn A, Turner DA, Martini A, Schwartz A for the APPD LEARN Crosswalk Study Group. (2024, September). Readiness for Unsupervised Practice for the 17 ABP General Pediatrics EPAs: Are graduating residents ready to provide the care patients need? Oral Presentation, American Board of Medical Specialties Conference, Chicago, IL.
Schumacher DJ, Kinnear B, Michelson C, Winn A, Turner DA, Martini A, Schwartz A for the APPD LEARN Crosswalk Study Group. (2024, August). “Readiness for Unsupervised Practice for the 17 ABP General Pediatrics EPAs: Are Graduating Residents Ready to Provide the Care Patients Need?” Short Presentation, 4th World Summit on Competency-based Education in the Health Professions, Basel, Switzerland.
Kakara Anderson HL, West DC, Turner DA, Schwartz A, Schumacher DJ for the APPD LEARN Crosswalk Study Group. (2024, April). "EPA Supervision Scale Ratings and Experiences of Inequity by Race/Ethnicity, Gender, and Socioeconomic Characteristics: Findings from the Crosswalk Study." Platform Presentation, APPD 2024 Annual Spring Meeting, Chicago, IL.
Schumacher DJ, Turner DA, Kinnear B, Winn A, Michelson C, Schwartz A for the APPD LEARN Crosswalk Study Group. (2024, April). Readiness for Unsupervised Practice for the 17 ABP General Pediatrics EPAs: Are graduating residents ready to provide the care patients need? Platform Presentation, APPD 2024 Annual Spring Meeting, Chicago, IL.
Pitts S, Schwartz A, Karnik R, High P, Stafford D, Aye T, Langhan M, Boyer D, Mink R. ( 2024, April). Are Pediatric Subspecialty Fellows Ready for Unsupervised Practice After Two Years of Training? Platform Presentation. APPD 2024 Annual Spring Meeting, Chicago, IL.
2023
Schumacher DJ, Kinnear B, Michelson C, Winn A, Turner DA, Martini A, Schwartz A for the APPD LEARN Crosswalk Study Group. (2023, September). How Many EPAs Should Be Assessed to Make a Reliable Overall Entrustment Decision About Pediatric Residents?” Platform Presentation, American Board of Medical Specialties Conference 2023, Chicago, IL.
2021
Zuniga L, Schwartz A, Batra M, Mahan JD, on behalf of the PRB-RSC. (2021, May). Burnout in the time of COVID: An unexpected story of stress and resilience. Pediatric Academic Societies, 2021 Virtual Meeting.
Zuniga L, Schuh A, Seo-Mayer P, Kapklein M, Cramton R, Sieplinga K, Kaushik R, Schwartz A, Batra M, Mahan JD on behalf of the PRB-RSC. (2021, March). Burnout in the time of COVID: You don't know until you measure it! APPD Spring Annual Meeting, Virtual.
2019
Mink R, Carraccio C, Herman B, Hsu D, Chess P, High P, Langhan M, McGann K, Curran M, Stafford D, Czaja A, Dammann C, Mahan J, Aye T, Schwartz A. (2019, May). Are Graduating Pediatric Fellows Meeting the Expected Level of Supervision for the Common Subspecialty Entrustable Professional Activities (EPAs)? Medical Education: Assessment: Pediatric Academic Societies, Baltimore, MD.
Reed S, Mink R, Li ST. (2019, May). Utility of Residency Milestones Reported to Fellowship Directors: A national survey of pediatric fellowship program directors. Medical Education: Assessment: Pediatric Academic Societies, Baltimore, MD.
2018
Park YS, Hicks PJ, Carraccio C, Margolis M, Schwartz A, for the PMAC Module 2 Study Group. (2018, November). Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth. RIME oral presentation. Annual Meeting of the Association of American Medical Colleges, Austin, TX.
Minimum Level of Supervision for Fellows at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. (May 2018). APA Presidential Plenary, Pediatric Academic Societies, Toronto, Canada.
Is the Expected Minimum Level of Supervision for Graduating Pediatric Fellows for the Common Entrustable Professional Activities Similar Across the Subspecialties? (May 2018). Medical Education 1: Top Oral Abstracts. Pediatric Academic Societies, Toronto, Canada.
Batra M, Mahan J, Schubert C, Wilson P, Staples B, Serwint JR, McClafferty H, Schwartz A, Kemper K.(2018, March). Burnout in Pediatric Residents: Findings From a National Longitudinal Survey. Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Lauden S, Wilson P, Fuast M, Mahan J, Batra M, Shubert C.(2018, March). Burnout & Who's Who in Pediatric Global Health: A Look at National Trends. Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
2017
Abramson E, Stevenson M, Naifeh M, Hammad H, Gerber L, LI ST. (2017, May). Scholarly Activity Training During Residency: A National Survey of Pediatric Residents and Program Directors. Pediatric Academic Societies, San Francisco, CA.
Gifford KA, Burke A, Li ST, Tancredi DJ, Trimm F, Schwartz A, Mahan J, Guralnick S, Guillot AP. (2017, May). Do Milestones Reported to the ACGME Better Describe Residents' Performance or their Programs' Assessment Practices? Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Bartlett K, Michelson C, Elliott S, Gibbs K, McGreevy J, Li ST, Sklansky DJ, Gonzalez del Rey J, Poynter S, Sharma T, Burman N, Gellin C, Thoreson L, Calaman S, Garfunkel L, Frohna J, Scott-Vernaglia S, Barnes M, Schwartz A. (2017, May). Association Between Milestone Assignments and Recommended Supervision Categorizations Among Clinical Competency Committee Members. Pediatric Academic Societies, San Francisco, CA.
Batra M, Mahan JD, Schubert CJ, Wilson PM, Staples BB, Serwint JR, McClafferty H, Schwartz A, Kemper KJ, for the Pediatric Residency Burnout and Resilience Consortium. (2017, May). Burnout in Pediatric Residents: A National Survey To Inform Future Interventions. Pediatric Academic Societies, San Francisco, CA.
Staples BB, Batra M, Mahan J, Schwartz A, Wilson P, McClafferty H, Schubert C, Kemper K, Serwint JR. (2017, May). Burnout Status and Milestone Performance in Pediatric Residents. Pediatric Academic Societies, San Francisco, CA.
Batra M, Mahan J, Shubert C, Wilson P, Staples BB, Serwint JR, McClafferty H, Schwartz A, Kemper K. (2017, April). Burnout in Pediatric Residents: A National Survey to Inform Future Interventions. Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA. Winner: 2017 Research Award.
2016
Schwartz A. (2016, December). Milestones-based research at APPD LEARN. ACGME Milestones Summit, Chicago, IL.
Mahan JD, Serwint JR, Schubert CJ, Wilson PM, Staples BB, McClafferty, Schwartz A, Kemper KJ, Batra M. (2016, November) Building a national consortium (The Pediatric Resident Burnout-Resilience Study Consortium) to understand and promote pediatric wellness. The ACGME Second Symposium on Physician Well-Being: Commitment to Change. Chicago, IL.
Schwartz, A. (2016, August). APPD LEARN: Building a Pediatric Education Research Network. AMSPDC Webinar
Kesselheim JC, Boyer D. (2016, April). Balancing education and service in Pediatric residency training: Results of a national survey. Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA. Winner: 2016 Research Award
Li S, Burke AE, Guillot A, Schwartz A, Gifford KA, Trimm F, Mahan JD, Guralnick S, Tancredi DJ. (2016, April). Pediatric Resident Minimum Milestone Expectations by Level of Training. Annual Spring Meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Li ST, Tancredi DJ, Schwartz A, Burke AE, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K. (2016, February). Pediatric Resident Minimum Milestone Expectations by Level of Training. Platform presentation at Pediatric Academic Societies (PAS) Annual National Conference, Baltimore, MD.
The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. (2016, February). Annual education meeting of the Accreditation Council for Graduate Medical Education, National Harbor, MD.
Creation and validation of entrustment scales for the common pediatric subspecialty entrustable professional activities (EPAs). (2016). Acad Pediatr 16:e1–e2.
2015
Andriole D, Jeffe D, Reilly K, Schwartz A, Bostrom D. (2015, November). Bridging Data Across Divides: Challenges of Aggregating Data from Multiple Organizations. Learn, Serve, Lead: 2015 AAMC, Baltimore, MD.
Li S, Tancredi D, Schwartz A, Burke A, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K, for APPD LEARN validity of resident self-assessment using pediatric milestones group. (2015, April). Which Sub-Competencies Best Identify Overall Unsatisfactory/ Marginal Resident Performance? Pediatric Academic Societies, San Diego, CA.
Li S, Tancredi D, Schwartz A, Burke A, Guillot A, Guralnick S, Mahan JD, Trimm RF, Gifford K, for APPD LEARN validity of resident self-assessment using pediatric milestones group. (2015, April) How Well Do Resident Milestone Self-Assessments Correlate with Clinical Competency Committee Milestone Assessments? Pediatric Academic Societies, San Diego, CA.
Gifford K, Li S, Trimm RF, Schwartz A, Gurlanick S, Tancredi DJ, Burke A, Mahan J, for For APPD LEARN validity of resident self-assessment group. (2015, April). How Do Clinical Competency Committees Function in Pediatric Residency? Pediatric Academic Societies, San Diego, CA.
Li S, Tancredi DJ, Trimm RF, Guillot A, Mahan JD, Burke A, Gifford KA, Schwartz A, Guralnick S, for APPD LEARN validity of resident self-assessment. (2015, March). How well do resident milestone self-assessments correlate with clinical competency committee milestone assessments? APPD Annual Spring Meeting, Orlando, FL. Winner, APPD Research Award
2013
Tenney-Soeiro R, Ronan JC, Weiss AK, Young D, Clyman SG, Schwartz A, Rose KM, Hicks PJ. (2013, April). Does the Quantity and Quality of Comments Provided on Intern and Sub-Intern Pediatrics Milestones Assessment Instrument Vary Based on Role of the Rater? Platform Presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Turner T, Hicks P, Rose K, Clyman S, Schwartz A, Poynter S, Haftel H, Ronan JC, Waggoner-Fountain L, Caputa G. (2013, April). Sub-Intern and Intern Performance Level on Pediatric Milestones. Platform Presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Chaffinch C, Poynter S, Turner T, Hicks P, for the APPD LEARN-NBME Milestones Assessment Group. (2013, February). Learner Feedback on the Pediatrics Milestones Assessment Project. Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
Poynter S, Turner T, Chaffinch C, Hicks P, for the APPD LEARN-NBME Milestones Assessment Group. (2013, Februray). Learner Feedback on the Pediatrics Milestones Assessment Project. 2013 ACGME Annual Educational Conference, Orlando, FL.
2012
Schwartz A. (2012, November). Assessment of Pediatrics Milestones and Development of a Collaborative Research Network for GME. Invited presentation, DeWitt Baldwin Seminar Series, ACGME, Chicago, IL.
POSTERS
2024
Schumacher DJ, Kinnear B, Michelson C, Winn A, Turner DA, Martini A, Schwartz A for the APPD LEARN Crosswalk Study Group. (2024, April). More Graduating Residents from Small Programs are Ready for Unsupervised Practice for the 17 ABP General Pediatrics EPAs When Compared to Medium and Large Programs. Poster Presentation, APPD 2024 Annual Spring Meeting, Chicago, IL.
Schumacher DJ, Kinnear B, Michelson C, Winn A, Turner DA, Martini A, Schwartz A for the APPD LEARN Crosswalk Study Group. (2023, September). How Many EPAs Should Be Assessed to Make a Reliable Overall Entrustment Decision About Pediatric Residents? Poster Presentation, AMA ChangeMedEd 2023, Chicago, IL.
Moffatt M, Schwartz A, Czaja A, Fussell J, Kamin D, Kesselheim J, Langhan M, Robinson B, Sauer C, Mink R. (2024, April). Equations to Predict Milestone Levels from Entrustable Professional Activities Level of Supervision Ratings for the Subspecialties. Poster Presentation, Association of Pediatric Program Directors, Chicago, IL.
Weiss P, Schwartz A, Boyer D, Chess P, Fussell J, McGann K, Moffat M, Mink R. (2024, May). Many Pediatric Fellows Do Not Meet the Minimum Level of Supervision at Graduation but Still Pass Their Subspecialty Certifying Exam. Poster Presentation, Pediatric Academic Societies, Toronto, Canada.
George R, Schwartz A, Chess P, Hsu D, Kamin D, Moffat M, Curran M, Mink R. (2024, May). Factors Associated with Early Identification of Pediatric Fellows Needing Remediation. Poster Presentation, Pediatric Academic Societies, Toronto, Canada.
Pitts S, Schwartz A, Karnik R, High P, Stafford D, Aye T, Langhan M, Boyer D, Mink R. (2024, May). Many Pediatric Subspecialty Fellows Do Not Achieve the Expected Minimum Level of Supervision for Graduation After Two Years of Training. Poster Presentation, Pediatric Academic Societies, Toronto, Canada.
2023
Mink R, Weiss P, Boyer D, Chess P, Fussell J, McGann K, Moffatt M, Schwartz A for APPD SPIN. (2023, March). Not All Pediatric Fellows are Meeting the Accepted Minimum Level of Supervision at Graduation. Poster Presentation, APPD 2023 Annual Spring Meeting, Atlanta, GA.
Kamin D, Schwartz A, Stafford D, Pitts S, Lopez M, Dammann C, Aye T, Mink R for APPD SPIN. (2023, March). Correlation Between Pediatric Fellow and Clinical Competency Committee Entrustment Ratings. Poster Presentation, APPD 2023 Annual Spring Meeting, Atlanta, GA.
Mink R, Rama J, Mehta J, James S, Dammann C, Schwartz A for APPD SPIN. (2023) Effect of the Covid-19 Pandemic on the Required Level of Supervision For Pediatric Fellows. Poster Presentation, Pediatric Academic Societies, Washington, DC.
Czaja A, Schwartz A, Mehta J, James S, Kesselheim J, Sauer C, Mink R for APPD SPIN. (2023) Validity Evidence for Level of Supervision Scales for Pediatric Subspecialty-Specific Entrustable Professional Activities. Pediatric Academic Societies, Washington, DC.
2020
Mink R, Carraccio C, Herman B, High P, Hsu D, Langhan M, Schwartz A. Factors Important in Determining Simple versus Complex Cases for the Pediatric Subspecialty Entrustable Professional Activity (EPA) Levels of Supervision, Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/ 2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.356.)
Mink R, Carraccio C, Herman B, Chess P, Dammann C, Hsu D, Kesselheim J, Moffat M, Myers A, Turner D, Czaja A, Schwartz A. Have Level of Supervision Ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) Changed Over Time? Pediatric Academic Societies, 2020. (Accepted as poster presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202823.369)
Mink R, Carraccio C, Herman B, Weiss P, Turner D, Pitts S, Moffat M, Kesselheim J, Czaja A, Schwartz A. Relationship Between Level of Supervision at the End of Residency Compared with Early Fellowship for the Entrustable Professional Activities (EPAs): A Pilot Study. Pediatric Academic Societies, 2020. (Accepted as platform presentation but due to COVID-19 related conference cancellation, not presented. Disseminated on 4/30/2020 on the PAS 2020 Meeting Program Guide, EPAS20202455.2.)
2019
Mink R, Carraccio C, Herman B, Hsu D, Chess P, High P, Langhan M, McGann K, Curran M, Stafford D, Czaja A, Dammann C, Mahan J, Aye T, Schwartz A. (2019, May). Are Graduating Pediatric Fellows Meeting the Expected Level of Supervision for the Common Subspecialty Entrustable Professional Activities (EPAs)? Pediatric Academic Societies, Baltimore, MD.
Mink R, Carraccio C, Herman B, Myers A, Fussell J, Turner D, Pitts S, Kesselheim J, Baffa J, Weiss P, Sauer C, Srivastava S, Moffatt M, Halbach S, Schwartz A. (2019, March). Do Pediatric Fellows Meet the Expected Level of Supervision at Graduation? Association of Pediatric Program Directors, New Orleans, LA.
Reed S, Mink R, Li ST. (2019, March). Utility of Residency Milestones Reported to Fellowship Directors: A national survey of pediatric fellowship program directors. Association of Pediatric Program Directors, New Orleans, LA.
2018
Smith S, Mensah-Bonsu N, Lopez M, Voigt R, Mink R, Carraccio C, Schwartz A, Herman B, Fussell J. (2018, September). Are Graduating Developmental-Behavioral Pediatrics Fellows Ready for Clinical Practice? A SPIN study. Poster Presentation at Society for Developmental and Behavioral Pediatrics 2018 Annual Meeting, Anaheim, CA.
Lopez M, Mensah-Bonsu N, Smith S, Voigt R, Mink R, Carraccio C, Schwartz A, Herman B, Fussell J. (2018, September). Using Entrustable Professional Activities to Assess Graduation Readiness in Developmental Behavioral Pediatrics: A SPIN Study. Poster Presentation at Society for Developmental and Behavioral Pediatrics 2018 Annual Meeting, Anaheim, CA.
Hu K, Hicks PJ, Margolis MJ, Carraccio CL, Osta AD, Schwartz A. (2018, August). Inter-program variance in ACGME Reportable Pediatrics Milestones. Can Residents' ACGME PM levels be compared among programs? Poster presentation at the 2018 Annual Meeting of the Association for Medical Education in Europe (AMEE 2018), Basel, Switzerland.
Mink R, Carraccio C, Herman B, Aye T, Chess P, Curran M, Dammann C, High P, Hsu D, Mahan J, McGann K, Srivastava S, Turner D, Weiss P, Schwartz A for the Subspecialty Pediatrics Investigator Network. (2018, May). Is the Expected Minimum Level of Supervision for Graduating Pediatric Fellows for the Common Entrustable Professional Activities Similar Across the Subspecialties? Pediatric Academic Societies, Toronto, Canada.
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carraccio C for the Subspecialty Pediatrics Investigator Network. (2018, May). Setting the Minimum Level of Supervision for Fellows at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. Pediatric Academic Societies, Toronto, Canada.
Mink R, Schwartz A, Herman B, Turner D, Stafford D, Sauer C, Myers A, Kesselheim J, Hsu D, High P, Chess P, Baffa J, Carracio C. (2018, March). Expected Fellow Minimum Level of Supervision at Graduation for the Common Pediatric Subspecialty Entrustable Professional Activities. Poser Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Mink R, Carracio C, Herman B, Baffa J, Fussell J, Kesselheim J, McGann K, Myers A, Sauer C, Stafford D, Weiss P, Schwartz A. (2018, March). Pediatric Fellows are Not Expected to Achieve "Entrustment" at Graduation from Fellowship. Poster Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
Abramson E, Stevenson M, Naifeh M, Mauer E, Gerber L, Li ST. (2018, March). Scholarly Activity Training During Residency: How Well Aligned are Program Directors and Residents? Poster Presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Atlanta, GA.
2017
Hicks P, Margolis M, Carraccio C, Clauser B, Winward M, Schwartz A, and the PMAC Module 1 Study Group. (2017, May). Pediatrics Milestones Assessment Collaborative: Development and implementation of an authentic workplacebased assessment system (WBAS). Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Barnes M, Bartlett K, Burman N, Calaman S, Elliott S, Frohna J, Garfunkel L, Gellin C, Gibbs K, Gonzalez del Rey J, Michelson C, McGreevy J, Li ST, Poynter S, Scott-Vernaglia S, Sharma T, Sklansky DJ, Thoreson L, Schwartz A. (2017, May). Concordance of Clinical Competency Committee and Program Director Recommended Supervision Categorization. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Schumacher DJ, King B, Thoreson L, Sklansky DJ, Sharma T, Scott-Vernaglia S, Poynter S, Li ST, McGreevy J, Michelson C, Gonzalez del Rey J, Gibbs K, Gellin C, Garfunkel L, Frohna J, Elliott S, Calaman S, Burman N, Bartlett K, Barnes M, Schwartz A. (2017, May). Key Factors for Recommending a Resident May Serve as a Supervisor: A National Study of Clinical Competency Committee Members. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Reed S, Batra M, Kemper K, Staples BB, Serwint JR, McClafferty H, Schubert C, Wilson P, Schwartz A, Mahan J. (2017, May). How Variable are Burnout and Stress in Pediatric Residents? An Exploratory Single Center Study from the Pediatric Resident Burnout-Resilience Study Consortium. Poster Presentation at Pediatric Academic Societies, San Francisco, CA.
Staples BB, Batra M, Mahan J, Schwartz A, Wilson P, McClafferty H, Schubert C, Kempter K, Serwint JR. (2017, April). Burnout Status and Milestone Performance in Pediatric Residents. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Schumacher DJ, King B, Barnes M, Bartlett K, Burman N, Calaman S, Elliott S, Frohna J, Garfunkel L, Gellin C, Gibbs K, Gonzalez del Rey J, Michelson C, McGreevy J, Li ST, Poynter S, Scott-Vernaglia S, Sharma T, Sklansky DJ, Thoreson L, Schwartz A. (2017, April). Concordance of Clinical Competency Committee and Program Director Recommended Supervision Categorization. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Gifford KA, Li ST, Schwartz A, Burke AE, Tancredi DJ, Guralnick S, Guillot AP, Trimm F, Mahan JD. (2017, April). Do Milestones Reported to the ACGME Describe Residents Performance or Their Programs Assessment Practices? Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Schumacher DJ, King B, Thoreson L, Sklansky DJ, Sharma T, Scott-Vernaglia, Poynter S, Li ST, McGreevy J, Michelson C, Gonzalez del Rey J, Gibbs K, Gellin C, Garfunkel L, Frohna J, Elliott S, Calaman S, Burman N, Bartlett K, Barnes M, Schwartz A. (2017, April). Key Factors for Recommending a Resident May Serve as a Supervisor: A National Study of Clinical Competency Committee Members. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
Abramson E, Stevenson M, Naifeh M, Hammad H, Gerber L, Li ST. (2017, April). Scholarly Activity Training During Residency: A National Survey of Pediatric Residents and Program Directors. Poster presentation at Annual Spring Meeting of the Association of Pediatric Program Directors, Anaheim, CA.
2016
Mink R, Carraccio C, Schwartz A, Dammann C, High P, Kesselheim J, McGann K, Herman B. (2016, May). Establishing a Medical Education Research Network for the Pediatric Subspecialties. Pediatric Academic Societies, Baltimore, MD.
Mink R, Carraccio C, Herman B, Dammann C, Mahan J, Pitts S, Sauer C, Schwartz A. (2016, May). Variability in Fellow Entrustment Across the Pediatric Subspecialties for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs). Pediatric Academic Societies, Baltimore, MD.
Gifford KA, Burke AE, Trimm F, Mahan JD, Schwartz A, Li ST, Guillot AP, Guralnick S. (2016, April). Learner self-assessment as a contribution to competency assessment. Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Schwartz A, Dammann CE, High PC, McGann KA, Herman BE for SPIN. (2016, April). Creation of a pediatric subspecialty educational research network. Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Aye T, Baffa JM, Chess PR, Fussell JJ, Sauer CG, Stafford DE, Weiss P, Schwartz A for SPIN. (2016, April). Do fellowship program directors (FPD) and clinical competency committees (CCC) agree in fellow entrustment decisions? Poster presentation at the annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DE, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Schwartz A for SPIN. (2016, April). The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. Annual spring meeting of the Association of Pediatric Program Directors, New Orleans, LA.
Mink RB, Carraccio CL, Herman BE, Weiss P, Turner DA, Stafford DE, Hsu DC, High PC, Fussell JJ, Curran ML, Chess PR, Schwartz A for SPIN. (2016, February). The link between milestone levels and fellow entrustment for the common pediatric subspecialty entrustable professional activities. Annual Education Meeting of the Accreditation Council for Graduate Medical Education, National Harbor, MD.
Mink RB, Carraccio CL, Herman BE, Turner DA, Myers A, Kesselheim JC, Hsu DC, Curran ML, Schwartz A, for the Subspecialty Pediatrics Investigators Network (SPIN). (2016, February). Reliability and validity of a supervision scale for the common pediatric subspecialty entrustable professional activities. Poster presentation. ACGME Annual Meeting, National Harbor, MD.
Balmer DF, Hicks PJ, Carraccio C, Schwartz A. (2016 February). Bridging the Gap between Educational Research and Practice: The Normalization Process Theory Poster presentation. Pediatric Academic Societies, Baltimore, MD.
2014
Schwartz A., Young, R., Hicks, P.J. (2014, November). Medical Education Research Networks: The case of APPD LEARN. Poster presentation at the 2014 American Association of Medical Colleges Medical Education Meeting, Chicago, IL.
2013
Chaffinch CN, Poynter SE, Turner T, Schwartz A. (2013, April). Learner Feedback on the Pediatrics Milestones Assessment Project. Poster presentation at the Association of Pediatric Program Directors/Council on Medical Student Education in Pediatrics 2013 Annual Spring Meeting, Nashville, TN.
2012
McGregor R, Burke A, Carraccio C, Haftel H, Hicks P, Schwartz A. (2012, November). APPD LEARN: An Educational Research Network in Pediatric GME Digitized Poster Presentation at Group on Educational Affairs, AAMC Annual Meeting, San Francisco, CA.
Join APPD LEARN
All APPD member programs are eligible to be members of APPD LEARN. Membership may be acquired at any time by contacting the APPD LEARN Project Manager and providing the required information.
APPD LEARN members receive the following benefits:
- Opportunities to participate in research studies advancing Pediatrics Graduate Medical Education (GME)
- Eligibility to propose new research studies to be conducted by APPD LEARN, including assistance with:
- Designing studies and preparing proposals
- IRB processes
- Project Management
- Access to CME-designated APPD LEARN training opportunities
- Additional evidence in support of ACGME program requirement II.B.5 (The faculty must establish and maintain an environment of inquiry and scholarship with an active research component)
While an APPD LEARN member program is participating in a research study, it becomes an active member. APPD LEARN active members receive:
- Early access to new educational methods and tools developed through APPD LEARN
- Opportunities for group authorship or acknowledgment in study publications
- Public and professional recognition for participation in educational research
- Access to consultation by educational research specialists
APPD LEARN members have the following responsibilities:
- The program director must indicate knowledge and approval of the program’s membership in APPD LEARN.
- A liaison within the program (who may also be the program director) must be identified.
- The liaison must respond to and complete the annual APPD LEARN member survey and needs assessment.
- The program must be willing to participate in at least one research study every two years.
- A site investigator must be identified for each study in which the program is participating. The site investigator (who may also be the liaison or program director) is responsible for maintaining contact with the APPD LEARN staff and the study PI, and overseeing the implementation of the study protocol at the program.
If you are an active APPD affiliated residency program interested in becoming a part of APPD LEARN, please email learn@appd.org with the following information:
1. Institution Name
2. Program Director Name
3. Email Address
4. Phone Number
Propose a Study
APPD LEARN member programs are eligible to propose studies to be conducted on the APPD LEARN network. See the links below for general proposal instructions.
Open calls for proposals
CFP 2022-1 (member-initiated proposals)
Proposals are reviewed three times per year, with submission due dates of January 31, May 31, and September 30 each year.
Sharing data through APPD LEARN
Projects involving any number of collaborating programs that are collecting resident-level data outside of APPD LEARN and that wish to archive and share their data through APPD LEARN should contact the APPD LEARN Director as early as possible in the project to discuss how data can be archived by APPD LEARN and APPD LEARN support for data de-identification and documentation.
Any APPD LEARN member program is eligible to submit a research proposal. APPD member programs that are not already APPD LEARN member programs must join APPD LEARN before or at the time of submission of a research proposal. Proposals may include collaborators who are not members of APPD LEARN, but an APPD LEARN member program must be the lead site for the proposal.
Proposals from APPD LEARN member programs may designate any affiliated faculty member(s) of any rank as project investigators, including principal investigator. However, all proposals must be approved by the Program Director and Department Head of the member program, and should be submitted by the program’s APPD LEARN liaison.
All studies that collect data through APPD LEARN are required to deposit their approved protocols and data into the APPD LEARN data repository in order to make them available to future investigators. Publications arising from APPD LEARN studies must acknowledge APPD LEARN and their manuscripts must be provided to APPD LEARN for archiving alongside the study data.
Please submit your proposal using the APPD Research Submission portal: https://apps.appd.org/survey_submission/.
To download a word document of the proposal submission fields, click here.
Proposals are reviewed three times per year by the APPD LEARN Proposal Review Committee (PRC) and APPD LEARN Director. Review cycles will group proposals submitted February 1-May 31, June 1-September 30, and October 1-January 31.
Criteria applied by the APPD LEARN PRC will include:
- Significance of the research question (including importance for Pediatric GME or medical education)
- Quality of the research plan (including study design, instrumentation, data analysis)
- Feasibility for APPD LEARN
Each proposal will be assigned a primary reviewer and at least two secondary reviewers. Reviewers will write narratives about each criterion and will assign each proposal an overall rating of excellent, very good, good, fair, and poor.
The Committee may make comments or suggest modifications to proposed studies to enhance any of these components. Committee recommendations are advisory. Final decisions about proposal approval or rejection will be made by the APPD LEARN Director.
Each study using the LEARN network will have an ad hoc project oversight committee, composed of the project principal investigator, APPD LEARN Director, LEARN Project Manager, and other members selected by the principal investigator and APPD LEARN Director. These committees will hold regular (biweekly or monthly) conference calls during the period of the study.
After formation of the oversight committee, the APPD LEARN Director and APPD LEARN Project Manager will assist in sampling and enrollment of APPD LEARN member sites into the study protocol and will work with participating sites to obtain IRB approval prior to data collection.
The Study Oversight Group is responsible for adopted processes for communicating with study memnbers and for scientific communications (e.g. manuscripts, presentations) that result from the study, consistent with the following APPD LEARN guidelines:
Request Data
Permitted users
- APPD LEARN members are permitted to obtain data from the repository at no charge, subject only to scientific and IRB approval discussed below.
- APPD members who are not members of LEARN may request access to repository data. These requests will be considered by the APPD LEARN Director and APPD LEARN Advisory Board, who will be responsible for determining any charges associated with providing repository data and any additional terms of use
- Investigators not associated with APPD may request access to repository data. These requests will be considered by the APPD LEARN Director and APPD LEARN Advisory Board, who will be responsible for determining any charges associated with providing repository data and any additional terms of use.
Scientific and budgetary approval
All data requests must include a summary of the data requested, research question to be answered by the data, and planned analyses. Data requests for projects that will involve additional data collection should explain whether and how newly collected data will be provided back to APPD LEARN; proposals that do not plan to share their data will be disadvantaged in review. APPD LEARN will provide a form for these data requests.
Requests received will be reviewed by the APPD LEARN Director for scientific merit and cost to APPD LEARN of providing data, and must be approved before data will be released. Requests that are denied may be appealed to the APPD LEARN Advisory Board, whose recommendations are binding and final.
IRB oversight
Because data maintained in APPD LEARN are not identifiable, use of APPD LEARN data by investigators who did not collect the data or interact with the subjects is likely to constitute research on “persons” rather than “human subjects”. In many institutions, however, investigators are required to obtain from their IRBs a determination that their proposed use of data does not constitute human subjects research. Investigators are responsible for satisfying their local requirements prior to obtaining and analyzing data.
To make a request for data, please complete and sign APPD LEARN's Data Request Form, and submit the form by emailing LEARN@appd.org. Procedures for obtaining data will be determined by the APPD LEARN Director for each request.
When several investigators are seeking to obtain data, priority will be given based on APPD LEARN membership status, scientific merit, and resource availability.
By downloading these Materials, I agree to the following:
- I will use the Materials only for academic or scientific purposes.
- I will not use the Materials to
- obtain information that could directly or indirectly identify subjects.
- produce links among the Distributor's datasets or among the Distributor's data and other datasets that could identify individuals or organizations.
- obtain information about, or further contact with, subjects known to me except where the use and/or release of such identifying information has no potential for constituting an unwarranted invasion of privacy and/or breach of confidentiality.
- I agree not to download any Materials where prohibited by applicable law.
- I agree not to use the Materials in any way prohibited by applicable law.
- I understand that although data sets may include documentation or codebooks that describe the instruments used to collect the data, these instruments are separately copyrighted, and access to the data set does not provide any license or rights to administer the instruments or reverse-engineer the instruments.
- I agree that any books, articles, conference papers, theses, dissertations, reports, or other publications that I create which employ data will reference the bibliographic citation accompanying this data and acknowledge the assistance of APPD LEARN in obtaining the data. These citations include the data authors, data identifier, and other information in accordance with the Recommended Standard (http://thedata.org/citation/standard) for social science data. I further agree to inform APPD LEARN about any such publications or presentations, and, subject to any copyright assignment or embargo agreements required by the publisher, to provide copies of publications or accepted-for-publication manuscripts to be archived alongside the data.
- The distributor makes no warranties, express or implied, by operation of Law or otherwise, regarding or relating to the dataset. I agree to indemnify and hold harmless APPD, APPD LEARN, and its officers, volunteers, agents, and employees for any claims arising out of my research using data provided through APPD LEARN.
Tools
The APPD LEARN Dataverse is a repository for all APPD LEARN study materials, including protocols and proposals, deidentified study data, and links to manuscripts reporting study results.
UIC IRB exemption for the APPD LEARN repository
Dr. Schwartz's HSPP/CITI education status for human subjects training, in case your IRB asks for evidence of CITI training.
The APPD LEARN de-identification system generates learner codes for use in reporting de-identified data to APPD LEARN in APPD LEARN studies. It is hosted at https://learn.cybermango.org/cgi-bin/xls-code.cgi. (This link only provides APPD LEARN ids - do not use it if your study has a study-specific link that also generates tailored survey urls.)
Details about this process and protecting learner privacy are available here
A description of APPD LEARN's survey platform and privacy protections is available here.
If your institution requires a data use agreement with APPD in order to provide data to APPD LEARN, this model agreement contains APPD LEARN's standard language and could be used.
APPD LEARN's authorship principles can be found here.
Individuals who submit data to a LEARN project (through either PRIME or SPIN) but who otherwise do not meet authorship criteria are considered collaborators in publications. The names of these individuals are PubMed searchable. We strongly support recognizing the academic contributions of collaborators and including information about participation in the collaborator's CV. One way to do this is to create a section in the CV entitled "Collaboration in National Studies" (or something similar) and listing the abstracts and publications under this heading. This will differentiate author vs. collaborator roles and avoid confusion.
Data Analysis
Best practices for working on data analyses with Alan
- Providing empty tables and mock-ups of desired figures is very useful to show what you'd like to see.
- Alan is supporting multiple projects at the same time, and often multiple papers/presentations within each project. This makes it challenging for him to quickly identify particular analyses he's run. You can make this much easier by:
- not renaming any analysis output files that he sends you (he can use the filename to quickly find your project/paper/etc on his computer), and
- attaching a copy of that analysis output file whenever asking questions about an analysis
