About the PME Educational Scholarship Review Committee
The committee has two primary functions:
Scientific review of studies - Ensuring that proposals meet acceptable standards of scientific rigor and merit, aligning with the HMS mission, values and priorities.
- Management and oversight of research burden on students – Reviewing the length, volume and timing of surveys and intervention studies to ensure that research and quality improvement efforts do not unduly impinge on students’ academic time.
The PME Educational Scholarship Review Committee is also responsible for discerning whether a study is considered Quality Improvement or Research.
Proposals will be reviewed for scientific merit, alignment with HMS curricular priorities and the level of research burden on students. Because we receive many requests to survey students, proposals that include survey instruments to be completed by students may not be approved. Requests to conduct surveys have increased in recent years, we encourage faculty who wish to conduct surveys to ensure that the survey instrument is of high quality, has been validated or sufficiently pilot-tested and is as brief as possible (preferably 15 minutes or less). Consultation is available for faculty who wish to have their survey reviewed prior to submitting a proposal.
PME Educational Scholarship Review Committee Members:
Dean for Medical Education — Edward M. Hundert, MD
Director of Educational Scholarship & Post-PCE Subcommittee Chair — Richard M. Schwartzstein, MD
Associate Dean for Medical Education Quality Improvement & Assessment Subcommittee Chair — John L. Dalrymrple, MD
Dean for Students — Fidencio Saldana, MD
Associate Dean for Medical Education Planning and Administration — Jane Neill
Director for Faculty Development, MD Program — Barbara Cockrill, MD
Director of the Office of Educational Quality Improvement — Carolyn Wood
Director of Curriculum Services — Michele Cohn
Registrar — Terese Galuszka
Evaluation and Assessment Senior Project Manager — Stephen Pelletier, PhD
PCE Subcommittee Chair — Sara Fazio, MD
Preclerkship Subcommittee Chair — Bernard Chang, MD
HST Subcomittee Chair — Richard Mitchell, MD, PhD
Questions related to the approval process can be directed to educationalscholarship_PME@hms.harvard.edu
Workshop on Research Methods
The Office of Educational Scholarship offers a workshop called Getting Started with Medical Education Research multiple times of year. To be notified of when the next workshop will be offered, email educationalscholarship_PME@hms.harvard.edu to get on the mailing list.
Do you have a research idea but don't know where to start? This hands-on, 90-minute workshop will help you to launch your work with confidence. Is your research idea feasible, interesting and publishable as an education research project? The workshop with be targeted for faculty who are new or relatively new to research, with a focus on evaluation of course or clerkship innovations. We will address the following questions, each customized to your specific project interests: What are the characteristics of an effective research question? How will this question, or set of questions, guide the design of your research project? Is there a way to create a randomized design, and if not, what is the best comparison group for your study? Do you use quantitative, qualitative, or mixed methods of research? What outcomes will best support valid inferences from your results? Enrollment is limited to 25 participants to maximize opportunities for individualized coaching and feedback. Participants will be asked to read one journal article and submit a one-paragraph description of their research idea prior to attending the workshop. Workshop sessions are led by the Associate Director for Education Research
By the end of this session, participants will be able to:
- Craft a feasible and answerable research question
- Choose an appropriate research design
- Identify a suitable comparison group
- Select meaningful outcomes
Before You Apply
HMS priorities for research and quality improvement studies involving students include:
- Studies related to the HMS curriculum (courses and clerkship) that are aimed at ongoing quality improvement
- Research provides evidence for specific pedagogic approaches that are not already demonstrated to be effective in existing literature
- Research that will contribute to theoretical or conceptual models of teaching and learning
Criteria and Policies
Policy for Course and Clerkship Directors on use of data from HMS students enrolled in their own course to inform QI studies
Course, Clerkship and Site Directors (Course Directors), in the context of their course/clerkship/theme leadership role, will naturally gather and review performance data, satisfaction data and general feedback from HMS students enrolled in their course to inform ongoing course improvements and enhancements. Data gathered within the context of a single course or clerkship does not require PME review/approval. Note that studies within a single clerkship that comprise data from multiple Harvard-affiliated hospital sites hosting students in that clerkship, will be defined as a study within a single course/clerkship.
Course Directors may directly survey their enrolled students using Harvard-supported secure platforms such as Canvas or Qualtrics. Such surveys must be conducted on top (vs instead of) all required PME surveys (course evaluation, faculty evaluation, PCE checklist, GQ, etc.), and Course Directors should be mindful not to unduly burden students with excess survey demand. In accordance with HMS policy on access to educational records and FERPA policy, Course Directors must safeguard student privacy by limiting data access to those with a business-need-to-know and de-identifying or aggregating any data shared more broadly (e.g. in the context of educational scholarship). Course Directors are free to publish QI research using data from their own course(s)/clerkship(s) assuming they do not compromise student privacy, but it is advisable to request an ex-post PME Educational Scholarship QI exemption to submit to journals in lieu of formal IRB approval.
If you are not sure whether your study may qualify as Educational Quality Improvement for the MD Program (which should be approved by the PME Office for Educational Scholarship) vs Human Subjects Research (which should be approved by the IRB), the IRB has developed a decision tool to assist investigators in determining if their activities are research, if their research involves human subjects, and/or if they need to submit an IRB application. The IRB Decision Tool should only be used by investigators who will act on behalf of a Harvard Longwood Campus School (HSPH, HMS, or HSDM) while conducting research activities. HarvardKey log in is required to use the tool.
Policy for Gathering and Using HMS Student/Faculty Data to inform Medical Education Research and QI outside the context of an individual Course or Clerkship
All research and QI studies involving HMS student/faculty data outside the context of an individual course or clerkship (including cross-course studies) must be approved by the PME Education Scholarship Review Committee Applications can be submitted on the PME Research QI Protocol Form.
Issuing Research Surveys to HMS Students/Faculty
For studies that involve surveys of HMS students/faculty, relevant materials (survey instruments, interview guides, consent forms and recruitment materials) must be approved by the PME Educational Scholarship Review Committee. Approved surveys and associated reminders will be issued by the Office of Educational Quality Improvement (OEQI) using Qualtrics survey software. OEQI staff will serve as honest brokers to de-identify the data before sharing it with investigators. If incentives such as gift cards are to be provided to respondents, PI’s should follow the Harvard guidelines on incentives. Requests to field surveys and de-identify data to support educational scholarship will be triaged, queued and accommodated subject to availability of staff time. Analysis of raw data is the responsibility of the PI and the study team.
Requesting HMS Education Data
HMS gathers and monitors a broad range of data about student performance, longitudinal trajectory and course/faculty evaluation to inform continuous quality improvement (CQI). These data are primarily intended to inform curricular governance and CQI within the MD program. However, on occasion, medical educators who hold HMS faculty appointments and who are directly involved in teaching and assessing HMS medical students may wish to pursue and publish educational QI studies that cut across courses or phases of the curriculum. In such cases, principal investigators must submit a research proposal via the PME Educational Scholarship Review process. If the study meets the required standards for scientific merit and alignment with HMS curricular priorities, and if it qualifies for an IRB QI exemption and/or IRB approval, PME staff will serve as honest brokers, pulling and de-identifying data for analysis by the principal investigator and the research team. Requests to field surveys and de-identify data to support educational scholarship will be triaged, queued and accommodated subject to availability of staff time. Analysis of raw data is the responsibility of the PI and the study team.
HMS affiliates who wish to participate in multi-institutional studies involving institutions outside Harvard and its affiliated hospitals using MD student/faculty data must seek approval from the PME Educational Scholarship Review Committee.IRB approval from the PI’s institution should be shared with the Longwood IRB.
Given the many demands on students’ time, only studies that are well designed and address HMS priorities can be considered for approval.
Is a Study Quality Improvement (QI) or Research?
Determining whether a study is Quality Improvement (QI) or Research is important because QI studies do not require IRB review and Research studies do. This determination is formally made through the PME Education Scholarship Study Review. It cannot be made by the study proposal writer.
The Harvard Longwood Campus (HLC) Institutional Review Board (IRB) has put together an IRB Decision Tool to assist investigators in determining if their activities are research, if their research involves human subjects, and/or if they need to submit an IRB application. This decision tool should only be used by investigators who will act on behalf of a HLC School (HSPH, HMS, or HSDM) while conducting research activities. HarvardKey log in is required to use the tool. The IRB Decision Tool can be found here: DECISION TOOL
In most cases, the PME Review can discern whether a study falls into the QI or Research category. If the PME Review is unsure whether or not a study falls into the category of research, the HMS IRB will provide guidance on this question. An HHS decision chart outlines the details of determining whether a project would be considered human subjects research.
QI studies, perhaps more familiar to clinicians in patient safety and quality approaches such as Plan-Do-Study-Act (PDSA), also apply to the continuous quality improvement of medical courses, clerkships, and curricula. Work that aims to assess and improve a specific course, program, or rotation is likely to fall into this category. Theoretical models may be employed to inform the study, but these are not always needed, and more exploratory, hypothesis-generating approaches are often prevalent. Studies with a QI approach can be published, but investigators should provide journals with a QI exemption letter from the PME Education Scholarship Review validating that the study was determined to be “not human subjects research.” Although human subjects, such as students, may be actively participating in the study, the study would not be considered “research.”
Information on publishing QI studies:
While it is always advisable to check with a specific journal editor or review articles in your targeted journal to ensure that they publish QI studies, in general, studies that have a “not human subjects research” determination can be published. In place of the usual description of IRB approval, you can simply provide journals with a QI exemption letter from the PME Education Scholarship Review Committee validating that the study was determined to be “not human subjects research.”
The US Department of Health and Human Services Office for Human Research Protections (OHRP) provides guidelines and regulations for research involving interaction with and/or data related to human subjects—in this case, medical students. The OHRP definition of research is as follows: “Research means a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge.” “Generalizable knowledge” is a key consideration in determining whether or not a study is considered research; that is, can the results be generalized to a larger population (of schools or students), or do the results primarily pertain to HMS? Studies that are designed to test theories or build knowledge in the field are likely to be considered research. Examples would include research on motivation, depression and burnout, or cognitive load.
IRB reviews fall into one of four categories:
- A determination of not human subjects research form (sometimes just called a “determination form,” which is a brief form used to determine if IRB review is needed or not from your hospital)
- Exempt review, which would receive an initial review but not require annual continuing review forms
- Expedited review (often required when there are data being collected that allow one to know the identity of specific students)
- Full board review, which is required when studies pose greater than minimal risk to subjects (this last category is rare in education research)
For more information see, refer to a description of criteria for each of these categories.
How To Apply
The proposal writer should start by reviewing the guidelines and submission information in the Before You Apply section. The Principal Investigator submits a Data Access Request to the PME Education Scholarship Review Committee. When the complete application has been submitted, the PI will receive a confirmation receipt by email.
The request will be screened for alignment with HMS curricular priorities, scientific merit and survey burden. Studies that address potentially sensitive topics will also require review and approval by the Dean for Medical Education. If the request is approved, you will be invited to complete the Online Submission Form to request an IRB QI exemption. If the request is not approved, you will receive a letter explaining why your data access and/or survey request has been denied, indicating a revise & resubmit or a rejection.
The PME Educational Scholarship Review Committee will review the Online Submission Form. The outcome of the review will be sent to the Principal Investigator. There are many requests to conduct research studies with students as participants, so unfortunately not all proposals can be approved. Applicants who receive conditional approval may choose to revise and resubmit their application. Feedback on resubmissions will be provided to investigators.
Approved studies will also be designated as either Research or Quality Improvement. If the committee deems your study to be QI, the study is IRB-exempt. The committee will send a letter granting IRB exemption. This letter can be submitted to journals with your manuscript in lieu of an IRB approval.
If the committee requires input from the IRB because it is a research study, they will refer you to the Longwood Campus. Research studies require HMS IRB review. The PME will send a copy of the PME Education Scholarship Study Review approval notice to the HMS IRB for all PME-approved research studies.
It is recommended that PI’s share their study with their hospital IRB in case any additional processes are needed.
Submit your application on the PME Research QI Protocol Form.
- Study Proposal – Upload either an IRB form or Word Document depending on whether it is a quality improvement or research proposal:
Quality Improvement proposals:
Investigators submit Quality improvement on an IRB form or in a Word document that contains the following: background, prior literature, importance of study, target population, sample size, justification, methods of data collection and analysis, expected response rate and information that will be provided to students.
Investigators submit research proposals either on their hospital's IRB form OR on the HMS IRB form.
Instruments and/or Measures – Upload surveys, interview guides and others. If instruments are not developed, you will be asked to submit the final instruments as an amendment to the study before data collection. Include any psychometric information that is available and relevant from previously validated instruments.
- Consent form(s), if applicable – HMS IRB Consent Form Templates
Letter(s) of Support from clerkship or course director(s), if applicable. If a study is to be conducted within a particular course or clerkship, documentation of the approval of the course/clerkship director should be included. A brief e-mail is acceptable.
- Recruitment materials, if applicable - Flyers, brochures, email text, study fact sheets
- Justification of rational for the selected student population – The submission link includes opportunities to explain study rational in detail. Examples: Why is it relevant for students rather than residents? Why year II students rather than year IV students? How much time would be required for student participation? If a new curricular component is being added to a course or clerkship, what existing elements will it replace?
A confirmation of receipt when the online proposal submission has been completed will be sent by email.