Research Design for Pilot

Research Design for Pilot

The experimental design for the pilot was constructed in a way to minimize any disruption to your current process while still allowing us to answer the question: does an additional prescreen designed to assess interpersonal skills (e.g., interpersonal communications, professionalism, leadership and empathy) using remote, on-demand interviewing and simulations result in greater benefits to your program’s match process and outcomes. In other words, does it improve: 1) efficiency in the process; 2) quality of candidates; and 3) likelihood of matching the candidates you most desire. More about the specific measures later.

Since it is a quality improvement, it does not need IRB approval.

You maintain your screening and selection practices for deciding whom you bring in for a site visit and face-to-face interview.   Let’s call the group you bring in your “green group.” You almost certainly had a “yellow group”, a group who were close and basically met your criteria but just didn’t make the cut for the face-face interview, often because you have a limited number of slots. The yellow group is the focus of our pilot.

For the pilot, we are asking that you open a few additional face-to-face interview slots. You will send us a fixed number of your “yellow” candidates. We will then use our remote interviewing/simulation process to evaluate the interpersonal and communication skills, empathy, leadership and professionalism on these “yellow” candidates and screen this group down. We will not evaluate technical skills or knowledge. We will then return a small number of candidates for you to include in your on-site interviews. Our “on-demand” interviews for candidates take about 30 minutes and can be completed anytime, 24-7 through any internet connected device (laptop, table or smart phone). We will provide a window for the candidates to complete the interview, typically on the order of a 1 to 2 week window.

You then agree to fold these additional candidates into your on-site interview process and blindly treat them as any other candidate in the mix.

We envision three primary measures to determine the impact of the prescreen:

• The first measure will be your ranking. Specifically, were the candidates who underwent a prescreen on interpersonal skills more likely to get a higher rank relative to their peers who went through the process without the prescreen?
• A second measure will be the selection ratio. Does the use of the prescreen positively impact the ratio of candidates that you interview to the number that you deem appropriate to your program.
• The third measure is the acceptance rate. Are prescreened candidates more likely to match to your program.

The results of the three measures can help answer the question: can a prescreen on interpersonal skills result in fewer on-site, face-to-face interviews while maintaining or improving candidate quality and achieving an enhanced match ratio for your better candidates.

The null hypothesis is, of course, no difference on the three measures.

We will aggregate data across several pilot programs to increase sample sizes.

We would like to limit the pilot to internal medicine, family medicine, pediatrics and emergency medicine programs.

It is not a perfect study and won’t definitively answer all our questions, but as a pilot it will certainly advance our understanding of how to better screen candidates for residency programs. While we all seek great breakthroughs, moving our understanding incrementally forward is the more common outcome. That is often  the role  of scientific inquiry in practice.

What’s in it for your program to participate in this pilot?  In addition to the opportunity to improve the three measures above, there are other benefits:

• A deeper understanding of the role that interpersonal skills play in resident selection along with other skills, such as tests scores, resume, various ERAS profile factors, etc.
• Finding ways to increase efficiency of an intensely busy process run by intensely busy people.
• Contributing to the body of research and literature in graduate medical education.
• How can I bring in better quality candidates onsite?
• Longitudinal research opportunities as we track candidates through your GME program

Thank you for your consideration. If you are interested in participating in a pilot, please submit your email below. There are a limited number of opportunities so we cannot guarantee everyone who is interested will be able to participate in this pilot.