New Idea Session on Competency-Based Evaluation of Predental Students from the data-driven evaluation side (predental advising and letters of evaluation).

1. Can progress in “Holistic admissions be measured”? Emil T. Chuck, Ph.D. Health professions Advisor Term Assistant Professor of Biology George Mason University 2010 ADEA Annual Session Saturday, February 27, 2010 3:00 to 3:25 pm, Potomac 5 (New idea session) Some data shown are preliminary and still being collected for future evaluation. Competency-Based Evaluation of Predental Applicants

3. Discuss whether the evaluation criteria fit professional competencies that are valued by the dental profession.

5. How prehealth advisors(?) view admissions Comment #184: “I must laugh at the credentials that this white student has. As an advisor for pre-med students at an Ivy-league school, my Asian students with 3.7-3.8 GPAs in the sciences have a rough time even getting interviews. A 3.3 GPA is a laughable GPA and even more laughable coming from a school like Pitt.” “Expecting a surge in U.S. medical schools” by AnemonaHartocollis (Feb 14, 2010 NYT) Peter Allen and Jessica Sidari, first-year students at The Commonwealth Medical College, a first-year school in Scranton, Pa Credit: Niko J. Kallianiotis for The New York Times

6. AAMC GPA/MCAT (2005-2007 accepted %)

7. “The system produces the result for which it was designed.”Bob Witzberg MD, Boston University SOM, AAMC Holistic Review Project

8. Holistic Review:Choosing from a pool of applicants Established substantive criteria for admissions that Reflect a justifiable balance among academic accomplishments and personal factors in applicants given the school’s mission-related goals. The school must have evidence for its admissions considerations with regard to: Academic background (major, GPA, letters, essays, etc.) Personal characteristics (background) Personal attributes (altruism, leadership, service) Personal experiences (relevant experiences, life experiences) Other criteria (please specify)

9. Competencies and Learning April 5, 2003, UNC ballroom dance club practice with RobertasMaleckis, Credit: Emil Chuck (personal)

10. Goals of competency-based evaluation Criteria should align with articulated technical standards of admissions (based on multiple personality theory). “Legal compliance” arguments tend to have more resistance to acceptance. Criteria should be transparent and independent of professional degree pursued by applicant to should reflect qualities desired for an interprofessional health care workforce. Can compare pools of applicants in the same year for different paths. Applicants should become familiar with competency-based evaluations as it will be a facet of their professional training and practice. Criteria should assist students with self-evaluation and references with confirmatory evaluation (360-evaluation) Quantitative measures should inform advising, program evaluation, and admissions outcomes.

11. GMU Prehealth Evaluation Rubric Compare to technical standards of admissions Academic foundation (science and non-science breadth/rigor) Scholarship and lifelong learning (problem-solving, research, entrepreneurship) Social intelligence (interpersonal, teamwork, collegial) Personal awareness (maturity, ethics, resiliency) Managerial experience (time, financial, project, leadership) Aesthetic or observational skills (art, music, literature) Kinesthetic dexterity (lab methods, sculpture, electronics) Communications skills (written/oral English, other) Knowledge of the profession and health care system Evaluators usually cannot comment on specific fitwith institutional mission and culture!

12. Competencies and Recommendations Newcomer: just starting out. Novice: follows instructions without questioning. Intermediate: memorizes steps, needs more practice. Proficient: can perform the tasks satisfactorily (minor difficulties). Competent: fluent; can improve the task/outcome by making minor adjustments. Expert: tests improvements to tasks in a systematic manner. Master: honored for exceptional skill development. No/Concern: general newcomer/novice to competency concepts. General: generally satisfactory/proficient in competencies Strong: shows additional development (competent) with competencies Enthusiastic: generally expert in competencies Highly: consistently honored (especially by peer review processes) to be exceptional

13. Example self-assessment Awards declared: I have made the dean’s list multiple times.

14. EY2011 preapplication pipeline

15. Rubric self-assessment EY2011 Applicants (n=147) Most confident:personal, emotional resilience and intelligence Least confident: academic foundation; mentoring/advising relationships.

16. Rubric self-assessment EY2011 Applicants (n=39) Predental Most confident:personal, emotional resilience and intelligence; observational skills; manual dexterity Least confident: academic foundation; mentoring/advising relationships.

17. Solicited evaluators assessment PRELIMINARY DATA ONLY (n=52), for EY2011 applicants, as of 24 Feb 2010. Solicited evaluators may overrate applicants’ skills development. Solicited evaluators rarely can comment on manual dexterity skills.

18. Committee evaluators assessment PRELIMINARY DATA ONLY (n=72), for EY2011 applicants, as of 26Feb 2010. Committee interviewers are more likely to detect lower-competency applicants (novice to proficient). Committee interviewers have more difficulty assessing manual dexterity & applicants’ collegial relationships with other advisors or faculty.

19. Evaluators’ summary assessment Committee EvaluatorsPRELIMINARY DATA ONLY (n=72), for EY2011 applicants, as of 26 Feb 2010. Solicited Evaluators PRELIMINARY DATA ONLY (n=52), for EY2011 applicants, as of 24 Feb 2010.

20. Solicited evaluators assessment PRELIMINARY DATA ONLY (n=53), for EY2011 applicants, as of 26 Feb 2010. Solicited evaluators have a relatively casual relationship or a knowledgeable relationship with applicants. 5/39 evaluators consider themselves “mentors” to applicant.

21. 360-assessment reveals calibration issues Applicants claim most confidence in their intrapersonal skills (personal resilience, hardship, evaluating criticism, faith). Applicants are least confident in academic skills and having strong mentoring relationships. Few solicited evaluators are “mentors” to applicants. Solicited evaluators will overrate/exaggerate the development of applicants’ skills (except for manual dexterity) compared to committee evaluators.

22. “Percent chance of admission” EY 2009

23. GPA applicants/accepted EY 2009

24. Average GPA’s for EY 2009 applicants All admissions decisions (GMU)

25. Average GPA’s for EY 2009 applicants All AADSAS 2009 admissions decisions (GMU)

26. Average GPA’s for EY 2010 applicants PRELIMINARY admissions decisions (GMU) as of February 19, 2010

27. Average GPA’s for AADSAS EY 2010 applicants PRELIMINARY AADSAS 2010 admissions decisions (GMU) as of February 19, 2010

28. Translating rubric to evaluators Competency-Based Holistic Evaluation of Prehealth Applicants Chuck ET. The Advisor (NAAHP), June 2009. Available online as PDF http://www.slideshare.net/etchuck/advisor-june-2009-competency-based-holistic-evaluations Writing better evaluation letters (solicited and committee) Competencies as a checklist of topics to cover. Training admissions readers and interviewers. Compare competencies against technical standards and admissions criteria. Communicate and provide feedback (compare with any competency assessments after D1/D2/D4 ).

29. Can holistic admissions be measured? Current challenges Applicants consider their competencies in their emotional maturity, integrity, and character as “expert”, as do solicited evaluators. Evaluators rate most applicants’ skills as “expert” though they do not usually have close mentoring relationships. Must better align evaluation rubric criteria with competency experiences/standards from professional education. Current opportunities with rubric “Research”-based standards of ideal health care professionals can be assessed, communicated, and “measured” with admissions outcomes. Improve prehealth advising and evaluation letters for applicants. Research any interprofessional (inter-institutional?) comparison of applicant holistic qualities. Introduce applicants to competency-based evaluation processes common in professional education and development.

31. Trend towards holistic competencies AAMC Holistic Review Project National Postdoctoral Association Core Competencies AAMC/HHMI Scientific Foundations for Future Physicians Assessment projects throughout medical education

32. It’s time for a checklist! Rubrics are checklists with an assessment angle.

33. Committee evaluators assessment PRELIMINARY DATA ONLY (n=65), for EY2011 applicants, as of 24 Feb 2010. Majority (84.6%) of committee evaluators suggest that reapplicants should have another set of MSAC interviews.

34. Average GPA’s for EY 2009 applicants All applicants (GMU)

35. Recommendations for EY 2009 applicants All admissions decisions (GMU)

36. Recommendations for EY 2010 applicants All admissions decisions (GMU) as of February 19, 2010

37. Recommendations for EY 2010 applicants All admissions decisions (GMU) as of February 19, 2010