Assessment

• The type of assessment tool that is appropriate depends on what you are trying to measure, the stakes of the assessment, and learner level.

• Stakes of examination, feasibility, cost, reliability and validity are all considerations when implementing an assessment tool.

• Choosing an assessment tool should begin with searching for available tools and deciding if they are sufficiently reliable and valid for your population.

• Assessment should align with specific and measurable learning objectives, following the SMART framework.
• Assessments should align with the relevant domain (knowledge, attitudes, or skills) and measure what they intend to measure for internal validity.
• Common assessment types for each domain include knowledge assessments (awareness), skills assessments (physical tasks), and attitude assessments (feelings or emotions).
• Assessments should be aligned with the appropriate level of mastery, considering Bloom’s taxonomy for hierarchical levels of complexity and specificity.
• It is important to consider the level of learners when designing assessments, starting with lower levels of Bloom’s pyramid for novice learners and progressing to higher levels for experienced learners.
• Assessment tools, such as multiple-choice tests or checklists, should be populated with content that aligns with learning objectives and instructional content.
• Available resources, such as time, personnel, and expertise in psychometrics, should be considered when planning assessments.
• Formative assessments, such as self-assessments, can be used when resources are limited, but high stakes exams require adequate rater training and multiple observations.
• Cost and the availability of reliable and valid assessment options, such as purchasing subject exams from reputable organizations, should also be taken into account.

Knowledge Assessment Types:

• Multiple-choice question tests are commonly used for assessing knowledge. They have strengths such as requiring limited resources and measuring recall and application of knowledge. However, developing a reliable and valid exam with aligned questions can be labor-intensive.
• High-quality exams can be purchased if resources are available.

Attitude Assessment Types:

• Self-report assessments allow learners to directly report their attitudes. The strength is obtaining data directly from learners, but weaknesses include potential social desirability bias and attitudes not persisting in different contexts.
• Observation feedback from others who observe learners’ behavior can provide a more objective approach compared to self-report. However, there may be issues with data quality, and only attitudes manifested in behavior can be reported.

Skills Assessment Types:

• Direct observation allows assessing skills by observing learners performing the skill. The strength is seeing the skill in action, but challenges include time-consumption and coordination difficulties.
• Simulation is an effective method for skills assessment, eliminating patient safety concerns. It allows for in-the-moment formative feedback but can be costly and time-consuming.

General Considerations When Developing Assessment Tools:

• Use verbal labels or descriptions for each anchor instead of numbers.
• Include options for “Not Applicable” or “Not Observed” when necessary.
• Seek guidance from literature and subject matter experts for appropriate scale anchors.
• Balance the need for details with concerns about rater fatigue when determining the number of items on a tool.
• Ensure consistency of anchor domains/elements across all points on the scale.
• Determine a minimum number of ratings required for sufficient reliability.
• Incorporate prompts for narrative feedback to focus on student behaviors and improvement suggestions.
• Provide assessor training before and during tool implementation.
• Maintain awareness of the current assessor and assessment culture.
• Establish processes for ongoing assessor support and feedback.

Deciding between pre-existing and self-developed assessment tools:

• Consider the nature of what you want to assess: existing tools are more readily available for attitudes, behaviors, skills, and performance assessments, while knowledge assessments may require building a new tool.
• Check if there is an existing tool that meets your needs and can be used or modified.
• Assess your time frame and balance the need for a tailored tool with time constraints.

Finding existing assessment tools:

• Start with literature and work with librarians to locate relevant resources.
• Utilize resources like the MedEd Portal, professional societies, and colleagues within and outside your institution.
• Consider related fields beyond medical education.

Assessing existing tools:

• Leverage university or specialty resources to evaluate strengths and weaknesses of found tools.

• Example scenarios:

  • Dr. B found a checklist from a business school to rate students’ slide presentations.
  • Dr. C modified a gait and balance scale from another institution to assess students’ gait evaluation ability.

Building your own assessment tool:

• Use a blueprint summarizing lesson content and objectives to guide the development of questions in alignment.
• Seek resources for guidance on writing knowledge-based questions.
• Consider a blended approach by modifying existing tools or strategies for your educational activity.

• Reliability and validity are essential for effective assessments.
• Reliability refers to the consistency of results obtained from an assessment tool.
  • Internal consistency reliability measures the correlation between items intended to measure the same construct.
  • Test-retest reliability assesses the consistency of results when the tool is administered multiple times over a period.
  • Parallel forms reliability determines how well different assessments measuring the same construct agree.
  • Inter-rater reliability assesses the degree of agreement among multiple raters assessing the same individual.
• Multiple measurements improve the accuracy of reliability assessments.
• Validity refers to the accuracy of an assessment in measuring the intended construct.
• Establishing validity involves gathering evidence of the assessment’s accuracy from various sources.
• Experts in the construct of interest play a role in determining validity evidence.
• Additional assessment development tools and resources, such as item analysis and E-value, can aid in assessing reliability and validity.
• Reaching out to experts in assessment or applied statistics, such as schools of education or public health, can provide further guidance in determining reliability and validity.