You Are Here:

Month: April 2015

Multiple Choice and Bloom’s Taxonomy

Posted on by Johnston, Tracy

Blooms_Taxonomy_pyramid_cake-style-use-with-permission

*Graphic from http://tips.uark.edu/using-blooms-taxonomy/

It is often thought that multiple choice questions will only test on the first two levels of Bloom’s Taxonomy: remembering and understanding.

However, the resources point out that multiple choice questions can be written for the higher levels:  applying, analyzing, evaluating, and creating.

First, we can recognize the different types of multiple choice questions.  While I have used all of these myself, it never occurred to me to classify them.

Source: http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf

Types:

Question/Right answer

Incomplete statement

Best answer

In fact, this source states:

…almost any well-defined cognitive objective can be tested fairly in a multiple choice format.

Advantages:

  • Very effective
  • Versatile at all levels
  • Minimum of writing for student
  • Guessing reduced
  • Can cover broad range of content

Can provide an excellent basis for post-test discussion, especially if the discussion addresses why the incorrect responses were wrong as well as why the correct responses were right.

Disadvantages:

  • Difficult to construct good test items
  • Difficult to come up with plausible distractors/alternative responses

They may appear too discriminating to students, especially when the alternatives are well constructed and are open to misinterpretation by students who read more into questions than is there.

So what can we do to make multiple choice questions work for higher levels of Bloom’s?

Source: http://www.uleth.ca/edu/runte/tests/

To Access Higher Levels in Bloom’s Taxonomy

Don’t confuse “higher thinking skills” with “difficulty” or “complicated”

      • use data or pictures to go beyond recall
      • use multiple choice to get at skill questions

Ideas:

  • Read and interpret a chart
  • Create a chart
  • “Cause and effect”  (e.g., read a map and draw a conclusion)

Another part of this source brings up the idea of using the “inquiry process” to present a family of problems that ask the student to analyze a quote or situation.

    • No more than 5 or 6 questions to a family
    • Simulates going through inquiry process, step-by-step
      • Identify the issue
      • Address advanced skill of organizing a good research question
      • Ask an opinion question (but not the student’s opinion)
      • Analyze implicit assumptions
      • Provide for a condition contrary to the facts, “hypothesize”

This source gives some good ideas, too.

Source: http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf

Develop questions that resemble miniature “cases” or situations.  Provide a small collection of data, such as a description of a situation, a series of graphs, quotes, a paragraph, or any cluster of the kinds of raw information that might be appropriate material.

Then develop a series of questions based on that material.  These questions might require students to apply learned concepts to the case, to combine data, to make a prediction on the outcome of a process, to analyze a relationship between pieces of the information, or to synthesize pieces of information into a new concept.

In short, multiple choice questions, when designed with good structure and strategies, can provide an in-depth evaluation of a student’s knowledge and understanding.  It can be challenging to write those good questions but the benefits are worthwhile.

I thought about writing a summary of what we have learned about multiple choice questions but found this funny little quiz to be better than anything I could come up with:

Can you answer these 6 questions about multiple-choice questions?

Multiple Choice Structure

Posted on by Johnston, Tracy

Taking a Test

One type of objective question is multiple choice.  We all know what it is but let’s look in detail at its description anyway.

Source: https://www.msu.edu/dept/soweb/writitem.html

Description of a multiple choice item:

Presents a problem or question in the stem of the item and requires the selection of the best answer or option. The options consist of a most-correct answer and one or more distractors or foils.

The major purpose of a multiple choice item is to identify examinees who do not have complete command of the concept or principle involved.

Properties:
• State the problem in the stem
• Include one correct or most defensible answer
• Select diagnostic foils or distractors such as:

o Clichés
o Common misinformation
o Logical interpretations
o Partial answers
o Technical terms or textbook jargon

The distractors must appear reasonable as the correct answer to the students who have not mastered the material.

So the structure of a multiple choice question is a stem followed by options.  The options contain one correct answer and a set of distractors.

The Stem

Some advice for constructing a good stem is

Source: http://www.iub.edu/~best/pdf_docs/better_tests.pdf

  • Write questions that test a significant concept, that are unambiguous, and that don’t give test-wise students an advantage
  • The stem should fully state the problem and all qualifications. Always include a verb in the statement
  • Items should measure students’ ability to comprehend, apply, analyze, and evaluate as well as recall
  • Include words in the stem that would otherwise be repeated in each option
  • Eliminate excessive wording and irrelevant information in the stem

Here are some examples of good and bad stem design:

Source:  http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/#stem

     

A stem that does not present a clear problem, however, may test students’ ability to draw inferences from vague descriptions rather serving as a more direct test of students’ achievement of the learning outcome.

  

If a significant learning outcome requires negative phrasing, such as identification of dangerous laboratory or clinical practices, the negative element should be emphasized with italics or capitalization.

  

A question stem is preferable because it allows the student to focus on answering the question rather than holding the partial sentence in working memory and sequentially completing it with each alternative

The best thought about the stem I have seen on the Internet:

Source: http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf

Way to judge a good stem: students who know the content should be able to answer before reading the alternatives.

The Options

Sometimes known as “the alternatives”, they are composed of one right answer and a group of “foils” or distractors.

One point that is emphasized regularly in the resources is that the distractors should all be plausible and attractive answers.

Source:  http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/

Common student errors provide the best source of distractors.

Alternatives should be stated clearly and concisely. Items that are excessively wordy assess students’ reading ability rather than their attainment of the learning objective

Alternatives should be mutually exclusive. Alternatives with overlapping content may be considered “trick” items by test-takers, excessive use of which can erode trust and respect for the testing process.

[Ed. Note:  I have some issues with this particular example but I get the point of their suggestion.]

Alternatives should be homogenous in content. Alternatives that are heterogeneous in content can provide cues to student about the correct answer.

  

The alternatives should be presented in a logical order (e.g., alphabetical or numerical) to avoid a bias toward certain positions.

Avoid complex multiple choice items, in which some or all of the alternatives consist of different combinations of options. As with “all of the above” answers, a sophisticated test-taker can use partial knowledge to achieve a correct answer.

Other suggestions from this source:

Alternatives should be free from clues about which response is correct. Sophisticated test-takers are alert to inadvertent clues to the correct answer, such differences in grammar, length, formatting, and language choice in the alternatives. It’s therefore important that alternatives

  • have grammar consistent with the stem.
  • are parallel in form.
  • are similar in length.
  • use similar language (e.g., all unlike textbook language or all like textbook language).

The alternatives “all of the above” and “none of the above” should not be used. When “all of the above” is used as an answer, test-takers who can identify more than one alternative as correct can select the correct answer even if unsure about other alternative(s). When “none of the above” is used as an alternative, test-takers who can eliminate a single option can thereby eliminate a second option. In either case, students can use partial knowledge to arrive at a correct answer.

The number of alternatives can vary among items as long as all alternatives are plausible. Plausible alternatives serve as functional distractors, which are those chosen by students that have not achieved the objective but ignored by students that have achieved the objective. There is little difference in difficulty, discrimination, and test score reliability among items containing two, three, and four distractors.

Keep the specific content of items independent of one another. Savvy test-takers can use information in one question to answer another question, reducing the validity of the test.

 

There is more to think about for multiple choice questions, which we will examine in the next post.

Objective or Subjective? Those are the Questions

Posted on by Johnston, Tracy

tobeornottobe

Now that we have studied general test writing strategies, ideas, and tips, it is time to pull our focus inward to the details of the questions themselves.

In general, question types fall into two categories:

  1. Objective
  2. Subjective

I needed specific definitions for these, which I found here.

Source: http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf

1. Objective, which require students to select the correct response from several alternatives or to supply a word or short phrase to answer a question or complete a statement.

Examples: multiple choice, true-false, matching, completion

2. Subjective or essay, which permit the student to organize and present an original answer

Examples: short-answer essay, extended-response essay, problem solving, performance test items

This source also suggests guidelines for choosing between them:

Essay tests are appropriate when:

  • The group to be tested is small and the test is not to be reused
  • You wish to encourage and reward the development of student skill in writing
  • You are more interested in exploring student attitudes than in measuring his/her achievement

Objective tests are appropriate when:

  • The group to be tested is large and the test may be reused.
  • Highly reliable scores must be obtained as efficiently as possible.
  • Impartiality of evaluation, fairness, and free from possible test scoring influences are essential.

Either essay or objective tests can be used to:

  • Measure almost any important educational achievement a written test can measure
  • Test understanding and ability to apply principles.
  • Test ability to think critically.
  • Test ability to solve problems.

And it continues with this bit of advice:

 The matching of learning objective expectations with certain item types provides a high degree of test validity:  testing what is supposed to be tested.

  • Demonstrate or show: performance test items
  • Explain or describe: essay test items

I wanted to see what different sources would say, so I also found this one.

Source: http://www.helpteaching.com/about/how_to_write_good_test_questions/

If you want the student to compare and contrast an issue taught during a history lesson, open ended questions may be the best option to evaluate the student’s understanding of the subject matter.

If you are seeking to measure the student’s reasoning skills, analysis skills, or general comprehension of a subject matter, consider selecting primarily multiple choice questions.

Or, for a varied approach, utilize a combination of all available test question types so that you can appeal to the learning strengths of any student on the exam.

Take into consideration both the objectives of the test and the overall time available for taking and scoring your tests when selecting the best format.

I am not sure that “multiple choice” should be the primary choice but I understand they are suggesting to avoid open-ended questions if you want to measure reasoning or analytic skills or general comprehension.

This bothers me a little.  It seems to me, from reviewing the previous posts in this blog, that an open-ended question could measure those skills.  The example that comes to mind is the question I had in botany about describing the cell types a pin might encounter when passing through a plant stem.  That was an essay question measuring general comprehension of plant tissues.

The following source brings up good points about analyzing the results.  It also notes that objective tests, when “constructed imaginatively,” can test at higher levels of Bloom’s Taxonomy.

Source: http://www.calm.hw.ac.uk/GeneralAuthoring/031112-goodpracticeguide-hw.pdf

Objective tests are especially well suited to certain types of tasks. Because questions can be designed to be answered quickly, they allow lecturers to test students on a wide range of material. … Additionally, statistical analysis on the performance of individual students, cohorts and questions is possible.

The capacity of objective tests to assess a wide range of learning is often underestimated. Objective tests are very good at examining recall of facts, knowledge and application of terms, and questions that require short text or numerical responses. But a common worry is that objective tests cannot assess learning beyond basic comprehension.

However, questions that are constructed imaginatively can challenge students and test higher learning levels. For example, students can be presented with case studies or a collection of data (such as a set of medical symptoms) and be asked to provide an analysis by answering a series of questions…

Problem solving can also be assessed with the right type of questions. …

A further worry is that objective tests result in inflated scores due to guessing. However, the effects of guessing can be eliminated through a combination of question design and scoring techniques. With the right number of questions and distracters, distortion through guessing becomes largely irrelevant. Alternatively, guessing can be encouraged and measured if this is thought to be a desirable skill.

There are, however, limits to what objective tests can assess. They cannot, for example, test the competence to communicate, the skill of constructing arguments or the ability to offer original responses. Tests must be carefully constructed in order to avoid the decontextualisation of knowledge (Paxton 1998) and it is wise to use objective testing as only one of a variety of assessment methods within a module. However, in times of growing student numbers and decreasing resources, objective testing can offer a viable addition to the range of assessment types available to a teacher or lecturer.

I like their point about how objective tests cannot test competence to communicate, construct arguments, or offer original answers.  Training our students to take only multiple choice tests (or simply answer “true” or “false”) does not help them to learn how to explain their thoughts or even ensure that they can write coherent sentences.

This is addressed by the second source and in previous posts.  The suggestion is to use a variety of test item types.  This can give you a better picture of what your students know, whereas using one single type can be biased against students who are not strong respondents to that type.

Strategy Summary

Posted on by Johnston, Tracy

summary

We are at the point of our investigation where we need to start looking in more detail at test construction.  Here is a brief summary that puts together the pieces of what we have learned so far.

Our challenges

Write an accurate measure of student achievement that

  • motivates students and reinforces learning,
  • enables us to assess student mastery of course objectives,
  • and allows us to recognize what material was or was not communicated clearly.

Some things we can do to accomplish this

In general, when designing a test we need to

  • consider the length of the test,
  • write clear, concise instructions,
  • use a variety of test item formats,
  • test early and/or frequently,
  • proofread and check for accuracy,
  • consider the needs of our disabled or non-native speaker students,
  • and use humor.

More specifically, our test goals are to

  • Assess achievement of instructional objectives,
  • measure important aspects of the subject,
  • accurately reflect the emphasis placed on important aspects of instruction,
  • measure an appropriate level of student knowledge,
  • and have the questions vary in levels of difficulty.

We should consider the technical quality of our tests

Quality means “conformance to requirements” and “fitness for use.”  The criteria are

  • offering cognitive complexity,
  • reviewing content quality,
  • writing meaningful questions,
  • using appropriate language,
  • being able to generalize about student learning from their test performance,
  • and writing a fair test with answers that represent what students know.

A useful tool is Bloom’s Taxonomy

It lists learning levels in increasing order of complexity:

  1. Remembering
  2. Understanding
  3. Applying
  4. Analyzing
  5. Evaluating
  6. Creating

To apply Bloom’s directly, we looked at

  • Lists of verbs associated with each level (some were discipline-specific),
  • question frames — nearly complete questions we can modify for our topics,
  • and knowledge domains — the kinds of knowledge that can be tested:
    • factual,
    • conceptual,
    • procedural,
    • and metacognitive.

Next up:  Learning what question types to use to achieve our goals.