You Are Here:

Month: October 2015

Make Me a Match!

Posted on by Johnston, Tracy

2-pattern-matching

One of my favorite (but not used all that much) test item types is the “matching exercise.”  One class I teach has quite a bit of vocabulary that my students just flat-out need to memorize.  Matching seems like a good, concise way of testing them with a minimum amount of pain on their part (writing the answers) and my part (creating the test).

The sources all agree on the definition:

A matching exercise consists of a list of questions or problems to be answered along with a list of responses.  The examinee is required to make an association between each question and a response.

(Source:http://www.iub.edu/~best/pdf_docs/better_tests.pdf)

I was pleased to see this same source describing the types of material that can be used:

The most common is to use verbal statements…  The problems might be locations on a map, geographic features on a contour map, parts of a diagram of the body, biological specimens, or math problems.

Similarly, the responses don’t have to be terms or labels, they might be functions of various parts of the body, or methods, principles, or solutions.

This other source, http://teaching.uncc.edu/learning-resources/articles-books/best-practice/assessment-grading/designing-test-questions, lists

  • terms with definitions
  • phrases with other phrases
  • causes with effects
  • parts with larger units
  • problems with solutions

As you can see, this test item format is well-suited for testing the Knowledge Level of Bloom’s Taxonomy, however several sources hint that it can apply to the Comprehension Level “if appropriately constructed.”

Only one source discusses in detail how to “aim for higher order thinking skills” by describing variations that address, for example, Analysis and Synthesis.  (http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf)

One variation is to give a Keylist or Masterlist, that is information about several objects, and have the student interpret the meaning of the information, do comparisons (least/greatest, highest/lowest, etc.), and translate symbols.  The example gives three elements from the periodic table with the properties listed below them but no title on the properties.  The questions ask “Which of the above elements has the largest atomic weight?” and “Which has the lowest melting point?” and other similar inquiries.

Another variation is a ranking example:

Directions:  Number (1 – 8) the following events in the history of ancient Egypt in the order in which they occurred, using 1 for the earliest event.

These directions are followed by a list of events.

While I see these variations more as the “fill-in-the-blank” types, their connections to matching properties to objects or events to a time line make it reasonable to treat them as matching types.

What are the advantages and disadvantages of matching exercises?

(Source: http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/)

These questions help students see the relationships among a set of items and integrate knowledge.

They are less suited than multiple-choice items for measuring higher levels of performance.

 

(Source: http://www.iub.edu/~best/pdf_docs/better_tests.pdf)

Because matching items permit one to cover a lot of content in one exercise, they are an efficient way to measure.

It is difficult, however, to write matching items that require more than simple recall of factual knowledge.

 

(Source:  http://teaching.uncc.edu/learning-resources/articles-books/best-practice/assessment-grading/designing-test-questions)

Maximum coverage at knowledge level in a minimum amount of space/prep time.

Valuable in content areas that have a lot of facts.

But

Time consuming for students.

There are design strategies that can reduce the amount of time it takes for students to work through the exercise, and others that don’t put so much emphasis on reading skills.  We’ll look at those in the next post.

Alternative-Response Design: Structure and Advice

Posted on by Johnston, Tracy

In the previous post we talked about the pros and cons of the alternative-response (e.g., true-false) types of questions as well as their application to Bloom’s Taxonomy.  Next we discuss aspects to consider when writing the questions.

I found this “Simple Guidelines” list helpful and informative.

(Source: http://webs.rtc.edu/ii/Teaching%20Resources/GuidelinesforWritingTest.htm)

  1. Base the item on a single idea.
  2. Write items that test an important idea
  3. Avoid lifting statements right from the textbook.
  4. Make the questions a brief as possible
  5. Write clearly true or clearly false statements.  Write them in pairs: one “true” and one “false” version and choose one to keep balance on the test.
  6. Eliminate giveaways:
    • Keep true and false statements approximately equal in length
    • Make half the statements true and half false.
    • Try to avoid such words as “all,” “always,” “never,” “only,” “nothing,” and “alone.” Students know these words usually signify false statements.
  7. Beware of words denoting indefinite degree.  The use of words like “more,” “less,” “important,” “unimportant,” “large,” “small,” “recent,” “old,” “tall,” “great,” and so on, can easily lead to ambiguity.
  8. State items positively.  Negative statements may be difficult to interpret.  This is especially true of statements using the double negative.  If a negative word, such as “not” or “never,” is used, be sure to underline or capitalize it.
  9. Beware of detectable answer patterns.  Students can pick out patterns such as (TTTTFFFF) which might be designed to make scoring easier.

All of this makes sense to me.  At first I objected to “Make half the statements true and half false” but when I thought about it, I wouldn’t do exactly half necessarily but maybe close to half.  In fact this source, http://teaching.uncc.edu/learning-resources/articles-books/best-practice/assessment-grading/designing-test-questions,  suggests making the ratio more like 60% false to 40% true since students are more likely to guess the answer is true.

I found other points to add to the guidelines list.  (Source:  http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf)

Two ideas can be included in a true-false statement if the purpose is to show cause and effect.

  • If a proposition expresses a relationship, such as cause and effect or premise and conclusion, present the correct part of the statement first and vary the truth or falsity of the second part.
  • When a true-false statement is an opinion, it should be attributed to someone in the statement.
  • Underlining or circling answers is preferable to having the student write them.
  • Make use of popular misconceptions/beliefs as false statements.
  • Write items so that the incorrect response is more plausible or attractive to those without the specialized knowledge being tested.
  • Avoid the use of unfamiliar vocabulary.
  • Determine that the questions are appropriately answered by “True” or “False” rather than by some other type of response, such as “Yes” or “No.”
  • Avoid the tendency to add details in true statements to make them more precise.  The answers should not be obvious to students who do not know the material.
  • Be sure to include directions that tell students how and where to mark their responses.

This same source gives you a nice tip for writing true-false items:

Write a set of true statements that cover the content, then convert approximately half of them to false statements.  State the false items positively, avoiding negatives or double negatives.

Most of this discussion has been about True-False questions but the category is really Alternative-Response.  Let’s look at the variations available to us.

(Source:  http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf)

  • The True-False-Correction Question
    In this variation, true-false statements are presented with a key word or brief phrase that is underlined.  It is not enough that a student correctly identify a statement as being false.  … the student must also supply the correct word or phrase which, when used to replace the underlined part of the statement, makes the statement a true one.This type of item is more thorough in determining whether students actually know the information that is presented in the false statements.

    The teacher decides what word/phrase can be changed in the sentence; if students were instructed only to make the statement a true statement, they would have the liberty of completely rewriting the statement so that the teacher might not be able to determine whether or not the student understood what was wrong with the original statement.

    If, however, the underlined word/phrase is one that can be changed to its opposite, it loses the advantage over the simpler true-false question because all the student has to know is that the statement is false and change is to is not.

  • The Yes-No Variation
    The student responds to each item by writing, circling or indicating yes-no rather than true-false.  An example follows:

What reasons are given by students for taking evening classes?  In the list below, circle Yes if that is one of the reasons given by students for enrolling in evening classes; circle No if that is not a reason given by students.

Yes   No   They are employed during the day.
Yes   No   They are working toward a degree.
Yes   No   They like going to school.
Yes   No   There are no good television shows to watch.
Yes   No   Parking is more plentiful at night.

  • The A-B Variation
    The example below shows a question for which the same two answers apply.  The answers are categories of content rather than true-false or yes-no.

Indicate whether each type of question below is a selection type or a supply type by circling A if it is a selection , B if it is supply.

Select     Supply
A      B            Multiple Choice
A      B            True-False
A      B            Essay
A      B            Matching
A      B            Short Answer

In summary, the sources all tend to agree that the best type of Alternative-Response items are those that are unambiguous (“true or false with respect to what?”), concisely written, covering one idea per question, and aimed at more than rote memorization.  We should avoid trick questions or questions that test on trivia.  And the best tests with A-R items have a lot of questions with a True-to-False ratio of 40:60.

Next test item type:  Matching!

Alternative-Response: True/False and Similar Items

Posted on by Johnston, Tracy

truefalse

My original intent for the title here was just “True or False Test Items” but one source pointed out that the best name is Alternative-Response.  That source is https://www.msu.edu/dept/soweb/writitem.html and defines alternative-response as

… a special case of the multiple-choice item format.  There are many situations which call for either-or decisions, such as deciding whether a specific solution is right or wrong, whether to continue or to stop, whether to use a singular or plural construction, and so on.  For such situations, the alternative-response item is an ideal measuring device.  

It goes on to point out the advantages of this item type:

Since only two options are possible, alternative-response items are generally shorter, and, therefore, require less reading time. Students may respond to more alternative-response items than other types of items in a given length of time.

But there is a major disadvantage:  “students have fifty-fifty probability of answering the item correctly by chance alone.”

It suggests making up for this by offering “a larger number of alternative-choice items than of other  types of items in order to achieve a given level of test reliability.”

There are other advantages and disadvantages, such as these offered by http://www.iub.edu/~best/pdf_docs/better_tests.pdf, which only addresses true-false items.

Strength:  They are relatively easy to write.

Limitation:  Items are often ambiguous because of the difficulty of writing statements that are unequivocally true or false.

To me this seems like a contradiction:  How can they be easy to write but difficult to make unambiguous?  This same source offers tips on writing good true-false items which we will address in the next post.

This source:  http://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf  brings up some other points on true-false items.

Good for:  

  • Knowledge level content
  • Evaluating student understanding of popular misconceptions
  • Concepts with two logical responses

Advantages:  

  • Can test large amounts of content
  • True-false are adaptable to the measurement of a wide variety of learning outcomes
  • Quick and easy to grade
  • If constructed well, can be highly reliable in assessing student knowledge

Disadvantages:  

  • It is difficult to discriminate between students that know the material and students who do not
  • Individual true-false items are less discriminating than individual multiple choice items
  • There is a tendency to write trivial true-false items, which lead students to verbatim memorization
  • True-false items are not amenable to concepts that cannot be formulated as propositions
  • The extent of students’ command of a particular area of knowledge is indicated by their success in judging the truth or falsity of propositions related to it

Bloom’s Taxonomy is of concern for us here, too.  Most sources I read indicate these types of questions items address only the first level of Bloom’s, Knowledge.  We saw that above with the source listing “Good for” bullet points.  Other sources say it bluntly:

True and false questions are best used when you are looking to test a student’s recall ability of specific facts or knowledge.

(Source:  http://www.helpteaching.com/about/how_to_write_good_test_questions)

However this source, http://www.iub.edu/~best/pdf_docs/better_tests.pdf, makes this statement:

Instructors generally use true-false items to measure the recall of factual knowledge such as names, events, dates, definitions, etc.  But this format has the potential to measure higher levels of cognitive ability, such as comprehension of significant ideas and their application in solving problems.

It goes on to give four examples, the first of which is just recall of facts but the others require the student to recall the important information and apply it correctly in order to answer without guessing.

T   F  1.  Jupiter is the largest planet in the solar system.

T   F  2. If Triangle ABC is isosceles and angle A measures 100 degrees, then angle B is 100 degrees.

T   F  3.  If a distribution of scores has a few extremely low scores, then the median will be numerically larger than the mean.

T   F  4.  The larger the number of scores in a distribution, the larger the standard deviation of the scores must be.

(In case you were wondering, the answers are T, F, T, F.)

The first example above measures recall of a specific fact.  The other examples, however, show how a true-false item can be written to measure comprehension and application.

We can see that true-false, alternative-response type questions have the potential to address higher levels of Bloom’s as well as cover a large portion of course material with minimal reading on the student’s part.  This question type can benefit our exams if its strengths and weaknesses are considered carefully when being used.

In the next post we will look at the recommendations for writing quality alternative-response questions.