Designing tests is an important part of assessing students understanding of course content and their level of competency in applying what they are learning. Whether you use low-stakes and frequent evaluations–quizzes–or high-stakes and infrequent evaluations–midterm and final–careful design will help provide more calibrated results.
Here are a few general guidelines to help you get started:
- Consider your reasons for testing.
- Will this quiz monitor the students’ progress so that you can adjust the pace of the course?
- Will ongoing quizzes serve to motivate students?
- Will this final provide data for a grade at the end of the quarter?
- Will this mid-term challenge students to apply concepts learned so far?
The reason(s) for giving a test will help you determine features such as length, format, level of detail required in answers, and the time frame for returning results to the students.
- Maintain consistency between goals for the course, methods of teaching, and the tests used to measure achievement of goals. If, for example, class time emphasizes review and recall of information, then so can the test; if class time emphasizes analysis and synthesis, then the test can also be designed to demonstrate how well students have learned these things.
- Use testing methods that are appropriate to learning goals. For example, a multiple choice test might be useful for demonstrating memory and recall, for example, but it may require an essay or open-ended problem-solving for students to demonstrate more independent analysis or synthesis.
- Help Students prepare. Most students will assume that the test is designed to measure what is most important for them to learn in the course. You can help students prepare for the test by clarifying course goals as well as reviewing material. This will allow the test to reinforce what you most want students to learn and retain.
- Use consistent language (in stating goals, in talking in class, and in writing test questions) to describe expected outcomes. If you want to use words like explain or discuss, be sure that you use them consistently and that students know what you mean when you use them.
- Design test items that allow students to show a range of learning. That is, students who have not fully mastered everything in the course should still be able to demonstrate how much they have learned.
Multiple choice exams
Multiple choice questions can be difficult to write, especially if you want students to go beyond recall of information, but the exams are easier to grade than essay or short-answer exams. On the other hand, multiple choice exams provide less opportunity than essay or short-answer exams for you to determine how well the students can think about the course content or use the language of the discipline in responding to questions.
If you decide you want to test mostly recall of information or facts and you need to do so in the most efficient way, then you should consider using multiple choice tests.
The following ideas may be helpful as you begin to plan for a multiple choice exam:
- Since questions can result in misleading wording and misinterpretation, try to have a colleague answer your test questions before the students do.
- Be sure that the question is clear within the stem so that students do not have to read the various options to know what the question is asking.
- Avoid writing items that lead students to choose the right answer for the wrong reasons. For instance, avoid making the correct alternative the longest or most qualified one, or the only one that is grammatically appropriate to the stem.
- Try to design items that tap students’ overall understanding of the subject. Although you may want to include some items that only require recognition, avoid the temptation to write items that are difficult because they are taken from obscure passages (footnotes, for instance).
- Consider a formal assessment of your multiple-choice questions with what is known as an “item analysis” of the test.
- Which questions proved to be the most difficult?
- Were there questions which most of the students with high grades missed?
This information can help you identify areas in which students need further work, and can also help you assess the test itself: Were the questions worded clearly? Was the level of difficulty appropriate? If scores are uniformly high, for example, you may be doing everything right, or have an unusually good class. On the other hand, your test may not have measured what you intended it to.
“Essay tests let students display their overall understanding of a topic and demonstrate their ability to think critically, organize their thoughts, and be creative and original. While essay and short-answer questions are easier to design than multiple-choice tests, they are more difficult and time-consuming to score. Moreover, essay tests can suffer from unreliable grading; that is, grades on the same response may vary from reader to reader or from time to time by the same reader. For this reason, some faculty prefer short-answer items to essay tests. On the other hand, essay tests are the best measure of students’ skills in higher-order thinking and written expression.”
(Barbara Gross Davis, Tools for Teaching, 1993, 272)
When are essay exams appropriate?
- When you are measuring students’ ability to analyze, synthesize, or evaluate
- When you have been teaching at these levels (i.e. writing intensive courses, upper-division undergraduate seminars, graduate courses) or the content lends it self to more critical analysis as opposed to recalling information
How do you design essay exams?
- Be specific
- Use words and phrases that alert students to the kind of thinking you expect; for example, identify, compare, or critique
- Indicate with points (or time limits) the approximate amount of time students should spend on each question and the level of detail expected in their responses
- Be aware of time; practice taking the exam yourself or ask a colleague to look at the questions
How do you grade essay exams?
- Develop criteria for appropriate responses to each essay question
- Develop a scoring guide that tell what you are looking for in each response and how much credit you intend to give for each part of the response
- Read all of the responses to question 1, then all of the responses to question 2, and on through the exam. This will provide a more holistic view of how the class answered the individual questions
How do you help students succeed on essay exams?
- Use study questions that ask for the same kind of thinking you expect on exams
- During lecture or discussion emphasize examples of thinking that would be appropriate on essay exams
- Provide practice exams or sample test questions
- Show examples of successful exam answers
Assessing your test
Regardless of the kind of exams you use, you can assess their effectiveness by asking yourself some basic questions:
- Did I test for what I thought I was testing for?
If you wanted to know whether students could apply a concept to a new situation, but mostly asked questions determining whether they could label parts or define terms, then you tested for recall rather than application.
- Did I test what I taught?
For example, your questions may have tested the students’ understanding of surface features or procedures, while you had been lecturing on causation or relation–not so much what the names of the bones of the foot are, but how they work together when we walk.
- Did I test for what I emphasized in class?
Make sure that you have asked most of the questions about the material you feel is the most important, especially if you have emphasized it in class. Avoid questions on obscure material that are weighted the same as questions on crucial material.
- Is the material I tested for really what I wanted students to learn?
For example, if you wanted students to use analytical skills such as the ability to recognize patterns or draw inferences, but only used true-false questions requiring non-inferential recall, you might try writing more complex true-false or multiple-choice questions.
Short Answer & Essay Tests
Strategies, Ideas, and Recommendations from the faculty Development Literature
- Do not use essay questions to evaluate understanding that could be tested with multiple-choice questions.
Save essay questions for testing higher levels of thought (application, synthesis, and evaluation), not recall facts. Appropriate tasks for essays include: Comparing: Identify the similarities and differences between
Relating cause and effect: What are the major causes of...? What would be the most likely effects of...?
Justifying: Explain why you agree or disagree with the following statement.
Generalizing: State a set of principles that can explain the following events.
Inferring: How would character X react to the following?
Creating: what would happen if...?
Applying: Describe a situation that illustrates the principle of.
Analyzing: Find and correct the reasoning errors in the following passage.
Evaluating: Assess the strengths and weaknesses of.
- Don't give students a choice of questions to answer.
There are three drawbacks to giving students a choice. First, some students will waste time trying to decide which questions to answer. Second, you will not know whether all students are equally knowledgeable about all the topics covered on the test. Third, since some questions are likely to be harder than others, the test could be unfair.
- Ask students to write more than one essay.
Tests that ask only one question are less valid and reliable than those with a wider sampling of test items. In a fifty-minute class period, you may be able to pose three essay questions or ten short answer questions.
- Give students advice on how to approach an essay or short-answer test.
To reduce students' anxiety and help them see that you want them to do their best, give them pointers on how to take an essay exam. For example:
- Survey the entire test quickly, noting the directions and estimating the importance and difficulty of each question. If ideas or answers come to mind, jot them down quickly.
- Outline each answer before you begin to write. Jot down notes on important points, arrange them in a pattern, and add specific details under each point.
Writing Effective Test Questions
- State the question clearly and precisely.
Avoid vague questions that could lead students to different interpretations. If you use the word "how" or "why" in an essay question, students will be better able to develop a clear thesis. As examples of essay and short-answer questions:
Poor: What are three types of market organization? In what ways are they different from one another?
Better: Define oligopoly. How does oligopoly differ from both perfect competition and monopoly in terms of number of firms, control over price, conditions of entry, cost structure, and long-term profitability?
Poor: Name the principles that determined postwar American foreign policy.
Better: Describe three principles on which American foreign policy was based between 1945 and 1960; illustrate each of the principles with two actions of the executive branch of government.
- Consider the layout of the question.
If you want students to consider certain aspects or issues in developing their answers, set them out in separate paragraph. Leave the questions on a line by itself.
- Write out the correct answer yourself.
Use your version to help you revise the question, as needed, and to estimate how much time students will need to complete the question. If you can answer the question in ten minutes, students will probably need twenty to thirty minutes. Use these estimates in determining the number of questions to ask on the exam. Give students advice on how much time to spend on each question.
- Decide on guidelines for full and partial credit.
Decide which specific facts or ideas a student must mention to earn full credit and how you will award partial credit. Below is an example of a holistic scoring rubric used to evaluate essays:
- Full credit-six points: The essay clearly states a position, provides support for the position, and raises a counterargument or objection and refutes it.
- Five points: The essay states a position, supports it, and raises a counterargument or objection and refutes it. The essay contains one or more of the following ragged edges: evidence is not uniformly persuasive, counterargument is not a serious threat to the position, some ideas seem out of place.
- Four points: The essay states a position and raises a counterargument, but neither is well developed. The objection or counterargument may lean toward the trivial. The essay also seems disorganized.
- Three points: The essay states a position, provides evidence supporting the position, and is well organized. However, the essay does not address possible objections or counterarguments. Thus, even though the essay may be better organized than the essay given four points, it should not receive more than three points.
- Two points: The essay states a position and provides some support but does not do it very well. Evidence is scanty, trivial, or general. The essay achieves it length largely through repetition of ideas and inclusion of irrelevant information.
- One point: The essay does not state the student's position on the issue. Instead, it restates the position presented in the question and summarizes evidence discussed in class or in the reading.
- Read the exams without looking at the students' names.
Try not to bias your grading by carrying over your perceptions about individual students. Some faculty ask students to put a number or pseudonym on the exam and to place that number / pseudonym on an index card that is turned in with the test, or have students write their names on the last page of the blue book or on the back of the test.
- Skim all exams quickly, without assigning any grades.
Before you begin grading, you will want an overview of the general level of performance and the range of students' responses.
- Choose examples of exams to serve as anchors or standards.
Identify exams that are excellent, good, adequate, and poor. Use these papers to refresh your memory of the standards by which you are grading and to ensure fairness over the period of time you spend grading.
- Grade each exam question by question rather than grading all questions for a single student.
Shuffle papers before scoring the next question to distribute your fatigue factor randomly. By randomly shuffling papers you also avoid ordering effects.
- Avoid judging exams on extraneous factors.
Don't let handwriting, use of pen or pencil, format (for example, many lists), or other such factors influence your judgment about the intellectual quality of the response.
- Write comments on students' exams.
Write brief notes on strengths and weaknesses to indicate what students have done well and where they need to improve. The process of writing comments also keeps your attention focused on the response. And your comments will refresh your memory if a student wants to talk to you about the exam.
- Strive to balance positive and critical comments.
Focus on the organization and flow of the response, not on whether you agree or disagree with the students' ideas. Experiences faculty note, however, that students tend not to read their returned final exams, so you probably do not need to comment extensively on those.
- Read only a modest number of exams at a time.
Most faculty tire after reading ten or so responses. Take short breaks to keep up your concentration. Also, try to set limits on how long to spend on each paper so that you maintain you energy level and do not get overwhelmed. However, research suggests that you read all responses to a single question in one sitting to avoid extraneous factors influencing your grading (for example, time of day, temperature, and so on).
- If you can, read some of the papers twice.
Wait two days or so and review a random set of exams without looking at the grades you assigned. Rereading helps you increase your reliability as a grader. If your two score differ, take the average.
- Place the grade on the last page of the exam.
This protects students' privacy when you return or they pick up their tests.
Returning Essay Exams
- Return exams promptly.
A quick turnaround reinforces learning and capitalizes on students' interest in the results. Try to return tests within a week or so.
- Review the exam in class.
Give students a copy of the scoring guide or grading criteria you used. Let students know what a good answer included and the most common errors the class made. If you wish, read an example of a good answer and contrast it with a poor answer you created. Give students information on the distribution of scores so they know where they stand.
- Use groups to discuss test questions.
Some faculty break the class into small groups to discuss answers to the test. Unresolved questions are brought up to the class as a whole.
- Get feedback from the class about the test.
Ask students to tell you what was particularly difficult or unexpected. Find out how they prepared for the exam and what they wish they had done differently. Pass along to next year's class tips on the specific skills and strategies this class found effective.
- Keep a file of essay questions.
Include a copy of the test with your annotations on ways to improve it, the mistakes students made in responding to various question, the distribution of students' performance, and comments that students made about the exam. If possible, keep copies of good and poor exams.
The Strategies, Ideas and Recommendations Here Come Primarily From:
Gross Davis, B. Tools for Teaching. San Francisco, Jossey-Bass, 1993.
McKeachie, W. J. Teaching Tips. (10th ed.) Lexington, Mass.: Heath, 2002.
Walvoord, B. E. and Johnson Anderson, V. Effective Grading. San Francisco, Jossey-Bass, 1998.
And These Additional Sources...
Brooks, P. Working in Subject A Courses. Berkeley: Subject A Program, University of California, 1990.
Cashin, W. E. "Improving Essay Tests." Idea Paper, no. 17. Manhattan: Center for Faculty
Evaluation and Development in Higher Education, Kansas State University, 1987.
Erickson, B. L., and Strommer, D. W. Teaching College Freshmen. San Francisco:
Fuhrmann, B. S. and Grasha, A. F. A Practical Handbook for College Teachers. Boston:
Little, Brown, 1983.
Jacobs, L. C. and Chase, C. I. Developing and Using Tests Effectively: A Guide for Faculty.
San Francisco: Jossey-Bass, 1992.
Jedrey, C. M. "Grading and Evaluation." In M. M. gullette (ed.), The Art and Craft of Teaching.
Cambridge, Mass.: Harvard University Press, 1984.
Lowman, J. Mastering the Techniques of Teaching. San Francisco: Jossey-Bass, 1984.
Ory, J. C. Improving Your Test Questions. Urbana:
Office of Instructional Res., University of Illinois, 1985.
Tollefson, S. K. Encouraging Student Writing. Berkeley:
Office of Educational Development, University of California, 1988.
Unruh, D. Test Scoring manual: Guide for Developing and Scoring Course Examinations.
Los Angeles: Office of Instructional Development, University of California, 1988.
Walvoord, B. E. Helping Students Write Well: A Guide for Teachers in All Disciplines.
(2nded.) New York: Modern Language Association, 1986.