The Use of Valid and Secure Test Questions in Education
Using valid and secure test questions as a part of education serves two purposes:
- measuring student knowledge and
- supporting re-education when weaknesses are discovered.
What makes a test valid and secure
A well-constructed test must remain secure if it is to have “test validity”. The test must measure either achievement or particular criteria. In a classroom test, for example, test validity means the student who best understood the information (without cheating) would obtain the highest score. This also means that this student gained the knowledge required to progress in the course. An invalid exam is one that:
- has been compromised (the students know the questions on the test prior to taking it)
- produces a score that is meaningless: a) all students scored above 95% correct, b) the exam was too short, c) the items were not at the EMT level, and/or d) the exam included many distractors that no student chose.
Failing to secure test questions compromises the test validity.
This can happen when the test or individual test questions are distributed to students to review prior to the test being administered. When students share test questions with other students, this further destroys test validity, skews test scores, and discredits test measurements.
Tests that are too easy do not measure anything; therefore, they are not considered valid because they don’t separate the students who have gained the knowledge required from those who are poorly prepared. An invalid test does not challenge students; therefore, they have no measurement value.
Short exams, such as those containing only twenty questions, do not cover enough content and “guessing” the right answer increases the likelihood of passing.
Tests that include questions geared to another EMS level, (for example AEMT), are not valid for EMT students. Using distractors requiring knowledge from another level (wrong answers), however, is acceptable.
In order for a test question answer to have measurement value, it must be chosen by some percentage of the students. There is no defined percentage that must choose the wrong answer, but if 100 or more students never pick one of the possible answers, then it is an invalid choice.
Standard setting in test validity
When scores are spread, the exam mimics an achievement exam (highest score equals highest achiever). The most commonly known achievement tests are the ACT and SAT exams used as a profile for college entrance. This is how most teacher-produced exams are scored. Another exam form is criterion referenced, in which the exam possesses a meaningful “cut-score,” based on particular criteria and has been built using a standard setting process. The means the cut score (pass/fail score) is the standard that each student must attain to pass.
Standard setting is conducted via statistical analysis combined with “judgment.” For example, a teacher may write a test and decide that students must get 70% of the items correct to pass. In this case, one person set the criterion, which represents the lowest form of validity in standard setting. If this same instructor asks other teachers to review the exam and set a score, more “judges” are involved in the standard setting of that exam. Obviously, the more reviewers involved in meeting a criteria (pass/fail score), the more likely the judgment will be correct/valid.
Most instructors lack adequate item statistics, so poorly-performing items end up on their tests over and over again. In some cases, teachers are “mandated” to follow college-prescribed criteria: for example, 100% to 90% equals an “A.” Using college-mandated scores is taxing on teachers because they must calibrate each item based upon its difficulty to assure excellence over the knowledge base obtained in order to receive an “A.” What is more troubling is when students fall right at the line of either pass or fail. And setting expectations for passing the “first” block test are/should be different than the “last” block test.
The assumption is that a student’s breadth of knowledge increases as he/she progresses through a course. When teachers use the school-mandated grading system, the complexities increase, and the student’s knowledge is nearly impossible to calculate. This is even more difficult for the student taking the test because the range of scores is so large (30 items incorrect in a 100-question test).
If the criteria are set correctly, the students will “bunch up” around the cut score and nearly all will receive a “C” grade—not a very fair or accurate outcome.
Another complication with the achievement–type exam is when the test has more “moderate” questions than “easy” and “difficult” questions, the outcome will likely be a high score of 70%. If grades are awarded using a school-mandated % grade, then only one or two students will pass.
There are also methods for grading on a curve or “smoothing” a test but these too are complicated. Regardless of the type of test an instructor prepares (criterion or achievement), the teacher is in a difficult position to develop a truly valid exam.
Understanding testing, the creation of test questions, and how decisions are made regarding pass/fail is complex. Completely understanding testing concepts through this short and simple essay is not possible. There are ways to improve classroom testing and measurement, however, and Bill is willing to consult with teachers or EMS program faculty who are having difficulty regarding testing. Please review this hotlink regarding his services.
Using examinations to help students learn
Test items are also used for review. Instructors should discuss items that are missed by high achievers in the class. There are three reasons high achievers answer test items incorrectly:
- The question was flawed
- The instructor failed to properly teach the information
- The student misunderstood the information.
When questions are flawed, it is necessary for the instructor to reconstruct the question before it is used again on another test.
Flawed items can be identified when reviewing items missed by high achievers during examination reviews that occur during class time.
High achievers are those who are in the top 1/3 of the class according to rank.
Rank score is determined by placing the entire class from highest score to the lowest. If high achievers miss a test question a discussion should occur in class regarding that question. Questions missed by low achievers do not need to be discussed.
During class discussion of identified questions the instructor can either re-teach the content the test question covers and/or explain the meaning of the question to the students. After the discussion the teacher then should make a determination whether he/she feels the question needs to be rewritten or the subject matter from which the question arises needs more in-depth teaching in subsequent classes. Development of quality test questions is a teacher responsibility and likewise it is a teacher responsibility to teach the material that will be on the test. It is inappropriate to host a “test content review” a few days prior to the test. These commonly used “test reviews” provide false and inaccurate feedback on student performance.
Test-taking skills
Improving test-taking skills is one of the goals of EMT PASS and AEMT PASS.
Lessons on test-taking skills are included in each of these programs. Test-taking skill is like any other skill; it must be practiced. When teacher-made tests are not valid, the measurement value of the test is inadequate and the skill of taking a test cannot be practiced. Repetition is the key to skill acquisition. When teachers do not have adequate testing instruments available to them, their students can learn test-taking skills via EMT PASS and AEMT PASS. As with any skill acquisition, mistakes will be more frequent at the beginning. Students become more comfortable with skills over time. EMT PASS and AEMT PASS may be uncomfortable for the student at first, but as their test-taking skills improve, so will their performance (as long as they have the corresponding knowledge base).
Learning by studying test questions alone is not recommended and will not make the student a competent EMT or AEMT.
Reading the course books, attending lectures, being involved in study groups, and finding a mentor in EMS are all excellent and recommended ways for learning how to be a competent entry-level EMT or AEMT.
Exams with properly-constructed test items with a range of item difficulty and used with a proper test plan is the best method for building student confidence and success; they are particularly valuable for struggling students. Every student however, will benefit from a properly-constructed review program, such as EMT PASS and AEMT PASS.
