The Art of Item Development (Writing Test Questions)

While many people can recite colors within the spectrum of light, and industry has created paints to match those colors, yet few can paint pictures (create art) like Michelangelo.

An artist can work years in order to refine their craft enough for it to have value. This is the conundrum of those who develop tests.

The Teacher Challenge

The teacher has the most difficult job in test development. Sitting alone, on a computer, without any formal training, attempting to write valid items is extremely difficult. Item development is very time consuming and difficult even for the expert. Writing test items for EMT PASS took an average time of 15 minutes per item. Writing valid items requires broad based knowledge over more than just the content of the test. It takes knowledge of the entire EMT level, the AEMT level, Paramedic level; in fact, knowledge of the entire EMS domain. Every instructor does not typically possess this breadth of knowledge.

Someone drafting items for a certification exam may know the “pitfalls” of item construction. They may have sample questions present for their review. But it is still possible to be incapable of constructing a valid item. In addition, the specific language used in the question must be consistent throughout the exam in order to ensure that the student understands it.

Bill has witnessed some of the nation’s best educators who drafted items for the NREMT examination that were unacceptable.

A spectator at a football game may witness a play and think, “That play was so simple, even I could run it!” If that spectator was to attempt the play, however, they would not perform as well as the player. The same holds true for test items: it looks like it would be easy to write a question once it’s been already written.

Test Item Expertise

Bill Brown arrived at the NREMT with solid item writing skills. Throughout his career he “practiced” using those skills.

He drafted items for and attended over 100 item-writing meetings in his career.

He acquired the art of item writing. When someone purchases an original piece of artwork, he or she wants to be assured that  the artist has experience, delivers an appreciated work, and that the cost can be justified. Quality has a price. Students and instructors can be confident that EMT PASS and AEMT PASS are applications that are like the original piece of artwork—developed with the highest of expertise and value.

Regarding his item writing experiences, Bill says, “I’ve been present when hundreds of item writers have submitted their test question drafts. Less than 20% of the items presented pass through the review process without several changes to the original. Even rarer is one that makes it without a single change. In fact, about 25% of the items submitted are discarded immediately upon review as being unacceptable.”

Participating in one or two item writing meetings does not make someone capable of writing items similar to those appearing on the NREMT exam. Just as a true artist is not born producing just one or two painting, a person cannot become an expert item writer after just one or two item writing sessions.

Because the complexity of item writing prevents it from being easily explained, Bill would be pleased to further explain or demonstrate the art of item writing as a part of his consulting services. (hotlink to Web education consulting).