Standardization News

It's More Than Multiple Choice

A new ASTM subcommittee is addressing standards for job capability tests.

Doubtless, few people would argue with Julie Timmcke on this point: "Imagine you're in a plane and your pilot says, ‘Hey, folks, I just passed my multiple-choice test on landing so I'm going to give it a go.' You would be aghast."

"Of course, a pilot's landing skills would never be tested with just a multiple-choice test," says Timmcke, executive director of the Performance Testing Council, Emmaus, Pa. But such a scenario points to the need for performance testing.

Professionals in numerous fields use performance, or hands-on, tests in real-life or simulated settings to evaluate the ability of recent graduates, job candidates and seasoned employees to demonstrate that they have mastered or retained the skills needed for a particular position.

Describing performance testing, Wallace Judd, founder and president of Authentic Testing Corp., Gaithersburg, Md., says, "It's when what you have to do to demonstrate your skills is the same thing you have to do on the job."

Some say the field continues to evolve as more and more professions are evaluating multiple-choice and related tests as compared to more interactive approaches to testing. "In very high stakes areas, performance tests are just a given," Timmcke says. "How else would you know the pilot could land a plane? They have to test them actually doing a landing."

There are standards and then there are standards

The growth of the field due to burgeoning demand from federal and state governments, insurance companies and international trade organizations for performance tests, as well as the need to ensure the quality of these tests, brought those working in the performance testing arena to ASTM International's door.

"There are standards for testing, and most of these standards are predominantly formulated in response to issues related to multiple-choice tests," Judd says. What are needed are standards specially designed to address performance testing, according to Judd and Timmcke.

"Standards exist in the testing industry, but in some areas there is a gap or a difference pertaining to performance items and tests that may not be addressed in current standards," Timmcke says. "These gaps leave an opening for interpretation. We want to have the clearest understanding of what standards apply to performance testing and what steps can be taken to meet those standards."

That's why Timmcke, Judd and the board of the PTC contacted ASTM International about developing performance testing standards, and that's what led to the formation of ASTM Subcommittee E36.80 on Personnel Performance Testing and Assessment as part of Committee E36 on Accreditation and Certification in 2009. Today the subcommittee has about 35 members.

As Judd says, "When you take a multiple-choice test, you can go back and forth over the questions. In a performance test, you can't do that. You can't put the branch back on the tree after you've sawn it off. With performance tests you have different issues than you have with multiple-choice tests."

Different approach

The E36.80 process began when the PTC - a member-run, nonprofit consortium dedicated to innovation in the design, development and delivery of performance testing - initiated a proposal process, contacting a few organizations for input and inviting them to present to the PTC board of directors on developing performance testing standards.

"Lately, meaning the past four to five years, communities with standards needs - whether a trade association, a federal agency or an industry-driven consortium - have become wiser consumers of standards development services," says Pat Picariello, ASTM's director of developmental operations. "Groups have become more savvy in selecting service providers/standards organizations, and they're issuing requests for proposals and hitting the Federal Register. The PTC was serious enough about doing this, and doing it right, that it contacted multiple organizations."

The PTC board evaluated options for collaborating on the development of standards and chose ASTM because of the experience and depth of process ASTM brought to the table, according to Timmcke, who is serving as secretary of the new subcommittee.

Many tests, many fields

"Developing standards for performance testing was a need our members have a strong interest in since most are doing or want to be doing performance testing," Timmcke says of the PTC.

The fields that use performance testing, according to Judd and Timmcke, are diverse, including but not limited to:

The subcommittee leaders expect that all of those fields and many more may benefit from standards.

The start

Subcommittee E36.80 organized formally in October 2009 through a virtual meeting. Judd chairs the subcommittee as well as the task group of 12 people that will propose recommendations to the subcommittee. The task group met for the first time in December 2009 to establish a structure and set up a calendar. Judd said the task force expects to discuss issues with the subcommittee in early 2010.

Currently, members have one work item in process: WK25593, Practice for Performance Testing and Assessment, which will provide guidance for performance test sponsors, developers and delivery providers for the planning, design, development, administration and reporting of high quality performance tests and assessments, and to stakeholders from the user and/or consumer communities for determining the quality of performance tests and assessments. The standard will include requirements, processes and intended outcomes for the entities that issue, develop, evaluate and deliver the test, users and test takers who interpret the test, and the specific quality characteristics of performance tests or assessments.

According to Judd, this and other standards will address a range of areas, including, but not limited to, random access, time limits, practice items and familiarity with the tools needed to take certain performance tests in order to ensure equal opportunities for all test takers.

The Future

The next steps for the subcommittee are to build its membership and hold virtual meetings to develop content. The subcommittee's timetable is fairly aggressive; it expects to introduce a standard for balloting by the second quarter of 2010.

Judd, a test development and psychometric analysis consultant who analyzes the effectiveness of tests, says that he hopes the ultimate outcome of the group's work will accomplish a number of things, including:

Performance testing standards, Judd says, will protect the consumer and the test sponsors. "Standards protect the sponsor by defining what reasonable practice is. Standards protect the testing candidate by making sure the test preparation and delivery give the candidate a fair chance to demonstrate his or her skills. Standards create an appropriate compromise," he says.

There's another benefit as well. "Without standards, a test developer would never know when adequate quality assurance was complete. Without standards, a developer would not know when the statistics are adequate to publish the test. When you have a set of standards, it improves commerce," says Judd.

For her part, Timmcke is confident ASTM's standards will make a difference in her field and for the PTC. "Standards for any subject matter should provide clarity and guidance for practitioners," she says.

Patricia Quigley is an award-winning journalist and public relations practitioner who has written for local, regional, national and international publications. She resides in southern New Jersey, where she earned a B.A. in communication and M.A. in writing from Rowan University.