Practice Exchange

Ilene Gast, Associate Editor

This column highlights innovative assessment programs of interest to assessment professionals. If you are conducting a project that would interest the ACN readers, or if you know someone who is, please let me hear from you. I can be reached by phone at (202) 305-0590, fax at (202) 305-3664, or electronic mail at ilene.f.gast@justice.usdoj.gov.

The Beauty of Performance Measurement

by Charles Brooks

I will get right to the point. To apply any selection strategy correctly, you must begin with that which you are trying to predict - performance. This article describes some performance measurement research we conducted for the Division of Child Support Enforcement (CSE) in Georgia.

Our original intention was to use our performance measures to demonstrate the concurrent criterion validity of a test of reading and arithmetic that we developed for entry-level CSE Agents at the customer's request. We were able demonstrate validity quite nicely. We obtained validity coefficients ranging from .25 to .47 depending upon the specific dimension of performance in question.

Performance measurement, however, has uses that reach far beyond demonstrating the validity and utility of tests. With the "right" performance measures, you can demonstrate employee productivity and economic value resulting from other types of government/organizational programs, initiatives, and practices. I will talk more about the "right" performance measures later.

The First Step: the CSEA Performance Analysis

We operationalized the strategic objectives of the Division of Child Support Enforcement) as the amount of money collected by the agents. CSE agents have the job of collecting child support payments from absent parents. We selected the amount of money collected by CSE agents as our performance measure based on the results of an extensive analysis of their performance. (I will be happy to supply more information about the specifics of this analysis upon request.) Amounts collected by the agents ranged from about $172,000 to $1,700,000 with an average of $812,000. Consistent with the findings of Schmidt, Hunter, and Judiesch (1990), we found that top performing agents collect 42% more than average agents (SDp = $341,000). As you will see, we used this performance measure to demonstrate the worth of a variety of personnel management programs.

Applications of the CSEA Performance Analysis

A Comparison of Selection Processes.

Since the enactment of the Merit System Reform Act in Georgia, state governmental agencies are not required to use our selection services. Before the reform took place, we had implemented a new assessment of customer service and interpersonal skills, a video-based multiple-choice test (the Diplomat marketed by Ergometrics). Although we had achieved moderate concurrent validities (zero order r = .28 to .34) against supervisors' ratings of task performance, CSE chose to discontinue that test in favor of a system of their own design.

About two years after CSE discontinued use of the Diplomat assessment, we conducted a predictive validity study. Our criteria included the dollar value of collections and supervisors' rankings of agent's contribution to achieving agency objectives. The raw correlations between test scores and these two criteria were .45 and .48 respectively. The Diplomat was virtually uncorrelated with the reading and arithmetic test (r = .04). By combining the two tests, we achieved a multiple correlation of .70.

Furthermore, in our performance analysis, we were able to show that each of those agents hired using the Diplomat collected an average of $200,000 more agents selected by other means. Although the organizational culture of CSE did not value on customer service and interpersonal skills, (management preferred "tough types"), we demonstrated showed how valuable these skills and the test measuring them were to the children of Georgia. The agency collected $10 million more due to that test. Over 40% of that sum would have been paid directly out of the Georgia State's checking account in the form of welfare payments had the money not been collected from the absent parents. Not only was $10 million collected, but it also saved the state over $4 million in welfare payments.

We are not surprised that CSE has suddenly reconsidered the use of the video-based test; it pays off for the children! They are also more eager to heed our counsel regarding selection and productivity issues in general.

The Cost of a Software Glitch.

CSE purchased a multi-million dollar software program as part of re-engineering the CSE Agent job. The program was supposed to perform arithmetic calculations. It malfunctioned, and the Agents spent an average of 18% of their time using a calculator to do those calculations. We were able to show that the software error cost the state over $30 million dollars in salaries and lost collections in one year. Our analysis helped CSE to obtain a sizable refund from the vendor and to hire programmers to fix the problem with the software.

The Cost of Turnover.

Our analysis also enabled us to put a dollar value on turnover. We were able to show that for each agent who left, recruitment, training, and diminished productivity cost an average 120% of annual salary, plus over $676,000 in lost collections. Turnover at CSE is fairly high, and between the cost of hiring and retraining new agents and the lost revenues, the cost that added up to over $70 million annually. A new selection process, which is under development, will greatly reduce turnover.

Benefit of Better Selection.

We are in the process of developing a competency-based behavioral interview protocol to complete our selection process for CSE Agents. We will combine the interview protocol with the reading and arithmetic test, the Diplomat, and a writing sample test. Based on our own findings and on projections published in the research literature, we estimate that the new selection process will have a 70% accuracy rate in hiring top performers (i.e., agents who perform at or above the current level of one standard deviation above the mean). We expect to hire about 100 agents in the year following implementation of this process. Recall that top performers collect 42% more than average agents. That should translate to over $24 million annually in increased collections and salary savings.

The Bottom Line - You Do the Math . . .

Our research and the development of a customized selection process cost less than $160,000. These costs consist mostly of analysts' and subject matter experts' salaries and travel expenses. Add up the benefit figures we described in the preceding section. If we are only half as successful as is potentially possible, the benefits of the selection process will exceed $62 million. Not bad!

Epilogue: Developing the Right Performance Measures

Performance measurement begins with understanding the organization's mission and strategic objectives. If these are not clear, then it is not clear what you are trying to predict. Therefore, take steps to clarify and understand the organization's mission, strategic plan, and strategic objectives. Use these to develop strategic measures to determine whether the objectives - the results your organization is trying to achieve - are being accomplished. Focus on efficiency, effectiveness, outcomes, and results. Do not worry about input and process measures unless the other types of measures are impossible to obtain.

In defining performance objectives, begin with the top level of the organization. Then, cascade the objectives and measures down to each successive organizational level proceeding down to the level of work teams and individual contributors. Done correctly, this cascading process takes about four months. If you are not sure how to cascade performance objectives, hire a consultant. It is critical that this step be done right. Otherwise, you truly are operating in the dark, and you may, in fact, harm the organization.

Make sure any performance measures that you develop support overall organizational objectives. Be careful when specifying the measures because the measures will greatly influence what people do on the job. If the wrong measures are specified, the wrong results are obtained. In short, when it comes to program and process evaluation, including selection processes, "you get what you measure."

Reference

Individual differences in output variability as a function of job complexity. (1990). Hunter, John E.; Schmidt, Frank L.; Judiesch, Michael K. Journal of Applied Psychology, 75, 28-42.

Charles Brooks is a Senior Human Resources Consultant in the Staffing and Productivity Solutions Team of the Georgia Merit System. He is also an Honest-to-God Living Legend of the Blues in Atlanta, GA. For more information, he may be reached at 404-657-2143 or bro@gms.state.ga.us.