Practice Exchange

Ilene Gast, Associate Editor

This column highlights innovative public sector programs at the Federal, state, and local level. If you are conducting a project that would interest the ACN readers, or if you know someone who is, please let me hear from you. I can be reached by phone at (202) 305-0590, by fax at (202) 305-3664, or electronic mail at ifgast@aol.com.

Update: INS Begins Using Competency- based Assessments For Promoting Border Patrol Supervisors and Managers

by David Pollack, Greg Beatty, and Paul Usala

Overview of the Process

The U.S. Immigration and Naturalization Service (INS) has recently begun using its new competency- based promotional assessments for GS-11 through GS-15 supervisory and managerial Border Patrol Agent jobs. The procedure includes four assessments: an evaluation of past achievements, a decision-making exercise, a multiple-choice in- basket exercise, and a writing exercise.

The new procedure will immediately replace the traditionally-used rating and ranking method for filling supervisory and managerial vacancies in four Border Patrol Sectors. The assessments will be administered again in June, and after candidates participating in this phase have received their feedback, assessment scores will be used also to fill vacancies in four additional Border Patrol Sectors. Finally, the last group of candidates will be assessed in the fall of 1997, and then assessment scores will be used to fill all supervisory and managerial vacancies in the Border Patrol. This will include staff officer positions in regional offices and at INS Headquarters.

Initial Implementation

The initial implementation of the competency-based assessment process went extremely well. A total of 876 candidates completed all four of the assessments and have received detailed feedback about their strengths, weaknesses, and performance relative to other candidates at the same grade level.

The candidates who participated in the initial implementation represented all of the Border Patrol's 21 sectors and 24 of INS' 33 districts. In addition, INS Headquarters and three regional offices were represented. Assessments were administered in 42 locations across the country. Candidates from as far away as Agana, Guam; Havana, Cuba; Shannon, Ireland; and Rome, Italy participated in the process.

Of the 876 candidates who completed all four assessments, 781 (89%) are currently in the Border Patrol. The other 95 candidates work in other programs in INS and are eligible to compete for supervisory and managerial vacancies in the Border Patrol.

Overall, candidates scored very well on the assessments, reflecting the high skill level of INS employees and the rigor of the INS entry-level assessment process. Candidates were assigned to one of three score bands based on their performance on the assessments. Over 75% of candidates reached the two highest score bands, indicating potential for excellent performance as a Border Patrol supervisor or manager.

Candidate Feedback

Candidates who completed the assessments were given a brief survey with questions about their perceptions of the new process. The survey was designed so that candidates could provide their input as to the strengths and weaknesses of the system. About 150 individuals chose to respond to the survey.

Overall, candidates appeared to be cautiously optimistic about the new assessment process. Only 29% of candidates prefer the traditional rating and ranking method, and 29% chose to withhold judgment until after the new process has been used to fill vacancies. The remaining 42% of the candidates who responded to the survey explicitly endorsed the competency-based method for developing Best Qualified lists over the traditional method.

Candidates also provided feedback about specific parts of the assessment process. A minority of candidates indicated that they would prefer more time to complete the Past Achievement Record. A few comments also indicated difficulties with some aspects of the assessment facilities in a handful of locations. In the next phase of the competency- based assessment process, every attempt will be made to provide candidates with more than adequate time to complete the Past Achievement Record. Also, alternative assessment facilities are being identified to replace those facilities that candidates described as anything less than optimal.

Future of the Competency-Based Assessment Process

A parallel competency-based assessment process for the Adjudications Officer, Criminal Investigator, Deportation Officer, and Immigration Inspector occupations is under development and will be implemented during FY98. This process will parallel the Border Patrol system and will ensure that all INS Officer Corps supervisors and managers are promoted in a fair and objective manner.

For further information about the development, validation, and implementation of competency- based promotional and diagnostic assessments, please contact Magda Colberg, David Pollack, or Greg Beatty at the Research and Development Division of INS (202) 305-0600.

Return for Investment Model Guides Evaluation of INS Competency-based Assessments

by Ilene Gast

INS has developed a "Return for Investment Model" to guide its personnel assessment staff in determining the returns?monetary and non-monetary that result from the implementation of new personnel selection systems. The model specifies broad categories of costs incurred in producing the new system. Such costs include research and development for the assessment procedure, implementation expenses such as production, administration, and documentation, administrative oversight of the new procedure, and program evaluation.

The model also outlines potential payoffs of the new system. One form of payoff can be expressed in dollars. Dollar-valued returns can be expressed as selection utility, or the monetary value of improved performance attained through test use. It may also be possible to assign a dollar value to the increased efficiency of the new system; savings can be realized as a result of a decrease in staff time required to operate a testing system. Similarly, dollar value returns may result from increased effectiveness of a system. For example, an organization may experience a decrease in candidates' requests for information about the selection process. Reduced staff time in handling such requests translates directly into dollars.

Equally important are benefits that are not easily expressed in dollar terms. Many new assessment programs fail despite their ability to demonstrate a cost savings through improved selection. Rather, failure can often be attributed to difficulty in implementation or lack of acceptance by decision makers. Therefore, the model addresses the perceptions of direct customers (e.g., test takers, selecting officials) and indirect customers (e.g., prospective co-workers, members of the public). For example, the perspective of selecting officials can be gained through focus groups. Customer satisfaction surveys can be used to assess candidate satisfaction with test materials and procedures. Organizational climate surveys can be used to obtain the perspective of employees who are indirectly affected by the new system. One would begin by taking a pre-implementation measure of relevant attitudes, such as satisfaction with the professionalism and competence of individuals in the group that will be hired or promoted through the new system. Once a baseline has been established, subsequent administration of such survey items might be used as an indicator of program effectiveness. Finally, it is important to gauge public reaction to the new procedure by monitoring news media and public inquiries.

INS plans to apply the Return for Investment Model to all its assessment programs so that the Research and Development Division can compare new assessment systems to preexisting systems. The model is receiving its first test with the implementation of the promotional assessment system described in the preceding article. As part of the study, the project staff has developed a more detailed protocol that specifies all variables to be assessed. For more information, contact Ilene Gast at the Research and Development Division of INS (202) 305-0590.