0.3 MB
Key Findings
Two of the three admissions factors used by the U.S. Air Force Academy (USAFA) — the academic composite and leadership composite — predict outcomes of interest (e.g., graduation and promotion down the line).
But higher scores given by a panel of raters at USAFA who provide an overall rating of applicants (i.e., selection panel scores) are associated with a lower likelihood of success in many of the outcomes, which argues for removing selection panel scores from the selection composite formula and readjusting the formula accordingly.
The U.S. Military Academy's (USMA's) whole candidate score (WCS) — a composite of academic score, physical aptitude exam score, and community leadership score — has a statistically significant relationship with both USMA graduation and probability of promotion to O-5 in the Army. Thus, using WCS as the primary basis for admission decisions is warranted.
The U.S. Air Force Academy (USAFA), the U.S. Military Academy (USMA), and the U.S. Naval Academy (USNA) provide undergraduate education and training for students who will become commissioned officers in the Armed Forces. Like all institutions of higher education, military academies rely on a set of admission criteria, but beyond that, they are particularly concerned about the outcomes of those admitted, because the goal is for those who graduate to move on and become successful as commissioned officers.
RAND was asked to examine admission standards at these academies to see whether they were enrolling individuals who would both graduate andbecome successful officers, but the USNA chose not to participate. Thus, this brief summarizes the results of analyses of the applicant scoring systems that USAFA and USMA use for admission.
Assessment of USAFA Admission Standards
Context. The USAFA admissions process is highly competitive. Although the entire admissions process takes many months and includes many factors, final decisions are based on scores calculated as a weighted combination of the three elements in Table 1. The computed total score for a candidate — called the selection composite score — is used to rank-order candidates and determine admissions.
Table 1. Elements of USAFA Admission Decisions
Element
Description
Percent Weight
Academic Composite
Consists of SAT/ACT score and prior academic ranking (high school class rank, grade point average [GPA], transcript, strength of high school, rigor of curriculum)
60
Leadership Composite
Called the extracurricular composite; includes activities, leadership, and résumé
20
Selection Panel Score
Consists of the selection panel's evaluation of the candidate, which includes a review of admissions liaison officer (ALO) evaluations, writing samples, teacher evaluations, recommendations, and the candidate fitness assessment
20
This selection formula has been in place for decades and is designed to yield a class of students highly respected both in the Air Force and in the educational community, but can the formula be improved? The study begins to answer that question by examining how well the three elements in the table predict outcomes that matter to the Air Force.
The key outcomes of interest for performance at USAFA are GPA, failure to graduate for academic reasons, failure to graduate because of a desire for a career change, military performance average (MPA), and overall performance average (OPA). In terms of future performance, key outcomes of interest are promotion to O-4, O-5, and O-6. To explore relationships between admission factors and outcomes, researchers relied on statistical regression techniques using data on the nearly 35,000 cadets who attended USAFA from 1980 to 2011 that included records from three data sources: USAFA registrar admissions records, USAFA cadet records, and Air Force personnel records.
Findings. Two of the three existing admissions factors are useful in predicting outcomes at USAFA. The overall selection composite and some elements within it (including academic composite and leadership composite) are significant predictors of failure to graduate for academic reasons, failure to graduate because of choosing a career change, GPA, MPA, and OPA. High school class rank was also a significant predictor of graduating versus failing for academic reasons and of GPA, MPA, and OPA — but it was not a significant predictor of graduating versus choosing a career change. However, the selection panel score, an element of the overall selection composite, does not work as anticipated. It was statistically significant, but in the wrong direction: Higher scores were associated with a lower likelihood of success in many of the outcomes.
Recommendation. Based on the analyses, the Air Force and USAFA leadership should consider removing the selection panel score from the selection composite formula and instead use it solely to screen out people the selection panel identifies as problematic. Further, the researchers recommend adjusting the formula from Table 1 as shown in Table 2.
Table 2. Recommendation for Adjusting the Selection Composite Formula for USAFA
Element
Existing Percent Weight
Adjusted Percent Weight
Academic Composite
60
75
Leadership Composite
20
25
Selection Panel Score
20
0
Assessment of USMA Admission Standards
Context. As with the USAFA, admission to the USMA is highly complex. Researchers focused on whether applicants with better results on USMA's whole candidate score (WCS) have a greater probability of graduating and whether applicants with higher WCSs have a greater probability of remaining in the U.S. Army and being promoted early to an O-4 pay grade and then to O-5.
Researchers analyzed data from USMA applicants for graduation years 1992 to 2001 to identify whether the application scores can predict these outcomes. The elements that make up a WCS and their relative weights in the formula are shown in Table 3. As was true in the USAFA analyses, the researchers used logistic regression models to examine whether and how the score elements are related to the outcomes of interest.
Table 3. Elements of the USMA Whole Candidate Score
Element
Description
Percent Weight
Academic Score
Composite of college entrance examination rank score (based on SAT or ACT scores) and a converted high school rank score
60
Community Leadership Score
Composite of the athletic activities score, the extracurricular activities score, and the faculty appraisal score
30
Physical Aptitude Exam Score
Candidate fitness assessment
10
Findings. While important, a WCS is not the sole determinant of admission, even if the applicant is deemed academically, physically, and medically qualified. Many applicants who were not offered admission had higher WCSs than applicants who were. It is unclear whether this results from judgments about the essay component of applications that was not available to the researchers, class composition goals that USMA strives to meet, or some combination of both.
Still, the WCS has a statistically significant relationship with both graduation from the USMA and probability of promotion to O-5 in the Army. All subelements of the WCS — except the converted high school rank score — make significant contributions to predicting graduation. However, the converted high school rank score is significant in predicting only early O-4 promotion and O-5 promotion. Both the candidate fitness assessment and the athletic activities score are statistically significant predictors of graduation. The candidate fitness assessment is also significantly associated with higher probability of promotion to O-5. Academics and fitness are both important criteria in selecting candidates.
Also, the magnitude of the relationships observed are likely underestimates because of selection effects — we cannot observe the graduation and promotion outcomes for applicants not admitted to USMA, for those admitted to another service academy, or for those admitted to a Reserve Officers Training Corps (ROTC) program (in the case of promotion outcomes). Unlike with USAFA, researchers did not find that a different weighting or combination of the WCS components could improve the selection process. Thus, using WCSs as the primary basis for admission decisions at USMA is warranted.
Recommendation. Based on the analyses, the Army and USMA leadership should continue to use the WCS as the primary basis of admissions decisions.
Conclusions
U.S. military academies aim to graduate those who attend and ensure they become productive commissioned officers. There are substantial costs if those outcomes are not achieved — both a monetary cost in educating an individual who does not move on and an opportunity cost in that someone who could have moved on was not selected. The RAND team's rigorous analyses of the selection processes in the USAFA and USMA show that those processes do predict the relevant outcomes of interest.
No comments:
Post a Comment