Fire Figher Biodata: Summary of Validity and Fairness
Page 3 of 3 (Quick jump - 1, 2, 3)


Fairness Analyses


Fairness analyses must be conducted in the cross-validation sample (N = 365). This is necessary because over-fitting occurs through the predictor development process, and this overfitting distorts relationships between predictor scales and criterion measures in the derivation sample, making it inappropriate to conduct fairness analyses in that sample. The cross-validation sample contained insufficient samples of ethnic subgroups (i.e., African-Americans, Caucasians, and Hispanics) to support separate fairness analyses. Therefore, ethnic subgroups were combined to create minority and non-minority groups, which were used for purposes of fairness analyses. The frequency distribution for minority status within the cross-validation sample is provided below.



Correlational analyses were used to examine the relative relationships between (1) the criterion measures and (2) the biodata predictors with respect to minority status. As shown below, correlations between the biodata predictor scales and minority status are roughly parallel to the correlations between the corresponding criterion measures and minority status. For example, the biodata predictor of physical performance (BIO_PHYS) shows a low negative correlation with minority status (r = -.07), while the corresponding physical performance criterion (CRITFAC1) also shows a low negative correlation with minority status (r = -.09).

The pattern is similar for the biodata predictor of cognitive performance (BIO_COGN), which shows a low negative correlation with minority status (r = -.13). The correlation between the corresponding cognitive performance criterion (CRITFAC2) and minority status also shows a low negative coefficient (r = -.11). All of these negative correlations mean that non-minorities tend to score somewhat higher on these criteria and predictor scales compared to minority fire fighters.

The pattern is different for the biodata predictor of teamwork performance (BIO_TEAM), which shows a statistically significant positive correlation with minority status (r = .14, p < .01). The corresponding teamwork performance criterion (CRITFAC3) shows a near zero correlation with minority status (r = .03). This means that the biodata predictor for teamwork performance tends somewhat to favor minority fire fighters.

For purposes of fairness analysis, a more reliable measure was computed for overall fire fighter job performance. This measure (ALLCRIT) was computed as an equal-weighted sum of the three principal components of fire fighter job performance. The resulting overall performance criterion shows a very high correlation (r = .83, p < .001) with the single item rating of overall performance (OVERALL). In addition, the overall biodata predictor (ALL_BIO) shows the same cross-validity (r = .27, P < .001) with the computed criterion (ALLCRIT) as with the single item rating (OVERALL). Nonetheless, this computed criterion offers enhanced reliability for purposes of the fairness analysis.

Compared to the discrete biodata predictor scales, a different pattern is seen in correlations by minority status at the level of overall performance. Whereas the overall performance criterion (ALLCRIT) shows a low negative correlation with minority status (r = -.10), the overall biodata predictor composite (ALL_BIO) shows near zero correlation with minority status (r = -.03). This means that although minorities tend to be somewhat lower on the overall performance criterion, minority and non-minority fire fighters tend to score about equally on the overall biodata predictor.



Four separate regression analyses were conducted within the cross-validation sample to further assess fairness for the biodata predictors with regard to minority status. In these analyses, regression lines were compared for minority and non-minority groups for each of the four biodata predictors (BIO_PHYS, BIO_COGN, BIO_TEAM, and ALL_BIO) with the appropriate criterion used as the dependent variable (CRITFAC1, CRITFAC2, CRITFAC3, and ALLCRIT respectively).

The regression analyses revealed non-significant intercept differences and non-significant interactions (p > .05) for minority and non-minority treatment groups, leading to conclusions that common regression lines are most appropriate for minority and non-minority fire fighters for each of the four biodata predictors in relation to their corresponding criteria. As such, these analyses support the fairness as well as the validity of the biodata predictors for fire fighter job performance.

These results also provide evidence of construct validity for the biodata predictors. This evidence is seen by the low or negative correlations of the biodata predictors with off-diagonal criteria. For example, the biodata predictor for physical performance shows a .39 validity against the physical performance criterion (CRITFAC1), but has essentially zero correlation with the cognitive performance criterion (CRITFAC2, r = .00), and it shows a negative correlation with the teamwork criterion (CRITFAC3, r = -.18). Similar results are shown for the other two biodata predictor scales. This discriminant validity evidence supports the construct validity of the three biodata predictor scales. This means that each of the three biodata scales effectively predicts what it is intended to predict, and it does not predict other distinct components of fire fighter job performance, with which it should be unrelated.






Privacy Policy
Copyright © 1994 to 2018 e-Selex.com, San Diego, California. All rights reserved.

Selex, e-Selex, Selection Excellence, PreSelex, SkillSelex, JobSelex, WEBselex,
Selex Success, and SmartSelex are registered trademarks of e-Selex.com.
The Intelligent Choice, ACE, and Authorized Consulting Expert are trademarks of e-Selex.com.