Hello There, Guest!  

[ODP]Putting Spearman’s Hypothesis to Work: Job IQ as a Predictor of Employee Racial

#31
Any update?

Bryan
 Reply
#32
I have more comments, especially with regards to the theoretical framing. However, I've been too busy to devote time to this. I should have more time after the 1st July.
 Reply
#33
(2016-Jun-29, 05:16:16)bpesta22 Wrote: Any update?

Bryan

I will review it. Please let me some time. Thanks. (I have been busy with my own article, among other things...)
 Reply
#34
Jensen's method (correlated vectors)
When one has multiple, preferably many, indicators of a latent trait one can use Jensen's method to check whether this latent variable is related to a criterion variable or if its the other variance components (group factors or test/item specificity). This method requires that one has multiple indicators each of which have some measurement of how well they measure the latent trait. Usually factor loadings from factor analysis are used for this purpose, but if one has item-level data, one should use item response theory-based discrimination values instead.

It is true that jobs are basically mental tests of varying difficulty. SH for this kind of data would be the claim that the jobs where the g-job performance link is stronger would show larger group differences in job performance, assuming no other effects (such as selective hiring which is ubiquitous). However, hat is not the kind of data this study has. The data here are the racial proportions of each job and some information about the jobs. One cannot frame the current study as a test of SH.

However, I still think the present study is useful and I have no serious criticism of the methods used, but it's a study of something else. It's a study of what happens when one has groups with different mean ability levels and there are jobs that select for different levels of cognitive ability. In the absence of differential hiring, there should be more of the higher scoring groups in the jobs that recruit from higher up the cognitive ability distribution. Of course, we know that there is some differential hiring and using IQ tests without doing this may be illegal (but not in the military).
Preferably, one should try to model the racial proportions together based on demographic data and assumptions about the range each job recruits workers from. However, the authors prefer to take a simpler approach and check the simpler prediction that the jobs with higher means do have more persons from the higher ability groups. This is fine with me.

Write-up of regressions
The write-up of the additional regression analyses is not satisfactory. These regressions were not based on any pre-analysis hypothesizing as the writing says. They were entirely exploratory, post hoc regressions and should be clearly labeled as such. To do otherwise, is to HARK (http://psr.sagepub.com/content/2/3/196.abstract). I would write something like:

A reviewer* suggested using multiple regression by including the jobs' ratings for whether they involve working with people and things. This was done because there may be group differences in these preferences which may obscure the relationship to the mean levels of cognitive ability. A regression model was fit for each race% as the outcome and with the mean job IQ, people-interest rating and person-interest rating as the predictors. [3 tables of results] The mean IQ was a good predictor in all the models and, importantly, the small correlation seen for Asian% seemed to be due to a suppression effect from a confound with a relatively stronger preference among Asians for working with people.

I'm sorry, but the framing of the study has to be changed. This is not a test of Spearman's hypothesis.

* I usually name reviewers who give useful suggestions.
 Reply
#35
Emil, thanks for these comments.

(2016-Jul-04, 04:29:07)Emil Wrote: Jensen's method (correlated vectors)
When one has multiple, preferably many, indicators of a latent trait one can use Jensen's method to check whether this latent variable is related to a criterion variable or if its the other variance components (group factors or test/item specificity). This method requires that one has multiple indicators each of which have some measurement of how well they measure the latent trait. Usually factor loadings from factor analysis are used for this purpose, but if one has item-level data, one should use item response theory-based discrimination values instead.

It is true that jobs are basically mental tests of varying difficulty. SH for this kind of data would be the claim that the jobs where the g-job performance link is stronger would show larger group differences in job performance, assuming no other effects (such as selective hiring which is ubiquitous). However, hat is not the kind of data this study has. The data here are the racial proportions of each job and some information about the jobs. One cannot frame the current study as a test of SH.


SH: The more x requires g, the larger the race difference on x.

In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).

In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).

I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.

Quote:However, I still think the present study is useful and I have no serious criticism of the methods used, but it's a study of something else. It's a study of what happens when one has groups with different mean ability levels and there are jobs that select for different levels of cognitive ability. In the absence of differential hiring, there should be more of the higher scoring groups in the jobs that recruit from higher up the cognitive ability distribution. Of course, we know that there is some differential hiring and using IQ tests without doing this may be illegal (but not in the military).
Preferably, one should try to model the racial proportions together based on demographic data and assumptions about the range each job recruits workers from. However, the authors prefer to take a simpler approach and check the simpler prediction that the jobs with higher means do have more persons from the higher ability groups. This is fine with me.


Write-up of regressions
The write-up of the additional regression analyses is not satisfactory. These regressions were not based on any pre-analysis hypothesizing as the writing says. They were entirely exploratory, post hoc regressions and should be clearly labeled as such. To do otherwise, is to HARK (http://psr.sagepub.com/content/2/3/196.abstract). I would write something like:

A reviewer* suggested using multiple regression by including the jobs' ratings for whether they involve working with people and things. This was done because there may be group differences in these preferences which may obscure the relationship to the mean levels of cognitive ability. A regression model was fit for each race% as the outcome and with the mean job IQ, people-interest rating and person-interest rating as the predictors. [3 tables of results] The mean IQ was a good predictor in all the models and, importantly, the small correlation seen for Asian% seemed to be due to a suppression effect from a confound with a relatively stronger preference among Asians for working with people.

I'm sorry, but the framing of the study has to be changed. This is not a test of Spearman's hypothesis.

* I usually name reviewers who give useful suggestions.

Obviously, the harked analyses weren’t in the original submission. We harked to “Hopefully Appease Reviewer Kirkegaard” (a double hark!?).

The analyses you suggested never occurred to us as we submitted the original paper here. So, adding the analyses had to be post hoc and exploratory. In fact we start discussion of this by stating “A reviewer recommended we explore….” I think that language makes it patently obvious that the analyses to follow are post hoc. We should add more disclaimers?

It occurs to me that “harking by reviewer” is interesting, and is likely an indictment that authors didn’t do something initially that they should have.

Also, (1) I don’t see what three regression tables add but length to the paper, and (2) I’ve always never acknowledged reviewers by name because it seems to diminish the significant service they do for our profession, for free, when they peer review.
 Reply
#36
Bryan,

Quote: SH: The more x requires g, the larger the race difference on x.

In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).

In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).

I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.

In the way you frame it here, yes, that would a kind of SH test. But note that your analysis do not look at group differences directly, and neither does it look at under or over-representation directly. It only looks at representation (proportion of workings from each SIRE). I don't see how it can be a test of SH without actual group difference data being analyzed.

Quote:Obviously, the harked analyses weren’t in the original submission. We harked to “Hopefully Appease Reviewer Kirkegaard” (a double hark!?).

The analyses you suggested never occurred to us as we submitted the original paper here. So, adding the analyses had to be post hoc and exploratory. In fact we start discussion of this by stating “A reviewer recommended we explore….” I think that language makes it patently obvious that the analyses to follow are post hoc. We should add more disclaimers?

It occurs to me that “harking by reviewer” is interesting, and is likely an indictment that authors didn’t do something initially that they should have.

Of course, these extra analyses were added by my request. However, I did not advocate HARKing in the write-up.

For an example of what I had in mind, see this previous paper's review thread where another reviewer (not me!) suggested an exploratory analysis. The resulting write-up was:


(Robustness section)

During the peer review process L. J. Zigerell suggested that population density or total population might be obscuring the results. To test this, I created parallel versions of the dataset with controls. This was done simply by regressing (linear regression) each indicator on the control variable and saving the residuals. Then each control variable was used with three modes: 1) untransformed, 2) log transformed and 3) square root transformed. This was done to make the distribution more normal and suitable for the linear model used for the control. The population data was copied from Wikipedia (“List of Japanese prefectures by population,” 2015).

...
(Discussion section)

However, in an exploratory (unplanned) analysis, it was found that if one removes the effect of (the log of) population density, the usual S factor study pattern emerges: the loadings go in expected directions, S can be reliably extracted from different samples of indicators, S correlates strongly with cognitive ability, and Jensen's method indicates that the relationship is due mostly to S, not other variance.


Here's what you have:

Quote:A reviewer recommended we explore further the relatively weak correlation between percent Asian and IQ.

All good.

Quote:We therefore conducted multiple regression analyses testing a hypothesis that Asians gravitate more toward “people jobs” (i.e., values for People might suppress the IQ / percent Asian correlation).

I suggested adding the interest predictors. This writing suggests, to me at least, that there was a specific hypothesis about Asians gravitating towards people-jobs, whereas this was not the case.

I was simply saying that either people or things-interest may act as a suppressor variable, but no direction of effect was suggested. Here's the direct quote (from earlier) "Asians have substantially different job interests (no idea if this is true), which therefore throws the correlations off."

Quote:However, this analysis on percent Asian produced beta weights of .41 (IQ), .34 (People), and .00 (Things). It appears that values on People suppress the true correlation between Job IQ and percent Asian.

Nitpick: the true relationship not the true correlation.

Quote:Also, (1) I don’t see what three regression tables add but length to the paper, and (2) I’ve always never acknowledged reviewers by name because it seems to diminish the significant service they do for our profession, for free, when they peer review.

Okay with me. :)
 Reply
#37
(2016-Jul-05, 02:32:14)Emil Wrote: Bryan,

Quote: SH: The more x requires g, the larger the race difference on x.

In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).

In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).

I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.

In the way you frame it here, yes, that would a kind of SH test. But note that your analysis do not look at group differences directly, and neither does it look at under or over-representation directly. It only looks at representation (proportion of workings from each SIRE). I don't see how it can be a test of SH without actual group difference data being analyzed.


You make an interesting point here, and I'm not sure I can address it.

But, suppose I subtracted the population percents for each group from the race data by job. So, for example, since 12% of the USA labor force is Black, subtracting that number from the Black column would then truly produce numbers that measure over/under representation. These numbers get more negative as job IQ goes up (Blacks are underrepresented at high-IQ jobs, presumably because of g). The reverse (i.e., a race difference) happens when looking at Whites and Asians.

Wouldn't this reversal be the "actual group difference" you're looking for above? The race difference here is in the relative percent of Blacks and Whites holding low and high IQ jobs.

If so, my idea of subtracting out the population percents seems trivial, as the correlations would remain the same values?

Would it ever be possible to test SH by looking at group differences indirectly?

I'm not sure I'm correct here, and wonder what other readers think about this study being a test of SH?

I'll reply to your other comments after we resolve this issue, as it seems key to the paper's future.
 Reply
#38
Perhaps we can get Dalliard to give his thoughts. He is also knowledgeable on the SH literature. I'll send him an email.
 Reply
#39
(2016-Jul-05, 03:37:10)bpesta22 Wrote: Would it ever be possible to test SH by looking at group differences indirectly?

I'm not sure I'm correct here, and wonder what other readers think about this study being a test of SH?

I'll reply to your other comments after we resolve this issue, as it seems key to the paper's future.


With MCV one correlates the vector of group differences with the vector of g-loadings. In this case, (1) groups differences are indexed by the employment % differences, and (2) "g-loadings" are re-conceptualized as "cognitive complexity loadings" which are indexed by the IQ -- or test score on a test that claims to measure general cognitive ability -- requirements of the jobs. Emil feels that (2) is problematic.

I disagree; Linda Gottfredson made an equivalent argument and conducted an equivalent test (and her paper passed review). We will have to wait for another reviewer to adjudicate. If they agree with Emil, you will simply have to rephrase some of the sections. What you are doing would no longer count as a valid test of SH, but would be an application of it -- a putting SH to work, as you say -- in the sense that SH would predict this -- thought so might other hypotheses -- because g is the major predictor of job differences and if there were group g differences -- as opposed to narrow ability differences or psychometric bias differences -- one would expect employment rate disparities in line with the GMA loaded-ness of Jobs.
 Reply
#40
I like the article. That there is a prediction that the % of racial composition changes as IQ job (or complexity) increases is supportive of Spearman's hypothesis, also shows (or rather, suggests) that within-correlation of IQ*job complexity may also extend to between-group context (as Gottfredson compiled lot of research showing that this correlation holds within groups, white groups) as the hypothesis expected.

I have no problem with the consistency/stability of the correlations, and their magnitude and signs indeed are supportive of the studied hypothesis.

I would appreciate if you can describe a little bit more the variables of worker activity, namely, data, people, and things, because it may not be very clear to everyone (e.g., I have only a rough idea).

Quote:They score six (“speaking-signaling”) on people, and two (“operating-controlling) on things.

You forgot a bracket.

(2016-May-30, 21:37:35)Chuck Wrote: I was suggesting that you might qualify your statement. For example, for precision, I might say: Spearman's hypothesis would predict that group differences are larger on more g-loaded tests, assuming no countervailing psychometric bias. Likewise: Spearman's hypothesis would predict that employment differences are larger for more g-loaded fields, assuming no countervailing societal bias e.g., affirmative action or defacto quotas. If you think that the qualification is obvious, don't bother.


It has little to do with the core of this study but I don't remember that Jensen said anything like this. In fact, I think Jensen's reasoning is in accordance with Dolan, in that that if there is bias, any finding in support of SH would be unreliable. They however don't disagree about the method to detect bias (but that, you and I, we already know very well).
 Reply
 
Forum Jump:

Users browsing this thread: 1 Guest(s)