To Whom It Might Concern: Demographic Variations in Letters of Suggestion

Beverly Hirtle and Anna Kovner

Letters of advice from school advisors play a crucial position within the job marketplace for Ph.D. economists. At their greatest, they’ll convey vital qualitative details about a candidate, together with the candidate’s potential to generate impactful analysis. However at their worst, these letters supply a subjective view of the candidate that may be vulnerable to acutely aware or unconscious bias. There can also be similarity or affinity bias, a very troublesome challenge for the economics career, the place most school members are white males. On this submit, we draw on our latest working paper to explain how suggestion letters differ by the gender, race, or ethnicity of the job candidate and the way these variations are associated to early profession outcomes.

Methodology

We analyze the textual content of 6,365 suggestion letters obtained by a big U.S.-based analysis establishment for two,227 new Ph.D. job candidates throughout 4 latest annual recruiting cycles (2018 to 2021). We pair the advice letters with info provided by the candidates about their main and secondary fields of analysis curiosity, their Ph.D. granting establishment, and confidential details about their self-identified gender, race, and ethnicity. Data on gender and race/ethnicity was collected for statistical functions on a voluntary foundation from all job candidates to the group, not only for economists. The knowledge candidates submitted was not used within the hiring course of and was not supplied to hiring managers or these reviewing or interviewing job candidates.

We establish key traits of every letter to measure the standard of the advice. These measures embody general phrase depend and the variety of phrases related to “standout” and “grindstone” traits. Standout traits are associated to how candidates may excel relative to others and are captured by phrases reminiscent of “modern,” “extraordinary,” and “distinctive.” Grindstone traits are associated to effort and are captured by phrases reminiscent of “reliable,” “hard-working,” and “devoted.” In specializing in standout and grindstone phrases, we mirror a lot of the sooner literature each for economics and for different fields (see, for instance, research involving candidates for jobs in biochemistry, orthopedic surgical procedure, and common surgical procedure). Whereas standout phrases are unambiguously optimistic, grindstone phrases can have a extra blended connotation and are generally thought-about as “damning with faint reward.”

We additionally develop a novel measure of letter high quality primarily based on language contained in lots of the letters that gives the letter author’s suggestion for the caliber of hiring establishment acceptable for the candidate—particularly, whether or not the letter author recommends the candidate to a “high” division. General, about 10 p.c of the letters in our pattern include such a suggestion, so these suggestions are comparatively uncommon.

Demographic Variations in Letter High quality

A lot of the sooner analysis on letters of advice has centered on variations in letter traits by candidate gender (see right here and right here for 2 earlier examples in economics from European universities). Not like a few of this earlier work, we don’t discover statistically vital variations in letter size or the share of standout phrases in letters for feminine job candidates. We do, nonetheless, discover that letters for feminine candidates include increased shares of grindstone phrases, which as famous, have a doubtlessly ambiguous interpretation.

One novel result’s the connection between letters and race and ethnicity. Letters for candidates who self-identify as Asian are considerably shorter and include fewer standout phrases and extra grindstone phrases—findings that stay once we restrict the pattern to candidates from high 10 U.S. economics and finance packages, one solution to handle potential choice bias within the candidate pool. We additionally discover some variations in letters written for candidates who self-identify as Hispanic or Black; letters for these candidates include a decrease share of standout phrases. Whereas the position of race and ethnicity in letters of advice in economics has not been studied to our data, these outcomes are per these in another fields (for example, standout phrases are extra probably for use for white surgical residents).

Macroeconomics and Finance Are Harsh Graders

We additionally discover vital variations within the size and substance of letters by subdiscipline inside economics. Letters written for candidates who establish “finance” or “macroeconomics” as their main area of curiosity are shorter and use fewer grindstone phrases. Letters for candidates specializing in macroeconomics additionally use fewer standout phrases. Candidates specializing in finance usually tend to be receiving their diploma from a enterprise college than candidates in different fields, so the variations may mirror the kind of college moderately than the sphere per se. We additionally run specs containing a management for enterprise college because the Ph.D.-granting establishment and the differential outcomes proceed to carry in these specs.

Underrepresented Candidates Are Much less More likely to Be Beneficial to the High Departments

We discover significant variations by gender, race, and ethnicity in whether or not a letter recommends a candidate to a “high” economics division. Letters for feminine, Asian, and Black or Hispanic candidates are all considerably much less prone to embody such a suggestion. These variations persist once we management for the traits of the Ph.D.-granting establishment, once we management for the traits of the letter author (each feminine and Asian letters writers are much less prone to make such suggestions), and once we restrict our pattern to candidates graduating from high 10 economics and finance departments. These variations are each statistically and economically vital. Letters for feminine candidates are 18 p.c much less prone to include a “high” suggestion than letters for male candidates, a outcome that holds even when the letter author is feminine. Letters for Black or Hispanic candidates are 30 p.c much less prone to include this suggestion than letters for white candidates, whereas letters for Asian candidates are 45 p.c much less prone to have a “high” suggestion.

Early Profession Outcomes

However do these variations in letter high quality matter? To handle this query, we look at the influence of letter traits on early profession outcomes for the job candidates in our pattern. Particularly, we look at preliminary job placements (whether or not a candidate’s preliminary job is at a high 20 economics or finance division) and publications (the variety of high journal publications a candidate has inside two years of receiving their Ph.D.).

Controlling for candidate traits, area of curiosity, Ph.D.-granting establishment traits, and letter author traits, we discover that stronger letters are certainly related to higher early profession outcomes. Longer letters and a “high” suggestion are each positively related to the chance of getting a high 20 preliminary job and with the variety of high journal publications. The next share of standout phrases is related to extra high journal publications whereas a better share of grindstone phrases is negatively related to early profession publications. We discover proof that early profession outcomes are stronger for candidates from high 10 economics and finance packages and that some outcomes are weaker for feminine, Asian, and Black or Hispanic candidates, even after controlling for letter traits.

Summing Up

Taken collectively, our findings counsel that there are significant variations within the content material of advice letters correlated with the gender, race, and ethnicity of the candidate, in addition to with the candidate’s area of curiosity, and that these variations matter in predicting early profession outcomes. A key open query from our work is to grasp the explanations for these findings. Is that this a pure end result of preferences for similarity whereby underrepresented candidates are much less just like their letter writers? Does this signify variations within the kinds of matters that various kinds of candidates select to check? What results in the variations in candidates being really useful to “high” departments? Understanding the associations within the information is barely step one in serious about the way to weigh the qualitative info contained in letters of advice.

Beverly Hirtle is a monetary analysis advisor in Monetary Intermediation Coverage Analysis within the Federal Reserve Financial institution of New York’s Analysis and Statistics Group.  

Anna Kovner is an govt vp and the director of Analysis on the Federal Reserve Financial institution of Richmond.

The way to cite this submit:
Beverly Hirtle and Anna Kovner, “To Whom It Might Concern: Demographic Variations in Letters of Suggestion,” Federal Reserve Financial institution of New York Liberty Avenue Economics, November 15, 2024, https://libertystreeteconomics.newyorkfed.org/2024/11/to-whom-it-may-concern-demographic-differences-in-letters-of-recommendation/.