Anonymous Group Challenges Statistical Validity of Fireside Report

In a report dropped just after the close of the Hugo nomination deadline last night, the timing chosen purposefully (I’m told by a source) though the reasons are not obvious, “a group of writers and editors” has challenged the statistical basis of the Fireside Report which said last July,” We don’t need the numbers to know that racism is a problem in our field. But we have them.”

Published at Medium under the pseudonym “Lev Bronstein” (Leon Trotsky’s real name), “Bias in Speculative Fiction” counters the Fireside Report by applying additional statistical study methods to the data, or enlarging the field from which relevant data can be drawn.

Fireside Fiction’s July 2016 report “Antiblack Racism in Speculative Fiction” [FR] purports to have found pervasive racism in speculative fiction publishing. With 38 out of 2,039 (1.9%) published works authored by black-identifying authors, there is unquestionable underrepresentation. FR ascribes this disparity to widespread anti-black editorial bias. We share Fireside’s concerns about underrepresentation and commend its authors for raising awareness. At the same time we find the report to be fraught with error.

The article’s many examples include:

The misuse of the binomial distribution in FR is a significant cause for concern. Under the binomial distribution, we treat each publication as an independent random event with some fixed probability of occurrence. FR assumes that each submitted submission should have a 13.2% chance of black authorship. The probability of observing their data subject to this assumption is 3.207×10^–76. They provide no rationale for using population rather than occupational rates. Suppose instead that science fiction slush is submitted from a pool of professional writers uniformly at random. Under these assumptions we assume that there’s a a 4% probability that a story is written by a black author. The data becomes 68 orders of magnitude more likely. Still unlikely, yes, but this demonstrate the impact of our assumptions.

The article can be presumed to serve as a defense of editors in the speculative fiction field. It argues there is bias, but that it is exerted in the culture in ways not directly related to the fate of slushpile manuscripts, such as in the educational system where PoC may or may not take degrees in literature, and other things that discourage black people from becoming authors at all.

The authors of the article defended their choice to remain anonymous in these terms —

Who are you?

We’re a group of writers and editors. We have chosen to publish under a collective pseudonym. Our identities would only serve as a distraction.

It’s puzzling why a group that agrees in their preamble that “There is a race problem in speculative fiction and we need to make an effort to understand its causes” is unwilling to engage under their own names, essentially reducing this to a drive-by correction of somebody else’s homework.

Justina Ireland, an executive editor at FIYAH magazine, has responded at length on Twitter. Here are several of her tweets:

Another comment:

Brandon O’Brien, a poet and writer in Trinidad and Tobago, also has made some observations:

O’Brien has many other comments, though one in particular about “nebulous maths” begs the question:

Bear in mind this quote from the Fireside Report —

To adjust for the methodological flaws, as well as the fact that we don’t have access to submission-rate data concerning race and ethnicity either overall or by individual magazine, we used binomial distributions. The purpose of this was to find the probability that such numbers could be random?—?the chances that numbers like that could exist without biases in play (which could extend to biases that are literary in nature, such as story structure), systemic problems, and/or structural gaps. In the first binomial distribution we ran the data assuming that submission rates of black authors are equal to the proportion of the black population in the United States, which was 13.2% in 2015 (according to Census projections).

The Fireside Report picked U.S. population statistics as the battleground, treated them as a valid tool for analyzing racism, and made arguments based on their own analysis of them. It’s not fair in that context to say the Fireside Report is above criticism because there are PoC writing SFF throughout the world, or that we all know racism is a problem in the publishing industry (as it is elsewhere).

Troy L. Wiggins, the other FIYAH executive editor, questioned the motive behind the new article:

Update: Hours later the authors of the article took it down and left in place the statement, “We’ve been receiving threats. Forget we were ever here.”  

For as long as it lasts, the original post can be read in the Google cache file.

61 thoughts on “Anonymous Group Challenges Statistical Validity of Fireside Report

  1. Greg Hullender:

    “No, the SFF editors are not making it worse–not to a significant degree, anyway. If there were massive, systematic discrimination, both studies would have detected it. They did not.”

    And if discrimination existed and was not massive or not systematic? Should it then be ignored? Everything is allright then?

    And do you think that study is good enough to prove that no discrimination exists that will affect people from minorities, that the data is not skewed in any way depending on who has answered?

  2. Greg Hullender: If there were massive, systematic discrimination, both studies would have detected it. They did not. More subtle (or just less-effective) discrimination might still exist, but it would take more data to detect it.

    And you are missing my point again; at this point, I can only presume that this is deliberate on your part.

    Your first sentence here is a false claim. The second sentence is the root of the problem.

    The original study, while of interest for numerous reasons, is massively unsuited (as Fireside admits) for drawing any sort of concrete conclusions, and certainly not the claims of “fact” you have been making.

    How many minority authors have submitted short SFF fiction to genre venues? How many pieces did they submit? What is the actual rejection rate? There is no way to know. The Fireside report is based on data from the ones they knew about and those they could identify through outreach. It is hardly surprising that they would know about published minority genre authors to contact — after all, they got published, so they were noticed. The non-published ones? not so much.

    You are making utterly unsubstantiable claims based on evidence which does not — can not, by its very insufficiency — support those claims.

    Just stop it.

  3. Greg Hullender on March 19, 2017 at 6:58 pm said:

    @JJ
    Although a number of people have claimed that the article isn’t valid, none of them showed any evidence of having read or understood the document.

    I’ve found it very difficult to go through mainly because it has a very odd relationship (to put it politely and assuming no malice on Bronstein et al’s part) with the original report.

    So with the first section when it states “The report assumes that 13.2% of speculative fiction submissions come from black authors,” I could recall from the last time I’d read the report that the ‘assumption’ was purely for the purpose of using a binomial distribution as a comparison rather than FR more generally assuming that this was (or should be) the case. OK, maybe a poor use of “assumes” there but I’m just on the first proper section and I’m already having to compare what the Bronstein report is saying versus what FR actually says.

    The section on Tokenisation is similar. It leads with a quote from FR which implies it was a conclusion of the report. It is actually just a comment on the FR data spreadsheet explaining why they listed the magazines in order.
    Full quote from FR “Technically, this list overall would provide a more accurate presentation if it was ordered by the percentage of stories by black authors per
    zine rather than the number — 1 of 7 means something far different than 1 of 58. However, we decided to put them in order by number because we are seeing far too many “exactly ones,” which is suggestive of tokenization.”

    Just two sections in (where I was when I began asking the authors questions about the study on medium) and I was finding that the Bronstein report appeared to be misreading the FR report in a way I found odd – as in it appeared to be presenting things in FR as conclusions or core assumptions by FR and attempting to show those conclusions were false when they were not actually conclusions of FR.

    I didn’t ask a question about the tokenisation bit because I wanted to work through what they had done first (then everything went kablooey). I didn’t assume their approach for testing tokenisation via the FR data was invalid but I was confused by what they were attempting to do.

    Now, maybe it was just really poor framing on their part. Assume the analysis of tokenization is correct* – if so then if they had presented what they were doing as using the FR data to investigate the issue that would have made more sense rather than an apparent attempt to identify errors.

    Bronstein has stated in their introduction that “At the same time we find the report to be fraught with error.” and two sections in I was seeing stuff that was not uninteresting but which did not support the idea that FR was ‘fraught with error’.

    So two sections in and the Bronstein report looked like it was building up and knocking down strawmen rather than engaging with the FR report.

    I didn’t look at the section on Terraform in any depth. That left the criticism of FR using a binomial distribution as a comparison.

    As things progressed ‘missing the point’ began to look like to charitable an interpretation of the Bronstein report.

    *[I suspect it is flawed because it tests the hypothesis using the same total number of stories by black authors against the same number of magazines, whereas if tokenism was not present we’d expect either more stories or less stories by black authors I think- but I haven’t done any numbers on what difference that would make]

  4. I’m generally not inclined to give the benefit of the doubt concerning the motivations of people who are unwilling to publicly stand behind their alleged analysis.

  5. Camestros Felapton on March 19, 2017 at 8:12 pm said:
    *[I suspect it is flawed because it tests the hypothesis using the same total number of stories by black authors against the same number of magazines, whereas if tokenism was not present we’d expect either more stories or less stories by black authors I think- but I haven’t done any numbers on what difference that would make]

    OK – less worried about that now. It just looked like an odd result that the proportion didn’t seem to matter.

  6. Pingback: Rebuttal of the 2016 Cecily Kane Fireside Article | Lela E. Buis

  7. This is a couple of days late and I guess will be buried, but given that I have some math background, I feel like writing a comment.

    0. I welcome the archival of the pseudonym report by various sources including Fireside.

    1. The pseudonym report is correct to point out that it’s practically impossible to determine on which level the discrimination is happening when we have data points only the baseline population of black authors and the end results (underrepresentation of PoC authors in publications). To determine to which extent the discrimination done by editors, as everyone has pointed out, one really would need slush pile statistics.

    Of course, their choice of baseline from statistics of professional writers is somewhat arbitrary, too, but it does has a point: the number demonstrates that it’s very well possible that bulk of the systemic racism filtering happens far before any texts end up on someone’s desk in a publishing house. Both numbers are poor proxies for the real base rate and thus not worth much, except for illustrative purposes (that choice of base rate matters in these models).

    It’s also possible that maybe there are dynamical relationships: there are less professional black authors than proportion of black people in general population would suggest because black authors have more difficult time getting published combined with other reasons (for example, rate of literary hobby). Rejection rate in publishing industry might have an effect on the distribution of demographics of authors the submitting stories. But estimating such models (especially the contribution of systemic racism in publishing vs other reasons) is difficult with only this kind of data.

    2. A charitable interpretation is that some editors felt like they were unduly attacked of being racists by the Fireside report and the commentary that followed it. It’s kind of reasonable that someone in the publishing might feel like they were being accused of racist publication policy, when the conclusion of the Fireside report is “Speculative short fiction publishing is rife with antiblackness, and white speculative fiction writers and publishers need to stop pretending otherwise”.

    3. As an aside, I think it’s worthwhile to notice that numbers do not tell us the who is doing the discrimination and why. Someone above in the comments pointed out the some of the discrimination could be caused by simply because of the differences in the life experience: the editor thinks that in their personal opinion, a story written by a PoC person is “not interesting” because it’s about stuff they can’t relate to. Which is both sad, and a reasonable response from the editor, which makes the situation doubly sad. It is also very problematic, especially if all the editors in the industry are from the same social segment and thus PoC authors writing about PoC experiences find it more difficult to get published than they should, so it should be fought against.

    But at the same time I’d find it is terrible to accuse publishers of racism on these grounds. The data at hand does not answer which kind of filtering effects – reasonable or unreasonable – are taking place. On the other hand, it’s certainly possible that there’s happening something more disgusting like, say, throwing manuscripts into rubbish bin because of the name of the author on the first page does not sound “white” enough. Or (most likely) combination of different effects. But we don’t know, at least with this information alone.

    4. I think pseudonym was also correct to point out that the data is consistent with Poisson process instead of ‘tokenism’. However, curious thing is that when I Ctrl-F’d “token” in the original report on the Medium website I didn’t find anything. No mention of tokenism. Where did the claim of tokenism originate?

    5. As a math person I’m inclined like pseudonym report more because it elaborates on the statistical methodology more. Like the multiple hypothesis part about Terraform or the effect of established authors writing most of the stuff being published and having easier time having their stories published (instead of i.i.d. story model).

    Pseudonym’s act of taking the report down was certainly weird. Both charitable and uncharitable interpretations are possible. However, we do not gain more knowledge by attacking the results of the pseudonym by accusing that their motivations for writing the report (whatever they are) might have been racist, if the contents of the claims itself are correct. Given the accusations of trolling and worse, I sort of feel sympathetic to their decision to remain anonymous.

  8. a.n.: Given the accusations of trolling and worse, I sort of feel sympathetic to their decision to remain anonymous.

    Given that the so-called rebuttal spent a considerable amount of verbiage countering claims that the Fireside report did not actually make, I think that the accusations of trolling are fair. There is a serious amount of strawmanning going on there.

    I am rarely inclined to give the benefit of the doubt to people who engage in strawman arguments — and the fact that they did so anonymously just makes them look even worse.

  9. I don’t find anonymity inherently suspicious. Lots of people use varying degrees of pseudonymity depending on the situation; it doesn’t automatically imply they’re up to no good.

    I do find arguing with statements not made off-putting. To me, it implies that someone is arguing with an assumption rather than the people actually involved. When posting a rebuttal on an emotive topic it helps to double-check to make sure that isn’t what you’re doing.

  10. Pingback: Pixel Scroll 3/24/17 No, Mr. Bond, I Expect You To Scroll | File 770

Leave a Reply

Your email address will not be published. Required fields are marked *