I recently read a blog entry about gender distribution in submissions and publications that Edmund Schubert, the editor of Orson Scott Card’s InterGalactic Medicine Show (IGMS), posted on Magical Words. He quoted three emails from two assistant editors relating submission rates and then provided publication rates. He then made the claim that they publish women at the same proportion that women submit, calling the ratio “close enough.” I hear not everyone was impressed by this rigorous approach and the subsequent detailed analysis and conclusion. Rather than pick a side, I decided to look at his submission and publication numbers and see if there was a statistically significant conclusion to draw. After all, the data provided is exactly the kind of data I need for a Chi-Squared test. Isn’t it exciting!
I’m interested in looking at the magazine as a whole. Individual staff are an entirely different matter which Scott M. Roberts elected to take on. He’s the Assistant Editor in email 2 and he has his own blog post discussing his own numbers, which has lead him to a difficult conclusion. And possibly unicorn steaks. That is a fabulous illustration of using data to draw a conclusion instead of just supporting one. Nor am I talking about anthologies, which Cheryl Morgan recently looked into. Yay for data!
Back to the Chi-Squared test. This type of statistical test is one way of determine if a drug performs better than a placebo in testing. You start by assuming the drug has no effect and try to prove otherwise. The assumption under investigation here is that there is no gender bias at IGMS. That is, women and men are published at the same rates as they submit stories. Let’s see if I can disprove it with a 5% threshold. That is, let’s see if I can say that I am 95% confident the submission rate is different from the publication rate.
The statistically expected values for each case (E) can be compared to the observed submission and publication numbers (O) in a handy little contingency table table.
For this table, only the submission numbers reported by Eric James Stone in emails 1 and 3 were used as it is likely that there is overlap between the submissions reviewed by the two editors and I want to avoid double counting. Using the observed and expected values, the Chi-Squared value is well below the requirement needed to disprove the hypothesis at a 5% significance level, a 10% significance level, or at any reasonable level. I ran this again using just email 2 and the results were the same. In other words, the numbers indicate that there is no difference between submission and publication rates at IGMS, no bias towards or against women.
That’s not a new finding. Susan U. Linville looked at magazine submission and publication rates back in 2002. She concluded that low publication rates were due to low submission rates. In a 2007 follow up, she noted that in the years since that analysis had been done, submission rates hadn’t improved. Looking at even more recent publication data, the proportion of women published in Analog (14% from 9/10 to 7/11)*, and F&SF (18% from 6/10 to 6/11) hasn’t increased. Publication rates for Asimov’s (32% from 8/10 to 8/11) have increased to a value on par with IGMS publication rates.
Is a perception of bias keeping women from submitting to these venues? Does a lack of existence of bias matter? I’ve read more than a few suggestions that perception is reality and the low number of submissions is in part because of the low number of publications. Angry Black Woman has suggested that promoting diversity will lead to diversity. (That’s an oversimplification of the eloquent and detailed piece linked, but “close enough” for my purposes.) In terms of gender, that would mean publishing women’s work at a greater rate than it is submitted should lead to more work from women being submitted. This concept is testable.
Strange Horizons has consistently published fiction by female writers at a greater rate than it received submissions since in 2001. Jed Hartman continued to track these numbers which are available from their website. Let’s see about using this data set to test out the argument for diversity. If it’s true, there should be an upward trend in submission rates. If it’s false, the submission rates should remain flat as they have for the other publications discussed. The number of ambiguously named authors complicates matters, but I can’t make a good argument for dismissing that population from the analysis. That should make things interesting.
I started by graphing the percentages of submissions and publications. In the figure, I’ve plotted 3 possible submission rates. The red line represents the reported ratio of female submissions. The blue line represents the sum of female and ambiguous submissions, which can be considered an upper limit on the total possible submissions by women. The purple line is an average value that assumes half of ambiguous submissions are from women.
The first obvious observation is that the distance between the green publication line and any of the submissions lines hasn’t closed. Looking at just the red line, it appears to be either flat or declining over time. Obviously, “looking” isn’t a meaningful measure. Daniels’ test for trends is. This test works by ranking the values in a time series and determining how closely the rankings are in order as they move forward in time as seen in the next table.
The results for the blue and purple lines indicate that both follow a trend. The red line does not. So, assuming that at least half of the ambiguous names on submissions are women, there has been a demonstratable increase. I have no way of knowing if that is a valid assumption.
I also redid the Chi-Squared test comparing the 2010 Strange Horizons average submissions with the IGMS submissions, and found no difference. So, even with a demonstrated upward trend in submissions over the last 10 years, it isn’t yet a statistically significant increase for the purple line (half of the ambiguous names are women). By varying the submissions numbers, I found that I would have to assume at least 65% of ambiguous submissions to Strange Horizons are women before their submission rate become statistically higher than IGMS rates. I have no way of knowing if that is a valid assumption.
Well that was a lot of work to come to no conclusion. Hmm. I do know that even if all of the ambiguous submissions are women, the submission rate is still less than 50% after a decade of publication rates greater than 50%. So while perception might possibly be altering reality, it has not resulted in equality in submission rates.
From the numbers in the short fiction markets I’ve looked at, it is clear that the perceived imbalance is the result of the different submission rates for each gender and not a bias on the part of publications. The only bias I can demonstrate is in favor of women. Again, that isn’t a new finding, just one I want to reinforce.
I don’t know anything about the publishing industry. But Kristine Kathryn Rusch does. She recently wrote:
Right now, new professional magazines are appearing almost daily. By professional, I mean magazines that pay their authors—and not in copies, but in actual dollars. Twenty years ago, a science fiction short story had to sell to one of five markets or get retired. Now, a science fiction short story has a dozen markets or more. There are so many markets in my main short story genre that I’m not even familiar with all of them. And that doesn’t count markets in mystery, romance, horror, and mainstream.
So, no bias against women, and a growing market. At least, that’s my perception of reality with a little bit of math.
* Analog and Asimov’s are my own counts using the TOC’s from the Kindle editions I have archived. Numbers for F&SF were collected by Scott M. Roberts (same one), whom I contacted after seeing his comment on the Magical Words blog post. And then I asked him like 500 questions about his numbers and Schubert’s numbers all of which he kindly answered.
DMS is an over-educated engineer living and working in the Deep South. In her free time, she reads, plays video games, and travels. She wholeheartedly approves of getting into blue boxes with strange men. This is a follow-up to her previous article analyzing the frequency of Hugo Awards honoring female authors, A Novel Approach.