By Jameson Quinn
1. Intro and disclaimers
This is a summary of the discussion thread on the post by Kevin Standlee discussing three proposals for expanding the Hugo nominations process in order to help avoid problems like the current list of Hugo finalists.
In summarizing a thread that’s over 500 comments long, and that refers extensively to two other threads with hundreds of comments, I am of course going to simplify matters. In particular, I’m going to overemphasize the points of agreement, without listing every qualification, caveat, quibble, or outright objection that was brought up. Obviously you can read the thread yourself, but if you don’t, please imagine it to have all the disagreement and misunderstandings (as well as off-topic filk, execrable puns, cute references, hastily-constructed codes, etc.) that you’d expect.
2. Initial options: A+2, DN, and 3SV
Of the three initial proposals by Kevin, one of them (nicknamed “+2” or “A+2”) relied on giving new discretionary powers to Hugo administrators. In the discussion thread this idea encountered significant opposition, so it will not be discussed further here.
The other two proposals would both create a new intermediate round of voting, in which the initial votes have been used to create a “longlist” of 15 works which is publicized and voted on in some fashion in order to get the list of 5 finalists. In the Double Nomination with Approval Voting (“DN”) proposal, nominators would vote for (“approve”) the longlist items they liked the best; and in the 3-Stage Voting (“3SV”) proposal, a majority of nominators would be able to disqualify some of the 15 works, without affecting the ordering of the remaining works from the nominations phase.
3. Consensus: 3SV (+1?)
Many of the participants in the discussion began with a preference for DN; they felt better about voting for the longlist works they liked than about voting to reject the ones they felt were illegitimate. However, as the thread progressed, and the strengths and weaknesses of the two proposals were analyzed in more depth, the consensus shifted, and by the end of the thread proposals based primarily on 3SV were the clear winners. Many supported a proposal nicknamed 3SV+1, described below, which integrated some aspects of DN onto the base of 3SV.
4. Why 3SV and not DN?
Why did people’s preferences shift from DN to 3SV? Several reasons:
A. Under DN, voters would have just weeks to assimilate and vote on a list of 255 longlisted items. Many of the most careful nominators would barely vote for any; while the most prolific voters would probably be going mostly by kneejerk reactions. This is true for 3SV, too, but it is less of an issue as explained below.
B. DN conflates two questions: “Do I like this work and feel it may deserve a Hugo?” with “Do I feel that this work’s presence on the longlist or in the list of finalists would be a legitimate result of honest fan preference?” In 3SV, those questions are separate, and votes to disqualify a work are based on the second question alone — one which does not require fully reading/reviewing every longlist work.
C. Unlike DN, 3SV would deal decisively with the issue of “troll finalists”: that is, works promoted by slates explicitly in order that their shocking and/or offensive nature might cast discredit on the awards.
D. 3SV would be similar in spirit to the “no award” option, except that works thus eliminated would not take up space on the list of finalists, and awkward moments at the awards themselves would be minimized.
E. DN would open up new kinds of attacks on the list of finalists, such as actually increasing slate voter’s capacity to act as “kingmakers” and/or perform “area defense” against certain kinds of works. All they’d have to do was to have enough voting power to reverse the gap between two works which both have significant organic (non-slate) support. But under 3SV, actually eliminating a work would not be possible without a relatively high “quorum”* of voters, and we hope that community pressure would lead to a low background level of organic rejection votes, so a minority of slate voters would be unable to use rejection as a weapon.
So, tell me more about how 3SV would work
The details are still up for discussion, but the basic idea is as follows:
3-Stage Voting (3SV) adds a new round of voting to the Hugo Award process, called “semi-finals,” between the existing nominating ballot and the existing final ballot. In 3SV, the “longlist” of top 15 nominees (as selected by the same process as the finalists will be selected; that is, EPH or EPH+ if those have passed) are listed in a way that doesn’t show how many nominations they received. Eligibility for this voting is being debated (see below), but the original proposal is that it would be restricted to members (supporting and attending) of the current Worldcon (not the previous and following Worldcons). Eligible voters presented with this list, with a question on each of the fifteen semi-finalists in each category: “Is this work worthy of being on the Final Hugo Award Ballot?” with the choices being YES, NO, and ABSTAIN.
If a work gets more than a “quorum” of no votes, it is not eligible to become a finalist. There are several proposals for how to calculate the quorum. The formula may involve such things as the number of eligible voters, the number of “YES” and/or “ABSTAIN” votes, the turnout in round 1 or in previous years, etc. The idea is that the quorum should be high enough so that a minority of slate voters will be unable to reach it, but low enough that a clear majority of fans can pass it, even given reasonable turnout assumptions.
During the “semi-final” voting period, admins would also be checking the eligibility of the works on the longlist, and accepting withdrawals from the author (or other responsible party; henceforth, we’ll just say “author”) of those works. The admins would make a good-faith attempt to contact authors, but note that since the longlist is public, the admins may assume that non-responsive authors have heard of their presence on the longlist, and thus that any authors who do not explicitly withdraw their works would accept becoming finalists.
The finalists will be the top 5 works from the longlist that have not been declared ineligible by vote, found ineligible by the admins, or withdrawn by the authors. EPH or EPH+ would not be re-run after ineligible works or withdrawals.
After the Hugos are awarded, admins would publish the usual statistics (that is, for EPH, the votes and points-when-eliminated for each of the members of the longlist). They would also publish the reason for ineligiblity (voted out, ineligible, or withdrawn) for any work that otherwise would have been a finalist. They would also publish the anonymized set of vote totals for each longlist item in each category, where “anonymized” means that they would not indicate which title was associated with which vote total.
What’s this “3SV+1” you mentioned earlier?
The “+1” means that, in addition to 3SV, all voters who were eligible for round 1 may add a single nomination per category, for something from the longlist they did not already nominate, to their existing ballot, during the same “semi-final” voting period. These combined ballots would then be counted by the usual process (that is, the current system, EPH, or EPH+) to find the finalists.
How would all of this interact with EPH, EPH+, 4/6, and/or 5/6?
The short version is that without EPH a realistically-sized, well managed slate could hope to entirely take over the longlist in many categories; with EPH, it could take about 2/3 (around 10 slots); and with EPH+, it could take over about half (6-8 slots), or possibly a bit more with cleverer strategies (but not as much as EPH, even then). 4/6 or 5/6 don’t change that story by much, though they help a bit in keeping organic slots among the finalists in spite of “kingmaker” slates. And +1 helps push slates towards around 1-2 finalist slots; hurting them in the common case that they had been going to get more, but actually helping them if they had miscalculated and were heading for less than that.
Here’s a graph of how many slots slate nominators could have gotten in 2014, as a function of number of slate nominators, if they’d split 3 ways and had the same level of coordination as they did in 2015. Note that 2016 had many more nominators than 2014 so it would have taken more slate nominators to get the same effect.
Here’s a similar graph, but assuming the slate nominators are better coordinated:
Here’s yet another graph, but assuming the slate voters split only 2 ways. In theory, this is better for them if there are fewer of them, but worse if there are more, because they max out at 10 slots. However, as you can see from the graph, it’s really not that much better even for small numbers; the random “bootstrap sampling” effects almost overwhelm any advantage:
Have you thought of any downsides? What about…
Yes, kinda. We have people (both honest supporters and honest opponents) thinking of attacks. And there’s always room for more on this “red team”. So far, here are the criticisms we’ve come up with. First, for “3SV” (we’ll talk about “+1” below):
Couldn’t slate voters take over the shortlist? As you can see in the graphs “take over 6-10 slots of the longlist” is the only one that we think is a concern if (as we expect) EPH or EPH+ is in place.
Wouldn’t this just increase negativity? There are several safeguards against this becoming merely an excuse for people to campaign against works they happen not to like. First and most important is social pressure; it should be clear from the outset that this is a just safeguard against outright bad faith, not a chance to express differences in taste, and I believe that any Worldcon members who promote disqualifying a work just because they don’t like it will not get much support. Second, there’s eligibility. Various rules, discussed below in “open issues”, have been proposed to prevent a campaign to bring in Worldcon outsiders after the longlist is public. Third, there’s the quorum; if participation in the second-round voting is low, it will not be enough to pass the threshold to eliminate any work. Fourth, there’s the relatively short period of the semifinals, also discussed below. And fifth, there’s the fact that elimination votes for a specific work would never be publicized; only anonymized distributions of votes for each category. (In some cases, of course, the identity of which work got a certain vote total would be easy to guess, but that would still be just a guess.)
Wouldn’t this fundamentally change the nature of the Hugos? They have already been changed by the slate. Many of the people in the discussion felt that this change, though it would not go back to exactly as before, would still be a change in the right direction.
Would this be more work for the administrators? In some ways, yes, of course. However, in at least one important way, it would actually simplify their lives. Since the longlist would be public, it would be much easier for them to contact authors. On a related note, authors could not leak their status as finalists, because until the list of finalists came out, they would know no more than the public at large.
Would this allow some unanticipated downside? Obviously, we can never rule that out 100%. However, we do think we’ve been pretty thorough at exploring all the angles. Again, you can read the thread and decide for yourself.
And, downsides for the +1 addition:
Would this be tough to administrate? Not if EPH or EPH+ were in place, since any program capable of doing either of these would already be able to associate multiple nominations with the same nominator and make sure that invalid votes, such as a single nominator nominating a given work multiple times, were not counted. It has been suggested that a proposal to institute +1 should say that this change will sunset (require re-approval) if EPH or EPH+ ever does.
What are the open questions/issues?
Eligibility: who should be eligible to vote on the semifinal round, and who should be able to add +1? The former question is more fraught. Several people said that they would want eligibility to be restricted enough that outsiders can’t come in and get memberships after the longlist is published. Others said that it is important to let the community respond and that if membership spikes on seeing the longlist that could be a healthy thing. One compromise proposal (suggested by yours truly) was that in order to be eligible, you would need to be a current member (attending or supporting) of this year’s Worldcon, AND also have been eligible to nominate (whether or not you actually did so). So if you were a member of the prior or following year’s worldcon, you could sign up after seeing the longlist; but not if you weren’t.
Quorum size/formula: There’s been various discussion of this issue.
+1 as attached or separate: The overall consensus seems to be that, if we propose +1, it should be in a separate proposal from 3SV, even though they share certain aspects (such as the concept of a semi-final round). However, there are varying opinions on whether +1 is a good idea, either in general or as a proposal for this year in particular.
Whether to try EPH+ this year: I’ll talk about this more in comments.
The admin discretion 14-18 longlist thing: I was thinking that, if the time between closing round 1 nominations and announcing the longlist is short, admins might have a hard time cleaning the data perfectly. In that case, it helps them be more certain of the list they publish if the next work below the list was not a near-tie with the lowest work on the list. To allow them to avoid such near-ties, especially in cases where they aren’t 100% sure they’ve cleaned the data perfectly, I suggested allowing them discretion to decide how long the longlist for each category would be, between 14 and 18 works.
https://file770.com/?p=29020&cpage=14#comment-436327: Define “nominating membership” in a way that legally allows for the possibility of a code of conduct that could lead to one year’s worldcon revoking voting privileges from a member of the prior year’s; in other words, closes the loophole whereby prior-year members are immune from any consequences for their actions.
Moderated Shortlist for Member Consideration: https://file770.com/?p=29020&cpage=16#comment-436437 This is an alternative to 3SV, where an administrative committee would tentatively suggest eliminating or adding certain works, and that suggestion would have to be ratified by an up-or-down vote of the members of the current Worldcon.
Where do we go from here?
A group led by Colin Harris that includes Kevin Standlee is writing up a proposal are writing up a proposal and are surely watching these threads. When that proposal is ready, we could write up +1 as an amendment or as a standalone proposal. Decide about EPH+ and deal with that. Anything else?