Three Possible Hugo Voting Alternatives

By Kevin Standlee: In light of the revelation that modeling of E Pluribus Hugo does not result in quite the “magic bullet” that some may have hoped for, we may need to consider other changes to the Hugo Award voting system to deal with bad actors deliberately setting themselves against the wishes of the majority of the voting members of the World Science Fiction Society. Over the past few weeks, I have written up descriptions of three different proposals that attempt to deal with bad actors in different ways, and to make it more likely that the results of the Hugo Awards represent the wishes of the majority of the participating members, without the members having to resort to the 16-ton anvil that is No Award.

All of the proposals below are compatible with either of the proposals up for ratification this year (EPH and 4/6). They are not necessarily compatible with each other. None of them require a Very Strong Administrator picking and choosing individual members’ ballots or disqualifying individual finalists on what I call “ideological” grounds. All of them aim to give the majority of the members a strong voice in picking the Hugo Awards.

In this overview, try not to get too deeply bogged down in specific details. For example, all three proposal refer to the “Top 15.” This is a reference to the “long list” currently defined in the WSFS constitution thusly:

3.11.4: The complete numerical vote totals, including all preliminary tallies for first, second, . . . places, shall be made public by the Worldcon Committee within ninety (90) days after the Worldcon. During the same period the nomination voting totals shall also be published, including in each category the vote counts for at least the fifteen highest vote-getters and any other candidate receiving a number of votes equal to at least five percent (5%) of the nomination ballots cast in that category, but not including any candidate receiving fewer than five votes.

Therefore, the “top 15” can be defined in different ways, and there’s a reasonable argument to be made for many of them. Don’t get too tangled up in specifics. These proposals have general principles, and if you’re agreeable to the general idea of any of them, then we can discuss the specific values for some of the blank spots in them.

In the list below, the links lead to my LiveJournal where each proposal is listed in more detail.

Proposal 1: 3-Stage Voting

3-Stage Voting (3SV) adds a new round of voting to the Hugo Award process, called “semi-finals,” between the existing nominating ballot and the existing final ballot. In 3SV, the top 15 nominees (including any ties, and see the warning about specifics above) are listed in a way that doesn’t show how many nominations they received. The members (supporting and attending) of the current Worldcon (not the previous and following Worldcons) are presented with this list, with a question on each of the fifteen semi-finalists in each category: “Is this work worthy of being on the Final Hugo Award Ballot?” with the choices being YES, NO, and ABSTAIN.

If a sufficient quorum (which is why counting explicit abstentions is important) votes, and if a sufficient number vote NO, that semi-finalist is disqualified from further consideration. As currently proposed, the necessary NO vote is “more NO than YES votes,” but the exact amount needed to disqualify is negotiable. Remember, even if every vote is NO, if a quorum (minimum number of voters) doesn’t participate, the work cannot be disqualified. (This makes it difficult for a small group to campaign against a work.)

At the end of the semi-final round, any works disqualified by the members, withdrawn by the nominees, or disqualified by the Committee on technical grounds is out of the running. From among the remaining semi-finalists, the five that got the most votes in the nominating phase become the finalists, and the final ballot continues as it currently does. Note that in this case it doesn’t matter how many YES votes a semi-finalist got, only that it didn’t get a negative majority.

3SV effectively moves the votes on NO AWARD to the semi-finals, although it would remain a candidate on the final ballot. It allows the members of the current Worldcon (the ones who will be voting on the final ballot) to decide in advance which works they think deserve to be on the final ballot. It does this at a price, however, and that price is to carve 6-8 weeks out of an already relatively crowded schedule. The proposal moves the deadline by which you have to be a member in order to nominate up by a month, and in practice would require the nominating deadline to be earlier in the year.

Another drawback of 3SV is that it triples the number of nominees that the Worldcon Committee (the Hugo Award Administrators) need to vet and contact. However, as a trade-off, it gives the administrator roughly three times as much time to do this contact work, and makes the vetting and contacting process public. That is because the Administrator would not need to contact semi-finalists in advance of announcing the “longlist” of semi-finalists. While the semi-final ballot runs, the Administrator would be contacting semi-finalists to give them an opportunity to withdraw from consideration in the final ballot. Administrators could put out a public appeal if they are unable to contact a given semi-finalist. In addition, the Administrator can do eligibility confirmation in public, with the help of the many other people who will undoubtedly be checking over the list and asking questions. Should a semi-finalist be disqualified on technical grounds, the semi-finalist would not be replaced on the semi-final ballot. Ineligible works show up on the existing Top 15 lists now, and the semi-final round would be too short to allow for replacing longlist slots.

An incidental feature of 3SV is that the finalists would not know they’d made the final ballot until the shortlist was announced. They’d know they were semi-finalists, and during the semi-finals they would have had time to decline if they so choose, but it would be impossible for them to leak being a finalist because they would not know it themselves.

Additional drawbacks to 3SV include the fact that it is explicitly negative. It is a place where you vote against things. Some people are philosophically opposed to “down-voting” works. Furthermore, should the known bad actors stop trying to game the system, there is little need for this semi-final round, and you might find people not even bothering to participate in it, which might result in complaints that we’d added complexity and expense for no obvious reason.

3SV is a “vote against stuff” system. An alternative to it was the second proposal.

Proposal 2: Double Nominations with Approval Voting

Double Nominations with Approval Voting (DN/AV, sometimes just DN) is similar to 3-Stage Voting, in that there would be a semi-final round with the Top 15 (see warning at the beginning of the article) nominees listed in an order that would not reveal how many nominations they received. Only members (attending and supporting) of the current Worldcon would vote at this stage. But instead of voting against semi-finalists as you do in 3SV, you would vote for those semi-finalists who you think deserve to be on the final ballot. You could vote for one or all of the semi-finalists. The version as currently proposed included a single write-in slot as well, primarily as a safety valve. In this case, the number of votes a semi-finalist gets here is critical, because only the top five would continue to the finals.

In 3SV, the number of YES votes doesn’t matter as long as there are fewer NO votes (or insufficient ballots cast to qualify the election at all). The relative number of YES votes doesn’t matter as long as the semi-finalist isn’t disqualified, because it’s the original nominating ballot count that sends works on to the final round if they survive the weeding-out process of 3SV’s downvotes. However, in DN/AV, the five works with the most votes in the semi-final round go on to the final ballot, and the number of nominating ballots cast to get the works onto the longlist is irrelevant. Some have called for nominating counts to be used as a tie-breaker, as they envision a 15-way tie for the final ballot. I personally think this unlikely, and there have been as many as eight finalists on a Hugo ballot due to ties for the final position, so I think we could live without a tie-breaker.

All of the extra administrative issues of 3SV are shared by DN/AV, so I won’t go over them again here.

DN/AV eliminates the “negative” aspect of 3SV, in that you vote for semi-finalists, not against them.

Some have suggested that there’s an implication that “you need to have read all of the semi-finalists in order to vote on this ballot,” although I don’t think that is true in either case. DN/AV is a second nominating round. Just as the current nominating ballot makes no pretense that you should have read everything published last year, DN/AV doesn’t expect voters to have read all fifteen semi-finalists, but instead to pick those that the voters think worth considering on the final ballot, knowing that only the top five will appear there.

The biggest advantage to either 3SV or DN/AV is that they put the decision of “what to have on the final ballot” in the hand of the members who will be voting on that final ballot, and that the majority will of those voters will prevail. Neither system is particularly susceptible to gaming by small minorities. The biggest drawback to either of the first two proposals is that they add an additional round of voting, with administrative overhead and complexity.

Some people have asserted, with various degrees of strength, that the Committee (Hugo Award Administrators) should simply ignore “slate voters” or disqualify “obvious slate-generated finalists.” In my opinion, such proposals are hugely problematical, in that they give Administrators authority they have never had in the entire history of the Hugo Awards. However, there is one historical precedent to which we can look when a group of bad actors appears to have forced a work onto the ballot, and the Worldcon Committee tried to ameliorate the action without actually disqualifying anyone. We’ll consider this in the final proposal.

Proposal 3: Plus Two

In 1989, the Hugo Award Administrator noticed an odd situation, where a set of nominating ballots arrived in close order with a single nomination cast in a single category. This was enough to place a finalist on the ballot that seemed unusual to anyone who had been watching the kinds of things that had made the shortlist in recent years. The ballots all having included membership payments in the form of consecutively numbered money orders further raised suspicions. According to the coverage of the situation in File 770 at the time, the 1989 Worldcon committee said in a statement at the time that they didn’t do any investigation, but that one person wrote to the committee to inquire why he had a membership when he hadn’t paid for it. (After they made a public statement, some fans contacted the committee claiming to have cast these votes with innocent intent.) In any event, there has always been a strong ethos in the Hugo Awards to allow the membership to speak, not a small committee. On the other hand, this case seemed to be doing someone out of a Hugo Award finalist slot unfairly. The Committee took the unprecedented step of adding the sixth-place nominee to the shortlist. Not too long thereafter, one of the six finalists (the one that seemed to be odd compared to the others) withdrew. There is no evidence or suggestion that the finalist who withdrew had anything to do with the string of bullet-voted ballots. At most, this was a case of enthusiasts with more money than common sense or ethics.

The final proposal that I’ve written up would explicitly authorize the Worldcon Committee (in practice, the Hugo Award Administration Subcommittee set up by the Worldcon Committee) to take the action that the 1989 Worldcon did whenever they think that there was a pattern of unethical voting in the nominating round. The Committee would be authorized to add up to two additional finalists from among the Top 15. They would not reveal which finalists they added until after the Hugo Awards ceremony, when it would be included in the post-ceremony detailed results.

I have no expectation that the current people who have typically been part of the Hugo Administration Subcommittee would be the ones who would make the actual Plus 2 decisions. I expect that any sensible Worldcon Committee would recruit additional Administrators for their literary judgement, just as the World Fantasy Award does. Any Administrator, no matter whether they were recruited for computer skills, literary skills, or public relations skills, would be ineligible for that year’s Award just as the current Administrators are.

Should the Administrators determine that there was no need to do so, they could choose to not add additional finalists. Thus, if known Bad Actors desist from their wrecking ways, the Administrators could simply continue running things as we have done in the past, leaving it up to the five highest pluralities of nominations. This would simplify administration and require relatively little change.

This proposal allows the Committee to pull from the Top 15, rather than simply the next two finishers in the nominations, to minimize multiple-slating attacks on the system that would try to dominate the top seven rather than the top five positions. While 20% of the electorate has been able to dominate the first five positions relatively easily, it seems unlikely that such a small group could dominate the top fifteen unless their voting power grew to a majority of the entire electorate. Inasmuch as solutions that represent a majority of the electorate (even if you personally dislike the result) are not contrary to democratic process, there is nothing in this proposal that tries to “defend against it.” As I’ve said above and will continue to say, if you can command a majority of the voters, you get your way, even if I don’t like it. It is minorities dominating the process that troubles me.

Note that there’s nothing magic about adding two additional works. It could be one or more. The exact value is debatable. However, consider that we do hope that most people read all or most of the finalists, and therefore making the shortlist too long works against any good you might get from adding (say) five extra works to the ballot.

Broadly speaking, Plus Two is compatible with either (but not both) 3SV and DN/AV. That is, this proposal can be considered separately from either 3SV or DN/AV, whereas the first two proposals above are antithetical to each other, and only one of them could be reasonably considered at a time.

The biggest advantage of Plus Two is that it’s relatively simple, does not add a lot of administrative overhead, and allows the existing two-stage process to stay in place, with only a relatively minor change of adding works to the final ballot. It also adds the “human judgement element” that it appears that many people seem to think is necessary to combat bad-faith efforts to sabotage the Awards.

The biggest disadvantage is that it gives Administrators an authority that they’ve never actually had before, and that they have used only once before, without any explicit sanction. Administrators may be reluctant to serve on the Hugo Award Administration Subcommittee if they know that they may be responsible in some way for adding works to the Hugo Award ballot.

Conclusions

There are no “magic bullets” when it comes to tinkering with the Hugo Award rules. None of the proposals here is a perfect fix. Indeed, per Arrow’s Theorem, there is no such thing as a perfect voting system. However, we can try to move things around and minimize unfairness, usually at the expense of additional complexity. The political question then is how much complexity we can tolerate to improve perceived fairness.

The Instant Runoff Voting system that we use on the final ballot is a case of trading complexity (IRV boggles the minds of people who reject anything other than First Past the Post voting) for fairness (IRV usually returns the least-disliked candidate in an election, rather than the one with the largest plurality; in a field of more than two candidates like the Hugo Awards, it usually returns a consensus winner, not just a strong front-runner). Should we decide that the changes we’ve started with E Pluribus Hugo and 4/6 that are up for ratification this year are insufficient to tilt the field back toward perceived fairness, it behooves the members of WSFS meeting in Kansas City this summer to consider additional changes now, not later.

WSFS rules are intentionally complicated to change, in order to prevent concerns of a day or even of a single year to overwhelm the process. However, that doesn’t mean that we cannot start queuing up additional changes now while we continue to monitor how things proceed, in order to protect our own longer term interests.

I expect at least one of the proposals outlined here to be on the agenda of this year’s WSFS Business Meeting. We might even have all three of them.

815 thoughts on “Three Possible Hugo Voting Alternatives

  1. For those watching this thread: the new one will, I believe, go up soon. When it does, I think we should move discussion to there; it’s tough to follow multiple threads at once.

  2. Jameson, in the paper you said that ballots being 29% “all puppy” and 14% “at least half puppy” was roughly typical, and only 3% of the remaining ballots had less than half puppy picks, fewer than in other categories. That does make it more similar to your modeled slate vote (in which the preferences on the slate are not shared by any other voters).

    Perhaps only slightly more similar. But specific information would be useful in order to evaluate that.

  3. Version 0.2 of MSMC (Moderated Shortlist for Member Consideration – this really needs a better name!)

    On this run through I’ve tried to incorporate input provided by Camestros Felapton, Johan P, and Hampus Eckermann. Any misinterpretation is of course my own.

    With Camestros’ recommendation of extended eligibility, I’ve removed the prior WorldCon appointing the moderation panel.

    Additionally, I couldn’t square some of Johan’s input with the proposal. His input is great but would preclude (as I read it) the members seeing both the pre and post moderation versions of the shortlist. This would be fairer to the potential nominees. Unfortunately it would also mean membership accepting the moderated list on blind faith without seeing the other side of the coin. If anyone can square this I really like his ideas.

    I suspect actually calling out block voting in the removal section, as requested by Hampus, might also might be controversial and increase resistance at the Business Meeting. This could likely be dealt with by the expansion section and the other removal reasons. This needs to be a community doc though.

    >>>

    1) The current year’s WorldCon shall appoint a moderation panel (consisting of?) to review the nominations selected by the membership this year.

    2) Prior to publication the Hugo administrators shall provide the moderation panel the results of the nominating phase.

    3) The moderation panel may recommend removal of works which, in its sole judgement, are:

    (a) Ineligible due to not meeting necessary qualifications for the awards;
    (b) Intended to harass or defame individuals or groups;
    (c ) Intended to disrupt the process of the awards;
    (d) Placed on the ballot through cheating or fraud; or
    (e) Placed on the ballot through a block voting scheme.

    4) Removed works shall be replaced by the next highest finisher on the nomination list.

    5) Removed works shall have their eligibility for nomination automatically extended for a single one (1) year period.

    6) The moderation panel may recommend to expand the list of finalists, in one or more categories, with up to two slots for works that, in their sole judgement, were disadvantaged by procedural or voting irregularities.

    7) If the panel recommends to expand the list, the panel is encouraged to describe their reasons for this recommendation without naming the candidates involved.

    8) If the moderators recommend any removal of works or expansion of nomination slots they shall provide the moderated list to the administrators within (a timeframe)

    9) If the moderators present a moderated list to the administrators, the administrators shall present the members of this year’s WorldCon with memberships as of (a cutoff date) with both the moderated and unmoderated short lists.

    10) The administrators shall conduct a vote of those members within (a timeframe) to accept or reject the moderated list. The majority of votes shall determine which list, in its totality, to use for determining the Hugo awards for this WorldCon.

    11) The decision of which list to use shall be binding on the administrators.

    12) The administrators shall announce the Hugo finalists within (a timeframe) of this vote.

  4. Pingback: They’d Rather Free Ride: Hugos and Game Theory (Proposal Discussion Thread 4) | File 770

  5. @Stoic
    That is a mighty summary 🙂

    With this version traditionally inelligible works also get incorporated into the ratification steps Was this intentional? I’d assumed it would just be the other removed stuff but maybe it would be a popular idea.

  6. @Camestros

    Oops! No, I unintentionally opened a loophole. Fix it in version 0.3 if this gets traction? Or leave it if it’s a feature not a bug to most folks. In which case: of course that’s what I meant 🙂

    Moved a copy of 0.2 on to the new thread to try and keep discussion unified.

  7. Brian Z on May 22, 2016 at 9:19 am said:

    That do not include non-Members, or restrict which Members may participate? I meant the voting on the Hugo Award final ballot.

    Only members (attending and supporting) of the current Worldcon may vote on site selection. However, by your hedge, I think you’re excluding that because you do also have to buy a membership in Worldcon N+2 when you vote. That is, you have to be a member of the current Worldcon, and you have to pay your WSFS membership dues for Worldcon N+2 at the time you vote.

    Personally, I’ve voted every year now for more than 25 years, so I’m always buying 2 (previously 3) years ahead.

    No, I meant the times over the past year people wanted to restrict all Hugo voting to attending members. (I would guess you agree that is a bad idea.)

    I do.

    Kevin, you have drafted two proposals, DN and 3SV, which both remove voting rights.

    Nope. They create an entirely new election, not currently defined, and limit the electorate to the members of the current Worldcon. That’s not taking rights away from anyone, anymore than limiting voting on the final ballot to only members of the current Worldcon is taking rights away from anyone.

    Thought experiment: WSFS creates a European Regional SF Convention analogous to NASFIC. Membership is limited to current Worldcon (or potentially ERSFC depending on circumstances) members in the same way NASFiC elections are limited to Worldcon (or NASFiC, depending on circumstances) members. Nobody is having rights taken away from them, because it’s an election that never existed before.

    Aside: NASFiC is the one case where WSFS has set up an election where WSFS members (the members of Worldcon) aren’t eligible to vote, that being the case when a NASFiC selects the site of a future NASFiC; this could happen in 2019 if no more bids turn up for 2019-20. NASFiC members aren’t WSFS members, although they’re members of a WSFS-sanctioned convention.

    Yes, I can imagine that getting through the Business Meeting. I can imagine hundreds of people filing into the meeting in Kansas City specifically for that, and leaving the room as soon as it passed. I’m not being facetious.

    And you also think hundreds more will turn up in Helsinki to ratify it, then leave?

    I meant that talking about how being a member of WSFS and going to Worldcon every year is a very North American centric concept.

    Come back and say that again in 2020, when I expect that we will have had three out of four Worldcons outside of North America. Around that time, I’m pretty sure a bunch of my fellow Americans will be whining about the Death of Worldcon because we aren’t holding in the USA often enough.

    I know your attitude is well then cough up the annual dues, but the picture isn’t that simple.

    The membership dues are $50 this year and aren’t likely to be a lot more than that. I oppose limiting the franchise to only attending members.

    There is only a superficial resemblance. Nobody pays dues to WSFS. People who have joined a Worldcon become members of it for the year regardless of how they got their Worldcon membership. If it was a clear cut case of being supposed to pay annual dues, they would never have decided to start extending voting rights to last year’s members in the first place.

    Actually, the extension of rights to the previous year’s members was relating to dealing with a previous instance of unethical behavior and the reaction of WSFS to stop it. I’ve already explained this before, so I don’t feel then need to do it again.

    As I said before, perhaps if you were required to make two separate payments, $50 charged to WORLD SCIENCE FICTION SOCIETY and $160 to MASFFC INC, you’d see the distinction better. I don’t see this happening, however. It would be terribly inefficient, because WSFS would then need to pay $49 of that $50 to MASFFC to pay for the outsourced membership-administration services. (The tiny number of central things that individual Worldcons can’t do for themselves cost about $1/member/year.)

    Jim Henley on May 22, 2016 at 9:36 am said:

    1. Since what year have supporting members had nominating and/or voting rights?

    Since at least the early 1970s, and possibly earlier than that, but the documents are muddled. Ben Yalow would know. But certainly for at least fifty years.

    2. Since what year has it been possible to nominate for the Hugos online?

    Well, we were doing it in 2002 at ConJosé, but not much before that. I don’t remember who was the first one. Nine years earlier, ConFrancisco was considered cutting-edge by allowing members to fax their ballots.

    Bonus question: Since what year have online nomination forms predominated?

    The first year for which I can find a separate breakout is 2005: Nominating ballots 546 (436 electronic), Final ballot 684 ballots (552 electronic).

    So we can say that the overwhelming majority of ballots have been electronic for at least ten years.

  8. Kevin, thanks for these replies.

    That’s not taking rights away from anyone

    In plain English. They have the right to vote to choose the finalists. You are taking that away by allowing a fraction of the eligible nominators to invalidate the nominators’ choice of finalists. It’s not a new election. It’s taking the one stage election to determine the final ballot, splitting that election in two, then telling half (or whatever it may be) of the eligible voters to go jump in a lake.

    I’m pretty sure a bunch of my fellow Americans will be whining about the Death of Worldcon because we aren’t holding in the USA often enough.

    I agree. And you’ll tell THEM to pay “annual dues” in checks made out to conventions in distant lands they can’t hope to attend.

    There are plenty of legitimate reasons for not being a member every year, and it doesn’t make those fans less part of Worldcon fandom or make their contribution to choosing the finalists less valuable.

    Actually, the extension of rights to the previous year’s members was relating to dealing with a previous instance of unethical behavior and the reaction of WSFS to stop it.

    Why didn’t you just strictly reprimand those members for failing to keep up with their annual dues? And why extend the vote to future members when you could have just scolded them too?That a particular event provoked the discussions that led to one of those decisions is neither here nor there.

  9. BrianZ-you’d rather people were just grateful for whatever crumbs dropped from VD’s table. Theodore Bolton eating a sausage in front of Hugo Greyjoy.

  10. Brain Z, have you considered a career in politics? With your ultra-selective memory, impressive skill at misrepresentation, and absolute disregard for honesty, I think you’d do well.

  11. Brian, In plain English. They have the right to vote to choose the finalists.

    No. No, they do not. They have the right to NOMINATE finalists, some of which may, or may not, show up on the final ballot, based on what a whole bunch of other people are doing.

    The only people who have the right to “vote to choose the finalists” are the juries on juried awards.

  12. Pingback: To Say Nothing of the Dogs; or, How We Confound the Hugos’ Third Slump (Hugo voting proposal discussion 5) | File 770

Comments are closed.