Mini Hugo rocket carried into space and photgraphed by astronaut Kjell Lindgren in 2015.
Apart from the discussions Jameson Quinn has been leading here, another group of fans has been working on ideas for reforming the Hugo voting process. Yesterday they published the drafts of their three main motions and an amendment to EPH (given its first passage last year) as a Google document.
The three main motions do these things:
(1) Change the deadline you must be a Worldcon member to be eligible to nominate from January 31 to December 31 of the previous year.
(2) Restrict eligibility to nominate to members of the current and preceding Worldcon.
(3) Add a second round that allows members to vote out something that makes the initial long list (“Three Stage Voting”).
Colin Harris (co-chair of the 2005 Worldcon), Warren Buff, Kevin Standlee (co-chair of the 2002 Worldcon), Nicholas Whyte, and Colette Fozard each sponsor at least one of the several motions. Harris explains:
We plan to submit the motions officially in about a week; we are publishing them now to encourage discussion, rather than because we expect to change the text — but of course if people point out important things we’ve missed, we’ll take the opportunity to fix any issues.
Commenting specifically about the Three-Stage-Voting proposal, Harris says:
To be clear, my stance as the main mover on 3SV is simple. I wish this change was not necessary, but I believe that EPH and the other proposals already in hand will not achieve the necessary outcomes. In particular, I believe that guaranteeing a couple of broadly acceptable finalists per category is simply not a high enough bar for “success” in restoring the integrity, reputation and stability of the awards. I do not know if 3SV will pass, but I believe that the Business Meeting should have the opportunity to discuss this more direct option for tackling manipulation of the nomination process.
The text of the proposals follows the jump.
PROPOSALS TO REVISE THE HUGO VOTING PROCESS
This document provides an advance look at a set of related motions which are being submitted to the 2016 WSFS Business Meeting in relation to the Hugo voting process. The motions have been developed by a small discussion group including Worldcon and NASFiC Chairs and Hugo Award Administrators.
Three separate proposals are presented, plus an associated amendment to the EPH ratification provision. The motions are essentially independent in effect although they all relate to the Hugo voting process.
- December is Good Enough (move the Nomination Ballot eligibility cut-off date from January 31 to December 31 of the previous year)
- Two Years are Enough (limit Nomination to members of the current and preceding Worldcon).
- Three Stage Voting (3SV), or “The Only Winning Move is Not to Play”
- Amendment to EPH
SHORT TITLE: DECEMBER IS GOOD ENOUGH
Moved, to amend the WSFS Constitution by striking out and inserting text as follows:
3.7.1: The Worldcon Committee shall conduct a poll to select the finalists for the Award voting. Each member of the administering Worldcon, the immediately preceding Worldcon, or the immediately following Worldcon as of January 31 the end of the current previous calendar year shall be allowed to make up to five (5) equally weighted nominations in every category.
- Proposed by: Colette Fozard, Warren Buff, Nicholas Whyte
Discussion:
The administrative strain on Hugo Administrators has increased substantially in recent years, due to:
(a) extension of nominating rights to include members of the year N+1 Worldcon – requiring the Administrator to merge three sets of membership data, two of which are continually changing as new members join the respective conventions.
(b) a substantial rise in the number of Supporting members due to increased interest in the Awards.
Members also expect to receive their nominating rights (and PINs) rapidly after the ballot opens – typically in early January – while the Administrator has to continue merging membership data from that point until the eligibility deadline, with no control of the timeliness of the data feed from Worldcon N+1.
Changing the eligibility deadline will reduce this strain and ease the administration process. In addition, in the event that 3 Stage Voting passes, an earlier eligibility cut-off will ensure that the overall schedule remains manageable (e.g. for a nomination phase of 8 weeks starting on January 1, then a December 31 cut-off is significantly more robust and easier to handle than January 31).
SHORT TITLE: TWO YEARS ARE ENOUGH
Moved, to amend the WSFS Constitution by striking out and inserting text as follows:
3.7.1: The Worldcon Committee shall conduct a poll to select the finalists for the Award voting. Each member of the administering Worldcon, or the immediately preceding Worldcon, or the immediately following Worldcon as of January 31 of the current calendar year shall be allowed to make up to five (5) equally weighted nominations in every category.
Provided that members of the 2019 Worldcon will retain their nominating rights in the 2018 Hugo Awards.
- Proposed by: Warren Buff, Colin Harris
Discussion:
The extension of the nominating franchise from two Worldcons (N and N-1) to three by including year N+1 was implemented as part of the drive to expand the voting population at a time when member participation in the Hugos was substantially lower than it is now. This change has created some undesirable side effects, and we believe the most appropriate response is to return to the previous arrangement.
The inclusion of year N+1 creates a very high burden on the Administrators since (a) they have to merge three sets of membership data rather than two (b) two of these are in motion with new members joining up to the eligibility cut-off date (c) the most recently seated Worldcon is often still establishing robust data and membership processes at the point where the current Worldcon needs a reliable flow of quality data.
The removal of year N+1 will remove this burden while only marginally reducing the size of the voting pool. In a typical recent year, Worldcon N-1 might have 6,000 members at the eligibility cut-off, Worldcon N might have 3,500, and Worldcon N+1 might have 2,000 – but in the early months after being seated, Worldcon N+1’s membership will be dominated by people who participated in Site Selection and hence are also members of Worldcon N.
This change would also mean that individuals who only wish to join to participate in the nomination process would have to purchase memberships every two years rather than every three.
SHORT TITLE: THREE STAGE VOTING (3SV), OR “THE ONLY WINNING MOVE IS NOT TO PLAY”
Moved, to amend Section 3.7 (Nominations) and Section 3.8 (Tallying of Nominations) for the purpose of creating an intermediate stage in the Hugo Award selection process by striking out and inserting text as follows.
Section 3.7: Nominations.
3.7.1: The Worldcon Committee shall conduct a two-stage poll to select the finalists for the Award voting. Each In the Nominating stage, each member of the administering Worldcon, the immediately preceding Worldcon, or the immediately following Worldcon as of January 31 of the current calendar year shall be allowed to make up to five (5) equally weighted nominations in every category.
3.7.2: The Committee shall include with each nomination ballot a copy of Article 3 of the WSFS Constitution and any applicable extensions of eligibility under Sections 3.4.
3.7.3: Nominations shall be solicited only for the Hugo Awards and the John W. Campbell Award for Best New Writer.
3.7.4 3.8.2: The Worldcon Committee shall determine the eligibility of nominees and assignment to the proper category of works nominated in more than one category.
3.7.5 3.8.3: Any nominations for “No Award” shall be disregarded.
3.7.6 3.8.4: If a nominee appears on a nomination ballot more than once in any one category, only one nomination shall be counted in that category.
3.7.7 3.8.6: The Committee shall move a nomination from another category to the work’s default category only if the member has made fewer than five (5) nominations in the default category.
3.7.8 3.8.7: If a work receives a nomination in its default category, and if the Committee relocates the work under its authority under subsection 3.2.7 or 3.2.8, the Committee shall count the nomination even if the member already has made five (5) nominations in the more-appropriate category
Section 3.8: Tallying of Nominations. Qualification Stage.
3.8.1: Except as provided below, the final Award ballots shall list in each category the five eligible nominees receiving the most nominations. If there is a tie including fifth place, all the tied eligible nominees shall be listed.
[3.8.2, 3.8.3, 3.8.4, 3.8.6, and 3.8.7 moved to Section 3.7.]
3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.
3.8.1: The Qualification stage of the process shall be based on a long list of the top fifteen Qualifiers (including ties that include fifteenth place) from the nomination process in each category. Only WSFS members may vote in this stage.
3.8.2: The purpose of the Qualification Stage is to allow the membership to confirm their willingness to see each Qualifier taken forward as a potential Hugo Award Finalist.
3.8.3: In the Qualification Stage ballot, each voter may choose between the options “Accept”, “Reject”, and “Abstain” for each Qualifier in each category.
3.8.4: A Qualifier shall be eliminated from consideration for the Final Ballot if it meets the following two criteria:
(1) the number of “Reject” votes is at least 60% of the combined total of “Accept” and “Reject” votes;
(2) the number of “Reject” votes is at least the higher of 600 or 20% of the number of eligible voters.
3.8.5: The final Award ballots shall list in each category the five eligible Qualifiers who received the most nominations in the first stage Nominating Ballot and were not eliminated from consideration in the Qualification Stage. If there is a tie including fifth place, all the tied eligible nominees shall be listed.
Provided that unless this amendment is re-ratified by the 2023 Business Meeting, Sections 3.7 and 3.8 shall revert to their wording prior to the initial ratification of this amendment, and
Provided that the question of re-ratification shall be automatically be placed on the agenda of the 2023 Business Meeting with any constitutional amendments awaiting ratification.
- Proposed By: Colin Harris, Kevin Standlee, Nicholas Whyte, Colette Fozard, Warren Buff
Discussion:
The essential argument for this change is that it enables us to directly address slates and bad actors in a direct way – and a way that statistical solutions such as EPH and 4+6 cannot. We have put it forward in recognition of the fact that the reputation and integrity of the Hugo Awards is under sustained attack, and in the belief that a response is needed to this reality. We would of course have preferred it if such a response was not necessary. The rationale for the proposal is as follows.
- Statistical solutions may reduce the number of slate nominees; but not sufficiently to act as a deterrent to slate campaigners or bad actors. Ensuring 1-2 non-slate candidates per category is ultimately a pyrrhic victory in terms of the reputation of the Awards; securing 2-3 finalists per category offers a strong incentive for slate campaigners to continue their efforts indefinitely, and continues to exclude nominees that would otherwise have made the Final Ballot from being recognised.
- We have seen that the membership is willing to use No Award to reject finalists that they consider are not appropriate winners of the Hugo Award. However this test is applied when the damage is done. The proposal essentially moves the No Award test to an earlier stage of the process. The Qualification Ballot is not a ranking ballot, but specifically enables members to reject candidates that they believe have benefited from inappropriate promotion. (It is important that the Qualification Ballot is presented clearly as one in which “Reject” should be used only for candidates that members feel are not suitable for inclusion on the Final Ballot – typically due to abuse of process – and not as an opportunity to express preference among the acceptable candidates. The wording of the Qualification Ballot rubric will therefore need to be clearly presented.
- It is similarly important to note that the Qualification Ballot is not used to re-rank the candidates for inclusion on the Final Ballot. Once any rejected candidates have been eliminated, it is the ranking from the original Nomination Ballot that determines the ranking and hence which finalists should be taken forward.
Why now? We have been asked why we have brought this proposal forward now, when EPH has yet to be ratified and tested. Our concern is that the Awards are being progressively tainted, and that we cannot afford to wait until EPH has been tried before having a “Plan B” under way. (Assuming EPH is ratified in 2016 and is used in 2017, and does not resolve concerns satisfactorily, a new approach passed in 2017 and ratified in 2018 would only be in effect in 2019.)
Our suggestion is that 3SV should be passed now to ensure that there is a strong option in hand for protecting the future of the Awards. If this option is found to be unnecessary once the 2017 nomination process has been run, then 3SV need not be ratified.
If 3SV is felt to be necessary – to send a message that WSFS decisively rejects attempts to game the awards and will single out and reject candidates that gain from such tactics – then there will be a separate decision on whether EPH should continue to be retained as well. While EPH is a sophisticated algorithm, it would arguably be redundant in rejecting slates if 3SV is in place (and indeed, 3SV makes it easier to separate tactical slates from genuinely correlated votes such as fans of a TV series nominating multiple episodes of that show). Moreover, it imposes a significant administrative burden and is a less transparent algorithm for members to understand.
Our view is that the other measures which are up for ratification i.e. 4+6 and Nominee Diversity – would combine with 3SV to provide a robust way forward for the Awards, and that EPH should potentially be dropped if it does not materially resolve the slate issue. As such we have introduced an amendment for additional re-ratification of EPH in 2017, such that the Business Meeting could choose to drop EPH if it proceeds with 3SV. The BM may of course choose to continue with both solutions – they are complementary – but we feel that this option should be available to the meeting in Helsinki.
AMENDMENT TO EPH
Moved, to substitute the following new enacting clauses for the existing clauses in the E Pluribus Hugo Proposal to require annual re-ratification until 2022:
Moved, that unless this amendment is re-ratified by the 2022 Business Meeting, Section 3.A shall be repealed, and
Provided that the question of re-ratification shall be automatically be placed on the agenda of the 2022 Business Meeting with any constitutional amendments awaiting ratification.
Provided that unless this amendment is re-ratified by every Business Meeting between the initial ratification of this amendment and the 2022 Business Meeting, Section 3.A shall be repealed, and
Provided that the question of re-ratification shall be automatically placed on the agenda of each Business Meeting between now and the 2022 Business Meeting with any constitutional amendments awaiting ratification, unless any of those meetings should fail to re-ratify the amendment, in which case no further re-ratification votes shall be held.
- Proposed By: Colin Harris, Kevin Standlee
Discussion:
This is an adjunct to the 3SV motion. It provides additional opportunities to repeal EPH in the event that the Business Meeting decides to adopt 3SV (or an alternative solution) in place of EPH as a means to address the issues experienced in recent Hugo ballots. EPH can of course be retained alongside such additional measures, but the Business Meeting may not wish to do so in view of the administrative load and the loss of voter understanding associated with the complex algorithm.
Discover more from File 770
Subscribe to get the latest posts to your email.
@Andrew M
If it was three months (Feb, Mar, Apr) instead of two weeks, would that change your mind?
The only hole I can see is if most members didn’t bother to pick 5 nominees from the top-15 list. What am I missing?
Yes, that’s how I saw it. They aren’t different problems: the reason we may not be able to outvote the abuse is that not enough organic voters will vote at this stage, and the reason for that is that they won’t be able to read the stuff in time.
@StephenfromOttawa
That’s my other worry with this version of 3SV: have they set the bar so high that no work will ever end up getting rejected? I fear very much that we could pass it and then two years later learn that it didn’t work at all.
@Andrew M
If you allow year-round nominations for the reading list (aka the top-15 list), then generate that list on February 1, it seems it should be easy to allow three months for members to evaluate the top-15 list, and then allow three months for the final vote.
If it was three months (Feb, Mar, Apr) instead of two weeks, would that change your mind?
What would that be cutting time off – nomination, or final voting?
The absence of a Hugo packet (yes, I know we’re not entitled to one), which means we might have difficulty getting hold of some of these things, might also be a problem.
David Goldfarb:
EPH is not that hard. Do you really intend to claim that most members of Worldcon are stupid?
It is not actually hard to understand if one thinks about it. It is, for some people, hard to think about, hard to get in the right frame of mind – just as some people have difficulty reading a page full of symbols, even if they understand every symbol.
That not everybody did understand it is suggested by the cries of amazement that came from many quarters when it was demonstrated that it does not in fact produce strictly proportional results – even though its leading proponents had never claimed it did. The widespread expectation that it would suggests that much of the support for it was, as Kevin says, a matter of faith.
@Andrew M
I’m thinking close nominations for the top-15 list at the end of January and then open voting on the top-5 finalist list as soon as possible in February. Close voting for the top-5 finalist list at the end of April (as it is done now), announce the finalists and open final voting for the awards as soon as possible in May. Close voting at the end of July. Announce the winners in August.
In this view, nominations for the top 15 would run year round, from February to February, and any member could nominate as many works per category as he/she wanted to.
Three months is a lot of time to do that, though. It’s also enough time for authors and magazines to make stories temporarily available, as some have done this year. And, of course, lots of stuff is free online already.
@Andrew M
I think it’s a little worse than that. The proponents repeated over and over that if the slate voters amounted to 20% of the votes, then they’d get 20% of the nominees. So the discovery that, even with EPH, a 10% minority can get 80% of the nominations is pretty shocking. I expect there will be a lot of angry people at MidAmeriCon 2 who feel they were lied to.
I think the mistake is that the proponents predicted results based on how voting systems similar to EPH work in actual legislatures. What they neglected was to consider what happens in a race where there is only one actual party and everyone else is an independent. I’m not sure how they failed to realize this, though, since it’s easy to show the limits of EPH vs. the size of a slate given just the publicly available info on the Hugo website. I guess no one was trying very hard to break it. And, of course, almost everyone (me included) really, really wanted to believe in it.
And here is where I point out that 3SV, is by design, a way of adding human judgement to the process. If you are proposing authorizing the Hugo admin to explicitly disqualify works, I agree that is unpalatable to the BM voters at this time.
While I don’t know that admin disqualification will ever become palatable, some of the ideas currently on the table were unpalatable to a bunch of people at last year’s Business Meeting who are now thinking about voting for them.
There is a well-known track record of democracies under stress moving to adopt severe measures they were reluctant to use at the outset. Depending on which side of history one is looking at, it can look like Winston Churchill becoming PM, or like the Peloponnesian Wars.
@Christopher Hensley
Tossing out nominations is not quite where my preference would be. I think empowering the Admins, or some other panel, to expand the shortlist by the top 2 to 3 non-slate entries in years with voting irregularities would be the way to go. I’ve argued my reasons for preferring that solution in earlier threads. There is no appetite for that solution so it’s not worth re-litigating.
@Stoic Cynic
Isn’t that going to be on the agenda at the business meeting this year? I thought Kevin showed it to us a few weeks ago.
It occurs to me that this together with EPH+ might, together, guarantee 5 organic nominees in every category.
@Greg Hullender
Kevin was actually the first person I saw bring it up months ago. It’s not his preferred solution but, as I understand it, one he was asked to help draft language for. He was looking for sponsors, on other folks behalf, a couple weeks back. Based on reaction here I think it has no shot though. It’s probably better to focus on improving 3SV which I think has some momentum.
I think it’s a little worse than that. The proponents repeated over and over that if the slate voters amounted to 20% of the votes, then they’d get 20% of the nominees. So the discovery that, even with EPH, a 10% minority can get 80% of the nominations is pretty shocking. I expect there will be a lot of angry people at MidAmeriCon 2 who feel they were lied to.
Do you have documentation on that? Felice’s figures, showing that EPH was not guaranteed to produce a proportional result, were available before last year’s business meeting. I didn’t see any one saying that these figures were wrong, though some people suggested they were based on an unrealistic scenario. The general response seemed to be that EPH was better than what we had now, it was likely to prevent a sweep, and that should be sufficient, and anyway it was our only hope. The response was the same when the preliminary results of Bruce and Jameson’s paper were leaked earlier this year. So it was a bit weird that when the paper was actually published, there was such widespread shock.
Keith Watt explicitly recognised that EPH would not be effective in every circumstance, saying ‘I’m assuming that fandom will have a preference’, and suggesting that if it didn’t the slaters, who do have a preference, deserved to win. I don’t agree with this, first because in short fiction categories it’s clear that fandom has difficulty reaching a preference, and second because it ignores the fact that slate votes don’t represent a real preference. But he was not misrepresenting the capacities of EPH.
(There’s also the constant ‘No one is claiming that EPH is a panacea’ that we were greeted with throughout the year – though some people clearly were. And VD, for what it’s worth, clearly knew what its real effect would be, since he said its purpose was to ensure that TOR got at least one nominee.)
It’s certainly true that at the time it was first adopted EPH was being defended with an almost religious fervour, and there was a strong sense that people who raised questions about ti were being irresponsible, because it was our only hope. I was reminded of the argument discussed by J.L. Mackie:
If the enemy is advancing in overwhelming numbers, we should retreat.
But we should not retreat, because that would mean letting down our allies.
Therefore, the enemy is not advancing in overwhelming numbers.
The publication of the paper led to what seemed to me an equally wild stampede away from EPH – though it certainly remains possible that, along with increased participation and decreasing enthusiasm from slaters, it will be enough to do the trick.
There has been a lot of proposals over time and after a while, some kind of consensus grew. Also, there was an agreement on not putting out too many proposals.
To now, with just a few weeks to spare, start discussing other alternatives is mostly a waste of time. That train has passed.
@Andrew M.
I don’t think you’re intentionally trying to rewrite history, but I assure you I saw proportionality cited over and over.
While it doesn’t explicitly state proportionality, this part of the Sasquan presentation on EPH certainly implies better results than testing with newer data is showing:
The only way for a slate (or even a group of slates) to sweep a category is to have a majority of the total number of nomination ballots – in which case they deserve to win.
Even your post states Felice’s scenario was dismissed as an edge case.
Strong claims were made for EPH. They were made in good faith and with backing from tests using the ’84 data. There’s nothing shameful in that. People represented what they thought was correct. They were just wrong. Stating the claims weren’t made really stretches credibility though.
Andrew M on July 23, 2016 at 2:08 pm said:
Sure. It’s not hard to find. Have a look at the E Pluribus Hugo Slide Deck from Sasquan. The third-to-last slide says:
The first paragraphs is a pretty clear declaration of proportionality. And, of course, as we have seen, even with fewer than 10% of the ballots a slate can sweep an entire category.
For another example, here is Jameson Quinn on Making Light saying that EPH prevents slates from getting more than their fair share.
@Stoic Cynic
I agree entirely. I don’t think anyone deliberately lied. But I do think it’s important for people to admit to their mistakes, one of which was grossly over-hyping what EPH could actually do.
Here’s Nick White, also stating that EPH would produce proportional results. And here’s Patrick May, claiming he ran experiments on the 1985 data which confirmed that EPH really did produce proportional results.
Heck, there’s probably a post somewhere where I make that claim myself–based on the other things I had read. I certainly believed it. I didn’t realize it wasn’t going to work that way until a couple of weeks after Sasquan was over when I studied the detailed nomination stats.
So, yeah, 75% of the business meeting voted to adopt EPH on the strength of testimony by some really smart people who assured them it would work proportionately. Expect people to be upset. “We stuck this super-complicated thing into our rules on your say-so and you’re telling me it doesn’t even work like you said it would?” People have a right to be upset.
See that proposal up there titled “Amendment to EPH?” That’s what “upset” looks like. It really means “Repeal EPH ASAP.”
Unfortunately, 3SV, as described above, is also a test of faith. There’s no way to tell if it will work at all without actually doing it.
• The only way for a slate (or even a group of
slates) to sweep a category is to have a
majority of the total number of nomination
ballots – in which case they deserve to win.
OK, that’s weird. If a slate gets two hundred votes, a non-slate nominee needs at least forty to beat it. That’s a very straightforward point, and doesn’t need any modelling to show it. And it’s a well-known fact that in some categories things can be finalists with less than forty votes. I don’t see how they can have been unaware of that. Can they really have been assuming that the five best non-slate candidates split up the rest of the votes between them? But there were figures they had all seen that showed that this was not true.
The other points don’t seem to me to be decisive. It’s true that a slate’s percentage of spots on the ballot will increase as its percentage of votes increases (other things being equal, obviously), and that doesn’t need to be taken as asserting proportionality. The other line is about what EPH is designed to do.
The 1984 analysis said nothing about how EPH would handle slates, and no one thought it did; it was addressed to a different question, whether EPH would significantly perturb the results with a normal distribution.
Some people on Making Light thought that Felice’s results were atypical; the normal response here was to say that they did not matter. Everyone knew that EPH would allow slates to get some spots, so what was the fuss about?
I have repeatedly raised this issue throughout the year, and I have never got a flat ‘no, you are wrong’. What I have got is:
‘Preventing sweeps should be enough.’
‘It will work in conjunction with increased participation.’
‘Well, have you a better idea?’
In any case, I think this is enough to show that how EPH works is not totally perspicuous.
Oh – here’s one thing that may change the issue: ‘they only get a sweep if they get a majority’ is not the same as strict, or even vague, proportionality. If a slate gains 51% of the votes and gets a sweep, then 49% of the voters go unrepresented.
Now while it’s clearly not strictly true that you can only get a sweep under EPH if you get a majority, it is true that sweeps are unlikely under normal distributions (and in fact none would have occurred last year). If they were making some assumptions about normality they may have felt entitled to assert this. But that’s a much weaker claim than that 20% of the votes gets you 20% of the ballot.
And Patrick May’s result confirms this: he has a slate with 40% of the votes getting 60% of the ballot – that’s not proportionality. He is still underestimating the strength of slates, since (as Brad Templeton points out in comments) votes in 1985 were much less diffused than they are now. But he’s not claiming strict proportionality.
@Andrew M
I believed that too, up until “Space Raptor.” Now I’m less sure, although who knows? Perhaps it really will discourage the slaters enough that they go bother someone else. It’s a lot of work to keep doing just to get a bunch of No Award votes.
Of course if one of their non-hostage nominees actually wins this year, I think they’ll be fired up like never before.
@Andrew M
All I’ll say to that is that anyone going to MidAmeriCon II and trying to claim that EPH didn’t really promise proportionality should not expect that to be well-received.
We have gross homophobia on the current ballot. We have false rape accusations on the current ballot. Preventing sweeps is absolutely not enough. What we need is a way to also prevent abuse and harassment.
That is what 3SV is.
Well said.
@Hampus Eckerman
All of which vileness will be on display in the qualifying list. The qualifying list itself will receive it’s share of publicity and promotion simply because it becomes part of the public voting process. It waters down the issue since at least they’re not nominees, just qualifiers, but still it doesn’t fully solve that problem. That doesn’t mean 3SV doesn’t deserve support but let’s realize it’s limitations too.
I’d like to speak to a number of points that have been discussed here already. I’ll do so in separate posts. This one is just “ticky”.
@Hampus Eckerman
Okay, but I don’t think the proposals described in the article above are really going to work. Unless it’s amended, I expect I’ll vote against 3SV.
Maybe. And maybe not. One thing is sure. The abusive items will not be read during the Hugo ceremony. And they will not be proudly displayed on the Wikipedia page, as they aren’t finalists.
@Hampus Eckerman
Well there is the ceremony point (which I really hadn’t thought about) 🙂 Wiki, on the other hand, I’d be less certain of.
(I just mistakenly posted the below without putting my name and it got caught in moderation. Please discard the duplicate.)
First off: was EPH oversold? Short answer: somewhat, but not as badly as some people are saying here.
I, personally, was careful never to say that EPH would achieve perfect proportionality for slates (and I that includes the ML post linked above). But the fact is that in order to describe EPH, you have to give the design motivations, so you have to introduce the idea of proportionality. It’s not at all surprising that when you say “EPH was designed around proportionality, and would give results that are more proportional than the current system”, some people walk away thinking you said “EPH will give proportional results”.
I’d like to specifically address the claim in the slide deck: “The only way for a slate (or even a group of slates) to sweep a category is to have a majority of the total number of nomination ballots – in which case they deserve to win.” There should definitely have been a technical footnote to explain our thinking there, and I’m sorry that I left that out. That footnote would have said: “This assumes that the top organic nominee in each category will be supported by at least 10% of the voters (assuming naive slating) or about 12% (assuming complicated vote-management by the slate; the percentage threshold is in this case actually 250/21=11.9; the slate strategy is to have 11.9% support 5 works and the other 38.1% support 4). This is because a slate with less than 50% of the vote will be giving less than 10% points to each of its 5 items until one is eliminated.”
As it turns out, those unstated assumptions were too optimistic in a few categories such as Fan Artist, but not wildly so.
The larger point is that yes, I expected EPH would work slightly better than my subsequent testing has suggested it will. That’s why I’m now suggesting strengthening it to EPH+. But I never claimed it would guarantee perfect proportionality, and if I’d heard anybody else claiming that I would have corrected them. The long tail issue is slightly more serious than I imagined, but it was totally obvious from the start that it would be a factor.
If we pass 3SV, would we still need EPH or EPH+? I’d say the answer is strongly yes. Others have discussed how in past years 300 slate voters split three ways would have been enough to dominate or nearly-dominate the longlist in most categories. But even if it doesn’t go that far, slates would still be able to have an outsized influence on who gets to be finalists, as long as they nominated reasonably mainstream works (such as Seveneves this year). This is because the whole point of 3SV is that it doesn’t change how finalists are decided, it just makes some things ineligible. For something like Seveneves, which I think most people here would agree is not outright ineligible, slate votes could easily make sure it was a finalist even if (hypothetically) it had almost no organic support. And if the slate voters chose 5 things like that in a category, they could sweep, ensuring that whatever kind of thing they don’t like would never get a Hugo. I think that’s clearly not a healthy scenario.
So, as to the proposals: I think “December Enough” and “Two Enough” are fine, I support 3SV (though I’d suggest a tiny edit or two; I’ll get to that later), and I strongly oppose the “Amendment to EPH” as written.
Taking up the “amendment” first: I think it’s pretty clear that the intent of this proposal is to ensure that EPH is revoked ASAP. As I’ve said, I think that’s a bad idea, even if 3SV is in place; so I strongly oppose this amendment.
But even if you think we only need either 3SV or EPH but not both, the “amendment” is problematic. For one thing: why should EPH require annual re-ratification while 3SV doesn’t? From the voter perspective, EPH is transparent, while 3SV tightens the schedule, adds another round of voting, and requires engaging with a longlist of works one might not have read. For another: why should it take just 1 year to permanently kill EPH, given that it would take 2 years to reinstate it?
If these two objections were addressed, I’d go from strongly opposing this amendment to merely opposing it.
Now, as to 3SV.
I think that this basic idea is necessary. There’s some finalists this year that I think make the case pretty well; they simply have no business on the final ballot, period. The only ways to eliminate them would be through some kind of jury — which clearly will not fly as a proposal — or through something like 3SV.
I think that if you’re doing a second round of voting, you should be able to additionally refine the results from the first round, without just eliminating them. This would mean a separate amendment, something like “+2 against trolls” that’s been discussed on prior threads here. But there are enough complicated proposals that will go to the BM this year, so I think it’s fine to put that discussion off until next year.
In terms of the proposal text above: I think that the threshold for rejection (60% AND at least 600 AND at least 20% of eligible voters) is too high. I think it would be hard to mobilize those numbers on short notice even this year, when attention is high. I’d suggest 50% and 500 and 1/6 of eligible voters. While I see the point of the supermajority 60% requirement (allowing countermobilization to save a work), it just ends up complicating things.
That’s all I have to say about these proposals, in case anybody is waiting until I’m done to respond. I do have a happy announcement about my fundraiser, but I’m going to ask Mike to put that on the front page. (Mike, I’m sorry I haven’t gotten back to you about that sooner. I was trying to get in touch with the MAC scheduling people first, but I should not hold things up for that. Is it still OK to write something for the front page about this?)
Jameson Quinn: Any time you like!
@Standlee said:
Sorry, I disagree.
IRV, without the showdown rule, is a process of elimination. At each round, whichever option gets the fewest first choice or transferred votes is permanently eliminated, even if it happens to be second preference on every ballot.
Imagine a situation where there were both slate and non-slate works on the ballot, and every non-slate voter ranked at least one of the non-slate options above “no award”. In that case, “no award” would be the first choice of literally nobody, so it would be the first to be eliminated. Once that had happened, it could not win without the “showdown” rule.
I agree that the combination of IRV and the “showdown” rule is needlessly complex. But the solution would not be to remove the showdown rule. In fact, if it were up to me, I’d simply replace IRV with Majority Judgment (with a variant tiebreaker).
In Majority Judgment, each voter rates each work using a number of rating categories. For the Hugos, those categories might be: “Outstandingly worthy”, “Highly worthy”, “Worthy”, or “Unworthy”. You can rate multiple works at the same level and/or skip levels between works. (It’s easiest for voters if there are 4-6 category gradations, but any number is fine). To find the winner, you first tally all the highest-level ratings (“Outstandingly worthy”), and see if any work has a majority. If not, you add the next level (“Highly worthy”) to the tally, and again check if any work has a majority. You continue in this fashion until either one or more works have a majority or the only category left to tally is “unworthy”, in which latter case there is no award in that category. If two works reach a majority at the same threshold, you find which of them has the higher score, where the score is calculated by giving 1 point for every rating above the threshold, subtracting 1 point for every rating below, and dividing by the total number of voters who did not abstain on that work. (This will give a score between -1 and 1. This tiebreaker is actually “graduated majority judgment”; the standard tiebreaker originally proposed by the inventors of majority judgment will almost always give the same answer but is slightly harder to explain.)
MJ has several advantages over IRV:
1. In some cases, IRV can prematurely eliminate a candidate for having the lowest first-choice support, even if that candidate has a lot of second-choice support and would be the best compromise winner.
2. MJ does not force you to make fine distinctions between works you consider of roughly equal merit. I’d argue that insofar as the IRV outcome hinges on such basically-arbitrary decisions, it’s less democratic.
3. MJ is immune to the strange nonmonotonicity of IRV (as shown in the pictures here and the simulation toy here). This also means it’s less likely to hinge on a single ballot.
4. MJ is actually easier to explain and to count.
5. MJ, unlike IRV, actually meets the favorite betrayal criterion; it’s never strategically a bad idea to give your favorite work the highest rating. (With IRV, you might in some cases do best by giving a compromise work the highest rank, to prevent it from getting prematurely eliminated.)
….
I’m not suggesting that this should be proposed this year or even ever. MJ is better than IRV, but IRV seems to be up to the job, so there’s no pressing need to change it. But if you are upset about having a strange extra “showdown” rule for “No Award”, I think this would be the right way to fix that problem.
To show the problem with Kevin’s suggestion, imagine the following ballots:
800 voters: The Day After Tomorrow is Almost Enough > Yesterday’s Neices > No Award > Gibbous Moon
700 voters: TDATIAE > No Award > YN > GM
500 voters: YN > TDATIAE > No Award > GM
500 voters: YN > No Award > TDATIAE > GM
1500 voters: GM > No Award > TDATIAE > YM
Basically, what’s going on here is that there’s no overlap between GM fans and others (perhaps GM is a slate work), while TDATIAE and YN fans have some overlap but some people who rate only one of the two above No Award. With pure IRV, No Award is eliminated right away, then YN, leaving TDATIAE to win. But actually, a majority of voters (2000 of the 3500) rated No Award above TDATIAE.
@Hampus Eckerman
I’m all for it if it can be made to work. And if it doesn’t create an opportunity for the slaters to remove works they consider “abusive” because their authors were people of color. As drafted, I’m pretty sure 3SV satisfies that last condition, but that’s about it.
Greg Hullender on July 23, 2016 at 1:52 pm said:
There will be a proposal called Additional Finalists (because the originally desired title of “Hugo Plus Two” appears to be one of the variations coming out of the discussions here and has nothing to do with it). This is the one that allows the Worldcon Committee (in practice, the Hugo Award Administrators) to add up to two additional finalists picked from among the Section 3.11.4 “Top 15” list. It does not require the Committee to add finalists, nor does it require them to pick the next two nominees after the first five places are placed on the final ballot. It adds an element of human discretion to the process, to be used or not as the Committee decides.
The Committee would not be obliged to say whether they exercised their discretion (there are sometimes more than five finalists due to ties). You’d know when the Section 3.11.4 list was published after the Hugo Awards Ceremony. The Committee would not be required to say why they picked (or did not pick) any particular nominee as a an additional finalist.
I drafted this proposal (as I generally will do for anyone who asks) at the request of Lisa Hayes (my wife). My name is not on it. This should not be taken as either my endorsement or opposition to it. I’m very cautious as to what proposals I co-sponsor the year before I am presiding over the Busienss Meeting, inasmuch as I obviously have to recuse myself from presiding over the ratification debate and vote on items I’ve co-sponsored. Lisa is not a proxy for me, nor I for her.
Multiple Nominations is a proposal presented for those people who insist that we must have subjective human judgement inserted into the process. Now it doesn’t go as far as some want, in that it doesn’t say, “The Committee can arbitrarily disqualify any finalist they choose for any reason whatsoever,” but instead adopts a variation of what one Worldcon committee did (1989) when they decided that someone was playing games with the nomination process, and added an extra finalist to the ballot without any constitutional sanction.
Whether it has any chance is obviously debatable, but if it fails, the next time someone says, “You need a Human Judge to Make Hard Decisions,” we can say, “This was proposed, and the members of WSFS rejected the idea.”
Greg Hullender on July 23, 2016 at 3:19 pm said:
Nope. You can’t repeal something that hasn’t been ratified. In fact, if those of us who have put forward that amendment felt strongly that EPH would never work, we’d not bother; we’d just vote to try and defeat the ratification. What we do think is that if EPH doesn’t work the way people seem to think it will under true field conditions, we’re going to want to be able to repeal it immediately, not in two years, and certainly not in five. This gives EPH the five years originally planned, but requires that it show itself annually, and does not leave us with a parliamentary albatross around our necks until 2022 should it prove to be unworkable in practice.
Hampus Eckerman on July 23, 2016 at 4:56 pm said:
I can say with authority that the Hugo Awards web site does not intend to include the full list of qualifiers in the permanent record, because, as Hampus says, the qualifiers are not finalists. That doesn’t mean the qualifiers won’t appear somewhere on the web site, because we include Hugo announcements and the 3.11.4 “Top 15” lists, but they won’t be given the prominence given to the finalists and winners.
Jameson Quinn on July 23, 2016 at 5:46 pm said:
I’m pretty sure you mean a majority rather than 50%, but in any event, there’s a reason for 60%. The lead author of 3SV, Colin Harris, is British, and has just seen his country go through a vote that likely to have much bigger consequences than who wins a literary prize. Had the EU vote required 60%, the UK would be looking a lot different right now. This colors his perception on what sort of vote should be needed for disqualification.
Note that it does not appear that such hurdles would have been difficult last year. Disqualification is not meant to be used casually, but only for what a super-majority of the voters reject as unworthy of being considered as a Hugo Award Finalist.
Jameson Quinn:
I, personally, was careful never to say that EPH would achieve perfect proportionality for slates (and I that includes the ML post linked above). But the fact is that in order to describe EPH, you have to give the design motivations, so you have to introduce the idea of proportionality. It’s not at all surprising that when you say “EPH was designed around proportionality, and would give results that are more proportional than the current system”, some people walk away thinking you said “EPH will give proportional results”.
Thank you. I know that you said, when quizzed about this here earlier, that you never intended to assert perfect proportionality; I was relying on this, and all the evidence I saw bore it out.
There is no doubt that some enthusiasts for EPH did claim that it guaranteed proportionality, but I don’t think any of the voting experts responsible for devising the system did.
I do think, though, that this further confirms Kevin’s view that these matters are not perspicuous, and we have difficulty getting our brains round them. Many statements that were not in fact assertions of proportionality – some that were even inconsistent with it – were widely read as such. ‘You can only get a sweep if you get a majority’ is not actually a statement of proportionality at all; in a strictly proportional system a bare majority of the votes would get a bare majority of the balllot.* But first-past- the-post thinking is so ingrained that it strikes us as proportional; ‘they’ve got a majority, so of course they win, don’t they?’
*Actually, of course, you can’t have a strictly proportional system with discrete quantities; strict proportionality would mean that 29.5 of the votes gets 29.5 of the ballot, which would be interesting to see. But you know what I mean.
Thoroughly supporting 3SV and EPH/EPH+. I don’t think we can address slates and trolls effectively without both.
Election systems are ways to obtain a set of coherent preferences from a diverse group of people. Should we imagine that they will be “perspicuous”, crystal clear to all the voters?
Well, certainly that would be nice. But that seems like a pretty high bar to set. When buying a cell phone, most people don’t look for the one whose operating system has the easiest-to-read source code, but rather the one with the most easy-to-use interface and the least tendency to crash and drop calls. Some might consider a clearly-written, open-source operating system evidence of crash-resistance, but it’s only one kind of evidence.
Obviously, a voting system should not be remotely as complex as a smartphone operating system. Legitimacy is an important goal of a voting system, and black box systems nobody entirely understands do not foster legitimacy. So a good system should strive to be reasonably simple and understandable at various different levels, from the broad-strokes description down to the nitty gritty of vote counting.
But the fact is that inferring a single set of group preferences from many individual inputs is actually not an easy task. Condorcet gave examples where no one winner is obvious; Arrow showed that no system can avoid such examples in all cases; Gibbard and Satterthwaite showed that any system can encourage strategy, thus calling into question the honesty of ballots; and Sen showed that under simple assumptions it’s impossible to guarantee that issues won’t get entangled so that my small, private preferences could be overridden by society’s overall goals. Any well-designed voting system must pilot these various shoals carefully, and direct simplicity can easily lead them to founder.
Enough philosophizing. In practice, is EPH (or EPH+) too much of a black box? I’m probably not the best person to answer this question, as steeped in voting theory as I am. But one thing I can say is that I’ve been extremely impressed with fandom’s ability to engage with this question at a high level. That includes the wonks like me; but it also includes the people who aren’t and don’t want to be wonks, but who know how to find a wonk and decide whether to trust them.
YMMV.
@Kevin Standlee
Okay, that makes sense. Thanks for clarifying it for me.
Earlier, I said I strongly opposed the amendment, but would change that to tepid opposition if it addressed two issues. Let me clarify.
First, I think it’s wrong to make it easier to permanently repeal it than it would be to reinstate it. I’d draw the analogy to the Brexit vote: if the UK had stayed inside the EU it could in principle still have decided to leave unilaterally any time later, but if it finishes leaving then getting back in would be much harder. So, I think that if EPH is up for an annual vote, then the first time it fails that should only apply to the following year; it shouldn’t be repealed permanently unless it fails for (at least) two years running.
Second, I don’t see why EPH should be up for annual re-ratification, while 3SV isn’t. From what I know now, it seems they’re both good ideas; but I see no reason to foresee that EPH is more likely to prove a bad idea. So I’d prefer that if there are annual re-votes, it should apply to 3SV as well. I would not insist on this, it just seems fairer to me.
If these two objections are addressed, I’d probably still vote against the amendment, but I wouldn’t argue strongly against it, as I would for the current version.
Perhaps the simplest thing would be to say that the election system can be amended annually for the next five years.
@Greg: I would support that fully, though I suspect some people (possibly Kevin?) would oppose it as being an invitation to meeting-packing. (Personally, I don’t think bad-faith actors have the capacity to take over the business meeting.)
If we’re looking for more capacity to respond quickly to events, but to still have some kind of checks and balances, perhaps the proposal could be that a proposal related to the Hugo decision procedures that passes in the next 5 years can be ratified by an online vote using the same voting eligibility and IDs as the Hugo awards.
Would people support this? I would. If it has broader support, it would be important to get it drafted pretty soon.
Any two members have the right to introduce amendments to new business. Should members want to try and propose changes to the enacting clause of 3SV, they are free to do do.
Pingback: Top 10 Posts For July 2016 | File 770