The Sasquan and MidAmeriCon II committees responded to File 770’s query about the transfer of anonymized raw 2015 Hugo nominating ballot data for use in testing the proposed E Pluribus Hugo vote tallying method.
Linda Deneroff, Sasquan’s WSFS (World Science Fiction Society) Division Head, wrote:
Sasquan passed its nominating data to MidAmeriCon II for analysis in the EPH process. Neither Glenn [Glazer], John [Lorentz], Ruth [Lorentz] nor I were involved in the analysis.
Tammy Coxen. MidAmeriCon II WSFS Division Head, explained what was done with the data:
After EPH passed at Sasquan, the MidAmeriCon II Hugo Administration team publicly committed to testing the system so that real data about its efficacy could be made available to WSFS members before the business meeting where ratification would take place. As part of that testing, MidAmeriCon II was collaborating with two researchers (Bruce Schneier and Jameson Quinn) in evaluating the system. As previously announced, it was determined that the data was unable to be sufficiently anonymized for a general release, so the researchers were provided data under a non-disclosure agreement.
There was to have been a coordinated release of the research findings between MidAmeriCon II and the researchers, which would have made clear the circumstances under which the data had been shared. Planning was already underway regarding that release, but as noted, analysis is still occurring. Our intention is to jointly share the research findings when they are complete, which will be well in advance of the business meeting at MidAmeriCon II.
The previously announced concerns that Coxen refers to were discussed here in a September 2015 post, “Hitch in Sasquan Nominating Data Turnover”.
Discover more from File 770
Subscribe to get the latest posts to your email.
I repeat my questions and direct them specifically to MAC II.
Did Loncon 3 in 2014 also provide the prior year’s data as Mr. Schneier mentioned? What about other years?
Am I mistaken, or wasn’t the point of delegating awards to a subcommittee to preserve confidentiality and integrity of the ballot? Not to give those individuals unilateral authority to selectively share the ballots as they saw fit with no accountability?
Am I mistaken, but aren’t stupid insinuations just stupid insinuations that should be ignored? Because if a person wanted an answer for real, wouldn’t the person send a mail to the correct instance instead of just making comments on random forums?
Dave McCarty chose to respond in his official capacity to Ed’s question on this blog by stating there will be no repercussions for EPH creators having used my confidential ballot and yours in a blatant attempt to sway voter behavior during the voting period. I think he can answer my follow up question.
Oh, give it a rest Brian.
He possibly could, but I think he knows as well as everyone else that there is no reason to feed trolls.
This seems to me to be the crux of the matter. A general anonymised release would have been ideal, provided it was sufficiently anonymised so as not to expose WSFS to the privacy laws of the world. As it couldn’t be done, then giving access to researchers under a NDA seems like the next best solution.
I wish when trolls were trolling, they’d at least be somewhat passingly familiar with how WSFS works with a WorldCon and how WorldCon Committees interact with their Hugo Committees and what powers and responsibilities the Constitution actually give WorldCon/Hugo Committees.
Of course if they bothered to actually understand how all this works they wouldn’t be trollish twits.
I’d like to repeat my apology for the headaches I’ve caused. I hope our analysis will be useful enough to make up for it.
Brian Z wrote “in a blatant attempt to sway voter behavior during the voting period…”
? It’s not actually the voting period; it’s the nomination period.
? EPH only affects how nominations are tallied, not how votes are tallied.
?? The whole point of EPH is that you don’t have to change your nomination behavior at all, indeed, that there is no point in changing your nomination behavior because the system is more difficult to game…
Oh, I think I found the problem.
No wonder some people are so unhappy.
@ Jameson–as you have discovered, many people think your analysis will be very useful indeed, including people who don’t *want* the WorldCon business meeting to have a useful analysis. Hang in there; most of us are looking forward to hearing more about it.
@Jameson: Your comment about the effectiveness of raising the average number of nominations per ballot caught my eye. It’s certainly made me more determined to find enough to fill all my nomination slots.
A blatant attempt to sway voters to do WHAT, exactly? If it’s so blatant, you can surely point it out. So precisely what change in normal voting behavior is he “blatant”ly trying to sway voters to make?
@Jameson: Thanks for doing all this work. I’ve been eager to see these results. I get that you to some extent regret posting them, so it’s weird to thank you for it maybe, but still, I’m grateful.
@Brian Z: I’ve missed you. I fondly recall the puppy sturm und drang of last year, and your subtle, deft way of getting to the heart of the matter with your characteristic warmth, wit, and honesty. It’s good to see you’re still in peak form.
Didn’t the business meeting vote to ask that the data be released if it could be sufficiently anonymized? They found it impossible to anonymize it sufficient for general release, but adequately for release to researchers for the analysis we want, so that’s what they’ve done.
And of course, Brian Z knows this.
Your comment about the effectiveness of raising the average number of nominations per ballot caught my eye.
Which may be tricky. Increasing the absolute number of nomination ballots should be easy; increasing the number of nominations on each ballot less so. People often point out, when urging others to nominate, that you don’t have to nominate in every category, and you don’t have to fill out your ballot in categories where you do nominate. This is of course true. But when the aim is not just to increase the number of nominations because this is in general a good thing, but specifically to get more nominations in order to outvote slates, it becomes a problem. Being able to fill out one’s ballot in a lot of categories requires a degree of knowledge which a lot of fans don’t have. (Including me.)
@Jameson
Thank you for doing this work – it’s good to actually know the patterns in the data. Don’t worry that you won’t win over the PUAs, homophobes, and Dark Enlightenment nutters who go by the name “puppy.”
@Jameson Quinn
Then please accept our gratitude for giving us a sneak peek at it. I’ll bet your gratitude-to-headache ratio is around 200 to 3, so don’t let it get you down. 🙂
As far as encouraging more people to nominate and to nominate more things, my husband and I created Rocket Stack Rank after Sasquan, on the theory that more people would nominate if it wasn’t such a research project. (Even remembering what you read and liked after six or twelve months–and what length it was–is a chore.) So we put together all the info we thought we ourselves needed and shared it.
Have a look, if you get a chance, and see what you think. It’s free (it’s even ad-free).
@Andrew M: I was specifically thinking of it being a good thing to make an extra effort this year as the nominations will still be vulnerable to slating.
It’s encouraged me to do so. I’ll be a boost on both variables: first time nominator and pushing myself to get a full ballot.
I am a little baffled about Brian Z.’s insinuation that the release is intended to affect people’s nomination strategies, since this year, the nominations will be counted in FPTP, as has been custom since time immemorial. Even in Brian Z. world, I fail to see how this could possible affect people’s strategy for _this_ year.
Brian Z
I’m breaking my policy of not responding to you. *gasp*
If you want to know if the “not fully autonomous data” will be released to you it’s beyond obvious you need to contact the appropriate people again. Although with a little bit of reading comprehension of this article I’d say it’s also obvious the answer will be NO. Ask yourself:
1. Are you a highly respected researcher?
2. Have you shown you understand how things work at Worldcon and you are willing to work within the system (Seeing as you ask on file770 instead of writing a new email that would be No)
3. Have you shown respect for the people who run Worldcon giving them a reason to trust you (show examples)
4. Do you have a well prepared request for why you want the data, what you intend to do with it, and how it will benefit Worldcon (I’d love to read this)
Seriously can you answer those questions in a way where Worldcon admins working very hard to protect nominees identity should pass their data along to you objectively? You could post the answers here if you still think the admins are more likely to answer here than by email. It would beat your regular comment quality today.
People should nominate as they feel comfortable this year as usual. No one should feel obligated to nominate 5 in all categories. That’s how we get into a situation of few nominators. The more people who nominate the better. Obviously the more things you can nominate and more categories the better chance of cutting back slate effectiveness.
But look at RP2 slate this year it’s all over the place. SP4 lost steam a while back and we won’t know for another month what their recommendation list is going to look like.
Strategic nominating is not the way to keep the Hugos together this year. Nominate what you think is the best of what you’ve read, seen, know. Take advantage of the many resources out there this year if you want to learn more.
I’m repeating my post from the earlier thread:
Why? EPH is not in effect for this year’s Hugo nominating process. It must be approved by this year’s business meeting in order to go into effect for NEXT year. So how, exactly, can Mr. McCarty’s response to you be of any influence on what you are going to nominate?
You are going to nominate works that you have read and feel are Hugo-worthy, correct? NO ONE can have any influence on that, can they? Your nominations are based on your personal, considered judgement on the quality of the works you have read and nominated.
So your statement makes no sense.
=========
You’re still not making sense, Brian. The analysis has NO EFFECT ON THIS YEAR’S NOMINATING PROCESS, because EPH is not in use this year.
The analysis will support or not support the implementation of EPH for NEXT year, assuming it is voted in at THIS YEAR’S business meeting.
So, it is important that the analysis be complete and available prior to the business meeting. It has NO EFFECT on this year’s nominations. None. And, as has been pointed out a zillion times, it is designed so that a person doesn’t need to change anything about how they nominate.
Which is, READ AND NOMINATE stuff they think is worthy of a Hugo.
Unless there is some other agenda involved in their nominations …
@Tasha Turner
I think the real ask is just that people invest a couple of hours in it. As you say, there are lots of resources online to help people find things to read (or even just to remind them of what they’ve already read). A small investment of time can make the difference between nominating two or three things in three or four categories vs. four or five in over half the categories.
That’s the most anyone can reasonably ask.
@Brian Z
Out of curiosity, what sort of EPH result do you believe could sway voter behavior? Even if it were to take effect this year. It seems (either way) that voters would want to nominate as many qualified works as possible in as many categories as possible.
It is not quite true that Jameson’s comment could not influence nominator behavior this year. His analysis apparently showed that both adding more honest (non slate) nominators and encouraging those honest nominators to find more good things to nominate will help dilute slate nominations, and both those things are just as possible this year as in future years.
This leads us to contemplate the possiblity that “encourage more people to find more good stuff to nominate” may be precisely the change in voter behavior that disturbs Brian most, of course.
Considering all the nonsense he’s been spreading about EPH since last year, most of it outright lies, contradicting every analysis done prior to last year’s Worldcon, you may have nailed it.
@Cat – His analysis apparently showed that both adding more honest (non slate) nominators and encouraging those honest nominators to find more good things to nominate will help dilute slate nominations, and both those things are just as possible this year as in future years.
When I say math is not my strong suit, I’m understating the case. However, as soon as some raw numbers were released it was clear that an antidote to stating was more nominators and more nominees in all categories. I don’t think it is a surprise, in other words.
But this has always been good behavior. This year from what we’ve seen stated by people it looks like more people will be nominating as they get their pins. Since EPH doesn’t eliminate slates totally it will continue to be good behavior. So it’s not news that more people eligible to nominate doing so would be good with or without EPH.
Knowing if the data is only from one year of Hugo nominations or multiple years is valuable for many reasons.
The decision by the Hugo coordinators to selectively release data under a non-disclosure agreement to people who have the ability to do the analysis is reasonable.
What IS NOT a good idea is to unreasonably limit qualified individuals who are NOT a part of the “in crowd” of WorldCon to reexamine the data.
An analysis of various voting structures will not be trusted if only “one side” can do the examination.
But what the Hugo people are doing as a first step is very reasonable. Make the data as anonymous as you can. Have a nondisclosure agreement. Pick reasonable people to do the first wave of analysis.
I would also strongly caution the “File 770 regulars” in trying to paint everyone who has not been very happy with the Hugo winners in the last decade with an overly broad negative brush.
I have not thought much of the Hugo novel winners of recent decades. I also voted for the Hugos for the first time last year. I plan to nominate things I liked in 2015 this year. I have a lot more sympathy for the Sad Puppies than I do most of the traditional Hugo crowd.
But the critical thing is this. Be careful of alienating those whom you might be able to find common ground with. I think that if I get (and take) the time to review the work that I could have a reasonable interpretation and confirmation/disconfirmation of the findings done by the current researchers.
But letting “everyone” have a crack at the data would probably lead to meaningless chaos. Given the level of anger some have about the Hugos, the people in charge should use reasonable judgment about the data and the selection of those who wish to check the math and the assumptions.
Unfortunately my phone doesn’t run Stylish but on the plus side it’s been a pleasure watching Brian Z’s misrepresentations getting a thorough demolition. Again. Carry on.
When I asked the question, I certainly had no intention of giving anyone a club to beat David over the head with. I apologize for doing so.
If anyone doesn’t like how David (or anyone else involved) has handled this, feel free to step up and volunteer to do the work.
Again, my apologies to David.
Airboy:
How do you define “in crowd”? Do you mean that Jameson Quinn is part of the “in crowd”? “In crowd” of what? And what sides are you talking about?
Yes, that would be stupid. Just as it would be stupid to not recognize that conspiracy theorists complaining about “SJWs” aren’t people you should pander to. They are people you should not trust with sensitive data like this.
@airboy
I’m not sure Bruce Schneier can be considered part of the “Wordcon in-crowd.” I don’t think anyone has said what they’ll be doing post-publication yet, and it may well involve exactly what you want. Schneier is a strong advocate for open research and challenging results, after all, although the final decision may not be his.
Cheryl S.: as soon as some raw numbers were released it was clear that an antidote to stating was more nominators and more nominees in all categories. I don’t think it is a surprise, in other words.
Yeah, I’m a bit surprised that so many people seem to regard this as some great revelation; I thought this had been pretty clearly established during discussions prior to Sasquan last year.
What IS NOT a good idea is to unreasonably limit qualified individuals who are NOT a part of the “in crowd” of WorldCon to reexamine the data.
Please tell us who that ‘in crowd’ is. My experience is that the people who run Worldcon and the Hugo awards are all fans. They aren’t chosen from some list kept in a file drawer in a secret office, and they aren’t chosen by the publishing industry.
WSFS and the Hugo subcommittee have a reasonable interest in not releasing last year’s nominating data to everyone, given that the puppies are still active and still trying to game the as-yet-unaltered system.
Dear Airboy, when you go from saying this:
I would also strongly caution the “File 770 regulars” in trying to paint everyone who has not been very happy with the Hugo winners in the last decade with an overly broad negative brush.
And then jump to this:
I have not thought much of the Hugo novel winners of recent decades. I also voted for the Hugos for the first time last year.
All you have really done is demonstrate you don’t actually know what you’re talking about, since you seem to think the last couple of decades worth of Hugo winners weren’t good. Was it Mirror Dance you didn’t like? Or Blue Mars? Rainbow’s End?
@Mark
Actually Bruce has been around SF fandom for a while. While I wouldn’t call him a Worldcon insider he has definitely been involved writing restaurant guides etc for various cons including at least one Worldcon.
I think Jameson Quinn is less well connected.
airboy: I would also strongly caution the “File 770 regulars” in trying to paint everyone who has not been very happy with the Hugo winners in the last decade with an overly broad negative brush. I have not thought much of the Hugo novel winners of recent decades. I also voted for the Hugos for the first time last year. I plan to nominate things I liked in 2015 this year. I have a lot more sympathy for the Sad Puppies than I do most of the traditional Hugo crowd. But the critical thing is this. Be careful of alienating those whom you might be able to find common ground with. I think that if I get (and take) the time to review the work that I could have a reasonable interpretation and confirmation/disconfirmation of the findings done by the current researchers.
Look, you can try to pretend all you want that you would be some sort of “objective” evaluator of the data. But given your numerous past comments such as:
airboy: What the Hugo voting showed is that 4,500 voters out of a legion of SF readers decided to vote lockstep to no award anything nominated by people they dislike who voted in lockstep during nominations.
The irony is hilarious! SJWs voted a slate to protest slate nominees! And then they pat themselves on the back for being “open-minded!” Awesome!
airboy: The SJWs slate voted “No Award” 2,674 votes in protest of people they don’t like slate voting. Why 2,674 slate SJW votes?… So approximately 2,500 to 3,050 slate voted “no award” because they were not part of the SJW clique.
airboy: I find it hilarious that the author mentions that the Sad Puppies were “slating campaigns” while at the same time pointing out that the same tiny number of other individuals were nominated/won year after year after year. Clearly the previous behavior is more reliable “slating behavior” than Sad Puppies. And most clearly the “slating behavior” continued when the artists nominated “out of the norm:” were all “no awarded” last year.
airboy: Jim Baen revolutionized e-publishing. The monthly collections Baen sells prior to print publication provides more money directly to authors than traditional publication methods. Jim Baen also was responsible for a lot more SF/F being available to the public. But the Hugo Clique either did not realize, or decide to honor these accomplishments for the overall field of SF/F publishing.
The fact that you use the terms “in crowd” and “SJW” and “clique” and “slating” in this way, and the fact that you are castigating Hugo voters for not giving Baen a “Best Publisher” Award when no such category exists, demonstrates that 1) you are nowhere near being “objective”, and 2) you don’t have a good understanding of Worldcon or how the Hugo Awards process works. I can’t imagine why you think anyone would be willing to entrust the data to someone fitting those descriptions.
@andyl
I knew he was a fan but wasn’t aware of those things, so thank you. I suppose if airboy wishes to paint him as a biased member of the “in crowd” they could do so based on that, but that would ignore his strong academic standing and many other reasons he would have to ensure his work was unbiased.
airboy –
I’d strongly caution painting the “File 770 regulars” with a broad brush in assuming they’re all somehow very happy with all of the Hugo winners of the last decade and not as individuals who nominated and voted for different things aside from those who won. In any year of the Hugos you’ll see a bunch of people arguing over the works they felt were the best, and yet remain respectful and happy for those that won and for the work people voluntarily do to make it wonderful even if they’re disappointed a book that was special to them that year didn’t make it in.
What a fine crop of sea lions we have today. 🙂
@Meredith: I think it’s El Nino that brought them out.
The brackets we’ve run certainly show file770 members don’t agree on books. The discussions we have around books – the same. We discuss books a lot around here – previous Hugo winners, books which came out in 2015, “classics”, “golden age”, what to give kids to get them into SFF, what to give college students to get them into SFF, etc. Disagree politely is how every single one of those discussions has gone.
I can’t recall a single book/short where everyone involved opinion the discussion agreed.
airboy
On the one hand it is perfectly okay to not like some of the Hugo nominees and winners. I personally strongly disliked “The Day The World Turned Upside Down” for example. And not too long ago I read several Hugo nominees in best novel and was under impressed.
On the other hand it is not okay to turn around and converge on a set of mutually acceptable second bests chosen by another to gain more political power over the process. That is very different from “I didn’t really care for the last decade’s Hugo winners.” People were supposed to nominate their honest favorites. Instead Puppies basically turned their ballot over to someone who used them to reward his friends and mentors.
If you disliked the previous Hugo winners but are nevertheless planning to nominate your honest favorites without recourse to a slate, of course I honor that. I’m planning to do the same despite the bitter fact that the Puppies are almost certainly going to make it meaningless in terms of the nominations.
But given your apparent confusion on the subject of slates (thanks, JJ!) I kind of expect that the committee is going to hesitate a bit to commission you to handle partially anonymized nomination data to check whether EPH reduces their effectiveness. I’m not on the committee or anything, so I could be wrong. But that would be my guess.
The only way to conclude that Bruce is part of the “in crowd” is to assume that it includes anyone who has ever been nominated for a Hugo, whether or not they won.
By that logic, Larry Correia and Vox Day are part of the in crowd, and I’m not.
Speaking as an outsider, I’m prepared to wait for the results of the analysis.
Considering
1) EPH won’t be in the rules for this year’s nominations and
2) the deadline for nominations is in about 6 weeks, making it likely that this data won’t be released in time to have any effect on this year’s nominators, any caviling about it is simply mendacious fearful Puppy whining.
“More people should nominate a variety of good stuff” would hardly seem to be a controversial stance, let alone a political one.
Sorry it got you so much tsuris, Jameson, but thanks for your hard work.
When I cited your question and the reply, I was simply answering the charge that I had no business asking questions or looking for answers on this blog by providing an example of a question and an answer.
Mr. McCarty did not create this mess. (I’m still not clear why it landed on his doorstep.) We all appreciate the hard work of the Worldcon volunteers.
Though I do notice that fans are stepping up and offering to help – and getting insulted and jeered at for making the gesture. That can’t be good.
If by fans you mean conspiracy theorists.
@Cat:
More nominators was obvious, of course. What was less obvious was how much impact nominating just one more item in a category could have. I didn’t have time to follow all the discussions about the various nominating proposals, so if the average nominations per ballot came up in them, I missed it.
All of that is apparently irrelevant. I have been in email contact with the Hugo administrators since August of last year, Their final response:
The Hugo Administrators are committed to only allowing the EPH proposers access to the data it seems. They see no need to validate or replicate the results independently.
errhead: The Hugo Administrators are committed to only allowing the EPH proposers access to the data it seems. They see no need to validate or replicate the results independently.
I wouldn’t call either Quinn or Schneier “the EPH proposers”. They are two outside experts who were consulted by the people who were proposing EPH. You can Google the credentials of both of them.
I’d like to see the nomination data, and do some crunching on it, too. Two of my professional specialties are data normalization and analysis.
But frankly, if the floodgates are opened to anyone who requests it, it will end up being nothing but a huge argument and headache — one that ultimately distracts from the goal here, which is to devise and enact a methodology which de-magnifies the magnification power of slates, without attempting to decide what constitutes “a slate ballot” and eliminating or devaluing those votes deemed to be “slate votes”.
errhead, I really appreciated the effort you put into programming your own EPH simulator and your participation in this process. I know it’s a big disappointment that you are probably not going to be allowed to run it on last year’s data, and I’m sorry about that.
I’m hopeful that we’ll all be given information that is a lot more meaningful than what was shared here — information that will be truly useful in deciding whether to ratify the EPH process — and I hope that you’ll continue to weigh in when those results are released.