Tell Us What You Really Think

In a “Hugos/Sad Puppies Guest Post by Jameson Quinn”, “the guy who came up with the basic idea for the E Pluribus Hugo proposal”, does his best to unravel the coalition that passed it on its first reading at Sasquan.

In order to understand this, it’s important to see that the Sads actually did have the germ of a valid grievance: in past years, many Hugo nominators have been from a pretty small and insular group of authors, editors, and hardcore fans, who often know each other personally and whose vote is probably influenced to some extent by factors extraneous to the work itself. Writings from authors who are personally well-liked, or whose overall body of work is stronger than the individual writing, probably have had a bit of an unfair advantage in getting nominated.

Of course, that’s not to endorse the Sad Puppy point of view. Of their three complaints — that the Hugos have been too artsy-fartsy, that they have been too political, and that they have involved logrolling — the first two are sour grapes, the second two are hypocritical, and the relationship between the three exists only in their heads. Only the third could be even slightly legitimate as cause for organized action; but certainly not for the action they took, which was basically to vandalize the awards as a whole, without any hope of actually accomplishing their objectives.

The efficiency of Quinn’s self-sabotage is impressive when you consider that half of the post is wasted concern-trolling the Republican primaries.

233 thoughts on “Tell Us What You Really Think

  1. JJ: I don’t remember how many entries were in the 1984 data (and, obviously, it’s a lot less than this year), but I cleaned/normalized it all — manually — in just over an hour.

    I find that very difficult to believe. For one category, maybe. But did you catch all the variations of, for instance, Teresa Nielsen Hayden’s name?

  2. Nicholas Whyte: I find that very difficult to believe. For one category, maybe. But did you catch all the variations of, for instance, Teresa Nielsen Hayden’s name?

    Yep, I imported it into Excel, then cleaned one category at a time. That makes it a lot easier.

    ETA: Note that I am very fast with a keyboard, I can type >100 words per minute.

  3. @KevinStandlee:

    You obviously have never actually administered the Hugo Awards. Saying this tells me that Sasquan really needs to release very raw, very un-normalized data, so that everyone who is convinced that data normalization is trivial can spend the next few months fighting over it and never actually getting a chance to run the data through the EPH algorithm because there will never been 100% agreement upon what sort of data normalization algorithm to use.

    (And note that I’m a database programmer in my Day Jobbe, usually dealing with large quantities of geographic information, and the data is never normalized and is a huge pain in the neck to keep clean.)

    Yup, never administered the Hugos, probably never will unless the Dublin team’s bid wins and they’re absolutely desperate for help, which they probably won’t be given how many people Shamrockon usually has helping out and the sheer size of the tech community in Dublin (throw a rock, you’ll hit two facebook developers, three google developers and break a startup’s window).

    And Yes Please, release very raw data and let us nerd out on it. It might actually make your life easier (or other peoples). Even if we do that through NDAs and the like. Some of us are actually qualified experienced professionals who are able to help and are volunteering to do so for free (I’m a software engineer, this stuff is how I’ve paid my mortgage for twenty years as well as how I keep my brain from getting bored in between good books – and by my count you currently have a half-dozen equally-or-more-qualified professionals volunteering their time on here alone because this problem is just so damn nifty and the thing it helps is so valued and loved by all of us).

    Look, tell you what, how about this? How about we show you? How about we go off, build the tools we’re talking about, point them at the public domain raw 1984 data and we see how fast the tools are, how accurate they are and how much time it saves the admins, and then we start talking about NDAs to release the raw 2015 data to people to test those tools on more modern and larger datasets, and if that all works, we give the tools for free to all future Hugo admins (which can be easily done several different ways).

  4. by my count you currently have a half-dozen equally-or-more-qualified professionals volunteering their time on here alone because this problem is just so damn nifty and the thing it helps is so valued and loved by all of us

    I’m dangerously close to spending the afternoon running up a cleaner that could be crowdsourced, unfortunately it’s one of the two afternoons I’m paid to do other stuff so it’ll have to wait till this evening. No Excel involved, and minimal typing I suspect…

  5. I understand that anywhere the Hugos and EPH is discussed will be of interest to and read by those interested in the Hugos in general and the Puppy issues in particular.

    But the article is in a blog that is generally an academic one about law and politics that occasionally crosses over with pop culture, and the target audience is that blog’s audience. I’m part of that blog’s readership from long before this, and from that perspective there’s nothing wrong with the post, other than that (and this is in the comments there too) tying it to the Republican primaries is a bit of a reach. Quinn took off his Fan hat and put on his Analyst hat and is talking to other analysts. His personal take on the Puppies’ beliefs and how legit they are is just an aside.

    The main discussion is for other wonks about how EPH actually works and how such as system could be applicable in other aspects. I don’t find anything controversial about it. The particular slant on Trump is consistent with the overall liberal/progressive focus of the blog.

    Overall, I don’t have a problem with it. And I plan to be at Worldcon next year to, among other things, vote for EPH, so I’ll cancel out at least one fragile voter who takes such offense at this that it changes their vote.

    Also, as I’ve said all along, I don’t think the perception that the nominators have been a fairly small and somewhat homogenous group, and personal relationships with and liking of particular authors has helped them get nominated, is entirely off base. If nothing else, if a bunch of people like a particular author, they will be more likely have read the author’s output over the course of the year, and if they missed something, to make a point of seeking it out and considering it come nomination time. And people who have built a strong base of support within the fan and writer community have an advantage in this respect.

    I don’t think it’s unfair, and I don’t think it’s particularly new. I think when authors back in ye olden days made a lot of con appearances, hung out and partied with fans, and answered fan mail personally, supported aspiring and upcoming authors, and were generally congenial and approachable folks, they tended to, if not outright get more nominations, have their work looked at and considered.

    As has been pointed out repeatedly, back then you could literally read all or nearly all the SF output for the year and consider all of it. Even then, though, some fans who voted on the Hugos didn’t have time to read everything, but if they liked and felt a personal affinity with an author, they probably made a point of reading and voting for their stuff.

    Nowadays, with the volume of stuff to read, the feeling of a personal relationship between writer and audience is even more important in getting attention and “reach”. And readers can’t hope to read everything so it’s important to be an author people make a point of reading. One thing Hugo readers are likely to read is future work by people who have gotten previous Hugo nominations and awards.

    I don’t think Quinn is saying anything particular offensive in noticing this, and he is not the only one. I don’t even think there is anything particularly wrong or problematic in it, and the Puppy offense-taking about it is silly and overblown. More voters in the nomination process will lead to wider-ranging choices in the long run, and if nothing else this year has brought a lot of energized fresh blood into the Hugo-voting fandom. At least some of us will stick around for the long haul.

    The more I learn about EPH the more impressed I am with it, especially the part where it’s really easy for the average voter because it doesn’t make their personal voting process any more complicated. I don’t find Quinn’s article offensive at all, I don’t think we (file 770-ites or SFF fans in general) are the target audience, and Quinn’s statements were perfectly appropriate for what was his target audience.

    And as for the Puppies’ taking offense and making hay out of it (or attempting to) — What. Ever. There is literally not a word written about the Hugos NOT written by an avowed Puppy that is not offensive or potentially offensive to them. It is stupid to walk on a superhighway of eggshells because something might set off a marmot or a Shiite Catholic or a fiery ItaloTexan.

    If anything, the fact that Mike Glyer!! of VILE770!!! found Quinn’s article offensive might be more effective ammo for them (Look even our SWORN ENEMY thinks this guy’s a jerk!) than the original article.

  6. I’m dangerously close to spending the afternoon running up a cleaner that could be crowdsourced

    I’ve put in about ten hours to one already based on affinity propagation, but I want to tie it into both edit distances of various kinds and actual worldcat searches to verify eligibility and get the canonical form of the bibliographic citation, so it’s going to take me another weekend at least to get v0.0.1 running. It’s seriously nifty stuff (the problem, I mean).

  7. While the difficulties of cleaning up the Hugo nomination data and possible ways of easing them are interesting, I’m surprised how little comment I’ve seen to date on the difficulties of the task Katie Paulk et al propose to undertake. If I’ve understood correctly, they’ve promised to produce a recommendation list of 10 entries for every Hugo category (presumably plus the Campbell), ranked in order of popularity, based on suggestions offered online by the denizens of an unspecified but at least moderately large number of online forums willing to play host to a Hugo nominee suggestions thread. They’ve also said they’ll make the full list of suggestions for each category, also ranked by popularity, available for those interested. Could they design a process more obviously vulnerable to freeping and more likely, if an honest count of all the suggestions is to be undertaken, to add up to more unpaid work than they can possibly afford to take on, if that was their explicitly announced intention?

  8. @Jon – Indeed. I note that if she and her team manage to get on top of these problems* she’s half way towards the populist award that Puppies have sometimes said the Hugos should be. Bolt some statues, a ceremony at a friendly con, and some PR onto it** and I for one would be eager to see what the Puppy awards would come up with.

    But yeah. I really hope they’ve cleared their schedules for the early part of next year.

    * They don’t even need to be properly solved; “good enough” would do.
    ** Maybe a sponsor?

  9. The comical thing is that if Paulk really goes to all that trouble, and produces a fair and transparent list from a broad base of people who care, she will simply be replicating the Hugo nominations process as is, and she’ll hate the results.

    Unless someone games a slate within her slate. Hmm, who might want to do that?

  10. Niall McAuley: Unless someone games a slate within her slate. Hmm, who might want to do that?

    Yes, unless the Sad Puppies limit themselves to a very long list of Recommendations for each category, you can pretty much see how it’s going to go. It’s going to be a repeat of this year, and they’ll be all hurt and upset that their slate list idea doesn’t work the way they claim they wanted it to.

  11. No, I think what Niall McAuley is getting at is that as designed right now, our friend the five dimensional chess playing game designer can suggest to his associates that they nominate on the Sad Puppy reccommendation threads.

    I’m 50/50 on whether he actually does it. Then split 30/20 on trying to keep it quiet (cunning) or just puts it all out in the open (arrogant) and dares Paulk et al to do anything about it.

  12. Paulk can’t complain if Beale games her slate with his minions, since a) there are no rules and b) hey, slate!

  13. Of course Paulk can complain. More than that, she can blame whoever she wants. She can even do so with a complete lack of general awareness and irony if she really wants to. I doubt the fact that I will make elaborately faux-naive jokes about it will deter her. Nor should it!

  14. Except with EPH, you have to: (a) clean up the data further down the list (because you’re possibly eliminating slate entries); and (b) you have to clean up that data completly before you can run any programs on (rather than saying “Oh, look, here are some entries for GOT: The Mountain and the Viper”–I’ll add that count further up the list to the count I have for “Game of Thrones: The Mountain and The Viper”).

    NON. SENSE. EPH does not require more cleanup than you’re already doing. (This is what nominations have always been like.)

  15. So when Beale slates Paulk’s 2016 slate, it’ll be like:

    a) good advice that you didn’t take?
    b) rain on your wedding day?
    c) ten thousand spoons?
    d) ironic?

  16. PJ Evans:

    NON. SENSE. EPH does not require more cleanup than you’re already doing.

    Preamble: This is not a trick question, even though it will sound just like one.

    I was assuming EPH would run the data from every single vote. In which case it would need to clean up all the data, which apparently is not done now. Is my assumption wrong?

  17. Mike Glyer on September 2, 2015 at 4:28 pm said:

    PJ Evans:

    NON. SENSE. EPH does not require more cleanup than you’re already doing.

    Preamble: This is not a trick question, even though it will sound just like one.

    I was assuming EPH would run the data from every single vote. In which case it would need to clean up all the data, which apparently is not done now. Is my assumption wrong?

    For purposes of testing EPH, it would indeed be best to run the data on every vote. Should EPH be implemented after it’s tested and (arguendo) found to be satisfactory, how much difference would it make to business as usual if the Hugo admins “only” clean up the voting data to the current standards?

  18. Hmmm… to a first approximation, ‘dirty’ data is more-or-less random. While EPH is being tested, perhaps the effect of incomplete data-cleansing could be tested by giving each individual nomination an X% chance of being ignored..?

  19. Mike, I’d think that they cleaned up the data now, given that people do strange things with names, and leave out information as well, although with electronic forms there should be less work than with paper.
    (Be glad you don’t have the 1984 paper forms. I am really tempted to type those in (extremely anonymized) for the lulz.)

  20. @P J Evans

    Apparently the usual thing is to only clean up things with lots of nominations that are obvously the same, rather than clean the data and then run the program. I get there’s extra work involved to do it thoroughly, but I can’t help but feel that not cleaning the data as a first step is asking for trouble, even without E Pluribus Hugo.

  21. So the issue is whether EPH requires more data cleaning than the current system. In the current system, if you can determine that a given illegible nomination couldn’t possibly be referring to any of the top 6 candidates, you can safely ignore it. In EPH, you’d have to know that it doesn’t refer to any candidate with more votes than the 5th place has points.

    I hate to say this, but it is actually theoretically possible for the order of the early eliminations to affect the final winners in EPH. But that is improbable; not just on the “not in a million years” scale, but on roughly the scale of “if you ran the system daily on every planet in the galaxy until the heat death of the universe it still wouldn’t happen”. (I haven’t actually done the calculation, but I know that getting an election like that by pure random chance is definitely multiple dozens of orders of magnitude improbable, because you need a tricky unbroken pattern that repeats its way up the whole chain of candidates. Even creating a ballot set where that could happen is something that would take me at least a day, and maybe several, of head-scratching; and I have a good theoretical understanding of how it should be possible.)

    So, that purely abstract and ridiculous possibility aside, the fact is that in practice any level of “good enough” data cleaning that is acceptable or unacceptable currently would be the same under EPH. EPH does not create the need to clean things up, and while it is technically a little bit stricter, it does not need 100% perfection; 99.44% is good enough, just as with the current system.

  22. @Jameson Quinn

    Can you describe the edge case in which that occurs? I assume it’s getting slightly different candidates ‘in the arena’ in a different order, which ultimately makes something get into 4th place in points and never have to face elimination?

    Also, not to derail the EPH talk, but doesn’t approval voting incredibly incentivise strategic voting? Some Condorcet method would presumably give better results, no?

  23. The edge case involves a crazy dance where there are two “tracks” of candidates from weak up to strong, each of which overlaps in a quasi-slate-like way, and the eliminations end up being two type A then two type B but it can end up either A1 A2 B1 B2…. or B1 A1 A2 B2 B3…. In the end, the last 5 would either be 3 As and 2 Bs or vice versa. As I said, I think it’s possible, but it would seriously take me a day or more of work to even work out a ballot set where it happens. (Pathological elimination order issues with IRV are trivial to set up and could even happen by chance, and that’s where I got the idea, but with EPH it is way, way harder.)

    As for approval voting and strategic voting: with approval ballots, it’s tough to even define what you mean by “strategic voting”, because setting an approval cutoff is by its very nature arbitrary, so who’s to say it’s not always strategic? But approval can say two good things about strategy. First, there is never cause for a dishonest strategy under any possible full or partial information (as long as your utility for the candidates themselves doesn’t depend on the other ballots; in that case, you can set up any crazy shit but it makes no sense.) And second, if there is a Condorcet winner, then strategic voting will elect them.

    Yes, I understand that as a voter, approval can feel more strategic than, say, Condorcet or even IRV. But that’s mostly just an illusion; it’s basically just making the hidden strategies visible.

    In my opinion, the best strategy-resistant voting systems amount to forms of approval voting where the voting system helps figure out the best strategy for you. That is, there’s Majority Judgment, which is equivalent to rated Bucklin, and Bucklin just means counting more and more of each ballot as approval until some candidate passes 50% (highest median). This is a simple way for the voting system to find an approval cutoff that works best for the average voter. And there’s also SODA, where you delegate to your favorite candidate, and they work out the best approval ballot you could have voted, which they can do because when they do the votes are already tallied.

    But that’s all abstract theory. In practice, to avoid eternal infighting among voting reformers, approval is the clear and simple first step reform.

  24. Sorry: I meant to say that if there is a majority Condorcet winner, approval will elect them, and usually do so even if some significant fraction of the voters do not vote a stable Nash strategy. If there is no majority CW due to clone candidates, then you could have a chicken dilemma situation.

    Note that even this qualified statement does not hold for Condorcet voting itself. In Condorcet, there can be a Condorcet winner, yet still be a way for a more-organized minority to get a winner they like more than the CW by artificially creating a cycle.

  25. Looking through the data last night it does appear there has been some cleaning done as the papers were transcribed, all names have been entered as “surname, firstname” and names and titles appear to be flattened to lower case. We’re going to need more recent data where the nominations are predominantly from online entries eventually…

  26. Looking through the data last night it does appear there has been some cleaning done as the papers were transcribed

    A significant problem is that people generally don’t think of that work as being part of the cleaning process 🙁

    I suppose you could add errors at random, keeping the error-d version at most one or two units away from the original in edit distance, but that’s still a pretty faked up test set. And I haven’t yet found any public domain dataset that we could use in its stead.

  27. There are also some comments added along the lines of “(Same as X???)”.

    Anyway, tonight I did a complete cleaning run through in around 90 minutes, I’ll see how I did tomorrow as it’s just gone 22:40 and my eyes have had enough.
    I’ve created a MySQL table from the CSV file, created a second table with DISTINCT category, title, author from the first (too slow doing a DISTINCT from the original) and then a Lucene data set from the DISTINCT.

    A PHP script picks a random item, looks up the title and author in the Lucene set and displays anything with a score over 0.1. Tick a checkbox for each matching entry, submit it, the script creates a ‘cluster’ with what will be a canonical name and author, and link table entries between the cluster and DISTINCT entries. Repeat until bored or until there are no more unmatched DISTINCTs.

    Useability needs a bit of work, I speeded things up when I realised checking boxes by default when the Lucene score was over 0.4 saved a lot of clicking, and the submit button is in the wrong place. Also planned for tomorrow is mapping the cleaned clusters back to the original data set.

  28. Oh, they’d be interesting to deal with, all right. What I’d want to know is, does case count? Because I’d rather not have to do all caps.

    (Sample weirdness:
    the citadel of the autarch – wolfe – timescape* i believe this was ruled an ’83 title last year. if i’m wrong please substitute cherryh’s 40,000 in gehenna

  29. Anthony on September 3, 2015 at 4:49 am said:

    There was cleaning in getting it into the computer: it was all on paper ballots. (There are a bit over 500.)

Comments are closed.