[Introduction: Erin Underwood originally left this as a comment and has given permission to republish it as a post.]
By Erin Underwood: I’m not going to get into the specifics regarding the use of AI by this Worldcon’s program committee. So, please note that this post has nothing to do with that…. However, I do recognize the appeal of using a tool like ChatGPT to help reduce a substantial, unpaid workload that is getting harder and harder to manage every year.
Instead, I would like to mention that one of the biggest issues that any convention program team faces (whether local, regional, or international) is figuring out who all these people are who are applying to be on the program. This is even more difficult since so many people complain about filling out longer, more detailed surveys. After years of running program teams or being a part of program teams, I can safely say that the information provided by potential program participants is often insufficient to make them stand out from all the other people who are also asking to be on the program.
Most of the time, these committees and teams rely heavily on long-time volunteers who are familiar with hundreds, if not thousands, of creatives across disciplines and over multiple generations. As longer-term volunteers disappear, their wealth of knowledge also disappears. This leaves new volunteers who are largely only familiar with the people they have read, seen, or are familiar with in their more limited capacity as a newer volunteer. We NEED new volunteers, and they are often doing the very best they can with the experience and information they have available to them.
Some common panelist questions that I have heard from volunteers include:
-Who is David Gerrold? What does he write? I don’t see any new books?
-Fonda Lee is a fantasy writer. Why would you put her on a YA or economics panel?
-What kind of art does Bob Eggleton do? Why is he on a TV show panel?
These are just a few questions that long-timers know the answers to without even seeing a survey response, but new volunteers are desperately needed, even if they don’t have the same depth of knowledge. More to the point, new volunteers are needed who are willing to do lots (and I mean LOTS and LOTS) of work for free, and those people have to actually follow through on the amount of work that is required. When I decide to run a convention program, I put in many hundreds of unpaid volunteer hours that fill up all my free time outside of my job between the months of August-February during the year – or thousands of hours if I also choose to also volunteer to help with a Worldcon. That is often a hard combination to find, and it’s a lot to ask of a person.
I can’t tell you how many times I have also seen program surveys come back from famous authors who assume that everyone knows who they are, and they only put in a bio that is pithy and funny… but doesn’t really say anything about their expertise, work, and interests. Their surveys are often detailed about when they are and aren’t available, and the use of tags and topics are usually very high level like: science fiction, fantasy, horror, diversity, writing, etc., which isn’t terribly useful when you are looking for panelists who can talk about “post-battle recovery from a medical or mental perspective within a non-industrial fantasy world.”
Additionally, the number of new, emerging, and midlist writers who are not well-known and who have 1-10 short stories and 1-3 novels is absolutely overwhelming from a program team perspective. We want to include everyone, and that is not possible. So, we have to rely on the surveys …. and when you are just looking at the answers and bios within a survey, easily 80% of survey responses look identical. Plus, the surveys are often sparsely filled out because people don’t understand the kinds of information that a program team really needs to see to understand that Author A would be amazing on a quantum physics panel because she actually works as a researcher in that field, but her survey only talks about her work as a fantasy novelist… or that Artist B would be a great panelist for that topic on how to restore engines in a post-apocalyptic world because he enjoys salvaging old cars and restoring them in his free time, but you’d only know that if you checked out his Facebook page. These are just two examples. I could come up with any number of similar examples.
But the core of the issue here is that when I work on a program, I spend easily half of my time googling authors who didn’t fill out their survey with enough information or writing to them to ask follow-up questions. That may sound like a normal and natural thing to do, but when you are dealing with 350+ people (or 1,300+ people), that can lead to hundreds of extra hours of unpaid volunteer work when you only have about 10-15 hours of time to give to your volunteer work per week. I loved doing it. When I run programming, I start the programming process in August (or earlier) just so I can get the program out by mid-January, and over the decades that I have been doing this, I have written hundreds of thousands of words in notes regarding what I know about people, and for privacy reasons, you really can’t keep these kinds of notes laying round. Yes, there are databases that could be used but you need to get permission to capture certain data and if you ever change databases, you lose all that past data …. and all this energy and effort over the years has burnt me out. People ask why I stepped back from running program. It’s because I simply can’t give that much time to a convention at the expense of my own creative work and mental well-being any longer.
Part of the problem is that we are dealing with real people, with real feelings, with hopes of achieving their dreams, with desires for success, and with love for the community they are a part of … and a convention’s program is the heart of the community because it brings all of these people together with fans, friends, and colleagues to talk about the things they love. Nobody wants to leave anyone out. Nobody wants to feel left out. And nobody wants to cause someone else harm … and it’s so easy to do these things unintentionally, despite our best intentions.
I, for one, would love to use AI as a tool to help cut down on some of that research time so that I could perform much more targeted research that is much more effective. I want to do a good job. I know every other program volunteer wants to do a good job. I also know that getting people to volunteer to work on program is getting harder and harder to do because they are terrified of landing themselves in the center of an internet firestorm, and the people who have the most experience at navigating this space without catching themselves on fire are becoming fewer and fewer … those who are left are burning themselves out at the expense of their own health, mental well-being, and creative success.
I understand the frustration and anger toward LLMs, but I think that we need to grant a little grace and understanding … and even kindness … to the people who are donating their time and putting their hearts, blood, sweat, and tears into trying to create these events that bring our community together.
Sometimes it is enough to say “this action is unacceptable” without burning everything to the ground, including the people … people who are just like the people who are raging in their angry protests, resigning their posts, turning down their nominations, and walking away from the convention. Again, saying “no” is enough. Thoughtful, if not stern, reprimands work far better than internet firestorms.
On a personal note, I don’t think I will ever feel comfortable volunteering or possibly even attending another SF/F convention again. I’m not even sure if this is the kind of community I want to be a part of any longer when every year, almost without fail, I see my friends and colleagues publicly shredding each other to pieces as they tear down different conventions, traditions, and communities.
I don’t think we can solve this problem as long as Worldcon remains run by independent organizations each year. Without a systemic organizational structure that has continuous annual oversight from year to year as well as some number of paid employees to maintain the structural, procedural, and data integrity, we are doomed to repeat the annual Ground Hog Day activities of the SF/F community dining upon ourselves at the annual Worldcon banquet.
Erin Underwood is a writer, editor, and content producer based in the greater Boston area. She has edited several anthologies, including Futuredaze, Geek Theater, and The Grimm Future, and her fiction and nonfiction have appeared in various publications. Erin is also an active volunteer in the science fiction community. She holds an MFA and enjoys exploring the intersections of storytelling, technology, and fandom. Visit her at www.youtube.com/@ErinUnderwood
Discover more from File 770
Subscribe to get the latest posts sent to your email.
“I understand the frustration and anger toward LLMs, but I think that we need to grant a little grace and understanding … and even kindness … to the people who are donating their time and putting their hearts, blood, sweat, and tears into trying to create these events that bring our community together.”
Thank you for writing this, Erin.
People in fandom shredding other people with innuendoes or fake accusations is certainly something I’ve run into. I try to avoid the people who do such things. I also don’t think having paid employees for central roles is a good idea. The question may be how long fan-run conventions will remain viable.
That was one of last year’s kerfuffles, as I recall, with a very famous author who wanted to be on panels, without doing a survey.
I have attended panels filled with people with little knowledge of the topic. I am not advocating AI but some sort of vetting.
All so true, and reflecting my own experiences, especially the point on the modern lack of kindness.
Pingback: Robot Hallucinations | Cora Buhlert
I love this. I’m so tired of the Hugos/Worldcons being a trainwreck, and I don’t even go. How they are currently run is not working. A two year delay to fix things does not work well.
Thank you. An excellent piece.
I speak at a tech conferences and even if you’re known in the industry, they still expect a pretty robust application to speak and details of what you want to talk on and what format you want it to be in.
Panels for those also tend to involve multiple practice sessions with all the participants.
I also think people struggle with how time multiplies – if it takes 15 minutes to review or dig into somebody for vetting or other purposes and you’ve 1000 to get through that’s 10 days work that somebody or bodies have to do.
Here’s my problem with your paean to unsung volunteers, Erin. Yes, they are expending lots of time and effort. Yes, they aren’t paid. But the fact is that both Kathy Bond and SunnyJim have run Cons, including Worldcons before, and should have the good sense goddess gave a walnut. They didn’t think. Their post facto explanations stink of it. It might be better if the SMOFs who do Worldcon might take a couple of years off and see what new teams of volunteers can do.
I am a reasonably well known pro. I get invited to lots of cons, including the Nebulas, Balticon, and others. I have not been invited to be a guest at Worldcon since Kansas City. I was told by a former President of SFWA who was brought in to bail out San Jose that I wasn’t good enough to be on her panels. Frankly, after Spokane, San Jose, Chengdu, and now Seattle, I don’t even bother to apply to be on panels anymore. It’s almost like there’s a list and I’m on it. If I was a truly suspicious man, I would think it might be that I’ve edited some books for Baen. (The alternate history magazine I edited didn’t belong to Baen, even though Jason Sanford said it did…it belonged to the late Eric Flint.)
I have to be at Seattle, because I’m running the fan table for the Heinlein Society, of which I am Vice President. But right now, I’m seriously considering suggesting to the Heinlein Society Board that we not appear there.
I don’t see the problem with using AI to screen panelists. But that’s not really the issue, if that hadn’t happened there would be screaming about US politics or something else. It’s this addiction to drama and outrage that is a problem. And I don’t think changing the structure of Worldcon will solve it, there needs to be a culture change. Concoms just have to be more comfortable with saying no, saying that’s just the way it’s going to be and not trying to fix everything that’s criticized. Look at the reaction to DC changing the date of the con versus to the sponsor. The date change was a huge deal practically but didn’t get much attention because the concom clearly said that was their decision. While the sponsor blew up because they apologized and seemed to reconsider.
90% of the content on social media is posted by 10% of the people. Stuff on there can be ignored. Instead you can look at real data like membership sales and only act if you see a problem there.
Erin Underwood’s heartfelt and wise column keys in on the most important issues: respecting our common humanity.
And not just in theory or abstractions, but the real nitty gritty of volunteering (which I’ve done in shepherding half a dozen national and regional conventions, including being a division head at one 1980s Worldcon).
Without wide respect for everyone who volunteers, and realism about both the limited time we all have and the various fallibilities that we all are heir to, despite our best efforts, I fear fandom will fall – and not just Worldcons.
Yes, I do have concerns about AI (while being highly skeptical of both the hype and hopes about it and the wildest doomsday fears), especially to the extent that it violate individual rights and steals any creator’s work without their consent.
But it seems that much of the criticism of the Seattle Worldcon indulges in moral self-righteousness and virtue-signaling that if we’re not careful can become a lynch mob and bring all the walls down.
Why can’t we keep in mind, as Ms. Underwood does so humanely and reasonably, about the concrete practicalities and limitations of time and knowledge that actually circumscribe all of our lives and careers?
Because that’s the messy, imperfect reality of what we laughingly call civilization – and it’s a far thing from the constructivist/rationalist perfectionism that evokes the horrors of the Greek myth of Procrustes and his murderous bed.
If the best method of evaluating a problem is to see who’s voting with their feet, a number of program participants have withdrawn, some members have asked for refunds, and a Lodestar nominee withdrew from consideration. That’s a pretty clear indication that this is a problem for the convention’s main constituency (participants and members of the convention, as opposed to wider fandom and randos on the internet; though given the nature of Worldcon as a movable feast and WSFS as an ongoing concern, one hopes all individual concoms look at least a bit beyond their own convention).
I strongly suspect Seattle’s attendance already took a hit due to issues beyond their control; a lot of prospective international attendees have very good reason to be nervous about traveling to the U.S. Trying to do a postmortem and sort out who didn’t come because they worried about being detained versus who didn’t come because they object to the use of ChatGPT will require looking at more than just numbers. It’ll mean looking at private emails to the convention and also public statements. It’s hard to prove negatives or extrapolate the counterfactual of a Seattle Worldcon with Harris in the White House, but future bids (especially U.S. bids) are going to need to look at the qualitative and quantitative data.
I think There’s a bunch of really good observations with regards the need for good applications, but it does feel like it is blaming authors a little too much for the lack of useful information.
for ex if I look at the application sheet that Worldcon posted.
there were a few basic questions with regards to identity, socials and history being a panelist. a bullet point list of your topic interests and a 500 character box for:
No decent survey is going to solve the; i’m a funny writer, let me make a joke cause i’m popular.
But this does not look like the type of survey that will give you the useful responses you want from potential panelists. Let alone from first time panelists, or relatively new panelists.
telling people to be succinct, is certainly going to get you very sparse replies.
if you’re looking for good information – and you know what good information is – as a program maker, one of the things you also have to look at is how do we make the application forms in such a way that we get the best information.
This is a little bit of a two-way street. give examples of what you want to see and hear from potential panelists.
Make it easier on yourself to get good replies by drafting a solid application survey that features examples of the type of responses you’d like to see.
Help prospective panelists by telling them what kind of information you are a looking for.
All that said – the amount of hours volunteers put into these things to deliver the best possible experience for all fans and attendees cannot be understated. Yes mistakes are made, but they’re not made because they want to watch the world burn. How we the commenters react to their mistakes will also affect how mistakes are fixed, and how willingly volunteers want to work to give us visitors a great show.
I don’t think anyone signs up to spend hours googling for sexual misconducts scandals or worse. it sucks that we’re at this necessary stage, and i understand fully why this is the part that we want to automate. Hopefully we’ve learned some lessons and we can build better processes for next time that do not use tools built on the plundered spoils of participant creativity.
It must be weird to go through life thinking that everybody who is motivated by a desire to do the right thing is faking.
@walt boyes wrote
New teams of volunteers are welcomed and encouraged to mount a bid! We’d love for them to show us what they can do! But right now we are limping along with one bid in most years, so if everyone currently involved in Worldcon running takes a couple years off, there are no Worldcons.
@rcade: No, “moral self-righteousness and virtue-signaling” are motivated by desire to do the right thing. The problem comes when desire to -show- that you are motivated to do the right thing overwhelms the ability to figure out what might be a useful and productive right thing to do.
Erin Underwood is wise. A little more kindness and understanding would not hurt, especially when dealing with volunteers who are doing a lot of work.
Volunteering on Worldcon program can be very fulfilling. It is a great way to learn more about the interesting, highly knowledgeable, and often very nice people in the science fiction community. Plus learning more about topics of interest and the history of the field. You are working with a large team of other volunteers who are also interesting and often very nice people. There can be frustration and mistakes happen, but Worldcon program teams have high standards and even when they fall short, they do amazing things.
The first Worldcon I volunteered on, I was on the program team in a technical capacity. I put in 20-30 hours a week for two years. It was like having a second job. I don’t think I could do it like that again. But I was younger and had the energy. It was fun.
On some other Worldcons I’ve been in the brain trust, where you get to talk about program ideas with the program team. It’s not what I would consider work. It’s not a staff position. It is a lot of fun, if you’re into that sort of thing. And it does help the program team. It is one of the ways the team can pull in knowledge and experience from a much larger group of people.
I’ve also done the program for smaller conventions, and have friends who have been Worldcon program division heads or assistant division heads. Overall I am more optimistic about Worldcon programming and volunteering. It definitely helps to have comments like Erin’s reminding us of our core values.
The advice that I would add is to be aware that the program team is always working behind deadlines. When they post a questionnaire or ask for feedback, the sooner you can provide it, you are giving the gift of time, which is very valuable. It makes everything easier and less stressful.
I’m pretty sure this is a stupid question. Why can’t the information collected by one Programming Department be passed on to the next year?
I think that one of the reasons you have famous people not fill out the forms is because they have filled out so many forms, and it can feel like you’ve already told people this a thousand times. Also, is there anyway to standardize the form from year to year? Even if you couldn’t pass on the data, if the form was the same, then repeat prospective panelists could just grab what they’d done the previous year.
Also, is the vetting done from scratch every year? Other than word of mouth, I mean. Again, is there any way to pass on the copious notes and other information that you must collect every year to the following year?
Lydia Nickerson: While I can’t swear that information is never passed on, there are always people ready to veto the idea because they believe it violates European data collection laws. And for all I know they may be right.
The problem with “I, for one, would love to use AI as a tool to help cut down on some of that research time so that I could perform much more targeted research that is much more effective” is that, as many people have pointed out, LLMs are not a useful tool for doing that. They make stuff up, they have built in biases, and in particular the fact that Seattle gave ChatGPT prompts that included people’s names and words like “scandal”, “homophobia” and “racism” means that those names and those words will now forever be associated in ChatGPT, leading it to be more likely to output those associations.
Nobody — literally nobody — is against automating parts of the volunteers’ work to make it easier for them, and many people in the comments to other posts on this subject have suggested ways that that could have been done with python scripts and so forth. The problem is that people’s data was misused without their consent, in a way that may be actively harmful to them, using a tool that is actively harmful to all writers and to the world, and a tool that does not even do the thing they were trying to get it to do.
(Also, I strongly admire the Glasgow team’s solution to the entitled-author problem last year, of simply saying “he didn’t fill in the forms like everyone else, he doesn’t get to be on the panel”. If famous people, of whatever level of fame, won’t respect volunteers’ time, that should be a them problem.)
There are laws that govern the collection and sharing of personal information, and especially sensitive personal information. Since Worldcons move between countries, you can have conflicting laws that govern how data can be collected and who it can be shared with. And then there’s the threat of legal action – if you write things down and store them somewhere, they can be subpoenaed. Hacking and leaking would be other risks of having a centralized store of data, and then those things could lead to legal action. There’s a lot of risk about storing data, and volunteer-run conventions with slim budgets don’t want to incur those risks. Some might argue that the risks are not as big as people think they are, but folks are very cautious in this regard.
As to standardizing the form, I expect they are pretty similar from year to year. With our decentralized system there’s no mechanism to FORCE anyone to do anything a particular way, but folks don’t like to do more work than they have to, so there’s lots of borrowing and re-using.
If you are a participant who doesn’t like re-writing your information every year, you should save your previous answers to drop them into the next convention that asks you for info. You might type in your Bluesky handle before your Facebook one, or might be asked for your people you want to avoid being scheduled with before you enter your availability, but the basic things you’ll be asked for are the same.
@Andrew Hickey says
One, LLMs don’t actually work like that – they don’t automatically absorb prompts and incorporate them into the model. Prompts can be used to train future versions of the model, but it’s neither immediate or assured.
And two, why is this better than typing “Andrew Hickey homophobic?” and “Andrew Hickey racist?” into Google? While we don’t know for sure that a future version of Chat GPT will incorporate that prompt, we 100% know that Google pays attention to these things – otherwise it would not offer up those handy search autocompletes.
@AndrewHickey I actually believe there is a great deal of education needed within our community and within the general population of the U.S. and the world regarding what AI is and isn’t, what types of AI exist beyond just LLMs, and how to actually make good use of an LLM (because there are ways to use them that are extremely effective).
I am not going to talk about these topics in this post because I don’t want to diminish a conversation that is focused on volunteers, communication, and how we treat each other with an educational discussion focused on LLMs. I am looking at writing something for Mike or someone else on that topic… but I also don’t know if I have the time for it. And, yes, this is what I do for a living. So, I do have credibility to bring to this conversation…
I just don’t know if I have the heart to have this conversation here, but I am considering it now, which is more than I was willing to do before the Seattle Kerfuffle. I just can’t keep watching my friends, colleagues, and community members fighting with each other any longer. It’s time for conversations……
@Andrew Hickey – to echo Tammy’s point – none of the public “AI” models train their datasets that way – the ways they have trained them is bad enough thank you but that? Not a thing, and not likely to be for lots of reasons around the way they work. Holding that kind of context for random tokens takes up resources that even these outfits don’t like tying up.
Ironically – the one outfit that does train their search algorithms that way is Google as Dan Savage proved with what he did to Senator Rick Santorum a few years ago.
People suddenly doing a lot of Google searches on “Is Y homophobic” can tilt their input string predictor and their top results.
But that said, in all cases the scale of what was done here and could be done isn’t going to be logged in any meaningful way and certainly not stored anywhere except the account it was done with and then that can be removed easily.
General Observation:
There is a conflict between people wanting there to be retained and specific individual data and knowledge that can move between conventions in different countries [a random example – the con actually does, heavens help us, go to Saudi – do we want a database of people who speak well on trans and women rights issues going with it?] and people who are paranoid about any use of data such as their name which many of us share with literally 1000s of other humans.
I discovered while trying to get on the program at a Worldcon more than a decade ago that someone in charge had decided I am obsolete, or evil, or something, and so have given up trying. The fact that I was a Worldcon Fan GoH, am a Hugo winner, have a wealth of knowledge about fandom and the genre and am even an interesting panel participant are, apparently, no longer of interest or useful.
Also, being an old white guy is now seen as not in my favor.
Speaking as one of the “number of new, emerging, and midlist writers who are not well-known and who have 1-10 short stories and 1-3 novels” – thank you for this post. It is a helpful reminder that applying to be on a con is much like submitting a novel to a slush pile.
Having said that, I think we have a chicken-and-egg problem. If I don’t know that the concom is looking for (in my case) somebody who can talk intelligently about running large-scale IT projects, I may not put that in my bio. This is why I find that the two-step process of applying and then being presented with a list of panels to bid on is helpful.
I would also like to heartily say amen to the idea that we need ” a systemic organizational structure that has continuous annual oversight from year to year as well as some number of paid employees.” This goes for all aspects of Worldcon and the WSFS, most especially including Hugo voting.
On the logistics problem — it does seem like this is a problem that’s been solved in the mundane world for a long time, with databases of information (either curated or crowd-sourced like wikis) or the various review systems where folks can leave reviews and comments. A “yelp/wiki for panelists” would allow all the conventions to share this effort, and all it would take is one big convention like the worldcon telling every programming applicant, even GRRM, “start an entry for yourself, but your info in the self-provided section, and then others will add reviews.” Of course there is the SMOP, but the work would pay back manyfold. Of course, the mundane world had commercial reasons to build these systems. Though when it comes to wikis there’s lots of open source, and for those who already have a wikipedia page you just import that to keep it honest.
But as to the OP’s bigger point–I share the view that Fandom has become a nasty place, and the pile-on on Kathy Bond (whom I don’t know) is one recent and sad example.
I did a Google search for “Chris Gerrib” and immediately found lots of good information about his writing. It looks quite interesting, and if I were researching him for a program, I’d happily follow up on it. The IT manager info is more sparse but it is there. Yes, the SF community has gotten big and there are a lot of new people I don’t know, but we have the internet.
@Brad Templeton
There’s the problem of people leaving bad reviews just because they don’t like a person. (Or because they didn’t get the kind of service they thought they deserved.)
@Linda R I have attended panels filled with people with little knowledge of the topic.
So, so true.
Indeed my recent experience has been that about half the panels I attend have been very poor. So much so that these days if over 25% of a con’s programme is panels then I will not go.
Solo talks tend to be much better for various reasons including:
1) The speaker has to prepare.
2) An ill-informed speaker cannot hide behind other panellists
3) A speaker usually has to justify to a con why they are appropriate for the talk they propose to give.
And, of course, there is the lack of films at today’s convention’s Back in the 1970s – 1990s Eastercons and Worldcon (and even in the ’70s to early ’80s smaller cons such as Novacons) had film programmes. A film is invariably better than a poor panel. And today we have an explosion of remarkable independent SF offerings. Sadly, last year’s Worldcon was the first British venued Worldcon not to screen any films whatsoever! Conversely Loncon 3 had offerings from the British Film Institute and Sci-Fi London. (Alas the venue they held it in was too small for the number wishing to see the SFL films.)
Worldcon organisers might wish to recognise that many attendees wish to see films that have not had a general release. This is demonstrable by the number nominating and voting in the Dramatic Presentation long-form Hugos (invariably over a thousand) and also last year’s Glasgow film Hugo survey that revealed a majority support for films… But you would not think so from the programme organisers gave us.
@brad
Great idea! If every participant had to open themselves up to online ratings and abuse through fake reviews to request to be a participant at a Worldcon, that sure would reduce the applicant pool to a manageable amount!
I think you’re trying to solve the “what is this person good at part of the question. But when it comes to vetting, the point of such a database would be for information like “good panelist, don’t use as moderator” and “fine for solo items, does not play well with others” and “is rude to women panelists” all the way up to “flashed their breasts to a crowded room” or worse. A public database is a horrible place to put that if you don’t want to be sued or attacked by fans of that person. And a private database is subject to leaking and hacking, so not much less risky.