Status Report

File 770 was conspicuously missing in action after posting the Hugo nominees yesterday.

My internet service provider says there was no denial of service attack.

Nor was it a bandwidth issue – the plan I was on had unlimited bandwidth.

Frankly, I don’t understand the explanation but it amounts to the system thinking I had maxed out a different service metric. The solution was to upgrade. That’s why we’re back on the air. (Hooray!)

Every silver lining has a cloud, unfortunately. When they migrated my files to the new server, they lost everything after April 22. Tech was unable to say why.

I reconstructed all the lost posts using Google cache files. But I couldn’t make the new posts use the old permalinks, so if you want somebody to find them, send them the new link. And unfortunately the comments aren’t there, because these are “new” posts.

132 thoughts on “Status Report

  1. Hi Mike, I have been lurking for ages but as someone who works a lot with WordPress, I wanted to extend my sympathies for your server troubles. My guess why nothing after April 22 is that your host’s backups may be on a weekly basis so that was the copy they had of your site/database.

    You can install a plugin for making your own backups if you want to, you can even find some plugins that will do it automatically for you once a day, week, month, etc. I would recommend not keeping too many backups around as I imagine a site your site with all the comments, the ZIP file will be huge, but even just having the one file replaced each day with the new backup might be something worth considering. The alternative may be to pay your hosting to do their own backups more often.

    Also if you would like my help fixing your header so there isn’t so much white space at the top, I am a WordPress CSS wizard when it comes to tweaking themes and I could email you the css, just let me know where you’d like it emailed to (or posted here, don’t matter to me, it’s not top secret info heh heh)

    If I can think of a way to get those email-saved comments added back to their respective posts, I’ll let you know. I often solve these sorts of problems at my day job, though I am less of an IT person and mostly just a humble web designer.

    Hello all, btw. I enjoy reading all your discussions of books, I just don’t read as many new books as I’d like to these days hence the lurking. (sorry to repost, linked to the wrong account)

  2. From “A Scrolling Class Hero is Something to Be”, right before the site went down, a suggestion for robinareid’s collection of single-sexed planets that might otherwise be lost. I happened to keep that tab open…. Since there was no post by robinareid since, they might have missed it.

    Jenora Feuer on April 26, 2016 at 10:10 am said:

    @robinareid:
    Apologies if this has already been mentioned, haven’t read the later threads yet. But how about The Disappearance by Philip Wylie, from 1951? Basic idea starts with people waking up one morning to discover that there’s been some sort of timeline split, where one copy of Earth has all the men and one copy has all the women. And then following the two parallel tracks.

  3. Are there any WordPress experts here?

    I have solved the problem Mike describes — getting blog posts their old permalink back — by editing the MySQL database directly. I changed the ID field of each blog post in the wp_posts table.

    This worked for me, but it was a blog with under 10 posts and no comments so I didn’t have much to lose if I made a mistake.

    I think Mike can do the same thing, after making sure he’s got a backup of the entire database just in case, but I’d like to find out if I’ve overlooked something about how the WordPress database works.

  4. Hah! My evil plan of suggesting a Meredith Dragon in the Worldcon park worked as a Meredith-summoning spell! <cackles> <rubs hands together>

    Seriously, welcome back, Meredith; you’ve been missed.

  5. rcade, if you can do this, you’ll be our hero! And if you make it to Worldcon, I foresee drinks being appurtained for you… (not by me, alas; I won’t be there.)

  6. Echoing what several others have said: I would be happy to help support File 770 overhead via Patreon or similar methods, if OGH is interested in pursuing such means. This place gives me much joy.

  7. And if you make it to Worldcon, I foresee drinks being appurtained for you… (not by me, alas; I won’t be there.)

    I like the sound of that. I’m planning to attend this year, if work and finances permit.

  8. Good to see the File back! I was starting to worry about a ddos, to be honest.

    Not sure if this is the best place to put it, but I would like to nth Kyra’s request to OGH to have a separate Pixel Scroll and Puppy Roundup each day. The enforced downtime for me yesterday made me do some hard thinking, and I will need to take a huge step back from the probably inevitable vitriol that Vox Day and his Ilk bring to the game.

    I adore the geeky community I finally discovered here, and would love to continue to discuss SF/F works. The Rabids? Not so much.

    Also, hi Meredith! *waves* It’s good to see you.

  9. I solved my permalink problem by using exact titles and dates and file names when I had to recover an entire website of 50-100 files.

    I’ll be doing the same when I convert a non-Wordpress site over to WordPress. It had to be done by hand and took hours looking over the Google archive to make sure I had everything. I’m not looking forward to converting the website over at all.

    Having done this more than once I now use the title although if my titles were too long I’d use dates. This makes it easier to recreate permalinks if you need to.

  10. Not sure when this comment will appear as I still need my account approved to make comments without moderation but here’s something to keep in mind about changing page ID numbers in WordPress, it’s usually not just a case of changing the id once in the database but it depends what other queries are involved in how the database is set up (ie media page attachments, plugins, etc):

    https://wordpress.org/support/topic/how-do-i-change-page-id-number

    Disclaimer: I am not a developer so I sadly can’t be more specific than that, maybe you could start a support ticket or forum post on the WordPress site? If you want a quick and easy way to make your own WordPress database backup, I reccomend WP Clone, it makes a zip of the whole site and if need be, you can use to restore the site if something goes wrong.

  11. It wasn’t just you. Yesterday a bunch of my feeds were MIA. Wonder if an update upstream somewhere mangled things.

  12. File 770 is back up and running and Meredith made a most welcome appearance. A wonderful start to my day!

  13. Wonder if an update upstream somewhere mangled things.

    There was a WordPress update yesterday that included a database bug fix. But I don’t see anybody screaming bloody murder about that fix on WordPress’ support site, so it seems unlikely to have caused the File 770 interregnum.

  14. 770 is back!
    Meredith is back!

    It’s an “I just returned from my goddaughter’s First Communion in Tulsa” Day miracle!

  15. Emerging for lurkerdom to tell all of you that I donated to the gofundme. File 770 and the people here have saved my sanity numerous times during the puppies debacle. Thanks!

  16. @Cassy B:

    Since there was no post by robinareid since, they might have missed it.

    Thanks for retrieving that!

  17. Jenora Feuer, I’m hoping that rcade can resurrect all the old threads, but I’m glad to do my small part. Alas, I nerfed most of the tabs I had open by hitting “refresh”.

  18. I’m a WordPress intermediate, not an expert– but I’ll do what I can.

    Any SQL experts out here? Or better yet, GREP experts? I’m on the lookout for people who can parse large piles of File770 emails and strip them into an import format.

    Oh, and I’m looking for people who have File770 email archives.

  19. @Meredith: Hi!!! I’m very happy to see you poke your head up here. 😀

    @Will R.: “In other news, I’m starting to wonder if the Space Raptor isn’t going to get the rabids in the end.” – ROFLMAO!

    @Jim Henley: “Folks, we need to crowd-fund Mike’s new hosting plan.” – I was thinking something similar and mentioned it to Mike in an e-mail. I’d donate, though I wouldn’t want this to force him from fanzine into semiprozine. 😉

  20. In exchange for the lost posts, the problem where the front page wouldn’t show up-to-date comment numbers or the list of most recent comments seems to have been fixed.

    Hey, I wonder, if there have been server-side upgrades… ??????

    ETA: Curses, no it doesn’t.

  21. Will R. said:

    In other news, I’m starting to wonder if the Space Raptor isn’t going to get the rabids in the end.

    I see what you did there.

  22. @Glenn Hauman:

    Um… I’m a SQL expert as part of my career (ASP.Net & PHP developer – in fact, I was just toying around trying to explain unclosed data connections and memory leaks in layman’s terms for Mike since that sounds like what he’s talking about). Unfortunately, I haven’t really worked much in GREP. Not sure if that helps at all.

    I wouldn’t mind learning me some GREP if it helps, but that’d be on a scale of days and that wouldn’t help if you are hoping to get this done today or tomorrow. 🙂

  23. Cat: Also, if needed, I live in the US and will be driving to MidAmeriCon II.

    Hampus, if Cat’s willing to transport your items by vehicle, that would probably work better than me trying to ship them or take them by plane. If you haven’t purchased anything yet, hold off and see if she’d be willing to have Mike give you her address.

    Would that be okay with you, Cat?

  24. Cally: Mike: is there anything you’d like any or all of us to do to help you get back/recreate the posts and comments that were lost? Or are you going to just accept the week of amnesia and go forward from here? I’ll leave the tabs I had open from the last week Just In Case.

    While doing any of this is far beyond my own technical ability (which is limited to typing stuff into windows created by WordPress, or pushing the occasional button), a couple of people here are airing ideas for recovering comments. So if you have the ability to access any comments we lost hang onto them for a little while til we decide if we’re really going to try something.

    (If I had enough clones I think I could brute-force recreate any comments for which we had the text by logging in as other users and then afterwards editing timestamps using my admin access. I did this in January to move a pair of comments from a Scroll to the Star Wars spoiler thread. The amount of time to do that isn’t worth it.)

  25. I know it’s possible to use a plugin to import data from a spreadsheet as individual posts/pages (or in my case, as products for an e-shop plugin like Woocommerce with areas for price, SKU, description, etc) but I am not sure the same is possible with comments, though it would be what I consider an ideal solution if someone was willing to organize all the comments by copypasta into a master spreadsheet as the spreadsheet would allow for separate columns for items like attach to which post, poster name, comment, email, website, controlling comment order, etc. When I get a free moment, I will see what Google turns up, either as a comment importer or something like that.

  26. Glenn,

    Grep is probably not what you want. It is based on a view that sees lines of files as separate records of interest, whereas most elements of an email in a file extend over multiple lines. Really, unless your starting mailbox format is especially straightforward and the transformation you are interested in is especially simple, you need something that contains the rules to parse the mailbox format into a form you can work with, like one of the available modules in perl or python.

  27. As I think about the issue more, I don’t expect to find comments for the lost blog posts. If your ISP lost posts after April 22, it’s likely they lost all comments after that too. The most likely reason the ISP lost those posts is because April 22 was the date of their last backup of your site’s data.

    If people have the lost comments, I think an import file can be created to add them back to the blog. This wouldn’t require logging in as other users.

  28. @Ken Marable

    Do you have any sort of data dictionary for the WordPress database? I think cleansing the data wouldn’t be that bad but we’d need to know the data structures and relationships / dependencies. Also we’d probably need a current copy of the database to see things like unique identifier sequences. That might answer the structural questions too.

    From a skim at WordPress it sounds like an ID number associated to the Post ID would need to be created for each comment.

    Agreed that days is probably the scope and not hours without a lot of existing knowledge of WordPress back ends. Also anyone with access to the DB would have access to the data in the DB including e-mail addresses and such so trust is imperative.

    (Definitely not a WordPress expert of any sort here nor a DBA / Programmer though I can muddle around in SQL. I have helped cleanse a lot of data for migrations FWIW though)

  29. Is it normal for an ISP to not back up all data every day, at a minimum? That seems very odd to me.

  30. @ULTRAGOTHA – it depends what type of hosting you get, generally the cheaper options offer either a weekly backup and some don’t offer an automated backup at all. Some people prefer this, as they prefer to set up their own backup programs/protocol. Or some people just want the cheapest website possible and never even think about backups (until their site disappears *then* they panic lol).

    When it comes to creating simpler websites with just HTML it’s not as much of an issue, as you would have a copy of your site files on your computer anyway and if the site got wiped, you would just copy and paste the files back onto the server. It’s when a WordPress site with database using PHP is involved that it’s not as straight forward to restore a site, there are certain steps that have to be done regarding the database setup (it can be very confusing to picture because so many database-related elements are dynamic and don’t, on some level, really “exist” as a whole the way static files do, like a JPG file or HTML file… hopefully *I’m* not confusing things too much lol) and that’s why recovering the comments is not just a matter of adding them back to static HTML pages or recovering page IDs is a bit more complicated than just renaming files – because other database files might still have the old names/numbers listed so you have to find all the cross-references, so to speak.

    Sorry if I am being over-explainy, I do WordPress training for work, so I talk about WordPress a lot lol

  31. @IanP – oh geez lol Every time someone suggests to me “hey why not learn how to be a developer for web?” I think of that stuff and get cold sweats lol At least with web design, the worst that will happen if I mess up is that the site will look silly, it will still be there and usually work too

  32. Do you have any sort of data dictionary for the WordPress database? I think cleansing the data wouldn’t be that bad but we’d need to know the data structures and relationships / dependencies.

    It shouldn’t be necessary to do that. WordPress has an import format called WordPress Extended RSS (WXR) that pulls in blog posts and comments, putting them in the proper place in the database and indexing them correctly.

  33. @Sunhawk

    I’m a sysadmin, a role that provides so many opportunities to look incredibly silly.

  34. @IanP – if anyone ever invents an Undo button for the internet/databases, they will be as rich as the rich dude who financed the mysterious space machine in Contact lol

  35. Waves to Meredith.

    I spoke to Mr D about extracting the comments from emails and he said, as I knew he would*, “A few lines of PEARL should do it”. So, ticky, and I’ll look at the format.

    Rcade, what format does the importer need? Have you got any links? Neither of us knows anything about WordPress.

    *He likes PEARL, I like spreadsheets. We may argue about the most effective way to solve problems.

  36. Rcade, what format does the importer need? Have you got any links?

    Here’s an article documenting the format.

    If you create an export file of a WordPress blog (menu comment Tools, Export), that file is in WXR format. It is an RSS 2.0 file with some WordPress extensions.

  37. I work with RSS files all the time after serving as chairman of the RSS Advisory Board, so when we reach a point that a WXR file needs to be created for File 770 comment import, I can do that.

  38. Trying to think outside the box, as I will leave the database box to those way more skilled/expert in PHP than I am, if for some reason it’s not possible to repost comments like normal, maybe they could still be just stitched together into a simple .txt file that Mike could just upload and link to at the bottom of each post, with an explanation that they were from ThisDate to ThatDate. I think I’ve seen other websites do something like that, or just use a PasteBin page to host the text, like you might do when documenting a chatroom log.

  39. if anyone ever invents an Undo button for the internet/databases, they will be as rich as the rich dude who financed the mysterious space machine in Contact lol

    Well, such things exist, but they have to be baked in at a lower level. Look at things like the UDF filesystem on DVDs, where support for DVD-R (write once only) media meant that nothing written to the disk already could be overwritten, and instead you had to just add the changes after the originals. Or most formal source code control systems.

    The main flaws with such things tend to be:
    – They all have to be set up and actively used in advance of you needing them.
    – They all use a LOT more space as you need to be able to access or reconstruct all the old versions of the data.
    – They tend to be slower for much the same reason.
    It’s a long-solved problem, but nobody likes the side effects of the solution.

  40. Oh, and that’s on top of the fact that sometimes, for whatever reason (including legal requirements), you actually do need to be able to permanently delete things and ensure that they can’t be recovered afterwards.

Comments are closed.