You should pay attention to this post. It may be a little drier than I sometimes am, but I’m going to tell you how to cover your ass in the event of a website mishap. I learned through experience and figured I’d share.
So here’s what happened.
Last Wednesday, I decided to sleep in (which is rare), and Robin came into the bedroom after I’d laid around enough and said, “Apparently your site is down. You’ve got all these emails like, ‘Dude, what’s wrong with your site?’”
Now, I’m not prone to freakouts, so I took my time getting in to the office, turned on my screen, and went to my inbox. And it was indeed filled with “Dude, what’s wrong with your site?” emails.
So I went to my site. Which looked great, except that I got that “404 – Page not found” error on the front page, and didn’t see any posts, or any of my sidebar stuff. I logged in and saw that I currently had zero posts, zero pages, and zero comments.
As of the time I’m writing this, I actually have 131 posts, 90 pages, and 2145 comments. Those stats have changed some since last week, but even back then, I was able to notice that zero posts/pages/comments left a discrepancy.
(I should pause here to describe the sinking, panicked feeling I had at this point, but I honestly felt nothing. I’d wager that some of you reading this right now feel more sinking and more panicked that it could happen to your blog than I did. And it totally could. I don’t know, maybe I’m just confident or something, but it was strangely bland.)
It was cool, though, because I’d been doing the right things. I was prepared for something like this.
What are the first and second rules of computing?
The first rule of computing is to back up your files.
The second rule of computing is to back up your files.
Eventually, if you stay in this world long enough, someone will make a Fight Club reference and mention the First and Second Rules, and nerds will start pulling out flash drives.
So the good news was that I had been doing the backup thing… which was fortunate, because I found out later that thanks to some technicality, my webhost had not been making the backups they promised. So if you aren’t backing stuff up on some level right now, there’s a chance that your webhost is doing (or not doing) the same thing, and you’ll have a crash and be totally and completely fucked.
The good news is that I’d been backing up.
But the bad news is that you never know if a backup plan works until something bad happens and you need to restore. And in my case, what I found out was that the backup itself was completely fucked.
When I went in to the backup console and clicked the little “restore” button for my last backup, it just kind of kept working and kept working but never finished. I tried a manual restore with the same result — totally fucked.
A handful of posts and pages showed up. I think around 10 of each. It was nice to have some of my content back, but the other 121 posts? Gone. And that wasn’t the worst part, because for most people, old posts are old news anyway. Technically, I could repost (out of order, randomly) and reformat them based on the original text files, which I had.
But all of the stuff missing from the sidebars? And the other 80 pages? I didn’t even know what those pages were. I couldn’t recreate most of them if I had to. Few of them were linked from the main page; they were pages I referred to for pre-sale or post-sale – things like my 10-step process for clients to get hosting the correct way. Sales pages I had painstakingly written. Ugh. The thought of recreating all of that or losing it was not cool.
To attempt to keep a long story shorter, the problem was that my backup file was somehow corrupt. Andrew Norcross diagnosed it as some fuckery that occurred when I moved from LearnToBeYourOwnVA.com to JohnnyBTruant.com wherein I was kind of somehow pulling content from two different databases. (Don’t ask; I’m not a database guy and don’t understand it myself.) He said that the database and the calls it made for content just got so muddled up that the whole thing kind of died a little, and then died a lot.
Andrew meticulously rebuilt my database from the corrupt backup files, and then I reposted the recent post I had made and my assistant Amy re-added the comments people had made in the time between that most recent backup and the crash. I made a few more tweaks and in under 36 hours, I was back to 100%.
Back to 100%, but also really paranoid.
So here’s the bulletproof stuff I’ve put into place, with Andrew’s advice. It’s a bit of overkill, but I feel super-secure now that I’m protected going forward.
• The Bei Fen backup plugin: This is what I had before – the backup measure that allowed my site to rise from the dead, but which also gave me a screwed backup. I’ve decided that the screwed backup wasn’t the fault of this plugin, but it was that “two databases, overwrought” thing that caused that mess.
When I was originally looking for backup plugins, Bei Fen was the only one I found that would make a COMPLETE backup of the site (database PLUS all of your static pages, media, images, etc.) and not insist on emailing the giant resulting monstrosity to you. It puts the backup in a directory on your site. Upside? Quick and easy and convenient. Downside? It’s ON YOUR SITE. If the whole site gets corrupted by malware or something, then there goes your backup.
But I use it anyway still, as a catch-all. I have it make a new compete backup every Wednesday and replace the old backup (otherwise the space consumed gets HUGE). I moved my weekly backup to Wednesday because I found out that my webhost makes backups on Sundays. So I verified that they would keep making them, and between the two, my most recent complete backup will always only be a few days old.
(The files on my computer also function as a second but imperfect backup of files, media, etc… everything but the database.)
• The WordPress Database Backup plugin: As it sounds, this one makes a copy of the database only – but that’s arguably the most important part of a WordPress site. It will email you the database, so I set up a new email account using Google Apps (which is what runs my main email address) and have the databases mailed to that account daily. This means that the database backup is being stored off-site, and it’ll archive automatically… the databases will just accumulate in that email account until I go in and clear out the old ones.
Two considerations here: If the database is all fucked up and overburdened like mine was, it may not backup quite right and/or be corrupt. Also, if the database gets too big, your server won’t email it out. It’ll say, “No way, dude… attachment too big. Fuck off.” And it won’t tell you that it’s refusing to send them, and you’ll find out when you need one and your most recent one is from six months ago.
Both of those problems can be solved by this next piece:
• The WP-Optimize plugin: This one adds an area on your dashboard where you can go in and clear out the junk cluttering up your database, thus making it function more cleanly and keeping its size down.
Two things that will clutter you up like a bastard are spam comments and post revisions. I knew about spam, but the revisions thing took me by surprise. Apparently, by default, WordPress saves every single revision you make to a post, and those revisions NEVER GO AWAY. Go in and change “a” to “an” in a certain sentence? You’ve just created a post revision, and it takes up as much space in the database as the actual published page. Thanks to post revisions, I believe my database was 10-15 times as large as it should have been.
You have to actually remember to use WP-Optimize to clear the crap out, though, so don’t forget. Andrew suggested doing it weekly.
The three plugins above would have addressed my issues if the same thing happened in the future, but then there’s hacking and malware to worry about too. I hadn’t gotten hit by those, but didn’t really want to wait and find out what it was like. So I added this other stuff too:
• The Login Lockdown plugin: You can eventually break a password if you just keep trying until you get it. To combat this kind of “brute force” attack, what this plugin does is to allow someone to try only a specified number of login attempts before locking them out of the system for a specified period of time.
• The Secure WordPress plugin: There area bunch of geeky ways that hackers can get at your site and that I don’t totally understand. By making some changes to the way WordPress presents itself (doesn’t show the version number which would indicate version-specific weaknesses, removes certain error messages that give hackers tips on what they’re doing right and wrong), you can lock your site down even further. This plugin lets you control and change those weaknesses.
• Sucuri malware protection service: I did this one purely on the recommendation of Tony Clark, who’s the Copyblogger Clark that doesn’t come out in public as often as that Brian guy. (And no, they’re not related.) Earlier this year, Copyblogger had a huge malware infestation and subsequently got blacklisted by Google. Sucuri got them out of it and got them back on Google’s good side. What’s good enough for Copyblogger is good enough for me, so I signed up. It’s like $90 a year.
Basically, Sucuri monitors your site and looks for unexpected or nutty software, activity, database calls, and other stuff that I don’t understand. And it lets you know when something’s fishy, and gets you back up and running if bad things go down.
Based on what Tony told me, hackers can get in through your webhost sometimes, which means that the stuff that you can do to protect your site can end up being pretty irrelevant — you’re barricading the front door while they sneak in through the back. Sucuri covers your ass by watching all of your doors.
So now, I feel a lot safer. And given that all but Sucuri above are free, you really should do this stuff yourself. Because having a website problem sucks giant balls.
Be safe, y’all.
When I smiled and told you on Twitter that the situation would make for a good “Lessons Learned” post I never expected it would be 36 hours till your site was back. I’ve only got about half of the stuff listed on my site, guess I’d better get busy.
Just bookmarked this post–excellent reminder! I know quite a few people who lost their sites and blogs earlier this year in a webhost disaster, and a lot of them apparently didn’t have any backups of their files or databases.
So glad you got your site back up!
I should have mentioned that if your webhost totally blows up, you’d lose your full backup based on what I have above. So from time to time, I’ll use FTP to download the whole damn complete backup to my machine. Not a bad idea to add!
Thank you, Johnny!
Outstanding post, Johnny. Thanks for taking the time to share all this. I found it VERY helpful, especially since I am in the middle of cleaning up a malware attack and attempting to prevent future ones from happening again. I’ll tell as many people as I can about this post…
That’s a whole lot of plug-ins you have added. It’s going to keep the site safe but there is always a danger that too many plug-ins might slow down the site or break it somewhere.
But personally I think it’s better to be safe than sorry. So maybe some of the plug-ins functionality can be hardcoded into the site itself.
My schedule is to keep doing the database and the website backups once every 15 days to my local machine via FTP and phpMyAdmin and then mail it to myself. Thanks for the WP Optimize tip- it was always at the back of my mind that I needed to delete the drafts but I never got around to doing it.
Another tip concerning log-in: for installs that have been updated to 3.0+ go change the user name from the default Admin to something else. There are plug-ins for that though I prefer going into the DB through my web-panel and changing it from there.
For the post revision problem, you can just add this line in your wp-config.php:
define(‘WP_POST_REVISIONS’, 3);
The number ’3′ there specifies how many revisions it will keep.
If you don’t want to edit the file, or if you want more control, there’s the Revision Control plugin. Sets a global value for how many to keep, but you can change it on a per-post basis.
By the way, I know why you were so calm. You know that there are two kinds of bloggers: Those whose sites have blown up, and those whose sites are going to blow up. Now you’re in the safe group.
I have another option for you, for website backups: It’s a wordpress plugin called Automatic WordPress Backup, and what it does is back you up to an Amazon s3 storage account. There is some cost associated with getting the Amazon account, but it’s really minimal (google Amazon s3, and there’s a thing on that page to let you calculate your estimated usage).
The snazzy thing about this plugin is it backs up your ENTIRE site, not just the database (though it backs that up too), and if you need to restore you can do it from your website. You control all the options form your wordpress dashboard. It’s sweet – I’ll be signing up for it as soon as I get my next paycheck
Nicely done!
*goes to install plugins…*
I also use WordPress Backup (by BTE) to back up themes, uploads (all the images I’ve used in posts and such) and plugins
Andy
Johnny,
Thank you for having that disaster and then showing us the tools to prevent it. I had a similar panic with a plugin that did not play nice with Thesis. It shut my entire blog down and after 4 hours of going into my hosting account and deleting the piece manually, I fixed it… but man was that scary.
Now I know I need to get my ass in gear and that my host may not be as back-up worthy as I though they were.
-Joshua Black
The Underdog Millionaire
Thanks for the continuing good tips, everyone. I’m hoping Tim Gary from Mindcue will hop in here too, because he has a checklist that contains some of my stuff but a lot more that he’s giving away.
The “Admin” idea was on there too… good one.
Wow, what a panic that must have been. Glad you got it all sorted out and shared your experience here.
Couple more ideas to add
- I think some of the stuff in WP Secure is no longer relevant. The plugins folder now has an index.php by default, and hiding the version is kind of useless. Hackers just test for vulnerabilities that existed in older versions anyway.
- the makers of WordPress have a new service in beta now, called VaultPress http://vaultpress.com/ If you’re allergic to geekery and your blog is your business then the price may be worth it to you. I think it’s $15/month.
- If you’re trying to reduce plugins then you can limit post revisions by editing your wp-config.php. I imagine you’d probably be able to do some of the other things wp-optimize does there, too.
Johnny, you’re freakin’ me out. About the only thing I understood in your post was the f… words.
What is really scary is I have at least 25 drafts of every post, and I count on you to figure this stuff out …and your site crashed …and I don’t have a clue on how to do backups and … my house doesn’t have that many outlets for all those plug-ins (couldn’t resist) and what do I do now???????????
ps. If you put this information into a 1-2-3 pdf document, I bet it would make a fantastic ebook that would go viral. ohh, how about a title for Halloween, “How to keep zombies from hacking your website” or “Getting Zombees to protect your website” or ….. “Nightmare at Johnny B. Truant”
I’m getting some garlic to put on my monitor right now.
Thanks for sharing all this info, Johnny.
My sites (including my e-commerce site, http://ketubahworks.com, which is my bread and butter income!) were attacked by malware last December and shut down for a MONTH. Total nightmare. I’ve been doing regular backups myself since then, but have been looking for a better system.
I really appreciate all the info here, both your post, and the comments! Looking into valutpress and Automatic WordPress Backup…
Incredibly helpful post.
Thanks, Johnny.
Man…you guys are freakin’ me out! Well, at least now I have something to keep me occupied for this holiday weekend. No, really…I didn’t have anything else planned.
All kidding aside, thanks for the tips…I really will be spending the weekend implementing!
Jerry
Thanks for this Johnny! I will be showing it to my techie husband. Just a week ago all my posts mysteriously disappeared for a few hours and then came back. I had a backup but it was pretty old. I didn’t know about the post revision problem either, and I am a BIG reviser.
Ok, so I am not technologically savvy AT ALL. All this mumbo jumbo you just listed is way over my head. You got a dumbass version for dummies? Or a suggestion on who to call, perhaps a qualified computer fairy that will come set all this up? Thanks Johnny!
Listened to your call through Jon Morrow’s apprenticeship program. Very informative, and helpful!
xo
amanda
That’s why I run my websites on a RAID 1 machine,(finally) do daily backups of the databases and weekly of everything important.
Also, as Tzaddi suggested above, with a big blog like yours, I’d consider http://www.vaultpress.com
Anyway, not fun loosing data, happened a couple times for me, now I have (usually) multiple backups of everything.
OK, you rock (but you knew that). I didn’t know about WP-Optimize and after installing it I was able to delete 904 post revisions. Which is interesting since I have (ahem) 19 posts on my site.
Thanks, you!
Nanci
Hey, if one person gets their ass saved by this, that’d be fantastic. Great tips on VaultPress… I’ll have to check that out, thanks!
Okay, Johnny you’ve got us staying awake at night. Each time I get a new email update on this post I start to itch.
I need a one, two , three on what to do. Do I go with VaultPress, or ….???? What does that even mean????
Please hurry… arm is starting to bleed….
Johnny that crap was terrible..I’m on my game plan now.
“TrafficColeman “Signing Off”
Well dang if I didn’t get to reading through the comments after having read the article immediately after it was posted!
Thanks Johnny for mentioning me earlier in the comments!
It’s probably a little late to be chiming in on a 2 week old post, but here goes… I just did a quick audio interview with Johnny about this stuff. It’s only 30 minutes, but I still need to edit out the first 2 minutes of setup time, and probably a bit of my technical ramble towards the end, but it’s in the works.
My main focus for the past 3 weeks, since just before Johnny’s site troubles, has been two-fold:
1) Creating a document that would serve as a compilation of the best WordPress security tips all in one place (with a few references to external details.) Unfortunately this won’t be a paint-by-numbers guide, because that could end up being the size of a book. But it is intended as a valuable reference for the novice and techie alike.
2) Starting a service to handle the bulk of the work in that document for people who are not technically inclined (or who simply want someone else to do it for them). This is available *now* as a bit of a early/test-launch through my site..
Johnny’s pushed me to get this out the door sooner rather than later, and my feet still drag through the coals known as “perfectionism” (ok, they’re more like “stumps” than feet at this point). There’s a lot more to do, but I’ll have an initial document ready by sometime on Friday or sooner. It’ll be free, and available on my http://www.mindcue.com site.
Thanks everyone for the extra tips, I’ve tried to include them where appropriate, and will welcome suggestions once it’s finally ready.
I wanted to also point out that the WordPress site itself has a *ton* of valuable information on security, configuration, etc.. For security, this page on “hardening” WordPress is excellent: http://codex.wordpress.org/Hardening_WordPress