i think our rackspace servers got hacked or something the other day, rackspace never really figured it out either. a quick summary of what happened:
- tried deploying via capistrano, noticed a weird error "/bin/bash: Exec format error"
- tried to ssh in, even as root, and that wasn't working (some random errors and then getting disconnected)
- tried to login via rackspace console, had some issues there (this is when my blood pressure noticeably increased)
- rackspace did a hard password reset, console worked again -- went into rescue mode and mounted sda1 to get back into our server
- databases were nowhere to be found (and i think the site cached everything, so i didn't notice the database was missing to begin with), luckily my co-founder had a 3 day recent backup so my blood pressure plateaued there. there were random files and users added, so at this point we think someone managed to get into the server.
- found the binary ibdata left in /var/lib/mysql and downloaded that, luckily it was untouched so we lost no data (rackspace only makes server instance backups of 2gb servers, we use 4gb on everything)
- transferred everything to another staging server, and realized the iphone v1.1 was pointing to the original production server ip, so we had to rebuild the production server from scratch.
- had lots of permission problems when we thought it was a much more serious problem, wasted time looking into mysql sockets when all we needed was to fix the mysql privs and re-configure a few things in apache's httpd.conf
- finally got everything working, and still wondering WTF just happened just before going back to sleep
all of this happened from about 3am to noon, and luckily it looks as though nothing ever happened unless you tried using the app early yesterday am ;)