Gather ’round children, and I’ll tell you a tale of what happened to a Linux box when its sole filesystem was remounted read-only due to disk errors. This coincided with the backup server being taken offline with an errant circuit breaker.
I first became aware of something rotten in the state of Webserver when the sites hosted on it became messes of PHP errors in place of content. I could ssh in, but after entering my password I was greeted with:
-bash: /etc/profile: Input/output error
-bash: /home/steve/.profile: Input/output error
-bash-3.2$
That was a scary greeting.
lighttpd and ssh continued running, but PHP died, commands other than Bash builtins refused to run, and Bash profiles failed to load. I don’t know if some of this is due to damage or the partition being read-only. I’m pretty sure that commands expect /tmp, /var/run, and /var/lock to be writable. I now have those mounted as tmpfs as per the instructions on the Arch wiki here. The warning about lighttpd seems to not apply in my case. cURL seemed to run at first but died when I tried to do anything with it. I had hoped to POST files over, as scp, ftp, and sftp would not run. su still worked. Tunneling worked too, so I was able to still access other machines behind the firewall through the server even though I couldn’t run ssh from the machine itself. I ended up using cat to copy over text files. For binary files I had to get a great deal more creative. The only way I could interact with the server was over ssh; the server was an hour away and even if I did have physical access, mount refused to run (unable to write to /etc/mtab?) and I was afraid that the files I could access might be only buffered in memory and that rebooting into a LiveCD/USB would lose them. My options were limited. I had to use only Bash builtins to pull binary files off the server in text form. I modified a version of this hexdump script to pull files over ssh using | tee file.log to avoid having to copy-paste. tee takes output from stdout and redirects it to stdout and a file given as an argument. Here’s the script:
exec 3<"$1"
while read -s -u 3 -d '' -r -n 1 char
do
printf "%02x" "'$char"
done
I lacked a text editor and couldn't write to anything on the root filesystem. I found a tmpfs mount point (I used /lib/init/rw but /dev/shm would also work.) and stored the file by echoing the script line-by-line. In retrospect, I could have used \n and the -e (interpret backslash escapes) option to do it in one line: echo -e "exec 3<\"\$1\"\nwhile read -s -u 3 -d '' -r -n 1 char\ndo\nprintf \"%02x\" \"'\$char\"\ndone" > scriptfile. I ran it with bash scriptfile target_file.
All of this effort, though fun, ended up being unneeded as I had forgotten about my set-and-forget backups. Hooray rdiff-backup!
I ran mysqldump nightly and let rdiff-backup handle any differences. I restored it on the new machine with source mysql_dump.sql on a mysqladmin prompt, but as it contained users and privileges things got messy as the root and debian-sys-maint accounts were partially overwritten. I used mysqladmin to sort out the root password confusion and phpmyadmin to replace the debian-sys-maint password with the one found (in plaintext?!) in /etc/mysql/debian.cnf.
It was a fun puzzle even though it was ill-timed.
The Wolverine Soft 48-hour game competition revealed to me just how difficult physics engines are to make. I spent two days coding and recoding collision resolution only to get different sets of bizarre, game-breaking glitches. At least collision detection was easy because everything was a circle. It was fun and I'd like to do it again. Maybe I should become familiar with a physics library such as Bullet and ask for it to be approved for use in the competition. The guideline is unless it's an approved library, all code and assets (with exceptions for music and sound effects) must be created primarily on-site within the 48-hours. Next time I'll have to plan to do homework in advance. Ignoring homework for a weekend is inadvisable.
I am currently taking 17 credits, and the time management is very difficult, though has not yet proven to be entirely impossible. I'm considering taking classes at LCC this summer to lighten the load during the next school year. I applied to Camp CAEN to be a counselor, but they emailed back saying camp was ending due to the director retiring. The odd part is their website, as of this writing, has no mention of it that I can find. I'll have to see if I can get an internship over the summer.