Software – Steve Dougherty's Blog

Fun with Stable Diffusion

After Ars Technica ran this article on a Stable Diffusion mobile app, it seemed like a good time to give Stable Diffusion another shot. I had previously given up figuring out how to set up the desktop version. It’s polished! It includes example prompts to demonstrate what sorts of incantations make up a good prompt, and with that and a moderate wait for it to generate I had this:

8k resolution, beautiful, cozy, inviting, bloomcore, decopunk, opulent, hobbit-house, luxurious, enchanted library in giverny flower garden, lily pond, detailed painting, romanticism, warm colors, digital illustration, polished, psychadelic, matte painting trending on artstation

That was enough to hook me, so when I noticed that article also linked to stable-diffusion-webui, it became a great time to see what the same underlying image generation can do when it can draw ~300W continuously on my desktop instead of being limited to a phone’s resources. I quickly (and somewhat inadvertently) was able to generate a cat fractal:

cute ((cat)) with a bow, studio photo, soft lighting, 4k

This was my introduction to the sorts of artifacts I could expect to do battle with. Then I had an idea of how to use its ability to modify existing photos. After some finagling, I had a prompt ready, set it to replace the view outside the window, and left it running. When set to a very high level of detail and output resolution, it generated 92 images over about 6 hours. Of those, 22 seemed pretty good. Here is a comparison of my 2 favorites:

And a slightly different prompt where I had selected part of the window other than the glass:

a colorful photo of the circular wooden door to a hobbit hole in the middle of a forest with trees and (((bushes))), by Ismail Inceoglu, ((((shadows)))), ((((high contrast)))), dynamic shading, ((hdr)), detailed vegetation, digital painting, digital drawing, detailed painting, a detailed digital painting, gothic art, featured on deviantart

There were three sorts of artifacts or undesirable outputs that were frequent:

It focused too much on the “circular” part describing a hobbit hole door.
It added unsettling Hobbit-cryptids that brought to mind Loab.
The way I used the built-in inpainting to replace the outside meant both that the image was awkwardly separated into 3 largely independent areas, and those images often tried to merge with the area around the window. This makes total sense for its actual use case of editing an image, but I’d tried to configure it to ignore the existing image without success. In retrospect, I could have used the mask I made to manually put the images behind the window with conventional editing software.

It’s wild to use image generation to generate this without being able to even imagine painting it myself. It’s definitely not the system doing all the work – you have to come up with the right kind of prompt, adjust generation parameters, and curate the results. But it’s a lot cheaper and easier than art school and practice, which I feel uncomfortable about because this model was trained in part and without permission on art from those who did go to art school.

Adventures in VeraCrypt Bruteforcing

I wrote a script – one that facilitates bruteforcing VeraCrypt passphrases – with the intent that the user probably remembers most of the passphrase. It’s available here.

It didn’t end up working for me, outside of test cases to make sure the script itself worked, but hopefully it’ll prove useful for others. One day I realized I was in the process of forgetting my VeraCrypt FDE passphrase, even though I’d been using it for months. It was odd – as though the harder I tried to hold onto it the more damaged my recollection of it became. Even though I still think I remember it, the passphrase I remember is wrong. I used the script to check similar passphrases, but they too were wrong. Going forward, it seems wise to not rely on muscle memory, and instead repeat to oneself the actual content of one’s passphrases from time to time. Go over mnemonic devices. Fight against it slipping away.

Network Interfaces

If I set the scene with remotely configuring network interfaces at 3 AM you’ll probably already guess it was a bad idea. In retrospect I would have been better served by my reservations that I should do it when I was better rested and had given it some thought. As it was, I was configuring a bridge for a KVM according to the excellent Debian Handbook.

/etc/init.d/networking reloading the network interfaces didn’t bring the new ones up, so I tried restarting them. One of the last messages I saw was "Running /etc/init.d/networking restart is deprecated because it may not enable again some interfaces". The last message I saw was a DHCP release. Whoops. What I probably should have done is run ifup on the new interfaces directly instead of risking bringing down the interface I was connected over.

Freenet

Freenet is a medium for censorship-resistant communication. It allows people to communicate by publishing and retrieving data securely and anonymously. When someone runs Freenet on their computer it is called a node. Each node connects with a limited number of other nodes. When two nodes are connected they are one another’s peers. Every node communicates with the rest of the network solely through its peers.

Each node has some amount of storage reserved for a datastore. A datastore is a shared space in which each node keeps data. Freenet can be thought of as a distributed, encrypted storage device. It allows inserting data into and fetching data from the network-wide datastore made up of all the individual nodes’ datastores.

In order to do this, Freenet must be able to determine which nodes to store data on, and later be able to find that data again. The process of finding a piece of data, or a place to store it, is called routing.

This is where math comes in. In graph theory, there is a type of network called a small-world network. A small-world network contains relatively short routes between any two nodes. This is good, because longer routes are slower and less reliable. Some types of small-world networks are especially interesting because they allow finding short routes with only locally available information. This is essential for Freenet because its nodes must perform routing with only locally available information through their limited number of peers.

Here’s the concept: all nodes have a network location, which is unrelated to geographical location. An inherent characteristic of every request sent into the network is that it has an ideal location to be routed to. Nodes route requests by giving them to their peer whose location is closest to that ideal location. In order for this to be effective, the network must have a specific characteristic: it must have a good distribution of “link lengths,” which are differences between the locations of connected nodes.

Locations can be thought of as wrapped around a circle: 0 at one point, approaching 1 as it goes around, then wrapping back to 0. 0.3 is 0.2 away from 0.5, and 0.1 is 0.2 away from 0.9. This distance between peers’ locations is called the connection’s link length. On average, nodes must have many connections with shorter link lengths, and a few connections with longer link lengths. One can think of this as being able to quickly make large leaps on the location circle and also make small adjustments.

Depending on the network security level, a node can run in “opennet” mode. It will connect with nodes run by untrusted people the node’s operator does not know, called strangers. This is in contrast to the preferred mode of operation, called “darknet,” in which the node only connects to people the node operator knows in person, at least enough to be pretty sure they aren’t a secret agent or incredibly bad at securing their computer.

Okay, so this is all fine and good, but so what – what can it do? The most straightforward use is to insert a file and share the key with others so that they can retrieve it. The problem becomes how to tell other people. If all Freenet can do is act as a file storage device in which one can only retrieve files one already knows about, Freenet can’t do much.

While this is a limitation, many people have still built useful applications on top of Freenet. They tend to use a web of trust to discover files inserted by identities people create, and put together those files to present a responsive user experience locally. They assemble something like the database that would usually be on a centralized server locally from fetched files.

A bunch of plugins and external applications allow interactive communication over Freenet. There’s real-time chat, email, a cross between Twitter and a Facebook wall, and other applications which provide completely decentralized forum systems.

I collect and analyze data about Freenet to provide estimates of things like the size of the network and help better understand the network’s behaviour. Feel free to take a look! The links in the footer starting with “USK@” are links to things in Freenet and won’t work here.

URXVT

URXVT is a lightweight terminal emulator, (with an equally excellent page on the Arch Wiki) but I didn’t like the default color set, especially when using WeeChat. Here is why:

After putting the scrollbar on the right, using a larger XFT font, and using the same color set as Gnome Terminal’s “Tango” theme, things look much nicer:

Here’s the .Xresources to do so:

URxvt.background: #300a24
URxvt.foreground: #FFFFFF

URxvt.font: xft:DejaVu Sans Mono:size=12
URxvt.iconFile: /usr/share/icons/Humanity/apps/48/terminal.svg
URxvt.scrollBar_right: true

! gnome-terminal Tango theme
! black
URxvt.color0 : #2E2E34343636
URxvt.color8 : #555557575353
! red
URxvt.color1 : #CCCC00000000
URxvt.color9 : #EFEF29292929
! green
URxvt.color2 : #4E4E9A9A0606
URxvt.color10 : #8A8AE2E23434
! yellow
URxvt.color3 : #C4C4A0A00000
URxvt.color11 : #FCFCE9E94F4F
! blue
URxvt.color4 : #34346565A4A4
URxvt.color12 : #72729F9FCFCF
! magenta
URxvt.color5 : #757550507B7B
URxvt.color13 : #ADAD7F7FA8A8
! cyan
URxvt.color6 : #060698209A9A
URxvt.color14 : #3434E2E2E2E2
! white
URxvt.color7 : #D3D3D7D7CFCF
URxvt.color15 : #EEEEEEEEECEC

Copy and paste with URXVT took a bit of getting used to. Without additional setup, it requires using the Xorg paste buffer: selecting text copies; middle (scroll wheel) click pastes.

Windows Reinstall, IDEA vs Eclipse, Ubuntu Natty

Soon after installing SP 1, it looked like my Windows 7 installation was hanging on boot. A few impatient hard reboots later I had a consistent BSOD. (0x0000007B (0xFFFFF880009A9928, 0xFFFFFFFFC0000034, 0x00000000000000000 0x0000000000000000)) Startup repair didn’t help and neither did any utilities I could think of, whether available from startup repair or not. Safe mode wouldn’t boot and would hang on classpnp.sys. I tried fixmbr, and then realized that restoring GRUB when running with an encrypted root partition is more complicated. I should have just backed up the bootloader to an image, but I didn’t. Luckily, someone more knowledgeable than I had the same problem (though not for as foolish a reason) and I found their guide. sfc /scannow didn’t fix it. Some forums suggested deleting classpnp.sys, which didn’t help; it just started rebooting without displaying a message. Copying the classpnp.sys from a machine also running Home Premium 64-bit just restored it to the BSOD. I ended up reinstalling Windows 7. I moved my files to my Linux backup partition and back while booted into Linux, and in retrospect given how much fragmentation it caused I should have installed over it and just copied from the Windows.old directory on the new installation. I assume that the Linux NTFS drivers are to blame for the high levels of fragmentation, and that Windows would handle it far better.

While Eclipse works, it is rather sluggish and the UI is clunky. Thanks to ##java on Freenode, I’m now using IDEA. It’s somewhat different, and it doesn’t have the ANT error-checking that Eclipse has, but I prefer it. The interface is less cluttered and it feels easier to devote more of my screen to the actual source code. I think the reason is as simple that the buttons to hide windows are larger, toggle, and don’t move. The ANT problem I ran into took a while to figure out. IDEA was nice enough to offer to pull code down from GitHub, but although the resulting code compiled, the .jars didn’t run correctly. Being reduced to diffing the directories, I discovered that IDEA didn’t catch the ANT problems that Eclipse did, and upon fixing those it worked.

I’ve upgraded to Ubuntu Natty, and I don’t like the new interface. I don’t think it works very well. For instance, it’s very easy to remove an item from the dock, but dragging an item to change the order takes too long. This is probably due to the necessity of differentiating between scrolling the contents and dragging a single item. It’s not intuitive. Similarly, adding an icon to the bar is as easy as dragging, but it only stays in the bar if the thing that was dragged there remains. I’m glad GDM presents the Ubuntu Classic option. My desktop’s upgrade process to Natty was very much unlike the three other machines I’ve upgraded uneventfully. Not only did GRUB install unsuccessfully, which resulted in symbol not found : 'grub_env_export', (following the above guide again fixed it) but something about my motherboard led to a very long hang on boot after NET: Registered protocol family 1. At least there are bugs filed for the former and the latter, but I’m glad I wasn’t also hit by this one. Perhaps this will lead to me reading the release notes before an upgrade. However, I don’t see mention of these issues in the release notes, which worries me that it won’t be enough.

Gitolite, Lighttpd, and GitPHP

I had some trouble setting up GitPHP running under Lighttpd when using Gitolite. GitPHP is a PHP clone of gitweb, which allows web-browser-based repository viewing, and Gitolite is a very nice Git permissions manager. I needed the webserver to have read access to the repos, so I set the group to www-data, but as Gitolite was managing them, each commit would reset the permissions. I couldn’t get it working by adding www-data to the git group as suggested and would make sense, which I think is a lighttpd issue. The group sticky bit solved this problem by stopping the group owner from changing. I didn’t want the Gitolite configuration repos displayed on GitPHP too, so I pointed GitPHP at a different directory than the one the repos were actually in, and filled that with symbolic links to the repos it should display.

Progress

Faced with a persistent inability to complete homework to a satisfactory degree, my academic advisor and I decided to drop my physics lecture, which had yielded encouraging increases in available time. I’ve also discovered GitPHP, which works as a PHP equivalent to gitweb and works very nicely. I set up a private git repo (accessible through the user git with everyone’s public keys in authorized_keys for my Engineering 100 class. We’ll use it to coordinate parallel development of an E100-compliant processor and our project, which at this point looks like it’s going to be creating robots which interact with one another through sound.

Learning Through Disaster

Gather ’round children, and I’ll tell you a tale of what happened to a Linux box when its sole filesystem was remounted read-only due to disk errors. This coincided with the backup server being taken offline with an errant circuit breaker.

I first became aware of something rotten in the state of Webserver when the sites hosted on it became messes of PHP errors in place of content. I could ssh in, but after entering my password I was greeted with:

-bash: /etc/profile: Input/output error -bash: /home/steve/.profile: Input/output error -bash-3.2$

That was a scary greeting.

lighttpd and ssh continued running, but PHP died, commands other than Bash builtins refused to run, and Bash profiles failed to load. I don’t know if some of this is due to damage or the partition being read-only. I’m pretty sure that commands expect /tmp, /var/run, and /var/lock to be writable. I now have those mounted as tmpfs as per the instructions on the Arch wiki here. The warning about lighttpd seems to not apply in my case. cURL seemed to run at first but died when I tried to do anything with it. I had hoped to POST files over, as scp, ftp, and sftp would not run. su still worked. Tunneling worked too, so I was able to still access other machines behind the firewall through the server even though I couldn’t run ssh from the machine itself. I ended up using cat to copy over text files. For binary files I had to get a great deal more creative. The only way I could interact with the server was over ssh; the server was an hour away and even if I did have physical access, mount refused to run (unable to write to /etc/mtab?) and I was afraid that the files I could access might be only buffered in memory and that rebooting into a LiveCD/USB would lose them. My options were limited. I had to use only Bash builtins to pull binary files off the server in text form. I modified a version of this hexdump script to pull files over ssh using | tee file.log to avoid having to copy-paste. tee takes output from stdout and redirects it to stdout and a file given as an argument. Here’s the script:

exec 3<"$1" while read -s -u 3 -d '' -r -n 1 char do printf "%02x" "'$char" done

I lacked a text editor and couldn't write to anything on the root filesystem. I found a tmpfs mount point (I used /lib/init/rw but /dev/shm would also work.) and stored the file by echoing the script line-by-line. In retrospect, I could have used \n and the -e (interpret backslash escapes) option to do it in one line: echo -e "exec 3<\"\$1\"\nwhile read -s -u 3 -d '' -r -n 1 char\ndo\nprintf \"%02x\" \"'\$char\"\ndone" > scriptfile. I ran it with bash scriptfile target_file.

All of this effort, though fun, ended up being unneeded as I had forgotten about my set-and-forget backups. Hooray rdiff-backup!

I ran mysqldump nightly and let rdiff-backup handle any differences. I restored it on the new machine with source mysql_dump.sql on a mysqladmin prompt, but as it contained users and privileges things got messy as the root and debian-sys-maint accounts were partially overwritten. I used mysqladmin to sort out the root password confusion and phpmyadmin to replace the debian-sys-maint password with the one found (in plaintext?!) in /etc/mysql/debian.cnf.

It was a fun puzzle even though it was ill-timed.

The Wolverine Soft 48-hour game competition revealed to me just how difficult physics engines are to make. I spent two days coding and recoding collision resolution only to get different sets of bizarre, game-breaking glitches. At least collision detection was easy because everything was a circle. It was fun and I'd like to do it again. Maybe I should become familiar with a physics library such as Bullet and ask for it to be approved for use in the competition. The guideline is unless it's an approved library, all code and assets (with exceptions for music and sound effects) must be created primarily on-site within the 48-hours. Next time I'll have to plan to do homework in advance. Ignoring homework for a weekend is inadvisable.

I am currently taking 17 credits, and the time management is very difficult, though has not yet proven to be entirely impossible. I'm considering taking classes at LCC this summer to lighten the load during the next school year. I applied to Camp CAEN to be a counselor, but they emailed back saying camp was ending due to the director retiring. The odd part is their website, as of this writing, has no mention of it that I can find. I'll have to see if I can get an internship over the summer.

ADC

The ADC makes more sense now. It turns out Professor Atkins has been waiting as we figured out that the weirdness we’ve run into is due to tremendous electromagnetic interference. My math GSI was incredibly kind and willing to spend about an hour helping me fix the statistics. I don’t know why the corruption I ran into was occurring, but we did establish that what GSL calls total sum of squares is actually variance. I’ve added a real TSS function, as well as an output of absolute value of residual. Here’s the best graphs we got previously, rendered with the latest graphing routine:

ADC3 with latest graphing.

ADC5 with latest graphing.

ADC6 with latest graphing.

I didn’t want to disrupt the servo guy’s work much, so I moved Gumstix over to the power supply and set it back up. I didn’t use the breadboard to ground the unused ADCs, and put them all in the same alligator clip instead. I thought I would calibrate two more channels so that we’d have a usable input for the gyro reference voltage. I was very surprised with the results:

Behold ADC2!

Behold ADC7!

This makes so much more sense for many reasons. As I pointed out yesterday, there was a consistent, significant distortion under 1v. This is nowhere to be found in the new line. It’s actually a line, and there is only a minuscule difference in counts for the same voltages between graphs. This is acceptable as imperfections in the voltages we fed it as in this respect our power supply is… abstract. This line also goes up to 1024 at 2.5v, which is what it should actually do as it’s the maximum count at the maximum voltage. What I find amazing is how huge the effect of electromagnetic interference is! We got completely different information when using the breadboard, and even its imperfections were consistent! Professor Atkins revealed that she had let us spend hours on this fruitless calibration of electromagnetic interference so that we would thoroughly learn the importance of electromagnetically clean wiring. Lesson learned!