LiThoughts

Really, the picture says it all. I don’t even need to write anything here. Since it’s a blog, however, I’ll add a brief description of the situation. This customer was having squirrely issues with his network connection, both wired and wireless. I found that disabling the wireless card in the bios allowed the wired network to function, leading me to believe that his wireless miniPCI card had failed. After having to remove the keyboard and the entire upper bezel to get to the blasted thing, I can’t replace it with any card other than one the same model as the original.

Thanks for all the unmaintainability.

Due to various problems with Ubuntu Server, I’m migrating our server at the shop to CentOS.  Part of that migration included copying over the chroot environments that we use for PXE booting.  When it comes to copying large amounts of data over the network, especially when the permissions and ownership of that data is crucial, rsync is always my goto tool.  It can use compression to save network bandwidth, uses delta copies for files that are already copied but have changes, and has options to delete files from the destination that don’t exist in the source.  On the whole, rsync is pretty easy to use as well and the man page has EXCELLENT examples.  Here’s a simplified version of what I was using (the real one uses ssh tricks to get rsync to run as sudo on the other end):

rsync -avz 192.16 8.1.14:/nfsrootmav/ /nfsrootmav/

Looks good, right?  What this tells rsync to do is copy recursively and maintain ownership/permissions (-a), use verbose output messages (-v), and use compression during the copy (-z).  I ran this command and all my files copied happily and without complaint.  When I tried to test the copied chroot environment, however, it wasn’t quite right.  When I tried to shut the computer down (booted to the chroot over PXE) it logged out of Gnome instead.  The shutdown command in GDM didn’t work either.

I knew that the only difference between the original and the copy is that the copy…had been copied.  This lead me in the direction of permissions, though I couldn’t understand why they’d be different if rsync was running in archive mode.  I tried running diff on `ls` dumps of the original vs. the copied directories, but I never really got the outputs on the different servers right for the diff to tell me everything I wanted to know.  What I did glean from my diff was that if I looked at the uid’s and gid’s of the files numerically, the original and copy had different ownerships on some files!

What?  How could that be?  Then it occurred to me…I had copied the chroot from the host OS, not from within the chroot.  It turns out that by default rsync copies the user and group owners of a file by name, not uid and gid.  That makes sense, doesn’t it?  If I have a file that’s owned by, say, “messagebus” on one computer and I copy it to another, I want it to be owned by “messagebus” on the other computer too even if the “messagebus” user has a different numeric uid.  It went wrong here because files are stored with a numeric uid/gid, and any time an operation requests or sets the text user/group then they are simply looked up in /etc/passwd or /etc/group.  The files in my chroot had users that matched uids in it’s own /etc/passwd, but rsync was referencing the host OS’s /etc/passwd.

The solution?  rysnc has an option called “–numeric-ids”.  This will cause rsync to use the numeric uid’s and gid’s directly rather than trying to look up the plain text names.  I added that and my files copied properly.  The PXE-booted computer shutdown as it should and all was well with the world.

I actually never would have had this problem if I’d chrooted into my /nfsrootmav directory and rsynced out from there because rsync would have been looking at the right /etc/passwd and /etc/group, but then I never would have learned this great lesson.

A few weeks ago my wife and I went to Ohio to see some of her family.  During the trip she, her mom, and her sister wanted to go to the local Goodwill store to find some clothes.  This wasn’t an incredibly promising trip for me but I do like to browse through the old electronics; you never know what gems you might find.  Lo and behold, they had an old microprocessor trainer kit for less than $10 laying on a shelf in the original—though yellowed—cardboard box.  It made the whole shopping-with-three-girls thing worth it.

I haven’t had a chance to mess with it until today but I very quickly learned from the manual (yes, it still has it!) how to work the internal synthesizer to both play music on the fly and program it in for computer playback.  It was pretty hard to just play it like a piano since the notes weren’t laid out very nicely, so I wanted to come up with a good, short, geeky song or riff to test it out with.  What could be better than “Axel F”?  Perhaps “Still Alive”, but that’s a song for another day I suppose.  That one would be near impossible though…I don’t see any way of programming adding a rest of any kind and “Still Alive” has a lot of them.
EDIT: I did find out how to program rests into a song; I just wasn’t reading far enough ahead. I updated the video to one that includes the rests in Axel-F, and maybe I’ll try Still Alive in the future.

Many thanks to my lovely and talented wife Christina for finding the sheet music online and converting it to solfège for me.  The trainer uses solfège instead of the typical A-G notes.

Because of a very irritating (unsolved) problem I had with Ubuntu Lucid on my home computer I decided to try a different distro for a change. I’ve been all over the distro map since I started toying with Linux in 2003 or so. I started out with Red Hat 9, then went to Slackware (10 at the time), Gentoo, Sabayon, then Ubuntu. I also briefly messed with some RHEL clone or other several years back (not CentOS…I don’t remember what it was..star something). Each had its benefits and its drawbacks but I’ve been on Ubuntu for the longest of any of the distros I’ve used, mainly because of how easy it is. Years ago I laughed at Ubuntu because I thought it was “the n00b distro” but I’ve really learned to respect the power of ease-of-use: if I’m not spending 3 hours getting X to work the way I want, I can be spending that time on more important things.

In my journey with Ubuntu and its lovely works-out-of-the-box(usually)-ness I have learned an important lesson, though. That lesson is that when you make something very easy to use you by nature are likely to sacrifice some stability. Usually that’s not a problem but in my case I sort of hit a productivity barrier since large disk reads would pretty much lock my computer up. By that I knew it was time to break out my rusty skills and go to something one degree less easy and one degree more stable.

I spent a few days deciding what that distro would be and I knew I wanted to try something I’d never used before. My first choice was Debian (the mother of Ubuntu) but there was a bug with the latest version (I don’t remember whether it was testing or stable) that kept it from even booting properly on my machine. My thoughts then drifted to Fedora. I’d used Red Hat 9 years ago as I mentioned previously and was eventually frustrated by the fact that Red Hat felt lead to do everything different from the way everyone else in the Linux community did it; these days though there’s so much documentation available online that the differences aren’t so hampering. Besides that, I’ve been getting the feeling that over time the differences have smoothed out a bit.

Still, there was my desire to try something totally new. That nagging urge is what eventually steered me to openSUSE. I knew it would be a good distro to learn since it’s one used a lot in industry; plus since it’s RPM based I knew I shouldn’t have an issue finding pre-built packages for any widely used application. Having made my decision, I downloaded the 11.4 ISO and loaded it up.

The installer was a nice, pretty, graphical one…not quite as simple as the Ubuntu installer but definitely manageable for someone who’d be using Linux for a while. This is one of the things I expected in the trade from easy to stable. The first thing about it that really impressed me was the fact that I didn’t have to reboot after installing. I was able to dive right in and used the installed version…I had never seen that before.

The layout of the desktop (Gnome, btw) took a little adjusting to get it the way I wanted it but that’s normal for me. I really liked the default theme and I’m still using it. It’s got dark grey panels and dark green titlebars but light window backgrounds and mostly light controls. It’s a pretty good mix of light and dark; I don’t like all dark themes. The default background is a rotating set of backgrounds that vary from very light green to darker green, seemingly following the time of day. All this is very refreshing to me after wanting to gouge my eyes out with my titanium spork every time I load a new version of Ubuntu and have to look at the default theme.

The fonts didn’t look very nice at first and a quick web search revealed that this was because openSUSE opted to disable sub-pixel hinting by default to avoid being victim to a M$ patent suit relating to ClearType. Everything is explained in this very nice article describing how to install the version of freetype with sub-pixel hinting enabled.

I also needed an article to learn how to install the proprietary NVidia drivers, and found it pretty easily as well. The nice thing about that article was that it included a special “one-click” install link that took me right to the software installation program and loaded the right package for me; I think I’ve seen “deb://” style links around the net that do that for Debian-based distros but I can’t find any right now.

Software installation is a good bit more complex on openSUSE; several times it would prompt me during updates or new installations asking question I didn’t exactly know how to answer. I can’t think of specific examples right now but it was usually a situation where some potential package conflict could occur and it wanted to know what I wanted to do. Usually there was an option to let it take care of it automatically and I just chose that. I found also that I had to add several repos to openSUSE’s software sources before I could get certain packages, some of them rather common. The biggest and most important of these is the packman repo which contains a lot of community builds for different software packages. Besides that, the videolan repository was also good for nicer versions of ffmpeg, mplayer, etc. Both of these are called community repositories; this is basically openSUSE’s version of a PPA. Since they are officially recognized by openSUSE, adding them is very easy:

1. Go to Yast → Software → Software Repositories
2. Click “Add” (near the bottom)
3. Choose “Community Repositories” and click Next
4. Now simply select the repos you want to add and click OK
5. Accept any offers to import GPG/signing keys

I still find myself having to look things up every now and again but familiarity is really starting to set in. I do miss the lightning-fast boot times made possible by ureadahead but besides that I don’t really look back too much. I still use Ubuntu on my work computer since it runs fine on it, so I’m not totally abandoning Ubuntu; I’ve just added another wrench to my toolbelt.

When I first started the Winning Side Ministries I had one code style…now I prefer another. It happens, right? It shouldn’t be that big of a deal; there are plenty of free tools available that will iterate through source code files and alter the formatting for you. What I’ve found in the last few days though is that not many of them really work all that well.

Our website is written in PHP so naturally I began looking for a solution that was tailored specifically to that language. There are only a few of them and I really couldn’t get any of them working in a satisfactory way. At that point I began looking at more generic solutions like GNU indent which is actually targeted at C but works OK with other languages that use C-style syntax. After a bit of toying around I got indent to make the changes I wanted it to make but it was also making other changes that I didn’t want it to make.

That’s when I realized…there were really only 2 formatting styles I wanted to change. Why not just fix it with some sort of regex substitution program? Enter sed (StreamEDitor). It’s really a nifty little program that does one thing really well (per the Unix philosophy) and that is take the text it’s given and perform the requested modifications. Now it’s really meant as a line based editor and some of my changes required multi-line substitution but it wasn’t really that hard to work around after googling for a bit.

Here’s the code style I wanted to convert from:
And here’s the style I wanted to convert to:

So we’re going from having 4 blank lines between functions to only 1, and going from having opening braces on a separate line to having them on the same line preceded by a space. Here’s the sed script I came up with:

That first line…I copied and pasted that from this answer to this stackoverflow question. I don’t know sed well enough to really explain it, but there’s some explanation in the answer comments there so check it out if you want to know more.  The basic idea is that sed can’t work with multiple newlines at once without a workaround since it expects to act upon every line individually.  The tr solution also proposed on that same question won’t work for me since tr works character by character and doesn’t support full regex’s as a result.

I did run into the problem described in the script’s comment, but only once. It wasn’t worth complicating the script to fix that though it may not be difficult. I don’t know…I didn’t bother.

To run the script you just call sed like so:

sed -i -f format_code.sed *.php

And you’re done! The “-i” makes it edit the files in place so you don’t have to do any kind of weird redirection or write a script around it. It just works.

It has been a long time since I’ve written anything, I know. I’ve been so busy recently with several different things; I’ve wanted to write about all of it but I just haven’t had the time. Maybe I’ll cover a little bit more of it soon but right now I wanted to talk about an interesting problem we had with the new server at the computer shop.

On the hardware side it’s a Dell PowerEdge 1950 with a quad-core Xeon 2.33GHz CPU, 4 GB RAM, and a mirrored 500 GB RAID array. It’s not what you’d call cutting edge in the data-center but it’s a far cry from the dorky little single-core Pentium 4 thing we had been using.

Software-wise it’s running Ubuntu 10.04 Server which is taking care of our file-server needs along with housing our squid caching proxy and PXE boot environments. Since this server is equipped with dual gigabit LAN we bit the bullet and bought a couple of gigabit switches as well, which has really made a difference in Quickbooks’ performance alone.

For various reasons (not least of which is Quickbooks) all of our workstations in the shop but mine are running Windows which means file sharing must be done over SMB using Samba. I set everything up as usual with mostly defaults in my smb.conf. The shares were added as usershares with full write access by everyone for the sake of simplicity; that shouldn’t be a problem since iptables is blocking all traffic originating from the WAN port.

Since everyone was so excited about the gigabit stuff, my boss fired up a couple of file transfers and began listening to mp3 files over SMB just to see what kind of performance we could get. The funny thing was that every so often the music would just hang for no reason for 20-30 seconds at a time. During the hang his computer wouldn’t be able to browse the file shares at all and all file transfers would hang as well. I checked the system logs on his computer and dmesg on the server and neither mentioned anything related to the problem. Finally I found the Samba session logs (these are in /var/log/samba on Ubuntu) which are unique per client and saw lines in it like this:

"Unable to connect to CUPS server localhost:631 - Connection timed out"

Timed out? Yeah, that sounds like it. A bit more investigation led me to realize the timing of the pauses and the errors seemed to correlate. Searching for that error on the Internet was very difficult, though; it seemed like everybody with a Samba client log posted on the web had that error and nearly all the posts I found we’re asking about unrelated issues.

Disgusted with searching for a bit, I tried the obvious solution of commenting out all printer related lines in smb.conf. No luck…apparently printing support is enabled by default even if it isn’t configured. Back to searching.

Finally I found one guy who was complaining about a big lag during log-in on password protected shares and was getting that same CUPS error. He figured out how to get Samba to stop trying to connect to CUPS using these configuration lines. He admits that they might not all be needed, but it doesn’t hurt my feelings to smash a bug a little harder than necessary:

load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes

Around the same time I also found that many people were saying that this particular configuration tweak was supposed to give better throughput when serving Windows clients:

socket options = TCP_NODELAY

I shamelessly applied both fixes at once.  At the time I just really needed it to work (we’d already moved our Quickbooks company file) and didn’t care too much which one fixed it so long as it was fixed.  And it was.

No one really mentioned the socket options tweak affecting any sort of hangs or delays so I’m certain that disabling print services in Samba was the real kicker.  Apparently there are very few people who set up Samba on a server without CUPS, but we have no need for a print server like that.  We don’t have a big copier or anything; our only printer is a small black and white laser at the front desk which is 30 feet and several walls away from our server.

So we have another problem solved and I feel pretty good about it. It’s nice to find the solution to a tough problem and I thank the good Lord for helping me.

Continuing on with the implementation of a custom CMS for the ministry website, I’ve been doing a lot more type checking than I used to do with PHP.  That’s actually one of the things that has always bothered me about languages like PHP; as a programmer I make a lot of mistakes and it’s a real boon if my programming language supports double checking me without any extra intervention, but PHP doesn’t offer much in the way of enforcing variable types.  Starting with PHP 5.1 we’ve had “type hinting”, which means I can force a parameter to a function/method to be an object of a certain type or an array.  Well, this doesn’t really help if my function needs a string, an int, or an array containing strictly typed elements.

What I’ve been doing to get around this is using type hinting wherever I can, then doing extra manual checks on the types of other parameters via the is_* family of functions.  If I get an incorrect type, I throw an InvalidArgumentException.  I’ve been using exceptions extensively in this project because of the multi-layered nature of it.  If I relied on return values alone for monitoring errors there would be layer after layer of if-statements and checks passing data back up the call stack, all the while making it very difficult for my functions and methods to return useful data to their callers without the use of output parameters.

One of the most recent things I’d done was to set up the page loading code to generalize error handling.  I am currently using a layout similar to this (not very pretty, I know—I’m working on it):

There’s actually a good bit more to it than that, but I’ve cut things out to make it less distracting.  Basically I get my page data, feed it to the template requested, and grab the parsed template output using $smarty->fetch().  If there is an exception thrown I check to see if the real exception message is safe to show the user or whether I need to use a general one, then I fetch the error page template to show the user instead.  Sounds OK, right?

When I tried testing this code be requesting an invalid photo album on the photo gallery page, I got two instances of the page header then the error message and page footer.  I thought I’d goofed up somewhere and included something wrong.  After more testing, I found that if the exception was thrown before the call to $smarty->fetch() (for instance if contentManger couldn’t find the page I fed it) there was no problem.  That got me to thinking about output buffering.

I knew smarty used output buffering in the implementation of fetch(), but I didn’t initially considering the implications that brought with it.  I haven’t looked at the code so I don’t know exactly how it’s written but the gist is something like this (again, my interpretation and that oversimplified):

See what’s happening? In the parseTheTemplate() section of the code, however smarty actually calls it, my template plugins get run as part of parsing the template. If one of my plugins throws an exception it gets passed back up the call stack until it gets caught by a catch block. What gets skipped? Yep, ob_get_clean(). That means any output that’s already been printed is still in the output buffer.  On the subsequent call to $smarty->fetch() it gets returned in with the rest of the output.  In order to fix it, all I had to do was add a call to ob_clean() like this:

No more problems.  Smarty could have handled this on it’s own by manually clearing the output buffer at the beginning of fetch() before anything was parsed but I’m sure there is a reason why it isn’t.  Perhaps there are people putting things in the output buffer before calling fetch() on purpose.

Recently at the shop we’ve been needing a few new monitors.  Instead of just buying new ones, we went to our local scrap guy to see if he had any LCD monitors that looked repairable; mostly we just fix the ones with bad capacitors in the power supply, which turns out to be a pretty high percentage.

We fixed 2 Dell monitors right off and I’m using one of them right now (score!).  The problem came when we tried to fix a really nice (well, it looks nice) Gateway 19″ wide screen.  It had 9 blown caps so I ordered those and soldered them in.  No problem.  It fired right up and looked beautiful…for an hour.

It turns out this monitor has a built-in anti-theft system.  Why you need an anti-theft system in a monitor, I have no idea.  It looked like we were going to be able to get it unlocked for us by Gateway support but they proved to be even worse at their jobs than you normally expect from outsourced tech support.

The problem stems from the time-frame in which the unlocking must occur.  The monitor will generate a 3 digit “challenge code” that Gateway uses to generated the unlock code, but the challenge code times out after 1 hour and a new one must be generated.  The guy at Gateway technical would take our challenge code and come back 12-24 hours later with an unlock code for us.  After 2 or 3 times of trying to explain to this guy that he had to respond within an hour for it to work I tried opening another ticket.  That technician basically said “go away”.

So now I have $19 and an hour of labor in a monitor that works, but I can’t use.

Moral of the story?  Buy a Dell.

I’d noticed recently that the ministry website (http://www.winningsideministries.org) had gotten woefully out of date in regard to it’s text content.  The news column on the main page is over a year old and the “about us” page doesn’t reflect the current state of the ministry at all.  I started to update the content but I got all hung up in figuring out whether the site was currently running the testing or stable version from svn, then making sure to upload the right file version, trying to remember to update the right repo…it was a mess.  I’d been using svn for the ministry website because I’d started working on it before I new git; more than that, I couldn’t store it on my free github account due to the fact that the database passwords would be there for the world to see.  The solution to that bit of the problem was simple.  I just set up my own git repo on the same server that hosts my svn repos and moved it there.  Since I was having so much trouble figuring out what version of what files were where, I just downloaded the current working version of the website and based the website off that.  It’s possible that I’ve lost a little bit of work that way, but as badly dissheveled as the codebase was I wasn’t sure that it mattered a whole lot in the long run.

After that I asked myself, “Why am I sitting here editing source code files to update the content of the website?”  Sure, building it that way got it up quick without me having to learn a lot of new stuff before-hand, but it made the maintenance a nightmare.  At that point I realized that for there to be any sort of efficiency in the task of keeping the content up to date I was going to have to completely (or nearly so) separate the content from the code, aka, use a CMS.

There are lots of good CMS’s out there, but most of them are really heavy from what I’ve seen.  A lot of them (like the WordPress CMS that powers this blog) are indeed very blog oriented, making it a bit more complex to have widely varying page setups.  I realize that if I’d spend the time requried I could very well make any one of the CMS’s available do just as good a job of handling our static content pages as it would our dynamic ones, but I wasn’t sure that’s what I wanted to do.  We have a very simple website with (currently) 9 working pages.  It just felt weird to use a CMS that took five times more code than the whole rest of the website.

I came to the conclusion that for our purposes all I really needed was a simple CMS that would take a database full of “page” rows and automatically put the content into the right spots in the Smarty template.  After a bit of playing around, the idea evoloved of using mod_rewrite and a single page-loader php file…a very elegant solution compared to what we had before.  What I needed then was a way to be able to mark up the content in a way that was non-code-oriented…very legible on it’s own, outside of the page itself.  Markdown was the first thing that came to mind.

I love markdown: it’s simple, elegant, and reads just like a plain-text email.  Everything about how it looked screamed, “You want to use me!  Pick me!  Pick me!” and so I did.  The only problem I ran into was the fact that markdown doesn’t support generating anything besides plain html tags—no class names, ID’s, or anything.  That wasn’t a big deal until I tried to add link anchors to our “about us” page…oops.  Can’t do it.  I found that the “extra” version of the PHP Markdown implementation allows adding custom ID’s to header tags, but I didn’t want to limit myself so badly that early on in the game.

Enter the next contender: textile.  Textile isn’t quite as plain-text looking as markdown, but it supports everything I need.  The issue I had with textile is that the version of the parser they have available for download on their website is ridiculously dated, though the link doesn’t say anything about that.  I was having weird problems like the last element of lists being bumped down into their own list…as in a 5 element ordered list would show as elements 1, 2, 3, 4 and 1.  Doing some research, I found out that this bug had been fixed long ago.  When I read that I immediately looked in the source code for some indication of a date…it had been released in 2006.  That’s a problem.  I did find, however, that the textile based CMS called TextPattern included the most recent version of the classTextile.php file (which is all you need to just use the textile parser) so I grabbed that, and the problem was solved.  You should be able to get the latest version here: TXP/classTextile.php:HEAD

I’ve already gotten the few completely static pages on the website converted so now all I have to do is get the mixed static & dynamic pages working.

It feels unbelievably good to confidently type `git rm` so many times in a console. 

Here at the shop we’ve been working with a customer recently who has been plagued with redirects.  Several of his computers had malware of various types, which we removed.  The problem was that he still had redirects when he got his computers back.  We’d sit and search for 15 or 20 minutes here at the shop (they were Google search redirects) and never see the first redirect, but he’d get them at his place.

At first we thought we were running into a type of malware that we hadn’t seen before and that none of our tools were detecting, but the longer we worked the surer we were that the computers were totally clean.  After fruitless hours of running scans which turned up nothing we came to the realization that we were up against something a little stranger than a piece of malware, and the idea that we couldn’t duplicate the redirects here at the shop kept cropping up in our discussions.  One of the other guys here at the shop had the idea that the bug might only rear it’s ugly head on certain ISP’s networks, but we called the customer back and found that they were using the same ISP as us.  He was on to something though…the common denominator was his network.

Then I remembered.

Not too long ago I’d read online that a lot of the older Linksys routers were vulnerable to remote attacks in which settings could be maliciously altered without the user’s knowledge.  I immediately called the customer and, sure enough, they were using an older Linksys router.

This morning I went to the customer’s place of business and tested the theory.  There in his router was the whole problem…static DNS entries that most certainly did NOT point to his ISP’s DNS servers.  Either by remote attack  or through a malware infection already installed on a PC in his network, someone had hijacked his DNS entries and kept his PC’s redirecting and getting infected.

I didn’t know all the details of the vulnerability, so I replaced the router completely.  It was a WRT54G v2, so it was really old anyway.

So, now we no longer have to pull out our hair and our customer can get back to business as usual.  Crazy days.