Like many IT professionals, I dabble a bit on the weekend. During the week, I’m mostly working in Microsoft Windows environments (I work for a local government agency, hence windows). But at night and on the weekends, I pitch in on some linux based websites (like this one). I had an interesting long-weekend while trying to upgrade Ubuntu on my servers.
One of my side gigs involved a website that collects donations for political purposes. We don’t actually process any donations on the site. We use one of the very well-known payment processors specializing in political donations. Still our bank requires that our website pass some security screenings in order to keep working with us.
Last year, we were notified of our site failing their security scans. The primary culprit: an old version of Apache2 (on linux). I managed to upgrade Apache2 using the apt-get / apt tools and our security errors went away. For a month or so, anyway. Then we got some security warnings regarding our SSL certificates. So we updated them using letsencrypt. Again, all seemed well.
Then two weeks ago, the latest warning popped up in my email. This time, the security folks were complaining about the Ubuntu 14.04 installation. “It needs to be upgraded” said the robotic voice I imagined as I read the email. So I started Googling “upgrade ubuntu 16.04….” Google gladly offered to complete my query: “to 18.04”. So I clicked a link and off I went.
The first time I dabbled in linux was with RedHat back in 1992 or so. Updating the operating system was a daunting task. You had to download sources, configure, build and install things using make tool. Quite arduous task when your hardware was based on Intel i386 or i486 chips and 8 Megabytes of memory. Ah… the good ole days.
But today, it’s easy-peasy. Two commands and you’re done in most cases: “sudo apt-get update” and “sudo apt-get upgrade.” We’ve come a LONG WAY, baby! Except, of course when you need to upgrade major versions of your installation. In that case, there’s another command (or two) that you need to learn “sudo do-release-upgrade”. Pretty slick.
So I upgraded THIS server to 18.04. Went off without a hitch. I created a snapshot of the “money” server and proceeded to install the upgrade. Again, no problems. Both servers in a couple of hours. Man, linux was fun to use.
I re-ran my security screening software and waited on the results with bated breath. FAILURE the email said. How? Why? I just upgraded. Or so I thought. The details indicated that the 18.04 version had many, many vulnerabilities and needed patching. Back to Google and lo and behold, the current version of Ubuntu is 20.04, not 18.04. So time to upgrade.
Now the previous work I’d done on a holiday-weekend Sunday in the light of day. The next update I attempted on Tuesday night (mistake #1). I start the upgrade and I’m immediately greeted with a warning “the following packaged have been held back.” What does that even mean???
I tried updating the packages manually and started to run into problems. All of a sudden Apache2 stops responding and I can’t tell what version of PHP is intalled (both packages that were being held up, among others). After a few more failed attempts at updating these packages, I finally figured out that the 16 to 18 upgrade had removed some repositories that I’d used. So I added the PHP and Apache2 repositories “ppa:ondrej/php” and “ppa:ondrej/apache2” and finally upgraded these packages and some other dependent packages.
I was finally able to upgrade the server to Ubuntu 20.04. Despite all these struggles, it took about 1 hours to break, research, fix and upgrade that server. Not too bad, in the end.
Finally tonight, I was able to upgrade THIS server. One hiccup on this server: during the distro upgrade package lxd upgrade failed and interrupted the distro upgrade. Fortunately, enough error messages were displayed on-screen that I was able to start dpkg to finish the upgrade in short order. Again, break, research, fix and upgrade in less than one hour.
Kudos, Ubuntu! You’ve made a dreaded task almost enjoyable. (I actually enjoy the challenges since my day job consists of baby-sitting 3rd party contractors).
