Signs of a current Chromium crash in GCP Cloud Run when using go-rod

This is mainly here so that I can find it again if it ever comes up, or maybe you'll find it when you're googling a log message.

The tl;dr is that Chromium crashes in GCP Cloud Run (on Alpine Linux if that's relevant) at version 146.0.7680.80-r0.

Why?  I'm not quite sure.  Yesterday, when it was running version 144.0.7559.132-r4, it ran fine.

(We use Chromium under the hood with go-rod to generate PDFs as part of our application.)

In the logs, we saw this:

Connecting to browser at endpoint: ws://127.0.0.1:42590/devtools/browser/f2909d30-39b2-4d59-9fab-e81d79a8f006

Chromium path: /usr/bin/chromium
Recovered from panic: [*net.OpError] write tcp 127.0.0.1:58219->127.0.0.1:42590: use of closed network connection
Uncaught signal: 4, pid=280, tid=280, fault_addr=94169238392146.

As best as I can tell, signal 4 on Linux has to do with a processor instruction mismatch of some kind.  Anyway, the (automatic) Chromium version change was the only thing that could explain the problem, and we rolled back by using "alpine:3.22" instead of "alpine:latest" in our Docker image.

This instantly fixed the problem.  I don't know why it broke, but I know where it broke.  Hopefully they get this fixed up, and now we know that we have to actually test this in GCP if we ever want to un-pin the Alpine version and upgrade Chromium again.

Watch out for SNI when using an nginx reverse proxy

From time to time, I'll have a use case where some box needs to talk to some website that it can't reach (through networking issues), and the easiest thing to do is to throw an nginx reverse proxy on a network that it can reach (such that the reverse proxy can reach both).

The whole shtick of a reverse proxy is that you can access the reverse proxy directly and it'll forward the request on to the appropriate destination and more or less masquerade itself as if it were the destination.  This is in contrast with a normal HTTP proxy that would be configured separately (if supported by whatever tool you're trying to use).  Sometimes a normal HTTP proxy is the best tool for the job, but sometimes you can cheat with a tweak to /etc/hosts and a reverse proxy and nobody needs to know what happened.

Here, we're focused on the reverse proxy.

In this case, we have the following scenario:

1. Box 1 wants to connect to site1.example.com.
2. Box 1 cannot reach site1.example.com.

To cheat using a reverse proxy, we need Box 2, which:

1. Can be reached by Box 1.
2. Can reach site1.example.com.

To set up the whole reverse proxy thing, we need to:

1. Set up nginx on Box 2 to listen on port 443 (HTTPS) and reverse proxy to site1.example.com.
2. Update /etc/hosts on Box 1 so that site1.example.com points to Box 2's IP address.

At first, I was seeing this error message on the reverse proxy's "nginx/error.log":

connect() to XXXXXX:443 failed (13: Permission denied) while connecting to upstream, client: XXXXXX, server: site1.example.com, request: "GET / HTTP/1.1"

"Permission denied" isn't great, and that told me that it was something OS-related.

Of course, it was an SELinux thing (in /var/log/messages):

SELinux is preventing /usr/sbin/nginx from name_connect access on the tcp_socket port 443.

The workaround was:

setsebool -P nis_enabled 1

This also was suggested by the logs, but it didn't seem to matter:

setsebool -P httpd_can_network_connect 1

After fixing that, I was seeing:

SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: XXXXXX, server: site1.example.com, request: "GET / HTTP/1.1"

After tcpdump-ing the traffic from Box 1 and also another box that could directly talk to site1.example.com, it was clear Box 1 was not using SNI in its requests (SNI is a TLS extension that passes the host name in plaintext so that proxies and load balancers can properly route name-based requests).

It took way too long for me to find the nginx setting to enable it (I don't know why its disabled by default), but it's:

proxy_ssl_server_name on;

Anyway, the final nginx config for the reverse proxy on Box 2 was:

server {
  listen 443 ssl;
  server_name site1.example.com;

  ssl_certificate /etc/nginx/ssl/server.crt;
  ssl_certificate_key /etc/nginx/server.key;
  ssl_protocols TLSv1.2;
 
  location / {
    proxy_pass https://site1.example.com;
    proxy_ssl_session_reuse on;
    proxy_ssl_server_name on;
  }
}

As far as Box 1 was concerned, it could connect to site1.example.com with only a small tweak to /etc/hosts.

sed will blow away your symlinks by default

sed typically outputs to stdout, but sed -i allows you edit a file “in place”.  However, under the hood, it actually creates a new file and then replaces the original file with the new file.  This means that sed replaces symlinks with normal files.  This is most likely not what you want.

However, there is a flag to pass to make it work the way that you’d expect:

--follow-symlinks

So, if you're using sed -i, then you probably also want to tack on --follow-symlinks, too.

Dump your SSH settings for quick troubleshooting

I recently had a Jenkins job that would die, seemingly-randomly.  The only thing that really stood out was that it would tend to succeed if the runtime was 14 minutes or less, and it would tend to fail if the runtime was 17 minutes or more.

This job did a bunch of database stuff (through an SSH tunnel; more on that soon), so I first did a whole bunch of troubleshooting on the Postgres client and server configs, but nothing seemed relevant.  It seemed to disconnect ("connection closed by server") on these long queries that would sit there for a long time (maybe around 15 minutes or so) and then come back with a result.  After ruling out the Postgres server (all of the settings looked good, and new sessions had decent timeout configs), I moved on to SSH.

This job connects to a database by way of a forwarded port through an SSH tunnel (don't ask why; just understand that it's the least worst option available in this context).  I figured that maybe the SSH tunnel was failing, since I start it in the background and have it run "sleep infinity" and then never look at it again.  However, when I tested locally, my SSH session would run for multiple days without a problem.

Spoiler alert: the answer ended up being the client config, but how do you actually find that out?

SSH has two really cool options.

On the server side, you can run "sudo sshd -T | sort" to have the SSH daemon read the relevant configs and then print out all of the actual values that it's using.  So, this'll merge in all of the unspecified defaults as well as all of the various options in "/etc/sshd_config" and "/etc/sshd_config.d", etc.

On the client side, you can run "ssh -G ${user}@${host} | sort", and it'll do the same thing, but for all of the client-side configs for that particular user and host combination (because maybe you have some custom stuff set up in your SSH config, etc.).

Now, in my case, it ended up being a keepalive issue.  So, on the server side, here's what the relevant settings were:

clientalivecountmax 0
clientaliveinterval 900
tcpkeepalive yes

On the client (which would disconnect sometimes), here's what the relevant settings were:

serveralivecountmax 3
serveraliveinterval 0
tcpkeepalive yes

Here, you can see that the client (which is whatever the default Jenkins Kubernetes agent ended up being) enabled a TCP keepalive, but it set the keepalive interval to "0", which means that it wouldn't send keepalive packets at all.

According to the docs, the server should have sent out keepalives every 15 minutes, but whatever it was doing, the connection would drop after 15 minutes.  Setting "serveraliveinterval" to "60" ended up solving my problem and allowed my SSH sessions to stay active indefinitely until the script was done with them.

Little bonus section

My SSH command to set up the tunnel in the background was:

ssh -4 -f -L${localport}:${targetaddress}:${targetport} ${user}@${bastionhost} 'sleep infinity';

"-4" forces it to use an IPv4 address (relevant in my context), and "-f" puts the SSH command into the background before "sleep infinity" gets called, right after all the port forwarding is set up.  "sleep infinity" ensures that the connection never closes on its own; the "sleep" command will do nothing forever.

(Obviously, I had the "-o ServerAliveInterval=60" option in there, too.)

With this, I could trivially have my container create an SSH session that allowed for port-forwarding, and that session would be available for the entirety of the container's lifetime (the entirety of the Jenkins build).

QNAP, NFS, and Filesystem ACLs

I recently spent hours banging my head against a wall trying to figure out why my Plex server couldn't find some new media that I put on its volume in my QNAP.

tl;dr QNAP "Advanced Folder Permissions" turn on file access control lists (you'll need the "getfacl" and "setfacl" tools installed on Linux to mess with them).  For more information, see this guide from QNAP.

I must have turned this setting on when I rebuilt my NAS a while back, and it never mattered until I did some file operations with the File Manager or maybe just straight "cp"; I forget which (or both).  Plex refused to see the new file, and I tried debugging the indexer and all that other Plex stuff before realizing that while it could list the file, it couldn't open the file, even though its normal "ls -l" permissions looked fine.

Apparently the file access control list denied it, but I didn't even have "getfacl" or "setfacl" installed on my machine (and I had never even heard of this before), so I had no idea what was going on.  I eventually installed those tools and verified that while the standard Linux permission looked fine, the ACL permissions did not.

"sudo chmod -R +r /path/to/folder" didn't solve my problem, but tearing out the ACL did: "sudo setfacl -b -R /path/to/folder"

Later, I eventually figured out that it was QNAP's "Advanced Folder Permissions" and just disabled that so it wouldn't bother me again.

Encrypt your /home directory using LUKS and a spare disk

Every year or two, I rotate a drive out of my NAS.  My most recent rotation yielded me with a spare 1TB SSD.  My main machine only had a 250GB SSD, so I figured that I'd just replace my /home directory with a mountpoint on that new disk, giving my lots of space for video editing and such, since I no longer had the room to deal with my GoPro footage.

My general thought process was as follows:

1. I don't want to mess too much with my system.
2. I don't want to clone my whole system onto the new drive.
3. I want to encrypt my personal data.
4. I don't really care about encrypting the entire OS.

I had originally looked into some other encryption options, such as encrypting each user's home directory separately, but even in the year 2019 there seemed to be too much drama dealing with that (anytime that I need to make a PAM change, it's a bad day).  Using LUKS, the disk (well, partition) is encrypted, so everything kind of comes for free after that.

If you register the partition in /etc/crypttab, your machine will prompt you for the decryption key when it boots (at least Kubuntu 18.04 does).

One other thing: dealing with encrypted data may be slow if your processor doesn't support AES encryption.  Do a quick check and make sure that "aes" is listed under "Flags":

lscpu;

If "aes" is there, then you're good to go.  If not, then maybe run some tests to see how much CPU overhead disk operations use on LUKS (you can follow this guide, but stop before "Home setup, phase 2", and see if your overhead is acceptable).

The plan

1. Luks Setup
1. Format the new disk with a single partition.
2. Set up LUKS on that partition.
3. Back up the LUKS header data.
2. Home setup, phase 1
1. Copy everything in /home to the new partition.
2. Update /etc/crypttab.
3. Update /etc/fstab using a test directory.
4. Reboot.
5. Test.
3. Home setup, phase 2
1. Update /etc/fstab using the /home directory.
2. Reboot.
3. Test.

LUKS setup

Wipe the new disk and make a single partition.  For the remainder of this post, I'll be assuming that the partition is /dev/sdx1.

Install "cryptsetup".

sudo apt install cryptsetup; 

Set up LUKS on the partition.  You'll need to give it a passphrase.  I recommend something that's easy to type, like a series of four random words, but you do you).  You'll have to type this passphrase every time that you boot your machine up.

sudo cryptsetup --verify-passphrase luksFormat /dev/sdx1;

Once that's done, you can give it some more (up to 8) passphrases.  This may be helpful if you want to have other people access the disk, or if you just want to have some backups, just in case.  If there are multiple passphrases, any one of them will work fine; you don't need to have multiple on hand.

sudo cryptsetup --verify-passphrase luksAddKey /dev/sdx1;

The next step is to "open" the partition.  The last argument ("encrypted-home") is the name to use for the partition that will appear under "/dev/mapper".

sudo cryptsetup luksOpen /dev/sdx1 encrypted-home;

At this point, everything is set up and ready ready.  Confirm that with the "status" command.

sudo cryptsetup status encrypted-home;

Back up the LUKS header data.  If this information gets corrupted on the disk, then there is no way to recover your data.  Note that if you recover data using the header backup, then the passphrases will be the ones in the header backup, not whatever was on the disk at the time of the recovery.

sudo cryptsetup luksHeaderBackup /dev/sdx1 --header-backup-file /root/luks.encrypted-home.header;

I put mine in the /root folder (which will not be on the encrypted home partition), and I also backed it up to Google Drive.  Remember, if you add, change, or delete passphrases, you'll want to do make another backup (otherwise, those changes won't be present during a restoration operation).

If you're really hardcore, fill up the partition with random data so that no part of it looks special.  Remember, the whole point of encryption is to make it so that whatever you wrote just ends up looking random, so writing a bunch of zeros with "dd" will do the trick:

sudo dd if=/dev/zero of=/dev/mapper/encrypted-home;

Before you can do anything with it, you'll need to format the partition.  I used EXT4 because everything else on this machine is EXT4.

sudo mkfs.ext4 /dev/mapper/encrypted-home;

Home setup, phase 1

Once the LUKS partition is all set up, the next set of steps is just a careful copy operation, tweaking a couple /etc files, and verifying that everything worked.

The safest thing to do would be to switch to a live CD here so that you're guaranteed to not be messing with your /home directory, but I just logged out of my window manager and did the next set of steps in the ctrl+alt+f2 terminal.  Again, you do you.

Mount the encrypted home directory somewhere where we can access it.

sudo mkdir /mnt/encrypted-home; 

sudo mount /dev/mapper-encrypted-home /mnt/encrypted-home;

Copy over everything in /home.  This could take a while.

sudo cp -a /home/. /mnt/encrypted-home/;

Make sure that /mnt/encrypted-home contains the home folders of your users.

Set up /etc/crypttab.  The format is:

${/dev/mapper name} UUID="${disk uuid}" none luks

In our case, the /dev/mapper name is going to be "encrypted-home".  To find the UUID, run:

sudo blkid /dev/sdx1;

So, in my particular case, /etc/crypttab looks like:

encrypted-home UUID="5e01cb97-ceed-40da-aec4-5f75b025ed4a" none luks

Finally, tell /etc/fstab to mount the partition to our /mnt/encrypted-home directory.  We don't want to clobber /home until we know that everything works.

Update /etc/fstab and add:

/dev/mapper/encrypted-home /mnt/encrypted-home ext4 defaults 0 0

Reboot your machine.

When it comes back up, it should ask you for the passphrase for the encrypted-home partition.  Give it one of the passphrases that you set up.

Log in and check /mnt/encrypted-home.  As long as everything's in there that's supposed to be in there (that is, all of your /home data), then phase 1 is complete.

Home setup, phase 2

Now that we know everything works, the next step is to clean up your actual /home directory and then tell /etc/fstab to mount /dev/mapper/encrypted-home at /home.

I didn't want to completely purge my /home directory; instead, I deleted everything large and/or personal in there (leaving my bash profile, some app settings, etc.).  This way, if my new disk failed or if I wanted to use my computer without it for some reason, then I'd at least have normal, functioning user accounts.  Again, you do you.  I've screwed up enough stuff in my time to like to have somewhat nice failback scenarios ready to go.

Update /etc/fstab and change /dev/mapper/encrypted-home line to mount to /home.

/dev/mapper/encrypted-home /home ext4 defaults 0 0

Reboot.

When it comes back up, it should ask you for the passphrase for the encrypted-home partition.  Give it one of the passphrases that you set up.

Log in.  You should now be using an encrypted home directory.  Yay.

To confirm, check your mountpoints:

mount | grep /home

You should see something like:

/dev/mapper/encrypted-home on /home type ext4 (rw,relatime,data=ordered)

Now that everything's working, you can get rid of "/mnt/encrypted-home"; we're not using it anymore.

sudo rmdir /mnt/encrypted-home;

Shrinking a disk to migrate to a smaller SSD

I have finally migrated all of my storage to solid state drives (SSDs).  For my older machines, the 256GB SSDs were bigger than their normal hard drives, so Clonezilla worked perfectly well.  The very last holdout was my 2TB Proxmox server, since my largest SSD was 480GB.

Since I was only using about 120GB of space on that disk, my plan was to shrink the filesystems, then shrink the partitions, and then clone the disk with Clonezilla.

tl;dr both Clonezilla and grub can be jerks.

Even after shrinking everything down, Clonezilla refused to clone the disk because the SSD was smaller than the original (even though all of the partitions would fit on the SSD with plenty of room to spare).  Clonezilla did let me clone a partition at a time, so I did that, but the disk wouldn't boot.  To make a long story short, you need to make sure that you run "update-grub" and "grub-install" after pulling a disk-switcheroo stunt.  And yes, to do that, you'll need to use a live OS to chroot in (and yes, you'll need to mount "/dev", etc.) and run those commands.

Just that paragraph above would have saved me about six hours of time, so there it is, for the next person who is about to embark on a foolish and painful journey.

Shrink a disk

My Proxmox host came with a 2TB spinner disk, and I foolishly decided to use it instead of putting in a 256GB SSD on day one.  I didn't feel like reinstalling Proxmox and setting all of my stuff up again, and I certainly didn't want to find out that I didn't back up some crucial file or couldn't remember the proper config; I just wanted my Proxmox disk migrated over to SSD, no drama.

Remember, you aren't actually shrinking the disk itself; rather, you're reducing the allocated space on that disk by shrinking your filesystems and partitions (and, if you're using LVM, your logical volumes and physical volumes, too).

So, the goal is: shrink all of your partitions down to a size that they can be copied over to a smaller disk.

Remember, you don't have to shrink all of your partitions, on the big ones.  In my case, I had a 1MB partition, a 500MB partition, and a 1.9TB partition.  Obviously, I left the first two alone and shrunk the third one.

First: Shrink whatever's on the partition

How easy this step is depends on if you're using LVM or not.  If you are not using LVM and just have a normal filesystem on the partition, then all you need to do is resize the filesystem to a smaller size (obviously, you have to have less space in use than the new size).

If you are using LVM, then your partition is full of LVM stuff (namely, all of the data needed to make the logical volumes (LVs) that reside on that partition (which is the physical volume, or PV)).

Note: you cannot mess with a currently-mounted filesystem.  If you need to resize your root filesystem, then you'll have to boot into a live OS and do it from there.

(From here on, we'll use "sda" as the original disk and "sdb" as the new SSD.)

 Option 1: Filesystem

Shrink the filesystem using the appropriate tool for your filesystem.  For example, let's assume that we want to shrink "/dev/sda3" down to 50GB.  You could do this with an EXT4 filesystem via:

resize2fs -p /dev/sda3 50G

Option 2: LVM

Shrink whatever filesystems are on the logical volumes that are on the physical volume that you want to shrink.  For example, let's assume that we want to shrink the "your-lv" volume down to 50GB.

resize2fs -p /dev/your-vg/your-lv 50G

Then, shrink the logical volume to match:

lvresize /dev/your-vg/your-lv --size 50G

Now that the logical volume has been shrunk, you should be able to see free space in its physical volume.  Run "pvs" and look at the "PFree" column; that's how much free space you have.

Now shrink the physical volume:

pvresize /dev/sda3 --setphysicalvolumesize 50G

Second: Shrink the partition

Be very, very sure of what you want the new size of your partition to be.  Very sure.  The surest that you've ever been in your life.

Then, use "parted" or "fdisk" to (1) delete the partition, and then (2) add a new partition that's smaller.  The new partition must start on the exact same spot that the old one did.  Make sure that the partition type is set appropriately ("Linux" or "Linux LVM", etc.).  Basically, the new one should look exactly like the old one, but with the ending position smaller.

Clone a disk

Use Clonezilla (or any other live OS, for that matter) to clone the partitions to the new disk.  With Clonezilla, I still had to make the partitions on the new disk myself, so while you're there, you might as well just "dd" the contents of each partition over.

Create the new partitions and copy the data

On the new disk, create partitions that will correspond with those on the old disk.  You can create them with the same size, or bigger.  I created my tiny partitions with the same sizes (1MB and 500MB), but I created my third partition to take up the rest of the SSD.

Copy the data over using whatever tool makes you happy; I like "dd" because it's easy.  Be very, very careful about which disk is the source disk (input file, or "if") and which is the target disk (output file, or "of").  For example:

dd if=/dev/sda1 of=/dev/sdb1 bs=20M

dd if=/dev/sda2 of=/dev/sdb2 bs=20M

dd if=/dev/sda3 of=/dev/sdb3 bs=20M

If you create larger partitions on the new disk, everything will still work just fine; the filesystems or LVM physical volumes simply will have the original sizes.  Once everything is copied over, you can simply expand the filesystem (or, for LVM, expand the physical volume, then the logical volume, and then the physical volume) to take advantage of the larger partition.

Remove the original disk

Power off and remove the original disk; you're done with it anyway.

If you're using LVM, that original disk will only get in the way, since it'll get confused about having double the number of everything and won't set up your volumes correctly.

(From here on, we'll use "sda" as the new SSD since we've removed the original disk.)

Bonus: expand your filesystems

Now that everything's copied over to the new disk, you can expand your filesystems to take up the full size of the partition.  You may have to run "fsck" before it'll let you expand the filesystem.

For a normal filesystem, simply use the appropriate tool for your filesystem.  For example, let's assume that we want to grow "/dev/sda3" to take up whatever extra space there might be.  You could do this with an EXT4 filesystem via:

resize2fs /dev/sda3

In the case of LVM, you'll have to expand the physical volume first, then the logical volume, and then the filesystem (essentially, this is the reverse of the process to shrink them).  For example:

pvresize /dev/sda3

lvresize -l +100%FREE /dev/your-vg/your-lv

resize2fs /dev/your-vg/your-lv

Note: the "-l +100%FREE" arguments to "lvresize" tell it to expand to include all ("100%") of the free space available on the physical volume.

Fix grub

I still don't fully grok the changes that UEFI brought to the bootloader, but I do know this: after you pull a disk-switcheroo, grub needs to be updated and reinstalled.

So, using your live OS, mount the root filesystem from your new disk (and any other dependent filesystem, such as "/boot" and "/boot/efi").  Then create bind mounts for "/dev", "/dev/pts", "/sys", "/proc", and "/run", since those are required for grub to install properly.

Finally, chroot in to the root filesystem and run "update-grub" followed by "grub-install" (and give it the path to the new disk):

update-grub

grub-install /dev/sda

Conclusion

That should do it.  Your new disk should boot properly.

For a decent step-by-step guide on shrinking an LVM physical volume (but with the intent of creating another partition), see this article from centoshelp.org.

Troubleshooting PXE boot problems on a Dell server

At work, we ship "appliances"—servers that have a pre-installed operating system and software stack.  The general workflow to build one is:

1. The box is racked and plugged into the "build" network.
2. The box PXE boots to a special controller script.
3. Someone answers a few questions.
4. The script partitions and formats the disks and installs the OS and software.
5. The box is powered down, packaged, and shipped.

I recently spent days fighting with a new PXE boot "live OS", encountered error messages so lonely that few sites on the Internet reference them, debugged "initrd", and ultimately solved my problems by feeling really, really stupid.

I'll tell you the story of what happened in case you happen to try to walk down this path in the future, and I'll go into detail on everything as we go.  But first...

The Motivation

When you PXE boot, you get a "live OS", typically a read-only operating system with some tmpfs mounts for write operations.  This OS is often minimal, having just enough tools and packages to accomplish the goal of partitioning, formatting, and copying files.  Ours is done by mounting the root filesystem over NFS, so the smaller the OS, the better.

We had two problems that I wanted to solve:

1. The current live OS was hand-built on Ubuntu 12.04 with no documentation; and
2. Ubuntu 12.04's version of "lbzip2" is buggy and can't decompress some files.

My goal was to switch to Ubuntu 14.04 (which has a perfectly working version of "lbzip2") in a way that the live OS would be generated from a script (reliably and repeatably).

(If you're interested in building an Ubuntu filesystem from scratch that you can then use as a live OS, or make into an ISO image, or whatever, see debootstrap.)

The Execution

I created a few-dozen line script to bootstrap Ubuntu 14.04, install a kernel and some packages, and install the tools that it would need to run when it booted.  That part went well.

I copied the new OS to our NFS server, replacing the existing one.  That part went well.

All of our automated virtual-appliance builds continued to work (actually better and faster than before).  That part went well.

I met with the manufacturing team to get the formal sign-off that it would work with the physical Dell servers that we use.  This did not go well.

The Flaw In The Plan

Manufacturing booted up a box onto the build network but it crashed.

Here's what he reported:

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200

CPU: 4 PID: 1 Comm: init Not tainted 3.13.0-24-generic #47-Ubuntu

Hardware name:    /0JP31P, BIOS 2.5.2 01/28/2015

 ffff880802af0000 ffff8808041f5e48 ffffffff81715ac4 ffffffff81a4c480

 ffff8808041f5ec0 ffffffff8170ecc5 ffffffff00000010 ffff8808041f5ed0

 ffff8808041f5e70 ffffffff81f219e0 0000000000000200 ffff8808041f8398

Call Trace:

 [<ffffffff81715ac4>] dump_stack+0x45/0x56

 [<ffffffff8170ecc5>] panic+0xc8/0x1d7

 [<ffffffff8106a391>] do_exit+0xa41/0xa50

 [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60

 [<ffffffff8106a41f>] do_group_exit+0x3f/0xa0

 [<ffffffff8106a494>] SyS_exit_group+0x14/0x20

 [<ffffffff817266bf>] tracesys+0xe1/0xe6

------------[ cut here ]------------

WARNING: CPU: 4 PID: 1 at /build/buildd/linux-3.13.0/arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()

Modules linked in: nfs lockd sunrpc fscache

Okay, a kernel panic.  I did upgrade the kernel from the old Ubuntu 12.04 version.  In fact, the old live OS had a hand-built Gentoo kernel for arcane reasons.  So maybe...

The Red Herring

Well, I did install the Ubuntu 14.04 ".deb" package for the Fusion I/O Drive's "iomemory-vsl" kernel module.  Some of our equipment have I/O Drives, so the live OS needs to be able to talk to them.

The VMs (which don't have I/O Drives) worked fine, and the physical hardware (which do) did not work.

I assumed that there was some mismatch between Fusion I/O's kernel module and the Ubuntu 14.04 kernel.  I compiled the driver (using Fusion I/O's guide) and re-made my live OS.

Same result: kernel panic.

In addition, it turned out that physical hardware that did not have an I/O Drive failed, as well.

The Hint

I took out my phone and recorded the boot process at 120 frames per second to see the text before the kernel panic.  The messages that I'd been able to see were not helpful, and when the kernel panics, I lose the ability to scroll up in the history.  And since the system never gets up all the way, there's no way to access the logs.

Here's what I saw:

systemd-udevd[329]: starting version 204

Begin: Loading essential drivers ... done

Begin: Running /scripts/init-premount ... done

Begin: Mounting root file system ... Begin: Running /scripts/nfs-top ... done

FS-Cache: Loaded

RPC: Registered named UNIX socket transport module.

RPC: Registered udp transport module.

RPC: Registered tcp transport module.

RPC: Registered tcp NFSv4.1 backchannel transport module.

FS-Cache: Netfs 'nfs' registered for caching

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

ipconfig: no devices to configure

/init: .: line 252: can't open '/run/net-*.conf'

Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200

The short story here is that the "ipconfig" message repeating over and over means that the kernel can't find any network interfaces (at least other than "lo").

The long story is that when booting from network (NFS), the "initrd" init script is going to run some scripts to set up NFS.  One of these calls a function "configure_networking" (since NFS won't work without networking).  That function tries to figure out which device to use for networking.

For a box with multiple interfaces, this could be a bit tricky.  However, if you've toggled on bit b10 (0x2) in your PXE setup using the "IPAPPEND" directive, then you'll get an environment variable called "BOOTIF" that encodes the MAC address of the interface that originally PXE booted.  The "configure_networking" function then looks through the /sys/class/net/* files to see which interface has that MAC address.  From there, it will know the device and can set it up.

However, "ipconfig" claims that there are "no devices to configure".  Also, if you debug the "configure_networking" function, you'll see that there's only one file in /sys/class/net, and that's "lo".  So, the system has a loopback interface, but none of the Ethernet interfaces that I expected.

The Problem

Very simply put: the appropriate Ethernet driver (kernel module) was not present in the "initrd" file.

The "initrd" file (short for "initial RAM disk") contains some bootstrap scripts and kernel modules.  For example, for PXE booting, we get the kernel and the "initrd" file from TFTP (or HTTP) from the Ethernet card itself as part of the process.  Thus, it is up to the "initrd" file to get networking up and running so that it can mount the root filesystem over NFS.

I jumped through hoops trying to figure out how to get the "initrd" file to have enough (or the right) modules.  I eventually settled on bucking the trends set in the "Diskless Ubuntu" and "Diskless Debian" guides.

In my /etc/initramfs-tools/update-initramfs.conf, I set:

MODULES=most

instead of:

MODULES=netboot

The "MODULES" variable tells "update-initramfs" (and "mkinitramfs") which modules to include in the "initrd" file.  Obviously, less modules means less space.  So setting "MODULES" to "netboot" is a special setting that tries to pull in the bare minimum to get networking working.  I figured whatever; more modules is better than less modules, especially since the module for my Ethernet card isn't making it in there.

However, that still didn't help.

(Note: I haven't experimented with "MODULES=netboot" again; the "initrd" file that gets generated is under 20MB, and it takes under 1 second to transfer that on my network, so I'm not too interested in trimming that file down to the bare essentials at this point.)

My Ethernet card was some kind of Broadcom NetXtreme card, and the Internet seemed to think that the appropriate kernel module for it is "tg3".

After banging my head for way too long, I realized that "tg3" was not present in either the "initrd" file or the "/lib/modules/" directory.  I didn't realize that this was a problem because the Ubuntu 12.04 version of the system had compiled-in drivers (remember, I said that it was a Gentoo-built kernel); they were not built as modules (".ko" files).  So all of my grepping and comparing of files and directories seemed to tell me that a "tg3.ko" file was not necessary or relevant.

Once I realized that I needed the "tg3" module (and that it was compiled into the older kernel), I had to find out where to get it from.

The Solution

It turns out that there are two Linux kernel packages in Ubuntu:

1. linux-image; and
2. linux-image-extra.

The "linux-image-extra" package was key.  This package dropped in a lot of modules, including "tg3.ko".

So, in addition to installing "linux-image", I installed "linux-image-extra".  When I ran "update-initramfs", all of the modules got copied into the "initrd" file.  So when I booted the box, it successfully found my interfaces (for example, "eth2"), mounted the root NFS filesystem, and continued along its merry way.

FS-Cache: Netfs 'nfs' registered for caching

[...]

IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready

[...]

IP-Config: eth2 hardware address b0:83:XX:XX:XX:XX mtu 1500 DHCP RARP

IP-Config: no response after 2 secs - giving up

tg3 0000:01:00.0 eth2: Link is up at 1000 Mbps, full duplex

tg3 0000:01:00.0 eth2: Flow control is on for TX and on for RX

tg3 0000:01:00.0 eth2: EEE is disabled

IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

IP-Config: eth2 hardware address b0:83:XX:XX:XX:XX mtu 1500 DHCP RARP

IP-Config: no response after 3 secs - giving up

IP-Config: eth2 hardware address b0:83:XX:XX:XX:XX mtu 1500 DHCP RARP

IP-Config: eth2 guessed broadcast address 10.XXX.XX.XXX

IP-Config: eth2 complete (dhcp from 192.168.XX.XX):

 address: 10.XXX.XX.XXX    broadcast: 10.XXX.XX.XXX    netmask: 255.255.252.0

 gateway: 10.XXX.XX.X      dns0     : 192.168.XX.XX    dns1   : 10.XXX.X.XX

 domain : XXX.XXXXXX.com

 rootserver: 10.XXX.XXX.XX rootpath:

 filename: pxelinux.0

Begin: Running /scripts/nfs-premount ... done

Begin: Running /scripts/nfs-bottom ... done

done

Begin: Running /scripts/init-bottom ... done

The Lesson

Well, what did I learn here?

1. "ipconfig: no devices to configure" means that the kernel doesn't think that you have any Ethernet devices (either there aren't any, or you're missing the driver for it).
2. "linux-image-extra" has all the drivers that you probably want (it certainly has "tg3").
3. When I see "Running /scripts/nfs-premount" messages, that file lives in the "initrd" file.  It is totally possible to take apart an "initrd" file (with "cpio"), make changes to the scripts, and then put it back together (again with "cpio").
4. If it seems impossible that your I/O Drive kernel module is causing networking-related kernel panics, then you're probably right and you have a networking issue somewhere.
5. Not being able to set up NFS in a network-booting setup will result in a "kernel panic".  This surprised me; I figured that I'd get to an emergency console or something.  Kernel panics usually point to deep, scary problems, not something as simple as not getting an IP address.

Fixing the default Ubuntu snmpd configuration

SNMP is super helpful for performance and health monitoring of any production equipment.  It's lightweight, easy to understand, and very resilient when Bad Things happen to the network.  If you're not monitoring your production equipment with SNMP, then probably should look into that right away (we use SevOne NMS at work).

Getting "snmpd", the Linux SNMP daemon, up and running on Ubuntu is simply a matter of installing "snmpd":
sudo apt-get install snmpd;

Or is it?

Default configuration woes

Logging

By default, Ubuntu wants to log literally everything that "snmpd" does to syslog.  While I love the enthusiasm, this quickly leads to overflowing logs and the headache around them (plus it makes it impossible to find any event that's actually important).

How many times do you want to see messages like this in your logs?

Sep 11 16:48:23 your-server snmpd[19552]: Connection from UDP: [192.168.59.101]:49867->[10.129.11.219]
Sep 11 16:48:23 snmpd[19552]: last message repeated 199 times

The logging options are specified on the "snmpd" command line, and are thus configured in "/etc/default/snmpd".

The default logging settings are:

-Lsd

"-L" is for the logging options.  "s" is for syslog.  "d" is for the daemon facility.

What we want are these settings:

-LS 4 d

"-L" again is for logging options.  Capital "S" is for a priority-filtered syslog, with "4" being "warning-level or higher".  Again, "d" is for the daemon facility.

Port access

By default, Ubuntu locks down SNMP access to "localhost", so it's 100% useless from a monitoring perspective.  While I respect the security-mindedness displayed here, I need my boxes to actually respond to requests.

The access options are specified in the "snmpd.conf" file, which is located here: "/etc/snmp/snmpd.conf".

At the top of the file, there is a configuration item called "agentAddress".  By default, this limits requests to those originating locally.

agentAddress udp:127.0.0.1:161

There is usually a line following it that's commented out, and that's the one that we want.  Get rid of the line above and make sure that this one is enabled:

agentAddress udp:161,udp6:[::1]:161

This makes sure that any requests to port 161 (the standard SNMP port) will be allowed.

Permissions

Yes, yes, we should all be using SNMPv3's great user-based access-control mechanism, but for an internal-to-the-company server that can't be reached from the Internet, we can often afford to be lax.  And hey, I'm not stopping you from setting up SNMPv3 access control.  Go nuts.

Here, we're going to allow the community string of "public" to access everything about the box (but not make any changes at all).

The default configuration allows "public" to see some basic system information, but that's not good enough:

rocommunity public default -V systemonly

Get rid of that line and replace it with one that doesn't have the "systemonly" restriction:

rocommunity public

Restart "snmpd" and you'll be ready to respond to SNMP requests from your local management station.
sudo service snmpd restart;

It's 2013; use "getaddrinfo" for sockets

If you're anything like me, you've come to accept this as true:

• Connecting to anything via a scripting language is easy; if it's a domain name, then it'll be translated.  If there's a port number at the back, then it'll be extracted.  IPv4 or IPv6?  No problem.
• Doing the same amount of work in Linux using C/C++ is a nightmare.

Well, it turns out that that's not the case.  However, because of the usual man page obfuscation involving networking, it has seemed that way to me for years.

Calling "getaddrinfo"

The function "getaddrinfo" is meant to be the starting place for network connections.  It takes two strings, optional hints, and a pointer to use for your result structure.  Seems simple, right?

Conceptually, you call it in this way:

struct addrinfo* targetAddress = NULL;

int result = getaddrinfo(

   "your-thing.example.com",

   "80",

   NULL,

   &targetAddress

);

if( result != 0 ) {

   cout << "Error resolving address: " << gai_strerror( result ) << endl;

   // ...

}

// ...

freeaddrinfo( targetAddress );

Ultimately, this takes "your-thing.example.com", resolves it to an IP address (it could be IPv4 or IPv6, or both); takes "80", and resolves it to a port number (this could have been "http" and it would still work); and puts the resulting information into "targetAddress", which it allocates for you.

When you're done, free "targetAddress" using "freeaddrinfo".

And yes, this thing has its own error function, "gai_strerror".

Using a "struct addrinfo*"

The result of "getaddrinfo" can be immediately used by "socket":

int socketHandle = socket(

   targetAddress->ai_family,

   targetAddress->ai_socktype,

   targetAddress->ai_protocol

);

if( socketHandle < 0 ) {

   cout << "Could not create socket." << endl;

   // ...

}

int result = connect(

   socketHandle,

   targetAddress->ai_addr,

   targetAddress->ai_addrlen

);

if( result != 0 ) {

   cout << "Could not connect to target address." << endl;

   // ...

}

But don't you normally have to specify the kind of socket?  Is it a stream- or datagram-based?  Is it IPv4 or IPv6?  What network protocol should it use?

That's where things get interesting.  So, a "struct addrinfo*" actually represents a linked list.  "getaddrinfo" then returns a list of possible results.  If we told it only the domain name and port, then it would return many different "struct addrinfo*" instances, all linked together starting from the original one.  To access the next one, use "targetAddress->ai_next".

Now, that might be kind of cool in the sense that you can simply loop through the results and keep trying them until one works.  But you probably know in advance what you're looking for.  I know that I generally only want IP-version-agnostic name translation.

This is where hints come in.

The third parameter to "getaddrinfo" is another "struct addrinfo*", but this time, you get to fill it out, and it'll use the contents of that structure as filter criteria.

For example, if you wanted to connect to "your-thing.example.com:80" over UDP, then you would just set up the hint accordingly:

struct addrinfo hint;

memset( &hint, 0, sizeof( decltype(hint) ) );

// Allow either IPv4 or IPv6.

hint.ai_family = AF_UNSPEC;

// Use a datagram socket (since that's what UDP is).

hint.ai_socktype = SOCK_DGRAM;

// Don't worry about flags.

hint.ai_flags = 0;

// Use UDP, specifically.  UDP is protocol number 17.

hint.ai_protocol = 17;

struct addrinfo* targetAddress = NULL;

int result = getaddrinfo(

   "your-thing.example.com",

   "80",

   &hint, //< Specify the hint here.

   &targetAddress

);

Now, since your protocol information is all filled out already, you're either going to get back something that you can immediately use, or "getaddrinfo" will fail.  Either way, the guesswork is gone.

init, setrlimit, and ZeroMQ

Linux has the notion of "resource limits" per process.  This conceptually cool because you could cap potentially dangerous processes with regard to memory size, number of threads, real-time priority, and a bunch of other things (including number of open files, which we'll come to shortly).  Practically, this protects a box from crippling itself under the weight of a malicious (or simply buggy) process.

Resource limits come with two notions of limit: a "hard" limit and a "soft" limit.  The hard limit is the limit set by "the system" and acts as a true upper bound on the particular resource.  The soft limit is a limit that may be changed by the process--but only up until the hard limit.

This way, a process may be able to create a vast number of threads, but it must specifically increase its own limit in order to do so.  It's a tiny little check to make you really think about what you're doing, since you have to manually increase the resource limit within the process.

In the shell: ulimit

When in a shell, the "ulimit" command can be used to change these limits from that moment forward.  This is often used to allow any processes spawned by the shell to have a nonzero (usually infinite) sized core file, which can be used to debug a program that crashes.

In the code: setrlimit

In C/C++, you can use "setrlimit" to change the limits.

How hard is "hard"?

In addition to changing the soft limit, you can change the hard limit if you are root.  Let me say that one more time: you can change hard limits if you are root.

Why is this important?  When the "init" process starts, it sets the hard limit for the maximum number of files open at any one time to 1024.  I ran into this limit years ago, and I tried to change the soft limit, but to no avail.  I couldn't get my process (which must run out of "init") to set this limit higher than 1024, not even as root.  It wasn't until recently that I realized that I didn't try changing the hard limit as root.  I naively thought that a limit documented as "hard" would mean that nothing could change it.  Nope.  One quick call and that number was anything I wanted.

So, why was that important?  We've recently begun to use ZeroMQ in our code at work, and it's pretty cool.  Without getting into too many details, ZeroMQ is a a message-passing system that abstracts communication into "messages" that are sent and received from "sockets".  In theory these sockets have nothing to do with Linux sockets; they are magical mailboxes for magical postmen that deliver magical messages.  In practice, they are super clean wrappers around TCP, UDP, and Unix sockets.

What did I have that needed a thousand network connections?  Nothing.  That's where things got interesting.  ZeroMQ's documentation leads you to believe that you should give up on managing global task lists and mutexes and signaling in favor of its notion of a simple message queue and workers that pull from it.  It wants you to focus on the communication aspect of what you're doing rather than getting lost in mutex hell.  As icing on the cake, ZeroMQ provides "in-process" sockets that don't hit the networking stack at all and can only be used within a process.  Perfect for eliminating task lists and custom thread pool code.

What they don't tell you is that every single ZeroMQ "socket" internally uses the "socketpair" call to generate two real sockets for some internal signalling mechanism--in addition to any networking sockets that are needed.  So, if you really and truly adopted their paradigm, throwing in-process sockets everywhere to simplify all of your code, you would easily run out of file descriptors on a process that spawned from "init".

Another limit?

After allowing my process an unlimited number of open files, I found that ZeroMQ was throwing exceptions about "too many open files", even when I was only around 1,000 open files.  This made no sense at all, since my process limit was now over 65,000.  So what was going on?

It turns out that ZeroMQ internally has a maximum number of files that it wants to respect.  That's great, except that that limit can only be changed in "config.hpp", a random file in the source code of ZeroMQ.  In order to increase that limit to acceptable levels, I had to manually patch ZeroMQ's source code, recompile it, and redistribute it to my equipment.

Summary

• Resource limits place limits on how much a process may do.
• "root" can arbitrarily change these limits, including the unchangeable "hard" limits.
• "nofiles" is the limit for the maximum number of open files.  "no" here represents an abbreviation of the word "number" (for whatever reason that those four extra letters couldn't be trusted).
• To set the limit in C/C++, use "setrlimit" with RLIMIT_NOFILES.
• ZeroMQ creates two sockets for every ZeroMQ "socket" that it creates.
• "max_sockets" is ZeroMQ's own constant (defined in "config.hpp") for the number of sockets that it will allow itself to open.