Showing posts with label Ubuntu. Show all posts
Showing posts with label Ubuntu. Show all posts

Sunday, January 8, 2017

Running transmission as a user in Ubuntu 16.04

I recently bought a QNAP TS-451+ NAS for my house.  This basically meant that all of my storage would be backed by NFS mounts.  QNAP (currently) only supports NFSv3, so the UIDs and GIDs on any files have to match the UIDs and GIDs on all of my boxes for everything to work.

The "standard" first user (in my case, "codon") in Ubuntu has a UID of 1000 and a GID of 1000 (also called, uncreatively "codon").  I have this user set up on every box in my network, so this means that any "codon" user on any box can read and write to anything that any other "codon" user created.

However, I use transmission to download torrents.  Transmission runs as the "debian-transmission" user, with a UID of 500 and a GID of 500.  Because of this, "codon" had to use "sudo" to change the ownership of any downloaded files before moving them to their final destinations.  In addition, my Plex server couldn't see the files (because the "plex" user was in the "codon" group with GID 1000).

Without jumping through a bunch of hoops, I figured that the simplest solution would be to run transmission as the "codon" user.  This should have been the simplest way, but I ran into a bunch of problems and outdated guides.  Here, I present to you how to run transmission as a user in Ubuntu 16.04.

Background: systemd

Ubuntu 16.04 uses systemd for its "init" system.  Systemd is great, but it's a whole different game than the older SysV init or now-defunct Upstart init systems.

The magic of systemd is that you do not have to modify any system-owned files; this means that you will not get conflicts when you upgrade.  Instead, you simply supplement an existing unit (a "unit" is what systemd operates on, not an "init script") by creating a new file in a particular location.

Step 1: run as a different user

Explanation

The systemd unit for transmission is called "transmission-daemon", and it is defined here:
/lib/systemd/system/transmission-daemon.service
In principle, you could simply modify that file to change the "User=debian-transmission" line, but that's not the systemd way.  Instead, we're going to create a supplement file in "/etc/systemd/" that has a different "User=..." line.  When systemd reads its configuration, it'll read the original "transmission-daemon.service" file, and then it'll apply the changes in our supplement file.

The appropriate place to put these supplement files is the following directory:
/etc/systemd/${service-path}.d/
Here, ${service-path} refers to the path after "/lib/systemd".  In transmission's case, this is "system/transmission-daemon.service".

What to do

Stop transmission (if it's already running).
sudo systemctl stop transmission-daemon;
Create the supplement file directory for transmission.
sudo mkdir -p /etc/systemd/system/transmission-daemon.service.d;
Create a new supplement file called "run-as-user.conf".
sudo vi /etc/systemd/system/transmission-daemon.service.d/run-as-user.conf
and put the following text in it.
[Service]
User=codon
Obviously, use your desired username and not "codon".

Tell systemd to reload its units.
sudo systemctl daemon-reload;
To confirm that the changes went through, ask systemd to print the configuration that it's using for transmission.
sudo systemctl cat transmission-daemon;
You should see something like this:
# /lib/systemd/system/transmission-daemon.service
[Unit]
Description=Transmission BitTorrent Daemon
After=network.target 
[Service]
User=debian-transmission
Type=notify
ExecStart=/usr/bin/transmission-daemon -f --log-error
ExecReload=/bin/kill -s HUP $MAINPID 
[Install]
WantedBy=multi-user.target 
# /etc/systemd/system/transmission-daemon.service.d/run-as-user.conf
[Service]
User=codon
You can see that our supplement file was appended to the end.

Step 2: get the configuration directory setup

Explanation

Transmission loads its configuration from the user's home directory, in particular:
~/.config/transmission-daemon/
Since Ubuntu runs transmission as the "debian-transmission" user, the transmission configuration resides in that user's home directory, which happens to be:
/var/lib/transmission-daemon
And here's where things get strange.

The normal place for system-wide configuration files is "/etc/", and transmission tries to pass itself off as a normal, system-wide daemon.  To get around this, the main configuration file for transmission appears to be "/etc/transmission-daemon/settings.json", but it's actually:
/var/lib/transmission-daemon/.config/transmission-daemon/settings.json
The Ubuntu configuration just happens to symlink that file to "/etc/transmission-daemon/settings.json".

Now that we're going to be running our transmission as our user and not "debian-transmission", the configuration will be loaded from our user's home directory (in my case, "/home/codon").  Thus, the fake-me-out "/etc/transmission-daemon/settings.json" configuration file will not be used.

When transmission starts, if it doesn't have a configuration directory set up already, it will set one up on its own.  So here, we want to start transmission briefly so that it configures this directory structure for us.

What to do

Start transmission and then stop transmission.
sudo systemctl start transmission-daemon;
sudo systemctl stop transmission-daemon;
You should now have the following directory in your user's home directory:
.config/transmission-daemon/

Step 3: change "settings.json" and start transmission

Explanation

Transmission will load its configuration from "~/.config/transmission-daemon/settings.json" when it starts, so just update that file with whatever your configuration is and then start transmission.

Note that transmission will overwrite that file when it stops, so make your changes while transmission is not running.  If you make changes while transmission is running, then simply issue the "reload" command to it and it'll reload the configuration live.
sudo systemctl reload transmission-daemon;

What to do

Update "~/.config/transmission-daemon/settings.json" to suite your needs.

Then, start transmission:
sudo systemctl start transmission-daemon;
Make sure that everything's running okay:
sudo systemctl status transmission-daemon;
You should see something like this:
● transmission-daemon.service - Transmission BitTorrent Daemon
   Loaded: loaded (/lib/systemd/system/transmission-daemon.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/transmission-daemon.service.d
           └─run-as-user.conf
   Active: active (running) since Sun 2017-01-08 19:28:27 EST; 1min ago
 Main PID: 3271 (transmission-da)
   Status: "Uploading 54.18 KBps, Downloading 0.17 KBps."
   CGroup: /system.slice/transmission-daemon.service
           └─3271 /usr/bin/transmission-daemon -f --log-error

Jan 08 19:28:27 transmission systemd[1]: Starting Transmission BitTorrent Daemon...
Jan 08 19:28:27 transmission systemd[1]: Started Transmission BitTorrent Daemon.

Thursday, April 28, 2016

Troubleshooting PXE boot problems on a Dell server

At work, we ship "appliances"—servers that have a pre-installed operating system and software stack.  The general workflow to build one is:

  1. The box is racked and plugged into the "build" network.
  2. The box PXE boots to a special controller script.
  3. Someone answers a few questions.
  4. The script partitions and formats the disks and installs the OS and software.
  5. The box is powered down, packaged, and shipped.
I recently spent days fighting with a new PXE boot "live OS", encountered error messages so lonely that few sites on the Internet reference them, debugged "initrd", and ultimately solved my problems by feeling really, really stupid.

I'll tell you the story of what happened in case you happen to try to walk down this path in the future, and I'll go into detail on everything as we go.  But first...

The Motivation

When you PXE boot, you get a "live OS", typically a read-only operating system with some tmpfs mounts for write operations.  This OS is often minimal, having just enough tools and packages to accomplish the goal of partitioning, formatting, and copying files.  Ours is done by mounting the root filesystem over NFS, so the smaller the OS, the better.

We had two problems that I wanted to solve:
  1. The current live OS was hand-built on Ubuntu 12.04 with no documentation; and
  2. Ubuntu 12.04's version of "lbzip2" is buggy and can't decompress some files.
My goal was to switch to Ubuntu 14.04 (which has a perfectly working version of "lbzip2") in a way that the live OS would be generated from a script (reliably and repeatably).

(If you're interested in building an Ubuntu filesystem from scratch that you can then use as a live OS, or make into an ISO image, or whatever, see debootstrap.)

The Execution

I created a few-dozen line script to bootstrap Ubuntu 14.04, install a kernel and some packages, and install the tools that it would need to run when it booted.  That part went well.

I copied the new OS to our NFS server, replacing the existing one.  That part went well.

All of our automated virtual-appliance builds continued to work (actually better and faster than before).  That part went well.

I met with the manufacturing team to get the formal sign-off that it would work with the physical Dell servers that we use.  This did not go well.

The Flaw In The Plan

Manufacturing booted up a box onto the build network but it crashed.

Here's what he reported:
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200

CPU: 4 PID: 1 Comm: init Not tainted 3.13.0-24-generic #47-Ubuntu
Hardware name:    /0JP31P, BIOS 2.5.2 01/28/2015
 ffff880802af0000 ffff8808041f5e48 ffffffff81715ac4 ffffffff81a4c480
 ffff8808041f5ec0 ffffffff8170ecc5 ffffffff00000010 ffff8808041f5ed0
 ffff8808041f5e70 ffffffff81f219e0 0000000000000200 ffff8808041f8398
Call Trace:
 [<ffffffff81715ac4>] dump_stack+0x45/0x56
 [<ffffffff8170ecc5>] panic+0xc8/0x1d7
 [<ffffffff8106a391>] do_exit+0xa41/0xa50
 [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
 [<ffffffff8106a41f>] do_group_exit+0x3f/0xa0
 [<ffffffff8106a494>] SyS_exit_group+0x14/0x20
 [<ffffffff817266bf>] tracesys+0xe1/0xe6
------------[ cut here ]------------
WARNING: CPU: 4 PID: 1 at /build/buildd/linux-3.13.0/arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
Modules linked in: nfs lockd sunrpc fscache

Okay, a kernel panic.  I did upgrade the kernel from the old Ubuntu 12.04 version.  In fact, the old live OS had a hand-built Gentoo kernel for arcane reasons.  So maybe...

The Red Herring

Well, I did install the Ubuntu 14.04 ".deb" package for the Fusion I/O Drive's "iomemory-vsl" kernel module.  Some of our equipment have I/O Drives, so the live OS needs to be able to talk to them.

The VMs (which don't have I/O Drives) worked fine, and the physical hardware (which do) did not work.

I assumed that there was some mismatch between Fusion I/O's kernel module and the Ubuntu 14.04 kernel.  I compiled the driver (using Fusion I/O's guide) and re-made my live OS.

Same result: kernel panic.

In addition, it turned out that physical hardware that did not have an I/O Drive failed, as well.

The Hint

I took out my phone and recorded the boot process at 120 frames per second to see the text before the kernel panic.  The messages that I'd been able to see were not helpful, and when the kernel panics, I lose the ability to scroll up in the history.  And since the system never gets up all the way, there's no way to access the logs.

Here's what I saw:
systemd-udevd[329]: starting version 204
Begin: Loading essential drivers ... done
Begin: Running /scripts/init-premount ... done
Begin: Mounting root file system ... Begin: Running /scripts/nfs-top ... done
FS-Cache: Loaded
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
FS-Cache: Netfs 'nfs' registered for caching
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
/init: .: line 252: can't open '/run/net-*.conf'
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200

The short story here is that the "ipconfig" message repeating over and over means that the kernel can't find any network interfaces (at least other than "lo").

The long story is that when booting from network (NFS), the "initrd" init script is going to run some scripts to set up NFS.  One of these calls a function "configure_networking" (since NFS won't work without networking).  That function tries to figure out which device to use for networking.

For a box with multiple interfaces, this could be a bit tricky.  However, if you've toggled on bit b10 (0x2) in your PXE setup using the "IPAPPEND" directive, then you'll get an environment variable called "BOOTIF" that encodes the MAC address of the interface that originally PXE booted.  The "configure_networking" function then looks through the /sys/class/net/* files to see which interface has that MAC address.  From there, it will know the device and can set it up.

However, "ipconfig" claims that there are "no devices to configure".  Also, if you debug the "configure_networking" function, you'll see that there's only one file in /sys/class/net, and that's "lo".  So, the system has a loopback interface, but none of the Ethernet interfaces that I expected.

The Problem

Very simply put: the appropriate Ethernet driver (kernel module) was not present in the "initrd" file.

The "initrd" file (short for "initial RAM disk") contains some bootstrap scripts and kernel modules.  For example, for PXE booting, we get the kernel and the "initrd" file from TFTP (or HTTP) from the Ethernet card itself as part of the process.  Thus, it is up to the "initrd" file to get networking up and running so that it can mount the root filesystem over NFS.

I jumped through hoops trying to figure out how to get the "initrd" file to have enough (or the right) modules.  I eventually settled on bucking the trends set in the "Diskless Ubuntu" and "Diskless Debian" guides.

In my /etc/initramfs-tools/update-initramfs.conf, I set:
MODULES=most
instead of:
MODULES=netboot

The "MODULES" variable tells "update-initramfs" (and "mkinitramfs") which modules to include in the "initrd" file.  Obviously, less modules means less space.  So setting "MODULES" to "netboot" is a special setting that tries to pull in the bare minimum to get networking working.  I figured whatever; more modules is better than less modules, especially since the module for my Ethernet card isn't making it in there.

However, that still didn't help.

(Note: I haven't experimented with "MODULES=netboot" again; the "initrd" file that gets generated is under 20MB, and it takes under 1 second to transfer that on my network, so I'm not too interested in trimming that file down to the bare essentials at this point.)

My Ethernet card was some kind of Broadcom NetXtreme card, and the Internet seemed to think that the appropriate kernel module for it is "tg3".

After banging my head for way too long, I realized that "tg3" was not present in either the "initrd" file or the "/lib/modules/" directory.  I didn't realize that this was a problem because the Ubuntu 12.04 version of the system had compiled-in drivers (remember, I said that it was a Gentoo-built kernel); they were not built as modules (".ko" files).  So all of my grepping and comparing of files and directories seemed to tell me that a "tg3.ko" file was not necessary or relevant.

Once I realized that I needed the "tg3" module (and that it was compiled into the older kernel), I had to find out where to get it from.

The Solution

It turns out that there are two Linux kernel packages in Ubuntu:
  1. linux-image; and
  2. linux-image-extra.
The "linux-image-extra" package was key.  This package dropped in a lot of modules, including "tg3.ko".

So, in addition to installing "linux-image", I installed "linux-image-extra".  When I ran "update-initramfs", all of the modules got copied into the "initrd" file.  So when I booted the box, it successfully found my interfaces (for example, "eth2"), mounted the root NFS filesystem, and continued along its merry way.

FS-Cache: Netfs 'nfs' registered for caching
[...]
IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[...]
IP-Config: eth2 hardware address b0:83:XX:XX:XX:XX mtu 1500 DHCP RARP
IP-Config: no response after 2 secs - giving up
tg3 0000:01:00.0 eth2: Link is up at 1000 Mbps, full duplex
tg3 0000:01:00.0 eth2: Flow control is on for TX and on for RX
tg3 0000:01:00.0 eth2: EEE is disabled
IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
IP-Config: eth2 hardware address b0:83:XX:XX:XX:XX mtu 1500 DHCP RARP
IP-Config: no response after 3 secs - giving up
IP-Config: eth2 hardware address b0:83:XX:XX:XX:XX mtu 1500 DHCP RARP
IP-Config: eth2 guessed broadcast address 10.XXX.XX.XXX
IP-Config: eth2 complete (dhcp from 192.168.XX.XX):
 address: 10.XXX.XX.XXX    broadcast: 10.XXX.XX.XXX    netmask: 255.255.252.0
 gateway: 10.XXX.XX.X      dns0     : 192.168.XX.XX    dns1   : 10.XXX.X.XX
 domain : XXX.XXXXXX.com
 rootserver: 10.XXX.XXX.XX rootpath:
 filename: pxelinux.0
Begin: Running /scripts/nfs-premount ... done
Begin: Running /scripts/nfs-bottom ... done
done
Begin: Running /scripts/init-bottom ... done

The Lesson

Well, what did I learn here?
  1. "ipconfig: no devices to configure" means that the kernel doesn't think that you have any Ethernet devices (either there aren't any, or you're missing the driver for it).
  2. "linux-image-extra" has all the drivers that you probably want (it certainly has "tg3").
  3. When I see "Running /scripts/nfs-premount" messages, that file lives in the "initrd" file.  It is totally possible to take apart an "initrd" file (with "cpio"), make changes to the scripts, and then put it back together (again with "cpio").
  4. If it seems impossible that your I/O Drive kernel module is causing networking-related kernel panics, then you're probably right and you have a networking issue somewhere.
  5. Not being able to set up NFS in a network-booting setup will result in a "kernel panic".  This surprised me; I figured that I'd get to an emergency console or something.  Kernel panics usually point to deep, scary problems, not something as simple as not getting an IP address.

Monday, September 14, 2015

Fixing the default Ubuntu snmpd configuration

SNMP is super helpful for performance and health monitoring of any production equipment.  It's lightweight, easy to understand, and very resilient when Bad Things happen to the network.  If you're not monitoring your production equipment with SNMP, then probably should look into that right away (we use SevOne NMS at work).

Getting "snmpd", the Linux SNMP daemon, up and running on Ubuntu is simply a matter of installing "snmpd":
sudo apt-get install snmpd;

Or is it?

Default configuration woes

Logging

By default, Ubuntu wants to log literally everything that "snmpd" does to syslog.  While I love the enthusiasm, this quickly leads to overflowing logs and the headache around them (plus it makes it impossible to find any event that's actually important).

How many times do you want to see messages like this in your logs?
Sep 11 16:48:23 your-server snmpd[19552]: Connection from UDP: [192.168.59.101]:49867->[10.129.11.219]
Sep 11 16:48:23 snmpd[19552]: last message repeated 199 times


The logging options are specified on the "snmpd" command line, and are thus configured in "/etc/default/snmpd".

The default logging settings are:
-Lsd

"-L" is for the logging options.  "s" is for syslog.  "d" is for the daemon facility.

What we want are these settings:
-LS 4 d

"-L" again is for logging options.  Capital "S" is for a priority-filtered syslog, with "4" being "warning-level or higher".  Again, "d" is for the daemon facility.

Port access

By default, Ubuntu locks down SNMP access to "localhost", so it's 100% useless from a monitoring perspective.  While I respect the security-mindedness displayed here, I need my boxes to actually respond to requests.

The access options are specified in the "snmpd.conf" file, which is located here: "/etc/snmp/snmpd.conf".

At the top of the file, there is a configuration item called "agentAddress".  By default, this limits requests to those originating locally.
agentAddress udp:127.0.0.1:161

There is usually a line following it that's commented out, and that's the one that we want.  Get rid of the line above and make sure that this one is enabled:
agentAddress udp:161,udp6:[::1]:161

This makes sure that any requests to port 161 (the standard SNMP port) will be allowed.

Permissions

Yes, yes, we should all be using SNMPv3's great user-based access-control mechanism, but for an internal-to-the-company server that can't be reached from the Internet, we can often afford to be lax.  And hey, I'm not stopping you from setting up SNMPv3 access control.  Go nuts.

Here, we're going to allow the community string of "public" to access everything about the box (but not make any changes at all).

The default configuration allows "public" to see some basic system information, but that's not good enough:
rocommunity public default -V systemonly

Get rid of that line and replace it with one that doesn't have the "systemonly" restriction:
rocommunity public

Restart "snmpd" and you'll be ready to respond to SNMP requests from your local management station.
sudo service snmpd restart;

Sunday, September 13, 2015

Beware the AppScale firewall

If you haven't already looked into AppScale for your company's internal application needs, then you may want to spend some time looking into it.  In short, it's an open-source implementation of Google App Engine that can run on a "private" cloud.  Why is this cool?  Well, it lets me use all of the power and convenience of Google App Engine apps without having to put my app in the public cloud (high latency and billing), instead letting me use all the resources that I want in my company data center.

tl;dr: AppScale runs "iptables" on its own, so if you want to run an additional service (such as SNMP) on a node, then you'll have to configure AppScale to allow it.

My goal: SNMP monitoring

At work, I'm setting up an AppScale cluster to serve some internal applications.  The first thing that any server needs to do, once online, is provide performance statistics (via SNMP) to our performance management tool (in our case, we use SevOne NMS).

Typically, this is the world's easiest task:
  1. Install "snmpd" (apt-get install snmpd).
  2. Allow "snmpd" to respond to remote requests (duh).
  3. Fix Ubuntu's terribly verbose SNMP logging defaults.
Unfortunately, I fought with this for an hour because, no matter what I did, "snmpd" would not respond to any requests from my management tool, and nothing in the logs said why.  For reasons not perfectly clear to me, AppScale (for Ubuntu) runs on Ubuntu 12.04, so I thought that maybe there was some ancient security measure in place that I had forgotten about over the years.

I eventually stumbled on "iptables" as a culprit (it's never first on my list, but probably should be).  I ran "iptables -L -n" to list the current "iptables" rules, and sure enough, the system had some:
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:1080
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:1443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2812
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5222
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5555
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:6106
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8100
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:4380:4400
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:4343
ACCEPT     all  --  10.129.11.219        0.0.0.0/0          
ACCEPT     all  --  10.129.11.221        0.0.0.0/0          
DROP       all  --  0.0.0.0/0            0.0.0.0/0          

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

However, no amount of "iptables" magic would allow me to get the system to respond to SNMP requests.  I'd add a rule, and it might respond for a few seconds, but after that, my SNMP requests would time out again.  My rule?  Gone.

The AppScale firewall

It turns out that AppScale maintains the "iptables" setup for the box, and any change that you make will quickly be reverted by it.  This doesn't, in principle, bother me except that it's not really documented anywhere.  The only real mention of it is the Performance Tuning document, and even then, it's just a quick mention in order to get HAProxy stats from the box.

The AppScale firewall configuration lives in "appscale/firewall.conf" (the default installation guide had me put the "appscale" directory in "/root", so the file was located in "/root/appscale/firewall.conf" for me).  Once I saw what was going on, it was simply a matter of making a quick change to the file and waiting a few seconds (AppScale periodically re-reads the file and makes any changes live).

To tell AppScale to allow SNMP requests, I simply had to add the following line after the other "iptables -A" lines:
iptables -A INPUT -p udp -m udp --dport 161 -j ACCEPT # SNMP

Problem solved.

Additional resources

  1. The current default version of AppScale's "firewall.conf" can be found here.