Fixing the default Ubuntu snmpd configuration

SNMP is super helpful for performance and health monitoring of any production equipment.  It's lightweight, easy to understand, and very resilient when Bad Things happen to the network.  If you're not monitoring your production equipment with SNMP, then probably should look into that right away (we use SevOne NMS at work).

Getting "snmpd", the Linux SNMP daemon, up and running on Ubuntu is simply a matter of installing "snmpd":
sudo apt-get install snmpd;

Or is it?

Default configuration woes

Logging

By default, Ubuntu wants to log literally everything that "snmpd" does to syslog.  While I love the enthusiasm, this quickly leads to overflowing logs and the headache around them (plus it makes it impossible to find any event that's actually important).

How many times do you want to see messages like this in your logs?

Sep 11 16:48:23 your-server snmpd[19552]: Connection from UDP: [192.168.59.101]:49867->[10.129.11.219]
Sep 11 16:48:23 snmpd[19552]: last message repeated 199 times

The logging options are specified on the "snmpd" command line, and are thus configured in "/etc/default/snmpd".

The default logging settings are:

-Lsd

"-L" is for the logging options.  "s" is for syslog.  "d" is for the daemon facility.

What we want are these settings:

-LS 4 d

"-L" again is for logging options.  Capital "S" is for a priority-filtered syslog, with "4" being "warning-level or higher".  Again, "d" is for the daemon facility.

Port access

By default, Ubuntu locks down SNMP access to "localhost", so it's 100% useless from a monitoring perspective.  While I respect the security-mindedness displayed here, I need my boxes to actually respond to requests.

The access options are specified in the "snmpd.conf" file, which is located here: "/etc/snmp/snmpd.conf".

At the top of the file, there is a configuration item called "agentAddress".  By default, this limits requests to those originating locally.

agentAddress udp:127.0.0.1:161

There is usually a line following it that's commented out, and that's the one that we want.  Get rid of the line above and make sure that this one is enabled:

agentAddress udp:161,udp6:[::1]:161

This makes sure that any requests to port 161 (the standard SNMP port) will be allowed.

Permissions

Yes, yes, we should all be using SNMPv3's great user-based access-control mechanism, but for an internal-to-the-company server that can't be reached from the Internet, we can often afford to be lax.  And hey, I'm not stopping you from setting up SNMPv3 access control.  Go nuts.

Here, we're going to allow the community string of "public" to access everything about the box (but not make any changes at all).

The default configuration allows "public" to see some basic system information, but that's not good enough:

rocommunity public default -V systemonly

Get rid of that line and replace it with one that doesn't have the "systemonly" restriction:

rocommunity public

Restart "snmpd" and you'll be ready to respond to SNMP requests from your local management station.
sudo service snmpd restart;

Beware the AppScale firewall

If you haven't already looked into AppScale for your company's internal application needs, then you may want to spend some time looking into it.  In short, it's an open-source implementation of Google App Engine that can run on a "private" cloud.  Why is this cool?  Well, it lets me use all of the power and convenience of Google App Engine apps without having to put my app in the public cloud (high latency and billing), instead letting me use all the resources that I want in my company data center.

tl;dr: AppScale runs "iptables" on its own, so if you want to run an additional service (such as SNMP) on a node, then you'll have to configure AppScale to allow it.

My goal: SNMP monitoring

At work, I'm setting up an AppScale cluster to serve some internal applications.  The first thing that any server needs to do, once online, is provide performance statistics (via SNMP) to our performance management tool (in our case, we use SevOne NMS).

Typically, this is the world's easiest task:

1. Install "snmpd" (apt-get install snmpd).
2. Allow "snmpd" to respond to remote requests (duh).
3. Fix Ubuntu's terribly verbose SNMP logging defaults.

Unfortunately, I fought with this for an hour because, no matter what I did, "snmpd" would not respond to any requests from my management tool, and nothing in the logs said why.  For reasons not perfectly clear to me, AppScale (for Ubuntu) runs on Ubuntu 12.04, so I thought that maybe there was some ancient security measure in place that I had forgotten about over the years.

I eventually stumbled on "iptables" as a culprit (it's never first on my list, but probably should be).  I ran "iptables -L -n" to list the current "iptables" rules, and sure enough, the system had some:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:1080
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:1443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2812
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5222
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5555
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:6106
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8100
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:4380:4400
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:4343
ACCEPT     all  --  10.129.11.219        0.0.0.0/0          
ACCEPT     all  --  10.129.11.221        0.0.0.0/0          
DROP       all  --  0.0.0.0/0            0.0.0.0/0          

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

However, no amount of "iptables" magic would allow me to get the system to respond to SNMP requests.  I'd add a rule, and it might respond for a few seconds, but after that, my SNMP requests would time out again.  My rule?  Gone.

The AppScale firewall

It turns out that AppScale maintains the "iptables" setup for the box, and any change that you make will quickly be reverted by it.  This doesn't, in principle, bother me except that it's not really documented anywhere.  The only real mention of it is the Performance Tuning document, and even then, it's just a quick mention in order to get HAProxy stats from the box.

The AppScale firewall configuration lives in "appscale/firewall.conf" (the default installation guide had me put the "appscale" directory in "/root", so the file was located in "/root/appscale/firewall.conf" for me).  Once I saw what was going on, it was simply a matter of making a quick change to the file and waiting a few seconds (AppScale periodically re-reads the file and makes any changes live).

To tell AppScale to allow SNMP requests, I simply had to add the following line after the other "iptables -A" lines:

iptables -A INPUT -p udp -m udp --dport 161 -j ACCEPT # SNMP

Problem solved.

Additional resources

1. The current default version of AppScale's "firewall.conf" can be found here.

Net-SNMP and read_all_mibs

For better or for worse, Net-SNMP is the SNMP implementation for use with C/C++ (and Linux).  My feelings on Net-SNMP are a topic for another time, however.

Here, we'll be focused on using the "read_all_mibs" function to load the MIBs that have been deployed and how to interpret the result of that function.

Our major use cases revolve around:

• Finding information about an OID (known by its numeric form); and
• Showing the contents of the OID tree (from some arbitrary location).

Initializing Net-SNMP

If your MIBs are in a non-standard directory, or if you just want to take charge and tell Net-SNMP what to do, you'll need to specify the MIB directory before calling "init_snmp".  For example:

netsnmp_set_mib_directory( "/usr/local/snmp/mibs" );

Once you've done that, your next step is to initialize Net-SNMP.  You'll need to give it some arbitrary text (which is used for logging purposes, I believe).  This should normally be the name of your application or something otherwise unique.

init_snmp( "sense-codons" );

"init_snmp" will then load all of the MIBs in the directory that you specified, and it will spew errors all over the screen if your MIBs are in any way not perfect.  And trust me, if you put some proprietary MIBs in that directory, you'll see what I'm talking about.  The point is that the "init_snmp" function might take a minute or two to run; once that function returns, you can rest assured that the MIBs have been loaded and, as best as it could, Net-SNMP has organized them into an OID tree for you to use.

Root nodes

The top-level structure of the OID tree looks like this:

• ccitt(0)
• iso(1)
• joint-iso-ccitt(2)

This much will be understood by Net-SNMP even if there are no MIBs present.

Since this is understood to be a tree, I would have expected there to be an uppermost, root node in the tree, but that's not the case with Net-SNMP.  Rather, all of the top-level items are the roots of their own trees, and there is no root node.

Getting a root node

To obtain the first (that is, lowest-numbered) root node (remember, there are multiple root nodes), you call "read_all_mibs".  This returns a "struct tree*", which has the following useful fields:

• label, a "char*" that is the name of the node.
• subid, an unsigned integer that is the numeric ID of the node.
• next_peer, a "struct tree*" pointer to the next sibling of the node.  This will be NULL if there is no sibling node.
• child_list, a "struct tree*" pointer to the first child of the node.  This will be NULL if there is no child node.

With a basic set of MIBs, you should get back a node with subid=0, label=ccitt and a sibling of subid=1, label=iso.

Trusting node IDs

This whole setup seems pretty straightforward; you'd think that the usual tree-navigation strategies would work.  However, there is just one more item that I need to mention: node IDs are not technically unique.

Due to the magic and wonder of SNMP, it often is the case that two manufacturers (or even one manufacturer at two times) will redefine a node.  That is, the same node ID will have different names ("label" in the structure).

As far as I can tell, Net-SNMP will always store both names; it will choose one of them to be authoritative, and the others will be more or less shortcuts to the "real" name.  These are stored in the tree as siblings.

So, if you are looking for node ".0.0.8.341.1", under node ".0.0.8.341" there might be multiple "1" nodes.  Once you find the first subid=1 node, you should keep iterating through all of the "next_peer" nodes that have subid=1.  The last such node is the node that you actually one.

For example, assume that you've found the first such node; we'll call it "node".  This little one-line for-loop will update "node" to the last node with that node ID.  This will work even if it is the only such node.

int currentSubid = node->subid;

for( ; node && node->next_peer && node->next_peer->subid == currentSubid; node = node->next_peer );