Beware the AppScale firewall

If you haven't already looked into AppScale for your company's internal application needs, then you may want to spend some time looking into it.  In short, it's an open-source implementation of Google App Engine that can run on a "private" cloud.  Why is this cool?  Well, it lets me use all of the power and convenience of Google App Engine apps without having to put my app in the public cloud (high latency and billing), instead letting me use all the resources that I want in my company data center.

tl;dr: AppScale runs "iptables" on its own, so if you want to run an additional service (such as SNMP) on a node, then you'll have to configure AppScale to allow it.

My goal: SNMP monitoring

At work, I'm setting up an AppScale cluster to serve some internal applications.  The first thing that any server needs to do, once online, is provide performance statistics (via SNMP) to our performance management tool (in our case, we use SevOne NMS).

Typically, this is the world's easiest task:

1. Install "snmpd" (apt-get install snmpd).
2. Allow "snmpd" to respond to remote requests (duh).
3. Fix Ubuntu's terribly verbose SNMP logging defaults.

Unfortunately, I fought with this for an hour because, no matter what I did, "snmpd" would not respond to any requests from my management tool, and nothing in the logs said why.  For reasons not perfectly clear to me, AppScale (for Ubuntu) runs on Ubuntu 12.04, so I thought that maybe there was some ancient security measure in place that I had forgotten about over the years.

I eventually stumbled on "iptables" as a culprit (it's never first on my list, but probably should be).  I ran "iptables -L -n" to list the current "iptables" rules, and sure enough, the system had some:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:1080
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:1443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2812
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5222
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:5555
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:6106
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:8080:8100
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpts:4380:4400
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:17443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:4343
ACCEPT     all  --  10.129.11.219        0.0.0.0/0          
ACCEPT     all  --  10.129.11.221        0.0.0.0/0          
DROP       all  --  0.0.0.0/0            0.0.0.0/0          

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

However, no amount of "iptables" magic would allow me to get the system to respond to SNMP requests.  I'd add a rule, and it might respond for a few seconds, but after that, my SNMP requests would time out again.  My rule?  Gone.

The AppScale firewall

It turns out that AppScale maintains the "iptables" setup for the box, and any change that you make will quickly be reverted by it.  This doesn't, in principle, bother me except that it's not really documented anywhere.  The only real mention of it is the Performance Tuning document, and even then, it's just a quick mention in order to get HAProxy stats from the box.

The AppScale firewall configuration lives in "appscale/firewall.conf" (the default installation guide had me put the "appscale" directory in "/root", so the file was located in "/root/appscale/firewall.conf" for me).  Once I saw what was going on, it was simply a matter of making a quick change to the file and waiting a few seconds (AppScale periodically re-reads the file and makes any changes live).

To tell AppScale to allow SNMP requests, I simply had to add the following line after the other "iptables -A" lines:

iptables -A INPUT -p udp -m udp --dport 161 -j ACCEPT # SNMP

Problem solved.

Additional resources

1. The current default version of AppScale's "firewall.conf" can be found here.