init, setrlimit, and ZeroMQ

Linux has the notion of "resource limits" per process.  This conceptually cool because you could cap potentially dangerous processes with regard to memory size, number of threads, real-time priority, and a bunch of other things (including number of open files, which we'll come to shortly).  Practically, this protects a box from crippling itself under the weight of a malicious (or simply buggy) process.

Resource limits come with two notions of limit: a "hard" limit and a "soft" limit.  The hard limit is the limit set by "the system" and acts as a true upper bound on the particular resource.  The soft limit is a limit that may be changed by the process--but only up until the hard limit.

This way, a process may be able to create a vast number of threads, but it must specifically increase its own limit in order to do so.  It's a tiny little check to make you really think about what you're doing, since you have to manually increase the resource limit within the process.

In the shell: ulimit

When in a shell, the "ulimit" command can be used to change these limits from that moment forward.  This is often used to allow any processes spawned by the shell to have a nonzero (usually infinite) sized core file, which can be used to debug a program that crashes.

In the code: setrlimit

In C/C++, you can use "setrlimit" to change the limits.

How hard is "hard"?

In addition to changing the soft limit, you can change the hard limit if you are root.  Let me say that one more time: you can change hard limits if you are root.

Why is this important?  When the "init" process starts, it sets the hard limit for the maximum number of files open at any one time to 1024.  I ran into this limit years ago, and I tried to change the soft limit, but to no avail.  I couldn't get my process (which must run out of "init") to set this limit higher than 1024, not even as root.  It wasn't until recently that I realized that I didn't try changing the hard limit as root.  I naively thought that a limit documented as "hard" would mean that nothing could change it.  Nope.  One quick call and that number was anything I wanted.

So, why was that important?  We've recently begun to use ZeroMQ in our code at work, and it's pretty cool.  Without getting into too many details, ZeroMQ is a a message-passing system that abstracts communication into "messages" that are sent and received from "sockets".  In theory these sockets have nothing to do with Linux sockets; they are magical mailboxes for magical postmen that deliver magical messages.  In practice, they are super clean wrappers around TCP, UDP, and Unix sockets.

What did I have that needed a thousand network connections?  Nothing.  That's where things got interesting.  ZeroMQ's documentation leads you to believe that you should give up on managing global task lists and mutexes and signaling in favor of its notion of a simple message queue and workers that pull from it.  It wants you to focus on the communication aspect of what you're doing rather than getting lost in mutex hell.  As icing on the cake, ZeroMQ provides "in-process" sockets that don't hit the networking stack at all and can only be used within a process.  Perfect for eliminating task lists and custom thread pool code.

What they don't tell you is that every single ZeroMQ "socket" internally uses the "socketpair" call to generate two real sockets for some internal signalling mechanism--in addition to any networking sockets that are needed.  So, if you really and truly adopted their paradigm, throwing in-process sockets everywhere to simplify all of your code, you would easily run out of file descriptors on a process that spawned from "init".

Another limit?

After allowing my process an unlimited number of open files, I found that ZeroMQ was throwing exceptions about "too many open files", even when I was only around 1,000 open files.  This made no sense at all, since my process limit was now over 65,000.  So what was going on?

It turns out that ZeroMQ internally has a maximum number of files that it wants to respect.  That's great, except that that limit can only be changed in "config.hpp", a random file in the source code of ZeroMQ.  In order to increase that limit to acceptable levels, I had to manually patch ZeroMQ's source code, recompile it, and redistribute it to my equipment.

Summary

• Resource limits place limits on how much a process may do.
• "root" can arbitrarily change these limits, including the unchangeable "hard" limits.
• "nofiles" is the limit for the maximum number of open files.  "no" here represents an abbreviation of the word "number" (for whatever reason that those four extra letters couldn't be trusted).
• To set the limit in C/C++, use "setrlimit" with RLIMIT_NOFILES.
• ZeroMQ creates two sockets for every ZeroMQ "socket" that it creates.
• "max_sockets" is ZeroMQ's own constant (defined in "config.hpp") for the number of sockets that it will allow itself to open.