SipStack Event Loop
[draft started 2/28/2011 kw]
The SIP stack runs as one monolithic thread. That is, socket IO, transport layer management, message parsing, message encoding, and the transaction state machine all run in one thread. A particular application can choose to run higher level code (e.g., dialog layer, dialog users, and application) in the same thread as the SIP stack, or give the SIP stack its own thread.
footnote: Previously (when?) the stack supported running each transport in its own thread, but this was removed.
The SipStack object itself it threadsafe: upper layer code can call it from "other" threads (e.g., to send a message). The SipStack will queue the message (using thread-safe queue) and then request that its "own" thread wakeup. It does this using an AsyncProcessHandler.
Currently, there are two event loop "styles" that can be used by an owning thread to provide cycles to the SIP stack. The original style (denoted "FdSet Style" below) was built around select(). It is flexible and works well for (small) clients, but doesn't scale well to servers with many open sockets. The newer style (denoted "Callback Style" below) was added in late 2010 to support servers with 100k+ open sockets.
The Callback style has multiple backing implementations available. There is an implementation based on select() that is available on all platforms. Early performance results indicate that this implementation has comparable performance to the older FdSet Style.
The stack obtains cycles in 3 ways:
- By registering socket IO callbacks with an FdPollGrp object. See the pollGrp argument to SipStack constructor and discussion below.
- Via the SipStack methods:
- Via the asynchronous process handler. See "handler" argument to SipStack constructor and discussion above.
Generally, an instance of EventStackThread is used to provide a private thread to the stack.
In this style, socket IO is handled via callbacks, but timers are still handled synchronously. At some point it might make sense to convert the timers to callbacks as well. This remains to be investigated.
This class (in rutil) manage the socket IO callbacks for the stack. The class is misnamed, it should really be something like "SocketEventGrp" or such, but has the old name for legacy reasons. The FdPollGrp is not complete (it has pure virtual methods) and must be backed by an implementation object.
An implementation object is created via the static factory method:
FdPollGrp::create(const char *implName)
where implName is:
- NULL, empty string or "event". Will create instance of "best" event loop that is available on platform.
- "epoll". Instance of FdPollImplEpoll below.
- "fdset". Instance of FdPollImpFdSet below.
This implementation uses the FdSet class, which is a light wrapper around the select() system call. As such, it is available on all platforms. Due to limitations of the select system call, no more than around 1000 sockets can be open, and performance degrades with more than a hundred or so sockets. It is suitable for use by clients, which typically have only a handful of sockets open.
This implementation uses the epoll() system call. The key advantage of this implementation is that it allows 100k+ open sockets. It is available on Linux and some other POSIX platforms; it is not available on Windows.
This implementation is compiled in via the HAVE_EPOLL preprocessor macro or the configure script option "--enable-epoll"; see Configuration_Options.
This doesn't exist yet; just and idea! It should be pretty easy to make an FdPollGrp implementation that is backed by libevent. It would need to be libevent version 2+; version 1.x of libevent has globals that would be difficult to hide. Probably this could be added without any further change to the resip/stack classes. This would provide access to all the platform-specific optimizations built into libevent.
The SipStack class has methods:
buildFdSet() getTimeTillNextProcessMS() process()
The "owning" stack thread must call the above functions in infinite loop to provide the stack with cycles. The application could either make the calls from its own process (and thus its own event loop), or create an instance of StackThread or InterruptableStackThread to provide an autonomous event loop.
Eventually (my opinion) this style (and specifically the buildFdSet and process() methods) should go away. This is because there is sigificant branching (if statements) in the stack code to handle both styles.