Mahoney threading info

From reSIProcate
Jump to: navigation, search

This is a rough draft of threading information that I'm updating. If you found this page through the Search function, note that it's not quite ready for prime time. Don't link to it, but if you have any feedback, let me know. Thanks! --Jean 11:16, 1 Dec 2005 (PST)


Threading Model[edit]

ReSIProcate can run in one thread or separate threads. There is no boundary between DUM and the layer below, the layer above handles the threads.

Single Threaded Mode

  • One thread for the entire application
  • File descriptors are managed at the application level

Multi-Threaded Mode

  • Optional StackThread
  • DUM + DUM client must be one thread or provide sync

Note: The thread per transport mode is no longer supported. In the original design, each transport instance could run in its own thread. This capability was removed since it did not seem to increase performance and introduced other design issues. The stack itself cannot be run in multiple threads.


FIFOs[edit]

FIFOs (first in, first out), also known as queues, are a standard thread synchronization/buffering mechanism. FIFOs are the main way to move information between threads within reSIProcate. Since SIP is a message/event protocol, event FIFOs are a natural and error resistant mechanism for communicating among threads.

There is a FIFO between every layer of reSIProcate [*** resip/stack/doc/FifoFlow.pdf - outdated?? Yes, don't quite know how to update]:

  • between the Transaction State Machine and the application (TU FIFO)
  • between the timers and the transaction state machine (transaction FIFO)
  • between the transaction state machine and the transports (one or more transport FIFOs - outdated? yes)
  • between the transport and the network (kernel TX FIFO)
  • between the network and the transports (kernel RX FIFO)

The actual FIFO is a template class (abstractFifo.hxx). The getNext method will wait until there is something in the FIFO.

There are a few appearances of direct use of mutex; they should be viewed with suspicion and added to reluctantly. For example, the logger (appropriately) protects an external resource, std::cerr with a Mutex. For more information on mutexes, see [*** add link]


Creating Threads[edit]

The ThreadIf base class is a wrapper to create and spawn a thread. Provides virtual interface. To use ThreadIf, derive from it and override the virtual thread() method. You should define thread() such that it returns when isShutdown() is true.

To start the thread, call the run() method. The code in thread() will run in a separate thread.

Call shutdown() from the constructing thread to shut down the code. This will set the bool shutdown_ to true. The code in thread() should react properly to shutdown_ being set by returning. Call join() to join the code.

Sample:

  ...
  DerivedThreadIf thread;
  thread.run(); 
   ... do stuff ...
  thread.shutdown();
  thread.join();


Event Loop for Single-Threaded Implementations[edit]

These series of function calls are required in order to allow the stack to run entirely in a single thread:

  • build a file descriptor set of the transports, connections, and DNS provider by calling SipStack::buildFdSet. The application may need to add its own file descriptors as well.
  • find out when process must be called by calling FdSet::select.The application may need to add its own timers as well.
  • give processing time to stack components by calling SipStack::process.
  while (!dumShutDown)
  {
     FdSet fdset;
     stack->buildFdSet(fdset);
     int err = fdset.selectMilliSeconds(stack->getTimeTillNextProcessMS());
     assert ( err != -1 );
     stack->process(fdset);
     while(dumUas->process());
  }

SipStack::buildFdSet[edit]

The buildFdSet method iterates through the transports and other components (ie. DNS resolver) and adds file descriptors to a File Descriptor set or FdSet.

  • Anything in the stack or the application that uses a file descriptor needs to add itself to the FdSet.
  • This should start with an empty FdSet
  • Calling buildFdSet() will add the file descriptors for all of the transports, connections, and the DNS provider
  • Next, add any application-specific file descriptors

StackThread::getTimeTillNextProcess[edit]

The getTimeTillNextProcess method determines the minimum time until any timer in the application or stack will fire. If anything in the stack needs to process immediately, short circuit and return 0. (Currently, the DNS provider short circuits with a value of 50ms?)

FdSet::select[edit]

[*** found in rutil/Socket - header file is not named after the class] The file descriptors in the FdSet all default to a signaled state when added by the buildFdSet method. Calling select will cause the file descriptors to flip to the unsignaled state and the running thread to block until one of the file descriptors is signaled. Usually this means one of the following:

  • there is new data on the wire (ie. a SIP message or DNS result)
  • a socket is ready to be written to
  • a timer has fired

When select completes, the FdSet will be modified and those file descriptors that are ready are set to the signaled state. If a timeout occurs, then no file descriptors in the set will be signaled.

  • Call select() either in the application or in a StackThread [*** selectMS()] and pass in the two things constructed above [*** select() doesn't take those two things above. Matthias says to use selectMilliSeconds because you don't want to deal with a timeval.]
  • Select will block until something appears on the socket.
  • The select() call needs a timeout value that the stack can also determine.


SipStack::process[edit]

This call gives the stack cycles to run. The stack will iterate through the transports and other components giving them cycles to do work. After select returns, pass the now selected FdSet to SipStack::process. This work includes processes such as the following:

  • reading a message from the wire (SIP Message or DNS result)
  • writing a message to the wire or processing a timeout.
  • asking the StatisticsManager to generate its stats
  • giving the DNS provider / cache cycles
  • processing application and transaction timers
  • handling outstanding events affecting TransactionState
  • handling TU shutdown related events

Warning: Ensure that the FdSet you pass in has been generated by a call to buildFdSet and that is has been used in a select call. If you fail to call select, then the process call will think all file descriptors passed in are signaled, and this can cause unpredicable behavior.

Event Loop for Multi-Threaded Implementations[edit]

The stack can run in a separate thread from the TU (or application). The StackThread class can be used to take care of this for you. The stackThread class is an example of an event loop. Has shutdown been called on the thread? If not … blocking call. Can ask when the next timer will go off. Max sleep time of 24 milliseconds. The interruptableStackThread class is for clients. It will sleep for as long as it can. [*** What's the thinking behind the two stackthread classes?] [StackThread does FdSet, Select, Process stuff] [ReSIP diagrams - 18 and ReSIP diagrams - Main Event Loop shows some of this.]

  SipStack stack;
  StackThread stackThread(stack); //timers are managed by stackThread
  stackThread.run()  //run in its own thread (?)

[*** Robert's not sure the following looks good. Use code snippet for sending OPTIONs]

  While(usleep(10)) {
     SipMessage* msg = stack.receive(); //blocking call, get a message out of the FIFO on stack. Will give a message that is now the application’s memory. You have to clean it up. 
     if (msg) {
        //process the incoming sip message
        delete msg;
      }
   }


Ensuring Thread Safety in Multi-threaded Implementations[edit]

reSIProcate is designed to be a threadsafe library. To ensure thread safety, use the threadsafe FIFO classes for inter-thread communication and data sharing (see the FIFO documentation for details [*** Add link]). If you want to add functionality to the stack, and you want to share data between threads, *please* investigate the existing FIFOs before adding new ones.

The exceptions, as of revision 4878 of /main, are described below.

The multi-thread data access and communication classes are the following:

  • Mutex: a semaphore that can be locked by only one thread at a time. Serializes access to a data member (with Lock).
  • Condition: a condition variable that can be signaled or waited on. Used for scheduling.
  • Lock: a convenience class to lock a Lockable object (such as a Mutex) on construction, and unlock it on destruction. Note that Lock is exception safe -- in general, do not use Mutex lock/unlock directly.

Note: Although the classes ReadLock and WriteLock are defined in Lock, they are no longer used (as of 4878).

As of revision 4878, here are the uses of these classes outside the FIFO classes (external applications such as DUM are not included):

rutil/HeapInstanceCounter.cxx

allocationMutex serializes access to the AllocationMap (using Lock, as are all of the following).

rutil/Log.cxx

_mutex (poorly named) serializes access to the logging level.

rutil/ThreadIf.cxx

mShutdownMutex serializes access to the mShutdown member variable of the class.
A Condition called mShutdownCondition is signaled when mShutdown is set. This is for the implementation of the waitForShutdown(int ms) function.

stack/SipStack.cxx

shutDownMutex serializes access to mShuttingDown.
mAppTimerMutex serializes access to mAppTimers.

stack/StatisticsMessage.cxx

mMutex (poorly named) serializes access to the statistics data in the message in the functions 'loadIn' and 'loadOut'. These are only used within the StatisticsManager.

stack/TimeAccumulate.cxx

mMutex (poorly named) serializes access to the time and the count.


These classes are useful abstractions of multi-thread data access and data sharing concepts to support system independence between Win32 and POSIX based systems. If you want your application or reSIProcate addition to use these abstractions, please look at the doxygen output for Mutex, Condition, and Lock for more information.

DUM Threading Options[edit]

The Dialog Usage Manager (DUM) supports a few threading options, which are summarized below.

NOTE: There is no threading protection in the DUM methods themselves. If your application is multi-threaded, you MUST provide your own protection (mutexing) if you plan on calling dum->process and/or other DUM APIs from different threads.


Single Threaded[edit]

Your application, DUM, the stack, and transports all run in the same thread. This is how the BasicCall sample works. A process loop is required:

  while (!dumShutDown)
  {
     FdSet fdset;
     stack->buildFdSet(fdset);
     int err = fdset.selectMilliSeconds(stack->getTimeTillNextProcessMS());
     assert ( err != -1 );
     stack->process(fdset);
     while(dumUas->process());
  }

More information can be found here [*** add link for info below]

Separate Stack Thread[edit]

You can have DUM and a proxy in the same app and they would share the same stack. Potentially make stack thread object if you have multiple.Your application and DUM run in the same thread, the stack and transports run in another thread. To run in this mode, you must use or emulate the functionality in StackThread.cxx to start and run the stack thread. Then you must periodically call the dum->process() method from your application so that DUM can peform its processing. Note: DUM is not part of the core stack, it sits on top.


Separate DUM and/or Stack Threads[edit]

If you are SUA only, run DUM in its own thread (a dum thread), if it's event-driven. Can use an interruptable thread. A class called DumThread is in the works that will allow you to run all of reSIProcate in its own thread (or threads) and your application in another thread. You can use StackThread and DumThread together to have the stack and DUM run in separate threads. In this scenario your application runs in its own thread. [*** DumThread exists but isn't commented.]


This diagram [1] shows the reSIProcate Transaction and Transport Architecture. Common colours indicate related, thread connected components. [**** But it doesn't]