freetdm: - Update ftmod_sangoma_isdn to use core state advance

- Added locking documentation (docs/locking.txt)
	 - Updated core state advance to use state processor in span
           rather than pushing the processor in the function arguments
This commit is contained in:
Moises Silva 2010-12-31 09:42:15 -05:00
parent 903d136481
commit 39d58e0452
7 changed files with 189 additions and 44 deletions

View File

@ -0,0 +1,125 @@
Last Updated: Fri 30 Dec, 2010
== Background ==
FreeTDM is a threaded library. As such, locking considerations must be taken when using it and when writing code inside the library.
At the moment locks are not exposed to users. This means API users cannot acquire a lock on a channel or span structure. There is no
need for users to lock channels or spans since all their interactions with those structures should be done thru the FreeTDM API which
can (and in most cases must) internally lock on their behalf.
Internally, locking can be done either by the core or the signaling modules. To better understand the locking considerations we must
understand first the threading model of FreeTDM.
== Threading Model ==
At startup, when the user calls ftdm_global_init(), just one timing thread is created to dispatch internal timers. If you write
a signaling module or any other code using the scheduling API, you can choose to run your schedule in this timing thread or in
a thread of your choice. This is the only thread launched at initialization.
If the application decides to use ftdm_global_configuration(), which reads freetdm.conf to create the spans and channels data
structures, then possibly another thread will be launched for CPU usage monitoring (only if enabled in the configuration cpu_monitor=yes
This thread sole purpose is to check the CPU and raise an alarm if reaches a configurable threshold, the alarm then is checked to avoid
placing or receiving further calls.
At this point FreeTDM has initialized and configured its channels input output configuration.
The user is then supposed to configure the signaling via ftdm_configure_span_signaling() and then start the signaling work
using ftdm_span_start(). This will typically launch at least 1 thread per span. Some signaling modules (actually just the analog one)
launches another thread per channel when receiving a call. The rest of the signaling modules currently launch only one thread per
span and the signaling for all channels within the span is handled in that thread. We call that thread 'the signaling thread'.
At this point the user can start placing calls using the FreeTDM call API ftdm_channel_call_place(). Any of the possible threads in
which the user calls the FreeTDM API is called 'the user thread', depending on the application thread model (the application using FreeTDM)
this user thread may be different each time or the same all the time, we cannot make any assumptions. In the case of FreeSWITCH, the most
common user of FreeTDM, the user thread is most of the cases a thread for each new call leg.
At this point we have identified 4 types of threads.
1. The timing thread (the core thread that triggers timers).
Its responsibility is simply check for timers that were scheduled and trigger them when the time comes. This means that if you decide
to use the scheduling API in freerun mode (where you use the core timer thread) you callbacks will be executed in this global thread
and you MUST not block at all since there might be other events waiting.
2. The CPU thread (we don't really care about this one as it does not interact with channels or spans).
3. The signaling thread.
There is one thread of this per span. This thread takes care of reading signaling specific messages from the network (ISDN network, etc) and
changing the channel call state accordingly and processing state changes caused by user API calls (like ftdm_channel_call_hangup for example).
4. The user thread.
This is a thread in which the user decides to execute FreeTDM APIs, in some cases it might even be the same than the signaling thread (because
most SIGEVENT notifications are delivered by the signaling thread, however we are advicing users to not use FreeTDM unsafe APIs from the
thread where they receive SIGEVENT notifications as some APIs may block for a few milliseconds, effectively blocking the whole signaling thread
that is servicing a span.
== Application Locking ==
Users of the FreeTDM API will typically have locking of their own for their own application-specific data structures (in the case of FreeSWITCH, the
session lock for example). Other application-specific locks may be involved.
== DeadLocks ==
As soon as we think of application locks, and we mix them with the FreeTDM internal locks, the possibility of deadlocks arise.
A typical deadlock scenario when 2 locks are involved is:
- User Thread - - Signaling Thread -
1. Application locks applock. 1. A network message is received for a channel.
2. Aplication invokes a FreeTDM call API (ie: ftdm_channel_call_hangup()). 2. The involved channel is locked.
3. The FreeTDM API attempts to acquire the channel lock and stalls because 3. The message processing results in a notification
the signaling thread just acquired it. to be delivered to the user via the callback function
provided for that purpose. The callback is then called.
4. The thread is now deadlocked because the signaling thread will never 4. The application callback attempts to acquire its application
release the channel lock. lock but deadlocks because the user thread already has it.
To avoid this signaling modules should not deliver signals to the user while holding the channel lock. An easy way to avoid this is
to not deliver signals while processing a state change, but rather defer them until the channel lock is released. Most new signaling modules
accomplish this by setting the span flag FTDM_SPAN_USE_SIGNALS_QUEUE, this flag tells the core to enqueue signals (ie FTDM_SIGEVENT_START)
when ftdm_span_send_signal() is called and not deliver them until ftdm_span_trigger_signals() is called, which is done by the signaling module
in its signaling thread when no channel lock is being held.
== State changes while locking ==
Only 2 types of threads should be performing state changes.
User threads.
The user thread is a random thread that was crated by the API user. We do not know what threading model users of FreeTDM will follow
and therefore cannot make assumptions about it. The user should be free to call FreeTDM APIs from any thread, except threads that
are under our control, like the signaling threads. Although it may work in most situations, is discouraged for users to try
to use FreeTDM APIs from the signaling thread, that is, the thread where the signaling callback provided during configuration
is called (the callback where FTDM_SIGEVENT_XXX signals are delivered).
A user thread may request state changes implicitly through calls to FreeTDM API's. The idea of state changes is internal to freetdm
and should not be exposed to users of the API (except for debugging purposes, like the ftdm_channel_get_state, ftdm_channel_get_state_str etc)
This is an example of the API's that implicitly request a state change.
ftdm_channel_call_answer()
Signaling modules should guarantee that upon releasing a lock on a channel, any state changes will be already processed and
not deferred to other threads, otherwise that leads to a situation where a state change requested by the signaling module is pending
to be serviced by another signaling module thread but a user thread wins the channel lock and attempts to perform a state change which will
fail because another state change is pending (and user threads are not meant to process signaling states).
ONLY one signaling thread per channel should try to perform state changes and processing of the states,
otherwise complexity arises and is not worth it!
At some point before we stablished this policies we could have 3 different threads doing state changes.
1. A user random thread could implcitly try to change the state in response to a call API.
2. The ftmod signaling thread could try to change the state in response to other state changes.
3. The lower level signaling stack threads could try to change the state in response to stack events.
As a result, lower level signaling stack thread could set a state and then let the signaling thread to
process it, but when unlocking the channel, the user thread may win the lock over the signaling thread and
may try to set a state change of its own and fail (due to the unprocessed state change)!
The rule is, the signaling module should never unlock a channel with states pending to process this way the user,
when acquiring a channel lock (inside ftdm_channel_call_answer for example) it will always find a consistent state
for the channel and not in the middle of state processing.

View File

@ -434,15 +434,14 @@ FT_DECLARE(char *) ftdm_channel_get_history_str(const ftdm_channel_t *fchan)
return stream.data;
}
FT_DECLARE(ftdm_status_t) ftdm_channel_advance_states(ftdm_channel_t *fchan, ftdm_channel_state_processor_t state_processor)
FT_DECLARE(ftdm_status_t) ftdm_channel_advance_states(ftdm_channel_t *fchan)
{
ftdm_channel_state_t state;
ftdm_channel_lock(fchan);
while (fchan->state_status == FTDM_STATE_STATUS_NEW) {
state = fchan->state;
ftdm_log_chan(fchan, FTDM_LOG_DEBUG, "Executing state processor for %s\n", ftdm_channel_state2str(fchan->state));
state_processor(fchan);
fchan->span->state_processor(fchan);
if (state == fchan->state && fchan->state_status == FTDM_STATE_STATUS_NEW) {
/* if the state did not change and is still NEW, the state status must go to PROCESSED
* otherwise we don't touch it since is a new state and the old state was
@ -451,7 +450,7 @@ FT_DECLARE(ftdm_status_t) ftdm_channel_advance_states(ftdm_channel_t *fchan, ftd
fchan->state_status = FTDM_STATE_STATUS_PROCESSED;
}
}
ftdm_channel_unlock(fchan);
return FTDM_SUCCESS;
}

View File

@ -620,7 +620,7 @@ static void ftdm_r2_on_call_init(openr2_chan_t *r2chan)
ftdm_sched_cancel_timer(r2data->sched, r2call->protocol_error_recovery_timer);
ftdm_log_chan_msg(ftdmchan, FTDM_LOG_DEBUG, "Cancelled protocol error recovery timer\n");
ftdm_set_state(ftdmchan, FTDM_CHANNEL_STATE_DOWN);
ftdm_channel_advance_states(ftdmchan, ftdm_r2_state_advance);
ftdm_channel_advance_states(ftdmchan);
}
}
@ -814,7 +814,7 @@ static void ftdm_r2_on_call_end(openr2_chan_t *r2chan)
ftdm_set_state(ftdmchan, FTDM_CHANNEL_STATE_DOWN);
/* in some circumstances openr2 can call on_call_init right after this, so let's advance the state right here */
ftdm_channel_advance_states(ftdmchan, ftdm_r2_state_advance);
ftdm_channel_advance_states(ftdmchan);
}
static void ftdm_r2_on_call_read(openr2_chan_t *r2chan, const unsigned char *buf, int buflen)
@ -846,7 +846,7 @@ static void ftdm_r2_recover_from_protocol_error(void *data)
goto done;
}
ftdm_set_state(ftdmchan, FTDM_CHANNEL_STATE_DOWN);
ftdm_channel_advance_states(ftdmchan, ftdm_r2_state_advance);
ftdm_channel_advance_states(ftdmchan);
done:
ftdm_channel_unlock(ftdmchan);
}
@ -1584,6 +1584,7 @@ static FIO_CONFIGURE_SPAN_SIGNALING_FUNCTION(ftdm_r2_configure_span_signaling)
span->set_channel_sig_status = ftdm_r2_set_channel_sig_status;
span->state_map = &r2_state_map;
span->state_processor = ftdm_r2_state_advance;
/* use signals queue */
ftdm_set_flag(span, FTDM_SPAN_USE_SIGNALS_QUEUE);
@ -1917,12 +1918,12 @@ static void *ftdm_r2_run(ftdm_thread_t *me, void *obj)
ftdm_clear_flag(ftdmchan, FTDM_CHANNEL_RX_DISABLED);
ftdm_clear_flag(ftdmchan, FTDM_CHANNEL_TX_DISABLED);
ftdm_channel_advance_states(ftdmchan, ftdm_r2_state_advance);
ftdm_channel_advance_states(ftdmchan);
r2chan = call->r2chan;
openr2_chan_process_signaling(r2chan);
ftdm_channel_advance_states(ftdmchan, ftdm_r2_state_advance);
ftdm_channel_advance_states(ftdmchan);
if (!call->accepted) {
/* if the call is not accepted we do not want users reading */

View File

@ -46,10 +46,9 @@ static ftdm_status_t ftdm_sangoma_isdn_stop(ftdm_span_t *span);
static ftdm_status_t ftdm_sangoma_isdn_start(ftdm_span_t *span);
ftdm_channel_t* ftdm_sangoma_isdn_process_event_states(ftdm_span_t *span, sngisdn_event_data_t *sngisdn_event);
static void ftdm_sangoma_isdn_advance_chan_states(ftdm_channel_t *ftdmchan);
static void ftdm_sangoma_isdn_poll_events(ftdm_span_t *span);
static void ftdm_sangoma_isdn_process_phy_events(ftdm_span_t *span, ftdm_oob_event_t event);
static void ftdm_sangoma_isdn_process_state_change(ftdm_channel_t *ftdmchan);
static ftdm_status_t ftdm_sangoma_isdn_process_state_change(ftdm_channel_t *ftdmchan);
static void ftdm_sangoma_isdn_process_stack_event (ftdm_span_t *span, sngisdn_event_data_t *sngisdn_event);
static void ftdm_sangoma_isdn_wakeup_phy(ftdm_channel_t *dchan);
static void ftdm_sangoma_isdn_dchan_set_queue_size(ftdm_channel_t *ftdmchan);
@ -270,13 +269,6 @@ ftdm_state_map_t sangoma_isdn_state_map = {
}
};
static __inline__ void ftdm_sangoma_isdn_advance_chan_states(ftdm_channel_t *ftdmchan)
{
while (ftdm_test_flag(ftdmchan, FTDM_CHANNEL_STATE_CHANGE)) {
ftdm_sangoma_isdn_process_state_change(ftdmchan);
}
}
static void ftdm_sangoma_isdn_process_phy_events(ftdm_span_t *span, ftdm_oob_event_t event)
{
sngisdn_span_data_t *signal_data = (sngisdn_span_data_t*) span->signal_data;
@ -457,7 +449,7 @@ static void *ftdm_sangoma_isdn_run(ftdm_thread_t *me, void *obj)
while ((ftdmchan = ftdm_queue_dequeue(span->pendingchans))) {
/* double check that this channel has a state change pending */
ftdm_channel_lock(ftdmchan);
ftdm_sangoma_isdn_advance_chan_states(ftdmchan);
ftdm_channel_advance_states(ftdmchan);
ftdm_channel_unlock(ftdmchan);
}
@ -470,11 +462,11 @@ static void *ftdm_sangoma_isdn_run(ftdm_thread_t *me, void *obj)
/* twiddle */
break;
case FTDM_FAIL:
ftdm_log(FTDM_LOG_ERROR,"%s:ftdm_interrupt_wait returned error!\n", span->name);
ftdm_log(FTDM_LOG_ERROR, "%s: ftdm_interrupt_wait returned error!\n", span->name);
break;
default:
ftdm_log(FTDM_LOG_ERROR,"%s:ftdm_interrupt_wait returned with unknown code\n", span->name);
ftdm_log(FTDM_LOG_ERROR, "%s: ftdm_interrupt_wait returned with unknown code\n", span->name);
break;
}
@ -536,7 +528,7 @@ ftdm_channel_t* ftdm_sangoma_isdn_process_event_states(ftdm_span_t *span, sngisd
break;
}
ftdm_channel_lock(ftdmchan);
ftdm_sangoma_isdn_advance_chan_states(ftdmchan);
ftdm_channel_advance_states(ftdmchan);
return ftdmchan;
}
@ -600,13 +592,14 @@ static void ftdm_sangoma_isdn_process_stack_event (ftdm_span_t *span, sngisdn_ev
sngisdn_process_rst_ind(sngisdn_event);
break;
}
if(ftdmchan != NULL) {
ftdm_sangoma_isdn_advance_chan_states(ftdmchan);
if (ftdmchan != NULL) {
ftdm_channel_advance_states(ftdmchan);
ftdm_channel_unlock(ftdmchan);
}
}
static void ftdm_sangoma_isdn_process_state_change(ftdm_channel_t *ftdmchan)
/* this function is called with the channel already locked by the core */
static ftdm_status_t ftdm_sangoma_isdn_process_state_change(ftdm_channel_t *ftdmchan)
{
ftdm_sigmsg_t sigev;
ftdm_channel_state_t initial_state;
@ -618,13 +611,12 @@ static void ftdm_sangoma_isdn_process_state_change(ftdm_channel_t *ftdmchan)
sigev.span_id = ftdmchan->span_id;
sigev.channel = ftdmchan;
/*first lock the channel*/
ftdm_channel_lock(ftdmchan);
/*clear the state change flag...since we might be setting a new state*/
ftdm_clear_flag(ftdmchan, FTDM_CHANNEL_STATE_CHANGE);
/* Acknowledge the state change */
ftdm_channel_complete_state(ftdmchan);
#ifdef FTDM_DEBUG_CHAN_MEMORY
if (ftdmchan->state == FTDM_CHANNEL_STATE_DIALING) {
ftdm_assert(mprotect(ftdmchan, sizeof(*ftdmchan), PROT_READ)==0, "Failed to mprotect");
ftdm_assert(mprotect(ftdmchan, sizeof(*ftdmchan), PROT_READ) == 0, "Failed to mprotect");
}
#endif
@ -879,11 +871,10 @@ static void ftdm_sangoma_isdn_process_state_change(ftdm_channel_t *ftdmchan)
}
#ifdef FTDM_DEBUG_CHAN_MEMORY
if (ftdmchan->state == FTDM_CHANNEL_STATE_DIALING) {
ftdm_assert(mprotect(ftdmchan, sizeof(*ftdmchan), PROT_READ|PROT_WRITE)==0, "Failed to mprotect");
ftdm_assert(mprotect(ftdmchan, sizeof(*ftdmchan), PROT_READ|PROT_WRITE) == 0, "Failed to mprotect");
}
#endif
ftdm_channel_unlock(ftdmchan);
return;
return FTDM_SUCCESS;
}
static FIO_CHANNEL_SEND_MSG_FUNCTION(ftdm_sangoma_isdn_send_msg)
@ -1098,6 +1089,7 @@ static FIO_CONFIGURE_SPAN_SIGNALING_FUNCTION(ftdm_sangoma_isdn_span_config)
span->get_span_sig_status = ftdm_sangoma_isdn_get_span_sig_status;
span->set_span_sig_status = ftdm_sangoma_isdn_set_span_sig_status;
span->state_map = &sangoma_isdn_state_map;
span->state_processor = ftdm_sangoma_isdn_process_state_change;
ftdm_set_flag(span, FTDM_SPAN_USE_CHAN_QUEUE);
ftdm_set_flag(span, FTDM_SPAN_USE_SIGNALS_QUEUE);
ftdm_set_flag(span, FTDM_SPAN_USE_PROCEED_STATE);

View File

@ -356,7 +356,6 @@ typedef enum {
"PROGRESS_MEDIA", "ALARM_TRAP", "ALARM_CLEAR", \
"COLLECTED_DIGIT", "ADD_CALL", "RESTART", "SIGSTATUS_CHANGED", "COLLISION", "FACILITY", \
"TRACE", "TRACE_RAW", "INDICATION_COMPLETED", "INVALID"
/*! \brief Move from string to ftdm_signal_event_t and viceversa */
FTDM_STR2ENUM_P(ftdm_str2ftdm_signal_event, ftdm_signal_event2str, ftdm_signal_event_t)
@ -653,7 +652,20 @@ typedef ftdm_status_t (*fio_span_get_sig_status_t) FIO_SPAN_GET_SIG_STATUS_ARGS;
typedef ftdm_status_t (*fio_span_poll_event_t) FIO_SPAN_POLL_EVENT_ARGS ;
typedef ftdm_status_t (*fio_span_next_event_t) FIO_SPAN_NEXT_EVENT_ARGS ;
typedef ftdm_status_t (*fio_channel_next_event_t) FIO_CHANNEL_NEXT_EVENT_ARGS ;
/*! \brief Callback for signal delivery (FTDM_SIGEVENT_START and friends)
* \note This callback is provided by the user during ftdm_configure_span_signaling
*
* \note You must NOT do any blocking during this callback since this function is
* most likely called in an internal signaling thread that can potentially be
* shared for all the channels in a span and blocking will delay processing
* (sometimes even audio processing) for other channels
*
* \note Although some simple FreeTDM APIs can work (ie: ftdm_span_get_id etc), the
* use of any FreeTDM call API (ie ftdm_channel_call_answer) is discouraged
*/
typedef ftdm_status_t (*fio_signal_cb_t) FIO_SIGNAL_CB_ARGS ;
typedef ftdm_status_t (*fio_event_cb_t) FIO_EVENT_CB_ARGS ;
typedef ftdm_status_t (*fio_configure_span_t) FIO_CONFIGURE_SPAN_ARGS ;
typedef ftdm_status_t (*fio_configure_t) FIO_CONFIGURE_ARGS ;

View File

@ -500,15 +500,15 @@ struct ftdm_span {
ftdm_span_stop_t stop;
ftdm_channel_sig_read_t sig_read;
ftdm_channel_sig_write_t sig_write;
/* Private I/O data per span. Do not touch unless you are an I/O module */
void *io_data;
ftdm_channel_state_processor_t state_processor; /*!< This guy is called whenever state processing is required */
void *io_data; /*!< Private I/O data per span. Do not touch unless you are an I/O module */
char *type;
char *dtmf_hangup;
size_t dtmf_hangup_len;
ftdm_state_map_t *state_map;
ftdm_caller_data_t default_caller_data;
ftdm_queue_t *pendingchans;
ftdm_queue_t *pendingsignals;
ftdm_queue_t *pendingchans; /*!< Channels pending of state processing */
ftdm_queue_t *pendingsignals; /*!< Signals pending from being delivered to the user */
struct ftdm_span *next;
};

View File

@ -86,15 +86,21 @@ typedef struct {
const char *file;
const char *func;
int line;
ftdm_channel_state_t state;
ftdm_channel_state_t last_state;
ftdm_time_t time;
ftdm_time_t end_time;
ftdm_channel_state_t state; /*!< Current state (processed or not) */
ftdm_channel_state_t last_state; /*!< Previous state */
ftdm_time_t time; /*!< Time the state was set */
ftdm_time_t end_time; /*!< Time the state processing was completed */
} ftdm_state_history_entry_t;
typedef ftdm_status_t (*ftdm_channel_state_processor_t)(ftdm_channel_t *fchan);
FT_DECLARE(ftdm_status_t) ftdm_channel_advance_states(ftdm_channel_t *fchan, ftdm_channel_state_processor_t processor);
/*!
* \brief Process channel states by invoking the channel state processing routine
* it will keep calling the processing routine while the state status
* is FTDM_STATE_STATUS_NEW, it will not do anything otherwise
*/
FT_DECLARE(ftdm_status_t) ftdm_channel_advance_states(ftdm_channel_t *fchan);
FT_DECLARE(ftdm_status_t) _ftdm_channel_complete_state(const char *file, const char *function, int line, ftdm_channel_t *fchan);
#define ftdm_channel_complete_state(obj) _ftdm_channel_complete_state(__FILE__, __FUNCTION__, __LINE__, obj)
FT_DECLARE(int) ftdm_check_state_all(ftdm_span_t *span, ftdm_channel_state_t state);
@ -171,16 +177,26 @@ struct ftdm_state_map {
};
typedef struct ftdm_state_map ftdm_state_map_t;
/*!\brief Set the state for a channel (the channel must be locked when calling this function)
* \note Signaling modules should use ftdm_set_state macro instead
* \note If this function is called with the wait parameter set to a non-zero value, the recursivity
* of the channel lock must be == 1 because the channel will be unlocked/locked when waiting */
FT_DECLARE(ftdm_status_t) ftdm_channel_set_state(const char *file, const char *func, int line,
ftdm_channel_t *ftdmchan, ftdm_channel_state_t state, int wait);
/*!\brief Set the state of a channel immediately and implicitly complete the previous state */
/*!\brief Set the state of a channel immediately and implicitly complete the previous state if needed
* \note FTDM_SIGEVENT_INDICATION_COMPLETED will be sent if the state change
* is associated to some indication (ie FTDM_CHANNEL_INDICATE_PROCEED)
* \note The channel must be locked when calling this function
* */
FT_DECLARE(ftdm_status_t) _ftdm_set_state(const char *file, const char *func, int line,
ftdm_channel_t *fchan, ftdm_channel_state_t state);
#define ftdm_set_state(obj, s) _ftdm_set_state(__FILE__, __FUNCTION__, __LINE__, obj, s); \
/*!\brief This macro is deprecated, signaling modules should always lock the channel themselves anyways since they must
* process first the user pending state changes then set a new state before releasing the lock
* this macro is here for backwards compatibility, DO NOT USE IT in new code since it is *always* wrong to set
* a state in a signaling module without checking and processing the current state first (and for that you must lock the channel)
*/
#define ftdm_set_state_locked(obj, s) \
do { \