Calling all Dispatchers

There are two items of general NetWeaver Identity Management maintenance that I get asked about frequently.

• How do you prevent deadlocks    
• What is the best way to configure my dispatchers in IDM?

All too often these issues are actually related as inefficient dispatcher setup can cause database deadlocks. In this blog entry I’d like to recommend some possible architecture scenarios that will help out with this. For the purposes of this discussion, we’ll be talking about a NetWeaver IDM 7.1 installation on Microsoft SQL Server 2008 R2. According to Microsoft:

A deadlock occurs when two or more tasks permanently block each other by each task having a lock on a resource which the other tasks are trying to lock. The following graph presents a high level view of a deadlock state where:

·         Task T1 has a lock on resource R1 and has requested a lock on resource R2.

·         Task T2 has a lock on resource R2 and has requested a lock on resource R1.

·         Because neither task can continue until a resource is available and neither resource can be released until a task continues, a deadlock state exists.

Attaching multiple dispatchers to the same task would then seem to create the potential for deadlocks to occur in the database, particularly if they are all trying to access the same rows in the various IDM tables. But wait, we’re supposed to be able to do this to encourage High Availability, Load balancing and failover, so what gives?

Well the secret lies in the architecture, of course. If the requests come from separate physical hosts, it is much easier for the both IDM and the database to manage the threads and requests. Let’s look at a couple of examples. First here is a basic dispatcher setup assuming one server with a couple of dispatchers on it.

This setup is OK since each dispatcher (D) is connected to a distinct Job (J) or set of jobs. Sometimes there is a need to have a configuration like this, which usually something like one dispatcher for provisioning jobs, one for housekeeping(HK) and one that provides some sort of elevated access for Directory Services(DS) access.

Based on this there might be a temptation to set up the dispatcher/job relationship to look something like this to provide additional fail over and support for some specific jobs. Consider the following example where I outline a scenario with a crossover between multiple dispatchers pointing to jobs in a many to one relationship:

This scenario is exactly what we do not want to have as it is most likely to create a deadlock scenario as having multiple dispatchers accessing the same jobs that are accessing the same database resources. What we have here is a potential for deadlocks as the system is trying to manage multiple database resources (rows) at the same time.

What tends to compound this situation is the way that the database is being accessed. Using the To Identity Store Pass creates the least amount of deadlock strain on the system, since this is under the direct control of the workflow system and its dispatchers, while using techniques such as the uIS_SetValue function, that can be called from anywhere, at any time, create the greatest possibility as the system is managing standard job based access with unforeseen access via uIS_SetValue.

This scenario should be replaced by something like this:

In this case the potential for deadlocks is significantly reduced since there is separate management of the database connections. It also provides a degree of load-balancing and failover since if the D₁ or D₃ Dispatchers are busy or unable to process an assigned task then the D₄ or D₅ dispatchers respectively can take over. A good resource for this can be found in the SAP document SAP NetWeaver Identity Management Identity Center Implementation Guide Optimizing Dispatcher Performance.

What also needs to be recognized is that it’s not always as much “where” the requests come from, but when the requests come. Care should be taken to monitor the frequency and duration of the larger and more intensive tasks and workflows to make sure that the more involved tasks do not run at the same time (For example the HCM load should probably not happen the same time that the Directory Service reconciliation is occurring). These should be the first candidates for having their down dedicated dispatchers if there is a need to run them on similar schedules and this cannot be avoided.

Given all of this, the thing to remember when considering dispatcher allocation is to make sure that there are not multiple dispatchers that are competing for the same set of Identity Store resources at the same time. As long as this is kept in mind when setting up dispatchers, the possibility of deadlocks is minimized.

A related question to this is how many dispatchers are needed to assign for provisioning and de-provisioning operations. I have always used 25,000 objects per dispatcher as a general rule. Based on this the scenario shown above would be good for an enterprise where there are a maximum of 50,000 users, roles, privileges or groups that are being managed in any given task.

As a final note, it is a good thing to remember that the dispatchers do not need to be installed on Microsoft Windows based systems only. Any UNIX/LINUX environment is just fine for setting up IDM Dispatchers. For more information, check out the SAP document: How To: Setting Up An Identity Management Dispatcher On A Unix Host Flavor.