Calling all Dispatchers

There are two items of general NetWeaver Identity Management maintenance that I get asked about frequently.

• How do you prevent deadlocks    
• What is the best way to configure my dispatchers in IDM?

All too often these issues are actually related as inefficient dispatcher setup can cause database deadlocks. In this blog entry I’d like to recommend some possible architecture scenarios that will help out with this. For the purposes of this discussion, we’ll be talking about a NetWeaver IDM 7.1 installation on Microsoft SQL Server 2008 R2. According to Microsoft:

A deadlock occurs when two or more tasks permanently block each other by each task having a lock on a resource which the other tasks are trying to lock. The following graph presents a high level view of a deadlock state where:

·         Task T1 has a lock on resource R1 and has requested a lock on resource R2.

·         Task T2 has a lock on resource R2 and has requested a lock on resource R1.

·         Because neither task can continue until a resource is available and neither resource can be released until a task continues, a deadlock state exists.

Attaching multiple dispatchers to the same task would then seem to create the potential for deadlocks to occur in the database, particularly if they are all trying to access the same rows in the various IDM tables. But wait, we’re supposed to be able to do this to encourage High Availability, Load balancing and failover, so what gives?

Well the secret lies in the architecture, of course. If the requests come from separate physical hosts, it is much easier for the both IDM and the database to manage the threads and requests. Let’s look at a couple of examples. First here is a basic dispatcher setup assuming one server with a couple of dispatchers on it.

This setup is OK since each dispatcher (D) is connected to a distinct Job (J) or set of jobs. Sometimes there is a need to have a configuration like this, which usually something like one dispatcher for provisioning jobs, one for housekeeping(HK) and one that provides some sort of elevated access for Directory Services(DS) access.

Based on this there might be a temptation to set up the dispatcher/job relationship to look something like this to provide additional fail over and support for some specific jobs. Consider the following example where I outline a scenario with a crossover between multiple dispatchers pointing to jobs in a many to one relationship:

This scenario is exactly what we do not want to have as it is most likely to create a deadlock scenario as having multiple dispatchers accessing the same jobs that are accessing the same database resources. What we have here is a potential for deadlocks as the system is trying to manage multiple database resources (rows) at the same time.

What tends to compound this situation is the way that the database is being accessed. Using the To Identity Store Pass creates the least amount of deadlock strain on the system, since this is under the direct control of the workflow system and its dispatchers, while using techniques such as the uIS_SetValue function, that can be called from anywhere, at any time, create the greatest possibility as the system is managing standard job based access with unforeseen access via uIS_SetValue.

This scenario should be replaced by something like this:

In this case the potential for deadlocks is significantly reduced since there is separate management of the database connections. It also provides a degree of load-balancing and failover since if the D₁ or D₃ Dispatchers are busy or unable to process an assigned task then the D₄ or D₅ dispatchers respectively can take over. A good resource for this can be found in the SAP document SAP NetWeaver Identity Management Identity Center Implementation Guide Optimizing Dispatcher Performance.

What also needs to be recognized is that it’s not always as much “where” the requests come from, but when the requests come. Care should be taken to monitor the frequency and duration of the larger and more intensive tasks and workflows to make sure that the more involved tasks do not run at the same time (For example the HCM load should probably not happen the same time that the Directory Service reconciliation is occurring). These should be the first candidates for having their down dedicated dispatchers if there is a need to run them on similar schedules and this cannot be avoided.

Given all of this, the thing to remember when considering dispatcher allocation is to make sure that there are not multiple dispatchers that are competing for the same set of Identity Store resources at the same time. As long as this is kept in mind when setting up dispatchers, the possibility of deadlocks is minimized.

A related question to this is how many dispatchers are needed to assign for provisioning and de-provisioning operations. I have always used 25,000 objects per dispatcher as a general rule. Based on this the scenario shown above would be good for an enterprise where there are a maximum of 50,000 users, roles, privileges or groups that are being managed in any given task.

As a final note, it is a good thing to remember that the dispatchers do not need to be installed on Microsoft Windows based systems only. Any UNIX/LINUX environment is just fine for setting up IDM Dispatchers. For more information, check out the SAP document: How To: Setting Up An Identity Management Dispatcher On A Unix Host Flavor.

Tutorial Follow up

One thing that I forgot to mention in my last post was about the content of the tutorials. When you take a look at them you're going to notice two things.

First off they are incredibly detailed and easy to follow.  If you've just come off of training, there's plenty of screenshots showing you how IDM should be configured which is great if you're learning something new (or seeing how it's been updated since last version)

If you're more familiar with IDM, the tutorials are great too.  They're all based on real life scenarios which you'll need to implement in your organization's  IDM setup. They're also very open ended, making it easy for IDM Pros to evolve the tutorial scenario into something that you can use.

Good luck and enjoy those tutorials!

Cutting the Gordian Provisioning Knot

I mentioned in a previous posting that SAP NetWeaver Identity Management has been called too complicated. I've even heard some people referring to it as the Gordian knot of IT Security. Well, anyone can "hack" through the knot, but smart people find a way to untangle it.

New versions of the product don't make this any easier, however there is a resource that is often overlooked. The SAP IDM team has created several fantastic tutorials that talk about many of challenges that we face on a daily basis.

I've found that with the release of 7.2, referring to the documentation in general and the tutorials in particular has been most helpful. I recently had an issue with Role and Privilege assignment that was driving me nuts until I looked at the new documentation. Along the way I learned some new things about Pending Values, the UI and of course, about how Privileges are assigned in IDM under the new release.

So remember, when the going gets tough, the tough read documentation and tutorials!

SQL Server Fulltext search updates

If you are using Fulltext search with a SQL Server implementation of IDM, be advised that there is a new installation guide that contains updated information about Fulltext search.  I was recently passed this information that you might find helpful...

There is a noiseword (2005)/stop-word(2008) list that contains words that are not indexed. Most of these makes sense (a, the, or, and, etc.) and does not affect searches in IdM. Our naming of privileges with the word "only" was slightly more unfortunate as its also considered a noiseword. This requires customization of the noiseword/stopword list for customers that want to search for the repository privilege.

 

Customizing stopwords in SQL Server 2008:

SQL Server 2008 uses stopwords stored in the database. To customize the list you need to make a copy of the system stopwords list and assign it to be used with the IdM full-text index (ftfull). This can be done using these commands or from the user interface of the SSMS.

CREATE FULLTEXT STOPLIST idmStopList FROM SYSTEM STOPLIST;
-- Remove the words you want to include in the index:
ALTER FULLTEXT STOPLIST idmStopList DROP 'only' LANGUAGE 1033;
ALTER FULLTEXT INDEX ON mxi_values SET STOPLIST idmStopList

Its also possible to view the stopwords using queries. An example listing languages blocking the word "only" follows:

-- To list all entries of 'only' stopwords in the stoplist (can be many languages):
SELECT * FROM sys.fulltext_stopwords WHERE stoplist_id = (SELECT stoplist_id FROM sys.fulltext_stoplists where name = 'idmStopList') and stopword = 'only'

-- To test a stoplist
SELECT special_term, display_term FROM sys.dm_fts_parser (' "a text like system priv ad only somethingsomething" ', 1033,(SELECT stoplist_id FROM sys.fulltext_stoplists where name = 'idmStopList'), 0)

Here we see that "a" and "like" are considered noise and not indexed, while "only" is indexed for exact matches.

Customizing noisewords in SQL Server 2005:

Modify the noiseZZZ.txt files in sql server folder, remove "only" and other things you want to include from ENG,ENU (and others) and then drop & recreate fulltext index:

drop fulltext index on mxi_values
drop fulltext catalog ftfull
CREATE FULLTEXT CATALOG ftfull WITH ACCENT_SENSITIVITY=OFF AS DEFAULT;
CREATE FULLTEXT INDEX ON mxi_values(searchvalue) KEY INDEX IX_MXI_VALUES_Value_ID; 

Hope you found this helpful! Thanks again to the folks that passed this info along!

Welcome, Microsoft

I'm always happy to see that IdM is gaining acceptance from customers and the Industry in general. In a list of The 10 sexiest Microsoft business teases for 2012 from ZD.NET by Mary Jo Foley, Identity Management finally makes the list, coming in at number 5.

Glad that IdM is finally being noticed as a core technology.  Sad to see that it's taken this long and only in the context of a major player like Microsoft, nothing about firms such as Sun (and Oracle), SAP, SailPoint, Aveksa, Siemens, etc., who have been doing this for years. not to mention that Microsoft has been in the IdM business for a few years now.