3/4/2009 Edited list of out-of-box management agents
So recently I was able to take a training class on ILM. This post contains some of the core concepts and info I learned in that class, along with a few interesting bits about our ILM implementation here. I’ve written once before about ILM here, Identity Lifecycle Manager: Directory synchronization nirvana, Delegated Group Management, Certificate Provisioning, and more, mostly as an intro without any technical depth. This post will pe much further into the technical details.
In general, ILM is a directory synchronization tool and a //certificate management and deployment tool. Since directories commonly hold identity information, this set of functionality neatly rolls up into the product name. ILM is commonly used to provision user accounts, and has some capability to manage password synchronization.
In a short time, Microsoft will release a new version of ILM, currently code-named ILM2. This product has a web portal front-end to it (more on that in a second), workflow functionality, and some very cool cross-product tie-ins. This combination of new features allows very cool new functionality, such as allowing you to provision groups that are dependent on something else. Imagine you know you need to be in a certain group to get access to something. ILM can capture your request via its web portal, or even via an Outlook snap-in. It then sends an email request along to the designated admin for the group, asking if you should be added. If the admin has the outlook snap-in, they can approve the request from within outlook. If they don’t use outlook, they can follow a link in the email to the web portal to approve/deny. I don’t know whether we’ll ever see this feature set of the product in use here at the UW, but it is fun to imagine. 🙂
Anyhow, let’s move back to the core product.
ILM is a state-based synchronization product. This means that it reads in the state of all the various data sources, compares them determining what has changed, and then acts on just the changes. As you might imagine, this has both advantages and disadvantages:
- it isn’t dependent on a specific order of events
- it isn’t reliant on installing agent code at each data source to send data to it, and any unreliability/security issues such agent code might bring with it
- it can enforce that the state be kept as expected
- Requires processing power to evaluate state differences
- the timeliness of changes aren’t necessarily as good as an event-based process, i.e. there is some synchronization latency
ILM comes out of the box with a wide variety of management agents for common directory and data source products. An ILM management agent is responsible for managing the flow of data to and from a data source.
- AD Global Address List
- Attibute-value pair (AVP) file-based
- Delimited text file
- Exchange 5.5
- Exchange 5.5 (bridgehead)
- Extensible connectivity
- Fixed-width text file
- IBM DB2 Universal Database
- IBM Directory Server
- LDIF file-based
- Lotus Notes
- Novell eDirectory
- Sun and Netscape directory servers
- Windows NT 4.0
There are also 3rd party management agents.
Conceptually, ILM has 2 object spaces you need to understand. Each management agent has its own connector space (CS). This includes all the data source objects for that management agent. And then there is a single metaverse space (MV). The metaverse space represents those connector space objects which have been projected or joined (more on what those two new words mean in a minute). In other words, the metaverse is the space where things come to together.
Each management agent defines which objects in its connector space should be projected or joined to the metaverse. To project means that the resulting object in the metaverse will consider this management agent’s object as authoritative for that object. And if you have no management agent’s projecting, then your metaverse will be empty. Put another way, projecting is the way to provision objects into the metaverse, and for each metaverse object there is a special relationship dependent upon which management agent or agents projected it. To join means what you might imagine; it connects an object in this management agent’s connector space with objects in the metaverse. Both projection and joining are dependent upon filters and rules that determine which objects should do what, and in what way. A projection filter determines which objects should project. A projection ruledetermines which connector space attributes should map to which metaverse attributes. A join filter determines which objects should attempt to connect to which metaverse objects. A join rule determines which connector space attributes should map to which metaverse attributes. Both join rules and projection rules can go either direction. In order to achieve some synchronization, you’ll need rules that go both in and out of the metaverse.
This leads us to schema, i.e. the definition of what kinds of objects there are and what kind of data can be associated with each object. Each data source comes with its own schema. Each management agent (and therefore its connector space) has a schema (which may or may not match the data source). And the metaverse has a schema that somehow melds all of this together.
Without any special skills, one can directly map an attribute in one connector space to an attribute in the metaverse. If you need to make any sort of changes, have any kind of logical dependency, then you need to use an extended rule, which involves writing some code. Which is not especially hard.
Moving back to concepts, the way in which data is moved around and synchronized is all tied to the management agents. Each management agent has a set of run profiles.
Each run profiles can be one or more of the following actions:
- Delta import (staged). Evaluate only changed objects in data source, and make changes only to CS.
- Full import (staged). Evaluate all objects in data source, and make changes only to CS.
- Delta import and delta sync. Evaluate only changed objects in data source, make changes to CS, then evaluate projection/join rules for only those CS objects which changed, make resulting changes to MV, follow any external attribute flow rules making changes to other management agent’s CS.
- Full import and delta sync. Evaluate all objects in data source, make changes to CS, then evaluate projection/join rules for only those CS objects which changed, make resulting changes to MV, follow any external attribute flow rules making changes to other management agent(s)’s CS(s).
- Full import and full sync. Evaluate all objects in data source, make changes to CS, then evaluate projection/join rules for all CS objects, make resulting changes to MV, follow any external attribute flow rules making changes to other management agent’s CS. A full sync is needed after a rules change to apply these new rules to CS and MV objects which didn’t change during a delta.
- Delta sync. Evaluate projection/join rules for only those CS objects which changed, make resulting changes to MV, follow any external attribute flow rules making changes to other management agent’s CS.
- Full sync. Evaluate projection/join rules for all CS objects, make resulting changes to MV, follow any external attribute flow rules making changes to other management agent’s CS.
- Export. Push *all* CS objects to data source per external attribute mapping.
So imports move data into the connector space.
Syncs move data from one connector space into the metaverse and back out to other connector spaces. Note that if you have many management agents, you need to run a sync on each of them to achieve complete synchronization.
Exports move data from the connector space back to the data sources. Exports are the only way to achieve some kind of change in the world outside ILM; without them, you are just playing around in an ILM universe.
We rolled ILM out depending heavily on a Microsoft Consulting engagement to get things in place so the UW Exchange deployment could have an address book which didn’t look rotten.
Our existing implementation of ILM connects an AD with an OpenLDAP directory, or more specifically UWWI with PDS. As you might have noticed, OpenLDAP wasn’t one of the out of the box management agents listed above. At the time of our implementation, there was a 3rd party management agent, but it had known problems which made it not acceptable for our deployment. Since then, an open-source openldap management agent which addresses those problems has been released, but we haven’t had a chance to evaluate it.
Our deployment uses an extensible management agent built around the LDIF management agent to connect to openldap. Our openldap directory dumps change logs to ILM, and ILM reads those in delta imports.
In the existing implementation, AD projects to the MV, and PDS joins. And only objects from PDS which are of class uwPerson are joined. The attribute flow is mostly online already, so I won’t go into that. Our AD has about 450000 users, while PDS has about 1.5 million objects.
One of the gotchas in our existing implementation is the high number of disconnectors. A disconnector is any connector space object which is not related to a metaverse object (i.e. joined or projected). The problem with lots of disconnectors is that every time you do a sync, *all* disconnectors are re-evaluated to see if they now join up. I’m in the process of investigating whether some design changes wouldn’t eliminate that problem in our implementation, and allow us to go from a 3 hour sync cycle to something much shorter. Currently the delta imports take about 5 minutes, while the delta sync takes 90 minutes.
Another gotcha is that while the MS consultant refined the rules a couple times before he left, it doesn’t appear that he ran a full sync after those rule changes. This means that only objects which have changed since then have had those updated rules applied. So there is inconsistency in UWWI in terms of what should be there. Now I may be the only one who has come across examples of this, but it really bugged me until I found out why.
The final notable gotcha in this space resolves around the “name” attributes. And there’s a complex story here, which for the sake of my sanity, I am not going to go into details about. The problems here are:
- the mapping logic is inconsistent depending on the initial state,
- not all UWWI objects get joined to a PDS object (b/c of the uwPerson join rule),
- the input validation and formatting from the source systems to the PDS attribute we use is non-existent,
- and our mapping logic code falls short of addressing all cases (but there’s no way you can address all possible cases given no input validation and inconsistent formatting).
Oh … and only employees have any real ability to change their name.
We’ve got an imagined fix for all of this, where the ‘manage your uw netid’ page would allow *every* uw netid to manage their name information in a consistent format, with input validation that would then flow through, but it hasn’t gotten enough priority to be resourced.