OpenLDAP Client Loses Connection to Active Directory

Near the start of the year, we had a number of RedHat 6.5 servers apply a set of platform updates. These updates included an OpenSSL update.

We then began to see a number of cross-platform issues when an application on the unix servers interacted with a service hosted on a Windows server over an encrypted channel. On the Windows side, the primary two services involved were Active Directory (LDAPS) and IIS (HTTPS). The error message on the client side was usually “Operations error”. Network packet inspection showed some funny business, including the Windows server closing the LDAP connection without notification to the client.

In some cases, specific to the AD involved issue, some folks have successfully implemented a workaround of changing their TCP port to 3268, following *many* online posts which suggest this as a solution. That workaround has two potential problems:

  • The GC port is a subset of all directory info. So you get some info, but not all of it. And you have no inkling that you are missing something. But if the GC has all the info you need, then no problem.
  • The GC port is read-only. No writes. If you don’t need to write, then no problem.

It’s taken us quite a while to troubleshoot what’s going on here, especially since there is an OS platform divide. There’s also confusion aplenty in how TLS relates to SSL. ūüôā

We initially believed that OpenSSL changed the default negotiation version to SSLv2. And that something about the Windows implementation of SSLv2 is incompatible with the OpenSSL implementation. Obvious workarounds based on that belief are forcing the unix client to choose SSLv3 and turning off SSLv2 on the Windows server, both good outcomes outside of this issue.

We’ve been deliberate in not jumping to turn off SSLv2 on the Windows servers because the impact isn’t clear. talks about how to go about doing that.

For the unix client —HTTPS—> IIS case, we’ve successfully had the client choose SSLv3, and resolved those incidents.

For the unix client —LDAPS—> AD case, we had more problems, mostly because openldap adds some complexity.

After much hair-pulling, we finally figured out that the openldap client doesn’t handle LDAP referrals after a LDAP bind well–even if they are configured to not follow referrals. Invisible to us, our domain began issuing some new LDAP referrals after we moved the DNS zone for the domain to AD-integrated DNS. The openldap client configuration needed to be updated to use the ldap_set_rebind_proc() function to allow an immediate rebind after connecting and receiving referrals.

The OpenSSL changes were red-herrings that threw us off and SSLv2 had nothing to do with the AD issue.

Hope this helps someone else. Especially folks who are getting the most common advice (which isn’t good advice) on the web for this which is to switch to port 3268 (which doesn’t suffer the same problem because it doesn’t issue those referrals).

Should We Sync Our AD Passwords to Office 365 (Azure Active Directory)?

This is a response I sent to this question on ActiveDir.


The usual objection to storing a password in Azure Active Directory (or anywhere that isn’t on-premise) is that you don’t have control of your credentials, and you don’t have the direct ability to enhance the at-rest or over-the-wire risks to those credentials. Let’s examine the logic of that a bit more closely …

For their on-premise services, very few organizations actually go to the strenuousness and periodicity of regulatory audits that Microsoft does for its cloud services. But meeting regulatory auditing doesn’t necessarily mean secure–as folks like Target might like to point out.

If you do sync your passsword, what actually is sent to Microsoft?

Turns out it is a hash of your password hash. So not the password and not even the NT hash.

If you don’t sync your password to AAD and only use federated authentication, do Microsoft’s cloud services ever have your password? Absolutely–yes. If your users use any of the various fat-clients to access Office 365 services, those fat-clients don’t support the security token protocols needed to do federation natively themselves. So they send the password over the wire to the O365 service. The O365 service proxies the federated authentication for the client–getting a federated token with your on-premise STS (or IdP) and using that token to get an AAD federated token it can use to access your O365 content and then send that content back to your fat-client. There are very few customers I know of that aren’t using any fat-clients with O365–so a very large number of users are currently sending their password to Microsoft over-the-wire. That isn’t quite the same as the at-rest credential risk, but most folks seem to gloss over this issue. I believe Microsoft is working on extending the fat clients they own, but they don’t control all fat clients, so this issue will likely remain as long as you have non-Microsoft based clients (e.g. iOS). More folks seem to be familiar with the MFA “app password” issue–which comes from exactly the same client protocol support issues.

So to summarize, there is no option that is “safe” if you define safe as your password doesn’t leave your on-premise organization. There are only options which may or may not meet your organization’s risk management stance. If that stance is that you can’t have at-rest credentials off-premise, then DirSync password sync isn’t for you.

A final thought: our organization had the same sort of initial reaction to storing credentials off-premise. But we’re now thinking about sticking a domain controller in an Azure VM–for a number of reasons that include reducing network traffic back to on-premise for Azure based VMs and to improve our geo-redundancy/business continuity stance. And in a year, we’ll probably also consider sticking a DC VM in Amazon EC2.¬† Bottom line: business needs often trump overly stringent security principles–and while security principles are nice things, it’s really all about risk management.

Active Directory Naming Attribute Restrictions

Last week someone asked a question on ActiveDir which many folks appreciated my response to. I’ve summarized the original question and included my response here for future reference.

Where should “prefix” words (Tussenvoegsels) in a name be stored in Active Directory? And how do I reflect cultural differences in names? For example, “Johannes van der Waals” vs. “Johannes Van Der Waals”.

The problem space here is bigger than you’ve outlined. It’s my understanding that there are cultures where the surname is presented first. There are also important name suffixes such as Jr., Sr., II (the second). As well as less important name suffixes. And name prefixes that you may consider important like Dr. There are also some regulations that may limit your ability to actually publish name data–in my sector, FERPA restricts our ability to publish student names when the student has blocked that. Case sensitivity is also a relevant factor in this space. And then there’s the issue of legal name vs. preferred name, which can be a highly political issue when transgender issues are taken into account.

I’d note that the LDAP specification includes support for attribute options. This is the idea that the attribute value might have different values depending on the context desired. It’s like a multi-valued attribute, but each value has additional context that “labels” when that value has meaning. And LDAP clients can request the right contextual value. In the context of this problem, imagine the clients could request the ‚ÄúBelgian‚ÄĚ option of the displayName. Or the ‚ÄúDutch‚ÄĚ option.

From a data modeling perspective, it’s much cleaner, and for example, it’d be a much better way to store a bunch of things in AD than what various product teams ended up with. Take for example, the displaySpecifier excerpt I sent earlier this week:

attributeDisplayNames (85): co,Country/Region; c,Country/Region Abbreviation; mSDS-PhoneticCompanyName,Phonetic Company Name; mSDS-PhoneticDepartment,Phonetic Department; mSDS-PhoneticDisplayName,Phonetic Display Name; mSDS-PhoneticFirstName,Phonetic First Name; mSDS-PhoneticLastName,Phonetic Last Name; distinguishedName,Distinguished Name; <snip>

An attribute option would have been a cleaner way to represent this data, instead of asserting an arbitrary delimiter of a comma to denote the data relationship. The Exchange product has a *lot* of examples in AD where this would be a better data modeling choice too–for example, the proxyAddresses attribute.

But AD doesn’t support attribute options. ÔĀĆ

So why did I bring it up? Well, I know a lot of places that use other LDAP directories to master their name data. And those that have tackled these harder issues leverage attribute options to accurately represent the name data in all these various formats.

And using an arbitrary delimiter‚ÄĒmimicking what Microsoft has done in a couple cases–isn‚Äôt really a good option here.

There’s another AD limitation too. By RFC, sn is supposed to be multivalued. But in AD, it’s single valued!¬†So with that attribute, you can’t even invent an arbitrary scheme to represent the various versions of the data you’d like to store. 8-10 years ago, when I pressed Microsoft PMs hard on this issue, I was told that yes, they recognized they screwed up, but there was no way they could fix it because it’d cause too many problems.

I’ve just talked about a bunch more challenges and not provided any help moving forward‚ÄĒbut I think that‚Äôs actually helpful, because context is important. ÔĀä

From a high level, I think you need to recognize that your best outcome is providing something pragmatically useful because the underlying product hasn’t given you the flexibility required to solve all these issues ideally. You do have the option of creating custom attributes, recognizing that none of the MS products that integrate with AD user data will leverage your custom attributes (I think Exchange is partially an exception here). Maybe that’s OK, and you use the displayName attribute to roll up all the bits of name data–both from the standard schema and your custom schema. But you’ll need to find a pragmatic compromise for the displayName value and the sn value–as those are single-valued and you really have no flexibility with those.

Diving deeper into the mechanics of which parts of the name data you store in which attributes, I’d bet there is a large diversity of choices taken by each AD deployment. The key here will be to make a decision for your implementation, document it, and then implement it.

Another thing to keep in mind when doing the data design is the ANR attribute set. The ANR attribute set by default is: givenName, sn, displayName, RDN, legacyExchangeDN, physicalDeliveryOfficeName, proxyAddresses. The ANR is a common search built into a lot of the MS integrated products, and this may influence where you decide to stick data. One thing to keep in mind about the ANR search is that if the ANR search has a space in it, it will break up those substrings and try both of those substrings against givenName & sn (in addition to all the other attributes). In other words, the ANR search makes it less important how you choose to format the displayName value. The ANR may influence your data design because if there’s name data you want to store, but most folks wouldn‚Äôt want to search on it, then you’d want to avoid putting it in that set of attributes. Or on the other side of the coin, maybe you’ll want to change the ANR attribute set from the default so more attributes are searched by default.

My AD has some name data standardization applied to it. But I have a mess when it comes to the source systems that supply the name data–and merging all those data sources. The source systems gave very little thought to these kinds of issues, in some cases have case insensitive data, there are a mix of legal and preferred names to choose from, and I have to deal with the FERPA issue I mentioned earlier. I’ve got a bunch of code to wrangle all of that, but to be honest, maintaining it and having to live with the limitations is the worst technical part of my job. For the past 5+ years, I’ve been pitching a different approach which ditches all of the source systems, and creates a new single source of preferred name data.¬† That new single source allows the users to assert their preferred name, but applies a set of name data rules at the point of input to ensure consistency and prevent deviation that is too great from the various source systems. This eliminates the need for the code I maintain by leveraging input validation, and eliminates the complexity associated with wrangling all the various source systems.

I hope all of that is helpful in some part.

UW Holidays for Outlook/Exchange 2014-2023

In the past I’ve pro­vided a cus­tom OUTLOOK.HOL file to add the UW Hol­i­days to Outlook and via Outlook to your Exchange calendar.

You can read for the first post where I men­tioned this mech­a­nism.

I‚Äôve just updated the Outlook.hol file noted in that post to include the UW¬†Holidays from 2014 thru 2023. Yeah, I got tired of doing this every year or two so I went for 10 years this time around. The link to that file is on the left … i.e.

Note that this newly updated OUTLOOK.HOL file is based on the stock Out­look file with UW Holidays from 2012 & 2013 and then UW Holidays 2014-2023 added. You’ll be clos­ing Out­look, replac­ing the stock file at C:\Program Files (x86)\Microsoft Office\Office14\1033 with this new copy (to be safe, make a copy of your stock file first), then via the File tool­bar, Options, Cal­en­dar, Add Hol­i­days…, Uncheck United States and Check UW Hol­i­days 2014-2023.

If you are running Outlook 2013, adjust the file location as needed.