Shockwave’s Blog

Having resolved the replication issues with the Active Directory Forest, the next step was to bring things up-to-date by moving to Server 2008 R2 for the Domain Controllers and raising the Forest Functional Level. To that end, this week I completed the ForestPrep and DomainPrep on the four domains in the forest without issue.

The problems came with the RODCPrep, after running the command I encountered an error with two of the application partitions, namely the ForestDNSZones and DomainDNSZones of the forest root domain, the errors were:

Adprep could not contact a replica for the partition DC=ForestDnsZones,DC=Domain,DC=com.
Adprep encountered an LDAP error
Error code: 0x0. Server extended error code:0x0, Server error message:(null)

Adprep failed the operation on the partition DC=ForestDnsZones,DC,Domain,DC=com

Adprep could not contact a replica for the partition DC=DomainDnsZones,DC=Domain,DC=com.
Adprep encountered an LDAP error
Error code: 0x0. Server extended error code:0x0, Server error message:(null)

Adprep failed the operation on the partition DC=DomainDnsZones,DC,Domain,DC=com

I had checked that the Infrastructure Master was available and I had the relevant permissions. So I checked the partitions in question and found that the fSMORoleOwner attribute in CN=Infrastructure,DC=ForestDNSZones,DC=Domain,DC=com and CN=Infrastructure,DC=DomainDNSZones,DC=Domain,DC=com had a strange value which made reference to a Domain Controller that had been removed from the forest some time ago, it looked something like this:

CN=NTDS Settings\0ADEL:7336dccf-9ddd-45c2-b3a1-42c72e17742c,CN=Server01\0ADEL:fe4f0c94-2ba5-4f28-945a-f225d3e4bd2a,CN=Servers,CN=Site,CN=Sites,CN=Configuration,DC=Domain,DC=com

A search on the web revealed an article from Microsoft (http://support.microsoft.com/kb/949257) which provided a script to change this to the correct value. Once changed (and given some time to replicate) I ran the ADPrep /RODCPrep again, which worked this time.

Some further reading by a colleague of mine brought him to an article by Ulf B. Simon-Weidner’s (http://msmvps.com/blogs/ulfbsimonweidner/archive/2008/07/31/how-many-infrastructure-masters-do-you-have.aspx) which revealed an interesting fact:

If you move your Infrastructure Master using the MMC or NTDSUtil, the fSMORoleOwner attribute is not updated automatically in your application partitions. Handy to know and thanks to Ulf for documenting this.

We’ve spent a great deal of time resolving the issues that existed with Active Directory infrastructure, however, one thing that still needed doing was to increase the Tombstone Lifetime from 60 to 180 days. If you’re reading this then you’re probably aware that Microsoft increased the default in Server 2003 SP1 and above, however, the 180 day default is only implemented in a Forest where the first Domain Controller was 2003 SP1 or above. If you simply upgrade the Domain Controllers in an existing Forest, it doesn’t change automatically.

Given the geographical distribution of Domain Controllers within my environment and the multiple child domain hierarchy, I wasn’t convinced that this was going to be as easy as changing the value. This was backed up by a great article by Ulf B. Simon-Weidner available HERE and Mark Parris HERE. You can also check out Ulf’s TechEd session HERE which discusses the issue.

Anyway, following Ulf’s advice, I devised the following the following checklist to increase the Tombstone Lifetime in my Forest.

Step1: Enable the diagnostic event for Garbage Collection on each Domain Controller in the Forest. You can do this by amending the registry (http://support.microsoft.com/kb/314980)

• Navigate to the following key in the registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\6 Garbage Collection
• Amend key to ‘3’ (This was my preference, for a description of the different logging levels, see the TechNet article linked in Step 1)

Step 2: Trigger Garbage Collection on each Domain Controller in the Forest

• Download ADMod from http://www.joeware.net/freetools/tools/admod/
• Run ADMod -b -sc gc on each Domain Controller in the Forest
• Check the event viewer to ensure that it ran

Step 3: Increase the Tombstone Lifetime

• Run ADSIEdit.msc on a Domain Controller in the Forest root
• Connect to the Configuration naming context
• Navigate to cn=Directory Service,cn=Windows NT,cn=Services,cn=Configuration,dc=YOURDOMAIN,dc=COM
• View the properties of cn=Directory Service
• Locate tombstoneLifetime and amend to 180

Step 4: Initiate replication to ensure that the new value is replicated to every Domain Controller.

• I won’t insult you by describing how I do this, but it involves using REPLMON

Step 5: Ensure Garbage Collection did not run in between

• Check your Directory Service events, Garbage Collection should appear as EVENT ID: 1646

Step 6: Ensure that each Domain Controller in the Forest have the new value

• You can achieve this by checking the configuration DB on each Domain Controller using the same process as described in Step 3 (without actually changing the value)

Unfortunately, I ran into a problem – During my tests, I found that manually triggering Garbage Collection did not reset the schedule, as a result you could manually trigger it on your Domain Controller only to find that 5 minutes later it gets triggered as part of the normal 12 hour cycle… Why is this a problem?

Well, imagine you had an account which had been tombstoned and is 59 days and 23 hours and 55 minutes old. When you manually trigger garbage collection, it will not be removed, however, what happens if Garbage Collection then runs 5 minutes after this as part of the normal cycle, before the new 180 tombstone lifetime value has been replicated to it? The answer is, it will be removed which is fine, unless another Domain Controller received the new 180 day tombstone lifetime before it’s garbage collection process runs – You’re then in a position where you have inconsistencies between Domain Controllers, not a nice position to be in.

To get around this, I ave devised an alternate approach (which I haven’t tested yet):

Step 1: Modify the garbageCollPeriod to 480 (This is it’s maximum value in hours for a Forest with a 60 day tombstone lifetime)

• Run ADSIEdit.msc on a Domain Controller in the Forest root
• Connect to the Configuration naming context
• Navigate to cn=Directory Service,cn=Windows NT,cn=Services,cn=Configuration,dc=YOURDOMAIN,dc=COM
• View the properties of cn=Directory Service
• Locate garbageCollPeriod and amend to 480

Step 2: Amend the Tombstone Lifetime to 180 days

• Run ADSIEdit.msc on a Domain Controller in the Forest root
• Connect to the Configuration naming context
• Navigate to cn=Directory Service,cn=Windows NT,cn=Services,cn=Configuration,dc=YOURDOMAIN,dc=COM
• View the properties of cn=Directory Service
• Locate tombstoneLifetime and amend to 180

Step 3: Amend the garbageCollPeriod back to the default 12 hours

• Run ADSIEdit.msc on a Domain Controller in the Forest root
• Connect to the Configuration naming context
• Navigate to cn=Directory Service,cn=Windows NT,cn=Services,cn=Configuration,dc=YOURDOMAIN,dc=COM
• View the properties of cn=Directory Service
• Locate garbageCollPeriod, select ‘edit’ then ‘clear’

WARNING: If you decide to simply increase the Tombstone Lifetime in your Forest and you don’t follow a similar procedure, you risk introducing lingering objects – Not a nice position to be in, trust me!

One of the issues I identified shortly after starting with my current employer was the replication topology – There had been some decisions made in the past which didn’t make a great deal of sense to me and a general lack of administration meaning that redundant sites had not been removed (as an example).

Since the issues were identified I have spent a lot of time supported by colleagues and a Microsoft Gold Partner (Silversands) cleaning this up and amending the topology. We have now move to a hub/spoke topology as we have an empty forest root and three child domains (representing the three regions). To give you an idea of the carnage before and the new hub/spoke design, have a look at the diagrams below:

Before:

After:

The difference, I hope you’ll agree is night and day… We can now accurately predict replication across the group. Before these changes, I had no idea how long something would take to replicate (if ever!).

For those of you that are interested, I created these diagrams using Microsoft Active Directory Topology Diagrammer, available HERE

This week I wanted to find out when an Active Directory domain integrated forward lookup zone was created. It had been configured to replicate to all DNS servers in the domain.

Initially I thought I might be able to retrieve this information using DNSCmd with the /ZoneInfo switch, alas this is not one of the attributes. Instead, I had to use ADSIEdit by connecting to the following distinguished name:

DC=DomainDNSZones,DC=domain,DC=com

On reviewing the properties, I was able to see the date the zone was created.

N.B. If the zone was replicated around the forest, I could have used DC=ForestDNSZones,DC=Domain,DC=com instead.

Okay, my new employer has a prexisting empty forest root with three child domains (North America, EMEA and Asia Pacific). Shortly after I started there was an issue where users in the Asia domain were unable to consume services being published on EMEA servers.

Whenever Asian users were trying to access anything, they’d get an unknown username/bad password message despite using the correct credentials.

Troubleshooting revealed that it was only users using credentials from other domains that were experiencing issues, this pointed to the trust between the EMEA domain and the empty forest root. To confirm this I checked the validity of the trust in AD Domains and Trusts which allows trust to be validated, this validation was successful on some domain controllers in the forest root, but not all. Further investigation revelaed that that password for the EMEA tree-root trust had been reset, but not replicated to all domain controllers in the forest root.

I was able to prove this by using ADSIEdit on each of the domain controllers in the forest root. By connecting to the domain context, navigating to the users container and finding the user account for the EMEA domain. This will be have the same name as the child domain with a $ suffix i.e. EMEADOMAIN$

If you review the properties of this account you will find the pwdLastSet value, displayed as a long integer:

Account Properties

You can convert this into a standard date/time format using w32tm /ntte as shown below:

w32tm /ntte

Once I compared this on all DCs, I found that it wasn’t consistent. The new password had not replicated around all domain controllers (which will be the subject of a future post). Anyway, to resolve the issue, it was necessary to force another password reset. To do this, I used the netdom command from a domain controller in the forest root using the syntax below:

NetDom trust EMEADOMAIN.ForestRoot.com /domain:FORESTROOT.com /UserD:FORESTROOT\EnterpriseAdmin PasswordD:* /UserO:EMEADOMAIN\DomainAdmin PasswordO:* /reset

Once I’d entered the password for the respective accounts, I waited for replication to occur and services were restored. Phew!

Having started a new job recently, one of the first things I wanted to do was get a handle on how the Active Directory infrastructure looked, to do this I decided to use my old friend the “Active Directory Topology Diagrammer” which is a free tool available from Microsoft.

My understanding was that most people were aware of this tool, however, the mere mention of it has been met with furrowed brows and confusion. As a result, I thought I’d kill two birds with one stone and create my first blog post in some time and share the link so you can download it.

By using ADTD what you will see is an automated diagram using Visio which will draw your AD topology including your domain(s), trust(s), sites and site links, Domain Controllers, OU structure and your Exchange Organisaion among others.

You will also get an excel spreadsheet which shows your Domain Controllers (including the OS and Service Pack), FSMO role location and identify which servers hold the Global Catalog.

For more information, download the tool and give it a go!

Download Active Directory Topology Diagrammer

Following a site move which involved moving a Domain Controller to an alternate site with a new IP addressing scheme we were required to change the IP address of this DC (as you might expect). Unfortunately, this move took a number of months to complete and as a result the computer account had expired causing replication issues.

If you experience similar issues, it’s pretty easy to resolve by following these steps:

1. Stop the KDC service and set the startup to manual.
2. Restarted the DC experiencing issues.
3. Use Netdom to reset the computer account password: Netdom ResetPWD /Server:Replication_Partner /UserD:DOMAIN\Administrator /PasswordD:*
4. Replication_Partner in this command was, in my case, the Domain Controller holding the PDCe FSMO role.
5. Once you’ve run the above command restart the DC once again.
6. Allow replication to occur.
7. If the KDC service does not start automatically and change it’s startup to ‘Automatic’ then you might want to do that.

After completing the above steps, give it some time. Remember, impatience is the undoing of many a good Systems Administrator. You may want to force replication from Active Directory Sites and Services then check your event viewer for issues.

We are currently suffering a problem starting Active Directory following a power down of the Domain… When we power up the first domain controller (PDCe) we get the usual delay before logging on, once logged on there are a number of event log errors and Active Directory and DNS will not start. This is a Server 2003 R2 Domain Controller with a Forest and Domain functional level of Server 2003.

When opening AD Users and Computers the following error is displayed: “The specified domain either does not exist or cannot be contacted” and you cannot expand AD Users and Computers as shown:

When opening the DNS console the following erroris displayed: “Cannot contact the DNS server”.

The DC in question is using 127.0.0.1 (as per the MS best practice) as its primary DNS and the IP of a different domain controller as the secondary DNS which wouldn’t be powered on at this stage.

In addition to this there the following associated events in the event logs for the DC:

Directory Services
1. Replication errors which I would expect given the other Domain Controllers aren’t powered on at this stage

2. Event ID 1126: Active Directory was unable to establish a connection with the global catalogue

DNS Server
1. Event ID 4013: The DNS server was unable to open the Active Directory. This DNS server is configured to use directory service information and can not operate without access to the directory. The DNS server will wait for the directory to start. If the DNS server started but the appropriate event has not been logged, then the DNS server is still waiting for the directory to start.

It appears that DNS will not start without Active Directory and Active Directory will not start without DNS!

After further investigation, it became evident that this issues relates to the network config. The server is a HP Proliant DL360 G5 using the HP Intelligent Networking Pack and 802.3ad LACP for the team, connected to a Cisco Catalyst 4948. If this team is dissolved then AD starts along with DNS.

Further testing revealed that AD and DNS would start fine once I disabled then enabled the team from “Network Connections” – Currently I have closed this as a Known Error with a workaround but the true cause remains a mystery. I would love to hear from anyone that has experienced the same issue.

You may or may not know that by default, any authenticated user has the ability to join 10 computers to your domain which is going to increase your attack surface, but all is not lost, you can remove this ability pretty easily. Just follow these simple steps (as a Domain Admin):

1. Run Adsiedit.msc
2. Expand the ‘Domain NC‘ node.
3. This node contains an object that begins with “DC=” and reflects the correct domain name.
4. Right-click this object, and then click Properties.
5. In the Select which properties to view box, click Both.
6. In the Select a property to view box, click ms-DS-MachineAccountQuota.
7. In the Edit Attribute box, type the number of workstations that you want users to be able to maintain concurrently.
8. Click Set, and then click OK.

Once you’re done, authenticated users will only be allowed to add the number of machines you sepcify (I’d suggest none!). This is all detailed in the Microsoft KB article HERE

We’ve been introducing a new HR and Payroll application recently within the organisation, as part of this we need some method of linking Active Directory (AD) accounts with employee records as clearly this will present untold benefits, like… errr, well, benefits!

Anyway, to achieve this we simply leveraged the power of the existing employeeID attribute within AD. To enable this you need to open a new MMC console and add the Active Directory Schema snap in –  You should do this on the Domain Controller which holds the Schema Admin FSMO role and that you are both a Domain Admin and a Schema Admin. If you can’t see the Schema snap in when you try to add it to your MMC then you’ve forgotten to run “regsvr32 schmmgmt.dll”.

Okay, now that you’ve done that you will find yourself in the AD Schema configuration, careful now! Here’s what you should do next:

1. Expand the Active Directory Schema object and then expand the Classes container.
2. Scroll down to the “user” class. Right-click on the “user” class and select Properties.
3. In the “user Properties” dialog box, click the Attributes tab and then click the Add button.
4. You should now be at the Select Schema Object window. Scroll down and select employeeID and then click OK.
5. In the “user Properties” dialog box, click OK.

Once complete this will have made the relevant changes… Now a warning (NOW a warning?!?) this is a Forest wide change and will impact all Domains within that Forest. But you knew that already, right?

Finally, now that we have this attribute, can you see it in Active Directory Users and Computers, well, no. You’re gonna have to get creative with your scripting. I have seen and heard of people using Visual Basic to add, change, delete these, however, given that we’re now in the Powershell world, that’d be my preference. So I intend to spend a little time looking at this. To date I have found the following code snippet:

Import-CSV C:\MyFile.csv | ForEach-Object { Set-QADUser $_.samAccountName
-ObjectAttributes @{employeeid=$_.employeeid }}

This uses the Quest AvtiveRoles cmdlets which can be found HERE and and -ObjectAttributes which will allow you to modify any AD property your heart desires. Once I’ve had a chance to play with this I will post a little more of a  Powershell update. I also need to come up with a clever way of working out which AD account links with which HR record in the first place to enter these IDs which should be fun!