SuperG Web Log

Logon troubleshooting

Ok so today I had a problem with an account that kept getting locked out due to bad passwords. The user had just changed their password. To try and find out where the issues is, I turned up the debug on netlogon on the DC. This way I can see where the logon event is coming from. Below is some good info for netlogon debugging.

Log files can be searched for logon problems

Do this on the PDC

First enable Debug mode for netlogon:

nltest /dbflag:0×2080ffff

Restart netlogon service

You can pull the log lines for bad password entered using 0xc000006A:

type %windir%\debug\netlogon.log | find /i “0xc000006A” > badpassword.txt

You can pull the log lines for locked out accounts 0xc0000234:

type %windir%\debug\netlogon.log | find /i “0xc0000234″ > lockout.txt

You can pull the log lines for usernames by using the username:

type %windir%\debug\netlogon.log | find /i “<username>” > user.txt

Afterwards disable debug mode:

nltest /dbflag:0×0

Restart netlogon service

Other returns to look for:

0xC0000234 User logon with Account Locked

0xC000006A User logon with Misspelled or bad Password

0xC0000072 User logon to account disabled by Administrator

0xC0000193 User logon with Expired Account

0xC0000070 User logon from unauthorized workstation

0xC000006F User logon Outside authorized hours

0xC0000224 User logon with “Change Password at Next Logon” flagged

0xC0000071 User logon with Expired Password

0xC0000064 User logon with Misspelled or Bad User Account

Debug flags:

////////////////////////////////////////////////////////////////////////

// Windows Server 2008, Windows Vista, Windows Server 2003, Windows 2000 Debug flags and their values

////////////////////////////////////////////////////////////////////////

#define NL_INIT 0×00000001 // Initialization

#define NL_MISC 0×00000002 // Misc debug

#define NL_LOGON 0×00000004 // Logon processing

#define NL_SYNC 0×00000008 // Synchronization and replication

#define NL_MAILSLOT 0×00000010 // Mailslot messages

#define NL_SITE 0×00000020 // Sites

#define NL_CRITICAL 0×00000100 // Only real important errors

#define NL_SESSION_SETUP 0×00000200 // Trusted Domain maintenance

#define NL_DOMAIN 0×00000400 // Hosted Domain maintenance

#define NL_2 0×00000800

#define NL_SERVER_SESS 0×00001000 // Server session maintenance

#define NL_CHANGELOG 0×00002000 // Change Log references

#define NL_DNS 0×00004000 // DNS name registration

//

// Very verbose bits

//

#define NL_WORKER 0×00010000 // Debug worker thread

#define NL_DNS_MORE 0×00020000 // Verbose DNS name registration

#define NL_PULSE_MORE 0×00040000 // Verbose pulse processing

#define NL_SESSION_MORE 0×00080000 // Verbose session management

#define NL_REPL_TIME 0×00100000 // replication timing output

#define NL_REPL_OBJ_TIME 0×00200000 // replication objects get/set timing output

#define NL_ENCRYPT 0×00400000 // debug encrypt and decrypt across net

#define NL_SYNC_MORE 0×00800000 // additional replication dbgprint

#define NL_PACK_VERBOSE 0×01000000 // Verbose Pack/Unpack

#define NL_MAILSLOT_TEXT 0×02000000 // Verbose Mailslot messages

#define NL_CHALLENGE_RES 0×04000000 // challenge response debug

#define NL_SITE_MORE 0×08000000 // Verbose sites

//

// Control bits.

//

#define NL_INHIBIT_CANCEL 0×10000000 // Don’t cancel API calls

#define NL_TIMESTAMP 0×20000000 // TimeStamp each output line

#define NL_ONECHANGE_REPL 0×40000000 // Only replicate one change per call

#define NL_BREAKPOINT 0×80000000 // Enter debugger on startup

Troubleshooting CAC Login

So we use smartcards to log onto the network, and i have a lot of users who call and say they are having problems logging on. When asked what the error is they state “its the system cannot log you on error”. Well there are around 20 different “the system cannot log you on errors”. Below is a partial list and possible cures.

@Axel Doux

This is in response to Axel Doux’s comment.

So I am assuming you created your EventLog something like this:

New-EventLog -LogName “My new EventLog” -Source “My sources”

And you can write events with:

write-eventlog -logname “My new EventLog” -Source “My sources” -Message “Some Error happened” -id 999

Then to read the log you can use:

Get-WinEvent –providerName “My sources”

You can even see it in the Event Viewer:

But when you make an Eventlog this way it makes it as a “Classic” eventlog

When you try and point to this as the destination you get and error that says that you cannot use Classic logs for destinations

If you look in this registry key you will see all the EventLogs that do show up in the destinations list, these are all Manifest-based Events:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels

While you can make Manifest-based EventLogs yourself, it is beyond the scope of what I do, and you would be better served finding a programmer (something in the .NET) to help if you want to make your own Manifest-based EventLogs.

Hope this helps you a little.

SuperG

Event Forwarding of Security Logs

This is a long one, sorry but there is a lot in information here.

So recently, where I work, we decided to collect a lot of the security events from all of our domain controllers. We are going to use the built-in “Windows Event Collector” service to do this. I am assuming you already have WinRM installed and working (WinRM qc)

From Server Manager expand Diagnostics > Event Viewer Then click on Subscriptions

This will start the prompt you to Subscription, click yes.

Clicking yes will make the “Windows Event Collector” service start Automatic (Delay Start)

You can now create a new Subscription by clicking the “Create Subscription…” in the action pane.

On the Subscription Properties windows fill out the information. In this case we are collecting the DC – Events. Select “Source computer initiated” for Subscription type.

Next in “Select Computer Group…” add the group that contains all the servers you want to collect events from.

In “Select Events…” add filter information to get just the events you want. In my case I want from the security log and only selected Event IDs

Next in the Advanced button Select Minimize Latency and HTTP. We will be changing this later to a custom config.

Now I want to change a few settings because I want to get events as soon as they happen. So I am going to set the “DeliveryMaxItems” to 1, you could also set “DeliveryMaxLatencyTime” if you wish but since you are only allowing one item each connection it will not wait for the timeout to deliver.

From an Admin Command Prompt:

wecutil ss “DC – Events” /cm:Custom /dmi:1

You can then use: wecutil gs “DC – Events” to see the change

Next we will have the source computers send the events as raw events vice “Rendered” events. This will remove some of the CPU usage from the source computer. We will do this by setting “ContentFormat”

This command could be combined with the above.

wecutil ss “DC – Events” /cf:Events

Now that I have the subscription setup I amd going to make a GPO to setup the Domain Controllers to receive the Subscription. Start Group Policy Management and create a new GPO linked to the OU that has the computers you are collecting the Events from… in my case it’s the Domain Controllers OU. In this GPO we need to set things in the “Event Forwarding”, WinRM Client” and WinRM Service”.

Event Forwarding:

“Configure the Configure the server address” Enable and add the collection server to the list.

Server=http://<Event Collectors FQDN>:5985/wsman/SubscriptionManager/WEC

WinRM Client:

“Trusted Hosts”

If you enable this policy setting, the WinRM client uses the list to determine if the destination Event Collector is a trusted entity.

WinRM Service:

“Allow automatic configuration of listeners”

Enter * for both

In order for the “Network Service” account to be able to access the Security log it needs to be in the “Event Log Readers” group.

I also added the “Domain Controllers” group (not sure if you need to do this but I was having a hard time getting this to work, see below for other errors I fixed )

You now MUST reboot each DC as the “Network Service” needs to update its group membership.

If all is well you should start seeing events in the “Forwarded Events”

And you should see all your DCs listed in Runtime Status. (this may take a few minuets to populate)

While setting this up I ran across a lot of errors and problems below is some of the errors and how I think I fixed them. Altogether I spent about 3 days setting up and troubleshooting this. I hope yours goes smother than mine!

*********************************************************

Windows Remote Management 10128

This command will show ACLs in the HTTP:

Netsh http show urlacl

After running this command you can see that http://+80/wsman is not listed

You can run run this to add if you wish and it will remove this error:

Netsh http add urlacl url=http://+80/wsman user=”Network Service”

You can do this for the the https one also

Netsh http add urlacl url=https://+443/wsman user=”Network Service”

***********************************************************

Windows Remote Management 10154

This one is because the Network Service does not have access to verify that the SPN is there.

This can be fixed by running:

dsacls “CN=AdminSDHolder,CN=System,DC=yourdomainname,DC=tld” /G “S-1-5-20:WS;Validated write to service principal name”

Example if you were microsoft

dsacls “CN=AdminSDHolder,CN=System,DC=microsoft,DC=com” /G “S-1-5-20:WS;Validated write to service principal name”

Reboot seems to be required…

*****************************************************************

Eventlog-ForwardingPlugin 105

(this one sucked)

This one was caused by a system level proxy setting. Removing this setting fixed the error.

Use this command to see if there is a system level proxy

Netsh winhttp show proxy

Use this command to remove the system level proxy

Netsh winhttp reset proxy

The above also caused the error below when trying to test connectivity with the collector.

The WinRM client cannot process the request because the server name cannot be resolved.

After removing the system proxy this error also went away and I was able to connect to the collector using the command line

***********************************************************

Eventlog-ForwardingPlugin 102

(this one sucked too REBOOT!!!!)

Error code is 5004

This one seems to have been caused by “Network Service” not having permissions to the security log.

Add “Network Service” to “Event Log Readers” group. I also added the “Domain Controllers” group as I am pulling the sec logs from them (not sure if you need to do this.)

This one requires a reboot, as group membership is set at logon and “Network Service” logs on at start up.

***********************************************************

Windows Remote Management 129

(still happens but does not seem to effect anything)

This one seems to not be a “real” error as it does not effect anything. This collection server is sending back a http 204 code. This seems to be normal behavior.

********************************************************************

RPC server is unavailable when running wecutil

For me at lest this is because the “Windows Event Collector” service was not running after starting the service this command runs fine.

SuperG

You can try this to get current logs:
WECUtil SS /cm:Custom /ree:true
REE is ReadExistingEvents
/ree:[VALUE]
A value that specifies which events are to be delivered for the subscription. VALUE can be true or false. When VALUE is true, all existing events are read from the subscription event sources. When VALUE is false, only future (arriving) events are delivered. The default is true when /ree is specified without a value, and the default is false if /ree is not specified.

I have found that this really needs to be set BEFORE any computers subscribe, if it is set AFTER then the computer does not send the old logs.

AD Undelete 2008R2 (Access is Denied)

In a previous blog I wrote about Deleted AD Objects and this one touches on that a little.

This morning a colleague of mine had a request to undelete a computer that had been deleted from the network.

Using powershell he attempted to recover the object but got an “Access is Denied” error.

Now, he does not have problems un-deleting other objects so its not an issue with the “deleted Objects”

OU and the object was not deleted very long ago so it has not dropped from the

Recycle Bin(default 180 days).

In fact he got the ObjectGUID by running the first part of the recover command using the Object Name.

So that means he can see the deleted object, he just can’t recover it. The account he is using is a Domain Admin account,

so why is he getting the Access is Denied message?

Not having all the information about the Object before it was deleted, I decided to check and see if I could find a

little more information about the Object. The “deleted Objects” OU is not viewable in most AD tools but you can use

LDP.exe to see the contents.

Start ldp.exe as an administrator.

Use the Connection menu in Ldp to connect and bind to a domain controller.

(Binding as logged on user if running LDP as administrator)

On the Options menu, click Controls.

In the Load Predefined list, click Return Deleted Objects.

Note: The 1.2.840.113556.1.4.417

control moves to the Active Controls window.

Under Control Type, click Server

, and the click OK.

On the View menu, click Tree

, type the distinguished name path of the deleted objects container in the domain where the deletion occurred,

and then click OK.

Note: The distinguished name path is also known as the DN path. For example,

if the deletion occurred in the contoso.com domain, the DN path would be the following path:

cn=deleted Objects,dc=contoso,dc=com

In the left pane of the window, double click the Deleted Object Container.

Note: As a search result of Idap query, only 1000 objects are returned by default.

For example, if more than 1000 objects exist in the Deleted Objects container, not all objects appear in this container.

If your target object does not appear, use ntdsutil, and then set the maximum number by using maxpagesize to get the search results .

A better option is to use the DN of the object you are looking for

Example:

CN=ComputerName\0ADEL:ce1dcaee-3041-4ece-aa9b-5fc3982d1afc,CN=Deleted Objects,DC=contoso,DC=com

This will give a lot more information about the object, including where the object came from.

lastKnownParent

Looking at this OU I found that for some reason Domain Admins was removed from the OU. After restoring the permissions , my colleague was able to recover the object
Super G