Interactive Guide to Authentication Protocols

🌐 Protocol Landscape

Active Directory environments rarely rely on a single authentication method. Understanding where each protocol fits prevents mismatched expectations and security gaps.

Integrated

Kerberos is the native Windows protocol. It delivers mutual authentication and delegated access inside trusted forests.

Legacy

NTLM handles fallbacks, workgroup joins, and down-level systems. It lacks ticketing and is vulnerable to relay attacks.

Federated

Protocol hand-offs (SAML, OpenID Connect, OAuth) extend sign-in beyond the domain and power modern SaaS SSO.

📌 Key takeaway: Map protocols to trust boundaries. Use Kerberos where the domain controls both client and service. Require federation for cloud apps. Limit NTLM to transitional scenarios and monitor it closely.

🔑 Kerberos Ticket-Granting Ticket (TGT) Flow

Kerberos uses symmetric keys and short-lived tickets. Follow the handshake below to see how the client, Key Distribution Center (KDC), and services exchange secrets.

Handshake Detail

Select a step on the left to see what is exchanged, the encryption keys involved, and typical failure codes.

Quick Checks

Validate SPNs

Duplicate or missing SPNs break TGS-REQ. Run setspn -X to catch duplicates.

Ticket Lifetime

Check klist for TGT expiry. Desynced time causes KRB_AP_ERR_SKEW.

PowerShell Quick Audit

Get-ADUser -Filter * -Properties ServicePrincipalName | Where-Object { $_.ServicePrincipalName -match "/" } | Select-Object Name, ServicePrincipalName | Export-Csv .\spn-audit.csv -NoTypeInformation

🛑 NTLM Challenge/Response

NTLM is challenge/response based. It lacks mutual authentication and is vulnerable to relay and credential forwarding.

Flow Snapshot

Client sends a NEGOTIATE message listing supported features.
Server responds with a challenge containing an 8-byte nonce.
Client encrypts the nonce with the user's NT hash and returns the response. Server validates via the domain controller.

Detect Usage

Enable NTLM auditing and monitor security events 4624 (logon) and 4776 (credentials validated).

Mitigate Risk

Prefer Kerberos, enable SMB signing, and use Protected Users or Extended Protection for Authentication to block relays.

GPO Baseline

# Set NTLM to audit mode first Set-ItemProperty -Path "HKLM:\\SYSTEM\\CurrentControlSet\\Control\\Lsa" -Name "RestrictSendingNTLMTraffic" -Value 2 # Harden LM/NTLM responses Set-ItemProperty -Path "HKLM:\\SYSTEM\\CurrentControlSet\\Control\\Lsa" -Name "LmCompatibilityLevel" -Value 5

📁 LDAP Bind Options

LDAP binds control how service accounts authenticate to directory services. Choose the right bind type for the sensitivity of data and the network path.

Simple Bind

Username and password are sent in clear text. Only safe over LDAPS (636) or StartTLS.

SASL / GSSAPI

Uses Kerberos for authentication. Supports channel binding and mutual auth.

Anonymous

Very limited access. Disable unless required for public directories.

TLS Checklist

Install a certificate with appropriate EKU (Server Authentication) on each DC.
Enable LDAP signing via Group Policy (Domain Controller: LDAP server signing requirements).
Test StartTLS with ldp.exe or Test-LdapStartTls.

🪪 Federation & Token-Based Access

When identities move beyond the domain boundary, claims-based protocols take over. Tokens carry signed assertions instead of shared secrets.

SAML 2.0

XML assertions delivered via browser redirects. Common for SaaS SSO integrations.

OpenID Connect

OAuth 2.0 + ID tokens. JSON payloads, ideal for modern web and mobile apps.

OAuth 2.0

Delegated authorization. Issues access tokens for APIs. Often paired with OIDC.

Token Anatomy

{ "aud": "https://graph.microsoft.com/", "iss": "https://sts.contoso.com/", "exp": 1712192047, "iat": 1712188447, "sub": "a1b2c3d4", "roles": ["User.Read", "Mail.Read"], "upn": "alex@contoso.com" }

Validate the signature (alg + kid), issuer, audience, and lifetime before exchanging tokens for backend calls.

🧭 Decision Matrix

Use the quick matrix below to choose the correct protocol for a workload.

Scenario	Recommended Protocol	Key Notes
Domain-joined client to file share	Kerberos	Supports mutual auth, constrained delegation, and automatic ticket renewals.
Legacy device without domain trust	NTLM (temporary)	Plan remediation. Limit to specific hosts using firewall rules.
3rd-party SaaS integration	SAML or OpenID Connect	Use Azure AD/ADFS as IdP. Map claims to app roles.
Custom line-of-business API	OAuth 2.0 + OIDC	Issue access tokens and use refresh tokens for long-lived sessions.
Directory service query by an app	LDAP w/ SASL	Require TLS, channel binding, and least-privilege service accounts.

🧪 Troubleshooting Playbook

Authentication failures are often timing, SPN, or policy related. Use the playbook to triage quickly.

Kerberos

Error KRB_AP_ERR_MODIFIED: Check SPNs, duplicate machine accounts.
KRB_ERR_CLIENT_REVOKED: Account disabled or password expired.
Use klist purge to clear cached tickets.

NTLM

Audit failures with Event IDs 4625 (logon) and 4771 (Kerberos pre-auth).
Verify channel binding on IIS/SMB to block relays.
Network traces expose NTLMSSP_NEGOTIATE packages.

Federation

Check token signing certificates and clock skew between IdP and SP.
Validate audience/issuer using jwt.ms or jwt.io.
Review claim rules for missing attributes.

Packet Capture Cheatsheet

Kerberos Filter

udp.port == 88 || tcp.port == 88

NTLM Filter

ntlmssp

LDAP StartTLS

ldap && tcp.port == 389

SAML POST

http.request.method == "POST" && frame contains "SAMLResponse"