In short

An IPSec VPN is a virtual private network built on IPsec (Internet Protocol Security), a suite of open standards that encrypts and authenticates traffic at the network layer of the internet. Two protocols do the core work: ESP (Encapsulating Security Payload), which encrypts data, and AH (Authentication Header), which only verifies integrity and is now discouraged by NIST. Keys are negotiated automatically, most often with IKEv2. IPSec VPNs are most common in site-to-site and enterprise remote access setups.

The full explanation

What IPsec actually is

IPsec stands for Internet Protocol Security. It is a suite of open standards for keeping communications private over public networks, and it is the most common network-layer security control in use today. Its main jobs are to encrypt Internet Protocol (IP) traffic between hosts and to create virtual private networks.

A VPN, or virtual private network, is a virtual network built on top of existing physical networks. It provides a secure way to send data and control information between computers or networks that are not on the same physical infrastructure, such as two offices connected only through the public internet.

One important distinction: people often say “IKEv2 VPN” or “IPsec VPN” as if they were different technologies, but terms such as “IPsec VPN,” “IKEv2 VPN,” and “L2TP/IPsec” all refer to IPsec-based VPN connections. IKEv2 is simply the protocol most commonly used to set them up.

The preferred spelling of the standard is “IPsec.” Other capitalizations, such as IPSec or IPSEC, are officially deprecated in the standards documents, although the alternate capitalization remains common in everyday writing. On this page, “IPsec” follows the standard; “IPSec VPN” reflects the common spelling of the search term.

Why the network layer matters

To understand IPsec, it helps to know where it operates. Internet communication is organized in layers: applications such as web browsers sit at the top, the transport layer handles connections for those applications, and the network layer, where the Internet Protocol itself lives, routes packets across networks.

IPsec works at the network layer. That placement is significant because it means IPsec protects all traffic carried over IP, no matter which application produced it. A web browser, an email client, a database replication process, and a voice call all get the same protection without any of them needing to know the VPN exists. This is one of the reasons IPsec became the standard choice for connecting whole networks rather than securing individual applications.

The security services IPsec provides

According to the architecture document that defines IPsec, it offers a defined set of security services:

  • Access control: deciding which traffic is allowed through.
  • Connectionless integrity: detecting whether packets have been altered in transit.
  • Data origin authentication: confirming that packets come from the expected sender.
  • Anti-replay protection: detecting and rejecting packets captured and resent by an attacker.
  • Confidentiality: encrypting traffic so outsiders cannot read it.
  • Limited traffic flow confidentiality: reducing how much information about communication patterns leaks.

Not every IPsec deployment uses all of these. The exact mix depends on how the connection is configured.

The two protocols: AH and ESP

IPsec does its packet-level work through two protocols: the Authentication Header (AH) and the Encapsulating Security Payload (ESP).

AH: authentication without encryption. The Authentication Header provides integrity and authentication for IP packets but provides no encryption. It authenticates as much of the IP header as possible, along with the packet contents. However, some IP header fields change as packets travel across networks, and the values of those fields cannot be known in advance by the sender, so AH’s protection of the header is necessarily partial. AH is identified by IP protocol number 51.

ESP: encryption plus authentication. The Encapsulating Security Payload can provide confidentiality through encryption, plus data origin authentication, integrity, and anti-replay protection. The standards allow ESP to be used in three ways: encryption only, integrity only, or encryption with integrity. Encryption-only operation is permitted by the specification but defends mainly against passive attackers; combining encryption with a strong integrity mechanism is the expected practice. ESP is identified by IP protocol number 50.

AH is on the way out. In its 2020 guidance, the US National Institute of Standards and Technology places AH in the legacy column and states plainly that it discourages AH’s use: it should not be implemented or deployed, and integrity-only ESP should be used instead when encryption is not desired. For new deployments, ESP is the protocol to understand.

Which algorithms does NIST recommend?

IPsec itself does not fix which ciphers get used; the two sides agree on them when the security association is negotiated, as the IKEv2 section describes. NIST’s 2020 guidance fills in that choice with a table of approved algorithms and options. For encryption, it recommends four AES modes with 128, 192, or 256-bit keys: AES-GCM, AES-CTR, AES-CBC, and AES-CCM. Triple DES (TDEA) sits in the legacy column, and NIST recommends upgrading any remaining TDEA use to AES. For integrity and key derivation, the recommended choices are the HMAC-SHA-2 family (HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512) plus AES-GMAC; HMAC-SHA-3 is listed as expected, a future addition, while HMAC-SHA-1 is legacy. AES-GCM deserves a note because it is an AEAD algorithm: it provides confidentiality and integrity in a single operation, while AES-CBC provides only encryption and must be paired with a separate integrity algorithm such as HMAC-SHA-256. For peer authentication, NIST recommends RSA, DSA, or ECDSA at 128-bit security strength, for example RSA with 3072-bit or larger keys. For the Diffie-Hellman exchange itself, groups 14 through 21 are recommended, with the newer groups 31 and 32 listed as expected.

Tunnel mode vs. transport mode

IPsec can be applied to packets in two modes, and this distinction is central to understanding how IPSec VPNs work.

Transport mode protects only the payload of each packet. The ESP or AH header is inserted after the original IP header and before the next-layer header (for example, the TCP header). The original IP addresses remain visible, so an observer can see which endpoints are communicating, but the content is protected. Transport mode is used for host-to-host connections, where the two endpoints themselves perform the protection.

Tunnel mode wraps the entire original IP packet, header and all, inside a new IP packet with a new IP header. The original addresses are hidden inside the encrypted payload, which gives observers less information about the actual endpoints and traffic patterns. This is the mode used for VPN tunnels between gateways, and it is the mode NIST recommends.

A useful way to remember the difference: in transport mode, IPsec protects what is being said but leaves the envelope’s addressing visible; in tunnel mode, IPsec seals the entire letter inside a new envelope.

Security associations: the rules of each connection

Every IPsec connection is governed by a security association (SA), which is the shared agreement between the two sides about how traffic is protected. An SA records the cryptographic algorithms in use, the keys, and parameters such as the anti-replay window. Each SA is one-directional: a full conversation between two endpoints requires one SA in each direction.

SAs are identified by three values: the destination address, the security protocol in use (AH or ESP), and a number called the Security Parameters Index (SPI). Alongside the SAs, IPsec implementations maintain a security policy database, which decides what each packet should receive: pass it through unprotected, discard it, or protect it with IPsec.

For most users this machinery is invisible, but the concept matters because it explains how the two sides of a VPN consistently apply the same protection to thousands of packets without negotiating each one individually.

IKEv2: the key exchange that makes it automatic

Security associations need keys, and keys need to be created and refreshed securely. That is the job of the Internet Key Exchange protocol, now in version 2 (IKEv2).

IKEv2 performs two exchanges to establish a connection:

  1. IKE_SA_INIT. The two sides exchange nonces and perform an ephemeral Diffie-Hellman exchange. This creates a shared seed key from which further keys are derived, and establishes the IKE security association itself, a secure control channel.
  2. IKE_AUTH. Over that protected channel, the sides authenticate each other and negotiate the child security association that will carry the actual data traffic: which IPsec protocol (typically ESP), which encryption algorithm, and which integrity algorithm.

Because IKEv2 uses an ephemeral Diffie-Hellman exchange, it provides perfect forward secrecy: session keys are not derived from long-term keys, so if long-term credentials are ever compromised, previously recorded traffic still cannot be decrypted. NIST recommends that rekeying use Diffie-Hellman exchanges of the same or greater strength, and suggests a security association lifetime of about 8 hours.

IKE runs over UDP ports 500 and 4500. Port 4500 comes into play when network address translation (NAT) is detected on the path: ESP as a raw IP protocol does not always pass cleanly through NAT devices, so the encrypted packets are wrapped in UDP to get through. This mechanism is why IPSec VPNs generally work from behind home routers and hotel Wi-Fi, environments where NAT is nearly universal.

IKEv1 and IKEv2: what changed

IKEv2 replaced the original IKE, which the standard describes as defined across three documents (RFCs 2407, 2408, and 2409), and the replacement was not backward compatible. The revision was mostly simplification. Where IKEv1 had eight different initial exchanges, IKEv2 has a single four-message exchange: the IKE_SA_INIT and IKE_AUTH pair described above, so connections set up with lower latency. IKEv2 also made the protocol reliable, with every message acknowledged and sequenced, which cut the child-SA exchange from three messages to two. And it increased robustness by holding back significant server-side work until the initiator proves it can receive messages at its claimed address, so a responder cannot be tricked into expensive work by an initiator that cannot even receive replies. NIST’s recommendation table places IKEv1 in the legacy column; within IKEv1, its main mode and quick mode exchanges were the sanctioned choices, while aggressive mode was already legacy. IKEv2 itself is now an Internet Standard.

What is L2TP/IPsec?

The introduction named L2TP/IPsec as one of the common labels for an IPsec VPN. L2TP, the Layer 2 Tunneling Protocol, is the successor to the older PPTP protocol: it encapsulates PPP packets inside UDP on port 1701, but for confidentiality and integrity of the data plane it depends on IPsec. (Some implementations add encryption at the PPP layer, which must confusingly be switched off so that IPsec can do the actual protecting.) Every real L2TP VPN therefore runs IPsec in transport mode underneath the L2TP layer, which is why the combination is called L2TP/IPsec; L2TP on its own is used by some ISPs, but that use is not a VPN. The IPsec layer is usually set up with IKEv1, often with a weak pre-shared key, although certificates are possible too. NIST’s assessment is blunt: even when deployed securely, L2TP/IPsec offers no advantage over IKEv2-based IPsec VPNs. Its extra encapsulation layers shrink the effective MTU and cause fragmentation problems, its transport-mode design works poorly behind NAT, and its implementations usually lack modern AEAD ciphers such as AES-GCM, which costs extra CPU and battery power on mobile devices. NIST recommends migrating L2TP/IPsec deployments to IKEv2-based IPsec VPNs.

Where IPSec VPNs are used

NIST’s guidance describes three primary IPsec VPN architectures:

  • Gateway-to-gateway (site-to-site). An encrypted tunnel connects the security gateways of two networks, such as a headquarters and a branch office. Hosts on either side communicate as if they were on the same private network, without needing any VPN software themselves. This is the classic IPSec VPN, and it uses tunnel mode.
  • Remote access. Individual devices, such as a laptop at home or on public Wi-Fi, connect to the corporate gateway. The tunnel runs between the user’s device and the gateway, giving the remote worker access to internal systems as if they were in the office.
  • Host-to-host. Two individual computers protect their traffic directly, end to end. This uses transport mode and is the least common of the three in enterprise VPN practice.

Beyond VPNs, IPsec also appears as a component inside many other protocols and technologies, including mobile phone networks, software-defined networking, and routing protocol protection.

IPSec VPNs vs. SSL/TLS VPNs

The most common alternative to IPsec is the SSL VPN. Despite the name, these now use TLS (the successor to SSL) rather than the older SSL protocol. TLS VPNs can run on TCP or, in the case of DTLS, on UDP.

The main practical difference is where protection happens. TLS operates at the transport layer, typically protecting application sessions such as web traffic, while IPsec operates at the network layer, protecting everything carried over IP. TLS-based VPNs have one notable advantage: their traffic can run on any configured port, usually 443 (the HTTPS port), which most networks allow through without inspection. That makes TLS VPN traffic much harder to block. On the other hand, when TLS VPNs run over TCP, they can suffer severe performance degradation under congestion because of two stacked TCP layers doing their own retransmission; DTLS over UDP avoids this.

TLS VPNs are usually implemented as applications, which can mean lower performance than kernel-based IPsec implementations, though this depends on the specific software. NIST publishes separate guidance for TLS-based VPNs, and organizations choose between the two based on their needs: IPsec for network-wide or gateway-level protection, TLS VPNs when easy traversal of restrictive networks or browser-based access matters most.

IPSec VPNs and consumer VPN apps

Many commercial VPN apps offer “IKEv2/IPsec” as one of several protocol choices, often alongside newer protocols such as WireGuard. In that context, IPsec is used for the remote-access architecture: the app on the user’s device builds an encrypted tunnel to the provider’s server, and all traffic is routed through it.

It is worth keeping perspective here. A consumer VPN app that uses IPsec protects the path between the device and the VPN provider’s server, not the path from the provider’s server to the destination website. The encryption the VPN provides stops at the provider. Any site that does not itself use HTTPS can still see and modify the user’s traffic beyond that point. This is true of every consumer VPN protocol, not just IPsec; it is a property of how the architecture works, not a flaw in the protocol.

How an IPSec VPN connection gets set up, step by step

Pulling the pieces together, here is what happens when an IPSec VPN connection is established:

  1. Policy decision. The initiator’s security policy database determines that traffic to the destination must be protected with IPsec.
  2. IKE_SA_INIT. The two sides exchange Diffie-Hellman parameters and nonces over UDP port 500, establishing the secure IKE control channel.
  3. IKE_AUTH. The sides authenticate each other and agree on the child security association: ESP in tunnel mode (typically), with negotiated encryption and integrity algorithms.
  4. Data flows. Packets are encrypted and authenticated according to the security association. Each packet carries a sequence number so the receiver can reject replays.
  5. Rekeying. Before the security association’s lifetime expires (NIST suggests about 8 hours), IKE negotiates fresh keys automatically, keeping the connection secure without interrupting it.

If a NAT device is detected during setup, both sides switch to UDP port 4500, wrapping ESP packets in UDP so they pass through the translator.

Limitations to be aware of

No security technology is complete, and IPsec’s limitations are well documented:

  • AH does not encrypt. Any remaining AH-based deployment authenticates traffic but leaves it readable. NIST’s guidance says not to deploy AH at all.
  • Encryption without integrity is fragile. The ESP specification explicitly warns that using encryption alone defends mainly against passive eavesdropping and can be insecure against active attackers.
  • End-to-end protection is not guaranteed. IPsec secures traffic between its two endpoints. If those endpoints are gateways, traffic inside each local network travels unprotected unless those networks are secured separately.
  • Configuration complexity. IPsec’s strength comes with real administrative cost. Weak pre-shared keys, outdated proposals kept for compatibility, and loose traffic selectors are the typical sources of real-world failures, not breaks in the cryptography itself.
  • NAT requires accommodation. Raw ESP does not always traverse NAT devices, which is why the UDP encapsulation mechanism exists and why some restrictive networks block it.

Summary

An IPSec VPN is a virtual private network built on IPsec, a suite of open standards that secures traffic at the network layer. ESP encrypts and authenticates packets; AH authenticates without encrypting and is now discouraged by NIST. Tunnel mode wraps whole packets for gateway-to-gateway VPNs, while transport mode protects host-to-host traffic directly. IKEv2 automates key exchange with perfect forward secrecy, and the whole system is standardized by the IETF with implementation guidance from NIST. For connecting networks and providing enterprise remote access over untrusted networks, it remains the default choice, with TLS-based VPNs as the most common alternative.

Frequently asked questions

Is an IPSec VPN the same as IKEv2?

IKEv2 is the key exchange protocol most commonly used to set up IPsec-based VPNs. “IKEv2 VPN” and “IPsec VPN” both describe the same kind of connection; the terms refer to the setup protocol and the packet-protection standards respectively.

What is the difference between AH and ESP?

AH (Authentication Header) provides authentication and integrity checking but no encryption. ESP (Encapsulating Security Payload) provides encryption and can also provide authentication and integrity. ESP is the protocol in current use; NIST’s 2020 guidance discourages new AH deployments.

What is tunnel mode vs. transport mode in IPsec?

In transport mode, IPsec protects the packet payload but leaves the original IP addresses visible; it is used for host-to-host connections. In tunnel mode, the entire original packet is encapsulated in a new IP packet, hiding the original addresses; it is used for gateway-to-gateway VPN tunnels and is the mode NIST recommends.

Which ports does IPSec use?

IKE, the key exchange protocol, runs over UDP ports 500 and 4500. Port 4500 is used when network address translation (NAT) is detected, with ESP packets wrapped in UDP so they pass through the translator. ESP data traffic itself uses IP protocol 50 and AH uses IP protocol 51.

Is IPSec VPN better than a TLS (SSL) VPN?

Neither is universally better; they solve different problems. IPsec protects all traffic at the network layer and is the standard choice for connecting networks and providing enterprise remote access. TLS VPNs operate at the transport layer, are typically easier to deploy for individual users, and their traffic is harder to block because it usually runs on the HTTPS port 443.

Does an IPSec VPN hide my traffic from my internet provider?

An IPSec VPN encrypts traffic between the device and the VPN endpoint, so the internet provider cannot read its contents. The provider can still see that encrypted traffic is flowing to the VPN endpoint. With a consumer VPN service, the VPN provider itself can see the traffic once it is decrypted, so the protection shifts trust from the internet provider to the VPN provider.

Sources & references

  • National Institute of Standards and Technology (NIST). SP 800-77 Rev. 1, Guide to IPsec VPNs. June 2020. DOI: 10.6028/NIST.SP.800-77r1.
  • Kent, S. and Seo, K. RFC 4301: Security Architecture for the Internet Protocol. IETF, December 2005.
  • Kent, S. RFC 4302: IP Authentication Header. IETF, December 2005.
  • Kent, S. RFC 4303: IP Encapsulating Security Payload (ESP). IETF, December 2005.
  • Kaufman, C. et al. RFC 7296: Internet Key Exchange Protocol Version 2 (IKEv2). IETF, October 2014.

Emma Hudson

Editor of Amberaisle — clear, thoughtful explainers for curious readers. More about the editor →