Menu Close

ISE PSN Got Stuck at Step 4

I came across a couple of issues during an ISE upgrade project I was working on recently which are not really common and you would rarely see them. These issues were, an ISE PSN got stuck at step 4 during the upgrade process, and a PAN went out of synch. This project was to upgrade a distributed ISE cube with six nodes from version 2.2 to 2.4 and all appliances were virtual.

Redirect ACL With C9300 Switches

In this post I will share with you one caveat and its fix with redirect ACL with C9300 switches. In the last few months I was working on a project for a medium size customer. The main requirements were to implement Firepower IPS, dot1x, pxGrid, AnyConnect client provisioning and posture assessment for both VPN and local clients. The customer has a few sites spread across the globe, and all of them are connected through VPLS. There are different network devices that we were working on and in one of the sites we had a stack of Cisco C9300 switches. The customer has ISE deployed for identity management.

FMC External Authentication with RADIUS

In this post, I am going to show you how to set up FMC external authentication with RADIUS. Why we would need that?!, simply put, to have a scalable solution in our environment that will allow us to manage accesses to our FMC appliance. Even if we configure the FMC with an external authentication server, we do still have the local admin account enabled that we can use in case the external authentication server is down.

Packet Capture in FMC

In this post, I am going to show you how to run a packet capture on Cisco Firepower Management Center (FMC). As we know, both FTD and FMC are Linux based which means we can rely on a few tools that are embedded in Linux operating system. In fact, when you log into the FMC or when you go into Expert mode on FTD, you will see that the majority of the commands you use are simply Linux commands.

Crypto Keypair without Domain Name

In this post I am going to show you how to configure crypto keypair without configuring host name or domain name on Cisco devices. A few network admins still have some confusion about if configuring the domain name on Cisco devices is a requirement to generate a crypto keypair or not. I believe this confusion comes from the error we get when we try to create a crypto keypair on Cisco devices before we’ve configured the domain name. The error would explicitly ask us to define the domain name first to generate the crypto keypair.

Snort HOME_NET and EXTERNAL_NET Variables

What are the Snort HOME_NET and EXTERNAL_NET Variables?! To know that let’s see how Snort rules work. Snort rules rely on variables to know what traffic they should inspect and what to ignore. Each Snort rule has a header where a bunch of variables are defined such as the action to be taken, protocol, source IP, source port, destination IP and destination port. The most important two bits among these variables are the source and destination IP addresses.

FDM Multiple Admin Accounts

In this post, I am going to show you how creating multiple admin accounts on FDM for GUI accesses can be possible by using some tools you would most likely have in your environment. First, as we know Firepower Device Management (FDM) does not support creating multiple admin accounts for FDM GUI accesses. This is a known limitation and as a result it would mean that all the admins will use the same admin account to log into the FTD. Of course this would lead to share the admin account credentials between the admins which could potentially breach our security.

ASA Site-to-Site VPN Failover “Preemption”

As we know, Cisco ASA IPsec site-to-site VPN preemption is not supported on Cisco ASA. Therefore, this means if the primary VPN peer recovers from a failure the VPN tunnel will remain active with the secondary VPN peer. In other words, if you configure a site-to-site VPN tunnel crypto map with two peers, one as the primary, and another as the secondary, the ASA will always try to initiate the tunnel with the primary peer first. If the primary peer fails and become unreachable, then the ASA will initiate the tunnel with the secondary peer.

When the primary peer comes back online, the ASA will not do anything, as long as the secondary peer is reachable. In other words, the primary peer will not take over the control from the secondary, instead the tunnel will remain initiated with the secondary peer.

In some cases, this would cause performance issues, or even productivity issues. Let’s assume the following scenario. Say we have two sites, one is a primary with our production environment, and another as a disaster recovery site with a limited production environment.

Or, let’s assume that we have same data replica across both the primary and the disaster recovery sites, but the secondary VPN peer connected to the disaster recovery site has less resources over the primary. Or maybe the WAN connection where the secondary peer is connected is much slower than the primary site. let’s see together how we can apply a workaround to get the Cisco ASA IPsec site-to-site VPN preemption to work.

 

Recommendation

 

For Cisco ASA IPsec site-to-site VPN preemption Cisco recommends configuring a SLA monitor to keep track of the primary peer IP address and creating a conditional route of the primary peer IP address tied to the SLA monitor. Then, to create an EEM applet to trigger the command “clear crypto ipsec sa peer <the secondary peer IP address>”. However, this command must be run only after the  the message ID 622001 is seen twice on the ASA. More about message ID 622001 below.

The command in itself will clear the tunnel with the secondary peer and as a result it will re-initiate the tunnel with the primary peer.

 

Message ID 622001

 

The message ID 622001 is generated every time the ASA adds or removes a route from its routing table. In our example, this means the ASA will generate the first message ID 622001 when the primary peer fails, and another message ID 622001 when the primary peer comes back online.

The message ID 622001 looks similar to these:

 

%ASA-6-622001: Adding tracked route 60.60.60.1 255.255.255.255 169.254.255.254, distance 1, table default, on interface outside

 

Or

 

%ASA-6-622001: Removing tracked route 60.60.60.1 255.255.255.255 169.254.255.254, distance 1, table default, on interface outside

 

Issue

 

The problem with that recommendation would arise if you have multiple conditional routes on the ASA that are tied to multiple SLA monitors. The reason behind this, is because as mentioned above every time the ASA adds or removes a route from/to its routing table, it will generate a message ID 622001. Therefore, the ASA will not make any distinction between which routes could cause the message ID 622001 generation.

As a result, the EEM applet would trigger the command to clear the VPN tunnel with no valid reason or even when it is not required. In other words, the EEM will trigger the clear crypto ipsec command each time the ASA generates two message ID 622001. That happens regardless of which conditional route has been added or removed to the ASA routing table.

 

Solution

 

Here is a workaround to make the ASA always initiate the VPN tunnel with the primary peer, as long as it is reachable. What I would do is configuring a SLA monitor, checking the availability of the primary peer, and creating a conditional route for the secondary peer pointing to a dummy next hop.

This means that as long as the primary peer IP address is reachable, the secondary peer IP route will always be installed in the routing table pointing to a fake next hop. In other words the secondary peer IP will never be reachable as long as the primary peer IP is alive.

When the primary peer fails, the track of the SLA monitor will cause the removal of the conditional route of the secondary peer that was pointing to the dummy next hop. As a result, now the route that will be used to reach the secondary peer will be the default route configured on the ASA.

When the primary peer comes back online, the SLA monitor will go again in UP state, and it will trigger the installation of the secondary peer route again. So, the ASA will add back the fake next hop route into its routing table to reach the secondary peer.

According to those route removals and additions, the VPN tunnel will automatically switch between the peers. When the primary peer is reachable, the VPN tunnel will always be initiated with the primary peer. And when the primary peer is unreachable, the VPN tunnel will be initiated with the secondary peer.

Here is the topology and the configuration I have used in my lab to prove this.

 

Topology

 

Cisco ASA IPsec site-to-site VPN preemption

 

ASA IPsec Site-to-Site VPN Preemption Configuration

 

asav-1:

 

HQ-ASA# sh ip
System IP Addresses:
 
Interface                Name              IP address      Subnet mask    Method
GigabitEthernet0/0      inside            192.168.0.1      255.255.255.0  CONFIG
GigabitEthernet0/1      outside            10.10.10.1      255.255.255.0  CONFIG

 

object-group network VPN-LOCAL
 network-object 192.168.0.0 255.255.255.0
object-group network VPN-REMOTE
 network-object 192.168.130.0 255.255.255.0
 
access-list VPN extended permit ip object-group VPN-LOCAL object-group VPN-REMOTE
 
nat (inside,outside) source static VPN-LOCAL VPN-LOCAL destination static VPN-REMOTE VPN-REMOTE no-proxy-arp route-lookup
 
crypto ipsec ikev1 transform-set ESP-AES256-SHA esp-aes-256 esp-sha-hmac
 
crypto map VPN 100 match address VPN
crypto map VPN 100 set peer 50.50.50.1 60.60.60.1
crypto map VPN 100 set ikev1 transform-set ESP-AES256-SHA
crypto map VPN interface outside
 
crypto isakmp identity address
crypto ikev1 enable outside
 
crypto ikev1 policy 10
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 86400
 
tunnel-group 50.50.50.1 type ipsec-l2l
tunnel-group 50.50.50.1 ipsec-attributes
 ikev1 pre-shared-key cisco?123
 
tunnel-group 60.60.60.1 type ipsec-l2l
tunnel-group 60.60.60.1 ipsec-attributes
 ikev1 pre-shared-key cisco?123
 
sla monitor 100
 type echo protocol ipIcmpEcho 50.50.50.1 interface outside
 timeout 1000
 frequency 3
sla monitor schedule 100 life forever start-time now
 
track 100 rtr 100 reachability
 
route outside 60.60.60.1 255.255.255.255 169.254.255.254 1 track 100
route outside 0.0.0.0 0.0.0.0 10.10.10.2 1

 

asav-2:

 

REMOTE-01# sh ip
System IP Addresses:
 
Interface                Name              IP address      Subnet mask    Method
GigabitEthernet0/0      outside            50.50.50.1      255.255.255.0  CONFIG
GigabitEthernet0/1      inside          192.168.130.1      255.255.255.0  CONFIG

 

object-group network VPN-LOCAL
 network-object 192.168.130.0 255.255.255.0
object-group network VPN-REMOTE
 network-object 192.168.0.0 255.255.255.0
 
access-list VPN extended permit ip object-group VPN-LOCAL object-group VPN-REMOTE
 
nat (inside,outside) source static VPN-LOCAL VPN-LOCAL destination static VPN-REMOTE VPN-REMOTE no-proxy-arp route-lookup
 
crypto ipsec ikev1 transform-set ESP-AES256-SHA esp-aes-256 esp-sha-hmac
 
crypto map VPN 100 match address VPN
crypto map VPN 100 set peer 10.10.10.1
crypto map VPN 100 set ikev1 transform-set ESP-AES256-SHA
crypto map VPN interface outside
 
crypto isakmp identity address
crypto ikev1 enable outside
 
crypto ikev1 policy 10
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 86400
 
tunnel-group 10.10.10.1 type ipsec-l2l
tunnel-group 10.10.10.1 ipsec-attributes
 ikev1 pre-shared-key cisco?123

 

asav-3:

 

object-group network VPN-LOCAL
 network-object 192.168.130.0 255.255.255.0
object-group network VPN-REMOTE
 network-object 192.168.0.0 255.255.255.0
 
access-list VPN extended permit ip object-group VPN-LOCAL object-group VPN-REMOTE
 
nat (inside,outside) source static VPN-LOCAL VPN-LOCAL destination static VPN-REMOTE VPN-REMOTE no-proxy-arp route-lookup
 
crypto ipsec ikev1 transform-set ESP-AES256-SHA esp-aes-256 esp-sha-hmac
 
crypto map VPN 100 match address VPN
crypto map VPN 100 set peer 10.10.10.1
crypto map VPN 100 set ikev1 transform-set ESP-AES256-SHA
crypto map VPN interface outside
 
crypto isakmp identity address
crypto ikev1 enable outside
 
crypto ikev1 policy 10
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 86400
 
tunnel-group 10.10.10.1 type ipsec-l2l
tunnel-group 10.10.10.1 ipsec-attributes
 ikev1 pre-shared-key cisco?123

 

Verification

 

First, let’s very the VPN tunnel on the HQ-ASA:

 

HQ-ASA# sh vpn-sessiondb l2l
 
Session Type: LAN-to-LAN
Connection  : 50.50.50.1
Index        : 11058                  IP Addr      : 50.50.50.1
Protocol    : IKEv1 IPsec
Encryption  : IKEv1: (1)AES256  IPsec: (1)AES256
Hashing      : IKEv1: (1)SHA1  IPsec: (1)SHA1
Bytes Tx    : 292572                Bytes Rx    : 292572
Login Time  : 00:21:21 UTC Tue Jul 24 2018
Duration    : 0h:58m:10s

 

Now, let’s break the link with the primary peer and see what happens:

 

HQ-ASA# %ASA-6-622001: Removing tracked route 60.60.60.1 255.255.255.255 169.254.255.254, distance 1, table default, on interface outside

 

As you can see, after the VPN tunnel with the primary timed out the ASA reinitiated the tunnel with the secondary peer:
 

HQ-ASA# sh vpn-sessiondb l2l
 
Session Type: LAN-to-LAN
Connection  : 60.60.60.1
Index        : 33744                  IP Addr      : 60.60.60.1
Protocol    : IKEv1 IPsec
Encryption  : IKEv1: (1)AES256  IPsec: (1)AES256
Hashing      : IKEv1: (1)SHA1  IPsec: (1)SHA1
Bytes Tx    : 3528                  Bytes Rx    : 3528
Login Time  : 01:23:06 UTC Tue Jul 24 2018
Duration    : 0h:00m:42s

 

Here is after we re-enabled the link with the primary peer:
 

HQ-ASA# %ASA-6-622001: Adding tracked route 60.60.60.1 255.255.255.255 169.254.255.254, distance 1, table default, on interface outside

 

Because the secondary peer is now not reachable anymore due to the fake next hop we have on the conditional router, the VPN tunnel with the secondary peer timed out and the ASA reinitiated the tunnel with the primary peer:
 

HQ-ASA# sh vpn-sessiondb l2l
 
Session Type: LAN-to-LAN
Connection  : 50.50.50.1
Index        : 56849                  IP Addr      : 50.50.50.1
Protocol    : IKEv1 IPsec
Encryption  : IKEv1: (1)AES256  IPsec: (1)AES256
Hashing      : IKEv1: (1)SHA1  IPsec: (1)SHA1
Bytes Tx    : 0                      Bytes Rx    : 0
Login Time  : 01:25:01 UTC Tue Jul 24 2018
Duration    : 0h:00m:01s

 

If you are wondering how I could use the ? in the pre-shared key for the VPN tunnels, then here is the trick: When you type a pre-shared key, and you want to use the ? as a character, it would be enough to type ctrl+v right before typing the ? and then you continue typing the rest of the pre-shared key. That is valid on the ASA for anywhere you might need to use the ? as a normal character, not only for pre-shared keys.

This wraps up this Cisco ASA IPsec site-to-site VPN preemption post. I hope you enjoyed reading this post, and as always, I would love to hear your feedback. Thanks for reading!

Privilege Level 15 with Cisco ISE

In this post, I’m going to show you how to assign privilege level 15 with Cisco ISE through RADIUS. We know Cisco ISE amazingly supports network devices administration through TACACS+ protocol which allows granting different access levels and managing what command sets could be run in each level. However, this feature requires an additional license called Device Administration to be installed on ISE.

Scroll To Top