Menu Close

ISE PSN Got Stuck at Step 4

I came across a couple of issues during an ISE upgrade project I was working on recently which are not really common and you would rarely see them. These issues were, an ISE PSN got stuck at step 4 during the upgrade process, and a PAN went out of synch. This project was to upgrade a distributed ISE cube with six nodes from version 2.2 to 2.4 and all appliances were virtual.

I was doing the upgrade through CLI which is my favorite option over the GUI because it really gives much more control and visibility. It is also much more reliable in my opinion because it prevents running into potential browsers dependent issues such as sessions timing out or unrealistic percentage bars when provisioning the ISE upgrade image.

As expected, the plan to do the upgrade was to start with the secondary PAN and then moving to the PSNs, and finally upgrading the new secondary PAN which was the old primary PAN.

When I started the upgrade everything was progressing as normal and so far there were no major issues. That was the case until one PSN got stuck at step 4 which took really too long before it failed the upgrade process.

However, the second issue was with the secondary PAN which for some reason was keeping one of the PSNs in its deployment even though that PSN was already upgraded and added to the new upgraded deployment.

Because of that I was not able to upgrade the secondary PAN since one of the requirements is to complete the upgrade of all the other nodes before starting or being able to upgrade the secondary PAN itself.

Along with Cisco TAC we spent some time doing some troubleshooting to try to find out the root cause of these issues with no luck. To fix the issue with the PSN got stuck at step 4 there were no many options apart from rebuilding the PSN or to reset its ISE application configuration.

However, regarding the the issue with secondary PAN, the two options were to try to adjust ISE database which was kind of a risky option since that would corrupt the database, or to reset ISE application configuration.

The option that was more reasonable and the less risky to fix the issues on both nodes was to reset their ISE application configuration. The command I used to do so was application reset-config ise.

What this command does is basically it resets only ISE application configuration to the factory defaults but without touching any network configuration such as IP addresses, FQDN, NTP, DNS, etc, and most importantly it asks you if you want to keep the certificates on the node.

 

Here is where the ISE PSN got stuck at step 4 which stayed in that state for so long before it failed the upgrade:

 

ISE-PSN-01/admin# application upgrade proceed
Initiating Application Upgrade…
% Warning: Do not use Ctrl-C or close this terminal window until upgrade completes.
-Checking VM for minimum hardware requirements
STEP 1: Stopping ISE application…
STEP 2: Verifying files in bundle…
-Internal hash verification passed for bundle
STEP 3: Validating data before upgrade…
STEP 4: De-registering node from current deployment.

 

Here are the system logs generated whilst the ISE PSN got stuck at step 4:

 

root: info:[application:operation:isehourlycron.sh] Waiting up to 20 seconds for lock: APP_UPGRADE to complete
root: info:[application:operation:isehourlycron.sh] Database is still locked by lock: APP_UPGRADE. Aborting. Please try it later
root: info:[application:operation:isehourlycron.sh] % Error: Another ISE DB process (APP_UPGRADE) is in progress, cannot perform cleanup at this time

 

These were the error messages after the upgrade process has failed:

 

ISE-PSN-01/admin#
% Error: De-registering node from current deployment failed!
Starting application after rollback…
% Manual rollback required: Perform the following steps to revert node to its pre-upgrade state:
– Ensure that node is still present in current deployment from Primary UI; if it is not present, register this node back again.
% Application install or upgrade cancelled.

 

This screen shows when the secondary PAN was trying to do the upgrade:

 

ISE-PAN-01/admin# application upgrade proceed
Initiating Application Upgrade…
% Warning: Do not use Ctrl-C or close this terminal window until upgrade completes.
-Checking VM for minimum hardware requirements
STEP 1: Stopping ISE application…
STEP 2: Verifying files in bundle…
-Internal hash verification passed for bundle
STEP 3: Validating data before upgrade…
% Warning: All secondary nodes should be upgraded and inline posture nodes should be de-registered before upgrading Primay PAP.
Starting application after rollback…
% Error: The node has been reverted back to its pre-upgrade state.
% Application install or upgrade cancelled.
ISE-PAN-01/admin#

 

This screen shows how to apply the application reset-config command on the PAN:

 

ISE0-PAN-01/admin# application reset-config ise
Initialize your ISE configuration to factory defaults? (y/n): y
This ISE node is the primary administration node in a ISE deployment. It is recommended you first deregister all secondary nodes before resetting the configuration. Proceed with factory reset? (y/n): y
Leaving currently connected AD domains if any…
Please rejoin to AD domains from the administrative GUI
Retain existing ISE server certificates? (y/n): y
Reinitializing local ISE configuration to factory defaults…
Stopping ISE Monitoring & Troubleshooting Log Collector…
Stopping ISE Monitoring & Troubleshooting Log Processor…
ISE Identity Mapping Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
Stopping ISE Certificate Authority Service…
Stopping ISE Profiler Database…
Stopping ISE Monitoring & Troubleshooting Session Database…
Stopping ISE AD Connector…
Stopping ISE Database processes…
Enter the ISE administrator username to create[admin]: admin
Enter the password for ‘admin’:
Re-enter the password for ‘admin’:
Extracting ISE database content…
Starting ISE database processes…
Creating ISE M&T session directory…
Performing ISE database priming…
< omitted > 

 

To apply the application reset-config ise command on the PSN is the same but the steps would look slightly different than what you would see on the PAN.

I hope this was useful and thanks for reading.

Posted in Blog, ISE, Security

Leave a Reply

avatar
  Subscribe  
Notify of

Related Posts:

Scroll To Top
Share via
Copy link
Powered by Social Snap