ISE PSN Got Stuck at Step 4

I came across a couple of issues during an ISE upgrade project I was working on recently which are not really common and you would rarely see them. These issues were, an ISE PSN got stuck at step 4 during the upgrade process, and a PAN went out of synch. This project was to upgrade a distributed ISE cube with six nodes from version 2.2 to 2.4 and all appliances were virtual.

I was doing the upgrade through CLI which is my favorite option over the GUI because it really gives much more control and visibility. It is also much more reliable in my opinion because it prevents running into potential browsers dependent issues such as sessions timing out or unrealistic percentage bars when provisioning the ISE upgrade image.

As expected, the plan to do the upgrade was to start with the secondary PAN and then moving to the PSNs, and finally upgrading the new secondary PAN which was the old primary PAN.

When I started the upgrade everything was progressing as normal and so far there were no major issues. That was the case until one PSN got stuck at step 4 which took really too long before it failed the upgrade process.

However, the second issue was with the secondary PAN which for some reason was keeping one of the PSNs in its deployment even though that PSN was already upgraded and added to the new upgraded deployment.

Because of that I was not able to upgrade the secondary PAN since one of the requirements is to complete the upgrade of all the other nodes before starting or being able to upgrade the secondary PAN itself.

Along with Cisco TAC we spent some time doing some troubleshooting to try to find out the root cause of these issues with no luck. To fix the issue with the PSN got stuck at step 4 there were no many options apart from rebuilding the PSN or to reset its ISE application configuration.

However, regarding the the issue with secondary PAN, the two options were to try to adjust ISE database which was kind of a risky option since that would corrupt the database, or to reset ISE application configuration.

The option that was more reasonable and the less risky to fix the issues on both nodes was to reset their ISE application configuration. The command I used to do so was application reset-config ise.

What this command does is basically it resets only ISE application configuration to the factory defaults but without touching any network configuration such as IP addresses, FQDN, NTP, DNS, etc, and most importantly it asks you if you want to keep the certificates on the node.

 

Here is where the ISE PSN got stuck at step 4 which stayed in that state for so long before it failed the upgrade:

 

 

Here are the system logs generated whilst the ISE PSN got stuck at step 4:

 

 

These were the error messages after the upgrade process has failed:

 

 

This screen shows when the secondary PAN was trying to do the upgrade:

 

 

This screen shows how to apply the application reset-config command on the PAN:

 

 

To apply the application reset-config ise command on the PSN is the same but the steps would look slightly different than what you would see on the PAN.

I hope this was useful and thanks for reading.