I was working recently on a project to upgrade a distributed ISE cube comprised of two PANs and four PSNs. The upgrade was from version 2.2 to 2.4, and I was doing it through CLI which is my favorite option over the GUI. The plan was to start upgrading the secondary PAN then the PSNs and finally the new secondary PAN which originally was the primary PAN. The upgrade was progressing without any major issues on all nodes with the exception of two of them. The two nodes that gave me some issues were one of the PSNs and the secondary PAN. The PSN got stuck at step 4 while doing the upgrade, and then after a long time the upgrade process failed.

However, the secondary PAN was giving a different issue. For some reason it was keeping in the old deployment one of the PSNs which was already upgraded and added to the new deployment. The secondary PAN was seeing that PSN as if it was still active in the old deployment when it was not the case. Because of that I was not able to upgrade the secondary PAN since one of the requirements was to complete the upgrade on all the other nodes before starting the upgrade on the secondary PAN itself.

There was no reasonable and a less risky option to go for to fix those issues but to reset ISE applications on those two nodes. The command I used to do that was application reset-config ise. This command resets only ISE application to its factory defaults which means it maintains all the node network configuration such as IP addresses, FQDN, NTP, DNS etc and most importantly it asks you if you want to keep the certificates on the node.

 

PSN STUCK AT STEP 4:

 

ISE-PSN-01/admin# application upgrade proceed
Initiating Application Upgrade…
% Warning: Do not use Ctrl-C or close this terminal window until upgrade completes.
-Checking VM for minimum hardware requirements
STEP 1: Stopping ISE application…
STEP 2: Verifying files in bundle…
-Internal hash verification passed for bundle
STEP 3: Validating data before upgrade…
STEP 4: De-registering node from current deployment.

 

LOOKING AT THE SYSTEM LOGS WHILST THE PSN WAS STUCK AT STEP 4:

 

root: info:[application:operation:isehourlycron.sh] Waiting up to 20 seconds for lock: APP_UPGRADE to complete
root: info:[application:operation:isehourlycron.sh] Database is still locked by lock: APP_UPGRADE. Aborting. Please try it later
root: info:[application:operation:isehourlycron.sh] % Error: Another ISE DB process (APP_UPGRADE) is in progress, cannot perform cleanup at this time

 

POST PSN STUCK AT STEP 4:

 

ISE-PSN-01/admin#
% Error: De-registering node from current deployment failed!
Starting application after rollback…
% Manual rollback required: Perform the following steps to revert node to its pre-upgrade state:
– Ensure that node is still present in current deployment from Primary UI; if it is not present, register this node back again.
% Application install or upgrade cancelled.

 

SECONDARY PAN UPGRADE ATTEMPT:

 

ISE-PAN-01/admin# application upgrade proceed
Initiating Application Upgrade…
% Warning: Do not use Ctrl-C or close this terminal window until upgrade completes.
-Checking VM for minimum hardware requirements
STEP 1: Stopping ISE application…
STEP 2: Verifying files in bundle…
-Internal hash verification passed for bundle
STEP 3: Validating data before upgrade…
% Warning: All secondary nodes should be upgraded and inline posture nodes should be de-registered before upgrading Primay PAP.
Starting application after rollback…
 
% Error: The node has been reverted back to its pre-upgrade state.
% Application install or upgrade cancelled.
ISE-PAN-01/admin#

 

SECONDARY PAN ISE RESET:

 

ISE0-PAN-01/admin# application reset-config ise
Initialize your ISE configuration to factory defaults? (y/n): y
This ISE node is the primary administration node in a ISE deployment. It is recommended you first deregister all secondary nodes before resetting the configuration. Proceed with factory reset? (y/n): y
Leaving currently connected AD domains if any…
Please rejoin to AD domains from the administrative GUI
Retain existing ISE server certificates? (y/n): y
Reinitializing local ISE configuration to factory defaults…
Stopping ISE Monitoring & Troubleshooting Log Collector…
Stopping ISE Monitoring & Troubleshooting Log Processor…
ISE Identity Mapping Service is disabled
ISE pxGrid processes are disabled
Stopping ISE Application Server…
Stopping ISE Certificate Authority Service…
Stopping ISE Profiler Database…
Stopping ISE Monitoring & Troubleshooting Session Database…
Stopping ISE AD Connector…
Stopping ISE Database processes…
Enter the ISE administrator username to create[admin]: admin
Enter the password for ‘admin’:
Re-enter the password for ‘admin’:
Extracting ISE database content…
Starting ISE database processes…
Creating ISE M&T session directory…
Performing ISE database priming…
 
< omitted >
 

 

ISE application reset on the PSN is very similar to the PAN.

 

I hope this was useful and thanks for reading.

 

Share via:

feedback?