In this post I am going to show you how to delete the pending manager in FTD. The reason why we would have a pending manager in the first place would be right after we register a manager (FMC) in the FTD, but before we add that FTD to the FMC. That is basically a pre-req before we can add an FTD by the FMC.
The command we use to register an FMC from the FTD is configure manager add. The configure manager command can be run from the CLISH (>) mode which is where we are placed by default when we login into the FTD CLI. That command, as a minimum, requires us to define the FMC IP address, or the its hostname, and a registration key. The IP address can be IPv4 or IPv6.
There is an additional option called NAT-ID which can be used if the FTD we want to manage by the FMC is setting behind a NAT device. Take a look at the NAT-ID description at this post.
Until we add the FTD to the FMC, the FTD will keep showing the registered manager in pending state. And during that time, the FTD will keep trying periodically to connect to the FMC on port 8305/tcp which is the sftunnel default port. The sftunnel is basically a secure communication channel between the FTD and the FMC where all communications are encrypted. The sftunnel uses certificates to encrypt the traffic, and the default used cipher is AES256-GCM-SHA384. You can check the sftunnel status by using the command sftunnel-status from within CLISH mode.
Also, you can check the periodic attempts to connect to the FMC by the FTD by looking at the /var/log/messages file from the FTD expert (#) mode. To go into expert mode you can use the command expert from the CLISH. For any privileged command you need to use sudo before the actual command. For instance, sudo tail -f /var/log/messages. An alternative for this would be to switch to the root user. To do so, you can use the command sudo su -.
After you type the root password, there is no need to use sudo anymore. And btw, you can do pretty much everything from the expert mode by invoking the Perl scripts.
Until the FTD completes successfully the registration process with the FMC, it will not hash the registration key. Although you see it with the asterisks when you use the show managers command, but it is actually stored in clear text in the sftunnel.conf file stored in /ngfw/etc/sf/ and /etc/sf/ folders. Once the registration is successful, the registration key will be hashed, so it won't show in clear text anymore.
Now back to the main purpose of this post which is how to delete the pending manager in FTD. You might be wondering why should I show you how to do that when by nature that pending state will go away when you complete the manager registration!. Or you can also use the command configure manager delete!.
Well, the uncommon scenario I though about is what if for any reason two managers have been added to the FTD. And one was successfully registered, and the other one was left behind in pending state, which you don't need anymore?!. I know that should not happen, but it can happen.
The way I am going to show you how to delete the pending manager in FTD in this post is going to be by interacting directly with the FTD database. As you can guess, this would not be supported by Cisco, and you should not do it in production environment. But let's do it here for the sake of learning and demonstration.
Topology
Let's first check the managers status on the FTD
> show managers Host : 172.16.1.240 Registration Key : **** Registration : pending RPC Status : Type : Manager Host : 172.16.1.246 Registration : Completed
As you can see, the registration with the manager 172.16.1.240 is still in pending state. However, the registration with the manager 172.16.1.246 was successfully completed.
Now let's go and check the sftunnel.conf file, and see the parameters associated to each manager. To do so, we need to be in expert mode.
> expert
**************************************************************
NOTICE - Shell access will be deprecated in future releases
and will be replaced with a separate expert mode CLI.
**************************************************************
admin@ftd-01:~$ sudo su -
Password:
root@ftd-01:~#
The informational message states that the command expert will be deprecated in the future releases. FTDv-01 in our lab is running 6.5.0.4 code. However, that message is gone on FTD starting from version 6.6.0 which leads me to think that Cisco decided not to deprecate it anymore. The sudo su - command will save you typing sudo each time you want to run a super user command.
Before we cat or more for the sftunnel.conf file, let's see where that file is stored in the FTD
root@ftd-01:~# find / -name "sftunnel.conf" /ngfw/etc/sf/sftunnel.conf /etc/sf/sftunnel.conf
As you can see, the file is stored in two folders. However, if we list that file in the two folders, we see that it looks exactly the same.
root@ftd-01:~# ls -l /ngfw/etc/sf/sftunnel.conf -rw-rw-r-- 1 www www 1969 Apr 25 02:17 /ngfw/etc/sf/sftunnel.conf root@ftd-01:~# ls -l /etc/sf/sftunnel.conf -rw-rw-r-- 1 www www 1969 Apr 25 02:17 /etc/sf/sftunnel.conf
Now let's check the content of the sftunnel.conf file, and compare the parameters associated to the manager 172.16.1.240 and 172.16.1.246
root@ftd-01:~# cat /etc/sf/sftunnel.conf
# File creation initiated by /ngfw/usr/local/sf/bin/ActionQueueScrape.pl process to invoke SF::PeerManager::ConfigFiles::create_sftunnel_config from /ngfw/usr/local/sf/lib/perl/5.10.1/SF/PeerManager/PeerInfo.pm:875 at Apr 25 02:17:24
#
#proxysetup:
# proxy_port: port
# connect_type: 0; could be set to IPv4 or IPv6, defaults to both IPv4 and IPv6, otherwise exclusive.
# proxy_pidfile: optional string to pass to daemonizer
# proxy_state: 0 = active; 1 = inactive
proxysetup {
max_msg_size 0;
enable_dyn_routes 1;
connect_type 0;
proxy_port 8305;
proxy_port_ipv6 8305;
proxy_host 172.16.1.211;
proxy_ip 172.16.1.211;
proxy_ipv6 ;
proxy_state 0;
proxy_uuid << omitted >>;
no_host_check 1;
model_type Sensor;
max_services 20;
HEARTBEAT_INTERVAL 120;
MAX_SSL_CONN 500;
MAX_MSGS 20;
LOG_TUNNEL_LOAD 0;
};
#proxyssl:
# proxy_cert: local(default) certificate of the ssl server
# proxy_key: local(default) public key of the ssl server
# proxy_cacert: the certificate of the local certificate authority
# proxy_crl: local(default) certificate relocation list of the ssl server
# proxy_cipher: optional 1 = STRONG 2 = SNOOP 3 = NULL -- defaults to STRONG
proxyssl {
proxy_cert /etc/sf/keys/sftunnel-cert.pem;
proxy_key /etc/sf/keys/sftunnel-key.pem;
proxy_cacert /etc/sf/ca_root/cacert.pem;
proxy_crl /etc/sf/ca_root/crl.pem;
proxy_cipher 1;
};
peers_registered
{
172.16.1.246
{
role 1;
reg_key << omitted >>;
ip 172.16.1.246;
host fmcv-02.mylab.local;
sw_version 6.6.0;
upgrade_version ;
uuid c03f7f0c-782e-11ea-9020-4f2229f1252f;
priority 0;
}
}
peers_pending
{
172.16.1.240
{
role 1;
host 172.16.1.240;
ip 172.16.1.240;
reg_key BlueNetSecBlog;
uuid 172.16.1.240;
}
}
peers_routed
{
}
root@ftd-01:~#
In the peers_registered section, we see the manager 172.16.1.246 that was successfully registered with the FTD. However, in the peers_pending section we see the manager 172.16.1.240 that is still in pending state.
Notice how the registered manager shows the actual host name of the FMC, the software version and how the registration key is hashed. On the other side, the pending manager shows little comparing to the registered one, and it shows the registration key in clear text.
The sftunnel.conf is a text file, you can edit it. However, if you do so, that won't change anything. That's because the FTD does not parse the managers data from the sftunnel.conf file, instead it parses the data from its database. Therefore, if you edit the sftunnel.conf file, you will still see the same output when you issue the show managers command.
And when one of the managers processes that write the content of the sftunnel.conf file gets restarted, the sftunnel.conf content will be overwritten with the values stored in the FTD database. Those processes would be the sftunnel, sfmgr and sfmbservice.
Any easy way to restart the sftunnel channel services would be by using the manage_procs.pl script and selecting option 3. Once the services restarted you can exist selecting option 0.
Let's check the FTD MySQL database
I believe the FTD database is called MariaDB. The table we want to look at is called called EM_peers. In that table we find all the managers details, as well as their status.
To get access to the database we need to use the OmniQuery.pl Perl script. Then once we are into the database, we use the select command to show the content of the EM_peers table.
Any operation we do from within the database is irreversible, which means there is no way to recover any lost data or any potential corruption on the database tables. This is why you should never do this in production environment, unless instructed by Cisco TAC.
I am not sure if there is a way to show the table content or to interact with the database in one single command, rather than using OmniQuery.pl and then issuing the sub commands. If you know a way to do that please share it in the comments below.
root@ftd-01:~# OmniQuery.pl
DBI connect('database=external_data:host=:mysql_socket=/ngfw/var/run/mysql/mysql.sock','root',...) failed: Unknown database 'external_data' at /ngfw/usr/local/sf/lib/perl/5.10.1/SF/QueryEngine/Utils.pm line 424
DBI connect('database=external_schema:host=:mysql_socket=/ngfw/var/run/mysql/mysql.sock','root',...) failed: Unknown database 'external_schema' at /ngfw/usr/local/sf/lib/perl/5.10.1/SF/QueryEngine/Utils.pm line 424
----------------------------
OmniQuery v2.5.2
(c) 2020 Cisco Systems, Inc.
.:|:.:|:.
----------------------------
mdb.sfsnort> select * from EM_peers;
+---------------------+--------------+-----+-----------+--------------------------------------+---------+--------+------+-----------+--------------------------------------+--------------+------+----------+------------+------------+-----------------+-------------------+--------------+----------+-------------+------------------+
| name | ip | vip | vip_local | uuid | uuid_gw | active | role | reg_state | reg_key | last_changed | vnet | priority | persistent | sw_version | upgrade_version | mgmt_mac_address | model_number | model_id | primary_mgr | domain_uuid |
+---------------------+--------------+-----+-----------+--------------------------------------+---------+--------+------+-----------+--------------------------------------+--------------+------+----------+------------+------------+-----------------+-------------------+--------------+----------+-------------+------------------+
| 172.16.1.240 | 172.16.1.240 | | | 172.16.1.240 | | 1 | 1 | 1 | BlueNetSecBlog | 1587778967 | | 0 | 0 | | | | | | 0 | |
| fmcv-02.mylab.local | 172.16.1.246 | | | c03f7f0c-782e-11ea-9020-4f2229f1252f | | 1 | 1 | 0 | c6fb5c26-73a1-11ea-9f25-dc9836ca9f08 | 1587780823 | | 0 | 0 | 6.6.0 | | 00:0C:29:46:49:24 | 66 | E | 1 | |
| 172.16.1.211 | 172.16.1.211 | | | << omitted >> | | 1 | 0 | 0 | | 1587522429 | | 0 | 0 | 6.5.0.4 | | 00:0C:29:11:9C:B6 | 75 | A | 1 | |
+---------------------+--------------+-----+-----------+--------------------------------------+---------+--------+------+-----------+--------------------------------------+--------------+------+----------+------------+------------+-----------------+-------------------+--------------+----------+-------------+------------------+
3 rows in set
As you can see, we have both managers in the above table. However, the manager 172.16.1.240 reg_state is 1, which means in pending. Notice that all the details we have seen into the sftunnel.conf are matching to what we have on the EM_peers table.
We can use any value of any column to delete any entry from the database table. We will use the value name to delete the 172.16.1.240 manager. The command we are going to use is delete from EM_peers where name = '172.16.1.240';.
mdb.sfsnort> delete from EM_peers where name = '172.16.1.240'; Command executed but returns no results.
Let's check again the content of the EM_peers table
mdb.sfsnort> select * from EM_peers; +---------------------+--------------+-----+-----------+--------------------------------------+---------+--------+------+-----------+--------------------------------------+--------------+------+----------+------------+------------+-----------------+-------------------+--------------+----------+-------------+------------------+ | name | ip | vip | vip_local | uuid | uuid_gw | active | role | reg_state | reg_key | last_changed | vnet | priority | persistent | sw_version | upgrade_version | mgmt_mac_address | model_number | model_id | primary_mgr | domain_uuid | +---------------------+--------------+-----+-----------+--------------------------------------+---------+--------+------+-----------+--------------------------------------+--------------+------+----------+------------+------------+-----------------+-------------------+--------------+----------+-------------+------------------+ | fmcv-02.mylab.local | 172.16.1.246 | | | c03f7f0c-782e-11ea-9020-4f2229f1252f | | 1 | 1 | 0 | c6fb5c26-73a1-11ea-9f25-dc9836ca9f08 | 1587780823 | | 0 | 0 | 6.6.0 | | 00:0C:29:46:49:24 | 66 | E | 1 | | | 172.16.1.211 | 172.16.1.211 | | | << omitted >> | | 1 | 0 | 0 | | 1587522429 | | 0 | 0 | 6.5.0.4 | | 00:0C:29:11:9C:B6 | 75 | A | 1 | | +---------------------+--------------+-----+-----------+--------------------------------------+---------+--------+------+-----------+--------------------------------------+--------------+------+----------+------------+------------+-----------------+-------------------+--------------+----------+-------------+------------------+ 2 rows in set
Cool, now the 172.16.1.240 manager is gone. Let's exit from the database, and go into the CLISH mode. Once we are in the CLISH mode let's do show managers and see if we still see the manager 172.16.1.240 as pending.
> show managers Type : Manager Host : 172.16.1.246 Registration : Completed >
As you can see the manager 172.16.1.240 now is gone.
This wraps up this post on how to delete the pending manager in FTD.
Thank you for reading!
OmniQuery.pl -db mdb -e “select name from EM_peers;G”
Site has removed a backslash before the upper case G
Thank you for sharing that, I’ve just tried it and it worked.
Great article! Thank you very much for saving my time!
Thank you very much! Your article very helped me!!!