Menu Close

Palo Alto ‘Log Collection log forwarding agent’ is active but not connected

I was troubleshooting an issue with logging collection a couple of weeks ago between a Palo Alto PA-850 and a Panorama. The PA-850 was configured with a Log Forwarding to push its logs to Panorama, and the Panorama was configured with itself as the Collector as well as with a Collector Group with both the Collector (itself) and the Device Log Forwarding (PA-850). So, the configuration was looking good so far, however, no logs were showing up on Panorama Monitor page related to the PA-850. On the PA-850 CLI I was seeing 'Log Collection log forwarding agent' is active but not connected on the output of the show logging-status command similar to the below output:

admin@PA-820-GW> show logging-status

-----------------------------------------------------------------------------------------------------------------------------
      Type      Last Log Created        Last Log Fwded       Last Seq Num Fwded  Last Seq Num Acked         Total Logs Fwded
-----------------------------------------------------------------------------------------------------------------------------
> CMS 0
        Not Sending to CMS 0
> CMS 1
        Not Sending to CMS 1

>Log Collector 
'Log Collection log forwarding agent' is active but not connected


    config         Not Available         Not Available                        0                   0                        0
    system         Not Available         Not Available                        0                   0                        0
    threat         Not Available         Not Available                        0                   0                        0
   traffic         Not Available         Not Available                        0             8261280                        0
  hipmatch         Not Available         Not Available                        0                   0                        0
gtp-tunnel         Not Available         Not Available                        0                   0                        0
    userid         Not Available         Not Available                        0                   0                        0
     iptag         Not Available         Not Available                        0                   0                        0
      auth         Not Available         Not Available                        0                   0                        0
      sctp         Not Available         Not Available                        0                   0                        0
   decrypt         Not Available         Not Available                        0                   0                        0
globalprotect         Not Available         Not Available                        0                   0                        0


admin@PA-820-GW> 

Reviewing the configuration a couple of times I could not see anything that would have caused any issue to stop the PA-850 from sending its logs to Panorama. Then I took a look at the management plane management server logs (ms.log) and I saw it was throwing out some SSL errors similar to these:

admin@PA-820-GW> tail mp-log ms.log
2020-11-16 23:50:08.233 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2635): lcs agent: cs_load_certs_ex failed
2020-11-16 23:50:08.233 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=2
2020-11-16 23:50:09.242 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=1
2020-11-16 23:50:19.252 +0000 COMM: connection established. sock=58 remote ip=<< omitted >> port=3978 local port=48679
2020-11-16 23:50:19.253 +0000 lcs agent: Pre. send buffer limit=22600. s=58
2020-11-16 23:50:19.253 +0000 lcs agent: Post. send buffer limit=458752. s=58
2020-11-16 23:50:19.253 +0000 Error: cs_load_certs_ex(cs_common.c:648): keyfile not exists
2020-11-16 23:50:19.253 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2635): lcs agent: cs_load_certs_ex failed
2020-11-16 23:50:19.253 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=2
2020-11-16 23:50:20.262 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=1

So, I started looking at the secure communication channel's configuration between the PA-850 and Panorama to check if there was anything missing in there that would have caused this issue. Communication wise it was working though, the device could be managed through Panorama with no issues at all.

What I found was a little checkbox that was left unticked which was causing this issue. This little checkbox is in the Secure Communication Settings section in the Device > Management, and it is called Log Collector Communication. Basically the secure communication channel between the PA-850 and Panorama was not allowing the logs to passthrough, hence, I was seeing the 'Log Collection log forwarding agent' is active but not connected message on the device, and on Panorama the logs were not showing up at all.

After enabling that checkbox, the logs started flowing as expected, and they started showing up on the Monitor page on Panorama. Looking back at the show logging-status command on the PA-850, the 'Log Collection log forwarding agent' is active but not connected message was gone, and replaced with 'Log Collection log forwarding agent' is active and connected.

admin@PA-820-GW> show logging-status

-----------------------------------------------------------------------------------------------------------------------------
      Type      Last Log Created        Last Log Fwded       Last Seq Num Fwded  Last Seq Num Acked         Total Logs Fwded
-----------------------------------------------------------------------------------------------------------------------------
> CMS 0
        Not Sending to CMS 0
> CMS 1
        Not Sending to CMS 1

>Log Collector 
'Log Collection log forwarding agent' is active and connected to << omitted >>


    config         Not Available         Not Available                        0                   0                        0
    system         Not Available         Not Available                        0                   0                        0
    threat   2020/11/17 00:24:54   2020/11/17 00:24:59                  1006689             1006682                       21
   traffic   2020/11/17 00:24:56   2020/11/17 00:24:59                  8264683             8264683                     2768
  hipmatch         Not Available         Not Available                        0                   0                        0
gtp-tunnel         Not Available         Not Available                        0                   0                        0
    userid         Not Available         Not Available                        0                   0                        0
     iptag         Not Available         Not Available                        0                   0                        0
      auth         Not Available         Not Available                        0                   0                        0
      sctp         Not Available         Not Available                        0                   0                        0
   decrypt         Not Available         Not Available                        0                   0                        0
globalprotect         Not Available         Not Available                        0                   0                        0

admin@PA-820-GW>

This wraps up this post.

Thank You for reading!

Posted in Blog, L3, Palo Alto, Security, Tips & Tricks

Related Posts

>
Scroll To Top
Share via
Copy link
Powered by Social Snap