I was troubleshooting an issue with logging collection a couple of weeks ago between a Palo Alto PA-850 and a Panorama. The PA-850 was configured with a Log Forwarding to push its logs to Panorama, and the Panorama was configured with itself as the Collector as well as with a Collector Group with both the Collector (itself) and the Device Log Forwarding (PA-850). So, the configuration was looking good so far, however, no logs were showing up on Panorama Monitor page related to the PA-850. On the PA-850 CLI I was seeing 'Log Collection log forwarding agent' is active but not connected on the output of the show logging-status command similar to the below output:
admin@PA-820-GW> show logging-status ----------------------------------------------------------------------------------------------------------------------------- Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded ----------------------------------------------------------------------------------------------------------------------------- > CMS 0 Not Sending to CMS 0 > CMS 1 Not Sending to CMS 1 >Log Collector 'Log Collection log forwarding agent' is active but not connected config Not Available Not Available 0 0 0 system Not Available Not Available 0 0 0 threat Not Available Not Available 0 0 0 traffic Not Available Not Available 0 8261280 0 hipmatch Not Available Not Available 0 0 0 gtp-tunnel Not Available Not Available 0 0 0 userid Not Available Not Available 0 0 0 iptag Not Available Not Available 0 0 0 auth Not Available Not Available 0 0 0 sctp Not Available Not Available 0 0 0 decrypt Not Available Not Available 0 0 0 globalprotect Not Available Not Available 0 0 0 admin@PA-820-GW>
Reviewing the configuration a couple of times I could not see anything that would have caused any issue to stop the PA-850 from sending its logs to Panorama. Then I took a look at the management plane management server logs (ms.log) and I saw it was throwing out some SSL errors similar to these:
admin@PA-820-GW> tail mp-log ms.log
2020-11-16 23:50:08.233 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2635): lcs agent: cs_load_certs_ex failed
2020-11-16 23:50:08.233 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=2
2020-11-16 23:50:09.242 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=1
2020-11-16 23:50:19.252 +0000 COMM: connection established. sock=58 remote ip=<< omitted >> port=3978 local port=48679
2020-11-16 23:50:19.253 +0000 lcs agent: Pre. send buffer limit=22600. s=58
2020-11-16 23:50:19.253 +0000 lcs agent: Post. send buffer limit=458752. s=58
2020-11-16 23:50:19.253 +0000 Error: cs_load_certs_ex(cs_common.c:648): keyfile not exists
2020-11-16 23:50:19.253 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2635): lcs agent: cs_load_certs_ex failed
2020-11-16 23:50:19.253 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=2
2020-11-16 23:50:20.262 +0000 Error: pan_lcsa_tcp_channel_setup(src_panos/lcs_agent.c:2678): lcs_agent: SSL connect retry. sslerr=1
So, I started looking at the secure communication channel's configuration between the PA-850 and Panorama to check if there was anything missing in there that would have caused this issue. Communication wise it was working though, the device could be managed through Panorama with no issues at all.
What I found was a little checkbox that was left unticked which was causing this issue. This little checkbox is in the Secure Communication Settings section in the Device > Management, and it is called Log Collector Communication. Basically the secure communication channel between the PA-850 and Panorama was not allowing the logs to passthrough, hence, I was seeing the 'Log Collection log forwarding agent' is active but not connected message on the device, and on Panorama the logs were not showing up at all.
After enabling that checkbox, the logs started flowing as expected, and they started showing up on the Monitor page on Panorama. Looking back at the show logging-status command on the PA-850, the 'Log Collection log forwarding agent' is active but not connected message was gone, and replaced with 'Log Collection log forwarding agent' is active and connected.
admin@PA-820-GW> show logging-status ----------------------------------------------------------------------------------------------------------------------------- Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded ----------------------------------------------------------------------------------------------------------------------------- > CMS 0 Not Sending to CMS 0 > CMS 1 Not Sending to CMS 1 >Log Collector 'Log Collection log forwarding agent' is active and connected to << omitted >> config Not Available Not Available 0 0 0 system Not Available Not Available 0 0 0 threat 2020/11/17 00:24:54 2020/11/17 00:24:59 1006689 1006682 21 traffic 2020/11/17 00:24:56 2020/11/17 00:24:59 8264683 8264683 2768 hipmatch Not Available Not Available 0 0 0 gtp-tunnel Not Available Not Available 0 0 0 userid Not Available Not Available 0 0 0 iptag Not Available Not Available 0 0 0 auth Not Available Not Available 0 0 0 sctp Not Available Not Available 0 0 0 decrypt Not Available Not Available 0 0 0 globalprotect Not Available Not Available 0 0 0 admin@PA-820-GW>
This wraps up this post.
Thank You for reading!
