Unknown Messages not being forwarded if UNKNOWN is configured

[gbieli]

I'd like to foward Messages that could not been parsed by napalm logs to my ELK stack server.

Napalm logs receives this Message:

[root@salt-master ~]# tcpdump -vnne tcp port 8080 or udp port 514 -i ens160 
17:27:32.366652 00:50:56:b9:50:7b > 00:50:56:b9:ec:65, ethertype IPv4 (0x0800), length 231: (tos 0x0, ttl 64, id 14120, offset 0, flags [DF], proto UDP (17), length 217)
    10.83.21.224.54530 > 10.83.21.232.514: SYSLOG, length: 189
        Facility local7 (23), Severity warning (4)
        Msg: 95662: <snip>: Feb  7 17:27:31.358: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet0/1/1 (22), with sy-test16.<snip> GigabitEthernet1/0/1 (1).

Then napalm logs shows this in the log

[root@salt-master ~]# tail /var/log/napalm/logs 
2019-02-07 17:27:32,368,368 [napalm_logs.device][INFO    ] Syslog message not configured for os: ios tag CDP-4-NATIVE_VLAN_MISMATCH

This is my config:

[root@salt-master ~]# cat /etc/napalm/logs 
log_level: info
port: 514
disable_security: true
extension_config_path: /etc/napalm/additional_config/
device_whitelist:
  - ios
  - opengear
publisher:
  http:
    address: 'http://elk.<snip>:8080'
    method: POST
    headers:
      Content-Type: application/json
    backend: requests
    error_whitelist:
      - RXDATA
      - UNKNOWN
  zmq:
    address: 0.0.0.0
    port: 49017
    error_blacklist:
      - RXDATA

In my understanding of the documentation I would expect that my ELK server should receive this message unparsed or am I wrong?

[luke-orden]

HI,

I think send_raw: true should do what you are looking for:

http://napalm-logs.com/en/latest/publisher/index.html#send-raw-false

[gbieli]

Hi,

Thanks for the quick answer. I just tried that and had no luck.
I also tried that, still with no luck:

    error_whitelist:
      - RXDATA
      - UNKNOWN
      - RAW

So to summarize my trials: I'm not able to get any unparsed logs to my ELK server.
Any other ideas?

[luke-orden]

Can you share your full config file and all (if any) cli args you are using.

[gbieli]

Current config file:

log_level: info
port: 514
disable_security: true
extension_config_path: /etc/napalm/additional_config/
device_whitelist:
  - ios
  - opengear
publisher:
  http:
    address: 'http://elk.<snip>:8080'
    method: POST
    headers:
      Content-Type: application/json
    backend: requests
    error_whitelist:
      - RXDATA
      - UNKNOWN
      - RAW
  zmq:
    address: 0.0.0.0
    port: 49017
    error_blacklist:
      - RXDATA

Service file:

[Unit]
Description=napalm-logs
Documentation=https://napalm-logs.readthedocs.io/en/latest/
After=network.target

[Service]
ExecStart=/usr/bin/napalm-logs --disable-security
Type=simple
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

What I'd like to accomplish:

• All parsed logs for IOS and Opengear should go into the salt event bus, except for Opengear RXDATA, these messages should go parsed directly to my ELK server (this works fine)
• All messages that could not have been parsed should go directly to the ELK server so that I can have napalm-logs as my only syslog receiver (this is not working)
• I do not want to have duplicate logs in my ELK server, e.g. a parsed version and an unparsed version of the same syslog message. I guess if I use "RAW" in my whitelist for the ELK server, that would be the case. In my understanding of the documentation what I should have configured is the following like in the post above:

log_level: info
port: 514
disable_security: true
extension_config_path: /etc/napalm/additional_config/
device_whitelist:
  - ios
  - opengear
publisher:
  http:
    address: 'http://elk.<snip>:8080'
    method: POST
    headers:
      Content-Type: application/json
    backend: requests
    error_whitelist:
      - RXDATA
      - UNKNOWN
  zmq:
    address: 0.0.0.0
    port: 49017
    error_blacklist:
      - RXDATA

[luke-orden]

Hi,

I can't see the option send_raw: true in your config file. Please add it and try again.

[gbieli]

Hi,
If I specify RAW in the error_whitelist together with send_raw: true then it works. If one of them is missing, it does not work. Is this the desired behavior?

Config file new:

log_level: info
port: 514
disable_security: true
extension_config_path: /etc/napalm/additional_config/
device_whitelist:
  - ios
  - opengear
publisher:
  http:
    address: 'http://elk.<snip>:8080'
    method: POST
    headers:
      Content-Type: application/json
    backend: requests
    send_raw: true
    error_whitelist:
      - RXDATA
      - UNKNOWN
      - RAW
  zmq:
    address: 0.0.0.0
    port: 49017
    error_blacklist:
      - RXDATA