8. General Settings » Graphs & Storage

In a later chapter, you will see how to configure Sensors to generate traffic graphs, tops and accounting information for every IP that belongs to the monitored network. In Configuration » General Settings » Graphs & Storage, you can configure how much disk space will be used to store that data, as well as other storage-related settings.

Graph Storage Engine parameters allow you to select the third-party software that will be used for storing graph data:

► InfluxDB is a specialized database designed for handling time series data. It uses more RAM than the alternative (RRDTool, shown below), but it needs less storage space, it is faster in most cases, it can be configured for High Availability, and existing data is not deleted by configuration changes

► RRDTool is very stable, but in some cases it can be very slow (due to the disk seek time), it uses a lot of storage space (due to its preallocation of data), and some configuration changes like adding a decoder will reset the existing graph data

Wanguard uses by default Graph Storage Engine 1 as the only backend. However, it can be configured to use two backends simultaneously in the following combinations: RRDTool + InfluxDB, or InfluxDB (primary) + InfluxDB (secondary).

Click on the options button from the right-hand side to configure the selected Graph Storage Engine:

InfluxDBRRDTool (On Disk)RRDTool (In RAM)

InfluxDB 1.9 is the preferred choice for new installations.

● IP Graph Granularity sets the granularity (level of accuracy) for IP, AS, and Country graphs. The default value is 5 minutes. When using Flow Sensor, do not set a value less than the flow delay

● Storage Restrictions parameter excludes several classes of graphs from using the selected Graph Storage Engine

● InfluxDB URL should contain the full URL of the InfluxDB http(s) server (e.g. http://127.0.0.1:8086)

● InfluxDB Username and InfluxDB Password are optional and should be set if auth-enabled=true in influxdb.conf

● InfluxDB Database is optional. By default, the name of the InfluxDB database is “andrisoft”. This should be changed only when using a single InfluxDB for multiple Consoles

● InfluxDB Connector permits initializing the database, to delete it or to check its status. The most important status value shown here is the “cardinality estimation”, which is directly related to the RAM consumed by InfluxDB. High-cardinality data include IP graphs and AS graphs. If the RAM usage is too high, consider disabling IP graphing for large subnets, and then deleting and reinitializing the database

Graphing Engine Backend lets you choose from which Graph Storage Engine to pull the data when rendering graphs. The options button lets you select which graph file format to use: SVG or PNG. To properly view SVG files on macOS or Linux, the client may need to install the Consolas font on its computer.

Decoders represent internal functions (traffic dissectors) that differentiate and classify the underlying protocols of each packet or flow. Each decoder increases the size of the IP graph, top and accounting data, and causes a very small performance penalty. It is recommended to enable only the decoders you are interested in. You can create your own decoders in General Settings » Custom Decoders. Built-in decoders:

IP	Matches all IP packets, irrespective of higher protocols. Always enabled
TCP	Matches TCP traffic
TCP+SYN	Matches TCP traffic with SYN flag set and ACK unset. Flow Sensor counts one packet per flow
UDP	Matches UDP traffic
ICMP	Matches ICMP traffic
OTHER	Matches IP protocols that differ from TCP, UDP and ICMP
INVALID	Matches TCP or UDP port set to 0, or IP protocol set to 0
FLOWS	Matches flow records and replaces packets/s with flows/s. Works only with Flow Sensor
FLOW+SYN	Matches flow records with SYN flag set. Flow Sensor counts all packets per flow
FRAGMENT	Matches fragmented IP packets. Works only with Packet Sensor
TCP-NULL	Matches TCP traffic without TCP flags, indicative of reconnaissance sweeps
TCP+RST	Matches TCP traffic with RST flag set
TCP+ACK	Matches TCP traffic with SYN flag unset and ACK set
TCP+SYNACK	Matches TCP traffic with SYN flag set and ACK flag set
NETBIOS	Matches TCP traffic on source or destination port 139
QUIC	Matches Google’s QUIC protocol on UDP port 80 and 443
UDP-QUIC	Matches UDP traffic not part of the QUIC protocol
MEMCACHED	Matches UDP traffic on port 11211
HTTP	Matches TCP traffic on source or destination port 80
HTTPS	Matches TCP traffic on source or destination port 443
MAIL	Matches TCP traffic on source or destination ports 25, 110, 143, 465, 585, 587, 993, 995
DNS	Matches UDP traffic on source or destination port 53
SIP	Matches TCP or UDP traffic on source or destination port 5060
IPSEC	Matches IP traffic on IP protocol 50 or 51
WWW	Matches TCP traffic on source or destination ports 80, 443
SSH	Matches TCP traffic on source or destination port 22
NTP	Matches UDP traffic on source or destination port 123
SNMP	Matches UDP traffic on source or destination ports 161, 163
RDP	Matches TCP or UDP traffic on source or destination port 3389
YOUTUBE	Matches IP traffic going or coming from Youtube AS 43515, 36561, or from Youtube subnets
NETFLIX	Matches IP traffic going or coming from Netflix AS 55095, 40027, 2906, or from Netflix subnets
HULU	Matches IP traffic going or coming from Hulu AS 23286, or from Hulu subnets
FACEBOOK	Matches IP traffic going or coming from Facebook AS 54115, 32934, or from Facebook subnets

IP Sweep Graphing option allows storing IP graph data for IPv4 and/or IPv6 addresses that receive traffic without sending any traffic in return. Do not set it to Off when monitoring unidirectional links or asymmetric traffic. Activating this feature for IPv6 traffic is generally not advised. Given the vast number of IP addresses an IPv6 subnet can hold, an IP sweep could exhaust all available RAM with InfluxDB or fill up all the disk space when using RRDTool.

Sensor Top N (default: 20) specifies the maximum number of items stored for ordered sets of data, such as top Talkers, External IPs, ASNs, Countries, TCP/UDP ports, IP protocols, and so on.

Flow Sensor saves flow data on a local disk drive in the path defined in the Flow Collector Path field, under a directory structure that takes into account the Flow Data Hierarchy parameter.

Packet Sensor saves packet dumps on a local disk drive in the path defined in the Packet Dump Path field.

RRDTool stores all graph files on the Console server in the Graphs Disk Path. InfluxDB stores all graph data in /var/lib/influxdb/data/, so if you want to use a different path or drive, make sure it is symlinked to the original location.

Note

It is highly recommended to automate the deletion of old data and to monitor the disk usage of IP graphs in General Settings » Data Retention.

8.1. Sensor and Applications Graph Troubleshooting

✔ Ensure that all Sensors run correctly by verifying the event log and by viewing live statistics from Reports » Devices » Overview. Please consult the troubleshooting guide for Packet Sensor, Flow Sensor, and SNMP Sensor

✔ Discontinuous Sensor graphs can be caused by enabling IP Accounting for too many/large subnets when there is a slow connection between the Sensor and the MySQL/MariaDB running on the Console server

8.2. IP/Subnet and Profiling Graph Troubleshooting

✔ Ensure that all Sensors run correctly by verifying the event log and by viewing live statistics displayed in Reports » Devices » Overview. Please consult the troubleshooting guide for Packet Sensor, Flow Sensor, and SNMP Sensor

✔ Generating IP and profiling graph data causes the biggest impact on the load of the Console server. Enable each feature (IP graphing, IP accounting, IP profiling) sequentially for each subnet, after making sure that the Console server can handle it. The storage requirements for each subnet are listed in the IP Zone, and the current disk usage in General Settings » Data Retention

✔ The internal process used for saving IP graph data is /opt/andrisoft/bin/genrrds_ip. If it is overloading the Console server or the event log contains warnings such as “Updating IP graph data takes longer than 5 minutes”, use InfluxDB or RRDCacheD, RAM/SSD updating method, faster disk drivers, enable IP graphing for fewer subnets, or deploy a Sensor Cluster configured to aggregate IP graph data

✔ The internal process used for generating IP or subnet graphs is /opt/andrisoft/bin/gengraph_ip. Console users launch the process for each requested IP or subnet graph. If the Console server gets too loaded by gengraph_ip, execute “killall gengraph_ip” and configure InfluxDB or RRDCacheD. When launched, the process stops only when the graph is generated. This process can be slow when users request subnet graphs for subnets not specifically defined in the IP Zone. It is not possible to throttle the number of graphs requested by users

8.3. AS and Country Graph Troubleshooting

✔ Ensure that all Sensors run correctly by verifying the event log and by viewing live statistics from Reports » Devices » Overview. Please consult the troubleshooting guide for Packet Sensor, Flow Sensor, and SNMP Sensor

✔ To enable AS and Country graphs, set the Stats Engine parameter to either Extended in Flow Sensor’s configuration, or to Full in Packet Sensor’s configuration

✔ SNMP Sensor is not able to generate AS graphs or Country graphs due to the limitations of the SNMP protocol