Reports

From UntangleWiki
Jump to: navigation, search

Reports 128x128.png     Reports
Other Links:
Reports Description Page
Reports Video Demo
Reports Screenshots
Reports Forums
Reports FAQs




About Untangle Reports

Reports provides users with detailed statistics of the traffic and activity on your network.

These reports can be viewed online, either through the administration interface or through the separate reporting interface available to non-administrators reporting-only users.

Report summaries can be sent via email, which includes basic information and a link to view the online reports if the user has access.

Reports can backup your data in multiple formats to Google Drive for long term storage.

Reports also contains alerts which can send you alerts in real-time when critical events occur.

Web Filter - Reports


Settings

This section reviews the different settings and configuration options available for Reports.


Status

On this tab you can click View Reports to open up Reports in a new browser tab.


All Reports

ManageReports


Here you can manage the reports that are displayed in each application. Each report is broken out by category and listed in the same display order you will find on the corresponding reports tab.

With the Enabled check box, you can remove reports from the display within each app. This will remove reports from the Reports tab without deleting the report, useful for unused reports.

View can be used to bring up any report without leaving the Reports app. This is useful for easily viewing and comparing reports across different apps.

Edit is a very powerful tool, allowing you to manipulate nearly every aspect of a report.

You can also use the Edit button to copy a report as well. This allows you to keep the original report while also having a customized version to meet your needs. Click Edit, then Copy Report. Update the report title and other fields as necessary.

For custom reports, the Delete button can be used to permanently delete a report. Note that Untangle pre-defined reports can not be deleted from the system.


Data

  • Data Retention: This value controls how much time report data is kept on disk. Please note that increasing the number increases the amount of disk space that is needed for data storage.
  • 'Upload Data to Google Drive If enabled, and the Google Connector in Directory Connector is enabled, your daily data will be uploaded to google drive each night for safe storage.
  • 'Upload CSVs to Google Drive If enabled, and the Google Connector in Directory Connector is enabled, your daily CSV files will be uploaded to google drive each night for safe storage.
  • Google Drive Directory configures which subdirectory data will be uploaded to in google drive.
  • Import/Restore Data Backup Files imports data from a previous backup into the database. NOTE: this directly imports the SQL contents. If you have upgraded and the database schema has significantly changed since the time of the back, the import will not work correctly.


Alert Rules

Alert rules are evaluated on all events logged in the database and will log and/or alert the administrator when interesting or noteworthy events occur.

Each logged event is represented by a JSON object. As each event is logged to the database the alert rules are evaluated. The Events page details all of the logging events. If all of an alert rule's conditions match the logged event the action(s) configured in the alert rule is performed.

Enable Thresholds limits the alert from firing until it reaches a certain frequency threshold.

Exceeds Threshold Limit is the frequency limit for which this condition will match. If the frequency is greater than this value, then the threshold conditions matches.

Over Timeframe defines the time range, in seconds, to use to compute the frequency.

Grouping Field defines how to group thresholds by an attribute field in the events. This field is optional.

If Exceeds Threshold Limit is 100 and Over Timeframe is 60, then the threshold condition will only match when this rules other conditions match approximately 100 times over any 60 second period. If Group Field is set to "CClientAddr" then the threshold load is grouped by "CClientAddr" value in the event objects. Using the above example this would mean that the alert would only fire when a specific "CClientAddr" like "192.168.1.100" does something over 100 times within 60 seconds. The threshold value for other clients like "192.168.1.150" is tracked separately.

Log Alert logs the event to the Alert Event Log

Send Alert sends an email to all administrators' emails describing the event.

Limit Send Frequency limits the number of times a rule can send an alert email To once per the configured amount of minutes. For some cases, like a low disk space alert, this is useful to limit the number of alerts sent so that an alert is not sent every minute.

Adding Alert Rules

Since alerts are created on the raw messages passed through the system before logging, adding alerts can be a bit tricky. An example is helpful to describe how alert rules are created.

One of the included alert rules is the "WAN is offline" alert. This triggers whenever a WAN interface has been disconnected.

Alert Rules

In the conditions, you can see we are looking for two Field conditions. All conditions for alert rules will be field conditions, which simply means an entry in the logged event represented by a JSON object (there are no other choices in the drop down). In nearly all cases you will have a field condition with a class, which represents an application or system process responsible for logging and one or more additional conditions to alert.

Going back to our example, you can see the alert rule is monitoring the class WanFailoverEvent which is created by WAN Failover. Within that class we are looking for any log event object that contains an action = DISCONNECTED. In this case that means a WAN was found to be disconnected by WAN Failover and an alert is triggered.

You can view the Events page for various classes and conditions that can be monitored for alert events.

If you are having problems adding a specific alert, or have common alert rules you would like added to the default rules, let us know on the forums or through the support team.

Email

You can customize emailed reports using Report Templates. You can create as many as you wish with any combination of:

  • Interval: Daily, Weekly, Monthly. You can only use an interval that matches your Data retention days. So if you have 7, you can only do Daily or Weekly, not Monthly.
  • Mobile: Generate chart images more appropriate for a mobile device.
  • Reports: Select those reports under Config and Application sections. Text and chart reports are allowed but not event list reports. Reports for applications will be included only if that application is installed.

Additionally, you can copy the settings for an existing report.

The default Daily Reports template includes common text and chart reports for your system. This template is fixed and cannot be changed or modified.

Email Templates must be associated with Report Users.

Syslog

Reports supports the sending of all events via syslog messages. To use syslog simply install a syslog receiver on another server, then enable syslog and configure as necessary.. Some syslog products are easier to set up than others. Kiwi, a third-party syslog daemon, is a favorite of many Untanglers using Windows, while those on *nix can use rsyslog.


  • Host: The host name or IP address of the Syslog daemon that is authorized to receive syslog messages from the Untangle Server. Do not set the Host to the Untangle box itself - this will result in the hard drive filling up very quickly and most likely crashing the box.
  • Port: The UDP port to send syslog messages to the syslog daemon. 514 is the default as this is the default syslog port.
  • Protocol: The protocol to use to send syslog messages. The default is UDP.

WARNING: Syslog sends every single event to your syslog server. This is a very expensive operation, consuming both processing power and bandwidth. Syslog should only be used in special circumstances when something is actually done with the data on the syslog server.


Name Map

You can use the Name Map to manually configure the hostname for hosts. Untangle often can automatically determine the hostname for the IP automatically via DHCP or other methods. You can view the current names for currently active hosts in the Host Viewer

However, when Untangle is unable to automatically determine a hostname for an IP the Name Map provides a way to manually name them.

Reports

This applications reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Alerts Alerts over time.
Top Alerts The top alerts.
Alert Events Log of all alerts created by alert rules.


The tables queried to render these reports:



Accessing Reports

If a user is set up to receive email reports, they only need to view or download the HTML attachment to see an overview report. If they need more information or would like to drill down to specific users or machines, they can use the link in the email, which will open Reports on the Untangle if it is accessible from their location. Administrators can use the View Reports button in Reports settings to open the Reports.

To access Reports directly from a browser, you have two options:

Please note that to view Reports from outside the network you'll need to check Allow HTTPS on WANs at Config > Network > Advanced > Filter Rules. If you have changed the External HTTPS Port, you'll need to use the proper HTTPS port when connecting from the outside.


Report Viewer

Reports provide a graphical view of the network traffic and actions of your Untangle. Various reports are available within applications and base system components. The reports can be manipulated to drill down, customize, and export data in many ways using the Report Viewer.


Web Filter - Reports

There are five panels in the Report Viewer:

  • Application Selector (Green)(Left): This allows you to choose from the system and application report groupings. By selecting an option here, the results in the Report Selector section will be filtered to show just that application. When using the report viewer within an application, this pane is not shown.
  • Report Selector (Blue)(Left): This panel is broken into two areas containing reports and event log queries. The "Select Report" contains a list of pre-defined reports and event log queries. Saved custom reports will also appear in the reports list.
  • Report Chart (Yellow)(Top): Shows the currently selected report and contains options to change the type of chart, customize the report, change the report start and end times, and view the report in Event Log format. You can also interact directly with the report. Data series can be removed from the view using the legend and hovering over data series will show the values in reader friendly format.
  • Current Data (Orange)(Right): Displays the raw data that is being used to generate the report. Data points will be displayed in reader friendly format when hovering over the graph. The data can be exported to a CSV text file that can be viewed by your favorite spreadsheet or text editor. Additionally, by clicking the filter icon in this pane, conditions can be applied instantly. This window will only display with report charts and is not displayed for event reports.
  • Conditions (Red)(Bottom): Conditions can be used to filter the traffic information shown in reports and events. Multiple conditions can be added to drill down and inspect data. The available conditions will vary based on which application you are viewing.


Report Charts

The Report Chart contains several features to help manipulate the view of the report to your liking.

Web Filter - Report Viewer

Along the top and bottom toolbars you will find the following selections:

  • Top Toolbar:
    • Chart Type (if available): Choose from Line, Bar, Bar Overlapped, Bar 3D, Bar 3D Overlapped. This feature is not available for pie charts.
    • Customize: Build and save customized reports. Custom reports will be saved in the report selection.
    • View Events: View the individual events that were used to build the report in Events format.
    • Download: Download a .png image of the chart.
  • Bottom Toolbar:
    • Time Selection: Select the start and end time of the report. Beware that running queries (viewing reports) against a huge number of days can still be expensive if the server is busy.
    • Refresh: Force the writing of all events currently buffered in memory to the database, and then re-query the database for current data.
    • Auto Refresh: Automatically refresh every few seconds. This is useful to keep displayed while debugging an issue or if you want to see whats happening in real time.


The legend will appear at the bottom of the chart for line or bar charts, and to the right for pie charts. By clicking the fields in the legend a data series can be removed or re-added. This can help to remove clutter and focus on certain data series.

Note: Some queries are more expensive than others to run. Depending on your hardware and the current amount of traffic Untangle is processing it is possible that you can slow network traffic by running expensive queries.


Events

Event Log

Event reports show recent 1000 events sorted by time_stamp with the most recent events at the top. When opening an event report it will automatically refresh and show you the default query.

The columns along the top will show the relevant columns for the specific event report and type of event being viewed. The example above shows the Web Filter event log so you can see many columns related to the web request and what action was taken.

Along the top and bottom toolbars you will find the following selections:

  • Top Toolbar:
    • Filter: A filter can be used to instantly select any rows that match your filter string and display only those rows. Use the Case sensitive check box to match case and Clear Filters button to remove the filter and display all data.
    • Export: Export ALL events of the relevant query to a CSV text file that can be viewed by your favorite spreadsheet or text editor. This is necessary for large datasets. Browsers can not handle huge datasets in the DOM and will become not responsive if given too much data. As such, there is an 1000 event limit on events displayed in the UI, however the Export button will give you all events in a potentially very large text file. Generating and downloading the export may take some time.
  • Bottom Toolbar:
    • Number of Events: The default is to show 1,000 events. This can be increased to 10,000 or 50,000.
    • Time Selection: Select the start and end time of the report. Beware that running queries (viewing reports) against a huge number of days can still be expensive if the server is busy.
    • Refresh: Force the writing of all events currently buffered in memory to the database, and then re-query the database for current data.
    • Auto Refresh: Automatically refresh every few seconds. This is useful to keep displayed while debugging an issue or if you want to see whats happening in real time.

Finally, you have the page management which you can use to browse through the current events being displayed.

Note: Some queries are more expensive than others to run. Depending on your hardware and the current amount of traffic Untangle is processing it is possible that you can slow network traffic by running expensive queries. This can be especially true for queries that only return a few events because it will collect events up until 1000 events. If 1000 events don't exist it will scan the entire database and return whatever events do exist. For example, "Infected Web Events" in Virus Blocker typically only returns a few events. This query can take some time because it will scan the entire web request table looking for "Infected Web Events."



Conditions

The Conditions panel appears at the bottom panel and can be used to filter the queries used in both reports and events. Multiple conditions can be added to drill down and inspect data. Conditions can also be added to pie charts quickly from the Current Data window by using the filter icon.

The left hand drop down lists the available conditions that can be added. These will vary based on the application you are viewing. These can be matched to data by selecting an operator and entering the query string you're looking for. After entering a condition the report or event you are viewing will automatically refresh.

Conditions

In this example, we've added 2 conditions to see all traffic from a single client IP address (192.168.72.128) going to a specific domain (microsoft.com).

The Quick Add button also allows you to quickly create some commonly used conditions. A common use case for this is choosing which rack/policy will be queried. Once selected, this will automatically be added to the Conditions list. This also allows adding conditions for Hosts or Usernames based on the hosts and usernames currently known.

Note: Conditions that do not apply to the data being queried will be silently ignored. For example if there is a condition that says 'policy_id' '=' '1' all report entries will show the data for data when the policy_id = 1. So for example all the web filter reports will only show web filter data from the 1st policy. However, the data for Reports > System > CPU Load queries the system_stat_events table which contains no 'policy_id' column. In this case the condition will be silently ignored and the CPU load for the whole system is displayed.

Condition Operators

The second field in the condition is the logical operator that will be used in evaluating the condition value defined in the last field. In most use cases the default "=" operator is what you want to use. However, there are several other operators available that make the reports and alerts a whole lot more powerful.

A detailed outline of each operator is on the Operators page.

Conditions Example - Rack by Policy ID

In many cases, you may just want to see the traffic related to a specific rack within policy manager. This can be accomplished very easily by adding a condition using the Quick Add feature.

Quick Add


  1. Open Report Viewer or Reports tab.
  2. In the Conditions panel, select Quick Add.
  3. Choose Policy ID and the rack name.
  4. The conditions is applied and will remain applied as you switch between reports.


Alternately, you can manually enter the condition. To do this, go to Policy Manager > Settings and take not of the rack ID number. Then, in the drop down condition list, select Policy ID, select the operator =, and then enter the rack ID.

Conditions Example - Web Filter Categories

From pie charts, you can quickly add a condition from the Current Data window. This can be handy for use with the Web Filter category selection which we'll use for this example. Once the condition is applied, we can then use other reports to drill down to find out more information about the traffic such as which user might be responsible.

Quick Add


  1. Open Report Viewer or the Web Filter Reports tab.
  2. Select the Top Categories report (by size or requests). In our example, you can see Games was at the top.
  3. Next to Games, click the "filter" icon.
  4. The conditions window displays with the category name Games pre-populated.
  5. Click Done to add the condition.
  6. To find the user(s) or machine(s) generating the traffic you can click to any other report such as Top Hostnames or Top Usernames



Related Topics

Custom Reports

Reports FAQs

What is the difference between Reports and Events?

Events and Reports are now one in the same, provide real-time data for each individual application. The Events view will show the individual events that make up the reports, while reports will show a graphical interpretation of summarized events.

Why is Reports taking up all of my server's resources?

Check your Data Retention setting - if it's too high it will cause a lot of issues. Try setting it to the default of 7 to see if that helps.


Why am I not receiving an email with my Reports?

If Untangle is set to email you and you're not receiving the emails, try the Email Test at Config > Email - if you get the test mail successfully, you should also get the email from Reports. If not, you can check /var/log/exim4/mainlog and look for the error, or contact Untangle Support.


I just upgraded my Untangle box and my reports are missing. Why?

An update may have changed how Reports stores data - the next time scheduled reports are run the report index will be rebuilt, which will allow you to access the older data. Please allow one complete reporting cycle (Daily, Weekly or Monthly) if you only run that type of report.


What is the "others" column when looking at the charts in Reports?

When looking at the Top 10 of a Reports chart, others is made up of everything else not listed. You can see the Top 9 sites visited by users in a day, while others is there to give us a baseline, for example if we saw one or two users with a larger percentage than others, we'd probably want to do some investigating as to why that user is pushing more web traffic than a large portion of the organization (relative to total organization size).


The spam and phishing stats don't seem to add up. Why?

You may notice that Reports contains a certain number of phish or spam email, however the Event Logs/CSVs show a different number. This is because the graphs show the actual number of emails while the Event Logs/CSVs treat each recipient as an individual email so per-user/host reports are correct. An example is a single spam email sent to two users - it will only be counted as one (email) in the Reports, but two (delivered emails) in the Event Logs/CSVs.


Why is the timestamp column not displayed properly in Excel when I open the CSV?

To solve this please change the format of the first column to the Date format.