Statistics CollectorΒΆ
As explained in the previous article, once per day using the cron entry file in /etc/cron.daily/websafety_report the system runs the following statistics generation script.
# run the collector being root
python3 /opt/websafety-ui/bin/collector.py 2>&1
This script parses each entry in the Squid access log files and generates the daily statistics as CSV files in /opt/websafety-ui/var/stats folder. The statistics generator is also able to parse normal Squid access log files (without web filter entries). Both normal logs and GZIP logs are processed.
After statistics generator finishes its work we will have nice daily CSV files in the statistics storage in the form of 2019-12-02 folders with data.csv file in it. If required you can upload these files into Microsoft Excel too.
Sample CSV file is shown below.
epoch timestamp date hour_of_day response_time user_id squid_status status_code reply_size request_method request_url domain fldomain mime_type ws_iid ws_duration ws_timing ws_mtime ws_scanflags ws_categories ws_trusted ws_verdict ws_verdict_pass ws_verdict_adapt ws_verdict_block ws_verdict_rescan ws_verdict_skip ws_offensive ws_policy ws_member ws_module ws_msgtype ws_param1 ws_param2 ws_debug
1574746207 2019-11-26T06:30:07.056000 2019-11-26 6 35 192.168.4.3 NONE 200 0 CONNECT v10.vortex-win.data.microsoft.com:443 v10.vortex-win.data.microsoft.com microsoft.com - 0 0 0 0 0 0 0 skip 0 0 0 0 1 0 - - 0 0 - - -
1574746257 2019-11-26T06:30:57.315000 2019-11-26 6 50259 192.168.4.3 TCP_TUNNEL 200 4357 CONNECT v10.vortex-win.data.microsoft.com:443 v10.vortex-win.data.microsoft.com microsoft.com - 0 0 0 0 0 0 0 skip 0 0 0 0 1 0 - - 0 0 - - -
1574746584 2019-11-26T06:36:24.933000 2019-11-26 6 18 192.168.4.3 NONE 200 0 CONNECT tsfe.trafficshaping.dsp.mp.microsoft.com:443 tsfe.trafficshaping.dsp.mp.microsoft.com microsoft.com - 0 0 0 0 0 0 0 skip 0 0 0 0 1 0 - - 0 0 - - -