Access Log Analyzer

Once per day, using the cron entry file in /etc/cron.daily/websafety_report the system runs the following statistics generation script.

# run the collector being root
python3 /opt/websafety-ui/bin/collector.py 2>&1

This script parses each entry in the Squid access log files and generates the daily statistics as CSV files in /opt/websafety-ui/var/stats folder. The statistics generator is also able to parse normal Squid access log files (without web filter entries). Both normal logs and GZIP logs are processed.

After statistics generator finishes its work we will have nice daily CSV files in the statistics storage in the form of 2019-12-02 folders with data.csv file in it. If required you can upload these files into Microsoft Excel too.

Sample CSV file is shown below.

epoch   timestamp       date    hour_of_day     response_time   user_id squid_status    status_code     reply_size      request_method  request_url     domain  fldomain        mime_type       ws_iid  ws_duration     ws_timing       ws_mtime        ws_scanflags    ws_categories   ws_trusted      ws_verdict      ws_verdict_pass ws_verdict_adapt        ws_verdict_block        ws_verdict_rescan       ws_verdict_skip ws_offensive    ws_policy       ws_member       ws_module       ws_msgtype      ws_param1       ws_param2       ws_debug
1574746207      2019-11-26T06:30:07.056000      2019-11-26      6       35      192.168.4.3     NONE    200     0       CONNECT v10.vortex-win.data.microsoft.com:443   v10.vortex-win.data.microsoft.com       microsoft.com   -       0       0       0       0       0       0       0       skip    0       0       0       0       1       0       -       -       0       0       -       -       -
1574746257      2019-11-26T06:30:57.315000      2019-11-26      6       50259   192.168.4.3     TCP_TUNNEL      200     4357    CONNECT v10.vortex-win.data.microsoft.com:443   v10.vortex-win.data.microsoft.com       microsoft.com   -       0       0       0       0       0       0       0       skip    0       0       0       0       1       0       -       -       0       0       -       -       -
1574746584      2019-11-26T06:36:24.933000      2019-11-26      6       18      192.168.4.3     NONE    200     0       CONNECT tsfe.trafficshaping.dsp.mp.microsoft.com:443    tsfe.trafficshaping.dsp.mp.microsoft.com        microsoft.com   -       0       0       0       0       0       0       0       skip    0       0       0       0       1       0       -       -       0       0       -       -       -