I have tested a new Sguil agent released by Paul Halliday  last month to collect and store http traffic session into the Sguil database for web traffic analysis. If you are looking for a method to collect and mine web traffic session, this new agent is your tool. Here is an example how the httpry agent collects its log:
This Sguil client snapshot shows the traffic collected by the agent on a client and displayed in the order the web sites were accessed. Like other traffic collected by the Sguil framework, this traffic can request the packets to be analyzed with Wireshark.
Follow Paul's instructions  on how to install and configured the agent to have the traffic report to the Sguil database. I would also suggest to run a cronjob to rotate the httpry service once per day to empty the logfile, otherwise it will grow exponentially and the agent will stop processing. Consider adding sites you consider of no value to the /etc/ httpry_agent.exclude file to carefully select what you insert in your database.
Guy Bruneau IPSS Inc. gbruneau at isc dot sans dot edu
Community SANS SEC 503 coming to Ottawa Sep 2011
Jul 14th 2011
7 years ago