Sawmill Analytics

Analyse und Reporting
für Web | Netzwerk | Sicherheit

Zugriffs- und Datenanalyse von Server-Logs (Proxy, Mailserver, Firewall, Webserver) und Überwachung der Sicherheit & Performance, Schwachstellenanalyse.

Sawmill Analytics 8 | Loganalyse


Creating Custom Fields, Revisited

In the February 2007 Newsletter, we discussed creating custom database fields. The process involves:

  1. A Log Field
  2. A Log Filter
  3. A Database Field
  4. A Report, and adding it to the Reports Menu
  5. Optionally, creating a Cross-reference Group
In Sawmill 7, steps 1, 3 and 5 involved editing the profile CFG file; step 2 involved creating a filter with the Log Filter editor; and step 4 involved creating a report with the Report Editor. This was a somewhat involved and technical process, especially the CFG editing.

With the release of Sawmill 8, this has all gotten a lot easier. Not only is it possible to do all steps separately through the web interface (thanks to the new Log Field Editor, Database Field Editor, and Cross-reference Group Editor), but there is also a New Field Wizard to do all of this for you, in a single step. This newsletter describes using the New Field Wizard to create a custom field.

The New Field Wizard

The New Field Wizard, which is in the Config section of the profile, creates a non-aggregating (non-numerical) field, and everything associated with it. For instance, it could be used to create a field "real name" which computes the real name of a user, from the username in the log data, and displays it in a new Real Names report. Or, it can be used to extract URL parameter from the query string of a URL, and report on it as a separate field. We will be demonstrating the latter example.

The log data we're using in this example is an Apache web server access log, from an older web site which provided, among other things, Sawmill documentation for a previous version of Sawmill. It contains pages like this:


Sawmill's default behavior is to chop off everything after the question mark (to keep the database simple), so the reports will show this:


But if we want to know the different values of the "ho" ("help on") parameter (which help pages were requested), that won't do--we need this information in the database. We could disable the log filter that chops off the query string, but that would make the Page field much more complicated, and would also include all query strings, even if we only care about the "ho" parameter. Furthermore, the "ho" parameter really functions as a separate field, and it would be preferable to have a separate "Help Chapters" report which shows all the different values of the "ho" parameter, like a standard report. Finally, it would be nice to filter or zoom on this parameter like any other field. This is all possible if it is a custom field, so let's go to the New Field Wizard and set it up there. In Config -> More Options -> New Field Wizard,

New Field Wizard

New Field Wizard (Start Page)

Now we click "Click here to start the wizard" to start creating the field. In the first page that appears, we enter the name of the field. We'll call this field "Help Chapters":

New field wizard field name

New Field Wizard (Field Name)

Then we click Next, to choose the log field options:

New field wizard log field options

New Field Wizard (Log Field Options)

In most cases, you will want to leave these options alone; but if the field has a natural hierarchy in it (e.g., a pathname field, which has a hierarchy of directories), you can describe that hierarchy here, which will allow you to drill into it, in the final report. In this case, the field has no internal structure--it's just a single flat (non-hierarchical) value--so we'll leave the type as Flat, and the rest of the options at default values also. We could always change them later, in the Log Field Editor. Then we click Next, to choose the database field options:

New Field Wizard Database Field Options

New Field Wizard (Database Field Options)

These options specify the hierarchy levels to be tracked in the database, and since this is a non-hierarchical field, we can leave them at their defaults (again, you can always change them later in the Database Field Editor), and click Next, to choose the report field options:

New Field Wizard Report Field Options

New Field Wizard (Report Field Options)

Report Fields are a new concept in Sawmill 8--they correspond to columns in table reports. In earlier versions of Sawmill, database fields were used directly as columns of table reports but having Report Fields separate, allows for more flexibility, for instance by having a single hierarchical database field providing columns for Countries, Regions, and Cities (three report fields). But in most cases, the report field is based directly on the database field, with one report field per database field. That's what we'll do in this case. This is a simple "string" field, with no special formatting required, so we'll leave Display Format as "String." We'll omit the column label, to use the database label ("Help Chapters") as the label in reports. Then we click Next, to go to the Log Filter page:

New Field Wizard Log Filter

New Field Wizard (Log Filter)

In order for the custom field to have a value, it must be computed from a log filter (or parsed by the parser, but normally it's done with a filter). Here we choose the name of the filter; we'll get into the details of the filter to extract the field value later. We click Next, to go to the Report Options page:

Log Field Wizard Report

Log Field Wizard (Report Options)

This specifies the name of the report to be created, and whether it should be shown in dynamic report (e.g., when you click Reports in the web browser), or static reports (e.g., when you generate reports from the Scheduler), or both.

Finally, we click Finish. This creates the log field, creates the database field, creates the report field, creates the log filter, and creates the report, and adds it to the report menu. Now, the only thing that remains is to edit the log filter that was just created, to set the field's value. So we click Log Filters, check the new Help Chapters log filter to turn it on, choose "Expression" as the Type, and edit the Expression like this:

Log Filter

Log Filter

This uses a regular expression to extract everything after the "?ho+" section of that page, and put it in the Help Chapters log field.

Now rebuild the database (Config -> Database Info -> Rebuild Database), and view the reports, and you'll see the new field in its own report:

help chapters

Help Chapters Reports

This report shows all values of the "ho" URL parameter -- "(empty)" indicates that the hit was not a help_on hit, but some other URL. The Help Chapters field is now a full peer of all other fields in the profile; you can use it to generate reports, to zoom, to filter reports, to create pivot tables, etc.

[Article revision v1.0]

Professionelle Dienstleistungen

Sollten Sie die Anpassung von Sawmill Analytics nicht selbst vornehmen wollen, können wir Ihnen dies als Dienstleisung anbieten. Unsere Experten setzen sich gerne mit Ihnen in Verbindung, um die Reports oder sonstige Aspekte von Sawmill Analytics an Ihre Gegebenheiten und Wünsche anzupassen. Kontakt

Zur Tutorial-Übersicht

Weitere Informationen

      Live-Demonstrationen »    
© 1995-2011 HAAGE & PARTNER Computer GmbH · Impressum · Datenschutz · www.haage-partner.de