The "Visit" Data Structure - Page 8

Trying to track individual visitors via the entries in a web server's access log is something of an exercise in futility. With things like proxy servers and client-side caching getting in the way, the series of accesses that show up in the log from a particular hostname or IP address can give only an approximate picture of what individual visitors are doing. Multiple users sharing the same IP address can have their activity merged into what looks like a single, very active visitor. Conversely, a single visitor can show up in the logs via a different IP address on each request, defying efforts to abstract those requests into a meaningful "visit." A proxy server at a major ISP can cache the site's pages, then satisfy hundreds of requests that never get recorded in the server's logs. Even so, it's hard not to wonder what a log file would reveal if we could pluck out the requests corresponding to specific hosts and string them together to see what patterns emerge. Many users still browse from individual host addresses without intervening proxy servers; for these users, at least, the resulting "visit" tracking provides a fascinating look at the paths being followed through the site. It's also interesting to see how many incoming requests are actually being generated by robot "spider" programs, and to study the behavior of those programs as they interact with the server. Finally, it's an interesting programming exercise to see how we can assemble and present information on these "visits." As with the data structure we used to create the SprocketExpo exhibitor directory in Chapters and , we could really benefit in this case by taking advantage of Perl's support for multilevel data structures. A hash of hashes (that is, a hash whose values are themselves hash variables) would make the task of storing and accessing information on these visits significantly easier. As it is, though, we won't be learning how to use multilevel data structures for several more chapters. That's okay; we can fake it by using the conventional variables we've been using already, just as we did for the SprocketExpo example. For the purposes of this script, we're going to define a "visit" as a series of one or more requests received from the same host, with no more than 15 minutes elapsing between one request and the next. If we get another request from the same host but more than 15 minutes has elapsed since the last one, we will treat the new request as the start of a new "visit," counting it separately in our statistics. We may as well make that 15-minute visit timeout a configuration variable up at the top of the script and store it in seconds to make our computations easier:

my $expire_time   = 900; # seconds of inactivity to consider a
                         # "visit" ended (0 = forever)

Notice how the comment tells us we can set the $expire_time variable to 0 to make the expiration time "forever." We'll see how this works in a minute. A number of other variables, visible throughout the script and declared with my near the beginning, will be used to store the information on individual visits:

$total_visits This scalar will be incremented by one for each new visit processed. Besides being used in the script's report to tell us how many visits there were in all, this count will be used to generate a unique visit number for each visit.

%visit_num This hash will have keys consisting of hostnames or IP addresses, and values consisting of the currently "working" visit number corresponding to that host.

All of the following hash variables will have keys consisting of the visit number described previously:

%host Key is visit number, value is the hostname or IP address corresponding to that visit number.

%first_time Key is visit number, value is the date and time of that visit's first access.

%last_time Key is visit number, value is the date and time of that visit's last (that is to say, most recent) access.

%last_seconds Key is visit number, value is the number of seconds returned by the &get_seconds subroutine for the date and time of that visit's last access.

%referer Key is visit number, value is the HTTP_REFERER environment variable supplied for that visit's first access.

%agent Key is visit number, value is the user- agent string supplied for that visit's first access.

We'll add all these new variables to the big my declaration up at the top of the script:

my($begin_time, $end_time, $total_hits, $total_mb, $total_views,
  $total_visits, %visit_num, %host, %first_time, %last_time,
  %last_seconds, %page_sequence, %referer, %agent);

Storing the Data - Page 7
Perl for Web Site Management
The &store_line Subroutine - Page 9