Professional Web Applications Themes

Parsing dhcpd.leases and squid access.log - PERL Beginners

Problem: Need to create a hash like data structure that contains the key as an ip address. The dhcpd leases file contains all leases handed out and the time they where assigned using UTC time. Using the epoch time stamp in the access file and the ipaddress to get the actual host name of the machine that made the request seems to be a harder thing to do then I thought. I have the following script that seems to work but it just seems like a really awful way of doing it. Access.log sample ------- 1073511381.266 24 192.168.254.116 TCP_IMS_HIT/304 209 ...

  1. #1

    Default Parsing dhcpd.leases and squid access.log

    Problem: Need to create a hash like data structure that contains the key as
    an ip address. The dhcpd leases file contains all leases handed out and the
    time they where assigned using UTC time. Using the epoch time stamp in the
    access file and the ipaddress to get the actual host name of the machine
    that made the request seems to be a harder thing to do then I thought.

    I have the following script that seems to work but it just seems like a
    really awful way of doing it.

    Access.log sample
    -------
    1073511381.266 24 192.168.254.116 TCP_IMS_HIT/304 209 GET
    [url]http://daily.webshots.com/img/bg_lt_featposter_6x1.gif[/url] - NONE/- image/gif

    Dhcpd.leases sample
    lease 192.168.254.58 {
    starts 3 2003/12/17 14:00:22;
    ends 3 2003/12/17 14:10:22;
    tstp 3 2003/12/17 14:10:22;
    binding state free;
    hardware ethernet 00:d0:b7:e1:85:b3;
    uid "\001\000\320\267\341\205\263";
    client-hostname "rpeterson";
    }
    lease 192.168.254.61 {
    starts 1 2003/12/29 14:59:11;
    ends 1 2003/12/29 14:59:17;
    tstp 1 2003/12/29 14:59:17;
    binding state free;
    hardware ethernet 00:08:74:e4:ef:3a;
    uid "\001RAS \000\010t\344\357:\000\000\000\000\000\000";
    client-hostname "pkraus";
    }

    Script
    ------
    #!/usr/bin/perl

    use strict;
    use warnings;
    use Date::Simple;
    my ( %users, %ip, %dates );
    open ( DHCP, "<dhcpd-leases.txt" ) or die ("Could not open leases file
    $!\n");
    open ( OUT, ">newlog.txt" ) or die ("Could not open log file for writing $!
    \n");

    my ( $ip, $date, $hostname );
    while ( <DHCP> ) {
    $ip = $1 if ( /lease\s([\d\.]+)/ );
    $date = $1 if ( /starts\s\d\s([\d\/]+)\s/ );
    $date =~ s/\//-/g if ($date);
    $date =~ s/(\d\d\d\d-)(\d)-/${1}0$2/ if ($date);
    $date =~ s/(\d\d\d\d-)(\d)-/${1}0$2/ if ($date);
    $date =~ s/(\d\d\d\d-\d\d-)(\d)$/${1}0$2/ if ($date);
    # print "$date\n";
    if ( /hostname "(\w+)"/ ){
    $hostname = $1;
    push( {$dates{$ip}}, "$hostname|$date");
    $users{"$ip-$date"} = $hostname;
    }
    }

    open ( LOG, "<access.txt" ) or die ("Could not open Access Log $!\n" );
    while ( <LOG> ){
    my ($timestamp,$ip,$size,$site) = (split/\s+/, $_)[0,2,4,6];
    my timestamp = (gmtime($timestamp))[2,1,0,4,3,5];
    my $time = "($timestamp[0]:$timestamp[1]:$timestamp[2])";
    my $date = ($timestamp[5]+1900) . "-" . ( $timestamp[3]+1 ) . "-" .
    $timestamp[4];
    my $hostname;

    #lookup host name
    if ($dates{$ip}){
    foreach ({$dates{$ip}}){
    my record = split /\|/;
    $date =~ s/(\d\d\d\d-)(\d-)/${1}0$2/;
    $date =~ s/(\d\d\d\d-)(\d)-/${1}0$2/;
    $date =~ s/(\d\d\d\d-\d\d-)(\d)$/${1}0$2/;
    # print "Two Dates:Squid($date)\tdhcp($record[1])\n";
    my $squiddate = Date::Simple -> new ($date);
    my $dhcpddate = Date::Simple -> new ($record[1]);

    if ($squiddate < $dhcpddate){
    $hostname = $record[0];
    last if ($hostname ne $ARGV[0]);
    print OUT "$hostname|$time|$date|$size|$site\n";
    last;
    }
    }
    }
    }

    Paul Kraus
    -----------------------
    PEL Supply Company
    Network Administrator
    -----------------------
    800 321-1264 Toll Free
    216 267-5775 Voice
    216 267-6176 Fax
    [url]www.pelsupply.com[/url]
    -----------------------

    Paul Kraus Guest

  2. #2

    Default Re: Parsing dhcpd.leases and squid access.log

    Paul Kraus wrote:
    >
    > Problem: Need to create a hash like data structure that contains the key as
    > an ip address. The dhcpd leases file contains all leases handed out and the
    > time they where assigned using UTC time. Using the epoch time stamp in the
    > access file and the ipaddress to get the actual host name of the machine
    > that made the request seems to be a harder thing to do then I thought.
    >
    > I have the following script that seems to work but it just seems like a
    > really awful way of doing it.
    >
    [snip data samples]
    >
    > use strict;
    > use warnings;
    > use Date::Simple;
    > my ( %users, %ip, %dates );
    > open ( DHCP, "<dhcpd-leases.txt" ) or die ("Could not open leases file
    > $!\n");
    > open ( OUT, ">newlog.txt" ) or die ("Could not open log file for writing $!
    > \n");
    >
    > my ( $ip, $date, $hostname );
    > while ( <DHCP> ) {
    > $ip = $1 if ( /lease\s([\d\.]+)/ );
    > $date = $1 if ( /starts\s\d\s([\d\/]+)\s/ );
    > $date =~ s/\//-/g if ($date);
    > $date =~ s/(\d\d\d\d-)(\d)-/${1}0$2/ if ($date);
    > $date =~ s/(\d\d\d\d-)(\d)-/${1}0$2/ if ($date);
    > $date =~ s/(\d\d\d\d-\d\d-)(\d)$/${1}0$2/ if ($date);
    > # print "$date\n";
    > if ( /hostname "(\w+)"/ ){
    > $hostname = $1;
    > push( {$dates{$ip}}, "$hostname|$date");
    > $users{"$ip-$date"} = $hostname;
    > }
    > }
    >
    > open ( LOG, "<access.txt" ) or die ("Could not open Access Log $!\n" );
    > while ( <LOG> ){
    > my ($timestamp,$ip,$size,$site) = (split/\s+/, $_)[0,2,4,6];
    > my timestamp = (gmtime($timestamp))[2,1,0,4,3,5];
    > my $time = "($timestamp[0]:$timestamp[1]:$timestamp[2])";
    > my $date = ($timestamp[5]+1900) . "-" . ( $timestamp[3]+1 ) . "-" .
    > $timestamp[4];
    > my $hostname;
    >
    > #lookup host name
    > if ($dates{$ip}){
    > foreach ({$dates{$ip}}){
    > my record = split /\|/;
    > $date =~ s/(\d\d\d\d-)(\d-)/${1}0$2/;
    > $date =~ s/(\d\d\d\d-)(\d)-/${1}0$2/;
    > $date =~ s/(\d\d\d\d-\d\d-)(\d)$/${1}0$2/;
    > # print "Two Dates:Squid($date)\tdhcp($record[1])\n";
    > my $squiddate = Date::Simple -> new ($date);
    > my $dhcpddate = Date::Simple -> new ($record[1]);
    >
    > if ($squiddate < $dhcpddate){
    > $hostname = $record[0];
    > last if ($hostname ne $ARGV[0]);
    > print OUT "$hostname|$time|$date|$size|$site\n";
    > last;
    > }
    > }
    > }
    > }
    Hi Paul.

    This looked like a good puzzle! I think you're on the right lines, although
    there's a lot that could be made neater. For instance:

    - You're zero-padding the month and day fields twice - both before they're
    pushed onto dates and after they're taken out.

    - I would have written the date formatting like this:

    if (/starts/) {
    $date = sprintf '%04d/%02d/%02d', m|(\d+)/(\d+)/(\d+)|;
    }

    - It may be better to write

    push {$dates{$ip}}, "$hostname|$date"

    as

    push {$dates{$ip}}, [$hostname, $date]

    so that the two values remain separate and don't need a 'split' to resolve them.

    - This is how I'd generate the log time and date in the format you want.

    my timestamp = (gmtime($timestamp));
    $timestamp[5] += 1900;
    $timestamp[4] += 1;
    my $time = sprintf '%02d:%02d:%02d', timestamp[2,1,0];
    my $date = sprintf '%04d-%02d-%02d', timestamp[5,4,3];


    - You never read from the %users hash. Do you still need this structure?

    - Dates in the format YYYY-MM-DD can be compared with the string operators (lt,
    eq, gt etc.) so you don't need Date::Simple.

    - Finally I'm not sure about your final test, which drops out at the first DHCP
    entry dated after the log date. Surely it should be the last such entry?

    HTH for now.

    Rob



    Rob Dixon Guest

Similar Threads

  1. Squid Log Pr
    By Paul Kraus in forum PERL Beginners
    Replies: 1
    Last Post: January 5th, 02:12 PM
  2. dhcpd logging
    By Peter Lingmen in forum Linux / Unix Administration
    Replies: 0
    Last Post: September 9th, 02:30 PM
  3. Solution: ISC dhcpd and AIX 4.3.3
    By Gottfried Scheckenbach in forum AIX
    Replies: 0
    Last Post: July 23rd, 10:30 AM
  4. Configure Squid
    By Stef in forum Linux Setup, Configuration & Administration
    Replies: 2
    Last Post: July 17th, 03:51 PM
  5. squid user_auth
    By Botha, Francois in forum Debian
    Replies: 4
    Last Post: July 17th, 08:30 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139