Professional Web Applications Themes

Trying to automate web site interaction that requires username and password authentication - PERL Modules

Hi everyone, I've searched and searched for an answer with no luck. I'm trying to automate interaction with a secure web site to retrieve some data. Normal interaction starts at a login page where you enter a username and password. If successful, you are presented with a form to fill out. I have successfully extracted everything I need from the form and created a single long URL. I can easily enter one long URL to log into the site and then another longer URL to retrieve my data. I've used these URLs in the script. Unfortunately, the script doesn't work ...

  1. #1

    Default Trying to automate web site interaction that requires username and password authentication

    Hi everyone,

    I've searched and searched for an answer with no luck. I'm trying to
    automate interaction with a secure web site to retrieve some data.
    Normal interaction starts at a login page where you enter a username
    and password. If successful, you are presented with a form to fill
    out. I have successfully extracted everything I need from the form
    and created a single long URL. I can easily enter one long URL to log
    into the site and then another longer URL to retrieve my data. I've
    used these URLs in the script. Unfortunately, the script doesn't work
    (obviously). The entire script (although somewhat messy) is shown
    below with all of the personal data removed. When I run the script
    everything looks like it's running okay (I've turned on some sort of
    debug so I can watch the script run) except the final output is the
    "signon" web page again. It's as if the site thinks I never
    successfully logged in. Any ideas what is going wrong here? If
    necessary I can get the debug output and post as well (I just don't
    have it on this machine right now). Thanks in advance.

    use LWP;
    use LWP::UserAgent;
    use LWP::ConnCache;
    use LWP::Debug qw(+);
    use HTTP::Cookies;

    my $browser = LWP::UserAgent->new(keep_alive=>1);
    my $cookie_jar = HTTP::Cookies->new();
    my $cache = $browser->conn_cache(LWP::ConnCache->new());
    my $loginurl = "https://something-here";
    my $profileurl = "https://something-more-here";
    my $probeurl = "https://something-even-more-here";

    $browser->cookie_jar($cookie_jar);
    $browser->credentials("something.somewhere.acme.com:443","s omething.somewhere.acme.com","\acme.com"
    => "X");
    my netscape_like_headers = (
    'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
    'Accept-Language' => 'en-US',
    'Accept-Cht' => 'iso-8859-1,*,utf-8',
    'Accept-Encoding' => 'gzip',
    'Accept' => "image/gib, image/x-bitmap, image/jpeg,
    image/pjpeg, image/png, */*",
    );

    sub TryProbedata {
    #Retrieves probe data from web site

    my $response = $browser->post($probeurl, netscape_like_headers);

    if($response->is_error){
    die "Can't get probe data: $url -- ", $response->status_line;
    }
    else{
    print "Probe data retrieve was successful. Writing to
    logfile\n";
    $returndata = $response->content;
    print LOGFILE "$returndata\n";
    }
    }

    sub loadProfile {
    #Loads profile page

    my $response = $browser->post($profileurl, netscape_like_headers); #,

    if($response->is_error){
    die "Can't load profile: $url -- ", $response->status_line;
    }
    else{
    $response->status_line;
    print "Load profile success calling probe data retrieve
    procedure\n";
    TryProbedata();
    }
    }

    sub siteLogin {
    #Logs into a given web site using username and password

    my $response = $browser->post($loginurl, netscape_like_headers); #,

    if($response->is_error){
    die "Can't sign in: $url -- ", $response->status_line;
    }
    else{
    $response->status_line;
    print "Login success calling load profile procedure\n";
    loadProfile();
    }
    }

    ################################################## ################################################## #####
    #
    # Main Program Section
    #
    ################################################## ################################################## #####

    print "Probe Collect is starting.\n";
    open(LOGFILE, ">output.txt");
    siteLogin();
    close(LOGFILE);
    print "Program complete.\n";





    Bart Swordfish Guest

  2. #2

    Default Re: Trying to automate web site interaction that requires username and password authentication

    Bart Swordfish (someonecomcast.net) wrote:
    : Hi everyone,

    : I've searched and searched for an answer with no luck. I'm trying to
    : automate interaction with a secure web site to retrieve some data.

    Perhaps there are additional things being set in the headers, especially
    the referrer, or cookies, and you aren't setting them all correctly.

    If you run a local proxy then you can display the complete exchange with a
    regular browser, and then send your requests through the proxy to compare
    them.

    I have used Proxomitron to examine my interactions using a browser, e.g.
    IE has an option to select a proxy, Proxomitron has complete instructions
    in how to do this.

    I have not used it to examine output from lwp scripts, but I am pretty
    sure you can do the same thing by telling the lwp script about the proxy.


    --

    (Paying) telecommute programming projects wanted. Simply reply to this.

    Malcolm Dew-Jones Guest

  3. #3

    Default Re: Trying to automate web site interaction that requires username and password authentication

    In article <uf1la0h47o5ko1cjbthie3g4hfjcpv0t9r4ax.com>,
    Bart Swordfish <someonecomcast.net> writes:
    >I've searched and searched for an answer with no luck. I'm trying to
    >automate interaction with a secure web site to retrieve some data.
    [snip]
    >
    >use LWP;
    >use LWP::UserAgent;
    >use LWP::ConnCache;
    >use LWP::Debug qw(+);
    >use HTTP::Cookies;
    [snip]

    Use WWW::Mechanize. Your program will become a lot shorter and
    easier for you and others to understand. I have encountered
    exactly the symptoms you describe with bank sites and there are
    several possibilities, such as the presence of hidden variables,
    or inputs set by JavaScript. You may have to inspect the source
    quite carefully.

    --
    Peter Scott
    [url]http://www.perldebugged.com/[/url]
    *** NEW *** [url]http://www.perlmedic.com/[/url]
    Peter Scott Guest

Similar Threads

  1. Contribute CS3 Requires Password
    By Marje in forum Macromedia Contribute General Discussion
    Replies: 2
    Last Post: May 19th, 01:01 PM
  2. Need help with new site (password authentication)
    By acesum1 in forum Macromedia Flash Data Integration
    Replies: 0
    Last Post: September 21st, 06:02 PM
  3. Username and password from web based authentication
    By Bart Lemmens in forum Coldfusion - Advanced Techniques
    Replies: 3
    Last Post: June 30th, 04:58 AM
  4. automate dialup web authentication?
    By Georg Schwarz in forum Mac Networking
    Replies: 0
    Last Post: August 20th, 03:24 PM
  5. Replies: 0
    Last Post: August 12th, 05:11 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139