Net::Google 502 Bad gateway errors

[Ben Bullock]

Does anyone here use the Net::Google module? It seems to run very slowly,
and I keep getting "502 Bad gateway" errors. In the end I just downloaded
the search page using libwww and parsed it myself, which is almost instant.

--
sci.lang.japan FAQ: [url]http://www.sljfaq.org/afaq/afaq.html[/url]

[Henry McGuinness]

"Ben Bullock" <benkasminbullock@gmail.com> wrote in message
news:e4ef2p$bnv$1@ml.accsnet.ne.jp...

> Does anyone here use the Net::Google module? It seems to run very slowly,
> and I keep getting "502 Bad gateway" errors. In the end I just downloaded
> the search page using libwww and parsed it myself, which is almost
> instant.

Hi,
I think the "Bad Gateway" stuff is actually a Google api/server problem. I
had this and it was nothing to do with Net::Google or my query. (though you
might have posted some code :)) - I assume you have a google api key, by
the way. I looked at a google api newsgroup and this was a fairly regular
problem

Believe it or not the solution is probably: leave it a couple of minutes and
try again a few times. It does work, but sometimes rather sporadically. I
found the Yahoo api stuff much more reliable. have a look at that:
[url]http://developer.yahoo.com/[/url]

Others were having this problem in January and posted some code (though they
were using SOAP::Lite, adjust your code accordingly):

[url]http://groups.google.co.uk/group/google.public.web-apis/browse_frm/thread/6f1ef9463b961feb/091e5c9505d5f81a?lnk=st&q=google+api+502+bad+gatew ay&rnum=1&hl=en#091e5c9505d5f81a[/url]

cheers
Henry

[Ben Bullock]

"Henry McGuinness" <henry.mcguinness@dphpc.ox.ac.uk> wrote in message
news:e4hbvp$le9$1@news.ox.ac.uk...

>
> "Ben Bullock" <benkasminbullock@gmail.com> wrote in message
> news:e4ef2p$bnv$1@ml.accsnet.ne.jp...

>> Does anyone here use the Net::Google module? It seems to run very slowly,
>> and I keep getting "502 Bad gateway" errors. In the end I just downloaded
>> the search page using libwww and parsed it myself, which is almost
>> instant.

> I think the "Bad Gateway" stuff is actually a Google api/server problem. I
> had this and it was nothing to do with Net::Google or my query. (though
> you might have posted some code :)) - I assume you have a google api key,
> by the way. I looked at a google api newsgroup and this was a fairly
> regular problem

Yes, I've got one of those.

> Believe it or not the solution is probably: leave it a couple of minutes
> and try again a few times. It does work, but sometimes rather
> sporadically. I found the Yahoo api stuff much more reliable. have a look
> at that: [url]http://developer.yahoo.com/[/url]

OK, but what about just putting a search URL together then grabbing the page
via libwww and parsing it using regular expressions though? That works
almost every time, I never get any 502 errors, and the results are virtually
instant. I can't see why I would want to use this Net::Google thing in the
first place.

> Others were having this problem in January and posted some code (though
> they were using SOAP::Lite, adjust your code accordingly):

That is where the error comes in, it gives the line numbers from that code.
I'm not sure I'd want to spend hours fixing it though, because I'm far from
being an expert in Perl and it would take a lot of time for me to do so. I
suppose the nature of my question was to try to find out why anyone would
use Net::Google when it seems to endlessly choke and run very slowly, and
it's easy to get the stuff from Google via another method.

[John Bokma]

"Ben Bullock" <benkasminbullock@gmail.com> wrote:

> Does anyone here use the Net::Google module? It seems to run very
> slowly, and I keep getting "502 Bad gateway" errors.

Yes, blame it on Google. I read similar complaints in the python
newsgroup, and I replied there with: just retry.

Somehow the API is quite unstable on Google's side for at least the past
two weeks. About 3 weeks ago I couldn't even access the wsdl file most of
the times.


--
John Bokma Freelance software developer
&
Experienced Perl programmer: [url]http://castleamber.com/[/url]

[John Bokma]

"Henry McGuinness" <henry.mcguinness@dphpc.ox.ac.uk> wrote:

> Others were having this problem in January and posted some code
> (though they were using SOAP::Lite, adjust your code accordingly):

Net::Google uses SOAP::Lite, so that's probably not changing a single
thing :-)

OTOH, I suspect that the latest Net::Google has some bugs regarding
booleans, and hence some options are always on (or off, can't recall).

I haven't had time to look into this further, but it you notice odd
results, you might want to check what Net::Google is making SOAP::Lite do
(IIRC there is a lot of hacking going in order to talk to SOAP::Lite)

--
John Bokma Freelance software developer
&
Experienced Perl programmer: [url]http://castleamber.com/[/url]

[Henry McGuinness]

"Ben Bullock" <benkasminbullock@gmail.com> wrote in message
news:e4jddl$h67$1@ml.accsnet.ne.jp...

> "Henry McGuinness" <henry.mcguinness@dphpc.ox.ac.uk> wrote in message
> news:e4hbvp$le9$1@news.ox.ac.uk...

>> Believe it or not the solution is probably: leave it a couple of minutes
>> and try again a few times. It does work, but sometimes rather
>> sporadically. I found the Yahoo api stuff much more reliable. have a look
>> at that: [url]http://developer.yahoo.com/[/url]

>
> OK, but what about just putting a search URL together then grabbing the
> page via libwww and parsing it using regular expressions though? That
> works almost every time, I never get any 502 errors, and the results are
> virtually instant. I can't see why I would want to use this Net::Google
> thing in the first place.

Maybe, but watch google's terms of service on this. They don't like
"automated queries" and are not specific about all the types of query this
includes: [url]http://www.google.co.uk/intl/en/terms_of_service.html[/url]

>

>> Others were having this problem in January and posted some code (though
>> they were using SOAP::Lite, adjust your code accordingly):

>
> That is where the error comes in, it gives the line numbers from that
> code.

Yes. What I meant was you'll have to change their SOAP::Lite syntax to the
Net::Google query you want to do so you'll have:

....do {

#your Net::Google search here instead of the doGoogleSearch stuff...
if ( $@ ) { ......




cheers
Henry

[Ben Bullock]

"Henry McGuinness" <henry.mcguinness@dphpc.ox.ac.uk> wrote in message
news:e4k6m1$lcf$1@news.ox.ac.uk...

>
> "Ben Bullock" <benkasminbullock@gmail.com> wrote in message
> news:e4jddl$h67$1@ml.accsnet.ne.jp...

>> "Henry McGuinness" <henry.mcguinness@dphpc.ox.ac.uk> wrote in message
>> news:e4hbvp$le9$1@news.ox.ac.uk...

>

>>> Believe it or not the solution is probably: leave it a couple of minutes
>>> and try again a few times. It does work, but sometimes rather
>>> sporadically. I found the Yahoo api stuff much more reliable. have a
>>> look at that: [url]http://developer.yahoo.com/[/url]

>>
>> OK, but what about just putting a search URL together then grabbing the
>> page via libwww and parsing it using regular expressions though? That
>> works almost every time, I never get any 502 errors, and the results are
>> virtually instant. I can't see why I would want to use this Net::Google
>> thing in the first place.

>
> Maybe, but watch google's terms of service on this. They don't like
> "automated queries" and are not specific about all the types of query this
> includes: [url]http://www.google.co.uk/intl/en/terms_of_service.html[/url]

I'm guessing that would be more directed at people like scroogle
([url]http://www.scroogle.org/[/url]) than someone like me, who's unlikely to download
more than 100 pages a day.