Matching invalid characters in a URL

[Kevin Zembower]

I'm trying to throw out URLs with any invalid characters in them, like
'@". According to [url]http://www.ietf.org/rfc/rfc1738.txt[/url] :
Thus, only alphanumerics, the special characters "$-_.+!*'(),", and
reserved characters used for their reserved purposes may be used
unencoded within a URL.

I'd like to throw out a URL like
'http://jncicancerspectrum.oupjournals.org/cgi/content/full/jnci;91/3/252'
(even though this one works perfectly fine. Go figure.). I've tried:
if ($url =~ /^[^A-Za-z0-9$-_.+!*'(),]+$/) { #if there are any
invalid URL characters in the string
# Remember, special
regex characters lose their meaning inside []
print "Invalid character in URL at line $.: $url\n";
next;
}

According to my Camel, special regex characters are supposed to lose
their special functioning inside []. Yet, that obviously isn't true for
'-' used to separate the start and end of a range. I thought the fourth
'-' at '$-' was probably indicating a range, so I tried to escape it by
preceding it with a backslash or '\Q' but both gave strange errors about
uninitiated strings in concatenations.

Any suggestions? Thanks for your help and thoughts.

-Kevin Zembower

-----
E. Kevin Zembower
Unix Administrator
Johns Hopkins University/Center for Communications Programs
111 Market Place, Suite 310
Baltimore, MD 21202
410-659-6139

[Dan Anderson]

> Any suggestions? Thanks for your help and thoughts.

It is much easier to define the set all chars must be in then not. Use
the =! which is the complement of all charachters matched by =~.
Alternatively, I believe there is a c option you can use.

-Dan

[Wiggins D Anconia]

> > Any suggestions? Thanks for your help and thoughts.

>
> It is much easier to define the set all chars must be in then not. Use
> the =! which is the complement of all charachters matched by =~.
> Alternatively, I believe there is a c option you can use.
>
> -Dan

That (I presume) should be !~ instead of != to complement =~ as opposed
to ==. When trying to include a dash in a character class (and not make
it a range), [], place it as the first character in the class, when
including a carat ^ do NOT place it as the first character (as that
negates the class).

The other problem I would see is in more complex URLs, for instance @
can be used to separate the authentication portion of a URL from the
rest, a colon can indicate a port, and your example where semi-colons
can be used to separate key/value pairs in the query string. You can
likely catch 99% of bad URLs, just depends on how important that other
1% is....

Good luck,

[url]http://danconia.org[/url]

[Wiggins D Anconia]

> I'm trying to throw out URLs with any invalid characters in them, like
> '@". According to [url]http://www.ietf.org/rfc/rfc1738.txt[/url] :
> Thus, only alphanumerics, the special characters "$-_.+!*'(),", and
> reserved characters used for their reserved purposes may be used
> unencoded within a URL.
>
> I'd like to throw out a URL like
> 'http://jncicancerspectrum.oupjournals.org/cgi/content/full/jnci;91/3/252'
> (even though this one works perfectly fine. Go figure.). I've tried:
> if ($url =~ /^[^A-Za-z0-9$-_.+!*'(),]+$/) { #if there are any
> invalid URL characters in the string
> # Remember, special
> regex characters lose their meaning inside []
> print "Invalid character in URL at line $.: $url\n";
> next;
> }
>
> According to my Camel, special regex characters are supposed to lose
> their special functioning inside []. Yet, that obviously isn't true for
> '-' used to separate the start and end of a range. I thought the fourth
> '-' at '$-' was probably indicating a range, so I tried to escape it by
> preceding it with a backslash or '\Q' but both gave strange errors about
> uninitiated strings in concatenations.
>
> Any suggestions? Thanks for your help and thoughts.
>

Did you mean to leave out those characters the RFC mentions are reserved
for some schemes,

"The characters ";", "/", "?", ":", "@", "=" and "&" are the characters
which may be reserved for special meaning within a scheme."

They should be in the class as well, since you are negating it right?
Just trying to understand completely so I don't throw you off with any
dumb remarks...

[url]http://danconia.org[/url]

[Dan Anderson]

On Fri, 2004-01-09 at 16:54, Wiggins d Anconia wrote:

> > > Any suggestions? Thanks for your help and thoughts.

> >
> > It is much easier to define the set all chars must be in then not. Use
> > the =! which is the complement of all charachters matched by =~.
> > Alternatively, I believe there is a c option you can use.
> >
> > -Dan

>
> That (I presume) should be !~ instead of != to complement =~ as opposed
> to ==. When trying to include a dash in a character class (and not make
> it a range), [], place it as the first character in the class, when
> including a carat ^ do NOT place it as the first character (as that
> negates the class).

Ooops... My apologies. I was typing too quick and without much
caffeine. :-(

-Dan

[Kevin Zembower]

Thank you all for some first thoughts and clarifying questions.

I'm trying to discard any URL with any character that is not an upper- or lower-case
letter, digit, or the characters $-_.+!*'(), . I realize that some other characters can be
used in special circumstances, but I don't have to allow for any of these in my program.

I thought that my perl statement:
if ($url =~ /^[^A-Za-z0-9$-_.+!*'(),]+$/) { #if there are any invalid URL characters in the string
# Remember, special regex characters lose their meaning inside []
print "Invalid character in URL at line $.: $url\n";
next;
}
is saying:
if the variable $url contains any characters not in the set [A-Za-z0-9$-_.+!*'(),]+$/), print "Invalid ..."

So, I think I need help in two areas; Do I have my logic backwards because I'm trying to match any
character in a variable, and, How do I write the match statement to do what I want.

Thanks, again, for all your help and suggestions.

-Kevin

>>> Wiggins d Anconia <wiggins@danconia.org> 01/09/04 05:01PM >>>

> I'm trying to throw out URLs with any invalid characters in them, like
> '@". According to [url]http://www.ietf.org/rfc/rfc1738.txt[/url] :
> Thus, only alphanumerics, the special characters "$-_.+!*'(),", and
> reserved characters used for their reserved purposes may be used
> unencoded within a URL.
>
> I'd like to throw out a URL like
> 'http://jncicancerspectrum.oupjournals.org/cgi/content/full/jnci;91/3/252'
> (even though this one works perfectly fine. Go figure.). I've tried:
> if ($url =~ /^[^A-Za-z0-9$-_.+!*'(),]+$/) { #if there are any
> invalid URL characters in the string
> # Remember, special
> regex characters lose their meaning inside []
> print "Invalid character in URL at line $.: $url\n";
> next;
> }
>
> According to my Camel, special regex characters are supposed to lose
> their special functioning inside []. Yet, that obviously isn't true for
> '-' used to separate the start and end of a range. I thought the fourth
> '-' at '$-' was probably indicating a range, so I tried to escape it by
> preceding it with a backslash or '\Q' but both gave strange errors about
> uninitiated strings in concatenations.
>
> Any suggestions? Thanks for your help and thoughts.
>

Did you mean to leave out those characters the RFC mentions are reserved
for some schemes,

"The characters ";", "/", "?", ":", "@", "=" and "&" are the characters
which may be reserved for special meaning within a scheme."

They should be in the class as well, since you are negating it right?
Just trying to understand completely so I don't throw you off with any
dumb remarks...

[url]http://danconia.org[/url]

[Charles K. Clarkson]

KEVIN ZEMBOWER <KZEMBOWE@jhuccp.org]> wrote:
:
: I'm trying to discard any URL with any character that is not
: an upper- or lower-case letter, digit, or the characters
: $-_.+!*'(), . I realize that some other characters can be
: used in special circumstances, but I don't have to allow for
: any of these in my program.
:
: I thought that my perl statement:
: if ($url =~ /^[^A-Za-z0-9$-_.+!*'(),]+$/) {
: print "Invalid character in URL at line $.: $url\n";
: next;
: }
: is saying:
: if the variable $url contains any characters not in the set
: [A-Za-z0-9$-_.+!*'(),]+$/), print "Invalid ..."

No. It is saying if ALL characters are invalid ...

Ignore the character class and look at the rest. There
is no room for a valid character:

/
^ # start at the beginning of the string
[^A-Za-z0-9$-_.+!*'(),]+
$ # end at the end of the string
/

Your anchors are dragging you down. You want to find the
first invalid character. After that it doesn't matter. This
should be fine.

/[^A-Za-z0-9$-_.+!*'(),]/


HTH,

Charles K. Clarkson
--
Head Bottle Washer,
Clarkson Energy Homes, Inc.
Mobile Home Specialists
254 968-8328

[Rob Dixon]

Kevin Zembower wrote:

>
> Thank you all for some first thoughts and clarifying questions.
>
> I'm trying to discard any URL with any character that is not an upper- or lower-case
> letter, digit, or the characters $-_.+!*'(), . I realize that some other characters can be
> used in special circumstances, but I don't have to allow for any of these in my program.
>
> I thought that my perl statement:
> if ($url =~ /^[^A-Za-z0-9$-_.+!*'(),]+$/) { #if there are any invalid URL characters in the string
> # Remember, special regex characters lose their meaning inside []
> print "Invalid character in URL at line $.: $url\n";
> next;
> }
> is saying:
> if the variable $url contains any characters not in the set [A-Za-z0-9$-_.+!*'(),]+$/), print "Invalid ..."
>
> So, I think I need help in two areas; Do I have my logic backwards because I'm trying to match any
> character in a variable, and, How do I write the match statement to do what I want.

Hi Kevin.

Take note of Charles' points, but also note that Perl is trying to expand
the built-in variable $- into your regex. This is almost certainly zero
unless you're using formats, so you're including the digit zero for a second
time instead of dollar and dash.

If you escape the dollar and code

[A-Za-z0-9\$-_.+!*'(),]

instead, then your class will include all characters from dollar up to
underscore. So you need to escape both dollar and dash:

if ($url =~ /[A-Za-z0-9\$\-_.+!*'(),]/) {
# Remember MOST special regex characters lose their meaning inside []
print "Invalid character in URL at line $.: $url\n";
next;
}

HTH,

Rob