Punctuation as noise

[Michael Campbell]

> I have noticed an odd effect already, though. The

> symbol/alpha ratio is fairly low (1-2) for smaller
> programs and larger (4-6) for larger programs. I've
> tried it on sources ranging from 10 lines to 2000
> lines.

In my scrpts directory, I have a half dozen each very small ruby and perl
scripts. For the ruby stuff I got s:n of 2.4. For the perl stuff 2.7.

Note: I use some old-habit stuff from perl and c, so I might slip in a
semicolon on EOL and I definitely avoid "poetry" mode at all costs
(including not using parens around if conditions), as it's Vogon poetry to
my eyes, so my ruby probably isn't idiomatic ruby; or at least not as RUBY
idiomatic ruby as is yours.

[Paul Brannan]

On Thu, Aug 21, 2003 at 02:49:19AM +0900, Hal E. Fulton wrote:

> I'd be curious to see people's results on a large
> corpus of code (Ruby, Perl, etc.).

On 27K lines of integration test code (Ruby), I get:
Signal/noise: 458705/119801 = 3.8

On 10K lines of C++ extension code that the above tests test:
Signal/noise: 163676/40803 = 4.0

On 1.3K lines of unit test code (C++), I get:
Signal/noise: 28549/8335 = 3.4

On 544 lines of unit test code (Perl), I get:
Signal/noise: 9365/4350 = 2.2

On 4K lines of vim scripts that I regularly use, I get:
Signal/noise: 500/73 = 6.8

On the 63K lines of Ruby code that come with 1.8:
Signal/noise: 912431/218858 = 4.2

On the 60K lines of C extension code that comes with 1.8:
Signal/noise: 823062/251038 = 3.3

On the 71K lines of C code that make up the Ruby 1.8 interpreter:
Signal/noise: 904042/313624 = 2.9

> I have noticed an odd effect already, though. The
> symbol/alpha ratio is fairly low (1-2) for smaller
> programs and larger (4-6) for larger programs. I've
> tried it on sources ranging from 10 lines to 2000
> lines.

This is probably the result I would expect. If I'm writing a throwaway
script, I don't care what it looks like, so long as it works.

BTW, I played golf with your script and got:

noise = 0
alpha = 0
punc = '\',./`=[]\\;~!@#$%^&*()_+{}|:"<>?-'
white = " \t\r\n"
while buf = $stdin.read(512) do
punc_count = buf.count(punc)
white_count = buf.count(white)
noise += punc_count
alpha += buf.size - white_count - punc_count
end

ratio = alpha/noise.to_f
puts "Signal/noise: #{alpha}/#{noise} = #{'%4.1f' % ratio}"

It runs significantly faster this way.

Paul

[Paul Brannan]

Hmm, I just noticed that the script counts _'s as puntuation, and thus
has a significant bias toward camelCase. I changed the script to count
them as whitespace instead, and here are the results:

On Thu, Aug 21, 2003 at 04:32:42AM +0900, Paul Brannan wrote:

> On 27K lines of integration test code (Ruby), I get:
> Signal/noise: 458705/119801 = 3.8

Signal/noise: 458705/90543 = 5.1

> On 10K lines of C++ extension code that the above tests test:
> Signal/noise: 163676/40803 = 4.0

Signal/noise: 163676/26609 = 6.2

> On 1.3K lines of unit test code (C++), I get:
> Signal/noise: 28549/8335 = 3.4

Signal/noise: 28549/6064 = 4.7

> On 544 lines of unit test code (Perl), I get:
> Signal/noise: 9365/4350 = 2.2

Signal/noise: 9365/3710 = 2.5

> On 4K lines of vim scripts that I regularly use, I get:
> Signal/noise: 500/73 = 6.8

Signal/noise: 107184/21843 = 4.9
(I think the original result I gave here was wrong)

> On the 63K lines of Ruby code that come with 1.8:
> Signal/noise: 912431/218858 = 4.2

Signal/noise: 912431/200519 = 4.6

> On the 60K lines of C extension code that comes with 1.8:
> Signal/noise: 823062/251038 = 3.3

Signal/noise: 823062/211752 = 3.9

> On the 71K lines of C code that make up the Ruby 1.8 interpreter:
> Signal/noise: 904042/313624 = 2.9

Signal/noise: 904042/266126 = 3.4

> punc = '\',./`=[]\\;~!@#$%^&*()_+{}|:"<>?-'
> white = " \t\r\n"

punc = '\',./`=[]\\;~!@#$%^&*()+{}|:"<>?-'
white = " \t\r\n_"

Paul

[Hal E. Fulton]

----- Original Message -----
From: "Paul Brannan" <pbrannan@atdesk.com>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Wednesday, August 20, 2003 2:40 PM
Subject: Re: Punctuation as noise

> Hmm, I just noticed that the script counts _'s as puntuation, and thus
> has a significant bias toward camelCase. I changed the script to count
> them as whitespace instead, and here are the results:

Oh, thanks, have to regard that as a bug.

Hmm, if we cheated and considered @ alphanumeric, it would help
my theory even more.

But my theory doesn't appear to be really borne out anyway. I'm
sure a statistician would laugh.

I expected significant, noticeable differences between Ruby
and Perl and C and C++. Don't really see those here.

Hal

> On Thu, Aug 21, 2003 at 04:32:42AM +0900, Paul Brannan wrote:

> > On 27K lines of integration test code (Ruby), I get:
> > Signal/noise: 458705/119801 = 3.8

> Signal/noise: 458705/90543 = 5.1
>

> > On 10K lines of C++ extension code that the above tests test:
> > Signal/noise: 163676/40803 = 4.0

> Signal/noise: 163676/26609 = 6.2
>

> > On 1.3K lines of unit test code (C++), I get:
> > Signal/noise: 28549/8335 = 3.4

> Signal/noise: 28549/6064 = 4.7
>

> > On 544 lines of unit test code (Perl), I get:
> > Signal/noise: 9365/4350 = 2.2

> Signal/noise: 9365/3710 = 2.5
>

> > On 4K lines of vim scripts that I regularly use, I get:
> > Signal/noise: 500/73 = 6.8

> Signal/noise: 107184/21843 = 4.9
> (I think the original result I gave here was wrong)
>

> > On the 63K lines of Ruby code that come with 1.8:
> > Signal/noise: 912431/218858 = 4.2

> Signal/noise: 912431/200519 = 4.6
>

> > On the 60K lines of C extension code that comes with 1.8:
> > Signal/noise: 823062/251038 = 3.3

> Signal/noise: 823062/211752 = 3.9
>

> > On the 71K lines of C code that make up the Ruby 1.8 interpreter:
> > Signal/noise: 904042/313624 = 2.9

> Signal/noise: 904042/266126 = 3.4
>

> > punc = '\',./`=[]\\;~!@#$%^&*()_+{}|:"<>?-'
> > white = " \t\r\n"

> punc = '\',./`=[]\\;~!@#$%^&*()+{}|:"<>?-'
> white = " \t\r\n_"
>
> Paul
>
>

[Michael Campbell]

> > I have noticed an odd effect already, though. The

> > symbol/alpha ratio is fairly low (1-2) for smaller
> > programs and larger (4-6) for larger programs. I've
> > tried it on sources ranging from 10 lines to 2000
> > lines.

One thing I think I've noticed however is that the s:n code you posted takes
into account comments as "code". For heavily commented source, this might
skew the s:n ratio, no?

[Ryan Pavlik]

On Thu, 21 Aug 2003 05:01:30 +0900
"Michael Campbell" <michael_s_campbell@yahoo.com> wrote:

> > > I have noticed an odd effect already, though. The
> > > symbol/alpha ratio is fairly low (1-2) for smaller
> > > programs and larger (4-6) for larger programs. I've
> > > tried it on sources ranging from 10 lines to 2000
> > > lines.

>
> One thing I think I've noticed however is that the s:n code you posted takes
> into account comments as "code". For heavily commented source, this might
> skew the s:n ratio, no?
>

Another thing is that it doesn't take into account variance. For
instance, with this code, "@@@@@@@@@" would be just as noisy as
"#@!^$(*&%", and to me, variance has a _lot_ to do with noise.


--
Ryan Pavlik <rpav@users.sf.net>

"You mean super bouncy ball!" - 8BT

[Simon Strandgaard]

On Thu, 21 Aug 2003 05:20:16 +0900, Hal E. Fulton wrote:

>
> Anyway, I'd be curious as to other people's results on this little
> experiment.

I have just made a small script which can measure signal2noise
alphanum / (total-blank). I am also interested in hearing what your
signal2noise ratio are ?

Try the script at the bottom of this posting.

--
Simon Strandgaard


noise.rb => 75%

> ruby noise.rb noise.rb

alphas = 571
blanks = 156
others = 189
alpha/(alpha+others) = 0.751315789473684

>

readme's => 85%

> ruby noise.rb REA*

alphas = 1864
blanks = 540
others = 319
alpha/(alpha+others) = 0.853870819972515

> ls -la REA*

-rw-r--r-- 1 neoneye neoneye 1634 10 Aug 23:51 README
-rw-r--r-- 1 neoneye neoneye 259 3 Jul 07:46 README.ADMIN
-rw-r--r-- 1 neoneye neoneye 830 19 Aug 00:54 README.CVS

>

signal2noise in my AEditor project source => 85%

> wc -l *.rb | grep total

3663 total

> ruby noise.rb test.non_test/*.rb

alphas = 54765
blanks = 16721
others = 9551
alpha/(alpha+others) = 0.851498849430935

>

signal2noise in my AEditor project unittests => 80%

> wc -l *.rb | grep total

4998 total

> ruby noise.rb test.test/*.rb

alphas = 99522
blanks = 21985
others = 24864
alpha/(alpha+others) = 0.800106121267667

>

signal2noise in a c++ project, .cpp => 79% .h => 86%

> ruby noise.rb *.cpp

alphas = 51331
blanks = 13813
others = 13171
alpha/(alpha+others) = 0.795804781247093

> ruby noise.rb *.h

alphas = 31507
blanks = 7416
others = 4806
alpha/(alpha+others) = 0.86765070360477

>

The noise.rb scrip is here:

> expand -t2 noise.rb

def calc_history(res, file)
data = nil
File.open(file, "r"){|f| data = f.read}
ary = data.unpack('C*')
ary.each{|i| res[i]+=1}
res
end

def count_freq(history, *matchs)
count = [0] * matchs.size
count_bad = 0
history.each_index{|ascii|
ok = false
matchs.each_index{|i|
if matchs[i].index(ascii) != nil
count[i] += history[ascii]
ok = true
end
}
count_bad += history[ascii] unless ok
}
count << count_bad
count
end

# scan files
result = [0]*256
ARGV.each{|file| calc_history(result, file) }

# count frequency
match_alphanum = ('a'[0]..'z'[0]).to_a + ('A'[0]..'Z'[0]).to_a + ('0'[0]..'9'[0]).to_a
match_blanks = [32, 13, 10, 9]
alphas, blanks, others = count_freq(result, match_alphanum, match_blanks)

# output result
totals = alphas + others
ratio = alphas.to_f / totals.to_f
puts <<MSG
alphas = #{alphas}
blanks = #{blanks}
others = #{others}
alpha/(alpha+others) = #{ratio}
MSG

>

[Hal E. Fulton]

----- Original Message -----
From: "Mills Thomas (app1tam)" <app1tam@ups.com>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Wednesday, August 20, 2003 3:52 PM
Subject: Re: Punctuation as noise

> My grandfather was a student of language and a teacher. He told me once
> that punctuation (in writings representing English) exist to tell the

reader

> when to insert dead air (pauses or breaths). These reveal information to
> listeners.

I agree in essence. I'd say however that commas and pauses both
serve to communicate much the same information. I'd argue there
is not a one-to-one correspondence. But that's OT, and if you
want to discuss it, let's take that part offline.

> According to Claude Shannon and the Theory of Information, pure

information

> is indistinguishable from noise. Context is everything.

I agree again. I'm not using "noise" in a formal sense.

However, I do maintain that this:

if (x > y)

is no more clear than this:

if x > y

I prefer the latter. Many people prefer the former, but I believe
that is only a matter of habit. I don't see how parentheses
decrease ambiguity here.

In any case, one reason we construct grammars (and precedence
hierarchies) so carefully is IMO to avoid explicit disambiguation.

Sometimes we even live with the amiguity rather than use explicit
markers, as in the PLGS problem in conlangs.

Enough OT though.

Thanks,
Hal

[Simon Strandgaard]

On Thu, 21 Aug 2003 00:20:29 +0200, Simon Strandgaard wrote:

> I have just made a small script which can measure signal2noise
> alphanum / (total-blank). I am also interested in hearing what your
> signal2noise ratio are ?
>
> Try the script at the bottom of this posting.

thinking of translating ascii-historic into a signal2noise ratio:

There exists a grey area between alphanum and symbols.
Where some letters cause more noise than others... hmmm
If we consider looking at ruby-code, without any syntax hiliting, then
heredoc and comments is actually quite distibing.

This is just how I would prioritize such sig2noise-grey area.

top = much noise, low signal
bottom = low noise, much signal

@@
`` difficult to determine if its backquote or quote
''
$0 globals
// regex is not nice to the eye
\n
::
;
<<HERE we should ignore some lines
# we should ignore to end of line
""
[]
{} do-end is easy to the eye
@
() parantesis is gentle
..
+-/*= math is like our alphabet
_ underscore is quite harmless




--
Simon Strandgaard

[Eric Schwartz]

"Hal E. Fulton" <hal9000@hypermetrics.com> writes:

> For example, the parentheses in a C "if"
> statement annoy me. The terminating semicolon
> in many languages is slightly annoying. The
> frequent colons in Python bother me. And let's
> not even get into Perl.

One of the hardest things for me to get used to in Ruby still is the
lack of braces and semicolons. Without braces, I'm always nervous
about where my statments are beginning and ending (ruby-mode.el
doesn't seem to let me use {forward,backward}-sexp to move around
blocks), and without semicolons (or statment delimiter of choice), I'm
always a bit nervous about where a statement is ending.

It's not hard to write a test or two to verify my assumptions, but I
still feel a bit uneasy about it. Silly, I know, but no sillier
than your being annoyed by them, I'm sure.

I ran your program over my test harness code, and my Perl code is
about 2:1, where my Ruby is more like 3-4:1. I honestly don't know
why.

-=Eric
--
Come to think of it, there are already a million monkeys on a million
typewriters, and Usenet is NOTHING like Shakespeare.
-- Blair Houghton.

[rnicz]

"Hal E. Fulton" <hal9000@hypermetrics.com> wrote in message news:<03b301c36744$828b2520$0300a8c0@austin.rr.com >...

> I've been thinking for a day or so about
> signal vs. noise,,, in source code and
> perhaps in other contexts (ahem!!).
>
> This is just a crazy theory of mine, so
> flame away if you like.
>
> I personally find that the more "littered"
> a program is with punctuation and symbols,
> the less I like to look at it. (Yes, it's
> possible to have *too little* of that, but
> that's rare in programming languages.)

I think you would be pleased with FORTH language. In forth there is
only two kinds of syntactic elements - *whitespaces* which separates
*words* consisting of series of non-whitespaces. No punctuation, no
keywords, no reserved words. Everything what in other programing
languages is expressed using special syntactic elements in forth is
achived by appropriate sequences of 'words' in Reverse Polish
Notation. Every dot, star or semicolon have no other purpose as
execution of piece of code!

Unfortunately forth code is rather hard to read for programers used to
"normal" languages.

Simplicity of syntax makes forth very fast - it's parser simply
recognizes words separated by whitespaces and "executes" them. I think
that this is reason of your concern: simplicity of language's syntax
makes interpreter's work easier - so execution of code can be faster.

(For quick overview of forth look at
[url]http://www.wikipedia.org/wiki/Forth_programming_language[/url])
--

Best regards
RNicz

[Garance A Drosihn]

>Hal E. Fulton <hal9000@hypermetrics.com> wrote:

>>
>> As a very crude way of measuring this, I decided
>> to count alphanumerics vs. symbols in code
>> (counting whitespace as neither -- an arbitrary
> > decision on my part).

I understand what idea you're getting at, but I think this
script is too crude. It can be completely thrown off if
one script happens to use longer variable-names than
another, for instance. I tend to use longer names than
the average C programmer does. Also, when I'm writing a
script, the longer the script gets, the longer my variable
names are likely to get.

One possible side-effect: If it takes me more lines of
code to implement a given task one language than another
language, then the "longer" language is likely to have
longer variable names, and thus score lower (aka "better")
with this punctuation-scoring method.

The punctuation-scoring method would also favor languages
which use a lot of long keywords. I once worked with a
language which used long keywords by design. All reserved
words were more than <x> letters, because it assumed that
programmers would prefer to pick shorter variable names.
So, it did things like use 'whenever' instead of 'if'.

It also favors languages which force you to use words
for what most languages use punctuation for. I wonder
what kind of a score cobol would get using this method.
[at the other end of the scale, APL would not fare well
with this method, and for very very good reasons! :-) ]

So while this starts out sounding like an interesting
measurement, I think that as soon as you try to get
"hard numbers" (which look significant simply because
they are "objective values"), you'll find that the
measurement is pretty close to meaningless.

--
Garance Alistair Drosehn = [email]gad@gilead.netel.rpi.edu[/email]
Senior Systems Programmer or [email]gad@freebsd.org[/email]
Rensselaer Polytechnic Institute or [email]drosih@rpi.edu[/email]

[Drew Mills]

Hal E. Fulton wrote:

>
> I agree again. I'm not using "noise" in a formal sense.
>
> However, I do maintain that this:
>
> if (x > y)
>
> is no more clear than this:
>
> if x > y
>
> I prefer the latter. Many people prefer the former, but I believe
> that is only a matter of habit. I don't see how parentheses
> decrease ambiguity here.
>

My point was that

If (x > y)

communicates more information than

If x > y

if the human reader of your code (and, sometimes, the compiler)
comprehends better, then you have disambiguated successfully. I think
what you call habit, I might call experience. Or more precisely, a more
well-developed context in the mind of the reader.

Maybe we should all write for our target audience. In that case I
appreciate all the extra fluff.

Drew

[Ryan Pavlik]

On Thu, 21 Aug 2003 09:52:08 +0900
Eric Schwartz <emschwar@pobox.com> wrote:

<snip>

> One of the hardest things for me to get used to in Ruby still is the
> lack of braces and semicolons. Without braces, I'm always nervous
> about where my statments are beginning and ending (ruby-mode.el
> doesn't seem to let me use {forward,backward}-sexp to move around
> blocks), and without semicolons (or statment delimiter of choice), I'm
> always a bit nervous about where a statement is ending.

<snip>

Well, about semicolons, I dunno. Statement ends at the EOL, unless it
obviously doesn't (you end with a , or + or something).

However, blocks have presented an issue. I have a theory about
character variety which I won't go into depth here on (basically,
there's a certain level of character variety that makes things easy to
read at a glance; too much or too few non-alphanumerics and you have
problems), but I've come up with a solution that works for blocks for
me:

class Foo
end #c:Foo

module A
end #M:A

I would have issues, where in a class definition more that a screen, I
wouldn't always hit the right 'end', and end up adding the method
outside the class or the like. (Lots of module depth.) The issue went
away immediately as I started using block end tags. I use the
following:

#c:CLASS
#M:MODULE
#m:METHOD
#b:BLOCK <-- rarely used

Of course, as you said, in C etc., the editor shows you what block you
came from. At least in the ruby emacs mode, this isn't the case.

--
Ryan Pavlik <rpav@users.sf.net>

"Let super-dimensional physics take its course!" - 8BT

[Brett H. Williams]

On Aug 22, Ryan Pavlik wrote:

> On Thu, 21 Aug 2003 09:52:08 +0900
> Eric Schwartz <emschwar@pobox.com> wrote:
>
> <snip>

> > One of the hardest things for me to get used to in Ruby still is the
> > lack of braces and semicolons. Without braces, I'm always nervous
> > about where my statments are beginning and ending (ruby-mode.el
> > doesn't seem to let me use {forward,backward}-sexp to move around
> > blocks), and without semicolons (or statment delimiter of choice), I'm
> > always a bit nervous about where a statement is ending.

> <snip>
>
> Of course, as you said, in C etc., the editor shows you what block you
> came from. At least in the ruby emacs mode, this isn't the case.

I wrote an emacs lisp hack that acted like the highlight method of
match-paren. If I had a cursor on the beginning or end of the block, it
would automatically highlight the block for me.

But... it was not always perfect and sometimes stopped working--I never
figured out why.

Now that I've switched to vim I can just use %, which helps a lot (thanks
to Ned Konz for his matchit support).

But I still miss the visual feedback of the highlight. When it worked it
was very nice. When it intermittently stopped working I missed it. When
it went into an infinite loop I cursed it (If I could write emacs
extensions in Ruby instead of elisp, I could've fixed it

Certainly a forward-sexp should be easy to implement for a better elisp
hacker than myself--ruby-mode gives you the functions to determine if your
cursor is at the beginning or end of a block.

--
---------------------------------------------- | --------------------------
Brett Williams |
Agilent Technologies | [email]brett_williams@agilent.com[/email]
---------------------------------------------- | --------------------------

[Jason Williams]

In article <20030821110003.3869e22a.rpav@users.sf.net>, Ryan Pavlik wrote:

> I would have issues, where in a class definition more that a screen, I
> wouldn't always hit the right 'end', and end up adding the method
> outside the class or the like. (Lots of module depth.) The issue went
> away immediately as I started using block end tags. I use the
> following:
>
> #c:CLASS
> #M:MODULE
> #m:METHOD
> #b:BLOCK <-- rarely used

A tool to do this automatically would be pretty handy; any ideas?

[Austin Ziegler]

On Fri, 22 Aug 2003 03:15:22 +0900, Brett H. Williams wrote:

> But I still miss the visual feedback of the highlight. When it worked it
> was very nice. When it intermittently stopped working I missed it. When
> it went into an infinite loop I cursed it (If I could write emacs
> extensions in Ruby instead of elisp, I could've fixed it

This is where I like the context-sensitive highlighting, but also the
*folding* offered by the ruby syntax file for vim. I can collapse blocks and
see the code around them. The other thing that I find is that this *really*
helps with refactoring code.

-austin
--
austin ziegler * [email]austin@halostatue.ca[/email] * Toronto, ON, Canada
software designer * pragmatic programmer * 2003.08.24
* 00.17.56

[Brett H. Williams]

On Aug 24, Austin Ziegler wrote:

> On Fri, 22 Aug 2003 03:15:22 +0900, Brett H. Williams wrote:

> > But I still miss the visual feedback of the highlight. When it worked it
> > was very nice. When it intermittently stopped working I missed it. When
> > it went into an infinite loop I cursed it (If I could write emacs
> > extensions in Ruby instead of elisp, I could've fixed it

>
> This is where I like the context-sensitive highlighting, but also the
> *folding* offered by the ruby syntax file for vim. I can collapse blocks and
> see the code around them. The other thing that I find is that this *really*
> helps with refactoring code.

Yes, I'm just starting to touch the tip of the iceberg with vim folds.
I'll have to try collapsing the blocks and looking at surrounding code.
That sounds like an interesting idea for looking at a method at a higher
level. I sometimes fold methods to get a higher-level view of a class, but
this is potentially even more useful.

About the context-sensitive highlighting in vim--exactly what do you mean?
Is there a feature I'm not aware of? I do see that the 'end' is often
colored to match based on what it began on, but it does little in the
presence of nesting (I get one color for class/module/method ends, and
another for other ends).

--
---------------------------------------------- | --------------------------
Brett Williams | (970) 288-0475
Agilent Technologies | [email]brett_williams@agilent.com[/email]
---------------------------------------------- | --------------------------