Unix Tools - Linux Setup, Configuration & Administration

[Gregory]

When I read about Unix tools there seems to be a lot of overlap between
them, like

shells
awk
sed
Perl
Python
C
tcl/tk
Java
Lisp

What's a beginner to do that can only learn one at a time? If I learn
Perl, is there any reason at all to look at awk and sed? What does Python
do that Perl doesn't? I like the thought of the C/tcl/tk combination,
e.g. a simple way to put a graphical interface on my C apps, or so the
legends go, but has Java made that obsolete? What does anyone use Lisp
for?

Please share some opinions.


--
"Let us learn to dream, gentlemen, then perhaps we shall find the
truth... But let us beware of publishing our dreams before they have been
put to the proof by the waking understanding." -- Friedrich August Kekulé

[Alan]

On Fri, 24 Oct 2003 01:22:25 +0000 (UTC), Gregory L. Hansen <ucs.indiana.edu> wrote: 

The shell is a command interpreter which stands between you and the kernel.

Sed is an editor, the rest are programming languages with various focusses,
or, in the case of C, a general language.

The shell includes a high-level and immensely practical programming language,
and you would be very wise to learn an sh-compatible shell like bash or zsh,
before tackling any of the other languages.

And then, once you have a handle on 'sh', skip the rest and go for broke:
learn C, (which most of the others are written in :-)

Learn the stream editor sed because it is very, very useful.

--
Alan C
Post validation at http://tinyurl.com/rv0y

[Charles]

In article <bn9ush$j0e$uits.indiana.edu>,
Gregory L. Hansen <ucs.indiana.edu> wrote: 

Learn a shell first, choose the one that you can get someone to
answer your questions one on one in person. Best is some Bourne
like shell which will be good for scripting and interactive use.

Next learn your basic unix commands as are found in "Unix in a
Nutshell." This includes awk and sed.

Then decide if you want to learn C, then Perl and the other stuff.

Perl is too cryptic to learn initially, and awk and sed are much
more straughtforward, as is C (C++ is harder to master).

So much for some initial thoughts.


Chuck Demas

--
Eat Healthy | _ _ | Nothing would be done at all,
Stay Fit | | If a man waited to do it so well,
Die Anyway | v | That no one could find fault with it.
com | \___/ | http://world.std.com/~cpd

[Walt]

Alan Connor <yyy> wrote in message news:<JH%lb.3593$news.pas.earthlink.net>... 
>
> The shell is a command interpreter which stands between you and the kernel.
>
> Sed is an editor, the rest are programming languages with various focusses,
> or, in the case of C, a general language.
>
> The shell includes a high-level and immensely practical programming language,
> and you would be very wise to learn an sh-compatible shell like bash or zsh,
> before tackling any of the other languages.
>
> And then, once you have a handle on 'sh', skip the rest and go for broke:
> learn C, (which most of the others are written in :-)
>
> Learn the stream editor sed because it is very, very useful.[/ref]


The "Advanced Bash Scripting Guide" available at tldp.org is
worth reading. It has many commented scripts.

Walt R.

[Alan]

On 23 Oct 2003 21:46:42 -0700, Walt R <net> wrote: 
>>
>> The shell is a command interpreter which stands between you and the kernel.
>>
>> Sed is an editor, the rest are programming languages with various focusses,
>> or, in the case of C, a general language.
>>
>> The shell includes a high-level and immensely practical programming language,
>> and you would be very wise to learn an sh-compatible shell like bash or zsh,
>> before tackling any of the other languages.
>>
>> And then, once you have a handle on 'sh', skip the rest and go for broke:
>> learn C, (which most of the others are written in :-)
>>
>> Learn the stream editor sed because it is very, very useful.[/ref]
>
>
> The "Advanced Bash Scripting Guide" available at tldp.org is
> worth reading. It has many commented scripts.
>
> Walt R.[/ref]

A great doent, but starting with one of the free on-line tutorials, and
asking the folks at comp.unix.shell to help you out when you get stuck, is
a very good idea.

--
Alan C
Post validation at http://tinyurl.com/rv0y

[Tim]

Removed by Administrator

[Gregory]

In article <bna4r2$e99$std.com>,
Charles Demas <com> wrote: 
>
>Learn a shell first, choose the one that you can get someone to
>answer your questions one on one in person. Best is some Bourne
>like shell which will be good for scripting and interactive use.
>
>Next learn your basic unix commands as are found in "Unix in a
>Nutshell." This includes awk and sed.
>
>Then decide if you want to learn C, then Perl and the other stuff.
>
>Perl is too cryptic to learn initially, and awk and sed are much
>more straughtforward, as is C (C++ is harder to master).
>
>So much for some initial thoughts.[/ref]

I actually have some familiarity with shells, less now than I did through
disuse of most features besides starting up programs and sometimes piping
them around. And I know C and C++, but I haven't used them much in a Unix
context. And I have a fat Unix book that describes some of that stuff,
like sed and awk but not Perl, that I plan to go through when I get Linux
in my home. But Perl has been described as doing more than sed and awk
together, removing the need to pipe things around between the two and the
shell, so I wondered whether I even need to care about them. One thing I
have in mind is a fairly elaborate e-mail filter that can keep tallies,
filter on attachments, send notifications of spam deletions, recognize
spoofed words like pe.n!s, which are things that slocal on my Sun shell
account can't do, so off-hand Perl seemed like the one to use, based
strictly on what I've read about it but no knowledge of it.

At least one person likes Python for scientific programming, but I don't
know if that does things that Perl doesn't, or makes the job especially
easier than C or Fortran would.

--
"Let us learn to dream, gentlemen, then perhaps we shall find the
truth... But let us beware of publishing our dreams before they have been
put to the proof by the waking understanding." -- Friedrich August Kekulé

[Ed]

On 10/24/2003 9:12 AM, Gregory L. Hansen wrote:
<snip> 

You could download cygwin (www.cygwin.org) while you're waiting...
 

You never need to do pipe things between sed and awk since awk can do anything
sed can do. In the context of sed/awk solutions: for small things that sed can
do easily, only use sed, and for the rest only use awk.

and the 

No, procmail, formail and/or spamassasin are some tools for that job.

Ed.

[Gregory]

In article <com>,
Ed Morton <com> wrote: 
>
>You could download cygwin (www.cygwin.org) while you're waiting...[/ref]

A cygmac would actually be more useful to me.
 
>
>You never need to do pipe things between sed and awk since awk can do anything
>sed can do. In the context of sed/awk solutions: for small things that sed can
>do easily, only use sed, and for the rest only use awk.[/ref]

Okay.
 
>
>No, procmail, formail and/or spamassasin are some tools for that job.[/ref]

None of those exist on my Sun account, only slocal, and I don't have a lot
of disk space there. And rolling my own would likely be more
instructional, so I thought I'd give it a try.

--
"Let us learn to dream, gentlemen, then perhaps we shall find the
truth... But let us beware of publishing our dreams before they have been
put to the proof by the waking understanding." -- Friedrich August Kekulé

[Stephane]

2003/10/24, 10:12(-05), Ed Morton:
[...] 
[...]

Not always. Even GNU awk can't do easily things that sed can
do in one line.

sed 's/\(.\)\1/2\1/3'

replaces the third occurrence of two consecutive same chars 'X'
with "2X" (may not work with some old seds).

The same in awk:

awk '{
line=$0; l=length; n=3; o=""
for (i=1;i<=l;i++) {
c=substr($0,i,1);
if (c == o) {
if (!--n) {
line=substr($0,1,i-2) "2" substr($0, i)
break
}
o=""
} else
o=c
}
print line}'


gawk '{print gensub(/(.)\1/,"2\\1",3)}'

doesn't work.

There's no equivalent for 'y' in awk.

--
Stéphane ["Stephane.Chazelas" at "free.fr"]

[Ed]

On 10/24/2003 11:55 AM, Stephane CHAZELAS wrote: 
>
> [...]
>
> Not always. Even GNU awk can't do easily things that sed can
> do in one line.[/ref]

In the part you snipped, I quoted the OP as saying:
 [/ref]

I suppose I should've emphasised the "need" part more. Of course I agree that
sed can do some things more easily than awk but I don't think I've ever piped
them together and I don't expect I'll ever need to. Not to say I wouldn't do it
if appropriate for a quick-and-easy hack. In any case, I was really contrasting
the abilites of awk vs perl in this context rather than contrasting sed vs awk.
 

Good to know, though I've never used it in sed either. I'm more of a "tr" guy...

Ed.

[Wayne]

: ucs.indiana.edu (Gregory L. Hansen)
: When I read about Unix tools there seems to be a lot of overlap
: between them, like shells awk sed Perl Python C tcl/tk Java Lisp
: What's a beginner to do that can only learn one at a time?

Well... doesn't the question contain the answer?
You learn them one at a time. Though I think you actually
could incrementally learn more and more about several at once.

I'd start out learning enough of bash to know the quoting,
pipelining, and file redirection notation, then dig into perl.
Visit others as inspiration strikes you. I'm sure others would
say to learn Python before perl... YMMV.

: If I learn Perl, is there any reason at all to look at awk and sed?

Lexical parsimony.

perl -ne 'print if /foo/' grep foo
perl -pe 's/foo/bar/' sed 's/foo/bar/'
perl -lane 'print "$F[1] $F[3]"' awk '{print $2 " " $4}'

: What does Python do that Perl doesn't?

Cleaner OO semantics and code that looks less like line noise.

: I like the thought of the C/tcl/tk combination, e.g. a simple way to
: put a graphical interface on my C apps, or so the legends go, but has
: Java made that obsolete?

I'd say no. You get more bang for each line of code in tcl/tk,
in terms of what widgets you can put on the screen. Also, tcl/tk
(or perl/tk, or other similar) are better glue between GUI
and the unixoid pipe paradigm, which I find very useful.

: What does anyone use Lisp for?

Elisp is used to customize and/or extend emacs.
Various lisp dialects are used for all kinds of things, but (in my
experience) most often in academic projects or in artifical reasoning
engines and the such. But it *can* be used for most anything, from
numerical ysis to theorem proving, stopping at compiler generation
and database work in between; common lisp is a very flexible and
powerful language.


Wayne Throop org http://sheol.org/throopw

[Gregory]

In article <org>, Wayne Throop <org> wrote:
 

Perl, I've read, started out as a text processing language, while Python
was mentioned on sci.physics as popular for scientific programming. Is it
fair to say that Perl's strength remains with text while Python is
stronger with numbers, or are they functionally pretty much the same,
except that one looks more like line noise than the other?
 

Oddly enough, I hadn't seen tk mentioned except in the context of tcl and
inclusions in C. But if it can be glued on to just about anything, that
seems useful.
--
"Let us learn to dream, gentlemen, then perhaps we shall find the
truth... But let us beware of publishing our dreams before they have been
put to the proof by the waking understanding." -- Friedrich August Kekulé

[Jay]

Perl is a great language to know if you do alot of scripting.

--

Tequila Stuff - http://www.tequila-stuff.com




"Gregory L. Hansen" <ucs.indiana.edu> wrote in message
news:bn9ush$j0e$uits.indiana.edu... 

[Wayne]

: ucs.indiana.edu (Gregory L. Hansen)
: Perl, I've read, started out as a text processing language, while
: Python was mentioned on sci.physics as popular for scientific
: programming. Is it fair to say that Perl's strength remains with text
: while Python is stronger with numbers, or are they functionally pretty
: much the same, except that one looks more like line noise than the other?

I'd say Python is stronger with *objects* (including in some ways
stronger with operations on numbers). Perl has more text-slinging
built in as primitives, but most of those are available as modules
in python. Perl is more universally used in unix sysadmin-like tasks
and general admin pogramming that used to be done in C or shell,
but Python can do much the same things; perl just had a head start
and is more similar to familiar unixy tools.

: Oddly enough, I hadn't seen tk mentioned except in the context of tcl
: and inclusions in C. But if it can be glued on to just about
: anything, that seems useful.

The tk widget toolset is scriptable from most anything; tcl is probably
the cleanest, tersest notation for using tk since they were designed
together, but the underlying widget set can be bound to most any
interpreted language, and I've seen it used in perl quite a bit. Lots
of people are going to gtk instead of tk, but I'm not sure of the
tradeoffs there; my impression is that gtk isn't as scriptable, in that
it is more from the compiled-language environment while tk was designed
for scripting ground up. But in gtk I am only an egg.

Note that while tk has been glued to lots of scripting languages
(well... several at least), and could be glued to many more, it's not
exactly trivial to actually glue it to a new language; don't want to
give the wrong impression.


Wayne Throop org http://sheol.org/throopw

[Alan]

On Fri, 24 Oct 2003 18:55:14 +0200, Stephane CHAZELAS <invalid> wrote: 
> [...]
>
> Not always. Even GNU awk can't do easily things that sed can
> do in one line.
>
> sed 's/\(.\)\1/2\1/3'
>
> replaces the third occurrence of two consecutive same chars 'X'
> with "2X" (may not work with some old seds).
>
> The same in awk:
>
> awk '{
> line=$0; l=length; n=3; o=""
> for (i=1;i<=l;i++) {
> c=substr($0,i,1);
> if (c == o) {
> if (!--n) {
> line=substr($0,1,i-2) "2" substr($0, i)
> break
> }
> o=""
> } else
> o=c
> }
> print line}'
>
>
> gawk '{print gensub(/(.)\1/,"2\\1",3)}'
>
> doesn't work.
>
> There's no equivalent for 'y' in awk.
>
> --
> Stéphane ["Stephane.Chazelas" at "free.fr"][/ref]


Way to go Stephane! This is going in my sed docs for the next time
the awk/sed argument comes up.

--
Alan C
Post validation at http://tinyurl.com/rv0y

[Nico]

"Gregory L. Hansen" <ucs.indiana.edu> wrote in message
news:bnbul8$94s$uits.indiana.edu... 
>
> Perl, I've read, started out as a text processing language, while Python
> was mentioned on sci.physics as popular for scientific programming. Is it
> fair to say that Perl's strength remains with text while Python is
> stronger with numbers, or are they functionally pretty much the same,
> except that one looks more like line noise than the other?[/ref]

Python's strength is also much cleaner code. There are *so many* ways to put
a revolver with one bullet in it to your code in Perl, and so many "add-on"
modules that have been poorly written or tested but somebody used for
something else you really liked because they hadn't heard of the better and
cleaner tool, that it really makes my head explode.

I just went through this with Bugzilla. *DO NOT* to the "install
Bundle::Bugzilla" trick, or you will break a bunch of your tools that have
newer releases which the "Bundle" system will grab, but have never been
tested together. Bad, bad, basd use of bleeding edge code is rife.

[LEE]

>>>>> "Gregory" == Gregory L Hansen <ucs.indiana.edu> writes:

Gregory> When I read about Unix tools there seems to be a lot of
Gregory> overlap between them, like

Gregory> shells awk sed Perl Python C tcl/tk Java Lisp

Gregory> What's a beginner to do that can only learn one at a
Gregory> time?

If you already have something (e.g. Perl that you mentioned below), go
ahead with it. You are wasting time by hessitating.


Gregory> If I learn Perl, is there any reason at all to look at
Gregory> awk and sed?

Why not? Simple awk and sed programs a easier and faster to write
than Perl. Moreover, once you've got the common CONCEPTS
(e.g. regex), it takes just a few minutes to switch from one form of
implementation to another.

I learnt sed and awk first (as a part of Shell program) before Perl.
The time spent on these sed, awk and shell were not wasted. Rather,
my knowledge of these gave me a super quick start when I picked up
Perl. So, my advice is: don't worry. Once you've got familiar with
the concepts, you can pick up others pretty quickly. There are more
in common among these tools than differences. Time spent on learning
one thing won't be wasted, and will pay off sharply as you proceed to
learn another. The only thing that is really wasting you time now
seems to be your hessistation to start learning any of these tools.



Gregory> What does Python do that Perl doesn't?

That it calls itself "Python".

Given that these are general purpose languages, they're all
Turing-complete, and hence they have equivalent power. There is
nothing that you can do in one but not another. The question that
would really bother is how fast it is to do something, and how
efficient this fast solution is. "What can be done" is not the issue.


Gregory> I like the thought of the C/tcl/tk combination, e.g. a
Gregory> simple way to put a graphical interface on my C apps, or
Gregory> so the legends go,

It's not legend. In unix, I've seen so many softwares that has a GUI
shell implemented in Tcl/Tk coupled with a small and efficient
"engine" written in C/C++. A wonderful combination taking the bests
of both tools. Although Tcl/Tk is also available on Windows, most
Windows programs know nothing about it (and hence its advantages).
So, you seldom see this GUI-shell + engine approach on Windows. The
Windows programmers thus normally use a much worse design, mixing GUI
and algorithms into a plate of spaghetti.


Gregory> but has Java made that obsolete?

No. Many people still find Java, esp. Swing, slow.

I myself don't find Swing slow. I like Swing's design. Very very
object-oriented, and realizes many of the best "design patterns". I
think Swing is not slow IF you know how to use it properly. i.e. you
have to RTFM. Lazy programmer (those who can't read a book quietly)
would of course not get it right and hence only get something slow.


Gregory> What does anyone use Lisp for?

Anything. It's also a Turing complete language.

Take a look of the huge collection of Emacs-lisp packages. There are
news/mail readers, WWW browsers, info (a hyper-text doentation
system that predates the WWW), syntax-highlighters of many programming
languages, nice frontends to CVS, RCS, ls, gcc, gdb, perldb, ...



--
Lee Sau Dan +Z05biGVm-(Big5) ~{nJX6X~}(HZ)

E-mail: uni-freiburg.de
Home page: http://www.informatik.uni-freiburg.de/~danlee

[LEE]

>>>>> "Gregory" == Gregory L Hansen <ucs.indiana.edu> writes:

Gregory> I actually have some familiarity with shells, less now
Gregory> than I did through disuse of most features besides
Gregory> starting up programs and sometimes piping them around.

Familiarity with shell (interactive commands) is not familiarity with
shell programs. People seldom use variables on the interactive
commands, but they're sort of indispensable when writing shell
scripts. Although I also use 'while' and 'for' loops occasionally in
an interactive command line, I think I'm an exception. Most people
using the command line won't go that far. But loops are very often
seen in shell script programs.


Gregory> And I know C and C++, but I haven't used them much in a
Gregory> Unix context.

There shouldn't be much difference. They syntax is the same. The
semantics (of function calls in stdlib, stdio, etc.) is the same.
Only the OS-dependent parts needs some adjustments. That's a
relatively simple thing.


Gregory> And I have a fat Unix book that describes some of that
Gregory> stuff, like sed and awk but not Perl, that I plan to go
Gregory> through when I get Linux in my home.

Go ahead. As I mentioned in another posting, it is the *concepts*
that you learn from these tools that is more important. You reuse
these concepts when you learn Perl.


Gregory> But Perl has been described as doing more than sed and
Gregory> awk together,

Yes. sed and awk are very simple tools. I'm not sure if they're even
Turing complete. Perl, OTOH, is Turing complete.


Gregory> removing the need to pipe things around between the two
Gregory> and the shell,

No. When you need to tie two programs together as in
ls -lR / | mail com
You still need to make pipes and start the two programs (or you
delegate this task to a system() command).

But I know what you mean. Shell is relatively retarded: it doesn't
have many built-in commands. It relies a lot on "external" commands.
Even 'test' and 'expr' and 'bin' are external! (I'm talking about
/bin/sh, not bash.) The advantage is that the language remains very
simple. The cons are that even simple and frequent operations like
'test' needs to be done by an external program, requiring a fork() and
using pipes to fetch the results. Perl avoids this by providing a
very right set of built-in functions and add-on modules (see also
CPAN). In most cases, you can do a lot using Perl commands without
invoking an external program. That saves the fork() call and the
overhead of pipes.



Gregory> so I wondered whether I even need to care about them.

You shouldn't. Shell scripts are meant to be used to combine existing
commands together to do something useful. Not for sophisticated
programs. (I can't understand why some people write thousand-line
long shell scripts. I can only tolerate shell scripts up to 500
lines.) Not even for efficiency. It's the fast-prototyping and quick
development that counts. This is its strength. If your shell program
suffers from bad performance (but otherwise works correctly), you
should start thinking reimplementing it in C/C++. Having a working
(though inefficient) version helps a lot when reimplementing an
efficient version.


Gregory> One thing I have in mind is a fairly elaborate e-mail
Gregory> filter that can keep tallies, filter on attachments, send
Gregory> notifications of spam deletions, recognize spoofed words
Gregory> like pe.n!s, which are things that slocal on my Sun shell
Gregory> account can't do, so off-hand Perl seemed like the one to
Gregory> use, based strictly on what I've read about it but no
Gregory> knowledge of it.

Perl seems to suit that purpose well. As mentioned before, I would
first prototype it in Perl. It's quite easy in Perl and you may be
able to come up with something working within a few days. Then, let
it run to see if it has a performance "good enough". I think it will
be that case. If not, you should evaluate how it can be improved:
where are the bottlenecks? How can it be improved (in Perl only)? Is
it better if that part (not the whole program) is implemented in C
instead? ...


Gregory> At least one person likes Python for scientific
Gregory> programming, but I don't know if that does things that
Gregory> Perl doesn't, or makes the job especially easier than C
Gregory> or Fortran would.

I can't comment about Python, as I don't know it. Given the type of
task you've mentioned, I believe it's easier to do in Perl. I choose
the language(s) to use depending on the task, not on what language(s)
I like. Using a language that suits the task saves lots of time and
energy.



--
Lee Sau Dan +Z05biGVm-(Big5) ~{nJX6X~}(HZ)

E-mail: uni-freiburg.de
Home page: http://www.informatik.uni-freiburg.de/~danlee

[LEE]

>>>>> "Tim" == Tim Haynes <vegetable.org.uk> writes:

Tim> Well, I use the Scheme dialect for a few things - anything
Tim> where I think a clean functional approach will benefit. See
Tim> <http://www.lisp.org>, <http://www.schemers.org/>. And, of
Tim> course, my "editor" is emacs, which is written in emacs
Tim> lisp...

This is not completely true. The core of the editor (and a few
utilities such as 'movemail') is written in C. Then, all the rest is
in eLisp. The core is quite minimal: a eLisp engine plus some user
interface code and I/O code. Most functionalities of the editor are
left to the eLisp code which this engine runs. Since Emacs has so
many functionalities, the proportion of code written in eLisp is very
large. The proportion of C code in Emacs is very small.

So, most functionalities of Emacs are implemented in LISP, although
the Emacs core engine is written in C.


--
Lee Sau Dan +Z05biGVm-(Big5) ~{nJX6X~}(HZ)

E-mail: uni-freiburg.de
Home page: http://www.informatik.uni-freiburg.de/~danlee