sort w/o using an array

[John Fisher]

I am trying to figure out if there is a way to do a sort that doesn't involve putting an entire file in memory. This kind of thing is available in apps like syncsort, where you give it arguments and it uses disk space/virtual memory to do the work.
All the examples I am finding are slurping it into an array or even a hash. I have a 10 field comma delimited csv file that I need to sort first by one field and run it through one of my scripts. Then sort it by a different field and run it through another of my scripts. My thought was to write a script to read in the csv file and rearrange the fields so that the field to be sorted is first. Then use a unix sort. Then run another script to put the order by correctly.
A file mockup:
"field oneA",002,"field threeGG","field fourTT", etc...
"field oneD",004,"field threeAA","field fourZZ",etc..
Say sort field three first, then sort field four second.

TIA,
John

[John W. Krahn]

John Fisher wrote:

>
> I am trying to figure out if there is a way to do a sort that
> doesn't involve putting an entire file in memory. This kind of
> thing is available in apps like syncsort, where you give it
> arguments and it uses disk space/virtual memory to do the work.

Do a search for "tape sort" and/or "external sort" and/or "merge sort".
Also have a look at Volume 3 of "The Art of Computer Programming" by
Donald E. Knuth.

> All the examples I am finding are slurping it into an array or
> even a hash. I have a 10 field comma delimited csv file that I
> need to sort first by one field and run it through one of my
> scripts. Then sort it by a different field and run it through
> another of my scripts. My thought was to write a script to read
> in the csv file and rearrange the fields so that the field to
> be sorted is first. Then use a unix sort. Then run another
> script to put the order by correctly.
> A file mockup:
> "field oneA",002,"field threeGG","field fourTT", etc...
> "field oneD",004,"field threeAA","field fourZZ",etc..
> Say sort field three first, then sort field four second.

It sounds like you need to use a real database and create indexes for
the fields in question.


John
--
use Perl;
program
fulfillment

[Rob Dixon]

John W. Krahn wrote:

>
> John Fisher wrote:

> >
> > I am trying to figure out if there is a way to do a sort that
> > doesn't involve putting an entire file in memory. This kind of
> > thing is available in apps like syncsort, where you give it
> > arguments and it uses disk space/virtual memory to do the work.

>
> Do a search for "tape sort" and/or "external sort" and/or "merge sort".
> Also have a look at Volume 3 of "The Art of Computer Programming" by
> Donald E. Knuth.
>
>

> > All the examples I am finding are slurping it into an array or
> > even a hash. I have a 10 field comma delimited csv file that I
> > need to sort first by one field and run it through one of my
> > scripts. Then sort it by a different field and run it through
> > another of my scripts. My thought was to write a script to read
> > in the csv file and rearrange the fields so that the field to
> > be sorted is first. Then use a unix sort. Then run another
> > script to put the order by correctly.
> > A file mockup:
> > "field oneA",002,"field threeGG","field fourTT", etc...
> > "field oneD",004,"field threeAA","field fourZZ",etc..
> > Say sort field three first, then sort field four second.

>
> It sounds like you need to use a real database and create indexes for
> the fields in question.

You may want to consider using the DBD::CSV database driver so that
you can use SQL to access the data directly from the CSV file. You
should then be able to combine your two scripts into one, acting on
the output from two different select statements.

HTH,

Rob