Professional Web Applications Themes

Bourn Shell Scripts that Produce Multiple Files - FreeBSD

I wrote a Bourn Shell script which has a while loop in it that reads a file line by line. The output of the script is supposed to go to a file with a different name for each iteration of the loop. This scheme is obviously a rotten idea because all the new files end up created, but quite empty. If I take out the > $newfilename.txt directive, I get the proper output at stdout so the only problem is with changing the file name in the middle of the game several times. I think that either the output gets ...

  1. #1

    Default Bourn Shell Scripts that Produce Multiple Files

    I wrote a Bourn Shell script which has a while loop in it that
    reads a file line by line. The output of the script is supposed to go
    to a file with a different name for each iteration of the loop. This
    scheme is obviously a rotten idea because all the new files end up
    created, but quite empty. If I take out the > $newfilename.txt
    directive, I get the proper output at stdout so the only problem is
    with changing the file name in the middle of the game several times.

    I think that either the output gets lost in buffers or I am
    killing the association between the file descriptor that was first
    opened and all the new files that get created each time the loop runs.
    bash doesn't have anything like a fflush(); function,
    does it?

    I even tried piping the output through cat as in |cat 
    output to stdout and to any files you give to tee as arguments.

    The results are always the same. When I used tee, I could
    read proper output being sent through stdout, but those files were
    still as blank as ever.

    The man page for bash doesn't even contain the word flush nor
    does it discuss output buffering regarding redirection or "closing"
    open files.

    I actually made the script work by splitting it in to two
    shell scripts which caused the part producing the output to exit each
    time. That certainly caused output to go to the proper files, but I
    think there should be a way to make it all happen from one script.

    Thank you for any suggestions.

    Martin McCormick WB5AGZ Stillwater, OK
    OSU Information Technology Division Network Operations Group
    Martin Guest

  2. #2

    Default Re: Bourn Shell Scripts that Produce Multiple Files

    # Martin McCormick: 

    This sounds a bit like a truncation issue.

    If you do something like

    command1 > bar
    # some code
    command2 > bar

    then the second redirect will truncate the file to 0 bytes
    before redirecting the output from command2 into it.

    Try using >> instead of >, as it appends to the file.

    If this doesn't help, please post the script (or a simplified
    version thereof). We're not clairvoyant, you know... :)

    HTH,
    Mario
    Mario Guest

  3. #3

    Default Re: Bourn Shell Scripts that Produce Multiple Files

    Mario Hoerich writes: 

    Yes.

    My code was very similar to what you describe only more like:

    command1 > bar
    #some code
    command2 > bar2
    #should leave bar alone and open bar2
    #Instead, bar and bar2 both end up empty of anything.
     

    What's sad is, I think I tried that but forgot to remove some
    older versions of the same files created by a less elegant method so
    it looked like the new files had too many lines in them and I figured
    that appending didn't work either. I should have cleaned house first
    and things would have been fine.

    Appending did solve the problem.
     

    Here is the simplified origin of the problem:

    #! /bin/sh
    searchfor () {
    #line of awk that produces standard output
    return 0
    }
    while read currentnumber; do
    #This line does work according to your suggestion.
    searchfor >>$currentnumber.txt
    #This line was what I had which opens new files but never fills them:
    #searchfor >$currentnumber.txt
    done <~/numbers

    The value held in $currentnumber.txt changes with each loop
    iteration so we should be writing to bar2, bar3, etc and leaving bar1
    alone. It apparently does not work that way. It reminds me of what
    happens in C if a program ends without closing an open file for
    whatever reason. The buffer never gets emptied so the file is left
    either empty or partially filled depending upon luck and how much data
    got written to the disk before the abnormal end.

    My short-term problem is solved so thanks again, but it
    appears that even opening new files without appending them confuses
    the shell on previously-opened files such that you do not receive any
    data in any of the files.

    Martin McCormick WB5AGZ Stillwater, OK
    OSU Information Technology Division Network Operations Group
    Martin Guest

Similar Threads

  1. shell scripts and cron
    By TB in forum Linux / Unix Administration
    Replies: 2
    Last Post: February 7th, 05:10 AM
  2. [ANN] Protect you Unix Shell Scripts!
    By Bungisoft, in forum Linux Setup, Configuration & Administration
    Replies: 6
    Last Post: December 2nd, 10:48 PM
  3. [ANN] Protect you Unix Shell Scripts!
    By Bungisoft, Inc. in forum Linux / Unix Administration
    Replies: 3
    Last Post: November 9th, 02:43 PM
  4. Timers in shell scripts
    By Chris in forum Mac Programming
    Replies: 3
    Last Post: November 7th, 07:28 PM
  5. Converting unix shell scripts to dos batch files
    By Nikolaos Giannopoulos in forum Linux / Unix Administration
    Replies: 6
    Last Post: July 11th, 05:38 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139