Professional Web Applications Themes

Speed of file vrs dbase access - Perl / CGI

fatted (fatted) wrote on MMMDXC September MCMXCIII in <URL:news:4eb7646d.0306300038.2be15c80posting.goo gle.com>: }} A request for some data is processed by my perl script. The data is }} currently stored in plain text files on a linux system. Each file's }} data contains less than 600 characters. The data from 3 files will be }} required (read into the script) per request. I now have the }} opportunity, to place the data in a MySQL table. A database handle }} (using the DBI module) is already opened to the database where this }} new table would be stored.(MySQL database and data files ...

  1. #1

    Default Re: Speed of file vrs dbase access

    fatted (fatted) wrote on MMMDXC September MCMXCIII in
    <URL:news:4eb7646d.0306300038.2be15c80posting.goo gle.com>:
    }} A request for some data is processed by my perl script. The data is
    }} currently stored in plain text files on a linux system. Each file's
    }} data contains less than 600 characters. The data from 3 files will be
    }} required (read into the script) per request. I now have the
    }} opportunity, to place the data in a MySQL table. A database handle
    }} (using the DBI module) is already opened to the database where this
    }} new table would be stored.(MySQL database and data files are stored on
    }} machine where perl script is run).
    }}
    }} Would database access be quicker or slower than direct file access to
    }} the current linux files?


    First of all, that's not a Perl question. Second, it depends. It depends
    on how the data is stored, and how you make requests. Reading a single
    file from disk might even be faster than doing a single request to the
    database, if that data still needs to be fetched from disk. But if you
    need to do a linear search over all the different files, while the database
    can make use of an index, the latter is much faster than the former.
    If you know which file you need, but you have a gazillion files, it will
    take relatively long for the OS to even find the file.

    The only way to know for sure is it do some extensive testing, under
    production cirstances.



    Abigail
    --
    $_ = "\nrekcaH lreP rehtona tsuJ"; my $chop; $chop = sub {print chop; $chop};
    $chop -> () -> () -> () -> () -> () -> () -> () -> () -> () -> () -> () -> ()
    -> () -> () -> () -> () -> () -> () -> () -> () -> () -> () -> () -> () -> ()
    Abigail Guest

  2. #2

    Default Re: Speed of file vrs dbase access

    "Abigail" <abigailabigail.nl> wrote in message
    news:slrnbg07ro.el.abigailalexandra.abigail.nl...
    > fatted (fatted) wrote on MMMDXC September MCMXCIII in
    > <URL:news:4eb7646d.0306300038.2be15c80posting.goo gle.com>:
    <snippity snip>
    > First of all,
    Thanks for replying Abigail.
    > that's not a Perl question.
    I think it is I want to carry out data access using a Perl script.
    (We could argue that this is a Perl newsgroup not an OS / dbase
    newsgroup, but I think from previous posts to the group, usually, people
    don't mind briefly discussing these things, especially when popular
    modules and built in functions are used).
    > Second, it depends. It depends
    > on how the data is stored, and how you make requests.
    Data currently is stored in a plain text file on linux system, less than
    600 ascii chars per file, perhaps 100 files. Accessed using perls open
    command and <>.
    I'm thinking of storing contents of each file in a table, 1 file content
    per row. The data would then be accessed using the DBI module interface
    to MySQL (select statement). With the provision that a database handle
    had already been created (dbase handle used previously in script).
    In both cases the MySQL database and plain text files would be stored on
    the same machine as the perl script.
    > The only way to know for sure is it do some extensive testing, under
    > production cirstances.
    I was just wondering whether there would be appreciable differences,
    between the two, perhaps answers of the "Are you crazy doing it like
    that, stick to ..." variety (and maybe a because .


    Fatted Guest

  3. #3

    Default Re: Speed of file vrs dbase access

    Juha Laiho <Juha.Laihoiki.fi> wrote in message news:<bdq20r$kic$1ichaos.ichaos-int>...
    > "Fatted" <fatted> said:
    > >Data currently is stored in a plain text file on linux system, less than
    > >600 ascii chars per file, perhaps 100 files. Accessed using perls open
    > >command and <>.
    > >I'm thinking of storing contents of each file in a table, 1 file content
    > >per row.
    >
    > For these both: do you tend to keep just one sheet of paper in each
    > deskside drawer?
    I think if you had read the original post or the rest of the post
    partially quoted above, its quite clear what I'm talking about In
    the mean time I'll assume your comment was extremely witty (if not
    down right hilarious .
    > So, the organisation looks _very_ strange to me. And as your amount
    > of data is overall ~60kB, there isn't much difference how you access it.
    If the script is called 4 times per second, small differences can add
    up.
    > It seems to me that you're using the file name as a key to the contained
    > information. And by some magic you always know which file you want to
    > open. Is this so? (instead of just reading through all the files just to
    > find out that it was actually the last one that you really needed)
    My question was about the speed of access to data in a file vrs data
    in a table of a MySQL database from a perl script (I was especially
    interested in whether access to a MySQL database with a previously
    created database handle would be quicker than a system file access).
    The filename containing the relevent data is chosen earlier in the
    script, depending on input received by the script. No Magic, Just
    not relevent to my question. You assumed I knew which table to access,
    were I to use the MySQL database; so you could have continued your
    assumption that I also knew which files to access.
    > If this is so (that you have now 100 pieces of data, and are using
    > the file name as the search key, and the search key is unique), I
    > think I'd like to introduce you to DB_File. It allows you to have
    > simple key-value -databases, and retrieve single data items by the
    > key.
    >
    > Where any SQL database would enter the arena is when you have more
    > than one data relation - which you currently apparently don't have.
    > For this they'd be overkill -- but certainly usable; most possibly
    > with a model of having one table with one key column and one data
    > column.
    My question was over access speed. Not storage requirements As I
    mentioned above the filename to be accessed is chosen in the script
    based on the data received. There are several criteria for choosing
    which filename to access, which would be mimicked by have additional
    information on each row in the database (each row containing a column
    with the data). A select - where statement would be used to get the
    data, instead of logic loops in the perl script.

    Thanks for your reply.
    fatted Guest

Similar Threads

  1. MS Access Query Speed
    By pkonshak in forum Coldfusion Database Access
    Replies: 0
    Last Post: August 17th, 01:45 PM
  2. Speed of data access in arrays
    By Steven James Samuel Stapleton in forum PHP Development
    Replies: 6
    Last Post: September 9th, 09:37 PM
  3. Replies: 1
    Last Post: August 15th, 11:50 AM
  4. Access dbase connection string errors
    By J. Muenchbourg in forum ASP
    Replies: 3
    Last Post: August 8th, 10:15 PM
  5. #14562 [Asn->Csd]: dBase: Unable to obtain dBase file structure
    By zak@php.net in forum PHP Development
    Replies: 0
    Last Post: July 15th, 06:23 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139