#40494 [NEW]: Memory problem with ZipArchive::addFile()

Ask a Question related to PHP Bugs, Design and Development.

  1. #1

    Default #40494 [NEW]: Memory problem with ZipArchive::addFile()

    From: foster dot graeme at gmail dot com
    Operating system: Linux
    PHP version: 5.2.1
    PHP Bug Type: Zip Related
    Bug description: Memory problem with ZipArchive::addFile()

    Description:
    ------------
    When adding files to an archive, (using successive ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. This
    can result in an out of memory error, a temporary fix is to close the
    archive and then reopen it within the php code.
    An idea solution would be to compress the file when it is added, probably
    in function _zip_replace(), but I don't know what the implications of this
    would be. It would certainly require a rewrite of the ugly function
    zip_close().


    --
    Edit bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    --
    Try a CVS snapshot (PHP 4.4): [url]http://bugs.php.net/fix.php?id=40494&r=trysnapshot44[/url]
    Try a CVS snapshot (PHP 5.2): [url]http://bugs.php.net/fix.php?id=40494&r=trysnapshot52[/url]
    Try a CVS snapshot (PHP 6.0): [url]http://bugs.php.net/fix.php?id=40494&r=trysnapshot60[/url]
    Fixed in CVS: [url]http://bugs.php.net/fix.php?id=40494&r=fixedcvs[/url]
    Fixed in release: [url]http://bugs.php.net/fix.php?id=40494&r=alreadyfixed[/url]
    Need backtrace: [url]http://bugs.php.net/fix.php?id=40494&r=needtrace[/url]
    Need Reproduce Script: [url]http://bugs.php.net/fix.php?id=40494&r=needscript[/url]
    Try newer version: [url]http://bugs.php.net/fix.php?id=40494&r=oldversion[/url]
    Not developer issue: [url]http://bugs.php.net/fix.php?id=40494&r=support[/url]
    Expected behavior: [url]http://bugs.php.net/fix.php?id=40494&r=notwrong[/url]
    Not enough info: [url]http://bugs.php.net/fix.php?id=40494&r=notenoughinfo[/url]
    Submitted twice: [url]http://bugs.php.net/fix.php?id=40494&r=submittedtwice[/url]
    register_globals: [url]http://bugs.php.net/fix.php?id=40494&r=globals[/url]
    PHP 3 support discontinued: [url]http://bugs.php.net/fix.php?id=40494&r=php3[/url]
    Daylight Savings: [url]http://bugs.php.net/fix.php?id=40494&r=dst[/url]
    IIS Stability: [url]http://bugs.php.net/fix.php?id=40494&r=isapi[/url]
    Install GNU Sed: [url]http://bugs.php.net/fix.php?id=40494&r=gnused[/url]
    Floating point limitations: [url]http://bugs.php.net/fix.php?id=40494&r=float[/url]
    No Zend Extensions: [url]http://bugs.php.net/fix.php?id=40494&r=nozend[/url]
    MySQL Configuration Error: [url]http://bugs.php.net/fix.php?id=40494&r=mysqlcfg[/url]
    foster dot graeme at gmail dot com Guest

  2. Similar Questions and Discussions

    1. #39784 [NEW]: Can't make an instance of ZipArchive
      From: itsmegawtf at gmail dot com Operating system: Gentoo linux PHP version: 5.2.0 PHP Bug Type: Zip Related Bug...
    2. #39742 [NEW]: Class 'ZipArchive' not found
      From: eric dot leseint at ac-poitiers dot fr Operating system: Windows 2003 server PHP version: 5.2.0 PHP Bug Type: ...
    3. #39506 [NEW]: Archive corrupt with ZipArchive::addFile method
      From: hscheidegger at zumbach dot ch Operating system: Windows XP PHP version: 5.2.0 PHP Bug Type: Zip Related Bug...
    4. #38944 [NEW]: ZipArchive exits with SEGV
      From: judas dot iscariote at gmail dot com Operating system: linux PHP version: 5CVS-2006-09-25 (CVS) PHP Bug Type: Zip...
    5. memory problem/memory leak
      Hi I have a problem with shockwave player running in Internet Explorer. My program consists of a controller movie which loads in content files in...
  3. #2

    Default #40494 [Opn->Asn]: Memory problem with ZipArchive::addFile()

    ID: 40494
    Updated by: [email]tony2001@php.net[/email]
    Reported By: foster dot graeme at gmail dot com
    -Status: Open
    +Status: Assigned
    -Bug Type: Zip Related
    +Bug Type: Feature/Change Request
    Operating System: Linux
    PHP Version: 5.2.1
    -Assigned To:
    +Assigned To: pajoye


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 10:22:25] foster dot graeme at gmail dot com

    Description:
    ------------
    When adding files to an archive, (using successive
    ZipArchive::addFile() commands) the compression doesn't happen until
    the file is closed. This can result in an out of memory error, a
    temporary fix is to close the archive and then reopen it within the php
    code.
    An idea solution would be to compress the file when it is added,
    probably in function _zip_replace(), but I don't know what the
    implications of this would be. It would certainly require a rewrite of
    the ugly function zip_close().



    ------------------------------------------------------------------------


    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    tony2001@php.net Guest

  4. #3

    Default #40494 [Asn->Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    Updated by: [email]pajoye@php.net[/email]
    Reported By: foster dot graeme at gmail dot com
    -Status: Assigned
    +Status: Bogus
    Bug Type: Feature/Change Request
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    "When adding files to an archive, (using successive
    ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. "

    Yes, we do it while finalizing the archive.

    " This can result in an out of memory error, "

    You will run out of file ID before running out of memory. It does not
    really use many memory, only the file names and file handlers.

    I suppose you are talking about the file handlers?

    "It would certainly require a rewrite of the ugly function
    zip_close()"

    What is ugly in this function? Or do you have a portable way to lock a
    file until the archive creation is done?

    I think you refer to the file handlers limitation. There is already a
    bug about it and I plan to add a special (less safe) mode. This mode
    will allow one to add only the paths without checks, errors will occur
    only when the archive is closed. But that's a feature addition not a
    bug fix.

    I close this bug (not a bug > bogus).

    Thanks for your report!


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 10:22:25] foster dot graeme at gmail dot com

    Description:
    ------------
    When adding files to an archive, (using successive
    ZipArchive::addFile() commands) the compression doesn't happen until
    the file is closed. This can result in an out of memory error, a
    temporary fix is to close the archive and then reopen it within the php
    code.
    An idea solution would be to compress the file when it is added,
    probably in function _zip_replace(), but I don't know what the
    implications of this would be. It would certainly require a rewrite of
    the ugly function zip_close().



    ------------------------------------------------------------------------


    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    pajoye@php.net Guest

  5. #4

    Default #40494 [Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    User updated by: foster dot graeme at gmail dot com
    Reported By: foster dot graeme at gmail dot com
    Status: Bogus
    -Bug Type: Feature/Change Request
    +Bug Type: Zip Related
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    Maybe I need to explain this problem a little more.

    I am trying to archive a folder on the server, at the moment it
    contains 5609 folders and 11,221 files. The script loops through the
    files adding them to the archive using the addFile() method. After the
    first 1002 files I get a ZIPARCHIVE::ER_OPEN. If I close the archive
    and the open it again I still have that error. However, if I close the
    archive and open it before I get that error then I can archive all
    11,221 files.

    Since closing the file and re-opening fixes the problem (so long as I
    do that before I get the error) Then may I suggest that closing an
    archive will clear the status. Obviously, it would be good if this
    wasn't necessary, in thatthe code could catch the problem and allocate
    extra file handles if that is the problem.


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 11:41:24] [email]pajoye@php.net[/email]

    "When adding files to an archive, (using successive
    ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. "

    Yes, we do it while finalizing the archive.

    " This can result in an out of memory error, "

    You will run out of file ID before running out of memory. It does not
    really use many memory, only the file names and file handlers.

    I suppose you are talking about the file handlers?

    "It would certainly require a rewrite of the ugly function
    zip_close()"

    What is ugly in this function? Or do you have a portable way to lock a
    file until the archive creation is done?

    I think you refer to the file handlers limitation. There is already a
    bug about it and I plan to add a special (less safe) mode. This mode
    will allow one to add only the paths without checks, errors will occur
    only when the archive is closed. But that's a feature addition not a
    bug fix.

    I close this bug (not a bug > bogus).

    Thanks for your report!

    ------------------------------------------------------------------------

    [2007-02-15 10:22:25] foster dot graeme at gmail dot com

    Description:
    ------------
    When adding files to an archive, (using successive
    ZipArchive::addFile() commands) the compression doesn't happen until
    the file is closed. This can result in an out of memory error, a
    temporary fix is to close the archive and then reopen it within the php
    code.
    An idea solution would be to compress the file when it is added,
    probably in function _zip_replace(), but I don't know what the
    implications of this would be. It would certainly require a rewrite of
    the ugly function zip_close().



    ------------------------------------------------------------------------


    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    foster dot graeme at gmail dot com Guest

  6. #5

    Default #40494 [Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    Updated by: [email]pajoye@php.net[/email]
    Reported By: foster dot graeme at gmail dot com
    Status: Bogus
    Bug Type: Zip Related
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    See:

    [url]http://pecl.php.net/bugs/bug.php?id=9443[/url]

    "it would be good if this wasn't necessary, in thatthe code could catch
    the problem and allocate extra file handles if that is the problem."

    This is not something I can control. The operating system defines it
    and there is no way for me to increase this value.

    I suggest you to close and reopen it every 1000 or so (or even 255 if
    you want to go on the safest way, ie old windows).

    Future releases will have a different mode, where the checks will done
    only when you close the archives.


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 13:14:57] foster dot graeme at gmail dot com

    Maybe I need to explain this problem a little more.

    I am trying to archive a folder on the server, at the moment it
    contains 5609 folders and 11,221 files. The script loops through the
    files adding them to the archive using the addFile() method. After the
    first 1002 files I get a ZIPARCHIVE::ER_OPEN. If I close the archive
    and the open it again I still have that error. However, if I close the
    archive and open it before I get that error then I can archive all
    11,221 files.

    Since closing the file and re-opening fixes the problem (so long as I
    do that before I get the error) Then may I suggest that closing an
    archive will clear the status. Obviously, it would be good if this
    wasn't necessary, in thatthe code could catch the problem and allocate
    extra file handles if that is the problem.

    ------------------------------------------------------------------------

    [2007-02-15 11:41:24] [email]pajoye@php.net[/email]

    "When adding files to an archive, (using successive
    ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. "

    Yes, we do it while finalizing the archive.

    " This can result in an out of memory error, "

    You will run out of file ID before running out of memory. It does not
    really use many memory, only the file names and file handlers.

    I suppose you are talking about the file handlers?

    "It would certainly require a rewrite of the ugly function
    zip_close()"

    What is ugly in this function? Or do you have a portable way to lock a
    file until the archive creation is done?

    I think you refer to the file handlers limitation. There is already a
    bug about it and I plan to add a special (less safe) mode. This mode
    will allow one to add only the paths without checks, errors will occur
    only when the archive is closed. But that's a feature addition not a
    bug fix.

    I close this bug (not a bug > bogus).

    Thanks for your report!

    ------------------------------------------------------------------------

    [2007-02-15 10:22:25] foster dot graeme at gmail dot com

    Description:
    ------------
    When adding files to an archive, (using successive
    ZipArchive::addFile() commands) the compression doesn't happen until
    the file is closed. This can result in an out of memory error, a
    temporary fix is to close the archive and then reopen it within the php
    code.
    An idea solution would be to compress the file when it is added,
    probably in function _zip_replace(), but I don't know what the
    implications of this would be. It would certainly require a rewrite of
    the ugly function zip_close().



    ------------------------------------------------------------------------


    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    pajoye@php.net Guest

  7. #6

    Default #40494 [Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    User updated by: foster dot graeme at gmail dot com
    Reported By: foster dot graeme at gmail dot com
    Status: Bogus
    Bug Type: Zip Related
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    Okay thanks for the explanation, I understand the problem a little
    better. I still think that it would be nice if there was some way for
    the system to manage this.

    I was thinking along the lines of a function to flush the files so that
    the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was opened.


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 13:23:36] [email]pajoye@php.net[/email]

    See:

    [url]http://pecl.php.net/bugs/bug.php?id=9443[/url]

    "it would be good if this wasn't necessary, in thatthe code could catch
    the problem and allocate extra file handles if that is the problem."

    This is not something I can control. The operating system defines it
    and there is no way for me to increase this value.

    I suggest you to close and reopen it every 1000 or so (or even 255 if
    you want to go on the safest way, ie old windows).

    Future releases will have a different mode, where the checks will done
    only when you close the archives.

    ------------------------------------------------------------------------

    [2007-02-15 13:14:57] foster dot graeme at gmail dot com

    Maybe I need to explain this problem a little more.

    I am trying to archive a folder on the server, at the moment it
    contains 5609 folders and 11,221 files. The script loops through the
    files adding them to the archive using the addFile() method. After the
    first 1002 files I get a ZIPARCHIVE::ER_OPEN. If I close the archive
    and the open it again I still have that error. However, if I close the
    archive and open it before I get that error then I can archive all
    11,221 files.

    Since closing the file and re-opening fixes the problem (so long as I
    do that before I get the error) Then may I suggest that closing an
    archive will clear the status. Obviously, it would be good if this
    wasn't necessary, in thatthe code could catch the problem and allocate
    extra file handles if that is the problem.

    ------------------------------------------------------------------------

    [2007-02-15 11:41:24] [email]pajoye@php.net[/email]

    "When adding files to an archive, (using successive
    ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. "

    Yes, we do it while finalizing the archive.

    " This can result in an out of memory error, "

    You will run out of file ID before running out of memory. It does not
    really use many memory, only the file names and file handlers.

    I suppose you are talking about the file handlers?

    "It would certainly require a rewrite of the ugly function
    zip_close()"

    What is ugly in this function? Or do you have a portable way to lock a
    file until the archive creation is done?

    I think you refer to the file handlers limitation. There is already a
    bug about it and I plan to add a special (less safe) mode. This mode
    will allow one to add only the paths without checks, errors will occur
    only when the archive is closed. But that's a feature addition not a
    bug fix.

    I close this bug (not a bug > bogus).

    Thanks for your report!

    ------------------------------------------------------------------------

    [2007-02-15 10:22:25] foster dot graeme at gmail dot com

    Description:
    ------------
    When adding files to an archive, (using successive
    ZipArchive::addFile() commands) the compression doesn't happen until
    the file is closed. This can result in an out of memory error, a
    temporary fix is to close the archive and then reopen it within the php
    code.
    An idea solution would be to compress the file when it is added,
    probably in function _zip_replace(), but I don't know what the
    implications of this would be. It would certainly require a rewrite of
    the ugly function zip_close().



    ------------------------------------------------------------------------


    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    foster dot graeme at gmail dot com Guest

  8. #7

    Default #40494 [Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    Updated by: [email]pajoye@php.net[/email]
    Reported By: foster dot graeme at gmail dot com
    Status: Bogus
    Bug Type: Zip Related
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    "I still think that it would be nice if there was some way for the
    system to manage this."

    It is in the TODO list. As I said three times already in this
    discussion. The solution is to add different modes:
    - commit at the end when the archive is close
    - immediate addition (will be much slower)

    And again, it is in my TODOs already. I cannot tell when they will be
    available (I do it on my free time).

    In the meantime a simple:

    if (($zip->numFiles % $yourlimit) == 0) {close; reopen;}

    will do it.



    "the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was
    opened."

    This solution does not work.The limit is arbitrary. There is no way to
    get an exact value (and I doubt php is the only running process).



    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 14:02:51] foster dot graeme at gmail dot com

    Okay thanks for the explanation, I understand the problem a little
    better. I still think that it would be nice if there was some way for
    the system to manage this.

    I was thinking along the lines of a function to flush the files so that
    the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was opened.

    ------------------------------------------------------------------------

    [2007-02-15 13:23:36] [email]pajoye@php.net[/email]

    See:

    [url]http://pecl.php.net/bugs/bug.php?id=9443[/url]

    "it would be good if this wasn't necessary, in thatthe code could catch
    the problem and allocate extra file handles if that is the problem."

    This is not something I can control. The operating system defines it
    and there is no way for me to increase this value.

    I suggest you to close and reopen it every 1000 or so (or even 255 if
    you want to go on the safest way, ie old windows).

    Future releases will have a different mode, where the checks will done
    only when you close the archives.

    ------------------------------------------------------------------------

    [2007-02-15 13:14:57] foster dot graeme at gmail dot com

    Maybe I need to explain this problem a little more.

    I am trying to archive a folder on the server, at the moment it
    contains 5609 folders and 11,221 files. The script loops through the
    files adding them to the archive using the addFile() method. After the
    first 1002 files I get a ZIPARCHIVE::ER_OPEN. If I close the archive
    and the open it again I still have that error. However, if I close the
    archive and open it before I get that error then I can archive all
    11,221 files.

    Since closing the file and re-opening fixes the problem (so long as I
    do that before I get the error) Then may I suggest that closing an
    archive will clear the status. Obviously, it would be good if this
    wasn't necessary, in thatthe code could catch the problem and allocate
    extra file handles if that is the problem.

    ------------------------------------------------------------------------

    [2007-02-15 11:41:24] [email]pajoye@php.net[/email]

    "When adding files to an archive, (using successive
    ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. "

    Yes, we do it while finalizing the archive.

    " This can result in an out of memory error, "

    You will run out of file ID before running out of memory. It does not
    really use many memory, only the file names and file handlers.

    I suppose you are talking about the file handlers?

    "It would certainly require a rewrite of the ugly function
    zip_close()"

    What is ugly in this function? Or do you have a portable way to lock a
    file until the archive creation is done?

    I think you refer to the file handlers limitation. There is already a
    bug about it and I plan to add a special (less safe) mode. This mode
    will allow one to add only the paths without checks, errors will occur
    only when the archive is closed. But that's a feature addition not a
    bug fix.

    I close this bug (not a bug > bogus).

    Thanks for your report!

    ------------------------------------------------------------------------

    [2007-02-15 10:22:25] foster dot graeme at gmail dot com

    Description:
    ------------
    When adding files to an archive, (using successive
    ZipArchive::addFile() commands) the compression doesn't happen until
    the file is closed. This can result in an out of memory error, a
    temporary fix is to close the archive and then reopen it within the php
    code.
    An idea solution would be to compress the file when it is added,
    probably in function _zip_replace(), but I don't know what the
    implications of this would be. It would certainly require a rewrite of
    the ugly function zip_close().



    ------------------------------------------------------------------------


    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    pajoye@php.net Guest

  9. #8

    Default #40494 [Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    User updated by: foster dot graeme at gmail dot com
    Reported By: foster dot graeme at gmail dot com
    Status: Bogus
    Bug Type: Zip Related
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    Would it be possible to add a brief description of this situation to
    the documentation, for example the following could be added to the
    description of ZipArchive::addFile

    Description

    bool ZipArchive::addFile ( string filename [, string localname] )

    Adds a link to the ZIP archive from a given path. When the archive is
    closed the link is checked to ensure that the file still exists and
    will then be compressed and added to the archive. If a lot of files are
    being added then the number of file handles permitted by the OS may be
    exceeded, if that occurs then the status will be set to
    ZIPARCHIVE::ER_OPEN. This can be avoided by closing the archive before
    the limit is reached and then reopening the archive.

    for example:

    if ($zip->numfile % $limit == 0)
    {
    $zip->close();
    $zip->open($filename,ZIPARCHIVE::CREATE);
    }


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 14:35:34] [email]pajoye@php.net[/email]

    "I still think that it would be nice if there was some way for the
    system to manage this."

    It is in the TODO list. As I said three times already in this
    discussion. The solution is to add different modes:
    - commit at the end when the archive is close
    - immediate addition (will be much slower)

    And again, it is in my TODOs already. I cannot tell when they will be
    available (I do it on my free time).

    In the meantime a simple:

    if (($zip->numFiles % $yourlimit) == 0) {close; reopen;}

    will do it.



    "the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was
    opened."

    This solution does not work.The limit is arbitrary. There is no way to
    get an exact value (and I doubt php is the only running process).


    ------------------------------------------------------------------------

    [2007-02-15 14:02:51] foster dot graeme at gmail dot com

    Okay thanks for the explanation, I understand the problem a little
    better. I still think that it would be nice if there was some way for
    the system to manage this.

    I was thinking along the lines of a function to flush the files so that
    the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was opened.

    ------------------------------------------------------------------------

    [2007-02-15 13:23:36] [email]pajoye@php.net[/email]

    See:

    [url]http://pecl.php.net/bugs/bug.php?id=9443[/url]

    "it would be good if this wasn't necessary, in thatthe code could catch
    the problem and allocate extra file handles if that is the problem."

    This is not something I can control. The operating system defines it
    and there is no way for me to increase this value.

    I suggest you to close and reopen it every 1000 or so (or even 255 if
    you want to go on the safest way, ie old windows).

    Future releases will have a different mode, where the checks will done
    only when you close the archives.

    ------------------------------------------------------------------------

    [2007-02-15 13:14:57] foster dot graeme at gmail dot com

    Maybe I need to explain this problem a little more.

    I am trying to archive a folder on the server, at the moment it
    contains 5609 folders and 11,221 files. The script loops through the
    files adding them to the archive using the addFile() method. After the
    first 1002 files I get a ZIPARCHIVE::ER_OPEN. If I close the archive
    and the open it again I still have that error. However, if I close the
    archive and open it before I get that error then I can archive all
    11,221 files.

    Since closing the file and re-opening fixes the problem (so long as I
    do that before I get the error) Then may I suggest that closing an
    archive will clear the status. Obviously, it would be good if this
    wasn't necessary, in thatthe code could catch the problem and allocate
    extra file handles if that is the problem.

    ------------------------------------------------------------------------

    [2007-02-15 11:41:24] [email]pajoye@php.net[/email]

    "When adding files to an archive, (using successive
    ZipArchive::addFile()
    commands) the compression doesn't happen until the file is closed. "

    Yes, we do it while finalizing the archive.

    " This can result in an out of memory error, "

    You will run out of file ID before running out of memory. It does not
    really use many memory, only the file names and file handlers.

    I suppose you are talking about the file handlers?

    "It would certainly require a rewrite of the ugly function
    zip_close()"

    What is ugly in this function? Or do you have a portable way to lock a
    file until the archive creation is done?

    I think you refer to the file handlers limitation. There is already a
    bug about it and I plan to add a special (less safe) mode. This mode
    will allow one to add only the paths without checks, errors will occur
    only when the archive is closed. But that's a feature addition not a
    bug fix.

    I close this bug (not a bug > bogus).

    Thanks for your report!

    ------------------------------------------------------------------------

    The remainder of the comments for this report are too long. To view
    the rest of the comments, please view the bug report online at
    [url]http://bugs.php.net/40494[/url]

    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    foster dot graeme at gmail dot com Guest

  10. #9

    Default #40494 [Bgs]: Memory problem with ZipArchive::addFile()

    ID: 40494
    Updated by: [email]pajoye@php.net[/email]
    Reported By: foster dot graeme at gmail dot com
    Status: Bogus
    Bug Type: Zip Related
    Operating System: Linux
    PHP Version: 5.2.1
    Assigned To: pajoye
    New Comment:

    Yes, it can be added to the doc.

    However your explanation is not correct. The files are open and kept
    open until the archive is closed (that's why you reach the handlers
    limit). It is the only safe way to lock a file and be sure it exists
    when when we finalize the archive.


    Previous Comments:
    ------------------------------------------------------------------------

    [2007-02-15 15:15:13] foster dot graeme at gmail dot com

    Would it be possible to add a brief description of this situation to
    the documentation, for example the following could be added to the
    description of ZipArchive::addFile

    Description

    bool ZipArchive::addFile ( string filename [, string localname] )

    Adds a link to the ZIP archive from a given path. When the archive is
    closed the link is checked to ensure that the file still exists and
    will then be compressed and added to the archive. If a lot of files are
    being added then the number of file handles permitted by the OS may be
    exceeded, if that occurs then the status will be set to
    ZIPARCHIVE::ER_OPEN. This can be avoided by closing the archive before
    the limit is reached and then reopening the archive.

    for example:

    if ($zip->numfile % $limit == 0)
    {
    $zip->close();
    $zip->open($filename,ZIPARCHIVE::CREATE);
    }

    ------------------------------------------------------------------------

    [2007-02-15 14:35:34] [email]pajoye@php.net[/email]

    "I still think that it would be nice if there was some way for the
    system to manage this."

    It is in the TODO list. As I said three times already in this
    discussion. The solution is to add different modes:
    - commit at the end when the archive is close
    - immediate addition (will be much slower)

    And again, it is in my TODOs already. I cannot tell when they will be
    available (I do it on my free time).

    In the meantime a simple:

    if (($zip->numFiles % $yourlimit) == 0) {close; reopen;}

    will do it.



    "the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was
    opened."

    This solution does not work.The limit is arbitrary. There is no way to
    get an exact value (and I doubt php is the only running process).


    ------------------------------------------------------------------------

    [2007-02-15 14:02:51] foster dot graeme at gmail dot com

    Okay thanks for the explanation, I understand the problem a little
    better. I still think that it would be nice if there was some way for
    the system to manage this.

    I was thinking along the lines of a function to flush the files so that
    the archive can be partially built prior to the ulimit being reached.
    This could be set as 250, with the ability to overload it. Maybe this
    would only be triggered if a flag was set when the archive was opened.

    ------------------------------------------------------------------------

    [2007-02-15 13:23:36] [email]pajoye@php.net[/email]

    See:

    [url]http://pecl.php.net/bugs/bug.php?id=9443[/url]

    "it would be good if this wasn't necessary, in thatthe code could catch
    the problem and allocate extra file handles if that is the problem."

    This is not something I can control. The operating system defines it
    and there is no way for me to increase this value.

    I suggest you to close and reopen it every 1000 or so (or even 255 if
    you want to go on the safest way, ie old windows).

    Future releases will have a different mode, where the checks will done
    only when you close the archives.

    ------------------------------------------------------------------------

    [2007-02-15 13:14:57] foster dot graeme at gmail dot com

    Maybe I need to explain this problem a little more.

    I am trying to archive a folder on the server, at the moment it
    contains 5609 folders and 11,221 files. The script loops through the
    files adding them to the archive using the addFile() method. After the
    first 1002 files I get a ZIPARCHIVE::ER_OPEN. If I close the archive
    and the open it again I still have that error. However, if I close the
    archive and open it before I get that error then I can archive all
    11,221 files.

    Since closing the file and re-opening fixes the problem (so long as I
    do that before I get the error) Then may I suggest that closing an
    archive will clear the status. Obviously, it would be good if this
    wasn't necessary, in thatthe code could catch the problem and allocate
    extra file handles if that is the problem.

    ------------------------------------------------------------------------

    The remainder of the comments for this report are too long. To view
    the rest of the comments, please view the bug report online at
    [url]http://bugs.php.net/40494[/url]

    --
    Edit this bug report at [url]http://bugs.php.net/?id=40494&edit=1[/url]
    pajoye@php.net Guest

Posting Permissions

  • You may not post new threads
  • You may post replies
  • You may not post attachments
  • You may not edit your posts

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139