How to unload a module from memory?

[Eric J. Roode]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have a long-running process, and I would like to detect when one (or
more) of its dependent modules change, so that it can reload that module.

I go through %INC and note the last-modification timestamp of each file,
and periodically check those files to see if the lastmod time has changed.
That's the easy part. The hard part is unloading the module from memory.

My first thought was simply to delete $INC{$module} and then eval "use
$module" (yes, I know that the format of the keys to %INC is different than
the argument to 'use' -- work with me here; I'm just illustrating. The
actual program does convert back and forth between formats as necessary).

That seemed to work, but when I do the "use" again, I get lots of warnings
due to all the subroutines being re-defined as the subordinate modules get
re-used and recompiled.

So I thought I would remove all subordinate subroutines and other data
structures by looping over the packages and doing %{$module.'::'}=(); for
each one. That worked *most* of the time, but for one module, the script
died with a "that type of scalar cannot be promoted" error. Which
befuddled me, since I couldn't see any reason why that module would fail
and other modules would work.

So I'm at a loss. Is there a safe and sane way to unload and reload a
module when it changes, without exiting and restarting the process?

Thanks,
- --
Eric
$_ = reverse sort qw p ekca lre Js reh ts
p, $/.r, map $_.$", qw e p h tona e; print

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPvuQVmPeouIeTNHoEQKCUgCePGmhwDRHKPqFsi/R7BMr4WKzuH8AnRhK
pUkszNvEyHRB3uJWQP3LHCLH
=p0aX
-----END PGP SIGNATURE-----

[Martien Verbruggen]

On Thu, 26 Jun 2003 19:32:00 -0500,
Eric J. Roode <REMOVEsdnCAPS@comcast.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have a long-running process, and I would like to detect when one (or
> more) of its dependent modules change, so that it can reload that module.
>
> I go through %INC and note the last-modification timestamp of each file,
> and periodically check those files to see if the lastmod time has changed.
> That's the easy part. The hard part is unloading the module from memory.
>
> My first thought was simply to delete $INC{$module} and then eval "use
> $module" (yes, I know that the format of the keys to %INC is different than
> the argument to 'use' -- work with me here; I'm just illustrating. The
> actual program does convert back and forth between formats as necessary).
>
> That seemed to work, but when I do the "use" again, I get lots of warnings
> due to all the subroutines being re-defined as the subordinate modules get
> re-used and recompiled.
>
> So I thought I would remove all subordinate subroutines and other data
> structures by looping over the packages and doing %{$module.'::'}=(); for
> each one. That worked *most* of the time, but for one module, the script
> died with a "that type of scalar cannot be promoted" error. Which
> befuddled me, since I couldn't see any reason why that module would fail
> and other modules would work.
>
> So I'm at a loss. Is there a safe and sane way to unload and reload a
> module when it changes, without exiting and restarting the process?

[I'm not sure how much of the following is applicable to your
particular process, but here are my thoughts on this anyway.]

While re-evaluating modules and reloading them can work for many
modules (think for example of the CPAN module), it is not necessarily
possible, or a good idea, for all modules or for all module changes.
It is also not at all trivial without the cooperation of the module
internally.

Take for example an OO module, i.e. a class. Your program starts, and
instantiates a few objects of that class. At that point, you have
references to blessed data structures, say, for example, hashes. The
module author in the meantime has decided that the implementation of
her class is too slow for many applications, and has changed the
internal representation to a pseudo-hash, or an array. You reload the
module, and suddenly the data structure makes no sense to the
internals of the module.

While this is a drastic example, even a reorganisation of deeper data
structures or renaming of some of the internally used keys of the hash
structure could have this result. If the module author suddenly
decides that a whole bunch of work that is, today, done in a module
will in the future be done in a sub-module, and an object of that
class is now kept in the original, you are in trouble. If a module
author decides that instead of using the internal attribute
$self->{_color}, it now becomes $self->{_colour}, you're in trouble.

If the module author has changed the public interface of the module,
you're in even more trouble, since methods you rely on are suddenly no
longer available, or have a different calling signature.

For the above reasons, I'd be very unwilling to ever consider
reloading modules in a running process, unless I could be absolutely
certain that it was possible. Instead, I'd upgrade the modules outside
of the production tree, check that my process still works, and then
I'd re-execute my long-running process to load the new modules, and
rebuild the data structures. Much less work.

In the case that you need to have persistence of data, I'd use files
or a database for that, because of the danger of internal data
structure changes, and just save the data that I need to be persistent.

If you i(also) need to keep the service available (a TCP port or some
other mechanism), I'd probably come up with some mechanism where a
thin parent process provides the service, and marshalls off the work
to child processes, or independent processes via some sort of IPC
mechanism. While the children or other processes are not available,
the parent process just waits.


For example, your service process could fire off a child that writes
to a FIFO, which is read by the process that you need to reload. If
the process isn't reading, the child will block, but the parent still
provides the service. As soon as the reader is back, and has processed
the request, things continue as normal, without the service ever
having been down.

Another possibility is using files in conjunction with SYSV IPC
queues. Your service process writes information in a file, then puts
some message on the queue. Your long-running process gets the message
off the queue, reads the file, processes it, writes a result, and puts
a message on the queue, signalling to the service process that it
can read the response. Again, when the processor is not running, the
service is not interrupted. It will just wait until it gets a response
via the queue.

Alternatively, your parent loads no modules. For each request, it
spawns a child that does load the modules and processes the requests.
As long as you know that it will all work with the new modules, you
can put them in place at any time. The problem with this approach is
that if a child starts while you're halfway through installing your
modules, it could break. The other mechanisms let you control better
when you restart your processor.


Which of these is most applicable is up to the application. I've used
all of those approaches in code at some time or another.

Martien
--
|
Martien Verbruggen | The world is complex; sendmail.cf reflects
Trading Post Australia | this.
|