Professional Web Applications Themes

thread gurus please help... - Ruby

following is a _simulation_ of a rather complex gui i'm working on. the gui run a procedure which, itself, is communication with many other processes via Open3::popen3 (i _need_ the stderr)... anyhow, i'm having fits getting the gui to update and do all the io (select/read, etc). this code seems to work fine but i am very conerned that something might bite me later... if anyone would care to comment on the interactions between the io and threads and if this is a reccomended approach i'd love to hear it... remember this code might look a bit funny but it ...

  1. #1

    Default thread gurus please help...


    following is a _simulation_ of a rather complex gui i'm working on. the gui
    run a procedure which, itself, is communication with many other processes via
    Open3::popen3 (i _need_ the stderr)... anyhow, i'm having fits getting the gui
    to update and do all the io (select/read, etc). this code seems to work fine
    but i am very conerned that something might bite me later... if anyone would
    care to comment on the interactions between the io and threads and if this is
    a reccomended approach i'd love to hear it... remember this code might look a
    bit funny but it demonstrative of what my code actually does. thanks in
    advance for any help/advice

    ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====
    require 'tk'
    require 'open3'
    require 'io/wait'
    require 'io/nonblock'
    require 'thread'
    $VERBOSE=nil
    STDOUT.sync = STDERR.sync = true
    Thread.abort_on_exception = true

    threads = []

    #setup gui
    root = TkRoot.new
    label_a=TkLabel.new root, :text=>'label_a'
    label_b=TkLabel.new root, :text=>'label_b'
    label_a.pack
    label_b.pack

    # start the gui running
    threads <<
    Thread.new{ Tk.mainloop }

    # simulation of my processing loop - which is communcating with another
    # process and updating a widget with it's output
    def update label, program
    command="ruby -e '#{ program }'"
    i,o,e = Open3::popen3 command
    i.close

    # we run in a thread so entire process is not blocked on io
    # we DO need to wait for it to complete though, because in the real code i
    # depend on values which are returned...
    Thread.new do
    o_done = false
    e_done = false
    loop do
    rios, = select [o, e], nil, nil
    rios.map do |rio|
    break if o_done and e_done
    if rio.eof?
    rio == o ? o_done = true : e_done = true
    next
    end
    # i like this better, but it hangs
    # putting a critical section results in deadlock
    # a semaphore is too slow...
    #rio.nonblock{ text = rio.read }

    # less elegant but works...
    text = ''
    sleep(0.42) and Thread.pass until rio.ready?
    text << rio.getc while rio.ready?
    label.configure :text=>text
    end
    end
    end.join
    end


    # start background processing which will update the labels
    threads <<
    Thread.new{
    update label_a, '4.times{p Time.now; sleep 1}'
    }
    threads <<
    Thread.new{
    update label_b, '4.times{|i| p "foobar_#{ i }"; sleep 1}'
    }

    # wait for everyone to finish
    #threads.map{|t| t.join}
    sleep
    ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====

    -a
    --
    ================================================== =============================
    | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    | PHONE :: 303.497.6469
    | ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
    | URL :: http://www.ngdc.noaa.gov/stp/
    | TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
    ================================================== =============================

    Ara.T.Howard Guest

  2. #2

    Default Re: thread gurus please help...

    On Wed, 25 Feb 2004, Ara.T.Howard wrote:

    forgot to mention that i have $VERBOSE=nil so i don't see the

    "...fork terminates thread..."

    err msg. i assume that, in this case (forking done in open3) it is OK if that
    thread is terminated in the child - it obviously seem to continue running in
    the parent...

    -a
     

    --
    ================================================== =============================
    | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    | PHONE :: 303.497.6469
    | ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
    | URL :: http://www.ngdc.noaa.gov/stp/
    | TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
    ================================================== =============================

    Ara.T.Howard Guest

  3. #3

    Default Re: thread gurus please help...

    Hi,

    At Thu, 26 Feb 2004 09:04:47 +0900,
    Ara.T.Howard wrote in [ruby-talk:93709]: 

    In [ruby-dev:22861], Akira Tanaka supposed use of sysread
    instead of non-blocking mode.

    # As for blocking on non-blocking IO, I'll investigate later.

    --
    Nobu Nakada


    nobu.nokada@softhome.net Guest

  4. #4

    Default Re: thread gurus please help...

    Ara.T.Howard wrote: 

    I've never seen forking have an adverse effect on threads in the parent.
    But there is a case where terminating all but one thread, which is what
    happens in the child, has an adverse effect.

    require 'thread'

    m = Mutex.new

    Thread.new { m.synchronize { sleep 1 } }

    fork do
    m.synchronize { puts "Didn't get here." }
    end

    m.synchronize { puts "Got here." }

    Process.wait

    This outputs the warning you saw and "Got here.".

    I've gotten around this by explicitly removing the dead threads that
    hold a mutex or are waiting for it. (Code available, if anyone's
    interested.)

    But that doesn't look like your problem, since the child is firing up a
    whole new ruby.


    Joel Guest

  5. #5

    Default Re: thread gurus please help...

    On Thu, 26 Feb 2004, Joel VanderWerf wrote:
     
    >
    > I've never seen forking have an adverse effect on threads in the parent.
    > But there is a case where terminating all but one thread, which is what
    > happens in the child, has an adverse effect.
    >
    > require 'thread'
    >
    > m = Mutex.new
    >
    > Thread.new { m.synchronize { sleep 1 } }
    >
    > fork do
    > m.synchronize { puts "Didn't get here." }
    > end
    >
    > m.synchronize { puts "Got here." }
    >
    > Process.wait
    >
    > This outputs the warning you saw and "Got here.".[/ref]

    wow. that is really bizarre. if i understand correctly all but the 'main'
    thread is killed so the thread that is sleeping with the lock sort vaporizes
    and leaves the mutex in an inconsistent (or locked at least) state?
     

    you are braver than i ;-)
     

    sort of, the actual fork in my code are many satelite image processing steps
    and it is done from within popen3 prior to an exec so i should be _ok_.

    thanks for the reply - i'm feeling alot more confident now.

    -a
    --
    ================================================== =============================
    | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    | PHONE :: 303.497.6469
    | ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
    | URL :: http://www.ngdc.noaa.gov/stp/
    | TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
    ================================================== =============================

    Ara.T.Howard Guest

  6. #6

    Default Re: thread gurus please help...

    On Thu, 26 Feb 2004 net wrote:
     
    >[/ref]
     

    wow. tried babelfish but i'm a bit lost on that one ;-)

    what is a 'bean jam ball'? sounds like something good...

    i tried replacing with sysread but am now failing with

    sysread for buffered IO (IOError)

    ....

    for now i seem to be having luck with (this is after a select because i am
    multiplexing stdout and stderr from another process)

    text = ''
    sleep(0.042) and Thread.pass until rio.ready?
    text << rio.getc while rio.ready?
    label.configure :text=>text

    but reading only a character at a time when the io object is ready i hope to
    never block. this is what i had hoped to accomplish by

    io.nonblock{ io.read }

    but this does seem to work fine - i don't really know how to say 'read
    everything that is available but do not block' from within a multi-threaded
    application...
     

    thanks. i'd be quite interested to hear what you find.

    -a
    --
    ================================================== =============================
    | EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
    | PHONE :: 303.497.6469
    | ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
    | URL :: http://www.ngdc.noaa.gov/stp/
    | TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
    ================================================== =============================

    Ara.T.Howard Guest

  7. #7

    Default Re: thread gurus please help...

    In article <ngdc.noaa.gov>,
    "Ara.T.Howard" <ngdc.noaa.gov> writes:
     

    You use eof? or other stdio related methods.
    Fortunately, you don't need such methods in this case.
     

    You don't need nonblock because sysread doesn't block if some data are
    available. Note that IO.select blocks until some data are available.

    For example, your [ruby-talk:93618] can be modified as follows.

    require 'open3'
    STDOUT.sync = STDERR.sync = true

    command = <<'End'
    ruby -e "
    STDOUT.sync = true
    10.times {
    STDOUT.puts Time.now; sleep 1
    STDERR.puts Time.now; sleep 1
    }"
    End
    i,o,e = Open3::popen3 command
    i.close

    inputs = [o, e]
    p inputs
    t = Thread.new do
    until inputs.empty?
    rios, = select inputs
    rios.each do |rio|
    p 'start...'
    begin
    text = rio.sysread(8192)
    rescue EOFError
    inputs.delete rio
    next
    end
    p [rio, text]
    p 'finish'
    end
    end
    end

    t.join

    Also, IO.select can be removed with two threads.

    require 'open3'
    STDOUT.sync = STDERR.sync = true

    command = <<'End'
    ruby -e "
    STDOUT.sync = true
    10.times {
    STDOUT.puts Time.now; sleep 1
    STDERR.puts Time.now; sleep 1
    }"
    End
    i,o,e = Open3::popen3 command
    i.close

    t1 = Thread.new do
    loop {
    begin
    text = o.sysread(8192)
    rescue EOFError
    break
    end
    puts "STDOUT: #{text.inspect}"
    }
    end
    t2 = Thread.new do
    loop {
    begin
    text = e.sysread(8192)
    rescue EOFError
    break
    end
    puts "STDERR: #{text.inspect}"
    }
    end

    t1.join
    t2.join

    Note that sysread may block in this case when no data is available.
    But I think it is not a problem because you cannot do any computation
    on the no data. I think it is even good behaviour because it avoids
    busy loop.
    --
    Tanaka Akira


    Tanaka Guest

  8. #8

    Default Re: thread gurus please help...

    Hi,

    At Thu, 26 Feb 2004 13:54:48 +0900,
    Ara.T.Howard wrote in [ruby-talk:93721]: 

    'Bean jam' is sweet paste made boiling red beans and sugar. It
    looks black in common, but there is white one and others. And
    Japanese are in the habit of eating rice balls with bean jam in
    each equinox.
     

    See [ruby-talk:93722] about sysread.

    Anyway, I feel non-blocking IO should never block in this case,
    simpler code which can reproduce your issue.

    require 'io/nonblock'

    unless rio = IO.popen("-")
    loop do
    p Time.now
    sleep 1
    end
    end

    Thread.new do
    until rio.eof?
    #Thread.critical = true # blocks w/o this
    text = rio.nonblock{ rio.read }
    #Thread.critical = false # blocks w/o this
    print text
    end
    end.join


    Index: io.c
    ================================================== =================
    RCS file: /cvs/ruby/src/ruby/io.c,v
    retrieving revision 1.261
    diff -u -2 -p -d -r1.261 io.c
    --- io.c 25 Feb 2004 12:17:39 -0000 1.261
    +++ io.c 26 Feb 2004 06:38:59 -0000
    -889,4 +889,10 rb_io_to_io(io)
    }

    +#ifndef O_NONBLOCK
    +# ifdef O_NDELAY
    +# define O_NONBLOCK O_NDELAY
    +# endif
    +#endif
    +
    /* reading functions */
    long
    -903,4 +909,8 rb_io_fread(ptr, len, f)
    long i = READ_DATA_PENDING_COUNT(f);
    if (i <= 0) {
    +#if defined(HAVE_FCNTL) && defined(F_GETFL) && defined(O_NONBLOCK)
    + i = fcntl(fileno(f), F_GETFL);
    + if (i == -1 || (i & O_NONBLOCK)) break;
    +#endif
    rb_thread_wait_fd(fileno(f));
    i = READ_DATA_PENDING_COUNT(f);
    -5508,10 +5518,6 Init_IO()
    rb_file_const("CREAT", INT2FIX(O_CREAT));
    rb_file_const("EXCL", INT2FIX(O_EXCL));
    -#if defined(O_NDELAY) || defined(O_NONBLOCK)
    -# ifdef O_NONBLOCK
    +#ifdef O_NONBLOCK
    rb_file_const("NONBLOCK", INT2FIX(O_NONBLOCK));
    -# else
    - rb_file_const("NONBLOCK", INT2FIX(O_NDELAY));
    -# endif
    #endif
    rb_file_const("TRUNC", INT2FIX(O_TRUNC));


    --
    Nobu Nakada


    nobu.nokada@softhome.net Guest

Similar Threads

  1. Newbie:Using ASP.NET thread pool thread to dispatch TCP data, etc.
    By Navin Mishra in forum ASP.NET Web Services
    Replies: 0
    Last Post: March 22nd, 06:45 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139