From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Nala Ginrut Newsgroups: gmane.lisp.guile.devel Subject: Re: [PATCH] Add current-suspendable-io-status parameter Date: Wed, 15 May 2019 17:31:31 +0800 Message-ID: References: <87pnomjcs9.fsf@netris.org> <87lfzaj6ya.fsf@netris.org> <878sv8kcqb.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="125667"; mail-complaints-to="usenet@blaine.gmane.org" Cc: guile-devel To: Mark H Weaver Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Wed May 15 11:35:51 2019 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hQqKA-000WUU-V0 for guile-devel@m.gmane.org; Wed, 15 May 2019 11:35:47 +0200 Original-Received: from localhost ([127.0.0.1]:34035 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hQqK9-000463-Td for guile-devel@m.gmane.org; Wed, 15 May 2019 05:35:45 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:53610) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hQqK3-00042r-EP for guile-devel@gnu.org; Wed, 15 May 2019 05:35:40 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hQqGJ-0003Ev-J1 for guile-devel@gnu.org; Wed, 15 May 2019 05:31:48 -0400 Original-Received: from mail-yb1-xb44.google.com ([2607:f8b0:4864:20::b44]:37415) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hQqGJ-0003Au-Cu for guile-devel@gnu.org; Wed, 15 May 2019 05:31:47 -0400 Original-Received: by mail-yb1-xb44.google.com with SMTP id p134so708352ybc.4 for ; Wed, 15 May 2019 02:31:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=T2crYpd8d4obxUh9qPfvQ9XKOYW8Tf5KodWpC80AQT8=; b=mHf6QUYLBy9LMm7CMyVOEAoTvLkc8dX121GxtX8i//GTM8x816RwvVRRbcOOhUo5wG gEC7YnUhfY70ex2rJ9jA1NO0G+q3gLl+O755MZJcjMB2Ks//VthkalKwZy1ov+Bo2OmU RnIlRto7YzI1rcxRsGdQ/Ms92IyYyRM3pHhZfwWYUjQ6FpTI02q+IjLGz9dFabYbLt4l hoDAuwORH48V/clhcjIMFlR7TCB0b/zgYIgMpx++egyEfuaIqcsYUBAuCV8PPH4NvzeW gZnGqoSgh8bQZc8ROmeG+qvWn++3MRJa33o06M30DK3eAlzhwGg91aI49FFAm45+W7TJ Yg9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=T2crYpd8d4obxUh9qPfvQ9XKOYW8Tf5KodWpC80AQT8=; b=YecSu/ZT2s4U/1UhzqXM4iw/Umd9SLJjqU/PXjYEgQhtc+LYO52X2bvJnVGz0zCDjI M4Yh1GRmsOtU/dIJGVA4ctB30kFJksNR9RxojWaVkzcs3uJo8NiefIvjUrGhE1hKn4Y5 GPJaKh/s6aEnartlFbdYCTcGjEeOE2lI6D73egRuZovFt7Q8OODsppNXPyQxUK+fmVBR ocIJckvAV/X2xIqASUedhC8JttNn9xYcPNK9AJAx0OEqo6+l+ukRJjnMtTGKzVBwMWXB ErsyJ8i76ep7FrctvlruQVdRwLuBRlzHgtHI+MlmxswzeOXWkEMLq5Eh0HoRz2WPDqbF QixA== X-Gm-Message-State: APjAAAWNX/meRF7K4u37cqQj87t3VjTIkfBmQwNnDa+fKNaJfnPtUbpj G8PCptbOD6d4ShWDMiOKgtvwnEBsSVVSfV9g+ow= X-Google-Smtp-Source: APXvYqyOZItQl1wIwzFlgD//zdvc/n1xh6e4DBLVyMpXQd/GG3d+laOl0+UY51Zuvil0xI8UL8Bb3RszHeZb67W1i7Y= X-Received: by 2002:a25:c048:: with SMTP id c69mr12107197ybf.347.1557912704037; Wed, 15 May 2019 02:31:44 -0700 (PDT) In-Reply-To: <878sv8kcqb.fsf@netris.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::b44 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.org gmane.lisp.guile.devel:19915 Archived-At: hi Mark! I think your approach works for the cases which use put-bytevector* directly. Are you assuming the server-core only handles suspendable-port with put-bytevector in http-read or http-write? If so, then your approach is not enough. When the users enabled suspendable-port, every I/O operation would be non-blocking, and be managed by overridden I/O methods rebind by suspendable-port . That is to say, every I/O operation should have the ability to return the current I/O status for the scheduler. That's why I put it in read-bytes and write-bytes. If we add a new put-bytevector* function, and if it is the only function which returns current I/O status, then we can't tell the scheduler the progress when users are trying to access database for a bigger data, or access a file on the disk, or any other potential I/O operation registered in HTTP handler by users. I think it's better to patch read-bytes and write-bytes for any I/O operations, otherwise, we have to patch all the overridden functions in suspendable-ports to make sure all I/O operations are handled correctly by suspendable-port. Hope I understand your idea correctly. Best regards. On Wed, May 15, 2019 at 4:23 AM Mark H Weaver wrote: > > Hi Nala, > > Nala Ginrut writes: > > > On Tue, May 14, 2019 at 7:01 AM Mark H Weaver wrote: > >> I guess what you want is the ability to see incremental reports on the > >> progress of your large I/O operations. Is that right? If we are going > >> to add an API for this, it needs to be reliable, and always give reports > >> in terms of the high-level requests that the user gave. > > > > Yes, that's exactly what I want. We need to get the progress of I/O > > operation when it's blocking > > so that we can compute a fair priority for the tasks. > > > >> My preferred approach would be something like this: we could add a > >> 'put-bytevector-some' I/O primitive which writes some bytes from a > >> bytevector, blocking only as needed to write at least one byte. It > >> would return the number of bytes written. This can be used to implement > >> an efficient variant of 'put-bytevector' that gives you access to the > >> real-time progress information. > > > > I'm not sure if put-bytevector-some does the work, I'll list my concerns: > > > > 1. All I/O will be managed by Guile when we enabled suspendable-port. > > That is to say, from the users side, users never know their I/O > > operations are blocking or not. It's transparent to users. > > Guile will guarantee the I/O operations to be finished by managing all > > the blocking I/O mechanisms. > > Users can only interact with the task with read or write waiter, which > > are registered by users themselves. > > In this scenario, users are out of control of I/O operations. And they > > have no way to get the progress of I/O, since there's > > no way to pass this status to the waiter function except for > > parameters in my patch. > > > > 2. suspendable-port module has already provided a bunch of overridden > > bytevector-* functions. > > However, they're hidden from users. I think it's good since the > > purpose of suspendable-port is to abstract all these details from > > users. Users only consider the read-waiter and write-waiter for scheduling. > > If we provide the low-level bytevector functions to users to let them > > do the non-blocking I/O by themselves, just like most C framework > > does. Then Guile suspendable-port will lose a critical feature, > > although users can still implement asynchronous non-blocking I/O by > > themselves with a non-managed approach. Say, do the I/O, check result > > by themselves, and do the scheduling. > > Personally, I'm fine with this way, since I'm familiar with both ways. > > But managed I/O of suspendable-port is a good selling point for many > > inexperienced server-side developers, they can use it in Scheme just > > like IOCP or AIO. > > > > Of course, I may misunderstand your mind. > > Could you elaborate more about your approach? > > Here's what I had in mind. I'm not suggesting that you use asynchronous > non-blocking I/O. 'put-bytevector-some' would be like 'put-bytevector' > except that it can write less than you asked it to. Ideally it would > block only as needed to write at least one byte. It would be analogous > to POSIX write(2). > > Here's an untested draft implementation of 'put-bytevector*' which sets > the parameter 'current-io-status' to provide the information you wanted, > except that here the information will reliably refer to your own buffer. > The primitive it is built on 'put-bytevector-some' does not yet exist, > but we could add it. Note that 'put-bytevector-some' would still block, > and it would be suspendable. > > --8<---------------cut here---------------start------------->8--- > (define* (put-bytevector* port bv > #:optional > (start 0) > (count (bytevector-length bv))) > (let loop ((start start) (count count)) > (unless (zero? count) > (let ((written (parameterize ((current-io-status (cons start count))) > (put-bytevector-some port bv start count)))) > (loop (+ start written) (- count written)))))) > --8<---------------cut here---------------end--------------->8--- > > We can already do something analogous on the reader side, using either > 'get-bytevector-some' or the already-implemented-but-not-yet-pushed > 'get-bytevector-some!'. > > What do you think? > > Mark