From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Daniel Dinnyes Newsgroups: gmane.lisp.guile.devel Subject: Hygienic rewrite of (ice-9 expect) Date: Sat, 15 Apr 2023 16:10:28 +0100 Message-ID: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000d8d69a05f9615da6" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="39952"; mail-complaints-to="usenet@ciao.gmane.io" To: guile-devel@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Sat Apr 15 22:40:58 2023 Return-path: Envelope-to: guile-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pnmhd-000A9V-Ln for guile-devel@m.gmane-mx.org; Sat, 15 Apr 2023 22:40:57 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pnmhN-0002Cu-RH; Sat, 15 Apr 2023 16:40:41 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pnhYV-0003EQ-0H for guile-devel@gnu.org; Sat, 15 Apr 2023 11:11:11 -0400 Original-Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pnhYR-0001bk-HP for guile-devel@gnu.org; Sat, 15 Apr 2023 11:11:10 -0400 Original-Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-3f0aabd1040so11196075e9.1 for ; Sat, 15 Apr 2023 08:11:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681571465; x=1684163465; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=7kObNLRDdEMNSgAMbwAHBGAw3uvtSo9UdLZBFDSAroo=; b=JBZWKJo7Dw3EhQR+uUjoyK0omhiIOsOEOjBZk/6vX4cT8tFCXIxKr5kd20JyHTPMLW cvyU4hDRmzt7kJl5LfcjJ21ugPrE0ofV93hGqCmIjE95b6C2grp9Eidnkh9WhBgs/2cC 5bUfeyVML33wpPpoQkNECbRfpg/WxqK3l4vIP9nZpNRdQ6nINybH+Alh6uy+A68Q1J5G q8QQA6GHo9L7s1jm4la8wfX569AWb8dTWNt/zDdvfWQCt0Ck7pLWdmd6jniOEywsqUUb qeGkPPeJT37hndHhW5CtB3zlEXBcoqTBtxVvfolCNyuHbh9KrlQvRDP0CCZ2fnDz3vZw uD9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681571465; x=1684163465; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=7kObNLRDdEMNSgAMbwAHBGAw3uvtSo9UdLZBFDSAroo=; b=dVj55sYcJybd6aNhuENze+V1bFIremP9bYTD37ckRggH6em0M+yaY6fv0sAFDBemWv LncJ6IbVHGdVbZsrwvE8FFCVHK6XDOD+gznMQAOhoTaA5aq3sAQEGSHStE9zz51a4ruN m403r/qhVi0xNnInwGTuEodhpU5GiLERVyVHS4Jn5CWyAeMH9QuLoTM7lIC4QEdktkq4 MPUla/KhaziksvTXX3p1SkNPsdRzA21VNE8tsmdD1kOerFgKk1mzB/UP+90DVU7UIgxh Vx543mT1/Hs37c53tGqDl/h35UXcneogtGAjZTJRl3m45AcImo66AUyrofrVOMKZ3JSl 1g9g== X-Gm-Message-State: AAQBX9fQlNcWV7xncvGZD+OsM0prpMZFa7ANnh7hCKEKeUkkMPB6GNAu dPulXDv3Yi6+vu0c/p1+7PBL5uiwZr1ciG1v7SyzLRijIO8ptQ== X-Google-Smtp-Source: AKy350YuXZjSp30ZtggDT3CpdmTf7Cs9Z5tOOuhe/jLZG21BbWBmxeBKQdB8N3p2snQbBP5eJzeYu3mr+C5yt0cKCyY= X-Received: by 2002:a5d:40cd:0:b0:2ef:4c83:b78e with SMTP id b13-20020a5d40cd000000b002ef4c83b78emr1726684wrq.4.1681571465187; Sat, 15 Apr 2023 08:11:05 -0700 (PDT) Received-SPF: pass client-ip=2a00:1450:4864:20::32b; envelope-from=dinnyesd@gmail.com; helo=mail-wm1-x32b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sat, 15 Apr 2023 16:40:40 -0400 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.devel:21804 Archived-At: --000000000000d8d69a05f9615da6 Content-Type: text/plain; charset="UTF-8" Hi everyone, I am new to this mailing list, and writing this because I have encountered some issues with ice-9 expect : The procedural expect macro has a serious problem: This syntax expects a procedure as the test expression, with 2 arguments (actual content read, and boolean EOF flag). The primary problem is that when the test expression is not a identifier bound to such a procedure, but instead an expression which creates such a procedure (aka "factory"), then the expression will be evaluated anew for each character read from expect-port. This is a serious problem: when constructing that procedure has significant computational costs, or side effects (eg. opening a port to a new log file), as it can be seen in my code here . (This is a branch of some of my KVM-based E2E testing code, which uses the original expect macro implementation, just to demonstrate the problem). I had written a workaround for this problem, by binding the evaluation of the test expression outside the expansion of the expect macro (as it can be seen here ). The problem with this was, that it doesn't support the full capabilities of the original expect macro, and debilitates it by only allowing for a single conditional branch. Also, I haven't verified, but I suspicious it would possibly even break the behaviour for the => syntax of the expect-strings macro. To implement a more complete version of this (and also as an exercise to understand hygienic macros), I've attempted to rewrite my workaround using syntax-case. I've ran into some issues, and my implementation didn't expand as I expected. After some discussion on IRC, we've concluded that this was because the current (ice-9 expect) implementation is written using unhygienic defmacro syntax, which doesn't work well together with the hygienic syntax-case. It was suggested I should rather rewrite expect completely to be hygienic instead. I've eventually completed the rewrite using syntax-case here , and would be happy to contribute it back to the ice-9 module! I would point out the following about this implementation: 1. It works!!! I had some quite extensive E2E testing code written before here , which I think really stretches the possibilities with (ice-9 expect). Switching to the new implementation was simple, and works flawlessly! 2. It is backwards compatible! I've gone extra lengths to make sure the code would be a backwards compatible, a drop-in replacement. 3. Introduced new feature for the procedural expect macro, such that the second argument of the procedure gets passed the currently read char, or #f if it was an EOF character. There is multiple reasons this is superior to passing just a boolean eof? flag argument: 1. the same logic can be implemented as with the eof? boolean, just by negating the condition 2. passing the currently read char avoids having to do the inefficient operation to retrieve the currently read char by cutting off the last character from the end of the content. Currently it is necessary to do this, if the logic wants something additional to be done with the current char (eg. as it can be seen in my code here ) 3. one such case of additional use is making the expect-char-proc property obsolete, as the predicate procedure is now able to execute such logic too, besides other more advanced logging scenarios (like it can be seen in my code here : here besides logging everything to STDOUT, and also to a main log-file, I also log for each expect macro call the contents it tries to compare against into a separately named minor log-file (useful for debugging), and also to ensure that we actually do not write to STDOUT when using expect on multiple parallel processes) 4. To ensure that everything stays backwards compatible, this new feature is only enabled if the new expect-pass-char? binding is set to #t in the lexical context (like here ). Otherwise, by default this feature is disabled, and boolean eof? flag argument is passed just like before. 4. There is a better support for using default value/expressions for the various parameters the macros depend on in their lexical contexts. The old macro had to export default values defined in its module, so as to ensure the defaults are available. This either clutters the module's global context where (ice-9 expect) is used, or the defaults are not available when using something like ((ice-9 expect) #:select (expect expect-string)). The new defaults are calculated based on the lexical context where the macros are used, and expands into the correct default values when they are not defined. For example when expect-port is not defined in the lexical context, then the default value used is going to be the result of the (current-input-port) expression. I would like someone to review whether I've implemented this correctly, as it seems the current-input-port identifier default for expect-port gets captured in the expansion. Not sure why that happens. 5. The expect-strings macro supports the => syntax, which passes the output(s) of the test predicate to a procedure, instead of just evaluating a body. This is fully supported, but additionally, the => syntax now also works with the procedural expect macro too! I haven't verified if the original implementation did this too, but at least it wasn't described in the documentation! 6. As an added bonus, I have added a simplistic implementation for an interact macro. It uses threads to read from STDIO, and expects expect-port to be an input-output-port, so that it can output the lines read from STDIO into the same port. Not too advanced, as it doesn't support a full terminal interface, with autocomplete, etc... but works well enough for my use-case to interact with a KVM process via serial port. 7. mnieper said that I am not using syntax-case idiomatically, and my style is more in the form of a syntax-rules-based state-machine. I am quite happy with how it is currently, but would be open to be convinced if there is a superior way of expressing all this. 8. I currently have various tests sprinkled around my implementation, to manually demonstrate expansions, and executions. Also, I have added some automated unit tests for the bind-locally helper procedure, which I used for checking whether parameters are defined in the lexical scope. 9. I am not happy with how I am managing my project structure, and how my modules can reference each other (having to call add-to-load-path everywhere). Also, I think the way I am using srfi-64 tests is a bit bothersome, as they always get executed when my modules are loaded, and I have to globally disable logging to files. I would like some recommendations how to do this better! If the community likes this implementation, I would be happy to apply/rebase my changes onto the official repo, and then we can go from there! Best regards, Daniel Dinnyes --000000000000d8d69a05f9615da6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi everyone,

I am new to thi= s mailing list, and writing this because I have encountered some issues wit= h ice-9 expect:

Th= e procedural expect macro has a serious problem:

<= div>This syntax expects a procedure as the test expression, with 2 argument= s (actual content read, and boolean EOF flag). The primary problem is that = when the test expression is not a identifier bound to such a procedure, but= instead an expression which creates such a procedure (aka "factory&qu= ot;), then the expression will be evaluated anew for each character read fr= om expect-port. This is a serious problem: when constructing that procedure= has significant computational costs, or side effects (eg. opening a port t= o a new log file), as it can be seen in my code here. (This is a branch of some of my KVM-based E2E testing code, whi= ch uses the original expect macro implementation, just to demonstrate the p= roblem).

I had written a workaround for this probl= em, by binding the evaluation of the test expression outside the expansion = of the expect macro (as it can be seen here). T= he problem with this was, that it doesn't support the full capabilities= of the original expect macro, and debilitates it by only allowing for a si= ngle conditional branch. Also, I haven't verified, but I suspicious it = would possibly even break the behaviour for the =3D> syntax of the expect-strings macro.

To implement a= more complete version of this (and also as an exercise to understand hygie= nic macros), I've attempted to rewrite my workaround using syntax-case. I've ran into some issue= s, and my implementation didn't expand as I expected. After some discus= sion on IRC, we've concluded that this was because the curre= nt (ice-9 expect) implementati= on is written using unhygienic defmac= ro syntax, which doesn't work well together with the hygienic syntax-case. It was suggested I = should rather rewrite expect completely to be hygienic instead.
<= br>
I've eventually completed the rewrite using syntax-case <= a href=3D"https://github.com/dadinn/common/blob/feature/expect/expect.scm" = target=3D"_blank">here, and would be happy to contribute it back to the= ice-9 module!
I would point out the following about this impleme= ntation:
  1. It works!!! I had some quite extensive E2E testi= ng code written before here, which I think really stretches the poss= ibilities with (ice-9 expect). Switching to the new implementation was simp= le, and works flawlessly!
  2. It is backwards compatible! I've = gone extra lengths to make sure the code would be a backwards compatible, a= drop-in replacement.
  3. Introduced new feature for the procedural= expect macro, such that the s= econd argument of the procedure gets passed the currently read char, or #f = if it was an EOF character. There is multiple reasons this is superior to p= assing just a boolean eof? flag argument:
      1. the same logi= c can be implemented as with the eof? boolean, just by negating the conditi= on
      2. passing the currently read char avoids having to do the inefficient operation to r= etrieve the currently read char by cutting off the last character from the = end of the content. Currently it is necessary to do this, if the logic want= s something additional to be done with the current char (eg. as it=20 can be seen in my code here)
      3. one such case of additional use is making the expect-char-proc property obsolete, as the predicate procedure= is now able to execute such logic too, besides other more advanced logging scenarios (like it can be seen in my code here: here besides logging everything to STDOUT, and also to a main log-file, I also log for each expect macro call the contents it tries to compare=20 against into a separately named minor log-file (useful for debugging), and = also to ensure that=20 we actually do not write to STDOUT when using expect on multiple parallel processes)
      4. To ensure that everything stays backwards compatible, t= his new feature is only enabled if the new expect-pass-char? binding is set to #t in the lexical context = (like here). Otherwise, by default this f= eature is disabled, and boolean eof?<= /span> flag argument is passed just like before.
  4. There is a better support for using default value/expressions for th= e various parameters the macros depend on in their lexical contexts. The ol= d macro had to export default values defined in its module, so as to ensure= the defaults are available. This either clutters the module's global c= ontext where (ice-9 expect) is= used, or the defaults are not available when using something like ((ice-9 expect) #:select (expect expect-strin= g)).=C2=A0 The new defaults are calculated based on the lexical cont= ext where the macros are used, and expands into the correct default values = when they are not defined. For example when expect-port is not defined in t= he lexical context, then the default value used is going to be the result o= f the (current-input-port) exp= ression. I would like someone to review whether I've implemented this c= orrectly, as it seems the current-inp= ut-port identifier default for expect-port gets captured in the expansion. Not sure why that happe= ns.
  5. The expect-strings macro supports the =3D> syntax, which passes the output(s) of the t= est predicate to a procedure, instead of just evaluating a body. This is fu= lly supported, but additionally, the = =3D> syntax now also works with the procedural expect macro too! = I haven't verified if the original implementation did this too, but at = least it wasn't described in the documentation!
  6. As an added= bonus, I have added a simplistic implementation for an interact macro. It = uses threads to read from STDIO, and expects expect-port to be an input-out= put-port, so that it can output the lines read from STDIO into the same por= t. Not too advanced, as it doesn't support a full terminal interface, w= ith autocomplete, etc... but works well enough for my use-case to interact = with a KVM process via serial port.
  7. mnieper said tha= t I am not using syntax-case idiomatically, and my style is more in the for= m of a syntax-rules-based stat= e-machine.
    I am quite happy with how it is currently, but would be open = to be convinced if there is a superior way of expressing all this.
  8. = I currently have various tests sprinkled around my implementation, to manua= lly demonstrate expansions, and executions.
    Also, I have added some auto= mated unit tests for the bind-locally= helper procedure, which I used for checking whether parameters are = defined in the lexical scope.
  9. I am not happy with how I am managing= my project structure, and how my modules can reference each other (having = to call add-to-load-path everywhere).
    Also, I think the way I am using s= rfi-64 tests is a bit bothersome, as they always get executed when my modul= es are loaded, and I have to globally disable logging to files.
    I would = like some recommendations how to do this better!
If the commu= nity likes this implementation, I would be happy to apply/rebase my changes= onto the official repo, and then we can go from there!

Best regards,
Daniel Dinnyes

=

--000000000000d8d69a05f9615da6--