#+title: Opensmtpd Records Task List #+AUTHOR: Joshua Branson (service (@ (gnu services mail) opensmtpd-service-type) ((@ (gnu services mail) opensmtpd-configuration) (config-file …))) * tasks ** PROJ I have decent data structures. now let's get some good code. [0/6] *** why are good data structures important? **** nckx's advice: use a simple 1-1 mapping "...as I think Guix services ought to faithfully wrap the native syntax whenever possible (implement alternative simple APIs on top of that — fine)." -nckx from irc on #guix **** To follow nckx's advice, one might create the == like this: #+BEGIN_SRC scheme (service opensmtpd-service (opensmtpd-configuration (includes ...) (tables ...) (pkis ...) (filters ...) (listen-on ...) (actions ...) (matches ...))) #+END_SRC Defining the service this way, makes it VERY easy from a development point of view. But it makes it possible for users to create simple mistakes when defining the service. For example, it is possible to define an nginx service that will successfully reconfigure the system. BUT after reboot nginx refuses to start. Why? Who knows. Guix won't tell you. Neither will the Shepherd. To fix this, the user has to go digging into the nginx logs, and he might not know where to find those. If possible, when the user specificies a == that has obvious errors, then the guix services should make reconfigure fail and print a helpful error message. **** BUT it would be better if the service uses better datastructures. I should follow nckx's advice, and Linus' advice: good programmers use good datastructures. If you have good datastructures, then your code will almost write itself. It might make the service a little harder to develop, but end-users will find the service easier to use. This would eliminate common errors like misspellings and give appropriate error messages. Practically it would ensure each == has a corresponding ==, creating a table name and then misspelling the table name later, and defining a table but never using it, etc. **** Example configuration #+BEGIN_SRC scheme (service opensmtpd-service-type (let ([interface "lo"] [creds-table (opensmtpd-table-configuration (name "creds") (data (list (cons "joshua" "$6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86."))))] [receive-action (opensmtpd-action-local-delivery-configuration (name "receive") (method (opensmtpd-maildir-configuration (pathname "/home/%{rcpt.user}/Maildir") (junk #t))) (virtual (opensmtpd-table-configuration (name "virtual") (data (list "josh" "jbranso@dismail.de")))))] [filter-dkimsign (opensmtpd-filter-configuration (name "dkimsign") (exec #t) (proc (string-append "/path/to/dkimsign -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " "/path/to/dkimsign-key user nobody group nobody")))] [smtp.gnucode.me (opensmtpd-pki-configuration (domain "smtp.gnucode.me") (cert "opensmtpd.scm") (key "opensmtpd.scm"))]) (opensmtpd-configuration (mta-max-deferred 50) (queue (opensmtpd-queue-configuration (compression #t))) (smtp (opensmtpd-smtp-configuration (max-message-size "10M"))) (srs (opensmtpd-srs-configuration (ttl-delay "5d"))) (listen-ons (list (opensmtpd-listen-on-configuration (interface interface) (port 25) (secure-connection "tls") (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (decision "disconnect") (message "433 No FCRDNS")))) (pki smtp.gnucode.me)) ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on-configuration (interface interface) (port 465) (secure-connection "smtps") (pki smtp.gnucode.me) (auth creds-table) (filters (list filter-dkimsign))) (opensmtpd-listen-on-configuration (interface interface) (port 587) (secure-connection "tls-require") (pki smtp.gnucode.me) (auth creds-table) (filters (list filter-dkimsign))))) (matches (list (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "relay"))) (options (list (opensmtpd-option-configuration (option "for any")) (opensmtpd-option-configuration (option "from any")) (opensmtpd-option-configuration (option "auth"))))) (opensmtpd-match-configuration (action receive-action) (options (list (opensmtpd-option-configuration (option "from any")) (opensmtpd-option-configuration (option "for domain") (data (opensmtpd-table-configuration (name "domain-table") (data (list "gnucode.me" "gnu-hurd.com")))))))) (opensmtpd-match-configuration (action receive-action) (options (list (opensmtpd-option-configuration (option "for local")))))))))) #+END_SRC :OldConfigurationSyntax: #+BEGIN_SRC scheme (service opensmtpd-service-type (opensmtpd-configuration (pkis (list (opensmtpd-pki-configuration ...))) (tables (list (opensmtpd-table-configuration ...) (opensmtpd-table-configuration ...))) (listen-ons (list (opensmtpd-listen-on-configuration ...) (opensmtpd-listen-on-configuration ...))) (actions (list (opensmtpd-action ...) (opensmtpd-action ...))) (matches (list (opensmtpd-match-configuration ...) (opensmtpd-match-configuration ...))) (filter-chains (list (opensmtpd-filter-chain (name "dropDumbEmails") (filter-names (list "nofcrdnsDisconnect" "nordnsDisconnect"))))) (filter-phases (list (opensmtpd-filter-phase-configuration ...) (opensmtpd-filter-phase-configuration ...))))) #+END_SRC Here you have to define the =pki=s twice! You define it once in the =pkis= section, and then you reference it later. This could potentially cause a mispelling error. That would be silly to debug as an end-user. :END: *** PROJ tweek the code for == & == records [4/7] **** Why I chose the current datastructures of == & == According to the man page I have a four kinds of filters: #+BEGIN_EXAMPLE 1. filter chain-name chain {filter-name [, ...]} Register a chain of filters chain-name, consisting of the filters listed from filter-name. Filters part of a filter chain are executed in order of declaration for each phase that they are registered for. A filter chain may be used in place of a filter for any direc‐ tive but filter chains themselves. 2. filter filter-name phase phase-name match conditions decision Register a filter filter-name. A decision about what to do with the mail is taken at phase phase-name when matching conditions. Phases, matching conditions, and decisions are described in MAIL FILTERING, below. 3. filter filter-name proc proc-name Register "proc" filter filter-name backed by the proc-name process. 4. filter filter-name proc-exec command Register and execute "proc" filter filter-name from command. If command starts with a slash it is executed with an absolute path, else it will be run from “/gnu/store/2d13sdz76ldq8zgwv4wif0zx7hkr3mh2-opensmtpd-6.8.0p2/libexec/opensmtpd”. #+END_EXAMPLE =chain-name= could be easily represented as a list of filters. in the opensmtpd-configuration-filter fieldname: #+BEGIN_SRC scheme (opensmtpd-configuration (listen-on (filter (list (opensmtpd-filter-configuration) (opensmtpd-filter-configuration) (opensmtpd-filter-configuration))))) #+END_SRC For example, this is probably easier #+BEGIN_SRC scheme (opensmtpd-configuration (actions (list (opensmtpd-action (name "relay") (method (opensmtpd-relay-configuration (domain (opensmtpd-table-configuration ;;(name "domains") ;; with some smart coding, the name would NOT be needed. (data (list "gnucode.me" "gnu-hurd.com")))))))))) #+END_SRC than the alternative: #+BEGIN_SRC scheme (opensmtpd-configuration (tables (list (opensmtpd-table-configuration (name "domains") (data (list "gnucode.me" "gnu-hurd.com"))))) (actions (list (opensmtpd-action (name "relay") (method (opensmtpd-relay-configuration (domain "domains"))))))) #+END_SRC **** some example code for each of the 3 types of filters 1. filter phase #+BEGIN_SRC scheme (opensmtpd-filter-phase-configuration (name "phase") (phase "connect") (options (list (opensmtpd-option-configuration (option "src") (not #t) (regex #t) (table (opensmtpd-table-configuration (name "src-option-table") (data (list "cat" "hot"))))))) (decision "reject") (message "We do not want spam here!")) #+END_SRC #+RESULTS: 2. filter proc this is a filter-proc #+BEGIN_SRC scheme (opensmtpd-filter (name "proc") (proc "dkimsign")) #+END_SRC 3. filter proc-exec #+BEGIN_SRC scheme (opensmtpd-filter (name "proc") (exec #t) (proc "dkimsign")) #+END_SRC ***** Why am I doing the data structure like the above? filter-proc and proc-exec as defined in man smtpd.conf can both use the same record. That works just fine. But filter-phase is a different beast. I do NOT want someone to accidentally define something like the following which is BAD data: #+BEGIN_SRC scheme (opensmtpd-filter (name "proc") (exec #t) (proc "dkimsign")) #+END_SRC **** NO Is it advantageous/desireable to merge == & When a user creates a filter, he is either going to create a =~ or an ~= NOT both. If we define separate records, then it is impossible for a user to accidentally define a filter record using fieldnames from both filter types. eg: #+BEGIN_SRC scheme (opensmtpd-filter-configuration (name "filter") (exec #t) (proc "dkimsign") (phase "connect")) ;; this phase should NOT be there. this is a #+END_SRC If == & == are separte then the above would correctly result in an error message for free. **** TODO make fieldname 'proc' accept a list of strings and/or a s Suppose you want to do dkimsigning in smtpd.conf. Here is how you might register the official opensmtpd dkimsign filter: #+BEGIN_EXAMPLE filter "dkimsign" proc-exec "filter-dkimsign -d -s \ -k /gnu/store/2d13sdz76ldq8zgwv4wif0zx7hkr3mh2-opensmtpd-6.8.0p2/etc/dkim/private.key" user _dkimsign group _dkimsign #+END_EXAMPLE For example my hacky code to do dkimsigning looks like: #+BEGIN_SRC scheme filter \"dkimsign\" \ proc-exec \"" path-to-filter-dkimsign " -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " etc-dkimsign-key-file " \" \ user nobody group nogroup #+END_SRC Here is some example code of how we could create an == that registers a dkimsign filter. The code below probably will NOT work. #+BEGIN_SRC scheme (let ((etc-dkimsign-key-file "filename.key") (path-to-dkimsign-key "/etc/opensmtpd/"))) (opensmtpd-filter-configuration (name "dkimsign") (proc (list (file-append opensmtpd-filter-dkimsign "/libexec/opensmtpd/filter-dkimsign") " -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " ~#(let ([UID-nobody (passwd:uid (getpw "nobody"))] [GID-root (group:gid (getgr "root"))] [GID-nogroup (group:gid (getgr "nogroup"))]) ;; #o550 user root can read/open the directory ;; and the group "root" can read/open the directory. ;; change these two lines to (mkdir-p) ? (unless (file-exists? "/etc/opensmtpd") (mkdir "/etc/opensmtpd" #o550)) ;; root can read/write/execute on directory dkimsign ;; group "root" can read and execute (unless (file-exists? "/etc/opensmtpd/dkimsign") (mkdir "/etc/opensmtpd/dkimsign" #o750)) (copy-file path-to-dkimsign-key etc-dkimsign-key-file) ;; make the etc-dkimsign-key-file to owned by nobody and group nogroup. (chown "/etc/opensmtpd" UID-nobody GID-root) (chown "/etc/opensmtpd/dkimsign" UID-nobody GID-root) (chown etc-dkimsign-key-file UID-nobody GID-nogroup) "/etc/opensmtpd/dkimsign/2021-09-22-rsa1024-gnucode.me.key") "user nobody group nogroup")) (exec #)) #+END_SRC Here is the full for how I currently run opensmtpd: #+BEGIN_SRC sh :dir ~/prog/gnu/guix/guix-config/linode-guix-system-configuration/ :results raw cat opensmtpd.scm #+END_SRC #+RESULTS: #+BEGIN_SRC scheme (define-module (opensmtpd) #:use-module (guix gexp) #:use-module (guix records) #:use-module (gnu packages mail) ;;for finding location of filter-dkimsign #:export ( %smtpd.conf )) ;; to create credentials for now, I need to do the following: ;; find /gnu/store -name '*encrypt*' | grep opensmtpd ;; /gnu/store/blah/opensmtpd/encrypt (define creds (plain-file "creds" ;; this is my joshua's password for server. This can be found on dobby's /home/joshua/.authinfo/ "joshua $6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86.")) (define vdoms (plain-file "vdoms" "gnucode.me gnu-hurd.com")) (define vusers (plain-file "vusers" "joshua@gnucode.me joshua jbranso@gnucode.me joshua postmaster@gnucode.me joshua")) (define path-to-filter-dkimsign (file-append opensmtpd-filter-dkimsign "/libexec/opensmtpd/filter-dkimsign")) (define path-to-dkimsign-key (string-append (getcwd) "/email-dkim-ssh-keys/2021-09-22-rsa1024-gnucode.me.key")) (define etc-dkimsign-key-file "/etc/opensmtpd/dkimsign/2021-09-22-rsa1024-gnucode.me.key") ;; FIXME: This should become a derivation. Currently it just runs when I evaluate ;; %smtpd.conf. For example it should look like this? ;; (define build-exp ;; #~(begin ;; (mkdir #$output) ;; (chdir #$output) ;; (symlink (string-append #$coreutils "/bin/ls") ;; "list-files"))) ;; I will need to extend the opensmtpd service, to create a directory ;; in etc. This line needs to be added to etc-service. ;; (service-extension etc-service-type opensmtpd-etc-service) ;; I'll then need to create a opensmtpd-etc-service procedure. ganeti has ;; a good example. ;; It should also use the /etc service, which is a service for creating ;; directories and files in /etc ? (define (create-etc-dkimsign-key-file) #~(let ([UID-nobody (passwd:uid (getpw "nobody"))] [GID-root (group:gid (getgr "root"))] [GID-nogroup (group:gid (getgr "nogroup"))]) ;; #o550 user root can read/open the directory ;; and the group "root" can read/open the directory. ;; change these two lines to (mkdir-p) ? (unless (file-exists? "/etc/opensmtpd") (mkdir "/etc/opensmtpd" #o550)) ;; root can read/write/execute on directory dkimsign ;; group "root" can read and execute (unless (file-exists? "/etc/opensmtpd/dkimsign") (mkdir "/etc/opensmtpd/dkimsign" #o750)) (copy-file path-to-dkimsign-key etc-dkimsign-key-file) ;; ;; ;; make the etc-dkimsign-key-file to owned by nobody and group nogroup. (chown "/etc/opensmtpd" UID-nobody GID-root) (chown "/etc/opensmtpd/dkimsign" UID-nobody GID-root) (chown etc-dkimsign-key-file UID-nobody GID-nogroup) etc-dkimsign-key-file)) (define %smtpd.conf (mixed-text-file "smtpd.conf" " # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. # borrowed from the archlinux guix # https://wiki.archlinux.org/index.php/OpenSMTPD#Simple_OpenSMTPD/mbox_configuration # My TLS certificate and key table aliases file:/etc/aliases pki smtp.gnucode.me cert \"/etc/letsencrypt/live/gnucode.me/fullchain.pem\" pki smtp.gnucode.me key \"/etc/letsencrypt/live/gnucode.me/privkey.pem\" # for now I am NOT using the virtual credentials # table creds { joshua = $6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86. } table creds \"" creds "\" table vdoms \"" vdoms "\" # table vdoms { gnucode.me, gnu-hurd.com } # table vusers { joshua@gnucode.me = joshua, jbranso@gnucode.me = joshua, postmaster@gnucode.me = joshua } table vusers \"" vusers "\" # this totally works! run this as user nobody! # info about dkimsign ...ing # https://openports.pl/path/mail/opensmtpd-filters/dkimsign # sudo -u nobody /gnu/store/g17vdv4l03bacn7qbdpb5v8l8vgdxcld-opensmtpd-filter-dkimsign-0.5/libexec/opensmtpd/filter-dkimsign -d gnucode.me -s 2020 -c relaxed/relaxed -k etc-dkimsign-key-file /home/joshua/linode-guix-system-configuration/email-dkim-ssh-keys/20201004-gnucode.me.key user nobody group nogroup filter \"dkimsign\" \ proc-exec \"" path-to-filter-dkimsign " -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " etc-dkimsign-key-file ;;(create-etc-dkimsign-key-file) " \" \ user nobody group nogroup # port 25 is used only for receiving from external servers, and they may start a # TLS session if the want. listen on eth0 port 25 tls pki smtp.gnucode.me # For sending messages from outside of this server, you need to authenticate and use # TLS listen on eth0 port 465 smtps pki smtp.gnucode.me auth filter \"dkimsign\" listen on eth0 port 587 tls-require pki smtp.gnucode.me auth filter \"dkimsign\" # users logged-in/ssh-ed into the system can send email listen on lo port 25 tls pki smtp.gnucode.me # receive email action action \"receive\" maildir \"/home/%{rcpt.user}/Maildir\" junk virtual # action send the email to the world action \"send\" relay # We accept to send email from any mail from authenticated users match for any from any auth action \"send\" #finally we receive any incoming email # maybe the next \"from any\" should be changed to \"for rdns\". match from any for domain action \"receive\" match for local action \"receive\"")) (define-module (opensmtpd) #:use-module (guix gexp) #:use-module (guix records) #:use-module (gnu packages mail) ;;for finding location of filter-dkimsign #:export ( %smtpd.conf )) ;; to create credentials for now, I need to do the following: ;; find /gnu/store -name '*encrypt*' | grep opensmtpd ;; /gnu/store/blah/opensmtpd/encrypt (define creds (plain-file "creds" ;; this is my joshua's password for server. This can be found on dobby's /home/joshua/.authinfo/ "joshua $6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86.")) (define vdoms (plain-file "vdoms" "gnucode.me gnu-hurd.com")) (define vusers (plain-file "vusers" "joshua@gnucode.me joshua jbranso@gnucode.me joshua postmaster@gnucode.me joshua")) (define path-to-filter-dkimsign (file-append opensmtpd-filter-dkimsign "/libexec/opensmtpd/filter-dkimsign")) (define path-to-dkimsign-key (string-append (getcwd) "/email-dkim-ssh-keys/2021-09-22-rsa1024-gnucode.me.key")) (define etc-dkimsign-key-file "/etc/opensmtpd/dkimsign/2021-09-22-rsa1024-gnucode.me.key") ;; FIXME: This should become a derivation. Currently it just runs when I evaluate ;; %smtpd.conf. For example it should look like this? ;; (define build-exp ;; #~(begin ;; (mkdir #$output) ;; (chdir #$output) ;; (symlink (string-append #$coreutils "/bin/ls") ;; "list-files"))) ;; I will need to extend the opensmtpd service, to create a directory ;; in etc. This line needs to be added to etc-service. ;; (service-extension etc-service-type opensmtpd-etc-service) ;; I'll then need to create a opensmtpd-etc-service procedure. ganeti has ;; a good example. ;; It should also use the /etc service, which is a service for creating ;; directories and files in /etc ? (define (create-etc-dkimsign-key-file) #~(let ([UID-nobody (passwd:uid (getpw "nobody"))] [GID-root (group:gid (getgr "root"))] [GID-nogroup (group:gid (getgr "nogroup"))]) ;; #o550 user root can read/open the directory ;; and the group "root" can read/open the directory. ;; change these two lines to (mkdir-p) ? (unless (file-exists? "/etc/opensmtpd") (mkdir "/etc/opensmtpd" #o550)) ;; root can read/write/execute on directory dkimsign ;; group "root" can read and execute (unless (file-exists? "/etc/opensmtpd/dkimsign") (mkdir "/etc/opensmtpd/dkimsign" #o750)) (copy-file path-to-dkimsign-key etc-dkimsign-key-file) ;; ;; ;; make the etc-dkimsign-key-file to owned by nobody and group nogroup. (chown "/etc/opensmtpd" UID-nobody GID-root) (chown "/etc/opensmtpd/dkimsign" UID-nobody GID-root) (chown etc-dkimsign-key-file UID-nobody GID-nogroup) etc-dkimsign-key-file)) (define %smtpd.conf (mixed-text-file "smtpd.conf" " # This is the smtpd server system-wide configuration file. # See smtpd.conf(5) for more information. # borrowed from the archlinux guix # https://wiki.archlinux.org/index.php/OpenSMTPD#Simple_OpenSMTPD/mbox_configuration # My TLS certificate and key table aliases file:/etc/aliases pki smtp.gnucode.me cert \"/etc/letsencrypt/live/gnucode.me/fullchain.pem\" pki smtp.gnucode.me key \"/etc/letsencrypt/live/gnucode.me/privkey.pem\" # for now I am NOT using the virtual credentials # table creds { joshua = $6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86. } table creds \"" creds "\" table vdoms \"" vdoms "\" # table vdoms { gnucode.me, gnu-hurd.com } # table vusers { joshua@gnucode.me = joshua, jbranso@gnucode.me = joshua, postmaster@gnucode.me = joshua } table vusers \"" vusers "\" # this totally works! run this as user nobody! # info about dkimsign ...ing # https://openports.pl/path/mail/opensmtpd-filters/dkimsign # sudo -u nobody /gnu/store/g17vdv4l03bacn7qbdpb5v8l8vgdxcld-opensmtpd-filter-dkimsign-0.5/libexec/opensmtpd/filter-dkimsign -d gnucode.me -s 2020 -c relaxed/relaxed -k etc-dkimsign-key-file /home/joshua/linode-guix-system-configuration/email-dkim-ssh-keys/20201004-gnucode.me.key user nobody group nogroup filter \"dkimsign\" \ proc-exec \"" path-to-filter-dkimsign " -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " etc-dkimsign-key-file ;;(create-etc-dkimsign-key-file) " \" \ user nobody group nogroup # port 25 is used only for receiving from external servers, and they may start a # TLS session if the want. listen on eth0 port 25 tls pki smtp.gnucode.me # For sending messages from outside of this server, you need to authenticate and use # TLS listen on eth0 port 465 smtps pki smtp.gnucode.me auth filter \"dkimsign\" listen on eth0 port 587 tls-require pki smtp.gnucode.me auth filter \"dkimsign\" # users logged-in/ssh-ed into the system can send email listen on lo port 25 tls pki smtp.gnucode.me # receive email action action \"receive\" maildir \"/home/%{rcpt.user}/Maildir\" junk virtual # action send the email to the world action \"send\" relay # We accept to send email from any mail from authenticated users match for any from any auth action \"send\" #finally we receive any incoming email # maybe the next \"from any\" should be changed to \"for rdns\". match from any for domain action \"receive\" match for local action \"receive\"")) #+END_SRC **** TODO what does rewrite needs value mean? Should it be a number? this is for == from the documentation rewrite value the command parameter is rewritten with value **** DONE sanitize the fieldname 'filters'. I can probably reuse existing code from the sanitize procedure found in == fieldname 'filters'. **** DONE write a get-opensmtpd-filters procedure This procedure takes all the values of fieldname 'filters' and fieldname 'filters'. It returns a list of , , and filter-chains, which is a list of and . An example of what this might return is: #+BEGIN_SRC scheme (list (list (opensmtpd-filter)) (list (opensmtpd-filter-phase-configuration)) (list (opensmtpd-filter) ; this list is a filter-chain (opensmtpd-filter-phase-configuration)) (list (opensmtpd-filter-phase-configuration) ; this list is also a filter chain (opensmtpd-filter) (opensmtpd-filter))) #+END_SRC These are some example bits of code that I can test my resulting code on. All unique filters of and . 4 of them. One listen-on has no filters. #+BEGIN_SRC scheme (let ([interface "lo"] [filter-dkimsign (opensmtpd-filter (name "dkimsign") (exec #t) (proc (string-append "/path/to/dkimsign -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " "/path/to/dkimsign-key user nobody group nobody")))]) (opensmtpd-configuration (listen-on-socket (opensmtpd-listen-on-socket-configuration-configuration (filters (list (opensmtpd-filter (name "rspamd") (proc "rspamd")))))) (listen-ons (list (opensmtpd-listen-on-configuration (interface interface) (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (decision "disconnect") (message "No FCRDNS"))))) ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on (interface interface) (port 27) (filters (list filter-dkimsign))) (opensmtpd-listen-on-configuration (port 29) (interface interface)) (opensmtpd-listen-on-configuration (interface interface) (filters (list (opensmtpd-filter (name "rspamd") (proc "rspamd") (exec #t))))))))) #+END_SRC 4 unique filters. One of the filters is a filter chain. #+BEGIN_SRC scheme (let ([interface "lo"] [filter-dkimsign (opensmtpd-filter (name "dkimsign") (exec #t) (proc (string-append "/path/to/dkimsign -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " "/path/to/dkimsign-key user nobody group nobody")))]) (opensmtpd-configuration (listen-on-socket (opensmtpd-listen-on-socket-configuration-configuration (filters (list (opensmtpd-filter (name "spamassassain") (proc "spamassassain")) (opensmtpd-filter-phase-configuration (name "rdns") (phase "data") (options (list (opensmtpd-option-configuration (option "rdns") (not #t)))) (decision "reject") (message "No RDNS")) (opensmtpd-filter (name "block") (proc "block")) (opensmtpd-filter-phase-configuration (name "auth") (phase "commit") (options (list (opensmtpd-option-configuration (option "auth") (regex #t) (not #t) (table (opensmtpd-table-configuration (name "auth-table") (data (list ".*@gmail.com" ".*@dismail.de"))))))) (decision "junk")))))) (listen-ons (list (opensmtpd-listen-on-configuration (interface interface) (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (decision "disconnect") (message "No FCRDNS"))))) ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on-configuration (interface interface) (port 27) (filters (list filter-dkimsign))) (opensmtpd-listen-on-configuration (port 29) (interface interface)) (opensmtpd-listen-on-configuration (interface interface) (filters (list (opensmtpd-filter (name "rspamd") (proc "rspamd") (exec #t))))))))) #+END_SRC No filters at all. #+BEGIN_SRC scheme (get-opensmtpd-filters (opensmtpd-configuration)) ; no filters at all. #+END_SRC This one prints rspamd twice! The get-opensmtpd-filters procedure returns a duplicate filter. While get-opensmtpd-filters does return a duplicate filter. #+BEGIN_SRC scheme (opensmtpd-configuration (listen-on-socket (opensmtpd-listen-on-configuration-socket-configuration (filters (list (opensmtpd-filter (name "rando") (proc "rando")))))) (listen-ons (list (opensmtpd-listen-on-configuration (port 25) (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (decision "disconnect") (message "No FCRDNS")) (opensmtpd-filter (name "rspamd") (proc "rspamd"))))) (opensmtpd-listen-on-configuration (port 465) (filters (list (opensmtpd-filter (name "rspamd") (proc "rspamd")) (opensmtpd-filter (name "block") (proc "block"))))) (opensmtpd-listen-on-configuration (port 587)) (opensmtpd-listen-on-configuration (port 999) (filters (list (opensmtpd-filter (name "bogofilter") (proc "bogofilter")))))))) #+END_SRC #+BEGIN_SRC scheme (opensmtpd-configuration (listen-ons (list (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter (name "bogofilter") (proc "bogofilter"))))) (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter (name "noFRDNS") (proc "noFRDNS")) (opensmtpd-filter (name "rspamd") (proc "rspamd")))))))) #+END_SRC **** PROJ by which method should I turn the filters in == into strings? [1/3] in (opensmtpd-configuration->mixed-text-file ) do either 1. one line of code. faster, but violates the convention set by the other lines of code around it. #+BEGIN_SRC scheme ;; write out all the filters (opensmtpd-filters->string (get-opensmtpd-filters record)) #+END_SRC This task in done: [[*(opensmtpd-filters->string (get-opensmtpd-filters record))][(opensmtpd-filters->string (get-opensmtpd-filters record))]] 2. two lines of code. slower, but follows the convention set by the other lines of code around it. #+BEGIN_SRC scheme (opensmtpd-configuration-fieldname->string record get-opensmtpd-filters-and-filter-phases opensmtpd-filter-and-filter-phase->string) (opensmtpd-configuration-fieldname->string record get-opensmtpd-filter-chains opensmtpd-filter-chain->string) #+END_SRC ***** DONE (opensmtpd-filters->string (get-opensmtpd-filters record)) Have one procedure that prints out all filters. #+BEGIN_SRC scheme ;; write out all the filters (opensmtpd-filters->string (get-opensmtpd-filters record)) #+END_SRC ***** PROJ 4 procedures: get-filter-and-filter-phases, filter-and-filter-phases->string, get-filter-chains, filter-chains->string The bonus with this method is that I can add these two lines in opensmtpd-configuration->mixed-text-file and keep a consistent coding framework: #+BEGIN_SRC scheme ;; write out all the filters and filter-phases (opensmtpd-configuration-fieldname->string record get-opensmtpd-filters-and-filter-phases opensmtpd-filter-and-filter-phase->string) ;; write out all the filter chains (opensmtpd-configuration-fieldname->string record get-opensmtpd-filter-chains opensmtpd-filter-chain->string) #+END_SRC **** DONE fix the sanitize procedure for == fieldnames 'phase-name', 'decision', etc. [5/5] ***** DONE sanitize == so that fieldname 'decision' option "reject" and "disconnect" requires a 'message'. :LOGBOOK: CLOCK: [2022-04-01 Fri 22:45]--[2022-04-02 Sat 04:13] => 5:28 :END: This message must be RFC compliant. The message must start with 4xx or 5xx status code. #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "junk") (phase "connect") (decision "junk") (options (list (opensmtpd-option-configuration (option "rdns"))))) (opensmtpd-filter-phase-configuration (name "src") (phase "connect") (options (list (opensmtpd-option-configuration (option "src") (data (opensmtpd-table-configuration (name "src-table") (data (list "cat" "hat"))))))) (decision "reject"))))) #+END_SRC #+RESULTS: #+BEGIN_EXAMPLE fieldname: 'decision' options "disconnect" and "reject" require fieldname 'message' to have a string. ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `bad!' with args `(#< name: "src" phase: "connect" options: (#< option: "src" not: #f regex: #f table: #< name: "src-table" file-db: #f values: ("cat" "hat") type: #:5894:22 (x)>>>) decision: "reject" message: #f value: #f>)'. Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. #+END_EXAMPLE #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "junk") (phase "connect") (decision "junk") (options (list (opensmtpd-option-configuration (option "rdns"))))) (opensmtpd-filter-phase-configuration (name "src") (phase "connect") (options (list (opensmtpd-option-configuration (option "src") (data (opensmtpd-table-configuration (name "src-table") (data (list "cat" "hat"))))))) (decision "reject") (message "322 Hello"))))) fieldname: 'decision' options "disconnect" and "reject" require fieldname 'message' to have a string. The 'message' string must be RFC commpliant, which means that the string must begin with a 4xx or 5xx status code. ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `bad!' with args `(#< name: "src" phase: "connect" options: (#< option: "src" not: #f regex: #f data: #< name: "src-table" file-db: #f data: ("cat" "hat") type: #:1153:21 (x)>>>) decision: "reject" message: "322 Hello" value: #f>)'. Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. #+END_SRC ***** DONE sanitize == so that fieldname 'decision' option "rewrite" requires a 'value'. :LOGBOOK: CLOCK: [2022-04-01 Fri 22:45]--[2022-04-02 Sat 04:13] => 5:28 :END: #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (value #f) (decision "rewrite")) ;; there needs to be a value here. rewrite requires a value! ))) $12 = #< interface: "lo" family: #f auth: #f auth-optional: #f filters: (#< name: "noFRDNS" phase: "commit" options: (#< option: "fcrdns" not: #t regex: #f data: #f>) decision: "rewrite" message: #f value: 343>) hostname: #f hostnames: #f mask-src: #f disable-dsn: #f pki: #f port: #f proxy-v2: #f received-auth: #f secure-connection: #f tag: #f> scheme@(opensmtpd-records) [10]> (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (decision "rewrite")) ;; there needs to be a value here. rewrite requires a value! ))) fieldname: 'decision' option "rewrite" requires fieldname 'value' to have a number. ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `bad!' with args `(#< name: "noFRDNS" phase: "commit" options: (#< option: "fcrdns" not: #t regex: #f data: #f>) decision: "rewrite" message: #f value: #f>)'. Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. #+END_SRC ***** DONE sanitize == so that fieldname 'decision' option "junk" and "bypass" have no message or value #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") ))) (decision "junk") (message "This is not a good email."))))) ; there should NOT be a message here! junk has no message. #+END_SRC ***** DONE sanitize the options too. rdns requires a table for instance: #+BEGIN_EXAMPLE At each phase, various conditions may be matched. The fcrdns, rdns, and src data are available in all phases, but other data must have been already submitted before they are available. fcrdns forward-confirmed reverse DNS is valid rdns session has a reverse DNS rdns session has a reverse DNS in table src
source address is in table helo
helo name is in table auth session is authenticated auth
session username is in table mail-from
sender address is in table rcpt-to
recipient address is in table #+END_EXAMPLE ***** DONE sanitize make sure that 'junking happens before phase 'committed'. #+BEGIN_EXAMPLE Descisions can be taken at any phase, though junking can only happen before a message is committed. #+END_EXAMPLE #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "junk-after-commit") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") ))) (decision "junk"))))) #+END_SRC *** PROJ make fieldnames that need a table accept a value of table [3/4] **** DONE opensmtpd-action-local-delivery-configuration [3/3] ***** DONE alias == [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 03:53] :END: ****** DONE change the sanitize portion of the fieldname alias in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-action-local-delivery-configuration (alias (opensmtpd-table-configuration (name "My-table") (data (list "gnu-hurd.com" "gnucode.me"))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-action-local-delivery-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-action-local-delivery-configuration->string) (opensmtpd-action-local-delivery-configuration (alias (opensmtpd-table-configuration (name "My-table") (data (list "gnu-hurd.com" "gnucode.me")))))) #+END_SRC ***** DONE userbase == [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 03:53] :END: ****** DONE change the sanitize portion of the fieldname userbase in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-action-local-delivery-configuration (userbase (opensmtpd-table-configuration (name "this") (data (list "job" "done"))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-action-local-delivery-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-action-local-delivery-configuration->string) (opensmtpd-action-local-delivery-configuration (userbase (opensmtpd-table-configuration (name "this") (data (list "job" "done")))))) #+END_SRC ***** DONE virtual [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 03:53] :END: ****** DONE change the sanitize portion of the fieldname virtual in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-action-local-delivery-configuration (virtual (opensmtpd-table-configuration (name "this") (data (list "job" "done"))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-action-local-delivery-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-action-local-delivery-configuration->string) (opensmtpd-action-local-delivery-configuration (virtual (opensmtpd-table-configuration (name "this") (data (list "job" "done")))))) #+END_SRC **** DONE opensmtpd-relay-configuration [4/4] ***** DONE helo-src :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:00] :END: ***** DONE domain == [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:00] :END: ****** DONE change the sanitize portion of the fieldname domain in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-action-relay-configuration (domain (opensmtpd-table-configuration (name "this") (data (list "gnucode.me" "gnu-hurd.com"))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-action-relay-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-action-relay-configuration->string) (opensmtpd-action-relay-configuration (domain (opensmtpd-table-configuration (name "this") (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ***** DONE auth == [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:00] :END: ****** DONE change the sanitize portion of the fieldname == in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-action-relay-configuration (auth (opensmtpd-table-configuration (name "this") (data (list "gnucode.me" "gnu-hurd.com"))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-action-relay-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-action-relay-configuration->string) (opensmtpd-action-relay-configuration (auth (opensmtpd-table-configuration (name "this") (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ***** DONE src srcaddress | :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:01] :END: ****** DONE change the sanitize portion of the fieldname == in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-action-relay-configuration (src (opensmtpd-table-configuration (name "this") (data (list "gnucode.me" "gnu-hurd.com"))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-action-relay-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-action-relay-configuration->string) (opensmtpd-action-relay-configuration (src (opensmtpd-table-configuration (name "this") (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC Use the string scraddress or list table for the source IP address. **** DONE opensmtpd-listen-on-configuration [3/3] ***** DONE auth == [3/3] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:01] :END: ****** DONE change the sanitize portion of the fieldname 'auth' in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "My-table") (data '(("joshua" . "$some$Long$EncrytpedPassword")))))) #+END_SRC #+RESULTS: ****** DONE change relevant portions in opensmtpd-listen-on-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-listen-on-configuration->string) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "credentials") (data '(("joshua" . "$someLongEncrytpedPassword"))))))) #+END_SRC ****** DONE sanitize the == so that it can only be an opensmtpd-table-configuration, whose fieldname values are an assoc-list #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "the") (data (list "the" "cat"))))) #+END_SRC #+RESULTS: == fieldname: 'auth' is of type boolean, or an == record whose fieldname 'values' are an assoc-list. ***** DONE auth-optional == [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:01] :END: ****** DONE change the sanitize portion of the fieldname 'auth-optional' in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "My-table") (data '(("joshua" . "$some$Long$EncrytpedPassword")))))) #+END_SRC #+RESULTS: AND the below code will correctly result in an error! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "My-table") (data '("joshua" "$some$Long$EncrytpedPassword"))))) #+END_SRC #+RESULTS: : == fieldname: 'auth' is of type boolean, or an == record whose fieldname 'values' are an assoc-list : (eg: (opensmtpd-table-configuration (name "table") (data '("joshua" . "$encrypted$password")))). ****** DONE change relevant portions in opensmtpd-listen-on-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-listen-on-configuration->string) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "credentials") (data '(("joshua" . "$someLongEncrytpedPassword"))))))) #+END_SRC ***** DONE hostnames == [2/2] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:01] :END: ****** DONE change the sanitize portion of the fieldname 'hostnames' in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (hostnames (opensmtpd-table-configuration (name "My-table") (data '(("joshua" . "$some$Long$EncrytpedPassword")))))) #+END_SRC #+RESULTS: AND the below code will correctly result in an error! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (hostnames (opensmtpd-table-configuration (name "My-table") (data '("joshua" "$some$Long$EncrytpedPassword"))))) #+END_SRC #+RESULTS: : == fieldname: 'hostname' is of type boolean, or an == record whose fieldname 'values' are an assoc-list : (eg: (opensmtpd-table-configuration (name "table") (data '("joshua" . "$encrypted$password")))). ****** DONE change relevant portions in opensmtpd-listen-on-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-listen-on-configuration->string) (opensmtpd-listen-on-configuration (hostnames (opensmtpd-table-configuration (name "credentials") (data '(("joshua" . "$someLongEncrytpedPassword"))))))) #+END_SRC **** TODO opensmtpd-match [20/24] ******* NO list approach Guix probably won't like the list approach. #+BEGIN_SRC scheme (openmstpd-match (for (list 'not "for domain regex" (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC #+BEGIN_SRC scheme (openmstpd-match (for (list "! for domain" (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ******* opensmtpd-options-configuration approach I like this one quite a bit. This method is a little bit more verbose. Well I guess it's a lot more verbose. But it's easier for me to properly parse what the user wants. I would sanitize the options in the opensmtpd-match-configuration-for, openmsmtpd-match-from, opensmtpd-match-configuration-auth, opensmtpd-match-configuration-helo, opensmtpd-match-configuration-mail-from, opensmtpd-match-configuration-rcpt-to fieldnames. ********* for #+BEGIN_SRC scheme (openmstpd-match (for (opensmtpd-options-configuration (not #t) (method "domain regex") ;; valid options for "for" are "domain" or "domain regex" (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC Do I want a regex fieldname? Probably not. It makes it more verbose... #+BEGIN_SRC scheme (openmstpd-match (for (opensmtpd-options-configuration (not #t) (regex #t) (method "domain") ;; valid options for "for" are "domain" (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ********* from #+BEGIN_SRC scheme (openmstpd-match (from (opensmtpd-options-configuration (not #t) (method "rdns regex") ;;valid options for from are "auth" "auth regex", "mail-from" "mail-from regex", ;; "rdns", "rdns regex", "src", "src regex" (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC Do I want a regex fieldname? #+BEGIN_SRC scheme (openmstpd-match (from (opensmtpd-options-configuration (not #t) (regex #t) (method "rdns") ;;valid options for from are "auth", "mail-from", "rdns", "src" (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ********* auth #+BEGIN_SRC scheme (openmstpd-match (auth (opensmtpd-options-configuration (not #t) (method "auth regex") (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC Do I want a regex fieldname? #+BEGIN_SRC scheme (openmstpd-match (auth (opensmtpd-options-configuration (not #t) (regex #t) (method "auth") ;; valid options for auth are "auth" or this method can be left blank. (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ********* mail-from #+BEGIN_SRC scheme (openmstpd-match (mail-from (opensmtpd-options-configuration (not #t) (method "mail from") (opensmtpd-table-configuration (data (list "gnucode.me" "gnu-hurd.com")))))) #+END_SRC ****** I tweak opensmtpd-match record and add a opensmtpd-options #+BEGIN_SRC scheme (opensmtpd-match-configuration (name "action-name") (options (list (opensmtpd-options-configuration (method "for domain regex"))) )) #+END_SRC ****** PROJ many of these options are not completely sanitized. For example: "for domain" requires a domain | BUT this record, which does not have a domain gives no errors: #+BEGIN_SRC scheme (opensmtpd-match-configuration (for (opensmtpd-option-configuration (option "for domain"))) (action (opensmtpd-action-local-delivery-configuration (name "local") ))) #+END_SRC And there are a ton of other examples of this. ****** DONE for domain :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:17] :END: The datastructures work: #+BEGIN_SRC scheme (opensmtpd-match-configuration (name "local") (for (opensmtpd-option-configuration (option "for domain") (value (opensmtpd-table-configuration (name "this") (data (list "helo" "hello")))))) (action (opensmtpd-action-local-delivery-configuration))) #+END_SRC #Results : $4 = #< name: "local" action: #< method: "mbox" alias: #f ttl: #f user: #f userbase: #f virtual: #f wrapper: #f> for: #< option: "for domain" not: #f regex: #f value: #< name: "this" file-db: #f values: ("helo" "hello") type: #:148:97 (x)>>> from: #f auth: #f helo: #f mail-from: #f rcpt-to: #f tag: #f tls: #f> ****** DONE for domain regexp :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:19] :END: the datastructure works #+BEGIN_SRC scheme (opensmtpd-match-configuration (name "local") (for (opensmtpd-option-configuration (regex #t) (option "for domain") (value (opensmtpd-table-configuration (name "this") (data (list "helo" "hello")))))) (action (opensmtpd-action-local-delivery-configuration))) #+END_SRC #Results : $4 = #< name: "local" action: #< method: "mbox" alias: #f ttl: #f user: #f userbase: #f virtual: #f wrapper: #f> for: #< option: "for domain" not: #f regex: #f value: #< name: "this" file-db: #f values: ("helo" "hello") type: #:148:97 (x)>>> from: #f auth: #f helo: #f mail-from: #f rcpt-to: #f tag: #f tls: #f> ****** DONE make opensmtpd-match-configuration->string work print for rcpt the appropriate match lines if some values now accept Seems to work: #+BEGIN_SRC scheme (opensmtpd-match-configuration->string (opensmtpd-match-configuration (name "local") (for (opensmtpd-option-configuration (regex #t) (option "for domain") (value (opensmtpd-table-configuration (name "this") (data (list "helo" "hello")))))) (action (opensmtpd-action-local-delivery-configuration)))) $6 = "match for domain regex == action \"local\" \n" #+END_SRC also seems to work #+BEGIN_SRC scheme (opensmtpd-match-configuration->string (opensmtpd-match-configuration (name "local") (for (opensmtpd-option-configuration (option "for domain") (value (opensmtpd-table-configuration (name "this") (data (list "helo" "hello")))))) (action (opensmtpd-action-local-delivery-configuration)))) $7 = "match for domain == action \"local\" \n" #+END_SRC ****** DONE for rcpt :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:19] :END: ****** DONE for rcpt regexp :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:19] :END: ****** DONE from auth user | :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:19] :END: #+BEGIN_SRC scheme (opensmtpd-option-configuration (regex #t) (option "from auth") (value (opensmtpd-table-configuration (name "this") (data (list "helo" "hello"))))) $8 = #< option: "from auth" not: #f regex: #t value: #< name: "this" file-db: #f values: ("helo" "hello") type: #:224:14 (x)>>> #+END_SRC ****** DONE from auth regex user | :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:20] :END: ****** DONE make sure opensmtpd-option-configuration->string works for from auth if they use #+BEGIN_SRC scheme (opensmtpd-option-configuration->string (opensmtpd-option-configuration (regex #t) (option "from auth") (value (opensmtpd-table-configuration (name "this") (data (list "helo" "hello")))))) $10 = "from auth regex == " #+END_SRC ****** DONE from mail-from sender | :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:20] :END: ****** DONE from mail-from regexp :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:20] :END: ****** DONE from rdns :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:20] :END: ****** DONE from rdns regex :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:20] :END: ****** DONE from src
:LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:21] :END: ****** DONE from src regex
:LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:21] :END: ****** TODO do some more sanitizing of these last couple of options There may be some way to specify invalid data. For example: tls does not support regex, not, or value fields. The below code should be an error. #+BEGIN_SRC scheme (display (opensmtpd-match-configuration->string (opensmtpd-match-configuration (tls (opensmtpd-option-configuration (option "tls") ;; this should be auth!!! NOT "helo" (regex #t) (not #t) (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "kitten")))))) (from (opensmtpd-option-configuration (option "from rdns") (value (opensmtpd-table-configuration (name "table") (data (list "cat" "hat")))))) (action (opensmtpd-action-local-delivery-configuration (name "matches")))))) match from rdns
! tls regex action "matches" #+END_SRC ****** DONE auth :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:21] :END: Well I need to fix this bug. Basically, I can make an with a table of values for 'option' "auth". And I can print that table with (opensmtpd-option-configuration->string)...That works fine... But if I put that same record into an == ...for some reason that 'auth' table is not being printed. #+BEGIN_SRC scheme (opensmtpd-option-configuration (option "auth") (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "hat"))))) $20 = #< option: "auth" not: #f regex: #f value: #< name: "mytable" file-db: #f values: ("cat" "hat") type: #:860:40 (x)>>> scheme@(opensmtpd-records) [4]> (opensmtpd-option-configuration->string $20) $21 = "auth == " scheme@(opensmtpd-records) [4]> (opensmtpd-match (name "matches") (auth (opensmtpd-option-configuration (option "auth") (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "kitten")))))) (from (opensmtpd-option-configuration (option "from rdns") (value (opensmtpd-table-configuration (name "table") (data (list "cat" "hat")))))) (action (opensmtpd-action-local-delivery-configuration))) $22 = #< name: "matches" action: #< method: "mbox" alias: #f ttl: #f user: #f userbase: #f virtual: #f wrapper: #f> for: #f from: #< option: "from rdns" not: #f regex: #f value: #< name: "table" file-db: #f values: ("cat" "hat") type: #:876:89 (x)>>> auth: #< option: "auth" not: #f regex: #f value: #< name: "mytable" file-db: #f values: ("cat" "kitten") type: #:873:63 (x)>>> helo: #f mail-from: #f rcpt-to: #f tag: #f tls: #f> scheme@(opensmtpd-records) [5]> (opensmtpd-match-configuration->string $22) $23 = "match from rdns =
= auth action \"matches\" \n" ;; THERE IS SUPPOSED TO BE a "auth " here scheme@(opensmtpd-records) [5]> #+END_SRC ****** TODO [!] auth ****** TODO auth regex :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:21] :END: This does NOT show the regex for the auth option. or the table why? #+BEGIN_SRC scheme (opensmtpd-match-configuration (auth (opensmtpd-option-configuration (option "auth") ;; this should be auth!!! NOT "helo" (regex #t) (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "kitten")))))) (from (opensmtpd-option-configuration (option "from rdns") (value (opensmtpd-table-configuration (name "table") (data (list "cat" "hat")))))) (action (opensmtpd-action-local-delivery-configuration (name "matches")))) $9 = #< action: #< name: "matches" method: "mbox" alias: #f ttl: #f user: #f userbase: #f virtual: #f wrapper: #f> for: #f from: #< option: "from rdns" not: #f regex: #f value: #< name: "table" file-db: #f values: ("cat" "hat") type: #:144:52 (x)>>> auth: #< option: "auth" not: #f regex: #t value: #< name: "mytable" file-db: #f values: ("cat" "kitten") type: #:141:15 (x)>>> helo: #f mail-from: #f rcpt-to: #f tag: #f tls: #f> scheme@(opensmtpd-records) [4]> (display (opensmtpd-match-configuration->string $9)) match from rdns
auth action "matches" ;; there should be a regex in there. #+END_SRC ****** DONE helo :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:21] :END: #+BEGIN_SRC scheme (opensmtpd-match-configuration (helo (opensmtpd-option-configuration (option "helo") ;; this should be auth!!! NOT "helo" (regex #t) (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "kitten")))))) (from (opensmtpd-option-configuration (option "from rdns") (value (opensmtpd-table-configuration (name "table") (data (list "cat" "hat")))))) (action (opensmtpd-action-local-delivery-configuration (name "matches")))) $10 = #< action: #< name: "matches" method: "mbox" alias: #f ttl: #f user: #f userbase: #f virtual: #f wrapper: #f> for: #f from: #< option: "from rdns" not: #f regex: #f value: #< name: "table" file-db: #f values: ("cat" "hat") type: #:252:52 (x)>>> auth: #f helo: #< option: "helo" not: #f regex: #t value: #< name: "mytable" file-db: #f values: ("cat" "kitten") type: #:249:15 (x)>>> mail-from: #f rcpt-to: #f tag: #f tls: #f> scheme@(opensmtpd-records) [5]> (opensmtpd-match-configuration->string $10) $11 = "match from rdns
helo regex action \"matches\" \n" #+END_SRC ****** DONE mail-from :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:22] :END: #+BEGIN_SRC scheme (display (opensmtpd-match-configuration->string (opensmtpd-match-configuration (mail-from (opensmtpd-option-configuration (option "mail-from") ;; this should be auth!!! NOT "helo" (regex #t) (not #t) (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "kitten")))))) (from (opensmtpd-option-configuration (option "from rdns") (value (opensmtpd-table-configuration (name "table") (data (list "cat" "hat")))))) (action (opensmtpd-action-local-delivery-configuration (name "matches")))))) match from rdns
! mail-from regex action "matches" #+END_SRC ****** DONE mail-from regex :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:22] :END: #+BEGIN_SRC scheme (display (opensmtpd-match-configuration->string (opensmtpd-match-configuration (mail-from (opensmtpd-option-configuration (option "mail-from") ;; this should be auth!!! NOT "helo" (regex #t) (not #t) (value (opensmtpd-table-configuration (name "mytable") (data (list "cat" "kitten")))))) (from (opensmtpd-option-configuration (option "from rdns") (value (opensmtpd-table-configuration (name "table") (data (list "cat" "hat")))))) (action (opensmtpd-action-local-delivery-configuration (name "matches")))))) match from rdns
! mail-from regex action "matches" #+END_SRC ****** DONE rcpt-to :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:22] :END: ****** DONE rcpt-to regex :LOGBOOK: - State "TODO" from "TODO" [2021-11-02 Tue 04:23] :END: *** PROJ sanitize the == records in == & == **** PROJ testing the sanitize-list-of-options-for-match-configuration-assoc precodure [5/5] ***** DONE make sure each option is unique (no duplicate "for"s). #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (options (list (opensmtpd-option-configuration (option "for any")) (opensmtpd-option-configuration (option "for local")))) (action (opensmtpd-action-relay-configuration (name "relay"))))))) #+END_SRC ***** DONE make sure there is no duplicate from's #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (options (list (opensmtpd-option-configuration (option "from any")) (opensmtpd-option-configuration (option "from auth")))) (action (opensmtpd-action-relay-configuration (name "relay"))))))) #+END_SRC ***** DONE for any data and regex must be false #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (options (list (opensmtpd-option-configuration (option "for any") (regex #t)))) (action (opensmtpd-action-relay-configuration (name "relay"))))))) #+END_SRC ***** DONE 'rcpt-to' must have data #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (options (list (opensmtpd-option-configuration (option "rcpt-to")))) (action (opensmtpd-action-relay-configuration (name "relay"))))))) #+END_SRC ***** DONE 'tls' cannot have a 'data' or 'regex' #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (options (list (opensmtpd-option-configuration (option "tls") (data "hello") ))) (action (opensmtpd-action-relay-configuration (name "relay"))))))) #+END_SRC *** TODO sanitize fieldname 'matches' so that no two unique actions have the same name I definitely should sanitize 'matches' a bit more. For example, you could have two different actions, one for local delivery and one for remote, with the same name. I should make sure that all unique actions have unique names. Here is an example of two actions that have the same name, but different ttl values: #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (action (opensmtpd-action-local-delivery-configuration (name "my-local-delivery") (ttl "50m")))) ; 50 minutes (opensmtpd-match-configuration (action (opensmtpd-action-local-delivery-configuration (name "my-local-delivery") (ttl "50h"))))))) ; 50 hours #+END_SRC *** PROJ follow the style guide and style up my project [0/4] https://mumble.net/~campbell/scheme/style.txt :SchemeStyleGuide: #+BEGIN_SRC org Riastradh's Lisp Style Rules -*- outline -*- Copyright (C) 2007--2011 Taylor R. Campbell CC BY-NC-SA 3.0 This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License: . This is a guide to Lisp style, written by Taylor R. Campbell, to describe the standard rules of Lisp style as well as a set of more stringent rules for his own style. This guide should be useful for Lisp in general, but there are [or will be in the final draft] parts that are focussed on or specific to Scheme or Common Lisp. This guide is written primarily as a collection of rules, with rationale for each rule. (If a rule is missing rationale, please inform the author!) Although a casual reader might go through and read the rules without the rationale, perhaps reasoning that reading of the rationale would require more time than is available, such a reader would derive little value from this guide. In order to apply the rules meaningfully, their spirit must be understood; the letter of the rules serves only to hint at the spirit. The rationale is just as important as the rules. There are many references in this document to `Emacs', `GNU Emacs', `Edwin', and so on. In this document, `Emacs' means any of a class of editors related in design to a common ancestor, the EMACS editor macros written for TECO on ITS on the PDP-10 in the middle of the nineteen seventies. All such editors -- or `all Emacsen', since `Emacsen' is the plural of `Emacs' -- have many traits in common, such as a very consistent set of key bindings, extensibility in Lisp, and so on. `GNU Emacs' means the member of the class of editors collectively known as Emacsen that was written for the GNU Project in the middle of the nineteen eighties, and which is today probably the most popular Emacs. `Edwin' is MIT Scheme's Emacs, which is bundled as part of MIT Scheme, and not available separately. There are other Emacsen as well, such as Hemlock and Climacs, but as the author of this document has little experience with Emacsen other than GNU Emacs and Edwin, there is little mention of other Emacsen. This guide is a work in progress. To be written: - Indentation rules for various special operators. - Philosophical rambling concerning naming. - Rules for breaking lines. - Many more examples. - A more cohesive explanation of the author's principles for composing programs, and their implications. - Rules for writing portable code. - Some thoughts concerning extensions to the lexical syntax. - Rules for writing or avoiding macros. - Some unfinished rationale. - More on documentation. - The `Dependencies' subsection of the `General Layout' section should be put in a different section, the rest of which has yet to be written, on organization of programs, module systems, and portable code. Feedback is welcome; address any feedback by email to the host mumble.net's user `campbell', or by IRC to Riastradh in the #scheme channel on Freenode (irc.freenode.net). Feedback includes reports of typos, questions, requests for clarification, and responses to the rationale, except in the case of round brackets versus square brackets, the argument surrounding which is supremely uninteresting and now not merely a dead horse but a rotting carcass buzzing with flies and being picked apart by vultures. As this document has grown, the line between standard Lisp rules and the author's own style has been blurred. The author is considering merging of the partition, but has not yet decided on this with certainty. Opinions on the subject are welcome -- is the partition still useful to keep the author's biases and idiosyncrasies out of the standard rules, or has the partition with its arbitrary nature only caused disorganization of the whole document? Unfortunately, this document is entirely unscientific. It is at best a superstition or philosophy, but one that the author of this document has found to have improved his programs. Furthermore, the author is somewhat skeptical of claims of scientific analyses of these matters: analyzing human behaviour, especially confined to the set of excellent programmers who often have strong opinions about their methods for building programs, is a very tricky task. ,* Standard Rules These are the standard rules for formatting Lisp code; they are repeated here for completeness, although they are surely described elsewhere. These are the rules implemented in Emacs Lisp modes, and auxiliary utilities such as Paredit. The rationale given here is merely the author's own speculation of the origin of these rules, and should be taken as nothing more than it. The reader shall, irrespective of the author's rationale, accept the rules as sent by the reader's favourite deity, or Cthulhu if no such deity strikes adequate fear into the heart of the reader. ,** Parentheses ,*** Terminology This guide avoids the term /parenthesis/ except in the general use of /parentheses/ or /parenthesized/, because the word's generally accepted definition, outside of the programming language, is a statement whose meaning is peripheral to the sentence in which it occurs, and *not* the typographical symbols used to delimit such statements. The balanced pair of typographical symbols that mark parentheses in English text are /round brackets/, i.e. ( and ). There are several other balanced pairs of typographical symbols, such as /square brackets/ (commonly called simply `brackets' in programming circles), i.e. [ and ]; /curly braces/ (sometimes called simply `braces'), i.e. { and }; /angle brackets/ (sometimes `brokets' (for `broken brackets')), i.e. < and >. In any balanced pair of typographical symbols, the symbol that begins the region delimited by the symbols is called the /opening bracket/ or the /left bracket/, such as ( or [ or { or <. The symbol that ends that region is called the /right bracket/ or the /closing bracket/, such as > or } or ] or ). ,*** Spacing If any text precedes an opening bracket or follows a closing bracket, separate that text from that bracket with a space. Conversely, leave no space after an opening bracket and before following text, or after preceding text and before a closing bracket. Unacceptable: (foo(bar baz)quux) (foo ( bar baz ) quux) Acceptable: (foo (bar baz) quux) Rationale: This is the same spacing found in standard typography of European text. It is more aesthetically pleasing. ,*** Line Separation Absolutely do *not* place closing brackets on their own lines. Unacceptable: (define (factorial x) (if (< x 2) 1 (* x (factorial (- x 1 ) ) ) ) ) Acceptable: (define (factorial x) (if (< x 2) 1 (* x (factorial (- x 1))))) Rationale: The parentheses grow lonely if their closing brackets are all kept separated and segregated. ,**** Exceptions to the Above Rule Concerning Line Separation Do not heed this section unless you know what you are doing. Its title does *not* make the unacceptable example above acceptable. When commenting out fragments of expressions with line comments, it may be necessary to break a line before a sequence of closing brackets: (define (foo bar) (list (frob bar) (zork bar) ;; (zap bar) )) This is acceptable, but there are other alternatives. In Common Lisp, one can use the read-time optional syntax, `#+' or `#-', with a feature optional that is guaranteed to be false or true -- `#+(OR)' or `#-(AND)' --; for example, (define (foo bar) (list (frob bar) (zork bar) ,#+(or) (zap bar))). Read-time optionals are expression-oriented, not line-oriented, so the closing brackets need not be placed on the following line. Some Scheme implementations, and SRFI 62, also support expression comments with `#;', which are operationally equivalent to the above read-time optionals for Common Lisp: (define (foo bar) (list (frob bar) (zork bar) #; (zap bar))) The expression is placed on another line in order to avoid confusing editors that do not recognize S-expression comments; see the section titled `Comments' below for more details. However, the `#;' notation is not standard -- it appears in neither the IEEE 1178 document nor in the R5RS --, so line comments are preferable for portable Scheme code, even if they require breaking a line before a sequence of closing brackets. Finally, it is acceptable to break a line immediately after an opening bracket and immediately before a closing bracket for very long lists, especially in files under version control. This eases the maintenance of the lists and clarifies version diffs. Example: (define colour-names ;Add more colour names to this list! '( blue cerulean green magenta purple red scarlet turquoise )) ,*** Parenthetical Philosophy The actual bracket characters are simply lexical tokens to which little significance should be assigned. Lisp programmers do not examine the brackets individually, or, Azathoth forbid, count brackets; instead they view the higher-level structures expressed in the program, especially as presented by the indentation. Lisp is not about writing a sequence of serial instructions; it is about building complex structures by summing parts. The composition of complex structures from parts is the focus of Lisp programs, and it should be readily apparent from the Lisp code. Placing brackets haphazardly about the presentation is jarring to a Lisp programmer, who otherwise would not even have seen them for the most part. ,** Indentation and Alignment The operator of any form, i.e. the first subform following the opening round bracket, determines the rules for indenting or aligning the remaining forms. Many names in this position indicate special alignment or indentation rules; these are special operators, macros, or procedures that have certain parameter structures. If the first subform is a non-special name, however, then if the second subform is on the same line, align the starting column of all following subforms with that of the second subform. If the second subform is on the following line, align its starting column with that of the first subform, and do the same for all remaining subforms. In general, Emacs will indent Lisp code correctly. Run `C-M-q' (indent-sexp) on any code to ensure that it is indented correctly, and configure Emacs so that any non-standard forms are indented appropriately. Unacceptable: (+ (sqrt -1) (* x y) (+ p q)) (+ (sqrt -1) (* x y) (+ p q)) Acceptable: (+ (sqrt -1) (* x y) (+ p q)) (+ (sqrt -1) (* x y) (+ p q)) Rationale: The columnar alignment allows the reader to follow the operands of any operation straightforwardly, simply by scanning downward or upward to match a common column. Indentation dictates structure; confusing indentation is a burden on the reader who wishes to derive structure without matching parentheses manually. ,*** Non-Symbol Indentation and Alignment The above rules are not exhaustive; some cases may arise with strange data in operator positions. ,**** Lists Unfortunately, style varies here from person to person and from editor to editor. Here are some examples of possible ways to indent lists whose operators are lists: Questionable: ((car x) ;Requires hand indentation. (cdr x) foo) ((car x) (cdr x) ;GNU Emacs foo) Preferable: ((car x) ;Any Emacs (cdr x) foo) ((car x) (cdr x) ;Edwin foo) Rationale: The operands should be aligned, as if it were any other procedure call with a name in the operator position; anything other than this is confusing because it gives some operands greater visual distinction, allowing others to hide from the viewer's sight. For example, the questionable indentation ((car x) (cdr x) foo) can make it hard to see that FOO and (CDR X) are both operands here at the same level. However, GNU Emacs will generate that indentation by default. (Edwin will not.) ,**** Strings If the form in question is meant to be simply a list of literal data, all of the subforms should be aligned to the same column, irrespective of the first subform. Unacceptable: ("foo" "bar" "baz" "quux" "zot" "mumble" "frotz" "gargle" "mumph") Questionable, but acceptable: (3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 2 3 8 4 6 2 6 4 3 3 8 3 2 7 9 5 0 2 8 8 4 1 9 7 1 6 9 3 9 9 3) Acceptable: ("foo" "bar" "baz" "quux" "zot" "mumble" "frotz" "gargle" "mumph") ("foo" "bar" "baz" "quux" "zot" "mumble" "frotz" "gargle" "mumph") Rationale: Seldom is the first subform distinguished for any reason, if it is a literal; usually in this case it indicates pure data, not code. Some editors and pretty-printers, however, will indent unacceptably in the example given unless the second subform is on the next line anyway, which is why the last way to write the fragment is usually best. ,** Names Naming is subtle and elusive. Bizarrely, it is simultaneously insignificant, because an object is independent of and unaffected by the many names by which we refer to it, and also of supreme importance, because it is what programming -- and, indeed, almost everything that we humans deal with -- is all about. A full discussion of the concept of name lies far outside the scope of this document, and could surely fill not even a book but a library. Symbolic names are written with English words separated by hyphens. Scheme and Common Lisp both fold the case of names in programs; consequently, camel case is frowned upon, and not merely because it is ugly. Underscores are unacceptable separators except for names that were derived directly from a foreign language without translation. Unacceptable: XMLHttpRequest foreach append_map Acceptable: xml-http-request for-each append-map ,*** Funny Characters There are several different conventions in different Lisps for the use of non-alphanumeric characters in names. ,**** Scheme ,***** Question Marks: Predicates Affix a question mark to the end of a name for a procedure whose purpose is to ask a question of an object and to yield a boolean answer. Such procedures are called `predicates'. Do not use a question mark if the procedure may return any object other than a boolean. Examples: pair? procedure? proper-list? Non-examples: member assoc any every Pronounce the question mark as if it were the isolated letter `p'. For example, to read the fragment (PAIR? OBJECT) aloud, say: `pair-pee object.' ,***** Exclamation Marks: Destructive Operations Affix an exclamation mark to the end of a name for a procedure (or macro) whose primary purpose is to modify an object. Such procedures are called `destructive'. Examples: set-car! append! Avoid using the exclamation mark willy nilly for just *any* procedure whose operation involves any kind of mutation or side effect; instead, use the exclamation mark to identify procedures that exist *solely* for the purpose of destructive update (e.g., SET-CAR!), or to distinguish a destructive, or potentially destructive (in the case of linear-update operations such as APPEND!), variant of a procedure of which there also exists a purely functional variant (e.g., APPEND). Pronounce the exclamation mark as `bang'. For example, to read the fragment (APPEND! LIST TAIL) aloud, say: `append-bang list tail.' ,***** Asterisks: Variants, Internal Routines, Mutable Globals Affix an asterisk to the end of a name to make a variation on a theme of the original name. Example: let -> let* Prefer a meaningful name over an asterisk; the asterisk does not explain what variation on the theme the name means. Affix an asterisk to the beginning of a name to make an internal routine for that name. Again, prefer a meaningful name over an asterisk. Affix asterisks to the beginning and end of a globally mutable variable. This allows the reader of the program to recognize very easily that it is badly written! ,***** `WITH-' and `CALL-WITH-': Dynamic State and Cleanup Prefix `WITH-' to any procedure that establishes dynamic state and calls a nullary procedure, which should be the last (required) argument. The dynamic state should be established for the extent of the nullary procedure, and should be returned to its original state after that procedure returns. Examples: with-input-from-file with-output-to-file Exception: Some systems provide a procedure (WITH-CONTINUATION ), which calls in the given continuation, using that continuation's dynamic state. If returns, it will return to , not to the continuation of the call to WITH-CONTINUATION. This is acceptable. Prefix `CALL-WITH-' to any procedure that calls a procedure, which should be its last argument, with some arguments, and is either somehow dependent upon the dynamic state or continuation of the program, or will perform some action to clean up data after the procedure argument returns. Generally, `CALL-WITH-' procedures should return the values that the procedure argument returns, after performing the cleaning action. Examples: - CALL-WITH-INPUT-FILE and CALL-WITH-OUTPUT-FILE both accept a pathname and a procedure as an argument, open that pathname (for input or output, respectively), and call the procedure with one argument, a port corresponding with the file named by the given pathname. After the procedure returns, CALL-WITH-INPUT-FILE and CALL-WITH-OUTPUT-FILE close the file that they opened, and return whatever the procedure returned. - CALL-WITH-CURRENT-CONTINUATION is dependent on the continuation with which it was called, and passes as an argument an escape procedure corresponding with that continuation. - CALL-WITH-OUTPUT-STRING, a common but non-standard procedure definable in terms of OPEN-OUTPUT-STRING and GET-OUTPUT-STRING from SRFI 6 (Basic String Ports), calls its procedure argument with an output port, and returns a string of all of the output written to that port. Note that it does not return what the procedure argument returns, which is an exception to the above rule. Generally, the distinction between these two classes of procedures is that `CALL-WITH-...' procedures should not establish fresh dynamic state and instead pass explicit arguments to their procedure arguments, whereas `WITH-...' should do the opposite and establish dynamic state while passing zero arguments to their procedure arguments. ,** Comments Write heading comments with at least four semicolons; see also the section below titled `Outline Headings'. Write top-level comments with three semicolons. Write comments on a particular fragment of code before that fragment and aligned with it, using two semicolons. Write margin comments with one semicolon. The only comments in which omission of a space between the semicolon and the text is acceptable are margin comments. Examples: ;;;; Frob Grovel ;;; This section of code has some important implications: ;;; 1. Foo. ;;; 2. Bar. ;;; 3. Baz. (define (fnord zarquon) ;; If zob, then veeblefitz. (quux zot mumble ;Zibblefrotz. frotz)) ,* Riastradh's Non-Standard Rules Three principles guide this style, roughly ordered according to descending importance: 1. The purpose of a program is to describe an idea, and not the way that the idea must be realized; the intent of the program's meaning, rather than peripheral details that are irrelevant to its intent, should be the focus of the program, *irrespective* of whether a human or a machine is reading it. [It would be nice to express this principle more concisely.] 2. The sum of the parts is easier to understand than the whole. 3. Aesthetics matters. No one enjoys reading an ugly program. ,** General Layout This section contains rules that the author has found generally helpful in keeping his programs clean and presentable, though they are not especially philosophically interesting. Contained in the rationale for some of the following rules are references to historical limitations of terminals and printers, which are now considered aging cruft of no further relevance to today's computers. Such references are made only to explain specific measures chosen for some of the rules, such as a limit of eighty columns per line, or sixty-six lines per page. There is a real reason for each of the rules, and this real reason is not intrinsically related to the historical measures, which are mentioned only for the sake of providing some arbitrary measure for the limit. ,*** File Length If a file exceeds five hundred twelve lines, begin to consider splitting it into multiple files. Do not write program files that exceed one thousand twenty-four lines. Write a concise but descriptive title at the top of each file, and include no content in the file that is unrelated to its title. Rationale: Files that are any larger should generally be factored into smaller parts. (One thousand twenty-four is a nicer number than one thousand.) Identifying the purpose of the file helps to break it into parts if necessary and to ensure that nothing unrelated is included accidentally. ,*** Top-Level Form Length Do not write top-level forms that exceed twenty-one lines, except for top-level forms that serve only the purpose of listing large sets of data. If a procedure exceeds this length, split it apart and give names to its parts. Avoid names formed simply by appending a number to the original procedure's name; give meaningful names to the parts. Rationale: Top-level forms, especially procedure definitions, that exceed this length usually combine too many concepts under one name. Readers of the code are likely to more easily understand the code if it is composed of separately named parts. Simply appending a number to the original procedure's name can help only the letter of the rule, not the spirit, however, even if the procedure was taken from a standard algorithm description. Using comments to mark the code with its corresponding place in the algorithm's description is acceptable, but the algorithm should be split up in meaningful fragments anyway. Rationale for the number twenty-one: Twenty-one lines, at a maximum of eighty columns per line, fits in a GNU Emacs instance running in a 24x80 terminal. Although the terminal may have twenty-four lines, three of the lines are occupied by GNU Emacs: one for the menu bar (which the author of this guide never uses, but which occupies a line nevertheless in a vanilla GNU Emacs installation), one for the mode line, and one for the minibuffer's window. The writer of some code may not be limited to such a terminal, but the author of this style guide often finds it helpful to have at least four such terminals or Emacs windows open simultaneously, spread across a twelve-inch laptop screen, to view multiple code fragments. ,*** Line Length Do not write lines that exceed eighty columns, or if possible seventy-two. Rationale: Following multiple lines that span more columns is difficult for humans, who must remember the line of focus and scan right to left from the end of the previous line to the beginning of the next line; the more columns there are, the harder this is to do. Sticking to a fixed limit helps to improve readability. Rationale for the numbers eighty and seventy-two: It is true that we have very wide screens these days, and we are no longer limited to eighty-column terminals; however, we ought to exploit our wide screens not by writing long lines, but by viewing multiple fragments of code in parallel, something that the author of this guide does very often. Seventy-two columns leave room for several nested layers of quotation in email messages before the code reaches eighty columns. Also, a fixed column limit yields nicer printed output, especially in conjunction with pagination; see the section `Pagination' below. ,*** Blank Lines Separate each adjacent top-level form with a single blank line (i.e. two line breaks). If two blank lines seem more appropriate, break the page instead. Do not place blank lines in the middle of a procedure body, except to separate internal definitions; if there is a blank line for any other reason, split the top-level form up into multiple ones. Rationale: More than one blank line is distracting and sloppy. If the two concepts that are separated by multiple blank lines are really so distinct that such a wide separator is warranted, then they are probably better placed on separate pages anyway; see the next section, `Pagination'. ,*** Pagination Separate each file into pages of no more than sixty-six lines and no fewer than forty lines with form feeds (ASCII #x0C, or ^L, written in Emacs with `C-q C-l'), on either side of which is a single line break (but not a blank line). Rationale: Keeping distinct concepts laid out on separate pages helps to keep them straight. This is helpful not only for the writer of the code, but also for the reader. It also allows readers of the code to print it onto paper without fiddling with printer settings to permit pages of more than sixty-six lines (which is the default number for many printers), and pagination also makes the code easier to navigate in Emacs, with the `C-x [' and `C-x ]' keys (`backward-page' and `forward-page', respectively). To avoid excessively small increments of page-by-page navigation, and to avoid wasting paper, each page should generally exceed forty lines. `C-x l' in Emacs will report the number of lines in the page on which the point lies; this is useful for finding where pagination is necessary. ,*** Outline Headings Use Emacs's Outline Mode to give titles to the pages, and if appropriate a hierarchical structure. Set `outline-regexp' (or `outline-pattern' in Edwin) to "\f\n;;;;+ ", so that each form feed followed by an line break followed by at least four semicolons and a space indicates an outline heading to Emacs. Use four semicolons for the highest level of headings in the hierarchy, and one more for each successively nested level of hierarchy. Rationale: Not only does this clarify the organization of the code, but readers of the code can then navigate the code's structure with Outline Mode commands such as `C-c C-f', `C-c C-b', `C-c C-u', and `C-c C-d' (forward, backward, up, down, respectively, headings). ,*** Dependencies When writing a file or module, minimize its dependencies. If there are too many dependencies, consider breaking the module up into several parts, and writing another module that is the sum of the parts and that depends only on the parts, not their dependencies. Rationale: A fragment of a program with fewer dependencies is less of a burden on the reader's cognition. The reader can more easily understand the fragment in isolation; humans are very good at local analyses, and terrible at global ones. ,** Naming This section requires an elaborate philosophical discussion which the author is too ill to have the energy to write at this moment. Compose concise but meaningful names. Do not cheat by abbreviating words or using contractions. Rationale: Abbreviating words in names does not make them shorter; it only makes them occupy less screen space. The reader still must understand the whole long name. This does not mean, however, that names should necessarily be long; they should be descriptive. Some long names are more descriptive than some short names, but there are also descriptive names that are not long and long names that are not descriptive. Here is an example of a long name that is not descriptive, from SchMUSE, a multi-user simulation environment written in MIT Scheme: frisk-descriptor-recursive-subexpr-descender-for-frisk-descr-env Not only is it long (sixty-four characters) and completely impenetrable, but halfway through its author decided to abbreviate some words as well! Do not write single-letter variable names. Give local variables meaningful names composed from complete English words. Rationale: It is tempting to reason that local variables are invisible to other code, so it is OK to be messy with their names. This is faulty reasoning: although the next person to come along and use a library may not care about anything but the top-level definitions that it exports, this is not the only audience of the code. Someone will also want to read the code later on, and if it is full of impenetrably terse variable names without meaning, that someone will have a hard time reading the code. Give names to intermediate values where their expressions do not adequately describe them. Rationale: An `expression' is a term that expresses some value. Although a machine needs no higher meaning for this value, and although it should be written to be sufficiently clear for a human to understand what it means, the expression might mean something more than just what it says where it is used. Consequently, it is helpful for humans to see names given to expressions. Example: A hash table HASH-TABLE maps foos to bars; (HASH-TABLE/GET HASH-TABLE FOO #F) expresses the datum that HASH-TABLE maps FOO to, but that expression gives the reader no hint of any information concerning that datum. (LET ((BAR (HASH-TABLE/GET HASH-TABLE FOO #F))) ...) gives a helpful name for the reader to understand the code without having to find the definition of HASH-TABLE. Index variables such as i and j, or variables such as A and D naming the car and cdr of a pair, are acceptable only if they are completely unambiguous in the scope. For example, (do ((i 0 (+ i 1))) ((= i (vector-length vector))) (frobnicate (vector-ref vector i))) is acceptable because the scope of i is very clearly limited to a single vector. However, if more vectors are involved, using more index variables such as j and k will obscure the program further. Avoid functional combinators, or, worse, the point-free (or `point-less') style of code that is popular in the Haskell world. At most, use function composition only where the composition of functions is the crux of the idea being expressed, rather than simply a procedure that happens to be a composition of two others. Rationale: Tempting as it may be to recognize patterns that can be structured as combinations of functional combinators -- say, `compose this procedure with the projection of the second argument of that other one', or (COMPOSE FOO (PROJECT 2 BAR)) --, the reader of the code must subsequently examine the elaborate structure that has been built up to obscure the underlying purpose. The previous fragment could have been written (LAMBDA (A B) (FOO (BAR B))), which is in fact shorter, and which tells the reader directly what argument is being passed on to what, and what argument is being ignored, without forcing the reader to search for the definitions of FOO and BAR or the call site of the final composition. The explicit fragment contains substantially more information when intermediate values are named, which is very helpful for understanding it and especially for modifying it later on. The screen space that can be potentially saved by using functional combinators is made up for by the cognitive effort on the part of the reader. The reader should not be asked to search globally for usage sites in order to understand a local fragment. Only if the structure of the composition really is central to the point of the narrative should it be written as such. For example, in a symbolic integrator or differentiator, composition is an important concept, but in most code the structure of the composition is completely irrelevant to the real point of the code. If a parameter is ignored, give it a meaningful name nevertheless and say that it is ignored; do not simply call it `ignored'. In Common Lisp, variables can be ignored with (DECLARE (IGNORE ...)). Some Scheme systems have similar declarations, but the portable way to ignore variables is just to write them in a command context, where their values will be discarded, preferably with a comment indicating this purpose: (define (foo x y z) x z ;ignore (frobnitz y)) Rationale: As with using functional combinators to hide names, avoiding meaningful names for ignored parameters only obscures the purpose of the program. It is helpful for a reader to understand what parameters a procedure is independent of, or if someone wishes to change the procedure later on, it is helpful to know what other parameters are available. If the ignored parameters were named meaninglessly, then these people would be forced to search for call sites of the procedure in order to get a rough idea of what parameters might be passed here. When naming top-level bindings, assume namespace partitions unless in a context where they are certain to be absent. Do not write explicit namespace prefixes, such as FOO:BAR for an operation BAR in a module FOO, unless the names will be used in a context known not to have any kind of namespace partitions. Rationale: Explicit namespace prefixes are ugly, and lengthen names without adding much semantic content. Common Lisp has its package system to separate the namespaces of symbols; most Schemes have mechanisms to do so as well, even if the RnRS do not specify any. It is better to write clear names which can be disambiguated if necessary, rather than to write names that assume some kind of disambiguation to be necessary to begin with. Furthermore, explicit namespace prefixes are inadequate to cover name clashes anyway: someone else might choose the same namespace prefix. Relegating this issue to a module system removes it from the content of the program, where it is uninteresting. ,** Comments Write comments only where the code is incapable of explaining itself. Prefer self-explanatory code over explanatory comments. Avoid `literate programming' like the plague. Rationale: If the code is often incapable of explaining itself, then perhaps it should be written in a more expressive language. This may mean using a different programming language altogether, or, since we are talking about Lisp, it may mean simply building a combinator language or a macro language for the purpose. `Literate programming' is the logical conclusion of languages incapable of explaining themselves; it is a direct concession of the inexpressiveness of the computer language implementing the program, to the extent that the only way a human can understand the program is by having it rewritten in a human language. Do not write interface documentation in the comments for the implementation of the interface. Explain the interface at the top of the file if it is a single-file library, or put that documentation in another file altogether. (See the `Documentation' section below if the interface documentation comments grow too large for a file.) Rationale: A reader who is interested only in the interface really should not need to read through the implementation to pick out its interface; by putting the interface documentation at the top, not only is such a reader's task of identifying the interface made easier, but the implementation code can be more liberally commented without fear of distracting this reader. To a reader who is interested in the implementation as well, the interface is still useful in order to understand what concepts the implementation is implementing. Example: In this example of a single-file library implementing the skip list data structure, the first page explains the purpose and dependencies of the file (which are useful for anyone who intends to use it, even though dependencies are really implementation details), and the next few pages explain the usage of skip lists as implemented in that file. On the first page of implementation, `Skip List Structure', there are some comments of interest only to a reader who wishes to understand the implementation; the same goes for the rest of the file, none of which must a reader read whose interest is only in the usage of the library. Avoid block comments (i.e. #| ... |#). Use S-expression comments (`#;' in Scheme, with the expression to comment on the next line; `#+(OR)' or `#-(AND)' in Common Lisp) to comment out whole expressions. Use blocks of line comments for text. Rationale: Editor support for block comments is weak, because it requires keeping a detailed intermediate parse state of the whole buffer, which most Emacsen do not do. At the very least, #|| ... ||# is better, because most Emacsen will see vertical bars as symbol delimiters, and lose trying to read a very, very long symbol, if they try to parse #| ... |#, whereas they will just see two empty symbols and otherwise innocuous text between them if they try to parse #|| ... ||#. In any case, in Emacs, `M-x comment-region RET', or `M-;' (comment-dwim), is trivial to type. The only standard comments in Scheme are line comments. There are SRFIs for block comments and S-expression comments, but support for them varies from system to system. Expression comments are not hard for editors to deal with because it is safe not to deal with them at all; however, in Scheme S-expression comments, which are written by prefixing an expression with `#;', the expression to be commented should be placed on the next line. This is because editors that do not deal with them at all may see the semicolon as the start of a line comment, which will throw them off. Expression comments in Common Lisp, however, are always safe. In Common Lisp, the two read-time optionals that are guaranteed to ignore any form following them are `#+(OR)' and `#-(AND)'. `#+NIL' is sometimes used in their stead, but, while it may appear to be an obviously false optional, it actually is not. The feature expressions are read in the KEYWORD package, so NIL is read not as CL:NIL, i.e. the boolean false value, but as :NIL, a keyword symbol whose name happens to be `NIL'. Not only is it not read as the boolean false value, but it has historically been used to indicate a feature that might be enabled -- in JonL White's New Implementation of Lisp! However, the New Implementation of Lisp is rather old these days, and unlikely to matter much...until Alastair Bridgewater writes Nyef's Implementation of Lisp. ,** Documentation On-line references and documentation/manuals are both useful for independent purposes, but there is a very fine distinction between them. Do not generate documentation or manuals automatically from the text of on-line references. Rationale: /On-line references/ are quick blurbs associated with objects in a running Lisp image, such as documentation strings in Common Lisp or Emacs Lisp. These assume that the reader is familiar with the gist of the surrounding context, but unclear on details; on-line references specify the details of individual objects. /Documentation/ and /manuals/ are fuller, organized, and cohesive documents that explain the surrounding context to readers who are unfamiliar with it. A reader should be able to pick a manual up and begin reading it at some definite point, perusing it linearly to acquire an understanding of the subject. Although manuals may be dominated by reference sections, they should still have sections that are linearly readable to acquaint the reader with context. ,** Round and Square Brackets Some implementations of Scheme provide a non-standard extension of the lexical syntax whereby balanced pairs of square brackets are semantically indistinguishable from balanced pairs of round brackets. Do not use this extension. Rationale: Because this is a non-standard extension, it creates inherently non-portable code, of a nature much worse than using a name in the program which is not defined by the R5RS. The reason that we have distinct typographical symbols in the first place is to express different meaning. The only distinction between round brackets and square brackets is in convention, but the precise nature of the convention is not specified by proponents of square brackets, who suggest that they be used for `clauses', or for forms that are parts of enclosing forms. This would lead to such constructions as (let [(x 5) (y 3)] ...) or (let ([x 5] [y 3]) ...) or (let [[x 5] [y 3]] ...), the first two of which the author of this guide has seen both of, and the last of which does nothing to help to distinguish the parentheses anyway. The reader of the code should not be forced to stumble over a semantic identity because it is expressed by a syntactic distinction. The reader's focus should not be directed toward the lexical tokens; it should be directed toward the structure, but using square brackets draws the reader's attention unnecessarily to the lexical tokens. ,* Attribution #+END_SRC :END: **** TODO I have to get change (let ([x 5] [y 3])) -> (let ((x 5) (y 3))) **** TODO comments #+BEGIN_SRC scheme ;;;; Frob Grovel ;;; This section of code has some important implications: ;;; 1. Foo. ;;; 2. Bar. ;;; 3. Baz. (define (fnord zarquon) ;; If zob, then veeblefitz. (quux zot mumble ;Zibblefrotz. frotz)) #+END_SRC **** TODO literal data Strings If the form in question is meant to be simply a list of literal data, all of the subforms should be aligned to the same column, irrespective of the first subform. Unacceptable: ("foo" "bar" "baz" "quux" "zot" "mumble" "frotz" "gargle" "mumph") Questionable, but acceptable: (3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 2 3 8 4 6 2 6 4 3 3 8 3 2 7 9 5 0 2 8 8 4 1 9 7 1 6 9 3 9 9 3) Acceptable: ("foo" "bar" "baz" "quux" "zot" "mumble" "frotz" "gargle" "mumph") ("foo" "bar" "baz" "quux" "zot" "mumble" "frotz" "gargle" "mumph") **** TODO follow this syntax convention proc args :alignment: The operator of any form, i.e. the first subform following the opening round bracket, determines the rules for indenting or aligning the remaining forms. Many names in this position indicate special alignment or indentation rules; these are special operators, macros, or procedures that have certain parameter structures. If the first subform is a non-special name, however, then if the second subform is on the same line, align the starting column of all following subforms with that of the second subform. If the second subform is on the following line, align its starting column with that of the first subform, and do the same for all remaining subforms. In general, Emacs will indent Lisp code correctly. Run `C-M-q' (indent-sexp) on any code to ensure that it is indented correctly, and configure Emacs so that any non-standard forms are indented appropriately. Unacceptable: (+ (sqrt -1) (* x y) (+ p q)) (+ (sqrt -1) (* x y) (+ p q)) Acceptable: (+ (sqrt -1) (* x y) (+ p q)) (+ (sqrt -1) (* x y) (+ p q)) Rationale: The columnar alignment allows the reader to follow the operands of any operation straightforwardly, simply by scanning downward or upward to match a common column. Indentation dictates structure; confusing indentation is a burden on the reader who wishes to derive structure without matching parentheses manually. :END: If you have a procedure, then it's arguments should be on the same line. #+BEGIN_SRC scheme (proc args (proc (proc args) (proc args) (proc (proc args) (proc (proc (proc (proc args)))))) + (proc (proc args))) #+END_SRC *** TODO write various tests for == I have many bits of code in opensmtpd.org.archive that should result in an error. I should write some tests for this. ** NO should I modifiy some of the records to include a sanitize field? Probably not. It would be cool if this function ran automatically upon record initiation, but there's no to make it do that. ** TODO which sanitize function is better? Pick the better sanitize method and use that one. The sanitize function found in opensmtpd-listen-on-configuration-filters Or the sanitize function sanitize-list-of-options-for-match-configuration ? sanitize-list-of-options-for-match-configuration is probably faster. But is it? It is an iteratize loop that checks for all issues as it loops through the options. There is a lot of repetitive code in this procedure. BUT opensmtpd-listen-on-configuration-filters certainly seems easier to follow. ** TODO remove opensmtpd-table-type fieldname and instead move that it its own procedure outside of the record ONly use one function instead of ;; this procedure takes in one argument. ;; if that argument is an whose fieldname 'values' is an assoc-list, then it returns ;; #t, #f if otherwise. ;; TODO should I remove these two functions? And instead use the (opensmtpd-table-configuration-type) procedure? (define (table-whose-data-are-assoc-list? table) (if (not (opensmtpd-table-configuration? table)) #f (assoc-list? (opensmtpd-table-configuration-data table)))) ;; this procedure takes in one argument ;; if that argument is an whose fieldname 'values' is a list of strings, then it returns ;; #t, #f if otherwise. (define (table-whose-data-are-a-list-of-strings? table) (if (not (opensmtpd-table-configuration? table)) #f (list-of-strings? (opensmtpd-table-configuration-data table)))) And opensmtpd-table-type ** TODO OpenSMTPD Service documentation OpenSMTPD is an easy-to-use mail transfer agent (MTA). Its configuration file is throughly documented in man 5 =smtpd.conf=. OpenSMTPD *listens* for incoming mail and *matches* the mail to *actions*. The following records represent those stages: ~~, ~=, =~, ~~, and ~~. Additionally, each ~~ and ~~ may use a list of ~~, and/or ~~ records to filter email/spam. Also numerous records' fieldnames use ~~ to hold lists or key value pairs of data. A simple example configuration is below: #+BEGIN_SRC scheme (let ((smtp.gnu.org (opensmtpd-pki-configuration (domain "smtp.gnu.org") (cert "file.cert") (key "file.key")))) (service opensmtpd-service-type (opensmtpd-configuration (listen-ons (list (opensmtpd-listen-on-configuration (pki smtp.gnu.org)) (opensmtpd-listen-on-configuration (pki smtp.gnu.org) (secure-connection "smtps")))) (matches (list (opensmtpd-match-configuration (action (opensmtpd-action-local-delivery-configuration (name "local-delivery")))) (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "relay"))))))))) #+END_SRC - Scheme Variable: opensmtpd-service-type Service type for the OpenSMTPD ([[https://www.opensmtpd.org][https://www.opensmtpd.org]]) email server. The value for this service type is a ~~ record. - Data Type: opensmtpd-configuration Data type representing the configuration of OpenSMTPD. - ~package~ (default: ~opensmtpd~) The OpenSMTPD package to use. - ~config-file~ (default: ~#f~) File-like object of the OpenSMTPD configuration file to use. By default it listens on the loopback network interface, and allows for mail from users and daemons on the local machine, as well as permitting email to remote servers. Run ~man smtpd.conf~ for more information. - ~bounce~ (default: ~(list "4h")~) ~bounce~ is a list of strings, which send warning messages to the envelope sender when temporary delivery failures cause a message to remain in the queue for longer than string _delay_. Each string _delay_ parameter consists of a string beginning with a positive decimal integer and a unit s, m, h, or d. At most four delay parameters can be specified. - ~listen-ons~ (default: ~(list (opensmtpd-listen-on-configuration))~ ) ~listen-ons~ is a list of ~~ records. This list details what interfaces and ports OpenSMTPD listens on as well as other information. - ~listen-on-socket~ (default: ~(opensmtpd-listen-on-socket-configuration-configuration)~ ) Listens for incoming connections on the Unix domain socket. - ~includes~ (default: ~#f~) # TODO ~includes~ should support a list of string filenames or gexps. ~includes~ is a list of string _filenames_. Each filename's contents is additional configuration that is inserted into the top of the configuration file. - ~matches~ default: #+BEGIN_SRC scheme (list (opensmtpd-match-configuration (action (opensmtpd-action-local-delivery-configuration (name "local") (method "mbox"))) (for (opensmtpd-option-configuration (option "for local")))) (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "outbound"))) (from (opensmtpd-option-configuration (option "from local"))) (for (opensmtpd-option-configuration (option "for any"))))) #+END_SRC ~matches~ is a list of ~~ records, which matches incoming mail and sends it to a correspending action. The match records are evaluated sequentially, with the first match winning. If an incoming mail does not match any match records, then it is rejected. # TODO when the code supports mda-wrappers, add in this documentation. # - ~mda-wrappers~ - ~mta-max-deferred~ (default: ~100~) When delivery to a given host is suspended due to temporary failures, cache at most _number_ envelopes for that host such that they can be delivered as soon as another delivery succeeds to that host. The default is 100. - ~queue~ (default: ~#f~) ~queue~ expects an ~~ record. With it, one may compress and encrypt queue-ed emails as well as set the default expiration time for temporarily undeliverable messages. - ~smtp~ (default: ~#f~) ~smtp~ expects an ~~ record, which lets one specifiy how large email may be along with other settings. - ~srs~ (default: ~#f~) ~srs~ expects an ~~ record, which lets one set up SRS, the Sender Rewritting Scheme. - Data Type: opensmtpd-listen-on-configuration Data type representing the configuration of an ~~. Listen on the fieldname ~interface~ for incoming connections, using the same syntax as for ifconfig(8). The interface parameter may also be an string interface group, an string IP address, or a string domain name. Listening can optionally be restricted to a specific address fieldname ~family~, which can be either "inet4" or "inet6". - ~interface~ (default: "lo") The string interface to listen for incoming connections. These interface can usually be found by the command ~ip link~. - ~family~ (default: ~#f~) The string IP family to use. Valid strings are "inet4" or "inet6". - ~auth~ (default: ~#f~) Support SMTPAUTH: clients may only start SMTP transactions after successful authentication. If ~auth~ is ~#t~, then users are authenticated against their own normal login credentials. Alternatively ~auth~ may be an ~~ whose users are authenticated against their passwords. - ~auth-optional~ (default: ~#f~) Support SMTPAUTH optionally: clients need not authenticate, but may do so. This allows the ~~ to both accept incoming mail from untrusted senders and permit outgoing mail from authenticated users (using ~~ fieldname ~auth~). It can be used in situations where it is not possible to listen on a separate port (usually the submission port, 587) for users to authenticate. - ~filters~ (default: ~#f~) A list of one or many ~~ or ~~ records. The filters are applied sequentially. These records listen and filter on connections handled by this listener. - ~hostname~ (default: ~#f~) Use string "hostname" in the greeting banner instead of the default server name. - ~hostnames~ (default: ~#f~) Override the server name for specific addresses. Use a ~~ containing a mapping of string IP addresses to hostnames. If the address on which the connection arrives appears in the mapping, the associated hostname is used. - ~mask-src~ (default: ~#f~) If ~#t~, then omit the from part when prepending “Received” headers. - ~disable-dsn~ (default: ~#f~) When ~#t~, then disable the DSN (Delivery Status Notification) extension. - ~pki~ (default: ~#f~) For secure connections, use an ~~ to prove a mail server's identity. - ~port~ (default: ~#f~) Listen on the _integer_ port instead of the default port of 25. - ~proxy-v2~ (default: ~#f~) If ~#t~, then support the PROXYv2 protocol, rewriting appropriately source address received from proxy. - ~received-auth~ (default: ~#f~) If ~#t~, then in “Received” headers, report whether the session was authenticated and by which local user. - ~senders~ (default: ~#f~) Look up the authenticated user in the supplied ~~ to find the email addresses that user is allowed to submit mail as. - ~secure-connection~ (default: ~#f~) This is a string of one of these options: |----------------------+---------------------------------------------| | "smtps" | Support SMTPS, by default on port 465. | |----------------------+---------------------------------------------| | "tls" | Support STARTTLS, by default on port 25. | |----------------------+---------------------------------------------| | "tls-require-verify" | Like tls, but force clients to establish | | | a secure connection before being allowed to | | | start an SMTP transaction. With the verify | | | option, clients must also provide a valid | | | certificate to establish an SMTP session. | |----------------------+---------------------------------------------| - ~tag~ (default: ~#f~) Clients connecting to the listener are tagged with the given string tag. - Data Type: opensmtpd-listen-on-socket-configuration Data type representing the configuration of an ~~. Listen for incoming SMTP connections on the Unix domain socket =/var/run/smtpd.sock=. This is done by default, even if the directive is absent. - ~filters~ (default: ~#f~) A list of one or many ~~ or ~~ records. These filter incoming connections handled by this listener. - ~mask-src~ (default: ~#f~) If ~#t~, then omit the from part when prepending “Received” headers. - ~tag~ (default: ~#f~) Clients connecting to the listener are tagged with the given string tag. - Data Type: opensmtpd-match-configuration This data type represents the configuration of an ~~ record. If at least one mail envelope matches the options of one match record, receive the incoming message, put a copy into each matching envelope, and atomically save the envelopes to the mail spool for later processing by the respective ~~ found in fieldname ~action~. - ~action~ (default: ~#f~) If mail matches this match configuration, then do this action. Valid values include ~~ or ~~. - ~options~ (default: ~#f~) ~~ The fieldname 'option' is a list of unique ~~ records. Each ~~ record's fieldname 'option' has some mutually exclusive options: there can be one "for" and one "from" option. |---------------------------+--------------------------------| | for | from | |---------------------------+--------------------------------| | use one of the following: | only use one of the following: | |---------------------------+--------------------------------| | "for any" | "from any" | | "for local" | "from auth" | | "for domain" | "from local" | | "for rcpt-to" | "from mail-from" | | | "from socket" | | | "from src" | |---------------------------+--------------------------------| The following matching options are supported and can all be negated via (not #t). The options that support a table (anything surrounded with '<' and '>' eg:
), also support specifying regex via (regex #t). - =for any= Specify that session may address any destination. - =for local= Specify that session may address any local domain. This is the default, and may be omitted. - =for domain _domain_ | = Specify that session may address the string or list table _domain_. - =for rcpt-to _recipient_ | = Specify that session may address the string or list table _recipient_. - =from any= Specify that session may originate from any source. - =from auth= Specify that session may originate from any authenticated user, no matter the source IP address. - =from auth _user_ | = Specify that session may originate from authenticated _user_ or user list user, no matter the source IP address. - =from local= Specify that session may only originate from a local IP address, or from the local enqueuer. This is the default, and may be omitted. - =from mail-from _sender_ | = Specify that session may originate from _sender_ or table _sender_, no matter the source IP address. - =from rdns= Specify that session may only originate from an IP address that resolves to a reverse DNS. - =from rdns _hostname_ | = Specify that session may only originate from an IP address that resolves to a reverse DNS matching string or list string _hostname_. - =from socket= Specify that session may only originate from the local enqueuer. - =from src _address_ |
= Specify that session may only originate from string or list table address which can be a specific _address_ or a subnet expressed in CIDR-notation. - =auth= Matches transactions which have been authenticated. - =auth _username_ | = Matches transactions which have been authenticated for user or user list _username_. - =helo _helo-name_ | = Specify that session's HELO / EHLO should match the string or list table _helo-name_. - =mail-from _sender_ | = Specify that transactions's MAIL FROM should match the string or list table _sender_. - =rcpt-to _recipient_ | = Specify that transaction's RCPT TO should match the string or list table _recipient_. - =tag tag= Matches transactions tagged with the given _tag_. - =tls= Specify that transaction should take place in a TLS channel. Here is a simple example: #+BEGIN_SRC scheme (opensmtpd-option-configuration (not #t) (regex #f) (option "for domain") (data (opensmtpd-table-configuration (name "domain-table") (data (list "gnu.org" "dismail.de"))))) #+END_SRC The mail must NOT come from the domains =gnu.org= or =dismail.de=. - Data Type: opensmtpd-option-configuration - Data Type: opensmtpd-action-local-delivery-configuration This data type represents the configuration of an ~~ record. - ~name~ (default: ~#f~) ~name~ is the string name of the relay action. - ~method~ (default: ~"mbox"~) The email delivery option. Valid options are: - ~"mbox"~ Deliver the message to the user's mbox with mail.local(8). - ~"expand-only"~ Only accept the message if a delivery method was specified in an aliases or _.forward file_. - ~"forward-only"~ Only accept the message if the recipient results in a remote address after the processing of aliases or forward file. - ~~ Deliver the message to an LMTP server at ~~'s fieldname ~destination~. The location may be expressed as string host:port or as a UNIX socket. Optionally, ~~'s fieldname ~rcpt-to~ might be specified to use the recipient email address (after expansion) instead of the local user in the LMTP session as RCPT TO. - ~~ Deliver the message to the maildir in ~~'s fieldname ~pathname~ if specified, or by default to =~/Maildir=. The pathname may contain format specifiers that are expanded before use (see the below section about Format Specifiers). If ~~'s record fieldname ~junk~ is ~#t~, then message will be moved to the ‘Junk’ folder if it contains a positive ‘X-Spam’ header. This folder will be created under fieldname ~pathname~ if it does not yet exist. - ~~ Delegate the delivery to the ~~'s fieldname ~command~ (type string) that receives the message on its standard input. The ~command~ may contain format specifiers that are expanded before use (see Format Specifiers). - ~alias~ (default: ~#f~) Use the mapping table for aliases expansion. ~alias~ is an ~~. - ~ttl~ (default: ~#f~) ~ttl~ is a string specify how long a message may remain in the queue. It's format is =n{s|m|h|d}=. eg: "4m" is four minutes. - ~user~ (default: ~#f~ ) ~user~ is the string username for performing the delivery, to be looked up with getpwnam(3). This is used for virtual hosting where a single username is in charge of handling delivery for all virtual users. This option is not usable with the mbox delivery method. - ~userbase~ (default: ~#f~) ~userbase~ is an ~~ record for mapping user lookups instead of the getpwnam(3) function. The fieldnames ~user~ and ~userbase~ are mutually exclusive. - ~virtual~ (default: ~#f~) ~virtual~ is an ~~ record is used for virtual expansion. # TODO man 5 smtpd.conf says "The aliasing table format is described in # table(5)." What is virtual expansion? I do NOT know how to use ~virtual~ # properly. What sort of do I need? does the # below work? # (opensmtpd-table (name "virtual") (data '(("joshua" . "jbranso@dismail.de")))) # TODO fix this ~wrapper documentation~. Should it accept an # ? If so, then I need to write an # - ~wrapper~ (default: ) # TODO double check that these options are all correct - Data Type: opensmtpd-action-relay-configuration This data type represents the configuration of an ~~ record. - ~name~ (default: ~#f~) ~name~ is the string name of the relay action. - ~backup~ (default: ~#f~) When ~#t~, operate as a backup mail exchanger delivering messages to any mail exchanger with higher priority. - ~backup-mx~ (default: ~#f~) Operate as a backup mail exchanger delivering messages to any mail exchanger with higher priority than mail exchanger identified as string name. - ~helo~ (default: ~#f~) Advertise string heloname as the hostname to other mail exchangers during the HELO phase. - ~helo-src~ (default: ~#f~ ) Use the mapping ~~ to look up a hostname matching the source address, to advertise during the HELO phase. - ~domain~ (default: ~#f~) Do not perform MX lookups but look up destination domain in an ~~ and use matching relay url as relay host. - ~host~ (default: ~#f~) Do not perform MX lookups but relay messages to the relay host described by the string relay-url. The format for relay-url is =[proto://[label@]]host[:port]=. The following protocols are available: |------------+----------------------------------------------------------------| | smtp | Normal SMTP session with opportunistic STARTTLS (the default). | | smtp+tls | Normal SMTP session with mandatory STARTTLS. | | smtp+notls | Plain text SMTP session without TLS. | | lmtp | LMTP session. port is required. | | smtps | SMTP session with forced TLS on connection, default port is | | | 465. | |------------+----------------------------------------------------------------| Unless noted, port defaults to 25. The label corresponds to an entry in a credentials table, as documented in =table(5)=. It is used with the ="smtp+tls"= and ="smtps"= protocols for authentication. Server certificates for those protocols are verified by default. - ~pki~ (default: ~#f~) For secure connections, use the certificate associated with ~~ (declared in a pki directive) to prove the client's identity to the remote mail server. - ~srs~ (default: ~#f~) If ~#t~, then when relaying a mail resulting from a forward, use the Sender Rewriting Scheme to rewrite sender address. - ~tls~ (default: ~#f~) boolean or string "no-verify" When ~#t~, Require TLS to be used when relaying, using mandatory STARTTLS by default. When used with a smarthost, the protocol must not be ="smtp+notls://"=. When string ~"no-verify"~, then do not require a valid certificate. - ~auth~ (default: ~#f~) ~~ Use the alist ~~ for connecting to relay-url using credentials. This option is usable only with fieldname ~host~ option. - ~mail-from~ (default: ~#f~) string Use the string _mailaddress_ as MAIL FROM address within the SMTP transaction. - ~src~ (default: ~#f~) string | ~~ Use the string or ~~ sourceaddr for the source IP address, which is useful on machines with multiple interfaces. If the list contains more than one address, all of them are used in such a way that traffic is routed as efficiently as possible. - Data Type: opensmtpd-filter-configuration This data type represents the configuration of an ~~. This is the filter record one should use if they want to use an external package to filter email eg: rspamd or spamassassin. - ~name~ (default: ~#f~) The string name of the filter. - ~proc~ (default: ~#f~) # TODO let ~proc~ be a gexp The string command or process name. If ~proc-exec~ is ~#t~, ~proc~ is treated as a command to execute. Otherwise, it is a process name. - ~proc-exec~ (default: ~#f~) - Data Type: opensmtpd-filter-phase-configuration This data type represents the configuration of an ~~. In a regular workflow, smtpd(8) may accept or reject a message based only on the content of envelopes. Its decisions are about the handling of the message, not about the handling of an active session. Filtering extends the decision making process by allowing smtpd(8) to stop at each phase of an SMTP session, check that options are met, then decide if a session is allowed to move forward. With filtering via an ~~ record, a session may be interrupted at any phase before an envelope is complete. A message may also be rejected after being submitted, regardless of whether the envelope was accepted or not. - ~name~ (default: ~#f~) The string name of the filter phase. - ~phase-name~ (default: ~#f~) The string name of the phase. Valid values are: |-------------+-----------------------------------------------| | "connect" | upon connection, before a banner is displayed | |-------------+-----------------------------------------------| | "helo" | after HELO command is submitted | |-------------+-----------------------------------------------| | "ehlo" | after EHLO command is submitted | |-------------+-----------------------------------------------| | "mail-from" | after MAIL FROM command is submitted | |-------------+-----------------------------------------------| | "rcpt-to" | after RCPT TO command is submitted | |-------------+-----------------------------------------------| | "data" | after DATA command is submitted | |-------------+-----------------------------------------------| | "commit" | after message is fully is submitted | |-------------+-----------------------------------------------| - ~options~ (default ~#f~) A list of unique ~~ records. At each phase, various options, specified by a list of ~~, may be checked. The ~~'s fieldname 'option' values of: "fcrdns", "rdns", and "src" data are available in all phases, but other data must have been already submitted before they are available. Options with a =
= next to them require the ~~'s fieldname ~data~ to be an ~~. There are the available options: |-------------------+----------------------------------------| | fcrdns | forward-confirmed reverse DNS is valid | |-------------------+----------------------------------------| | rdns | session has a reverse DNS | |-------------------+----------------------------------------| | rdns
| session has a reverse DNS in table | |-------------------+----------------------------------------| | src
| source address is in table | |-------------------+----------------------------------------| | helo
| helo name is in table | |-------------------+----------------------------------------| | auth | session is authenticated | |-------------------+----------------------------------------| | auth
| session username is in table | |-------------------+----------------------------------------| | mail-from
| sender address is in table | |-------------------+----------------------------------------| | rcpt-to
| recipient address is in table | |-------------------+----------------------------------------| These conditions may all be negated by setting ~~'s fieldname ~not~ to ~#t~. Any conditions that require a table may indicate that tables include regexs setting ~~'s fieldname ~regex~ to ~#t~. - ~decision~ A string decision to be taken. Some decisions require an ~message~ or ~value~. Valid strings are: |----------------------+------------------------------------------------| | "bypass" | the session or transaction bypasses filters | |----------------------+------------------------------------------------| | "disconnect" message | the session is disconnected with message | |----------------------+------------------------------------------------| | "junk" | the session or transaction is junked, i.e., an | | | ‘X-Spam: yes’ header is added to any messages | |----------------------+------------------------------------------------| | "reject" message | the command is rejected with message | |----------------------+------------------------------------------------| | "rewrite" value | the command parameter is rewritten with value | |----------------------+------------------------------------------------| Decisions that involve a message require that the message be RFC valid, meaning that they should either start with a 4xx or 5xx status code. Descisions can be taken at any phase, though junking can only happen before a message is committed. - ~message~ (default ~#f~) A string message beginning with a 4xx or 5xx status code. - ~value~ (default: ~#f~) A number value. ~value~ and ~message~ are mutually exclusive. - Data Type: opensmtpd-option-configuration This data type represents the configuration of an ~~, which is used by ~~ and ~~ to match various options for email. - ~conditition~ (default ~#f~) A string option to be taken. Some options require a string or an ~~ via the fieldname data. When the option record is used inside of an ~~, then valid strings are: At each phase, various options may be matched. The fcrdns, rdns, and src data are available in all phases, but other data must have been already submitted before they are available. |---------------------+----------------------------------------| | "fcrdns" | forward-confirmed reverse DNS is valid | | "rdns" | session has a reverse DNS | | "rdns"
| session has a reverse DNS in table | | "src"
| source address is in table | | "helo"
| helo name is in table | | "auth" | session is authenticated | | "auth"
| session username is in table | | "mail-from"
| sender address is in table | | "rcpt-to"
| recipient address is in table | |---------------------+----------------------------------------| When ~~ is used inside of an ~~, then valid strigs for fieldname ~option~ are: "for", "for any", "for local", "for domain", "for rcpt-to", "from any" "from auth", "from local", "from mail-from", "from rdns", "from socket", "from src", "auth", "helo", "mail-from", "rcpt-to", "tag", or "tls". - ~data~ (default ~#f~) ~~ Some options require a table to be present. One would specify that table here. - ~regex~ (default: ~#f~) boolean Any options using a table may indicate that tables hold regex by prefixing the table name with the keyword regex. - ~not~ (default: ~#f~) boolean When ~#t~, this option record is negated. - Data Type: opensmtpd-table-configuration This data type represents the configuration of an ~~. - ~name~ (default ~#f~) ~name~ is the name of the ~~ record. - ~data~ (default: ~#f~) ~data~ expects a list of strings or an alist, which is a list of cons cells. eg: ~(data (list ("james" . "password")))~ OR ~(data (list ("gnu.org" "fsf.org")))~. - Data Type: opensmtpd-pki-configuration This data type represents the configuration of an ~~. - ~domain~ (default ~#f~) ~domain~ is the string name of the ~~ record. - ~cert~ (default: ~#f~) ~cert~ (default: ~#f~) ~cert~ is the string certificate filename to use for this pki. - ~key~ (default: ~#f~) ~key~ is the string certificate falename to use for this pki. - ~dhe~ (default: ~"none"~) Specify the DHE string parameter to use for DHE cipher suites with host pkiname. Valid parameter values are "none", "legacy", or "auto". For "legacy", a fixed key length of 1024 bits is used, whereas for "auto", the key length is determined automatically. The default is "none", which disables DHE cipher suites. - Data Type: opensmtpd-maildir-configuration - ~pathname~ (default: ~"~/Maildir"~) Deliver the message to the maildir if pathname if specified, or by default to =~/Maildir=. The pathname may contain format specifiers that are expanded before use (see FORMAT SPECIFIERS). - ~junk~ (default: ~#f~) If the junk argument is ~#t~, then the message will be moved to the =‘Junk’= folder if it contains a positive =‘X-Spam’= header. This folder will be created under pathname if it does not yet exist. - Data Type: opensmtpd-mda-configuration # Do we need a dataypte for mda configuration? # this could just be a gexp in the fieldname opensmtpd-configuration-mda - ~name~ The string name for this MDA command. - ~command~ Delegate the delivery to a command that receives the message on its standard input. The command may contain format specifiers that are expanded before use (see FORMAT SPECIFIERS). - Data Type: opensmtpd-queue-configuration - ~compression~ (default ~#f~) Store queue files in a compressed format. This may be useful to save disk space. - ~encryption~ (default ~#f~) Encrypt queue files with EVP_aes_256_gcm(3). If no key is specified, it is read with getpass(3). If the string stdin or a single dash (‘-’) is given instead of a key, the key is read from the standard input. - ~ttl-delay~ (default ~#f~) Set the default expiration time for temporarily undeliverable messages, given as a positive decimal integer followed by a unit s, m, h, or d. The default is four days ("4d"). - Data Type: opensmtpd-smtp-configuration Data type representing an ~~ record. - ~ciphers~ (default: ~#f~) Set the control string for SSL_CTX_set_cipher_list(3). The default is "HIGH:!aNULL:!MD5". - ~limit-max-mails~ (default: ~100~) Limit the number of messages to count for each sessio - ~limit-max-rcpt~ (default: ~1000~) Limit the number of recipients to count for each transaction. - ~max-message-size~ (default: ~35M~) Reject messages larger than size, given as a positive number of bytes or as a string to be parsed with scan_scaled(3). - ~sub-addr-delim character~ (default: ~+~) When resolving the local part of a local email address, ignore the ASCII character and all characters following it. This is helpful for email filters. ="admin+bills@gnu.org"= is the same email address as ="admin@gnu.org"=. BUT an email filter can filter emails addressed to first email address into a 'Bills' email folder. - Data Type: opensmtpd-srs-configuration - ~key~ (default: ~#f~) Set the secret key to use for SRS, the Sender Rewriting Scheme. - ~backup-key~ (default: ~#f~) Set a backup secret key to use as a fallback for SRS. This can be used to implement SRS key rotation. - ~ttl-delay~ (default: ~"4d"~) Set the time-to-live delay for SRS envelopes. After this delay, a bounce reply to the SRS address will be discarded to limit risks of forged addresses. - Format Specifiers Some configuration records support expansion of their parameters at runtime. Such records (for example ~~, ~~) may use format specifiers which are expanded before delivery or relaying. The following formats are currently supported: |---------------------+-------------------------------------------------------| | =%{sender}= | sender email address, may be empty string | | =%{sender.user}= | user part of the sender email address, may be empty | | =%{sender.domain}= | domain part of the sender email address, may be empty | | =%{rcpt}= | recipient email address | | =%{rcpt.user}= | user part of the recipient email address | | =%{rcpt.domain}= | domain part of the recipient email address | | =%{dest}= | recipient email address after expansion | | =%{dest.user}= | user part after expansion | | =%{dest.domain}= | domain part after expansion | | =%{user.username}= | local user | | =%{user.directory}= | home directory of the local user | | =%{mbox.from}= | name used in mbox From separator lines | | =%{mda}= | mda command, only available for mda wrappers | |---------------------+-------------------------------------------------------| Expansion formats also support partial expansion using the optional bracket notations with substring offset. For example, with recipient domain =“example.org”=: |------------------------+----------------------| | =%{rcpt.domain[0]}= | expands to “e” | | =%{rcpt.domain[1]}= | expands to “x” | | =%{rcpt.domain[8:]}= | expands to “org” | | =%{rcpt.domain[-3:]}= | expands to “org” | | =%{rcpt.domain[0:6]}= | expands to “example” | | =%{rcpt.domain[0:-4]}= | expands to “example” | |------------------------+----------------------| In addition, modifiers may be applied to the token. For example, with recipient =“User+Tag@Example.org”=: |--------------------------+-----------------------------------| | =%{rcpt:lowercase}= | expands to “user+tag@example.org” | | =%{rcpt:uppercase}= | expands to “USER+TAG@EXAMPLE.ORG” | | =%{rcpt:strip}= | expands to “User@Example.org” | | =%{rcpt:lowercasestrip}= | expands to “user@example.org” | |--------------------------+-----------------------------------| For security concerns, expanded values are sanitized and potentially dangerous characters are replaced with ‘:’. In situations where they are desirable, the “raw” modifier may be applied. For example, with recipient =“user+t?g@example.org”=: |---------------+-----------------------------------| | =%{rcpt}= | expands to “user+t:g@example.org” | | =%{rcpt:raw}= | expands to “user+t?g@example.org” | |---------------+-----------------------------------| *** some example ~~ that are probably out of date #+BEGIN_SRC scheme ;;this works! (opensmtpd-configuration->mixed-text-file (opensmtpd-configuration (smtp (opensmtpd-smtp-configuration (limit-max-rcpt 10))))) ;; (tables (list ;; (opensmtpd-table-configuration ;; (name "aliases") ;; (data ;; (list ;; (cons "webmaster" "root") ;; (cons "postmaster" "root") ;; (cons "abuse" "root")))) ;; ;; (opensmtpd-table-configuration ;; (name "vdoms") ;; (data (list "gnucode.me" ;; "gnu-hurd.com"))) ;; (opensmtpd-table-configuration ;; (name (opensmtpd-table-configuration ;; (name "virtual") ;; (data (list "root" "postmaster@gnu.org")))) ;; (data (list (cons "joshua@gnucode.me" "joshua") ;; (cons "jbranso@gnucode.me" "joshua") ;; (cons "postmaster@gnucode.me" "joshua")))))) ;; (filter-chains ;; (list ;; (opensmtpd-filter-chain ;; (name "dropDumbEmails") ;; (filter-names (list "nofcrdnsDisconnect" ;; "nordnsDisconnect"))))) ;; (filter-phases ;; (list (opensmtpd-filter-phase-configuration ;; (name "nofcrdnsDisconnect") ;; (phase-name "connect") ;; (options (list "!fcrdns")) ;; (decision "disconnect") ;; (message "You have not set up forward confirmed DNS.")) ;; (opensmtpd-filter-phase-configuration ;; (name "nordnsDisconnect") ;; (phase-name "connect") ;; (options (list "!rdns")) ;; ;; (decision "reject") ;; (message "You have not set up reverse DNS.")))) ;; (define example-opensmtpd-config-smaller (opensmtpd-configuration (listen-ons (list ;; this forum help suggests that I listen on 0.0.0.0 and NOT eth0 ;; https://serverfault.com/questions/726795/opensmtpd-wont-work-at-reboot ;; this listens for email from the outside world ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on-configuration (interface "wlp2s0") (port 465)))) (matches (list (opensmtpd-match-configuration (name "maildir") (action (opensmtpd-action-local-delivery-configuration (method (opensmtpd-maildir-configuration (pathname "/home/%{rcpt.user}/Maildir") (junk #t))) (virtual (opensmtpd-table-configuration (name "virtual") (data (list "root" "james@gnu.org")))))) (for (opensmtpd-option-configuration (option "for local")))))))) (define example-opensmtpd-config-small (let ([interface "wlp2s0"] [creds (opensmtpd-table-configuration (name "creds") (data (list (cons "joshua" "$6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86."))))] [receive-action (opensmtpd-action-local-delivery-configuration (name "receive") (method (opensmtpd-maildir-configuration (pathname "/home/%{rcpt.user}/Maildir") (junk #t))) (virtual (opensmtpd-table-configuration (name "virtual") (data (list "root" "james@gnu.org")))))] [smtp.gnucode.me (opensmtpd-pki-configuration (domain "smtp.gnucode.me") (cert "opensmtpd.scm") (key "opensmtpd.scm"))]) (opensmtpd-configuration (listen-ons (list ;; this forum help suggests that I listen on 0.0.0.0 and NOT eth0 ;; https://serverfault.com/questions/726795/opensmtpd-wont-work-at-reboot ;; this listens for email from the outside world (opensmtpd-listen-on-configuration (interface interface) (port 25) (secure-connection "tls") (pki smtp.gnucode.me)) ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on-configuration (interface interface) (port 465) (secure-connection "smtps") (pki smtp.gnucode.me) (auth creds)))) (matches (list (opensmtpd-match-configuration (action receive-action) (for (opensmtpd-option-configuration (option "for local"))))))))) (define example-opensmtpd-config (let ([interface "lo"] [creds (opensmtpd-table-configuration (name "creds") (data (list (cons "joshua" "$6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86."))))] [receive-action (opensmtpd-action-local-delivery-configuration (name "receive") (method (opensmtpd-maildir-configuration (pathname "/home/%{rcpt.user}/Maildir") (junk #t))) (virtual (opensmtpd-table-configuration (name "virtual") (data (list "josh" "jbranso@dismail.de")))))] [smtp.gnucode.me (opensmtpd-pki-configuration (domain "smtp.gnucode.me") (cert "opensmtpd.scm") (key "opensmtpd.scm"))]) (opensmtpd-configuration ;; (mta-max-deferred 50) ;; (queue ;; (opensmtpd-queue-configuration ;; (compression #t))) ;; (smtp ;; (opensmtpd-smtp-configuration ;; (max-message-size "10M"))) ;; (srs ;; (opensmtpd-srs-configuration ;; (ttl-delay "5d"))) (listen-ons (list ;; this forum help suggests that I listen on 0.0.0.0 and NOT eth0 ;; https://serverfault.com/questions/726795/opensmtpd-wont-work-at-reboot ;; this listens for email from the outside world (opensmtpd-listen-on-configuration (interface interface) (port 25) (secure-connection "tls") (pki smtp.gnucode.me)) ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on-configuration (interface "lo") (port 25) (secure-connection "tls") (pki smtp.gnucode.me)) (opensmtpd-listen-on-configuration (interface interface) (port 465) (secure-connection "smtps") (pki smtp.gnucode.me) (auth creds) ;;(filter ) ) (opensmtpd-listen-on-configuration (interface interface) (port 587) (secure-connection "tls-require") (pki smtp.gnucode.me) (auth creds)))) (matches (list (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "send"))) (for (opensmtpd-option-configuration (option "for any"))) (from (opensmtpd-option-configuration (option "from any"))) (auth (opensmtpd-option-configuration (option "auth")))) (opensmtpd-match-configuration (action receive-action) (from (opensmtpd-option-configuration (option "from any"))) (for (opensmtpd-option-configuration (option "for domain") (value (list "gnucode.me" "gnu-hurd.com"))))) (opensmtpd-match-configuration (action receive-action) (for (opensmtpd-option-configuration (option "for local"))))))))) #+END_SRC *** some example smtpd.conf configs *** serving multiple domains with one pki source: https://www.reddit.com/r/openbsd/comments/n41wkz/how_to_host_different_domains_for_an_email_server/ #+BEGIN_EXAMPLE ​pki mail.primary.domain cert​pki mail.primary.domain cert "/etc/ssl/mail.primary.domain.fullchain.pem" pki mail.primary.domain key "/etc/ssl/private/mail.primary.domain.key" filter check_dyndns phase connect match rdns regex { '.*\.dyn\..*', '.*\.dsl\..*' } \ disconnect "550 no residential connections" filter check_rdns phase connect match !rdns \ disconnect "550 no rDNS is so 80s" filter check_fcrdns phase connect match !fcrdns \ disconnect "550 no FCrDNS is so 80s" filter senderscore \ proc-exec "filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000" filter rspamd proc-exec "filter-rspamd" table usermap file:/etc/mail/usermap table credentials file:/etc/mail/credentials table domains { primary.domain, second.domain } listen on all tls pki mail.primary.domain \ filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } listen on egress port 465 smtps pki mail.primary.domain \ auth ~~ filter rspamd action "inbound" lmtp "/var/dovecot/lmtp" rcpt-to virtual ~~ #maildir junk alias action "outbound" relay helo mail.primary.domain match from any for domain ~~ action "inbound" match for local action "inbound" match from any auth for any action "outbound" match for any action "outbound" "/etc/ssl/mail.primary.domain.fullchain.pem" pki mail.primary.domain key "/etc/ssl/private/mail.primary.domain.key" filter check_dyndns phase connect match rdns regex { '.*\.dyn\..*', '.*\.dsl\..*' } \ disconnect "550 no residential connections" filter check_rdns phase connect match !rdns \ disconnect "550 no rDNS is so 80s" filter check_fcrdns phase connect match !fcrdns \ disconnect "550 no FCrDNS is so 80s" filter senderscore \ proc-exec "filter-senderscore -blockBelow 10 -junkBelow 70 -slowFactor 5000" filter rspamd proc-exec "filter-rspamd" table usermap file:/etc/mail/usermap table credentials file:/etc/mail/credentials table domains { primary.domain, second.domain } listen on all tls pki mail.primary.domain \ filter { check_dyndns, check_rdns, check_fcrdns, senderscore, rspamd } listen on egress port 465 smtps pki mail.primary.domain \ auth ~~ filter rspamd action "inbound" lmtp "/var/dovecot/lmtp" rcpt-to virtual ~~ #maildir junk alias action "outbound" relay helo mail.primary.domain match from any for domain ~~ action "inbound" match for local action "inbound" match from any auth for any action "outbound" match for any action "outbound" #+END_EXAMPLE ** PROJ nice things to have [0/9] *** TODO Should I delete ~~ ? or fieldname 'opensmtpd-configuration-mda-wrapppers'? ~~'s fieldname 'method' allows for an mda configuration. BUT instead of an mda-configuration record, you could just use a list of strings and/or gexps. #+BEGIN_EXAMPLE mda wrapper name command Associate command with the mail delivery agent wrapper named name. When a local delivery specifies a wrapper, the command associated with the wrapper will be ex‐ ecuted instead. The command may contain format specifiers (see FORMAT SPECIFIERS). #+END_EXAMPLE If I choose to NOT delete ~~, then should I delete 'opensmtpd-configuration-mda-wrapppers'? Also should I delete the opensmtpd-action-local-delivery-configuration-wrapper? *** TODO make the 'auth-optional' and 'auth' fieldnames for ~~ autoencrypt passwords. [0/0] Guix makes it pretty hard to find the openbsd binary file that encrypts passwords for you. If I can progmatically find this file, it would be nice to autoencrypt the users's passwords for you. What does this mean practically? #+BEGIN_SRC scheme (opensmtpd-table-configuration (name "credentials") (data '(("joshua@gnu.org" . "somePassword") ("postmaster@gnu.org") . "anotherSillyPassword"))) #+END_SRC Gets stored in /gnu/store/ in the =smtpd.conf= as something like: #+BEGIN_EXAMPLE table credentials { joshua@gnu.org = $some$Long$EncrytpedPassword, \ postmaster@gnu.org = $some$Long$Other$EncrytpedPassword } #+END_EXAMPLE You would need to encourage users NOT to have passwords in a public git repo. With guile-git, it might be possible to sanitize the config, to ensure that the passwords are NOT stored in the git repo. Alternatively, we could put the following in the documentation: #+BEGIN_SRC scheme (use-modules (passwords)) (opensmtpd-table-configuration (name "credentials") (data %passwords)) #+END_SRC *** PROJ Why does (opensmtpd-configuration) take so long to initialize? [0/1] For example, try to initialize this bit of code. It takes almost 5 seconds. #+BEGIN_SRC scheme (let ([interface "lo"] [creds-table (opensmtpd-table-configuration (name "creds") (data (list (cons "joshua" "$6$Ec4m8FgKjT2F/03Y$k66ABdse9TzCX6qaALB3WBL9GC1rmAWJmaoSjFMpbhzat7DOpFqpnOwpbZ34wwsQYIK8RQlqwM1I/v6vsRq86."))))] [receive-action (opensmtpd-action-local-delivery-configuration (name "receive") (method (opensmtpd-maildir-configuration (pathname "/home/%{rcpt.user}/Maildir") (junk #t))) (virtual (opensmtpd-table-configuration (name "virtual") (data (list "josh" "jbranso@dismail.de")))))] [filter-dkimsign (opensmtpd-filter (name "dkimsign") (exec #t) (proc (string-append "/path/to/dkimsign -d gnucode.me -s 2021-09-22 -c relaxed/relaxed -k " "/path/to/dkimsign-key user nobody group nobody")))] [smtp.gnucode.me (opensmtpd-pki-configuration (domain "smtp.gnucode.me") (cert "opensmtpd.scm") (key "opensmtpd.scm"))]) (opensmtpd-configuration (mta-max-deferred 50) (queue (opensmtpd-queue-configuration (compression #t))) (smtp (opensmtpd-smtp-configuration (max-message-size "10M"))) (srs (opensmtpd-srs-configuration (ttl-delay "5d"))) (listen-ons (list (opensmtpd-listen-on-configuration (interface interface) (port 25) (secure-connection "tls") (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") (not #t)))) (decision "disconnect") (message "No FCRDNS")))) (pki smtp.gnucode.me)) ;; this lets local users logged into the system via ssh send email (opensmtpd-listen-on-configuration (interface interface) (port 465) (secure-connection "smtps") (pki smtp.gnucode.me) (auth creds-table) (filters (list filter-dkimsign))) (opensmtpd-listen-on-configuration (interface interface) (port 587) (secure-connection "tls-require") (pki smtp.gnucode.me) (auth creds-table) (filters (list filter-dkimsign))))) (matches (list (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "relay"))) (for (opensmtpd-option-configuration (option "for any"))) (from (opensmtpd-option-configuration (option "from any"))) (auth (opensmtpd-option-configuration (option "auth")))) (opensmtpd-match-configuration (action receive-action) (from (opensmtpd-option-configuration (option "from any"))) (for (opensmtpd-option-configuration (option "for domain") (value (opensmtpd-table-configuration (name "domain-table") (data (list "gnucode.me" "gnu-hurd.com"))))))) (opensmtpd-match-configuration (action receive-action) (for (opensmtpd-option-configuration (option "for local")))))))) #+END_SRC **** TODO one area to look for speed up improvements would be in the sanitize function of =(opensmtpd-listen-on-configuration-filters)=. *** PROJ check the code base for places to use apply, map, fold, eval, or remove [3/4] **** DONE string-in-list would be a good place. maybe is-value-right-type #+BEGIN_SRC scheme (define (string-in-list? string list) (if (null? list) #f (if (and (string? (car list)) (string=? string (car list))) #t (string-in-list? string (cdr list))))) (define (string-in-list? string list) (primitive-eval (cons 'or (map (lambda (var) (string=? string var)) list)))) #+END_SRC **** DONE contains-duplicate #+BEGIN_SRC scheme (define (contains-duplicate? list) (if (null? list) #f (or ;; (let loop ([list (cdr list)] [1st (car list)]) (if (null? list) #f (if (equal? 1st (car list)) (data #t 1st) (loop (cdr list) 1st)))) ;; (contains-duplicate? (cdr list))))) (define (contains-duplicate? list) (if (null? list) #f (or (primitive-eval (cons 'or ; check if (car list) is in (cdr list) (map (lambda (var) (equal? var (car list))) (cdr list)))) ;; check if (cdr list) contains duplicate (contains-duplicate? (cdr list))))) #+END_SRC **** DONE using remove and flatten and map #+BEGIN_SRC scheme (define (get-opensmtpd-table-configurations value) (delete-duplicates (let loop ([list (flatten (cond ((opensmtpd-table-configuration? value) value) ((record? value) (let* ([ (record-type-descriptor value)] [list-of-record-fieldnames (record-type-fields )]) (map (lambda (fieldname) (get-opensmtpd-table-configurations ((record-accessor fieldname) value))) list-of-record-fieldnames))) ((and (list? value) (not (null? list))) (map (lambda (element-in-list) (if (record? element-in-list) (get-opensmtpd-table-configurations element-in-list) #f)) value))))]) (if (null? list) '() (if (opensmtpd-table-configuration? (car list)) (cons (car list) (loop (cdr list))) (loop (cdr list))))))) (define (get-opensmtpd-table-configurations value) (let loop ([list (flatten ;; turn (list '(1) '(2 '(3))) -> '(1 2 3) (cond ((opensmtpd-table-configuration? value) value) ((record? value) (let* ([ (record-type-descriptor value)] [list-of-record-fieldnames (record-type-fields )]) (map (lambda (fieldname) (get-opensmtpd-table-configurations ((record-accessor fieldname) value))) list-of-record-fieldnames))) ((and (list? value) (not (null? list))) (map (lambda (element-in-list) (if (record? element-in-list) (get-opensmtpd-table-configurations element-in-list) #f)) value))))]) (delete-duplicates (partition opensmtpd-table-configuration? list)))) #+END_SRC **** TODO using map, apply, and fold is certainly awesome, but is it less efficient? For example, list-of-type? using a named let is pretty efficient. It loops through the list once. #+BEGIN_SRC scheme (define (list-of-type? list proc?) (if (and (list? list) (not (null? list))) (let loop ([list list]) (if (null? list) #t (if (proc? (car list)) (loop (cdr list)) #f))) #f)) #+END_SRC BUT when I using map on this, it is slightly less efficient. It has to apply a simple procedure to each element in the list. Then it has to return the list of booleans. Then it has to build the primitive eval list, then it has to eval it. #+BEGIN_SRC scheme (define (list-of-type? list proc?) (if (and (list? list) (not (null? list))) (primitive-eval (cons 'and (map (lambda (var) (if (proc? var) #t #f)) list))) #f)) #+END_SRC *** PROJ improve [0/2] **** TODO it would be nice if ~~ supported aliasing tables, as described in man 5 table #+BEGIN_EXAMPLE Aliasing tables Aliasing tables are mappings that associate a recipient to one or many destinations. They can be used in two contexts: primary domain aliases and virtual domain mapping. action name method alias
action name method virtual
In a primary domain context, the key is the user part of the recipient address, whilst the value is one or many recipients as described in aliases(5): user1 otheruser user2 otheruser1,otheruser2 user3 otheruser@example.com In a virtual domain context, the key is either a user part, a full email address or a catch all, following selection rules described in smtpd.conf(5), and the value is one or many recipients as described in aliases(5): user1 otheruser user2@example.org otheruser1,otheruser2 @example.org otheruser@example.com @ catchall@example.com #+END_EXAMPLE Currently opensmtpd-table-configuration, does not support mapping a user to 5 email addresses. For example, if user 'dave' can email as 'postmaster@gnu.org', and 'other@gnu.org', and 5 other email addresses... does not support this kind of mapping. To support it, I may be able to just embed a table in smtpd.conf, or I may need to create an /etc/aliases table as man 5 aliases describes. **** TODO make an with file-db #t, auto convert the table into a berkley database via makemap See man 5 table and man smtpd.conf *** TODO writing out the pkis when there are no pkis gives the string "\n"...it might be better to give "" instead =(opensmtpd-configuration-fieldname->string example-opensmtpd-with-0-pkis opensmtpd-configuration-pkis opensmtpd-pki-configuration->string)= *** PROJ Can I make some improvements to my/sanitize procedure? [0/5] **** how does my/sanitize procedure work? ~(my/sanitize var "record-name" "fieldname" '(string? boolean? number?)')~ It is essentially asking? are you any of the following: string?, boolean?, number? If not, then error out with a helpful error message. **** How does my hard-coded sanitized procedure work? eg [[file:opensmtpd-records.scm::(filters opensmtpd-listen-on-configuration-filters][opensmtpd-listen-on-configuration-filters]] This hard coded sanitize is a little different than the my/sanitize procedure. I designed the thunk ~my/sanitize~, such that each thunk (string?, false?, boolean?) has a corresponding entry in the procedure [[file:opensmtpd-records.scm::define (list-of-procedures->string procedures][~(list-of-procedures->string procedures)~]]. However, it would be nice to have the sanitize invocation in opensmtpd-listen-on-configuration-filters use a my/sanitize invocation like so. #+BEGIN_SRC scheme (my/sanitize var "opensmtpd-listen-on-configuration" "filters" (list false? '(list-has-duplicates-or-non-filters "is a list in which each unique element is of type \n" "or .") '(some-filters-in-list-need-message? " fieldname: 'decision' options " "\"disconnect\" and \"reject\" require fieldname 'message'\n" "to have a string.\n") '(some-filters-in-list-need-value? " fieldname: 'decision' option " "\"rewrite\" requires fieldname 'value'\n" "to have a string.\n"))) #+END_SRC **** PROJ better error messages for my/sanitize calls that use a lambda instead of a defined function [0/4] THIS IS HARD TO DO... NOT DOING IT! I just chose to use a hard-coded error message baked into the lambda. I tried making my/sanitize better...but I could not get it to work. The hard-coded method just works: #+BEGIN_SRC scheme (phase-name opensmtpd-filter-phase-configuration-phase-name ;; string (default #f) (sanitize (lambda (var) (if (and (string? var) (or (string=? "connect" var) (string=? "helo" var) (string=? "mail-from" var) (string=? "rcpt-to" var) (string=? "data" var) (string=? "commit" var))) var (begin (display (string-append " fieldname: 'phase-name' is of type " "string. The string can be either 'connect'," " 'helo', 'mail-from', 'rcpt-to', 'data', or 'commit.'\n ")) (throw 'bad! var)))))) #+END_SRC Why? ~~ fieldnames only accept certain strings. I want to sanitize each fieldname to make sure that it's strings is one of those strings. How would I do this? For example, ~~ fieldname 'decision' uses a lambda to sanitize itself. This will result in an error message that is descriptive enough to solve the problem. If I decide to do this, then I probably should create a non-exported record. #+BEGIN_SRC scheme (opensmtpd-filter-phase-configuration (name "cat") (phase-name "connect") (options "fcrdns") (decision "bypasse")) #+END_SRC #+RESULTS: : (opensmtpd-filter-phase-configuration (name "cat") (phase-name "connect") (options "fcrdns") (decision "bypasse")) : fieldname 'bypasse' is of type. : #:972:0 (var)> : ice-9/boot-9.scm:1685:16: In procedure raise-exception: : Throw to key `bad!' with args `(#:972:0 (var)>)'. A solution may be to modify the my/sanitize procedure to accept something like #+BEGIN_SRC scheme (my/sanitize var "" "'fieldname'" (list ((lambda (var) ...) . "list of unique numbers or strings"))) #+END_SRC I have some example code [[file:opensmtpd-records.scm::;; TODO add in some code that accepts a (cons . cell) that of (proc . "error message") here.][here.]] It probably won't work, but it is a rough sketch of what could work. #+BEGIN_SRC scheme [(eq? (cons? (car procedures))) (cdr (car procedures))] #+END_SRC Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. ***** TODO Now make all the other sanitize sections that use a lambda use this new functionality: eg: #+BEGIN_SRC scheme (family opensmtpd-listen-on-configuration-family (default #f) (sanitize (lambda (var) (cond [(eq? #f var) ;; var == #f var] [(and (string? var) (or (string=? "inet4" var) (string=? "inet6" var))) var] [else (begin (display " fieldname 'family' must be string \"inet4\" or \"inet6\".\n") (throw 'bad! var))])))) #+END_SRC ***** TODO perhaps I can create an unexported record [0/2] fieldnames: 'procedure', 'error message'. ***** TODO Perhaps I could try to make my/sanitize work more like the opensmtpd-listen-on-configuration-filters does. It looks like tiny errors first. And shows you those relevent errors. When you fix those tiny errors it starts looking for harder errors. This is nice because when you get something wrong in the config, you get the specific error message. The way my/sanitize currently works, if you get one thing wrong, then you get 4 reasons for what you might have done wrong. ***** TODO the has lost of hard coded error checking. It would be nice to hook this up to my/sanitize. Along with other bits of the code. **** TODO rework my/sanitize to be like opensmtpd-listen-on-configuration-filters opensmtpd-listen-on-configuration-filters works like so: Is the variable (not (false? var)) Is the variable (need-some-messages?) Does the variable (need-some-value) ? else var. For example, this #+BEGIN_SRC scheme (define-record-type* opensmtpd-table-configuration make-opensmtpd-table-configuration opensmtpd-table-configuration? this-record (name opensmtpd-table-configuration-name ;; string (default #f) (sanitize (lambda (var) (my/sanitize var "opensmtpd-table-configuration" "name" (list string?))))) (file-db opensmtpd-table-configuration-file-db (default #f) (sanitize (lambda (var) (my/sanitize var "opensmtpd-table-configuration" "file-db" (list boolean?))))) #+END_SRC would become: #+BEGIN_SRC scheme (define-record-type* opensmtpd-table-configuration make-opensmtpd-table-configuration opensmtpd-table-configuration? this-record (name opensmtpd-table-configuration-name ;; string (default #f) (sanitize (lambda (var) (my/sanitize var "opensmtpd-table-configuration" "name" (list not-string?))))) (file-db opensmtpd-table-configuration-file-db (default #f) (sanitize (lambda (var) (my/sanitize var "opensmtpd-table-configuration" "file-db" (list not-boolean?))))) #+END_SRC This #+BEGIN_SRC scheme (secure-connection opensmtpd-listen-on-configuration-secure-connection (default #f) (sanitize (lambda (var) (cond [(boolean? var) var] [(and (string? var) (string-in-list? var (list "smtps" "tls" "tls-require" "tls-require-verify"))) var] [else (begin (display (string-append " fieldname 'secure-connection' can be " "one of the following strings: \n'smtps', 'tls', 'tls-require', " "or 'tls-require-verify'.\n")) (throw 'bad! var))])))) #+END_SRC would become (finish this thought exercise.) #+BEGIN_SRC scheme (secure-connection opensmtpd-listen-on-configuration-secure-connection (default #f) (sanitize (lambda (var) (list not-boolean (sanitize-proc-configuration string-in-list? (list "smtps" "tls" "tls-require" "tls-require-verify")) [else (begin (display (string-append " fieldname 'secure-connection' can be " "one of the following strings: \n'smtps', 'tls', 'tls-require', " "or 'tls-require-verify'.\n")) (throw 'bad! var))])))) #+END_SRC **** PROJ can we merge my/sanitize syntax into (guix records)? #+BEGIN_SRC scheme (define-record-type* opensmtpd-option-configuration make-opensmtpd-option-configuration opensmtpd-option-configuration? (documentation (list " uses to\n" "tweak various options.")) (sanitize (sanitize-configuration ; this sanitizes the whole record. (list (lambda (value) ... )))) (option opensmtpd-option-configuration-option (default #f) (sanitize (sanitize-configuration (list string?)))) (not opensmtpd-option-configuration-not (default #f) (sanitize (sanitize-configuration (list boolean?)) )) (regex opensmtpd-option-configuration-regex (default #f) (sanitize (sanitize-configuration '(boolean?)))) (value opensmtpd-option-configuration-value (default #f) (sanitize (sanitize-configuration ;; note that it is smart enough to realize that opensmtpd-table-configuration? is a record, ;; so the error message it returns is something like " fieldname is of ;; type ." (list false? string? opensmtpd-table-configuration?))))) #+END_SRC **** TODO maybe sanitize the ~~ fieldname 'matches' better #+BEGIN_SRC scheme (opensmtpd-configuration (matches (list (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "relay")))) 345 (opensmtpd-match-configuration (for (opensmtpd-option-configuration (option "for local"))) (action (opensmtpd-action-relay-configuration (name "relay"))))))) #+END_SRC #+BEGIN_EXAMPLE fieldname: 'matches' is of type a list of unique records. ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `bad!' with args `((#< action: #< name: "relay" backup: #f backup-mx: #f helo: #f domain: #f host: #f pki: #f srs: #f tls: #f protocols: #f ciphers: #f auth: #f mail-from: #f src: #f> for: #f from: #f auth: #f helo: #f mail-from: #f rcpt-to: #f tag: #f tls: #f> 345 #< action: #< name: "relay" backup: #f backup-mx: #f helo: #f domain: #f host: #f pki: #f srs: #f tls: #f protocols: #f ciphers: #f auth: #f mail-from: #f src: #f> for: #< option: "for local" not: #f regex: #f value: #f> from: #f auth: #f helo: #f mail-from: #f rcpt-to: #f tag: #f tls: #f>))'. Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. scheme@(opensmtpd-records) [24]> ,bt In current input: 2358:0 2 (_) In /home/joshua/prog/gnu/guix/guix-config/linode-guix-system-configuration/opensmtpd-records.scm: 278:8 1 (my/sanitize (#< action: #< name: "relay" backup: #f backup-mx: #f helo: #f domain…> …) …) In ice-9/boot-9.scm: 1685:16 0 (raise-exception _ #:continuable? _) #+END_EXAMPLE It is not obvious from the error message what is wrong. The error message should say #+BEGIN_SRC org ~= fieldname 'matches' is a list of unique =~. One of the items in the list is '345'. Throw to key `bad! with args 345 #+END_SRC Alternatively, we could define a guix specific record printer to make it easier to see the problem. #+BEGIN_SRC scheme fieldname: 'matches' is of type a list of unique records. ice-9/boot-9.scm:1685:16: In procedure raise-exception: Throw to key `bad!' with args: `(list (opensmtpd-match-configuration (action (opensmtpd-action-relay-configuration (name "relay")))) 345 (opensmtpd-match-configuration (for (opensmtpd-option-configuration (option "for local"))) (action (opensmtpd-action-relay-configuration (name "relay")))))'. Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue. scheme@(opensmtpd-records) [24]> ,bt In current input: 2358:0 2 (_) In /home/joshua/prog/gnu/guix/guix-config/linode-guix-system-configuration/opensmtpd-records.scm: 278:8 1 (my/sanitize (#< action: #< name: "relay" backup: #f backup-mx: #f helo: #f domain…> …) …) In ice-9/boot-9.scm: 1685:16 0 (raise-exception _ #:continuable? _) #+END_SRC **** TODO some of the error messages say "bad var #f". This is not very helpful. Where it is useful I should do a ~(throw 'bad! record)~ instead of ~(throw `bad! #f)~ *** PROJ add support for ~= fieldname 'senders': syntax "senders =~ [masquerade]" [0/4] :LOGBOOK: - State "TODO" from [2021-11-02 Tue 04:08] :END: **** TODO add a record type fieldnames: 'table (accepts '), and 'masquerade' (accepts boolean). **** TODO change the sanitize portion of the fieldname 'senders' in the The below code does work in a REPL. #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "My-table") (data '(("joshua" . "$some$Long$EncrytpedPassword")))))) #+END_SRC #+RESULTS: AND the below code will correctly result in an error! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "My-table") (data '("joshua" "$some$Long$EncrytpedPassword"))))) #+END_SRC #+RESULTS: : ~= fieldname: 'auth' is of type boolean, or an =~ record whose fieldname 'values' are an assoc-list : (eg: (opensmtpd-table-configuration (name "table") (data '("joshua" . "$encrypted$password")))). **** TODO change relevant portions in opensmtpd-listen-on-configuration->string This bit of code works in the repl too! #+BEGIN_SRC scheme :dir ~/prog/gnu/guix-config/linode-guix-system-configuration/ (add-to-load-path (dirname (current-filename))) (use-modules (opensmtpd-records)) ((@@ (opensmtpd-records) opensmtpd-listen-on-configuration->string) (opensmtpd-listen-on-configuration (auth (opensmtpd-table-configuration (name "credentials") (data '(("joshua" . "$someLongEncrytpedPassword"))))))) #+END_SRC **** TODO support the masquerade option Right now, senders just accepts an , but I am not allowing the user to turn on or off the masquerade option. *** TODO write out of examples of ~~ records that will fail to start or do not make sense. Provide appropriate error messages. There is a trend of guix services that "work" but are not dummy proof. For example, the XMPP service wants a cert in the format of "~ record and just by looking at the record tell, if you have done something silly that will make the service refuse to start or behave in a weird way, and provide you appropriate error messages so you don't have to go syntax hunting. Examples: - could be a filter that is defined but never used, which won't be possible once [[id:89603b3f-7580-4531-8aee-2c115c97adfe][remove opensmtpd-configuration-filters]] is done. - (listen-on (interface "doesNotExist")) - (smtp-configuration (smtp-max-message-size "10G")) Are you sure you want emails that large? - (pki (domain "name") (key "notAKeyfile.txt") (cert "notACertFile.txt") - (ca (file "NotACaFile.txt")) - (opensmtpd-filter-phase-configuration (name "filter") (phase "helo") (decision "bypass")) There is no fieldname =options= here. This has to be sanitized by ~~'s fieldname 'filters'. #+BEGIN_SRC scheme (opensmtpd-listen-on-configuration (filters (list (opensmtpd-filter-phase-configuration (name "noFRDNS") (phase "commit") (options (list (opensmtpd-option-configuration (option "fcrdns") ))) (decision "junk"))))) #+END_SRC * Some notes on working on the service workflows and such disabling centaur-tabs-mode seems to help. and NOT working in the console helps too. I think that having the geiser repl running via M-x geiser M-x geiser-load-file RET opensmtpd-records.scm ,m (opensmtpd-records) May be causing Emacs to move slowly after a while. It may be better to instead do: cd prog/gnu/guix-config/linode-system-configuration; guile -L . --listen=9999 And then in Emacs (as described here: https://www.nongnu.org/geiser/geiser_3.html) connect to the external repl via M-x geiser-connect