From: Andy Wingo <wingo@pobox.com>
To: Neil Jerram <neil@ossau.homelinux.net>
Cc: guile-devel <guile-devel@gnu.org>
Subject: Re: byte-order marks
Date: Tue, 29 Jan 2013 22:09:38 +0100 [thread overview]
Message-ID: <87y5fb1yst.fsf@pobox.com> (raw)
In-Reply-To: <878v7biykc.fsf@neil-laptop.ossau.uklinux.net> (Neil Jerram's message of "Tue, 29 Jan 2013 19:22:43 +0000")
On Tue 29 Jan 2013 20:22, Neil Jerram <neil@ossau.homelinux.net> writes:
> (define (read-csv file-name)
> (let ((s (utf16->string (get-bytevector-all (open-input-file file-name))
> 'little)))
>
> ;; Discard possible byte order mark.
> (if (and (>= (string-length s) 1)
> (char=? (string-ref s 0) #\xfeff))
> (set! s (substring s 1)))
>
> ...))
FWIW the procedure I had was:
(define (consume-byte-order-mark port)
(let ((enc (or (port-encoding port) "ISO-8859-1")))
(set-port-encoding! port "ISO-8859-1")
(case (peek-char port)
((#\xEF)
(read-char port)
(case (peek-char port)
((#\xBB)
(read-char port)
(case (peek-char port)
((#\xBF)
(read-char port)
(set-port-encoding! port "UTF-8"))
(else
(unread-char #\xBB port)
(unread-char #\xEF port)
(set-port-encoding! port enc))))
(else
(unread-char #\xEF port)
(set-port-encoding! port enc))))
((#\xFE)
(read-char port)
(case (peek-char port)
((#\xFF)
(read-char port)
(set-port-encoding! port "UTF-16BE"))
(else
(unread-char #\xFE port)
(set-port-encoding! port enc))))
((#\xFF)
(read-char port)
(case (peek-char port)
((#\xFE)
(read-char port)
(set-port-encoding! port "UTF-16LE"))
(else
(unread-char #\xFF port)
(set-port-encoding! port enc))))
(else
(set-port-encoding! port enc)))))
The encoding dance is because there is no unread-u8 from Scheme, only
unread-char.
Andy
--
http://wingolog.org/
next prev parent reply other threads:[~2013-01-29 21:09 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-28 21:42 byte-order marks Andy Wingo
2013-01-28 22:20 ` Mike Gran
2013-01-29 9:03 ` Andy Wingo
2013-01-29 8:22 ` Mark H Weaver
2013-01-29 9:03 ` Andy Wingo
2013-01-29 13:27 ` Ludovic Courtès
2013-01-29 14:04 ` Andy Wingo
2013-01-29 17:09 ` Mark H Weaver
2013-01-29 19:09 ` Mark H Weaver
2013-01-29 20:52 ` Ludovic Courtès
2013-01-29 20:53 ` Ludovic Courtès
2013-01-30 9:20 ` Andy Wingo
2013-01-30 21:18 ` Ludovic Courtès
2013-01-31 8:52 ` Andy Wingo
2013-01-31 4:40 ` [PATCHES] Discard BOMs at stream start for UTF-{8,16,32} encodings Mark H Weaver
2013-01-31 9:39 ` Andy Wingo
2013-01-31 10:33 ` Andy Wingo
2013-01-31 18:01 ` [PATCHES] Discard BOMs at stream start for UTF-{8, 16, 32} encodings Mark H Weaver
2013-01-31 21:42 ` Ludovic Courtès
2013-01-29 19:22 ` byte-order marks Neil Jerram
2013-01-29 21:09 ` Andy Wingo [this message]
2013-01-29 21:12 ` Neil Jerram
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87y5fb1yst.fsf@pobox.com \
--to=wingo@pobox.com \
--cc=guile-devel@gnu.org \
--cc=neil@ossau.homelinux.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).