From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
To: Utkarsh Singh <utkarsh190601@gmail.com>
Cc: 47885@debbugs.gnu.org, emacs-orgmode@gnu.org
Subject: Re: [PATCH] org-table-import: Make it more smarter for interactive use
Date: Tue, 20 Apr 2021 15:40:12 +0200 [thread overview]
Message-ID: <87im4h9irn.fsf@nicolasgoaziou.fr> (raw)
In-Reply-To: <87k0oyfj4y.fsf@gmail.com> (Utkarsh Singh's message of "Mon, 19 Apr 2021 19:53:25 +0530")
Hello,
Utkarsh Singh <utkarsh190601@gmail.com> writes:
> At first I was also reluctant in creating a new function but decided to
> do so because:
>
> + org-table-convert-region is currently doing two thing 'guessing the
> separator' and 'converting the region'. I thought it was a good idea to
> separate out function into it's atomic operations.
I understand, but there is sometimes a (difficult) line to draw between
"separating concerns" and "function proliferation". Anyway, that's fine
here.
> + Current guessing technique is quite basic as it assumes that data
> (file that has to be imported) has no error/inconsistency in it. I
> would like to show you the doc string of Python's CSV library
> implementation to guess separator (region inside """):
>
> """
> Looks for text enclosed between two identical quotes
> (the probable quotechar) which are preceded and followed
> by the same character (the probable delimiter).
> For example:
> ,'some text',
> The quote with the most wins, same with the delimiter.
> If there is no quotechar the delimiter can't be determined
> this way.
> """
>
> And if this functions fails then we have:
>
> """
> The delimiter /should/ occur the same number of times on
> each row. However, due to malformed data, it may not. We don't want
> an all or nothing approach, so we allow for small variations in this
> number.
> 1) build a table of the frequency of each character on every line.
> 2) build a table of frequencies of this frequency (meta-frequency?),
> e.g. 'x occurred 5 times in 10 rows, 6 times in 1000 rows,
> 7 times in 2 rows'
> 3) use the mode of the meta-frequency to determine the /expected/
> frequency for that character
> 4) find out how often the character actually meets that goal
> 5) the character that best meets its goal is the delimiter
> For performance reasons, the data is evaluated in chunks, so it can
> try and evaluate the smallest portion of the data possible, evaluating
> additional chunks as necessary.
> """
For the problem we're trying to solve, this sounds like over-engineering
to me. Do we want so badly to guess a separator?
> I tried to do similar in Elisp but currently facing some issues due to
> my inexperience in functional programming. Also moving the 'guessing'
> part out the function may lead to development of even better algorithm
> than Python counterpart.
>
> Modified version of concerned function:
>
> (defun org-table-guess-separator (beg0 end0)
> "Guess separator for `org-table-convert-region' for region BEG0 to END0.
>
> List of preferred separator:
> comma, TAB, semicolon, colon or SPACE.
>
> If region contains a line which doesn't contain the required
> separator then discard the separator and search again using next
> separator."
> (let* ((beg (save-excursion
> (goto-char (min beg0 end0))
> (line-beginning-position)))
> (end (save-excursion
> (goto-char (max beg0 end0))
> (line-end-position)))
Thinking again about it, this needs extra care, as end0 might end up on
an empty line. You tried to avoid this in your first function, but
I think this was not sufficient either. Actually, beg0 could also start
on an empty line.
This needs to be tested extensively, but as a first approximation,
I think `beg' needs to be defined as:
(save-excursion
(goto-char (min beg0 end0))
(skip-chars-forward " \t\n")
(if (eobp) (point) (line-beginning-position)))
and `end' as
(save-excursion
(goto-char (max beg end0))
(skip-chars-backward " \t\n" beg)
(if (= beg (point)) (point) (line-end-position)))
Then you need to bail out if beg = end.
> (sep-rexp '(("," "^[^\n,]+$")
sep-rexp -> sep-regexp
> ("\t" "^[^\n\t]+$")
> (";" "^[^\n;]+$")
> (":" "^[^\n:]+$")
> (" " "^\\([^'\"][^\n\s][^'\"]\\)+$")))
At this point, I suggest to use `rx' macro instead.
> (tmp (car sep-rexp))
> sep)
> (save-excursion
> (goto-char beg)
> (while (and (not sep)
> (if (save-excursion
> (not (re-search-forward (nth 1 tmp) end t)))
> (setq sep (nth 0 tmp))
> (setq sep-rexp (cdr sep-rexp))
> (setq tmp (car sep-rexp)))))
I suggest this (yes, I like pattern-matching, `car' and `cdr' are so
80's) instead:
(save-excursion
(goto-char beg)
(catch :found
(pcase-dolist (`(,sep ,regexp) sep-regexp)
(save-excursion
(unless (re-search-forward regexp end t)
(throw :found sep))))
nil))
Again all this needs to extensively tested, as there are a lot of
dangers lurking around.
Regards,
--
Nicolas Goaziou
next prev parent reply other threads:[~2021-04-20 13:41 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-19 4:43 [PATCH] org-table-import: Make it more smarter for interactive use Utkarsh Singh
2021-04-19 8:19 ` Nicolas Goaziou
2021-04-19 14:23 ` Utkarsh Singh
2021-04-20 13:40 ` Nicolas Goaziou [this message]
2021-04-20 17:15 ` Utkarsh Singh
2021-04-23 4:58 ` Utkarsh Singh
2021-04-27 20:21 ` bug#47885: " Nicolas Goaziou
2021-04-28 8:37 ` Utkarsh Singh
2021-04-28 16:38 ` Maxim Nikulin
2021-05-10 18:36 ` Utkarsh Singh
2021-05-12 17:08 ` Maxim Nikulin
2021-05-14 14:54 ` Utkarsh Singh
2021-05-15 9:13 ` Bastien
2021-05-15 10:10 ` Utkarsh Singh
2021-05-15 10:30 ` Bastien
2021-05-15 11:09 ` Utkarsh Singh
2021-05-17 5:29 ` Bastien
2021-05-17 16:27 ` Utkarsh Singh
2021-06-01 16:23 ` Maxim Nikulin
2021-06-01 17:46 ` Utkarsh Singh
2021-06-02 12:06 ` Maxim Nikulin
2021-06-02 15:08 ` Utkarsh Singh
2021-06-02 16:44 ` Maxim Nikulin
2021-06-04 4:04 ` Utkarsh Singh
2021-06-05 12:40 ` Maxim Nikulin
2021-06-05 17:50 ` Utkarsh Singh
2021-06-09 12:15 ` Maxim Nikulin
2021-09-26 8:40 ` Bastien
2021-05-16 16:24 ` Maxim Nikulin
2021-05-17 16:30 ` Utkarsh Singh
2021-05-18 10:24 ` Utkarsh Singh
2021-05-18 12:31 ` Maxim Nikulin
2021-05-18 15:05 ` Utkarsh Singh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87im4h9irn.fsf@nicolasgoaziou.fr \
--to=mail@nicolasgoaziou.fr \
--cc=47885@debbugs.gnu.org \
--cc=emacs-orgmode@gnu.org \
--cc=utkarsh190601@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.