"split-sentences"?

"split-sentences"?

[moasenwood--- via Users list for the GNU Emacs text editor]

Can I parse/split a string into sentences based on
human-language punctuation?

Did anyone do that already?

TIA

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: "split-sentences"?

[moasenwood--- via Users list for the GNU Emacs text editor]

moasenwood--- via Users list for the GNU Emacs text editor wrote:

> Can I parse/split a string into sentences based on
> human-language punctuation?
>
> Did anyone do that already?

I mean very mechanically is fine, no linguistics or anything.

So this

"'This sentence is spoken by Mr. W. E. B Dubois, Esq.!' played
through amazon.com alexa speakers?"

would be

("'" "This sentence is spoken by Mr" "." "W" "." "E" "." "B
Dubois" "," "Esq" "." "!" "'" "played through amazon" "."
"com" "alexa "speakers" "?")

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: "split-sentences"?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On Sat, Jan 23, 2021 at 07:38:49AM +0100, moasenwood--- via Users list for the GNU Emacs text editor wrote:
> moasenwood--- via Users list for the GNU Emacs text editor wrote:
> 
> > Can I parse/split a string into sentences based on
> > human-language punctuation?
> >
> > Did anyone do that already?
> 
> I mean very mechanically is fine, no linguistics or anything.
> 
> So this
> 
> "'This sentence is spoken by Mr. W. E. B Dubois, Esq.!' played
> through amazon.com alexa speakers?"
>
> would be
> 
> ("'" "This sentence is spoken by Mr" "." "W" "." "E" "." "B
> Dubois" "," "Esq" "." "!" "'" "played through amazon" "."
> "com" "alexa "speakers" "?")

Not exactly your result, but this comes close:

  (split-string
    "'This sentence is spoken by Mr. W. E. B Dubois, Esq.!' played through amazon.com alexa speakers?"
    "[[:punct:]][[:space:]]*")

=>

  (""
   "This sentence is spoken by Mr"
   "W"
   "E"
   "B Dubois"
   "Esq"
   ""
   ""
   "played through amazon"
   "com alexa speakers"
   "")

You can adjust the results by tweaking the regexp (try word
boundaries like '\<' and '\>' if you want to keep punctuation)
or the other split-string's optional params (e.g. drop the
empty matches, etc.).

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: "split-sentences"?

[Tomas Hlavaty]

On Sat 23 Jan 2021 at 09:41, <tomas@tuxteam.de> wrote:
> On Sat, Jan 23, 2021 at 07:38:49AM +0100, moasenwood--- via Users list for the GNU Emacs text editor wrote:
>> Can I parse/split a string into sentences based on
>> human-language punctuation?

not easily

>> Did anyone do that already?

https://www.unicode.org/reports/tr29/#Sentence_Boundaries

Does emacs expose unicode text functions?  For example to classify
characters, determine graphemes, words, sentences, line breaks etc?

>> I mean very mechanically is fine, no linguistics or anything.
>> 
>> So this
>> 
>> "'This sentence is spoken by Mr. W. E. B Dubois, Esq.!' played
>> through amazon.com alexa speakers?"
>>
>> would be
>> 
>> ("'" "This sentence is spoken by Mr" "." "W" "." "E" "." "B
>> Dubois" "," "Esq" "." "!" "'" "played through amazon" "."
>> "com" "alexa "speakers" "?")

That is not really split-sentences.

The example has two sentences.  Moreover the first sentence is a subject
of the second.

This would be represented something like this:

(sentence
  (sentence "This sentence is spoken by Mr. W. E. B Dubois, Esq.!")
  "played through amazon.com alexa speakers?")

but it depends, what do you want to achieve.

Re: "split-sentences"?

[moasenwood--- via Users list for the GNU Emacs text editor]

Tomas Hlavaty wrote:

>> I mean very mechanically is fine, no linguistics
>> or anything.
>> 
>> So this
>> 
>> "'This sentence is spoken by Mr. W. E. B Dubois, Esq.!'
>> played through amazon.com alexa speakers?"
>>
>> would be
>> 
>> ("'" "This sentence is spoken by Mr" "." "W" "." "E" "." "B
>> Dubois" "," "Esq" "." "!" "'" "played through amazon" "."
>> "com" "alexa "speakers" "?")
>
> That is not really split-sentences.
>
> The example has two sentences.  Moreover the first sentence is a subject
> of the second.
>
> This would be represented something like this:
>
> (sentence
>   (sentence "This sentence is spoken by Mr. W. E. B Dubois, Esq.!")
>   "played through amazon.com alexa speakers?")
>
> but it depends, what do you want to achieve.

I found this post, which even provides an example:

  https://lists.gnu.org/archive/html/help-gnu-emacs/2021-01/msg00385.html

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: "split-sentences"?

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Date: Sat, 23 Jan 2021 10:07:06 +0100
> 
> Does emacs expose unicode text functions?

It does expose some of them, although not necessarily under the names
used by the UCS.

> For example to classify characters, determine graphemes, words,
> sentences, line breaks etc?

We have get-char-code-property for Unicode character properties and
find-composition for finding grapheme clusters (Emacs doesn't care
about graphemes, unless you use these two terms as aliases).  For
words, sentences, and line breaks, we use our own definitions, and
generally don't support the Unicode delimiters like U+2028.

Re: "split-sentences"?

[Tomas Hlavaty]

On Sat 23 Jan 2021 at 11:48, Eli Zaretskii <eliz@gnu.org> wrote:
> We have get-char-code-property for Unicode character properties and
> find-composition for finding grapheme clusters (Emacs doesn't care
> about graphemes, unless you use these two terms as aliases).  For
> words, sentences, and line breaks, we use our own definitions, and
> generally don't support the Unicode delimiters like U+2028.

thanks, I'll have a look

Re: "split-sentences"?

[moasenwood--- via Users list for the GNU Emacs text editor]

tomas wrote:

> Not exactly your result, but this comes close:
>
>   (split-string "'This sentence is spoken by Mr. W. E.
>     B Dubois, Esq.!' played through amazon.com alexa
>     speakers?" "[[:punct:]][[:space:]]*")
>
> =>
>
>   (""
>    "This sentence is spoken by Mr"
>    "W"
>    "E"
>    "B Dubois"
>    "Esq"
>    ""
>    ""
>    "played through amazon"
>    "com alexa speakers"
>    "")
>
> You can adjust the results by tweaking the regexp (try word
> boundaries like '\<' and '\>'

*scratches my head*

> if you want to keep punctuation) or the other split-string's
> optional params (e.g. drop the empty matches, etc.).

Well, that's a start, for sure. Thanks :)

Silly me, I already used `split-string' 10 times...

-- 
underground experts united
http://user.it.uu.se/~embe8573
https://dataswamp.org/~incal

Re: "split-sentences"?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

On Sat, Jan 23, 2021 at 10:35:51AM +0100, moasenwood--- via Users list for the GNU Emacs text editor wrote:
> tomas wrote:
> 
> > Not exactly your result, but this comes close:

[...]

> > You can adjust the results by tweaking the regexp (try word
> > boundaries like '\<' and '\>'
> 
> *scratches my head*

A candidate for a sentence boundary is a word boundary
(plus some other conditions). This was at least my thought
process leading to that suggestion. It might be a bad
suggestion, though.

> > if you want to keep punctuation) or the other split-string's
> > optional params (e.g. drop the empty matches, etc.).
> 
> Well, that's a start, for sure. Thanks :)

You're welcome. Note that [:punct:] may be too broad a category:
does a sentence end with a comma? A semi-colon? A colon? What
about question and exclamation marks? What about the latter in
a language like Spanish, where they're parenthetical: "Ella
me preguntó ¿qué quieres?" (the parenthetical things make it
much easier to embed a question or an exclamation into something
else).

As always, the really interesting questions are left as exercises to
the reader... until you end with Natural Language Processing :-)

Possibly this is the danger Tomas Hlavaty is hinting at elsethread.

> Silly me, I already used `split-string' 10 times...

C'm on. Wetware caches are like that. Mine too.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: "split-sentences"?

[Eric Abrahamsen]

<tomas@tuxteam.de> writes:

> On Sat, Jan 23, 2021 at 10:35:51AM +0100, moasenwood--- via Users list
> for the GNU Emacs text editor wrote:
>> tomas wrote:
>> 
>> > Not exactly your result, but this comes close:
>
> [...]
>
>> > You can adjust the results by tweaking the regexp (try word
>> > boundaries like '\<' and '\>'
>> 
>> *scratches my head*
>
> A candidate for a sentence boundary is a word boundary
> (plus some other conditions). This was at least my thought
> process leading to that suggestion. It might be a bad
> suggestion, though.

You can see what Emacs thinks would make a good sentence boundary with
the (sentence-end) function.

Re: "split-sentences"?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

On Sat, Jan 23, 2021 at 09:46:40AM -0800, Eric Abrahamsen wrote:
> <tomas@tuxteam.de> writes:

[...]

> > A candidate for a sentence boundary [...]

> You can see what Emacs thinks would make a good sentence boundary with
> the (sentence-end) function.

Wow-wow-wow! Even more functions! Thanks for the hint :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]