bug#47260: Package GNU MediaGoblin as a Guix service

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

This is a "meta" bug to keep track of the progress of packaging GNU
MediaGoblin, a platform for publishing images/audio/video etc. See
https://mediagoblin.org/

We have a guix-env.scm in the upstream source which should always have
the latest copy of our packaging progress and instructions to run it:

https://git.savannah.gnu.org/cgit/mediagoblin.git/tree/guix-env.scm

Current plan is:

1. Add OGG support to libsndfile which is needed to package
python-soundfile [patch 47210]

2. Package python-soundfile (see above). After this the test suite
should pass 100% with pytest installed from PyPI [patch 47181]

3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
in Guix seem to be incompatible. After this our test suite should run
100% with only dependencies from Guix!

4. Package MediaGoblin itself. The build process is ./configure/make
which is a bit weird for a Python project.

5. Get a basic Guix service working, with sqlite3 and without the
offloaded media transcoding currently using Celery/RabbitMQ.

6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
improve the no-bundled-JavaScript video/audio playing experience.

7. Work out why H264 support is missing.

8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
implicated in many bugs anyway, so there may benefits to the project to
doing this anyway.

9. Figure out how to deal with translations.

10. Add a PostgreSQL database to the Guix service instead of sqlite3.

11. We win. Maybe :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

bug#47260: Package GNU MediaGoblin as a Guix service

[jgart via Bug reports for GNU Guix]

This sounds like a great project. I would love MediaGoblin to be in Guix also.

> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
> improve the no-bundled-JavaScript video/audio playing experience.

What are your thoughts on rewriting the jquery? 

Should MediaGoblin be using vanilla javascript instead?

Some other possibilities could be purescript (https://www.purescript.org) or mint (http://mint-lang.com), although mint and crystal are not in guix yet and mint uses preact (http://preactjs.com) as its' runtime since 0.8.0 (https://www.mint-lang.com/blog/mint-0.8.0).

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Fri, 19 Mar 2021, jgart wrote:

> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>
>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>> improve the no-bundled-JavaScript video/audio playing experience.
>
> What are your thoughts on rewriting the jquery? 
>
> Should MediaGoblin be using vanilla javascript instead?
>
> Some other possibilities could be purescript (https://www.purescript.org) or
> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
> (https://www.mint-lang.com/blog/mint-0.8.0).

Many of the functions we used to use jQuery for are now built into most
browsers from the last 10 years.

By far the most common are element selectors like:

  var panel = $('#header-panel')
  var arrow = $('.arrow')

which we just change to:

  var panel = document.querySelector('#header-panel')
  var arrow = document.querySelector('.arrow')

This is all vanilla JavaScript in individual files imported via <script>
without any sort of compilation process. This seems to be the simplest
path to get into distributions like Guix and Debian. The main problem
with purescript and others is that they require more dependencies, build
steps and tooling and are known by less people.

We have a few dependencies which will be harder to deal with such as the
video, audio and 3D model viewers. In the first instance, I think the
answer will be to do without this JavaScript entirely. That should be
feasible for video and audio given modern browser support. Perhaps we
don't ship 3D model support out of the box in distros for now.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2615 bytes --]


Ben Sturmfels via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> On Fri, 19 Mar 2021, jgart wrote:
>
>> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>>
>>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>>> improve the no-bundled-JavaScript video/audio playing experience.
>>
>> What are your thoughts on rewriting the jquery? 
>>
>> Should MediaGoblin be using vanilla javascript instead?
>>
>> Some other possibilities could be purescript (https://www.purescript.org) or
>> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
>> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
>> (https://www.mint-lang.com/blog/mint-0.8.0).
>
> Many of the functions we used to use jQuery for are now built into most
> browsers from the last 10 years.
>
> By far the most common are element selectors like:
>
>   var panel = $('#header-panel')
>   var arrow = $('.arrow')
>
> which we just change to:
>
>   var panel = document.querySelector('#header-panel')
>   var arrow = document.querySelector('.arrow')
>
> This is all vanilla JavaScript in individual files imported via <script>
> without any sort of compilation process. This seems to be the simplest
> path to get into distributions like Guix and Debian. The main problem
> with purescript and others is that they require more dependencies, build
> steps and tooling and are known by less people.
>
> We have a few dependencies which will be harder to deal with such as the
> video, audio and 3D model viewers. In the first instance, I think the
> answer will be to do without this JavaScript entirely. That should be
> feasible for video and audio given modern browser support. Perhaps we
> don't ship 3D model support out of the box in distros for now.

If you need support for m3u-playlists, you can use the player I wrote
here: https://www.draketo.de/software/m3u-player
→ https://www.draketo.de/software/m3u-player.js (save as utf-8)
(that m3u-playlists aren’t supported out of the box in most players is a
strange oversight, the code adds it for video- and audio-tags, License:
GPLv2 or later — just ask me if you need something else)

There’s also an enhanced version for Freenet, but that has lots of
performance-changes to work over high-latency networks and with paranoid
CSP-settings:
https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Mon, 22 Mar 2021, Dr. Arne Babenhauserheide wrote:

> If you need support for m3u-playlists, you can use the player I wrote
> here: https://www.draketo.de/software/m3u-player
> → https://www.draketo.de/software/m3u-player.js (save as utf-8)
> (that m3u-playlists aren’t supported out of the box in most players is a
> strange oversight, the code adds it for video- and audio-tags, License:
> GPLv2 or later — just ask me if you need something else)
>
> There’s also an enhanced version for Freenet, but that has lots of
> performance-changes to work over high-latency networks and with paranoid
> CSP-settings:
> https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802

Thanks Arne! I've forwarded this on to mediagoblin-devel@gnu.org so we
don't lose track of it.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 11277 bytes --]


Ben Sturmfels <ben@sturm.com.au> writes:

> On Mon, 22 Mar 2021, Dr. Arne Babenhauserheide wrote:
>
>> If you need support for m3u-playlists, you can use the player I wrote
>> here: https://www.draketo.de/software/m3u-player
>> → https://www.draketo.de/software/m3u-player.js (save as utf-8)
>> (that m3u-playlists aren’t supported out of the box in most players is a
>> strange oversight, the code adds it for video- and audio-tags, License:
>> GPLv2 or later — just ask me if you need something else)
>>
>> There’s also an enhanced version for Freenet, but that has lots of
>> performance-changes to work over high-latency networks and with paranoid
>> CSP-settings:
>> https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802
>
> Thanks Arne! I've forwarded this on to mediagoblin-devel@gnu.org so we
> don't lose track of it.

Thank you!

I added one change last week to support mobile browsers which answer
"maybe" to the query `mediaTag.canPlayType('audio/x-mpegurl')` (yes,
seriously, and it is in the spec :-) ).

Also I backported the not freenet specific changes:
- prefetch the next three tracks as blob and keep at most 10 tracks
  cached to allow for fast track skipping (and now actually release the
  memory)
- adjustments to allow for inlining and survive the non-utf8-encoding.
- continue automatically when fetch succeeded if playback was stopped
  because it reached the end (but not if paused).
- minimal mouseover for the back and forward arrows.

When a https-m3u-list refers to a http-file, it falls back from fetching
blobs to rewriting the src-part of the tag (because blobs cannot be
fetched from a less secure resource).

This is how it looks: https://www.draketo.de/software/m3u-player.html

The changes are included in https://www.draketo.de/software/m3u-player.js

You can use it like this:

<script src="m3u-player.js" defer="defer"></script>
<audio src="m3u-player-example-playlist.m3u" controls="controls">
not supported?
</audio>


To make this bug-report independent of my site, here’s the full code:



// [[file:m3u-player.org::*The script][The script:1]]
// @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2-or-Later
const nodes = document.querySelectorAll("audio,video");
const playlists = {};
const prefetchedTracks = new Map(); // use a map for insertion order, so we can just blow away old entries.
// maximum prefetched blobs that are kept.
const MAX_PREFETCH_KEEP = 10;
// maximum allowed number of entries in a playlist to prevent OOM attacks against the browser with self-referencing playlists
const MAX_PLAYLIST_LENGTH = 1000;
const PLAYLIST_MIME_TYPES = ["audio/x-mpegurl", "audio/mpegurl", "application/vnd.apple.mpegurl","application/mpegurl","application/x-mpegurl"];
function stripUrlParameters(link) {
  const url = new URL(link, window.location);
  url.search = "";
  url.hash = "";
  return url.href;
}
function isPlaylist(link) {
  const linkHref = stripUrlParameters(link);
  return linkHref.endsWith(".m3u") || linkHref.endsWith(".m3u8");
}
function isBlob(link) {
  return new URL(link, window.location).protocol == 'blob';
}
function parsePlaylist(textContent) {
  return textContent.match(/^(?!#)(?!\s).*$/mg)
    .filter(s => s); // filter removes empty strings
}
/**
 * Download the given playlist, parse it, and store the tracks in the
 * global playlists object using the url as key.
 *
 * Runs callback once the playlist downloaded successfully.
 */
function fetchPlaylist(url, onload, onerror) {
  const playlistFetcher = new XMLHttpRequest();
  playlistFetcher.open("GET", url, true);
  playlistFetcher.responseType = "blob"; // to get a mime type
  playlistFetcher.onload = () => {
    if (PLAYLIST_MIME_TYPES.includes(playlistFetcher.response.type)) { // security check to ensure that filters have run
      const reader = new FileReader();
      const load = onload; // propagate to inner scope
      reader.addEventListener("loadend", e => {
        playlists[url] = parsePlaylist(reader.result);
        onload();
      });
      reader.readAsText(playlistFetcher.response);
    } else {
      console.error("playlist must have one of the playlist MIME type '" + PLAYLIST_MIME_TYPES + "' but it had MIME type '" + playlistFetcher.response.type + "'.");
      onerror();
    }
  };
  playlistFetcher.onerror = onerror;
  playlistFetcher.abort = onerror;
  playlistFetcher.send();
}
function prefetchTrack(url, onload) {
  if (prefetchedTracks.has(url)) {
    return;
  }
  // first cleanup: kill the oldest entries until we're back at the allowed size
  while (prefetchedTracks.size > MAX_PREFETCH_KEEP) {
    const key = prefetchedTracks.keys().next().value;
    const track = prefetchedTracks.get(key);
    prefetchedTracks.delete(key);
  }
  // first set the prefetched to the url so we will never request twice
  prefetchedTracks.set(url, url);
  // now start replacing it with a blob
  const xhr = new XMLHttpRequest();
  xhr.open("GET", url, true);
  xhr.responseType = "blob";
  xhr.onload = () => {
    prefetchedTracks.set(url, xhr.response);
    if (onload) {
      onload();
    }
  };
  xhr.send();
}
function updateSrc(mediaTag, callback) {
  const playlistUrl = mediaTag.getAttribute("playlist");
  const trackIndex =  mediaTag.getAttribute("track-index");
  // deepcopy playlists to avoid shared mutation
  let playlist = [...playlists[playlistUrl]];
  let trackUrl = playlist[trackIndex];
  // download and splice in playlists as needed
  if (isPlaylist(trackUrl)) {
    if (playlist.length >= MAX_PLAYLIST_LENGTH) {
      // skip playlist if we already have too many tracks
      changeTrack(mediaTag, +1);
    } else {
      // do not use the cached playlist here, though it is tempting: it might genuinely change to allow for updates
      fetchPlaylist(
        trackUrl,
        () => {
          playlist.splice(trackIndex, 1, ...playlists[trackUrl]);
          playlists[playlistUrl] = playlist;
          updateSrc(mediaTag, callback);
        },
        () => callback());
    }
  } else {
    let url = prefetchedTracks.has(trackUrl)
        ? prefetchedTracks.get(trackUrl) instanceof Blob
        ? URL.createObjectURL(prefetchedTracks.get(trackUrl))
        : trackUrl : trackUrl;
    const oldUrl = mediaTag.getAttribute("src");
    mediaTag.setAttribute("src", url);
    // replace the url when done, because a blob from an xhr request
    // is more reliable in the media tag;
    // the normal URL caused jumping prematurely to the next track.
    if (url == trackUrl) {
      prefetchTrack(trackUrl, () => {
        if (mediaTag.paused) {
          if (url == mediaTag.getAttribute("src")) {
            if (mediaTag.currentTime === 0) {
              mediaTag.setAttribute("src", URL.createObjectURL(
                prefetchedTracks.get(url)));
            }
          }
        }
      });
    }
    // allow releasing memory
    if (isBlob(oldUrl)) {
      URL.revokeObjectURL(oldUrl);
    }
    // update title
    mediaTag.parentElement.querySelector(".m3u-player--title").title = trackUrl;
    mediaTag.parentElement.querySelector(".m3u-player--title").textContent = trackUrl;
    // start prefetching the next three tracks.
    for (const i of [1, 2, 3]) {
      if (playlist.length > Number(trackIndex) + i) {
        prefetchTrack(playlist[Number(trackIndex) + i]);
      }
    }
    callback();
  }
}
function changeTrack(mediaTag, diff) {
  const currentTrackIndex = Number(mediaTag.getAttribute("track-index"));
  const nextTrackIndex = currentTrackIndex + diff;
  const tracks = playlists[mediaTag.getAttribute("playlist")];
  if (nextTrackIndex >= 0) { // do not collapse the if clauses with double-and, that does not survive inlining
    if (tracks.length > nextTrackIndex) {
    mediaTag.setAttribute("track-index", nextTrackIndex);
      updateSrc(mediaTag, () => mediaTag.play());
    }
  }
}

/**
 * Turn a media tag into playlist player.
 */
function initPlayer(mediaTag) {
  mediaTag.setAttribute("playlist", mediaTag.getAttribute("src"));
  mediaTag.setAttribute("track-index", 0);
  const url = mediaTag.getAttribute("playlist");
  const wrapper = mediaTag.parentElement.insertBefore(document.createElement("div"), mediaTag);
  const controls = document.createElement("div");
  const left = document.createElement("span");
  const title = document.createElement("span");
  const right = document.createElement("span");
  controls.appendChild(left);
  controls.appendChild(title);
  controls.appendChild(right);
  left.classList.add("m3u-player--left");
  right.classList.add("m3u-player--right");
  title.classList.add("m3u-player--title");
  title.style.overflow = "hidden";
  title.style.textOverflow = "ellipsis";
  title.style.whiteSpace = "nowrap";
  title.style.opacity = "0.3";
  title.style.direction = "rtl"; // for truncation on the left
  title.style.paddingLeft = "0.5em";
  title.style.paddingRight = "0.5em";
  controls.style.display = "flex";
  controls.style.justifyContent = "space-between";
  const styleTag = document.createElement("style");
  styleTag.innerHTML = ".m3u-player--left:hover, .m3u-player--right:hover {color: wheat; background-color: DarkSlateGray}";
  wrapper.appendChild(styleTag);
  wrapper.appendChild(controls);
  controls.style.width = mediaTag.getBoundingClientRect().width.toString() + "px";
  // appending the media tag to the wrapper removes it from the outer scope but keeps the event listeners
  wrapper.appendChild(mediaTag);
  left.innerHTML = "&lt;"; // not textContent, because we MUST escape
                           // the tag here and textContent shows the
                           // escaped version
  left.onclick = () => changeTrack(mediaTag, -1);
  right.innerHTML = "&gt;";
  right.onclick = () => changeTrack(mediaTag, +1);
  fetchPlaylist(
    url,
    () => {
      updateSrc(mediaTag, () => null);
      mediaTag.addEventListener("ended", event => {
        if (mediaTag.currentTime >= mediaTag.duration) {
          changeTrack(mediaTag, +1);
        }
      });
    },
    () => null);
  // keep the controls aligned to the media tag
  mediaTag.resizeObserver = new ResizeObserver(entries => {
    controls.style.width = entries[0].contentRect.width.toString() + "px";
  });
  mediaTag.resizeObserver.observe(mediaTag);
}
function processTag(mediaTag) {
  const canPlayClaim = mediaTag.canPlayType('audio/x-mpegurl');
  let supportsPlaylists = !!canPlayClaim;
  if (canPlayClaim == 'maybe') { // yes, seriously: specced as you only know when you try
    supportsPlaylists = false;
  }
  if (!supportsPlaylists) {
    if (isPlaylist(mediaTag.getAttribute("src"))) {
      initPlayer(mediaTag);
    }
  }
}
document.addEventListener('DOMContentLoaded', () => {
  const nodes = document.querySelectorAll("audio,video");
  nodes.forEach(processTag);
});
// @license-end
// The script:1 ends here




Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

bug#47260: Package GNU MediaGoblin as a Guix service

[Christopher Lemmer Webber]

Ben Sturmfels via Bug reports for GNU Guix writes:

> On Fri, 19 Mar 2021, jgart wrote:
>
>> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>>
>>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>>> improve the no-bundled-JavaScript video/audio playing experience.
>>
>> What are your thoughts on rewriting the jquery? 
>>
>> Should MediaGoblin be using vanilla javascript instead?
>>
>> Some other possibilities could be purescript (https://www.purescript.org) or
>> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
>> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
>> (https://www.mint-lang.com/blog/mint-0.8.0).
>
> Many of the functions we used to use jQuery for are now built into most
> browsers from the last 10 years.
>
> By far the most common are element selectors like:
>
>   var panel = $('#header-panel')
>   var arrow = $('.arrow')
>
> which we just change to:
>
>   var panel = document.querySelector('#header-panel')
>   var arrow = document.querySelector('.arrow')
>
> This is all vanilla JavaScript in individual files imported via <script>
> without any sort of compilation process. This seems to be the simplest
> path to get into distributions like Guix and Debian. The main problem
> with purescript and others is that they require more dependencies, build
> steps and tooling and are known by less people.
>
> We have a few dependencies which will be harder to deal with such as the
> video, audio and 3D model viewers. In the first instance, I think the
> answer will be to do without this JavaScript entirely. That should be
> feasible for video and audio given modern browser support. Perhaps we
> don't ship 3D model support out of the box in distros for now.
>
> Regards,
> Ben

The 3d model support isn't MediaGoblin's most heavily used plugin.  Not
shipping support for that would be fine, I'd think.

 - Chris

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

Just an update:

On Fri, 19 Mar 2021, Ben Sturmfels wrote:

> 1. Add OGG support to libsndfile which is needed to package
> python-soundfile [patch 47210]

lfam pointed out that this has already been done in 46067, which is now
in core-updates awaiting merging into master:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46067

For the time being, we've also copied the updated libsndfile into
MediaGoblin's guix-env.scm.

> 2. Package python-soundfile (see above). After this the test suite
> should pass 100% with pytest installed from PyPI [patch 47181]

We now have a local copy of python-soundfile in MediaGoblin's
guix-env.scm which is passing all audio tests. Yay!

> 3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
> in Guix seem to be incompatible. After this our test suite should run
> 100% with only dependencies from Guix!

Discovered we'll also need to upgrade Guix's python-wtforms, but in the
mean time, installing only wtforms, pytest, pytest-xdist, pytest-forked
from PyPI allows us to pass the test suite 100% Getting closer!

> ...
> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
> implicated in many bugs anyway, so there may benefits to the project to
> doing this anyway.

I learnt that Celery has a Redis backend, so maybe we don't need to
rewrite just yet.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Tue, 30 Mar 2021, Ben Sturmfels wrote:

>> 3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
>> in Guix seem to be incompatible. After this our test suite should run
>> 100% with only dependencies from Guix!
>
> Discovered we'll also need to upgrade Guix's python-wtforms, but in the
> mean time, installing only wtforms, pytest, pytest-xdist, pytest-forked
> from PyPI allows us to pass the test suite 100% Getting closer!

Pytest issues have now been resolved. We now need ZERO packages from
PyPI for a basic install and full test suite run. :)

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Tue, 30 Mar 2021, Ben Sturmfels wrote:

> On Fri, 19 Mar 2021, Ben Sturmfels wrote:

>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>> implicated in many bugs anyway, so there may benefits to the project to
>> doing this anyway.
>
> I learnt that Celery has a Redis backend, so maybe we don't need to
> rewrite just yet.

It turns out that MediaGoblin's Celery-based media processing backend
work out of the box by simply configuring:

  [celery]
  BROKER_URL = "redis://"

(There seems to be an unrelated bug where media is marked as failed after
restarting Celery, possibly tied to sqlite. We've had reports of this
with a RabbitMQ broker too though.)


This means our shorter to-do list is now:

1. Upstream our new python-soundfile Guix package from guix-env.scm when
core-updates is merged.

2. Upstream our upgraded python-wtforms package.

6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
Video and audio are essentially functional without the NPM installed
players. Some later refinements perhaps.

4. Package MediaGoblin itself. The build process is ./configure/make
which is a bit weird for a Python project.

5. Get a basic Guix service working, with sqlite3 and without the
offloaded media transcoding currently using Celery task queue with a
Redis broker.

7. Work out why H264 support is missing.

8. Figure out how to deal with translations.

9. Add a PostgreSQL database to the Guix service instead of sqlite3.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Thu, 01 Apr 2021, Ben Sturmfels wrote:

> 7. Work out why H264 support is missing.

This is now fixed MediaGoblin's master branch guix-env.scm by adding
gst-libav to propagated inputs.

bug#47260: Package GNU MediaGoblin as a Guix service

[Léo Le Bouter via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Tue, 2021-04-06 at 00:17 +1000, Ben Sturmfels wrote:
> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
> 
> > 7. Work out why H264 support is missing.
> 
> This is now fixed MediaGoblin's master branch guix-env.scm by adding
> gst-libav to propagated inputs.

Hello!

I suggest not using propagated-inputs because they are likely to cause
conflicts in profiles when used. I suggest intead creating wrapper
scripts that append to the PATH variable.

Examples of such wrapper scripts can be found:

- 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=373e5fc96724fd38bb1263e4af90932ea36f596b
(PYTHONPATH more so)
- 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
(Append xdg-utils to PATH)

Let me know if there's any questions about creating such wrappers.

Léo 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Mon, 05 Apr 2021, Léo Le Bouter wrote:

> On Tue, 2021-04-06 at 00:17 +1000, Ben Sturmfels wrote:
>> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
>> 
>> > 7. Work out why H264 support is missing.
>> 
>> This is now fixed MediaGoblin's master branch guix-env.scm by adding
>> gst-libav to propagated inputs.
>
> Hello!
>
> I suggest not using propagated-inputs because they are likely to cause
> conflicts in profiles when used. I suggest intead creating wrapper
> scripts that append to the PATH variable.
>
> Examples of such wrapper scripts can be found:
>
> - 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=373e5fc96724fd38bb1263e4af90932ea36f596b
> (PYTHONPATH more so)
> - 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
> (Append xdg-utils to PATH)
>
> Let me know if there's any questions about creating such wrappers.

Thanks Léo, I'll look into this!

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1714 bytes --]

On Thu, 01 Apr 2021, Ben Sturmfels wrote:

> 5. Get a basic Guix service working, with sqlite3 and without the
> offloaded media transcoding currently using Celery task queue with a
> Redis broker.

Woo! After a lot of trial and error, I finally have a basic MediaGoblin
running entirely under Guix with no virtualenv trickery!

After applying the attached patch to my guix repo, I grab a copy of the
basic config files and enable audio and video:

  curl https://git.savannah.gnu.org/cgit/mediagoblin.git/plain/mediagoblin.example.ini > mediagoblin.ini
  curl https://git.savannah.gnu.org/cgit/mediagoblin.git/plain/paste.ini > paste.ini
  echo "[[mediagoblin.media_types.audio]]" >> mediagoblin.ini
  echo "[[mediagoblin.media_types.video]]" >> mediagoblin.ini

Build MediaGoblin, which downloads from our master branch and runs the
full test suite successfully:

  ~/ws/guix/pre-inst-env guix build mediagoblin

Then install MediaGoblin in a container (not working in a non-container
guix environment or without explicit "python"):

  ~/ws/guix/pre-inst-env guix environment --container --network --share=$HOME/.bash_history --ad-hoc mediagoblin python

Create an sqlite3 database and add a user:

  gmg dbupdate
  gmg adduser --username admin --password a --email admin@example.com
  gmg changepw admin a

Upload an image, audio and video via CLI:

  gmg addmedia admin image.jpg
  gmg addmedia admin audio.wav
  gmg addmedia admin video.mp4

Start the web interface:

  CELERY_ALWAYS_EAGER=true paster serve paste.ini

The web interface is working. Looks like we're missing some CSS
(probably due to files not being included in the setuptools package),
but that's a minor issue.

Getting there!

Regards,
Ben

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-MediaGoblin-package.patch --]
[-- Type: text/x-patch, Size: 12013 bytes --]

From 704cd6aa56f17d4f705d33f1f58ff03e2581a2da Mon Sep 17 00:00:00 2001
From: Ben Sturmfels <ben@sturm.com.au>
Date: Tue, 6 Apr 2021 12:48:47 +1000
Subject: [PATCH] Add MediaGoblin package.

---
 gnu/local.mk                 |   1 +
 gnu/packages/mediagoblin.scm | 266 +++++++++++++++++++++++++++++++++++
 2 files changed, 267 insertions(+)
 create mode 100644 gnu/packages/mediagoblin.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index f2d595f2cc..b2daa3cfe6 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -467,6 +467,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/python-science.scm		\
   %D%/packages/python-web.scm			\
   %D%/packages/python-xyz.scm			\
+  %D%/packages/mediagoblin.scm			\
   %D%/packages/toys.scm				\
   %D%/packages/tryton.scm			\
   %D%/packages/qt.scm				\
diff --git a/gnu/packages/mediagoblin.scm b/gnu/packages/mediagoblin.scm
new file mode 100644
index 0000000000..87649c965a
--- /dev/null
+++ b/gnu/packages/mediagoblin.scm
@@ -0,0 +1,266 @@
+;; Install with:
+;;
+;; ~/ws/guix/pre-inst-env guix environment --container --network --share=$HOME/.bash_history --ad-hoc mediagoblin
+;;
+;; Doesn't currently work when not in a container:
+;;
+;; ~/ws/guix/pre-inst-env guix environment --ad-hoc mediagoblin
+
+(define-module (gnu packages mediagoblin)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (guix packages)
+  #:use-module (guix licenses)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system python)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages xiph)  ; flac for embedded libsndfile
+  #:use-module (gnu packages autotools)
+  #:use-module (gnu packages base)
+  #:use-module (gnu packages certs)
+  #:use-module (gnu packages check)
+  #:use-module (gnu packages compression)  ; unzip for embedded python-wtforms
+  #:use-module (gnu packages databases)
+  #:use-module (gnu packages libffi)  ; cffi for embedded python-soundfile
+  #:use-module (gnu packages openldap)
+  #:use-module (gnu packages pdf)
+  #:use-module (gnu packages pkg-config)  ; embedded libsndfile
+  #:use-module (gnu packages python)
+  #:use-module (gnu packages python-crypto)
+  #:use-module (gnu packages python-web)
+  #:use-module (gnu packages python-xyz)
+  #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages gstreamer)
+  #:use-module (gnu packages glib)
+  #:use-module (gnu packages pulseaudio)
+  #:use-module (gnu packages rsync)
+  #:use-module (gnu packages ssh)
+  #:use-module (gnu packages time)
+  #:use-module (gnu packages video)
+  #:use-module (gnu packages version-control)
+  #:use-module (gnu packages xml)
+  #:use-module ((guix licenses) #:select (bsd-3 gpl2+) #:prefix license:))
+
+(define python-wtforms
+  (package
+    (name "python-wtforms")
+    (version "2.3.3")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "WTForms" version))
+       (sha256
+        (base32
+         ;; Interesting, if this has is that of a lower version, it blindly
+         ;; ignores the version number above and you silently get the older
+         ;; version.
+         "17427m7p9nn9byzva697dkykykwcp2br3bxvi8vciywlmkh5s6c1"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f))  ; TODO: Fix tests for upgraded version.
+    (propagated-inputs
+    `(("python-markupsafe" ,python-markupsafe)))
+    (native-inputs
+     `(("unzip" ,unzip)))  ; CHECK WHETHER NEEDED - not in `guix import` but is in old package.
+    (home-page "http://wtforms.simplecodes.com/")
+    (synopsis
+     "Form validation and rendering library for Python web development")
+    (description
+     "WTForms is a flexible forms validation and rendering library
+for Python web development.  It is very similar to the web form API
+available in Django, but is a standalone package.")
+    (license license:bsd-3)))
+
+
+;; Copied from guix/gnu/packages/pulseaudio.scm in the core-updates branch which
+;; adds flac/ogg/vorbis/opus support. This is required for building
+;; python-soundfile (March 2021).
+(define libsndfile
+  (package
+    (name "libsndfile")
+    (version "1.0.30")
+    (source (origin
+             (method url-fetch)
+             (uri (string-append "https://github.com/erikd/libsndfile"
+                                 "/releases/download/v" version
+                                 "/libsndfile-" version ".tar.bz2"))
+             (sha256
+              (base32
+               "06k1wj3lwm7vf21s8yqy51k6nrkn9z610bj1gxb618ag5hq77wlx"))
+             (modules '((ice-9 textual-ports) (guix build utils)))
+             (snippet
+              '(begin
+                 ;; Remove carriage returns (CRLF) to prevent bogus
+                 ;; errors from bash like "$'\r': command not found".
+                 (let ((data (call-with-input-file
+                                 "tests/pedantic-header-test.sh.in"
+                               (lambda (port)
+                                 (string-join
+                                  (string-split (get-string-all port)
+                                                #\return))))))
+                   (call-with-output-file "tests/pedantic-header-test.sh.in"
+                     (lambda (port) (format port data))))
+
+                 ;; While at it, fix hard coded executable name.
+                 (substitute* "tests/test_wrapper.sh.in"
+                   (("^/usr/bin/env") "env"))
+                 #t))))
+    (build-system gnu-build-system)
+    (propagated-inputs
+     `(("flac" ,flac)
+       ("libogg" ,libogg)
+       ("libvorbis" ,libvorbis)
+       ("opus" ,opus)))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)
+       ("python" ,python)))
+    (home-page "http://www.mega-nerd.com/libsndfile/")
+    (synopsis "Reading and writing files containing sampled sound")
+    (description
+     "Libsndfile is a C library for reading and writing files containing
+sampled sound (such as MS Windows WAV and the Apple/SGI AIFF format) through
+one standard library interface.
+
+It was designed to handle both little-endian (such as WAV) and
+big-endian (such as AIFF) data, and to compile and run correctly on
+little-endian (such as Intel and DEC/Compaq Alpha) processor systems as well
+as big-endian processor systems such as Motorola 68k, Power PC, MIPS and
+SPARC.  Hopefully the design of the library will also make it easy to extend
+for reading and writing new sound file formats.")
+    (license license:gpl2+)))
+
+;; Need soundfile for new Python 3 audio spectrograms. Can me merged into Guix
+;; once core-updates is merged.
+(define python-soundfile
+  (package
+    (name "python-soundfile")
+    (version "0.10.3.post1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "SoundFile" version))
+       (sha256
+        (base32
+         "0yqhrfz7xkvqrwdxdx2ydy4h467sk7z3gf984y1x2cq7cm1gy329"))))
+    (build-system python-build-system)
+    (native-inputs
+     `(("python-pytest" ,python-pytest)))
+    (propagated-inputs
+     `(("python-cffi" ,python-cffi)
+       ("libsndfile" ,libsndfile)
+       ("python-numpy" ,python-numpy)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'set-library-file-name
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let ((libsndfile (assoc-ref inputs "libsndfile")))
+               (substitute* "soundfile.py"
+                 (("_find_library\\('sndfile'\\)")
+                  (string-append "'" libsndfile "/lib/libsndfile.so.1'")))
+               #t))))))
+    (home-page "https://github.com/bastibe/python-soundfile")
+    (synopsis "An audio library based on libsndfile, CFFI and NumPy")
+    (description
+     "The soundfile module can read and write sound files, representing audio
+data as NumPy arrays.")
+    (license license:bsd-3)))
+
+;; =================================================================
+
+(define-public mediagoblin
+  (package
+    (name "mediagoblin")
+    (version "0.12.0.dev")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://git.savannah.gnu.org/git/mediagoblin.git")
+             (commit "3d72ccf4dfb106d8b9dcc5ac4cb754d6efa80649")))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32 "0i6h8781nlnnqrw8jf0n0264p2fvc3z9rcp8vzv4a7aghiqmn2b2"))))
+    (build-system python-build-system)
+    (arguments
+     `(
+       #:tests? #f
+       ;; #:phases (modify-phases %standard-phases
+       ;;            (replace 'check
+       ;;              (lambda _
+       ;;                (setenv "PYTHONPATH"
+       ;;                        (string-append (getcwd) ":"
+       ;;                                       (getenv "PYTHONPATH")))
+       ;;                ;; Translations needed for tests to pass. Probably
+       ;;                ;; should be done during build stage?
+       ;;                (invoke "./devtools/compile_translations.sh")
+       ;;                (invoke "pytest" "./mediagoblin/tests" "-rs" "--boxed")
+       ;;                #t)))
+                ))
+    (native-inputs
+     `(("nss-certs" ,nss-certs)
+       ("python-pytest" ,python-pytest)
+       ("python-pytest-forked" ,python-pytest-forked)
+       ("python-pytest-xdist" ,python-pytest-xdist)
+       ("python-webtest" ,python-webtest)))
+    ;; lle-bout suggests avoiding propagated-inputs and insteading creating
+    ;; wrappers scripts. See:
+    ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47260#44
+    (propagated-inputs
+     `(("python-alembic" ,python-alembic)
+       ("python-babel" ,python-babel)
+       ("python-celery" ,python-celery)
+       ("python-configobj" ,python-configobj)
+       ("python-dateutil" ,python-dateutil)
+       ("python-docutils" ,python-docutils)  ; What for?
+       ("python-email-validator" ,python-email-validator)
+       ("python-exif-read" ,python-exif-read)
+       ("python-feedgenerator" ,python-feedgenerator)
+       ("python-itsdangerous" ,python-itsdangerous)
+       ("python-jinja2" ,python-jinja2)
+       ("python-jsonschema" ,python-jsonschema)
+       ("python-ldap" ,python-ldap)  ; For LDAP plugin
+       ("python-lxml" ,python-lxml)
+       ("python-markdown" ,python-markdown)
+       ("python-oauthlib" ,python-oauthlib)
+       ("python-openid" ,python-openid)  ; For OpenID plugin
+       ("python-pastescript" ,python-pastescript)
+       ("python-pillow" ,python-pillow)
+       ("python-py-bcrypt" ,python-py-bcrypt)
+       ("python-pyld" ,python-pyld)
+       ("python-pytz" ,python-pytz)
+       ("python-requests" ,python-requests)  ; For Persona plugin, batchaddmedia
+       ("python-setuptools" ,python-setuptools)  ; What for?
+       ("python-soundfile" ,python-soundfile)
+       ("python-sphinx" ,python-sphinx)
+       ("python-sqlalchemy" ,python-sqlalchemy)
+       ("python-translitcodec" ,python-translitcodec)
+       ("python-unidecode" ,python-unidecode)
+       ("python-waitress" ,python-waitress)
+       ("python-werkzeug" ,python-werkzeug)
+       ("python-wtforms" ,python-wtforms)
+
+       ;; Audio/video media
+       ("gobject-introspection" ,gobject-introspection)
+       ("gst-libav" ,gst-libav)
+       ("gst-plugins-bad" ,gst-plugins-bad)
+       ("gst-plugins-base" ,gst-plugins-base)
+       ("gst-plugins-good" ,gst-plugins-good)
+       ("gst-plugins-ugly" ,gst-plugins-ugly)
+       ("gstreamer" ,gstreamer)
+       ("openh264" ,openh264)
+       ("python-gst" ,python-gst)  ; For tests to pass
+       ("python-numpy" ,python-numpy)  ; Audio spectrograms
+       ("python-pygobject" ,python-pygobject)
+
+       ;; PDF media.
+       ("poppler" ,poppler)
+
+       ))
+    (home-page "https://mediagoblin.org/")
+    (synopsis "Web application for media publishing")
+    (description "MediaGoblin is a web application for publishing all kinds of
+media.")
+    (license agpl3+)))
-- 
2.31.1

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Tue, 06 Apr 2021, Ben Sturmfels wrote:

> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
>
>> 5. Get a basic Guix service working, with sqlite3 and without the
>> offloaded media transcoding currently using Celery task queue with a
>> Redis broker.
>
> Woo! After a lot of trial and error, I finally have a basic MediaGoblin
> running entirely under Guix with no virtualenv trickery!
>
> After applying the attached patch to my guix repo...

Even simpler, I've now created a Guix channel for our work-in-progress
MediaGoblin package/service. See the README for instructions on enabling
the channel and installing MediaGoblin:

https://gitlab.com/BenSturmfels/mediagoblin-guix/-/blob/master/README.md

Once the channel is enabled, you can `guix install mediagoblin` and run
`gmg` commands and the web interface. All documented in the above README.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

Just a progress update - Raghav and Jgart have now packaged and updated
python-wtforms, thank you both!

All that remains now for a passing test suite (and OGG Vorbis support)
is the modified libsndfile that's waiting in Guix's core-updates branch.
Very close now.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

I've now written up all the progress in our MediaGoblin Guix channel README:

https://git.sr.ht/~mediagoblin/mediagoblin-guix/tree/master/item/README.md

In short MediaGoblin can be installed as a Guix package with no external
dependencies or Python virtualenvs. After some slightly clumsy static
files configuration the web interface runs successfully based on an
SQLite database. The Celery backend run successfully. Images and video
can be uploaded and viewed successfully. Video plays via the stock
browser player but doesn't allow selection of video quality. Audio is
broken pending merge of updated libsndfile from Guix's "core-updates"
branch.

To try it out, follow the instructions in "Install via load-path",
followed by "Run MediaGoblin" from the README mentioned above.

Regards,
Ben

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

On Mon, 13 Sep 2021, Ben Sturmfels wrote:

I've now updated MediaGoblin to find static assets relative to the code,
and hacked in two new entrypoints `gmg serve` and `gmg celery`. Together
these changes mean that we can now mostly use `inputs` rather than
`propagated-inputs` and that the CSS and images are showing up when
installed as a user package (not service)

Here's the updated channel:

https://git.sr.ht/~mediagoblin/mediagoblin-guix/tree/master/item/README.md

I think we're close to being able to propose a patch for a MediaGoblin
package that can be run standalone. Haven't yet worked on a service.

bug#47260: Package GNU MediaGoblin as a Guix service

[Dr. Arne Babenhauserheide]

[-- Attachment #1.1: Type: text/plain, Size: 156 bytes --]

Hi,

I just added non-flickering video-change to the m3u-player. Attaching
the file. I thought that could be useful for MediaGoblin. The file is
attached.


[-- Attachment #1.2: m3u-player.js --]
[-- Type: application/octet-stream, Size: 11050 bytes --]

// [[file:m3u-player.org::*The script][The script:1]]
// @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2-or-Later
const nodes = document.querySelectorAll("audio,video");
const playlists = {};
const prefetchedTracks = new Map(); // use a map for insertion order, so we can just blow away old entries.
// maximum prefetched blobs that are kept.
const MAX_PREFETCH_KEEP = 10;
// maximum allowed number of entries in a playlist to prevent OOM attacks against the browser with self-referencing playlists
const MAX_PLAYLIST_LENGTH = 1000;
const PLAYLIST_MIME_TYPES = ["audio/x-mpegurl", "audio/mpegurl", "application/vnd.apple.mpegurl","application/mpegurl","application/x-mpegurl"];
function stripUrlParameters(link) {
  const url = new URL(link, window.location);
  url.search = "";
  url.hash = "";
  return url.href;
}
function isPlaylist(link) {
  const linkHref = stripUrlParameters(link);
  return linkHref.endsWith(".m3u") || linkHref.endsWith(".m3u8");
}
function isBlob(link) {
  return new URL(link, window.location).protocol == 'blob';
}
function parsePlaylist(textContent) {
  return textContent.match(/^(?!#)(?!\s).*$/mg)
    .filter(s => s); // filter removes empty strings
}
/**
 * Download the given playlist, parse it, and store the tracks in the
 * global playlists object using the url as key.
 *
 * Runs callback once the playlist downloaded successfully.
 */
function fetchPlaylist(url, onload, onerror) {
  const playlistFetcher = new XMLHttpRequest();
  playlistFetcher.open("GET", url, true);
  playlistFetcher.responseType = "blob"; // to get a mime type
  playlistFetcher.onload = () => {
    if (PLAYLIST_MIME_TYPES.includes(playlistFetcher.response.type)) { // security check to ensure that filters have run
      const reader = new FileReader();
      const load = onload; // propagate to inner scope
      reader.addEventListener("loadend", e => {
        playlists[url] = parsePlaylist(reader.result);
        onload();
      });
      reader.readAsText(playlistFetcher.response);
    } else {
      console.error("playlist must have one of the playlist MIME type '" + PLAYLIST_MIME_TYPES + "' but it had MIME type '" + playlistFetcher.response.type + "'.");
      onerror();
    }
  };
  playlistFetcher.onerror = onerror;
  playlistFetcher.abort = onerror;
  playlistFetcher.send();
}
function servedPartialDataAndCanRequestAll (xhr) {
  if (xhr.status === 206) {
    if (xhr.getResponseHeader("content-range").includes("/")) {
      if (!xhr.getResponseHeader("content-range").includes("/*")) {
        return true;
      }
    }
  }
  return false;
}
function prefetchTrack(url, onload) {
  if (prefetchedTracks.has(url)) {
    return;
  }
  // first cleanup: kill the oldest entries until we're back at the allowed size
  while (prefetchedTracks.size > MAX_PREFETCH_KEEP) {
    const key = prefetchedTracks.keys().next().value;
    const track = prefetchedTracks.get(key);
    prefetchedTracks.delete(key);
  }
  // first set the prefetched to the url so we will never request twice
  prefetchedTracks.set(url, url);
  // now start replacing it with a blob
  const xhr = new XMLHttpRequest();
  xhr.open("GET", url, true);
  xhr.responseType = "blob";
  xhr.onload = () => {
    if (servedPartialDataAndCanRequestAll(xhr)) {
      const endRange = Number(xhr.getResponseHeader("content-range").split("/")[1]) - 1;
      const rangeXhr = new XMLHttpRequest();
      rangeXhr.open("GET", url, true);
      rangeXhr.responseType = "blob";
      rangeXhr.setRequestHeader("range", "bytes=0-" + endRange);
      rangeXhr.onload = () => {
        prefetchedTracks.set(url, rangeXhr.response);
        if (onload) {
          onload();
        }
      };
      rangeXhr.send();      
    } else {
      prefetchedTracks.set(url, xhr.response);
      if (onload) {
        onload();
      }
    }
  };
  xhr.send();
}
function updateSrc(mediaTag, callback) {
  const playlistUrl = mediaTag.getAttribute("playlist");
  const trackIndex =  mediaTag.getAttribute("track-index");
  // deepcopy playlists to avoid shared mutation
  let playlist = [...playlists[playlistUrl]];
  let trackUrl = playlist[trackIndex];
  // download and splice in playlists as needed
  if (isPlaylist(trackUrl)) {
    if (playlist.length >= MAX_PLAYLIST_LENGTH) {
      // skip playlist if we already have too many tracks
      changeTrack(mediaTag, +1);
    } else {
      // do not use the cached playlist here, though it is tempting: it might genuinely change to allow for updates
      fetchPlaylist(
        trackUrl,
        () => {
          playlist.splice(trackIndex, 1, ...playlists[trackUrl]);
          playlists[playlistUrl] = playlist;
          updateSrc(mediaTag, callback);
        },
        () => callback());
    }
  } else {
    let url = prefetchedTracks.has(trackUrl)
        ? prefetchedTracks.get(trackUrl) instanceof Blob
        ? URL.createObjectURL(prefetchedTracks.get(trackUrl))
        : trackUrl : trackUrl;
    const oldUrl = mediaTag.getAttribute("src");
    // prevent size flickering by setting height before src change
    const canvas = document.createElement("canvas");
    if (!isNaN(mediaTag.duration)) { // already loaded a valid file so the size should fit
      // fix height to the height of the current video. Re-run after setting the source.
      mediaTag.height = (mediaTag.clientWidth * mediaTag.videoHeight) / mediaTag.videoWidth;
      // take screenshot of video and overlay it to mask flicker
      canvas.width = mediaTag.clientWidth;
      canvas.height = mediaTag.clientHeight;
      const context = canvas.getContext("2d");
      context.scale(mediaTag.clientWidth / mediaTag.videoWidth, mediaTag.clientHeight / mediaTag.videoHeight);
      context.drawImage(mediaTag, 0, 0);
      canvas.hidden = true;
      mediaTag.parentNode.insertBefore(canvas, mediaTag.nextSibling);
      canvas.style.position = "absolute";
      canvas.style.marginLeft = "-" + mediaTag.clientWidth + "px";
      canvas.hidden = false;
    }
    mediaTag.setAttribute("src", url);
    mediaTag.oncanplaythrough = () => {
      if (!isNaN(mediaTag.duration)) { // already loaded a valid file
        // fix height to the height of the current video. Re-run after setting the source.
        mediaTag.height = (mediaTag.clientWidth * mediaTag.videoHeight) / mediaTag.videoWidth;
      }
      // remove overlay
      canvas.hidden = true;
      canvas.remove(); // to allow garbage collection
    };
    setTimeout(() => canvas.remove(), 300); // fallback
    // replace the url when done, because a blob from an xhr request
    // is more reliable in the media tag;
    // the normal URL caused jumping prematurely to the next track.
    if (url == trackUrl) {
      prefetchTrack(trackUrl, () => {
        if (mediaTag.paused) {
          if (url == mediaTag.getAttribute("src")) {
            if (mediaTag.currentTime === 0) {
              mediaTag.setAttribute("src", URL.createObjectURL(
                prefetchedTracks.get(url)));
            }
          }
        }
      });
    }
    // allow releasing memory
    if (isBlob(oldUrl)) {
      URL.revokeObjectURL(oldUrl);
    }
    // update title
    mediaTag.parentElement.querySelector(".m3u-player--title").title = trackUrl;
    mediaTag.parentElement.querySelector(".m3u-player--title").textContent = trackUrl;
    // start prefetching the next three tracks.
    for (const i of [1, 2, 3]) {
      if (playlist.length > Number(trackIndex) + i) {
        prefetchTrack(playlist[Number(trackIndex) + i]);
      }
    }
    callback();
  }
}
function changeTrack(mediaTag, diff) {
  const currentTrackIndex = Number(mediaTag.getAttribute("track-index"));
  const nextTrackIndex = currentTrackIndex + diff;
  const tracks = playlists[mediaTag.getAttribute("playlist")];
  if (nextTrackIndex >= 0) { // do not collapse the if clauses with double-and, that does not survive inlining
    if (tracks.length > nextTrackIndex) {
    mediaTag.setAttribute("track-index", nextTrackIndex);
      updateSrc(mediaTag, () => mediaTag.play());
    }
  }
}

/**
 * Turn a media tag into playlist player.
 */
function initPlayer(mediaTag) {
  mediaTag.setAttribute("playlist", mediaTag.getAttribute("src"));
  mediaTag.setAttribute("track-index", 0);
  const url = mediaTag.getAttribute("playlist");
  const wrapper = mediaTag.parentElement.insertBefore(document.createElement("div"), mediaTag);
  const controls = document.createElement("div");
  const left = document.createElement("span");
  const title = document.createElement("span");
  const right = document.createElement("span");
  controls.appendChild(left);
  controls.appendChild(title);
  controls.appendChild(right);
  left.classList.add("m3u-player--left");
  right.classList.add("m3u-player--right");
  title.classList.add("m3u-player--title");
  title.style.overflow = "hidden";
  title.style.textOverflow = "ellipsis";
  title.style.whiteSpace = "nowrap";
  title.style.opacity = "0.3";
  title.style.direction = "rtl"; // for truncation on the left
  title.style.paddingLeft = "0.5em";
  title.style.paddingRight = "0.5em";
  controls.style.display = "flex";
  controls.style.justifyContent = "space-between";
  const styleTag = document.createElement("style");
  styleTag.innerHTML = ".m3u-player--left:hover, .m3u-player--right:hover {color: wheat; background-color: DarkSlateGray}";
  wrapper.appendChild(styleTag);
  wrapper.appendChild(controls);
  controls.style.width = mediaTag.getBoundingClientRect().width.toString() + "px";
  // appending the media tag to the wrapper removes it from the outer scope but keeps the event listeners
  wrapper.appendChild(mediaTag);
  left.innerHTML = "<"; // not textContent, because we MUST escape
                           // the tag here and textContent shows the
                           // escaped version
  left.onclick = () => changeTrack(mediaTag, -1);
  right.innerHTML = ">";
  right.onclick = () => changeTrack(mediaTag, +1);
  fetchPlaylist(
    url,
    () => {
      updateSrc(mediaTag, () => null);
      mediaTag.addEventListener("ended", event => {
        if (mediaTag.currentTime >= mediaTag.duration) {
          changeTrack(mediaTag, +1);
        }
      });
    },
    () => null);
  // keep the controls aligned to the media tag
  mediaTag.resizeObserver = new ResizeObserver(entries => {
    controls.style.width = entries[0].contentRect.width.toString() + "px";
  });
  mediaTag.resizeObserver.observe(mediaTag);
}
function processTag(mediaTag) {
  const canPlayClaim = mediaTag.canPlayType('audio/x-mpegurl');
  let supportsPlaylists = !!canPlayClaim;
  if (canPlayClaim == 'maybe') { // yes, seriously: specced as you only know when you try
    supportsPlaylists = false;
  }
  if (!supportsPlaylists) {
    if (isPlaylist(mediaTag.getAttribute("src"))) {
      initPlayer(mediaTag);
    }
  }
}
document.addEventListener('DOMContentLoaded', () => {
  const nodes = document.querySelectorAll("audio,video");
  nodes.forEach(processTag);
});
// @license-end
// The script:1 ends here

[-- Attachment #1.3: Type: text/plain, Size: 1850 bytes --]


Best wishes,
Arne


Ben Sturmfels via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> On Tue, 30 Mar 2021, Ben Sturmfels wrote:
>
>> On Fri, 19 Mar 2021, Ben Sturmfels wrote:
>
>>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>>> implicated in many bugs anyway, so there may benefits to the project to
>>> doing this anyway.
>>
>> I learnt that Celery has a Redis backend, so maybe we don't need to
>> rewrite just yet.
>
> It turns out that MediaGoblin's Celery-based media processing backend
> work out of the box by simply configuring:
>
>   [celery]
>   BROKER_URL = "redis://"
>
> (There seems to be an unrelated bug where media is marked as failed after
> restarting Celery, possibly tied to sqlite. We've had reports of this
> with a RabbitMQ broker too though.)
>
>
> This means our shorter to-do list is now:
>
> 1. Upstream our new python-soundfile Guix package from guix-env.scm when
> core-updates is merged.
>
> 2. Upstream our upgraded python-wtforms package.
>
> 6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
> Video and audio are essentially functional without the NPM installed
> players. Some later refinements perhaps.
>
> 4. Package MediaGoblin itself. The build process is ./configure/make
> which is a bit weird for a Python project.
>
> 5. Get a basic Guix service working, with sqlite3 and without the
> offloaded media transcoding currently using Celery task queue with a
> Redis broker.
>
> 7. Work out why H264 support is missing.
>
> 8. Figure out how to deal with translations.
>
> 9. Add a PostgreSQL database to the Guix service instead of sqlite3.
>
> Regards,
> Ben


-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

bug#47260: Package GNU MediaGoblin as a Guix service

[Ben Sturmfels via Bug reports for GNU Guix]

Thanks for the update Arne. This issue is specifically about Guix
packaging, so to save us losing track of your update, please post it to
the dedicated mediagoblin-devel@gnu.org thread we started a couple of
months back:

https://lists.gnu.org/archive/html/mediagoblin-devel/2021-03/msg00026.html

Thanks again,
Ben

On Wed, 05 May 2021, Arne Babenhauserheide wrote:

> Hi,
>
> I just added non-flickering video-change to the m3u-player. Attaching
> the file. I thought that could be useful for MediaGoblin. The file is
> attached.
>
>
> Best wishes,
> Arne
>
>
> Ben Sturmfels via Bug reports for GNU Guix <bug-guix@gnu.org> writes:
>
>> On Tue, 30 Mar 2021, Ben Sturmfels wrote:
>>
>>> On Fri, 19 Mar 2021, Ben Sturmfels wrote:
>>
>>>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>>>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>>>> implicated in many bugs anyway, so there may benefits to the project to
>>>> doing this anyway.
>>>
>>> I learnt that Celery has a Redis backend, so maybe we don't need to
>>> rewrite just yet.
>>
>> It turns out that MediaGoblin's Celery-based media processing backend
>> work out of the box by simply configuring:
>>
>>   [celery]
>>   BROKER_URL = "redis://"
>>
>> (There seems to be an unrelated bug where media is marked as failed after
>> restarting Celery, possibly tied to sqlite. We've had reports of this
>> with a RabbitMQ broker too though.)
>>
>>
>> This means our shorter to-do list is now:
>>
>> 1. Upstream our new python-soundfile Guix package from guix-env.scm when
>> core-updates is merged.
>>
>> 2. Upstream our upgraded python-wtforms package.
>>
>> 6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
>> Video and audio are essentially functional without the NPM installed
>> players. Some later refinements perhaps.
>>
>> 4. Package MediaGoblin itself. The build process is ./configure/make
>> which is a bit weird for a Python project.
>>
>> 5. Get a basic Guix service working, with sqlite3 and without the
>> offloaded media transcoding currently using Celery task queue with a
>> Redis broker.
>>
>> 7. Work out why H264 support is missing.
>>
>> 8. Figure out how to deal with translations.
>>
>> 9. Add a PostgreSQL database to the Guix service instead of sqlite3.
>>
>> Regards,
>> Ben

bug#47260: Wrapping binaries in MediaGoblin Guix Package

[jgart via Bug reports for GNU Guix]

Just a note that we'll probably have to wrap the binaries that are in this 
module when we finish making the guix package:

mediagoblin/media_types/pdf/processing.py

I see that pdftocairo, pdfinfo, and unoconv probably need to be wrapped to point
to the executable that is in /gnu/store.

See this issue for a similar discussion about wrapping binaries with guix:

https://issues.guix.gnu.org/50833

bug#47260: Wrapping binaries in MediaGoblin Guix Package

[Ben Sturmfels via Bug reports for GNU Guix]

On Tue, 05 Oct 2021, jgart wrote:

> Just a note that we'll probably have to wrap the binaries that are in this 
> module when we finish making the guix package:
>
> mediagoblin/media_types/pdf/processing.py
>
> I see that pdftocairo, pdfinfo, and unoconv probably need to be wrapped to point
> to the executable that is in /gnu/store.
>
> See this issue for a similar discussion about wrapping binaries with guix:
>
> https://issues.guix.gnu.org/50833

Thanks, I've made a note in guix-env.scm to remind me when we look at
PDF support in Guix.