all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / Atom feed
* bug#47260: Package GNU MediaGoblin as a Guix service
@ 2021-03-19 12:20 Ben Sturmfels via Bug reports for GNU Guix
  2021-03-19 15:50 ` jgart via Bug reports for GNU Guix
  2021-03-30  4:12 ` Ben Sturmfels via Bug reports for GNU Guix
  0 siblings, 2 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-03-19 12:20 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

This is a "meta" bug to keep track of the progress of packaging GNU
MediaGoblin, a platform for publishing images/audio/video etc. See
https://mediagoblin.org/

We have a guix-env.scm in the upstream source which should always have
the latest copy of our packaging progress and instructions to run it:

https://git.savannah.gnu.org/cgit/mediagoblin.git/tree/guix-env.scm

Current plan is:

1. Add OGG support to libsndfile which is needed to package
python-soundfile [patch 47210]

2. Package python-soundfile (see above). After this the test suite
should pass 100% with pytest installed from PyPI [patch 47181]

3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
in Guix seem to be incompatible. After this our test suite should run
100% with only dependencies from Guix!

4. Package MediaGoblin itself. The build process is ./configure/make
which is a bit weird for a Python project.

5. Get a basic Guix service working, with sqlite3 and without the
offloaded media transcoding currently using Celery/RabbitMQ.

6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
improve the no-bundled-JavaScript video/audio playing experience.

7. Work out why H264 support is missing.

8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
implicated in many bugs anyway, so there may benefits to the project to
doing this anyway.

9. Figure out how to deal with translations.

10. Add a PostgreSQL database to the Guix service instead of sqlite3.

11. We win. Maybe :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-19 12:20 bug#47260: Package GNU MediaGoblin as a Guix service Ben Sturmfels via Bug reports for GNU Guix
@ 2021-03-19 15:50 ` jgart via Bug reports for GNU Guix
  2021-03-21 23:28   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-30  4:12 ` Ben Sturmfels via Bug reports for GNU Guix
  1 sibling, 1 reply; 17+ messages in thread
From: jgart via Bug reports for GNU Guix @ 2021-03-19 15:50 UTC (permalink / raw)
  To: 47260

This sounds like a great project. I would love MediaGoblin to be in Guix also.

> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
> improve the no-bundled-JavaScript video/audio playing experience.

What are your thoughts on rewriting the jquery? 

Should MediaGoblin be using vanilla javascript instead?

Some other possibilities could be purescript (https://www.purescript.org) or mint (http://mint-lang.com), although mint and crystal are not in guix yet and mint uses preact (http://preactjs.com) as its' runtime since 0.8.0 (https://www.mint-lang.com/blog/mint-0.8.0).




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-19 15:50 ` jgart via Bug reports for GNU Guix
@ 2021-03-21 23:28   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-22  7:02     ` Dr. Arne Babenhauserheide
  2021-03-22 17:58     ` Christopher Lemmer Webber
  0 siblings, 2 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-03-21 23:28 UTC (permalink / raw)
  To: jgart; +Cc: 47260

On Fri, 19 Mar 2021, jgart wrote:

> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>
>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>> improve the no-bundled-JavaScript video/audio playing experience.
>
> What are your thoughts on rewriting the jquery? 
>
> Should MediaGoblin be using vanilla javascript instead?
>
> Some other possibilities could be purescript (https://www.purescript.org) or
> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
> (https://www.mint-lang.com/blog/mint-0.8.0).

Many of the functions we used to use jQuery for are now built into most
browsers from the last 10 years.

By far the most common are element selectors like:

  var panel = $('#header-panel')
  var arrow = $('.arrow')

which we just change to:

  var panel = document.querySelector('#header-panel')
  var arrow = document.querySelector('.arrow')

This is all vanilla JavaScript in individual files imported via <script>
without any sort of compilation process. This seems to be the simplest
path to get into distributions like Guix and Debian. The main problem
with purescript and others is that they require more dependencies, build
steps and tooling and are known by less people.

We have a few dependencies which will be harder to deal with such as the
video, audio and 3D model viewers. In the first instance, I think the
answer will be to do without this JavaScript entirely. That should be
feasible for video and audio given modern browser support. Perhaps we
don't ship 3D model support out of the box in distros for now.

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-21 23:28   ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-03-22  7:02     ` Dr. Arne Babenhauserheide
  2021-03-30  4:02       ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-22 17:58     ` Christopher Lemmer Webber
  1 sibling, 1 reply; 17+ messages in thread
From: Dr. Arne Babenhauserheide @ 2021-03-22  7:02 UTC (permalink / raw)
  To: Ben Sturmfels; +Cc: 47260, jgart

[-- Attachment #1: Type: text/plain, Size: 2615 bytes --]


Ben Sturmfels via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> On Fri, 19 Mar 2021, jgart wrote:
>
>> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>>
>>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>>> improve the no-bundled-JavaScript video/audio playing experience.
>>
>> What are your thoughts on rewriting the jquery? 
>>
>> Should MediaGoblin be using vanilla javascript instead?
>>
>> Some other possibilities could be purescript (https://www.purescript.org) or
>> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
>> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
>> (https://www.mint-lang.com/blog/mint-0.8.0).
>
> Many of the functions we used to use jQuery for are now built into most
> browsers from the last 10 years.
>
> By far the most common are element selectors like:
>
>   var panel = $('#header-panel')
>   var arrow = $('.arrow')
>
> which we just change to:
>
>   var panel = document.querySelector('#header-panel')
>   var arrow = document.querySelector('.arrow')
>
> This is all vanilla JavaScript in individual files imported via <script>
> without any sort of compilation process. This seems to be the simplest
> path to get into distributions like Guix and Debian. The main problem
> with purescript and others is that they require more dependencies, build
> steps and tooling and are known by less people.
>
> We have a few dependencies which will be harder to deal with such as the
> video, audio and 3D model viewers. In the first instance, I think the
> answer will be to do without this JavaScript entirely. That should be
> feasible for video and audio given modern browser support. Perhaps we
> don't ship 3D model support out of the box in distros for now.

If you need support for m3u-playlists, you can use the player I wrote
here: https://www.draketo.de/software/m3u-playerhttps://www.draketo.de/software/m3u-player.js (save as utf-8)
(that m3u-playlists aren’t supported out of the box in most players is a
strange oversight, the code adds it for video- and audio-tags, License:
GPLv2 or later — just ask me if you need something else)

There’s also an enhanced version for Freenet, but that has lots of
performance-changes to work over high-latency networks and with paranoid
CSP-settings:
https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-21 23:28   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-22  7:02     ` Dr. Arne Babenhauserheide
@ 2021-03-22 17:58     ` Christopher Lemmer Webber
  1 sibling, 0 replies; 17+ messages in thread
From: Christopher Lemmer Webber @ 2021-03-22 17:58 UTC (permalink / raw)
  To: Ben Sturmfels; +Cc: 47260, jgart

Ben Sturmfels via Bug reports for GNU Guix writes:

> On Fri, 19 Mar 2021, jgart wrote:
>
>> This sounds like a great project. I would love MediaGoblin to be in Guix also.
>>
>>> 6. Rewrite MediaGoblin's JavaScript code not to use jQuery. Maybe
>>> improve the no-bundled-JavaScript video/audio playing experience.
>>
>> What are your thoughts on rewriting the jquery? 
>>
>> Should MediaGoblin be using vanilla javascript instead?
>>
>> Some other possibilities could be purescript (https://www.purescript.org) or
>> mint (http://mint-lang.com), although mint and crystal are not in guix yet and
>> mint uses preact (http://preactjs.com) as its' runtime since 0.8.0
>> (https://www.mint-lang.com/blog/mint-0.8.0).
>
> Many of the functions we used to use jQuery for are now built into most
> browsers from the last 10 years.
>
> By far the most common are element selectors like:
>
>   var panel = $('#header-panel')
>   var arrow = $('.arrow')
>
> which we just change to:
>
>   var panel = document.querySelector('#header-panel')
>   var arrow = document.querySelector('.arrow')
>
> This is all vanilla JavaScript in individual files imported via <script>
> without any sort of compilation process. This seems to be the simplest
> path to get into distributions like Guix and Debian. The main problem
> with purescript and others is that they require more dependencies, build
> steps and tooling and are known by less people.
>
> We have a few dependencies which will be harder to deal with such as the
> video, audio and 3D model viewers. In the first instance, I think the
> answer will be to do without this JavaScript entirely. That should be
> feasible for video and audio given modern browser support. Perhaps we
> don't ship 3D model support out of the box in distros for now.
>
> Regards,
> Ben

The 3d model support isn't MediaGoblin's most heavily used plugin.  Not
shipping support for that would be fine, I'd think.

 - Chris





^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-22  7:02     ` Dr. Arne Babenhauserheide
@ 2021-03-30  4:02       ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-30  6:40         ` Dr. Arne Babenhauserheide
  0 siblings, 1 reply; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-03-30  4:02 UTC (permalink / raw)
  To: Dr. Arne Babenhauserheide; +Cc: 47260

On Mon, 22 Mar 2021, Dr. Arne Babenhauserheide wrote:

> If you need support for m3u-playlists, you can use the player I wrote
> here: https://www.draketo.de/software/m3u-player
> → https://www.draketo.de/software/m3u-player.js (save as utf-8)
> (that m3u-playlists aren’t supported out of the box in most players is a
> strange oversight, the code adds it for video- and audio-tags, License:
> GPLv2 or later — just ask me if you need something else)
>
> There’s also an enhanced version for Freenet, but that has lots of
> performance-changes to work over high-latency networks and with paranoid
> CSP-settings:
> https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802

Thanks Arne! I've forwarded this on to mediagoblin-devel@gnu.org so we
don't lose track of it.

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-19 12:20 bug#47260: Package GNU MediaGoblin as a Guix service Ben Sturmfels via Bug reports for GNU Guix
  2021-03-19 15:50 ` jgart via Bug reports for GNU Guix
@ 2021-03-30  4:12 ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-30 12:13   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
  1 sibling, 2 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-03-30  4:12 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter

Just an update:

On Fri, 19 Mar 2021, Ben Sturmfels wrote:

> 1. Add OGG support to libsndfile which is needed to package
> python-soundfile [patch 47210]

lfam pointed out that this has already been done in 46067, which is now
in core-updates awaiting merging into master:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46067

For the time being, we've also copied the updated libsndfile into
MediaGoblin's guix-env.scm.

> 2. Package python-soundfile (see above). After this the test suite
> should pass 100% with pytest installed from PyPI [patch 47181]

We now have a local copy of python-soundfile in MediaGoblin's
guix-env.scm which is passing all audio tests. Yay!

> 3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
> in Guix seem to be incompatible. After this our test suite should run
> 100% with only dependencies from Guix!

Discovered we'll also need to upgrade Guix's python-wtforms, but in the
mean time, installing only wtforms, pytest, pytest-xdist, pytest-forked
from PyPI allows us to pass the test suite 100% Getting closer!

> ...
> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
> implicated in many bugs anyway, so there may benefits to the project to
> doing this anyway.

I learnt that Celery has a Redis backend, so maybe we don't need to
rewrite just yet.

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-30  4:02       ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-03-30  6:40         ` Dr. Arne Babenhauserheide
  0 siblings, 0 replies; 17+ messages in thread
From: Dr. Arne Babenhauserheide @ 2021-03-30  6:40 UTC (permalink / raw)
  To: Ben Sturmfels; +Cc: 47260

[-- Attachment #1: Type: text/plain, Size: 11277 bytes --]


Ben Sturmfels <ben@sturm.com.au> writes:

> On Mon, 22 Mar 2021, Dr. Arne Babenhauserheide wrote:
>
>> If you need support for m3u-playlists, you can use the player I wrote
>> here: https://www.draketo.de/software/m3u-player
>> → https://www.draketo.de/software/m3u-player.js (save as utf-8)
>> (that m3u-playlists aren’t supported out of the box in most players is a
>> strange oversight, the code adds it for video- and audio-tags, License:
>> GPLv2 or later — just ask me if you need something else)
>>
>> There’s also an enhanced version for Freenet, but that has lots of
>> performance-changes to work over high-latency networks and with paranoid
>> CSP-settings:
>> https://github.com/freenet/fred/pull/721/files#diff-33cbf95723ae7b33eb205cf9adc3411b2098e27ba757e553406f689a4fafb802
>
> Thanks Arne! I've forwarded this on to mediagoblin-devel@gnu.org so we
> don't lose track of it.

Thank you!

I added one change last week to support mobile browsers which answer
"maybe" to the query `mediaTag.canPlayType('audio/x-mpegurl')` (yes,
seriously, and it is in the spec :-) ).

Also I backported the not freenet specific changes:
- prefetch the next three tracks as blob and keep at most 10 tracks
  cached to allow for fast track skipping (and now actually release the
  memory)
- adjustments to allow for inlining and survive the non-utf8-encoding.
- continue automatically when fetch succeeded if playback was stopped
  because it reached the end (but not if paused).
- minimal mouseover for the back and forward arrows.

When a https-m3u-list refers to a http-file, it falls back from fetching
blobs to rewriting the src-part of the tag (because blobs cannot be
fetched from a less secure resource).

This is how it looks: https://www.draketo.de/software/m3u-player.html

The changes are included in https://www.draketo.de/software/m3u-player.js

You can use it like this:

<script src="m3u-player.js" defer="defer"></script>
<audio src="m3u-player-example-playlist.m3u" controls="controls">
not supported?
</audio>


To make this bug-report independent of my site, here’s the full code:



// [[file:m3u-player.org::*The script][The script:1]]
// @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2-or-Later
const nodes = document.querySelectorAll("audio,video");
const playlists = {};
const prefetchedTracks = new Map(); // use a map for insertion order, so we can just blow away old entries.
// maximum prefetched blobs that are kept.
const MAX_PREFETCH_KEEP = 10;
// maximum allowed number of entries in a playlist to prevent OOM attacks against the browser with self-referencing playlists
const MAX_PLAYLIST_LENGTH = 1000;
const PLAYLIST_MIME_TYPES = ["audio/x-mpegurl", "audio/mpegurl", "application/vnd.apple.mpegurl","application/mpegurl","application/x-mpegurl"];
function stripUrlParameters(link) {
  const url = new URL(link, window.location);
  url.search = "";
  url.hash = "";
  return url.href;
}
function isPlaylist(link) {
  const linkHref = stripUrlParameters(link);
  return linkHref.endsWith(".m3u") || linkHref.endsWith(".m3u8");
}
function isBlob(link) {
  return new URL(link, window.location).protocol == 'blob';
}
function parsePlaylist(textContent) {
  return textContent.match(/^(?!#)(?!\s).*$/mg)
    .filter(s => s); // filter removes empty strings
}
/**
 * Download the given playlist, parse it, and store the tracks in the
 * global playlists object using the url as key.
 *
 * Runs callback once the playlist downloaded successfully.
 */
function fetchPlaylist(url, onload, onerror) {
  const playlistFetcher = new XMLHttpRequest();
  playlistFetcher.open("GET", url, true);
  playlistFetcher.responseType = "blob"; // to get a mime type
  playlistFetcher.onload = () => {
    if (PLAYLIST_MIME_TYPES.includes(playlistFetcher.response.type)) { // security check to ensure that filters have run
      const reader = new FileReader();
      const load = onload; // propagate to inner scope
      reader.addEventListener("loadend", e => {
        playlists[url] = parsePlaylist(reader.result);
        onload();
      });
      reader.readAsText(playlistFetcher.response);
    } else {
      console.error("playlist must have one of the playlist MIME type '" + PLAYLIST_MIME_TYPES + "' but it had MIME type '" + playlistFetcher.response.type + "'.");
      onerror();
    }
  };
  playlistFetcher.onerror = onerror;
  playlistFetcher.abort = onerror;
  playlistFetcher.send();
}
function prefetchTrack(url, onload) {
  if (prefetchedTracks.has(url)) {
    return;
  }
  // first cleanup: kill the oldest entries until we're back at the allowed size
  while (prefetchedTracks.size > MAX_PREFETCH_KEEP) {
    const key = prefetchedTracks.keys().next().value;
    const track = prefetchedTracks.get(key);
    prefetchedTracks.delete(key);
  }
  // first set the prefetched to the url so we will never request twice
  prefetchedTracks.set(url, url);
  // now start replacing it with a blob
  const xhr = new XMLHttpRequest();
  xhr.open("GET", url, true);
  xhr.responseType = "blob";
  xhr.onload = () => {
    prefetchedTracks.set(url, xhr.response);
    if (onload) {
      onload();
    }
  };
  xhr.send();
}
function updateSrc(mediaTag, callback) {
  const playlistUrl = mediaTag.getAttribute("playlist");
  const trackIndex =  mediaTag.getAttribute("track-index");
  // deepcopy playlists to avoid shared mutation
  let playlist = [...playlists[playlistUrl]];
  let trackUrl = playlist[trackIndex];
  // download and splice in playlists as needed
  if (isPlaylist(trackUrl)) {
    if (playlist.length >= MAX_PLAYLIST_LENGTH) {
      // skip playlist if we already have too many tracks
      changeTrack(mediaTag, +1);
    } else {
      // do not use the cached playlist here, though it is tempting: it might genuinely change to allow for updates
      fetchPlaylist(
        trackUrl,
        () => {
          playlist.splice(trackIndex, 1, ...playlists[trackUrl]);
          playlists[playlistUrl] = playlist;
          updateSrc(mediaTag, callback);
        },
        () => callback());
    }
  } else {
    let url = prefetchedTracks.has(trackUrl)
        ? prefetchedTracks.get(trackUrl) instanceof Blob
        ? URL.createObjectURL(prefetchedTracks.get(trackUrl))
        : trackUrl : trackUrl;
    const oldUrl = mediaTag.getAttribute("src");
    mediaTag.setAttribute("src", url);
    // replace the url when done, because a blob from an xhr request
    // is more reliable in the media tag;
    // the normal URL caused jumping prematurely to the next track.
    if (url == trackUrl) {
      prefetchTrack(trackUrl, () => {
        if (mediaTag.paused) {
          if (url == mediaTag.getAttribute("src")) {
            if (mediaTag.currentTime === 0) {
              mediaTag.setAttribute("src", URL.createObjectURL(
                prefetchedTracks.get(url)));
            }
          }
        }
      });
    }
    // allow releasing memory
    if (isBlob(oldUrl)) {
      URL.revokeObjectURL(oldUrl);
    }
    // update title
    mediaTag.parentElement.querySelector(".m3u-player--title").title = trackUrl;
    mediaTag.parentElement.querySelector(".m3u-player--title").textContent = trackUrl;
    // start prefetching the next three tracks.
    for (const i of [1, 2, 3]) {
      if (playlist.length > Number(trackIndex) + i) {
        prefetchTrack(playlist[Number(trackIndex) + i]);
      }
    }
    callback();
  }
}
function changeTrack(mediaTag, diff) {
  const currentTrackIndex = Number(mediaTag.getAttribute("track-index"));
  const nextTrackIndex = currentTrackIndex + diff;
  const tracks = playlists[mediaTag.getAttribute("playlist")];
  if (nextTrackIndex >= 0) { // do not collapse the if clauses with double-and, that does not survive inlining
    if (tracks.length > nextTrackIndex) {
    mediaTag.setAttribute("track-index", nextTrackIndex);
      updateSrc(mediaTag, () => mediaTag.play());
    }
  }
}

/**
 * Turn a media tag into playlist player.
 */
function initPlayer(mediaTag) {
  mediaTag.setAttribute("playlist", mediaTag.getAttribute("src"));
  mediaTag.setAttribute("track-index", 0);
  const url = mediaTag.getAttribute("playlist");
  const wrapper = mediaTag.parentElement.insertBefore(document.createElement("div"), mediaTag);
  const controls = document.createElement("div");
  const left = document.createElement("span");
  const title = document.createElement("span");
  const right = document.createElement("span");
  controls.appendChild(left);
  controls.appendChild(title);
  controls.appendChild(right);
  left.classList.add("m3u-player--left");
  right.classList.add("m3u-player--right");
  title.classList.add("m3u-player--title");
  title.style.overflow = "hidden";
  title.style.textOverflow = "ellipsis";
  title.style.whiteSpace = "nowrap";
  title.style.opacity = "0.3";
  title.style.direction = "rtl"; // for truncation on the left
  title.style.paddingLeft = "0.5em";
  title.style.paddingRight = "0.5em";
  controls.style.display = "flex";
  controls.style.justifyContent = "space-between";
  const styleTag = document.createElement("style");
  styleTag.innerHTML = ".m3u-player--left:hover, .m3u-player--right:hover {color: wheat; background-color: DarkSlateGray}";
  wrapper.appendChild(styleTag);
  wrapper.appendChild(controls);
  controls.style.width = mediaTag.getBoundingClientRect().width.toString() + "px";
  // appending the media tag to the wrapper removes it from the outer scope but keeps the event listeners
  wrapper.appendChild(mediaTag);
  left.innerHTML = "&lt;"; // not textContent, because we MUST escape
                           // the tag here and textContent shows the
                           // escaped version
  left.onclick = () => changeTrack(mediaTag, -1);
  right.innerHTML = "&gt;";
  right.onclick = () => changeTrack(mediaTag, +1);
  fetchPlaylist(
    url,
    () => {
      updateSrc(mediaTag, () => null);
      mediaTag.addEventListener("ended", event => {
        if (mediaTag.currentTime >= mediaTag.duration) {
          changeTrack(mediaTag, +1);
        }
      });
    },
    () => null);
  // keep the controls aligned to the media tag
  mediaTag.resizeObserver = new ResizeObserver(entries => {
    controls.style.width = entries[0].contentRect.width.toString() + "px";
  });
  mediaTag.resizeObserver.observe(mediaTag);
}
function processTag(mediaTag) {
  const canPlayClaim = mediaTag.canPlayType('audio/x-mpegurl');
  let supportsPlaylists = !!canPlayClaim;
  if (canPlayClaim == 'maybe') { // yes, seriously: specced as you only know when you try
    supportsPlaylists = false;
  }
  if (!supportsPlaylists) {
    if (isPlaylist(mediaTag.getAttribute("src"))) {
      initPlayer(mediaTag);
    }
  }
}
document.addEventListener('DOMContentLoaded', () => {
  const nodes = document.querySelectorAll("audio,video");
  nodes.forEach(processTag);
});
// @license-end
// The script:1 ends here




Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-30  4:12 ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-03-30 12:13   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
  1 sibling, 0 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-03-30 12:13 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter

On Tue, 30 Mar 2021, Ben Sturmfels wrote:

>> 3. Work out why python-pytest-6/python-pytest-xdist/python-pytest-forked
>> in Guix seem to be incompatible. After this our test suite should run
>> 100% with only dependencies from Guix!
>
> Discovered we'll also need to upgrade Guix's python-wtforms, but in the
> mean time, installing only wtforms, pytest, pytest-xdist, pytest-forked
> from PyPI allows us to pass the test suite 100% Getting closer!

Pytest issues have now been resolved. We now need ZERO packages from
PyPI for a basic install and full test suite run. :)

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-03-30  4:12 ` Ben Sturmfels via Bug reports for GNU Guix
  2021-03-30 12:13   ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-05 14:17     ` Ben Sturmfels via Bug reports for GNU Guix
                       ` (2 more replies)
  1 sibling, 3 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-04-01  2:03 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter, jgart

On Tue, 30 Mar 2021, Ben Sturmfels wrote:

> On Fri, 19 Mar 2021, Ben Sturmfels wrote:

>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>> implicated in many bugs anyway, so there may benefits to the project to
>> doing this anyway.
>
> I learnt that Celery has a Redis backend, so maybe we don't need to
> rewrite just yet.

It turns out that MediaGoblin's Celery-based media processing backend
work out of the box by simply configuring:

  [celery]
  BROKER_URL = "redis://"

(There seems to be an unrelated bug where media is marked as failed after
restarting Celery, possibly tied to sqlite. We've had reports of this
with a RabbitMQ broker too though.)


This means our shorter to-do list is now:

1. Upstream our new python-soundfile Guix package from guix-env.scm when
core-updates is merged.

2. Upstream our upgraded python-wtforms package.

6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
Video and audio are essentially functional without the NPM installed
players. Some later refinements perhaps.

4. Package MediaGoblin itself. The build process is ./configure/make
which is a bit weird for a Python project.

5. Get a basic Guix service working, with sqlite3 and without the
offloaded media transcoding currently using Celery task queue with a
Redis broker.

7. Work out why H264 support is missing.

8. Figure out how to deal with translations.

9. Add a PostgreSQL database to the Guix service instead of sqlite3.

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-04-05 14:17     ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-05 15:50       ` Léo Le Bouter via Bug reports for GNU Guix
  2021-04-06 12:01     ` Ben Sturmfels via Bug reports for GNU Guix
  2021-05-04 20:58     ` Dr. Arne Babenhauserheide
  2 siblings, 1 reply; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-04-05 14:17 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter, jgart

On Thu, 01 Apr 2021, Ben Sturmfels wrote:

> 7. Work out why H264 support is missing.

This is now fixed MediaGoblin's master branch guix-env.scm by adding
gst-libav to propagated inputs.




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-04-05 14:17     ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-04-05 15:50       ` Léo Le Bouter via Bug reports for GNU Guix
  2021-04-06 12:05         ` Ben Sturmfels via Bug reports for GNU Guix
  0 siblings, 1 reply; 17+ messages in thread
From: Léo Le Bouter via Bug reports for GNU Guix @ 2021-04-05 15:50 UTC (permalink / raw)
  To: Ben Sturmfels, 47260; +Cc: cwebber, jgart

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Tue, 2021-04-06 at 00:17 +1000, Ben Sturmfels wrote:
> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
> 
> > 7. Work out why H264 support is missing.
> 
> This is now fixed MediaGoblin's master branch guix-env.scm by adding
> gst-libav to propagated inputs.

Hello!

I suggest not using propagated-inputs because they are likely to cause
conflicts in profiles when used. I suggest intead creating wrapper
scripts that append to the PATH variable.

Examples of such wrapper scripts can be found:

- 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=373e5fc96724fd38bb1263e4af90932ea36f596b
(PYTHONPATH more so)
- 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
(Append xdg-utils to PATH)

Let me know if there's any questions about creating such wrappers.

Léo 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-05 14:17     ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-04-06 12:01     ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-07 13:15       ` Ben Sturmfels via Bug reports for GNU Guix
  2021-05-04 20:58     ` Dr. Arne Babenhauserheide
  2 siblings, 1 reply; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-04-06 12:01 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter, jgart

[-- Attachment #1: Type: text/plain, Size: 1714 bytes --]

On Thu, 01 Apr 2021, Ben Sturmfels wrote:

> 5. Get a basic Guix service working, with sqlite3 and without the
> offloaded media transcoding currently using Celery task queue with a
> Redis broker.

Woo! After a lot of trial and error, I finally have a basic MediaGoblin
running entirely under Guix with no virtualenv trickery!

After applying the attached patch to my guix repo, I grab a copy of the
basic config files and enable audio and video:

  curl https://git.savannah.gnu.org/cgit/mediagoblin.git/plain/mediagoblin.example.ini > mediagoblin.ini
  curl https://git.savannah.gnu.org/cgit/mediagoblin.git/plain/paste.ini > paste.ini
  echo "[[mediagoblin.media_types.audio]]" >> mediagoblin.ini
  echo "[[mediagoblin.media_types.video]]" >> mediagoblin.ini

Build MediaGoblin, which downloads from our master branch and runs the
full test suite successfully:

  ~/ws/guix/pre-inst-env guix build mediagoblin

Then install MediaGoblin in a container (not working in a non-container
guix environment or without explicit "python"):

  ~/ws/guix/pre-inst-env guix environment --container --network --share=$HOME/.bash_history --ad-hoc mediagoblin python

Create an sqlite3 database and add a user:

  gmg dbupdate
  gmg adduser --username admin --password a --email admin@example.com
  gmg changepw admin a

Upload an image, audio and video via CLI:

  gmg addmedia admin image.jpg
  gmg addmedia admin audio.wav
  gmg addmedia admin video.mp4

Start the web interface:

  CELERY_ALWAYS_EAGER=true paster serve paste.ini

The web interface is working. Looks like we're missing some CSS
(probably due to files not being included in the setuptools package),
but that's a minor issue.

Getting there!

Regards,
Ben

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-MediaGoblin-package.patch --]
[-- Type: text/x-patch, Size: 12013 bytes --]

From 704cd6aa56f17d4f705d33f1f58ff03e2581a2da Mon Sep 17 00:00:00 2001
From: Ben Sturmfels <ben@sturm.com.au>
Date: Tue, 6 Apr 2021 12:48:47 +1000
Subject: [PATCH] Add MediaGoblin package.

---
 gnu/local.mk                 |   1 +
 gnu/packages/mediagoblin.scm | 266 +++++++++++++++++++++++++++++++++++
 2 files changed, 267 insertions(+)
 create mode 100644 gnu/packages/mediagoblin.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index f2d595f2cc..b2daa3cfe6 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -467,6 +467,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/python-science.scm		\
   %D%/packages/python-web.scm			\
   %D%/packages/python-xyz.scm			\
+  %D%/packages/mediagoblin.scm			\
   %D%/packages/toys.scm				\
   %D%/packages/tryton.scm			\
   %D%/packages/qt.scm				\
diff --git a/gnu/packages/mediagoblin.scm b/gnu/packages/mediagoblin.scm
new file mode 100644
index 0000000000..87649c965a
--- /dev/null
+++ b/gnu/packages/mediagoblin.scm
@@ -0,0 +1,266 @@
+;; Install with:
+;;
+;; ~/ws/guix/pre-inst-env guix environment --container --network --share=$HOME/.bash_history --ad-hoc mediagoblin
+;;
+;; Doesn't currently work when not in a container:
+;;
+;; ~/ws/guix/pre-inst-env guix environment --ad-hoc mediagoblin
+
+(define-module (gnu packages mediagoblin)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (guix packages)
+  #:use-module (guix licenses)
+  #:use-module (guix download)
+  #:use-module (guix git-download)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system python)
+  #:use-module (gnu packages)
+  #:use-module (gnu packages xiph)  ; flac for embedded libsndfile
+  #:use-module (gnu packages autotools)
+  #:use-module (gnu packages base)
+  #:use-module (gnu packages certs)
+  #:use-module (gnu packages check)
+  #:use-module (gnu packages compression)  ; unzip for embedded python-wtforms
+  #:use-module (gnu packages databases)
+  #:use-module (gnu packages libffi)  ; cffi for embedded python-soundfile
+  #:use-module (gnu packages openldap)
+  #:use-module (gnu packages pdf)
+  #:use-module (gnu packages pkg-config)  ; embedded libsndfile
+  #:use-module (gnu packages python)
+  #:use-module (gnu packages python-crypto)
+  #:use-module (gnu packages python-web)
+  #:use-module (gnu packages python-xyz)
+  #:use-module (gnu packages sphinx)
+  #:use-module (gnu packages gstreamer)
+  #:use-module (gnu packages glib)
+  #:use-module (gnu packages pulseaudio)
+  #:use-module (gnu packages rsync)
+  #:use-module (gnu packages ssh)
+  #:use-module (gnu packages time)
+  #:use-module (gnu packages video)
+  #:use-module (gnu packages version-control)
+  #:use-module (gnu packages xml)
+  #:use-module ((guix licenses) #:select (bsd-3 gpl2+) #:prefix license:))
+
+(define python-wtforms
+  (package
+    (name "python-wtforms")
+    (version "2.3.3")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "WTForms" version))
+       (sha256
+        (base32
+         ;; Interesting, if this has is that of a lower version, it blindly
+         ;; ignores the version number above and you silently get the older
+         ;; version.
+         "17427m7p9nn9byzva697dkykykwcp2br3bxvi8vciywlmkh5s6c1"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f))  ; TODO: Fix tests for upgraded version.
+    (propagated-inputs
+    `(("python-markupsafe" ,python-markupsafe)))
+    (native-inputs
+     `(("unzip" ,unzip)))  ; CHECK WHETHER NEEDED - not in `guix import` but is in old package.
+    (home-page "http://wtforms.simplecodes.com/")
+    (synopsis
+     "Form validation and rendering library for Python web development")
+    (description
+     "WTForms is a flexible forms validation and rendering library
+for Python web development.  It is very similar to the web form API
+available in Django, but is a standalone package.")
+    (license license:bsd-3)))
+
+
+;; Copied from guix/gnu/packages/pulseaudio.scm in the core-updates branch which
+;; adds flac/ogg/vorbis/opus support. This is required for building
+;; python-soundfile (March 2021).
+(define libsndfile
+  (package
+    (name "libsndfile")
+    (version "1.0.30")
+    (source (origin
+             (method url-fetch)
+             (uri (string-append "https://github.com/erikd/libsndfile"
+                                 "/releases/download/v" version
+                                 "/libsndfile-" version ".tar.bz2"))
+             (sha256
+              (base32
+               "06k1wj3lwm7vf21s8yqy51k6nrkn9z610bj1gxb618ag5hq77wlx"))
+             (modules '((ice-9 textual-ports) (guix build utils)))
+             (snippet
+              '(begin
+                 ;; Remove carriage returns (CRLF) to prevent bogus
+                 ;; errors from bash like "$'\r': command not found".
+                 (let ((data (call-with-input-file
+                                 "tests/pedantic-header-test.sh.in"
+                               (lambda (port)
+                                 (string-join
+                                  (string-split (get-string-all port)
+                                                #\return))))))
+                   (call-with-output-file "tests/pedantic-header-test.sh.in"
+                     (lambda (port) (format port data))))
+
+                 ;; While at it, fix hard coded executable name.
+                 (substitute* "tests/test_wrapper.sh.in"
+                   (("^/usr/bin/env") "env"))
+                 #t))))
+    (build-system gnu-build-system)
+    (propagated-inputs
+     `(("flac" ,flac)
+       ("libogg" ,libogg)
+       ("libvorbis" ,libvorbis)
+       ("opus" ,opus)))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)
+       ("python" ,python)))
+    (home-page "http://www.mega-nerd.com/libsndfile/")
+    (synopsis "Reading and writing files containing sampled sound")
+    (description
+     "Libsndfile is a C library for reading and writing files containing
+sampled sound (such as MS Windows WAV and the Apple/SGI AIFF format) through
+one standard library interface.
+
+It was designed to handle both little-endian (such as WAV) and
+big-endian (such as AIFF) data, and to compile and run correctly on
+little-endian (such as Intel and DEC/Compaq Alpha) processor systems as well
+as big-endian processor systems such as Motorola 68k, Power PC, MIPS and
+SPARC.  Hopefully the design of the library will also make it easy to extend
+for reading and writing new sound file formats.")
+    (license license:gpl2+)))
+
+;; Need soundfile for new Python 3 audio spectrograms. Can me merged into Guix
+;; once core-updates is merged.
+(define python-soundfile
+  (package
+    (name "python-soundfile")
+    (version "0.10.3.post1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "SoundFile" version))
+       (sha256
+        (base32
+         "0yqhrfz7xkvqrwdxdx2ydy4h467sk7z3gf984y1x2cq7cm1gy329"))))
+    (build-system python-build-system)
+    (native-inputs
+     `(("python-pytest" ,python-pytest)))
+    (propagated-inputs
+     `(("python-cffi" ,python-cffi)
+       ("libsndfile" ,libsndfile)
+       ("python-numpy" ,python-numpy)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'set-library-file-name
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let ((libsndfile (assoc-ref inputs "libsndfile")))
+               (substitute* "soundfile.py"
+                 (("_find_library\\('sndfile'\\)")
+                  (string-append "'" libsndfile "/lib/libsndfile.so.1'")))
+               #t))))))
+    (home-page "https://github.com/bastibe/python-soundfile")
+    (synopsis "An audio library based on libsndfile, CFFI and NumPy")
+    (description
+     "The soundfile module can read and write sound files, representing audio
+data as NumPy arrays.")
+    (license license:bsd-3)))
+
+;; =================================================================
+
+(define-public mediagoblin
+  (package
+    (name "mediagoblin")
+    (version "0.12.0.dev")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://git.savannah.gnu.org/git/mediagoblin.git")
+             (commit "3d72ccf4dfb106d8b9dcc5ac4cb754d6efa80649")))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32 "0i6h8781nlnnqrw8jf0n0264p2fvc3z9rcp8vzv4a7aghiqmn2b2"))))
+    (build-system python-build-system)
+    (arguments
+     `(
+       #:tests? #f
+       ;; #:phases (modify-phases %standard-phases
+       ;;            (replace 'check
+       ;;              (lambda _
+       ;;                (setenv "PYTHONPATH"
+       ;;                        (string-append (getcwd) ":"
+       ;;                                       (getenv "PYTHONPATH")))
+       ;;                ;; Translations needed for tests to pass. Probably
+       ;;                ;; should be done during build stage?
+       ;;                (invoke "./devtools/compile_translations.sh")
+       ;;                (invoke "pytest" "./mediagoblin/tests" "-rs" "--boxed")
+       ;;                #t)))
+                ))
+    (native-inputs
+     `(("nss-certs" ,nss-certs)
+       ("python-pytest" ,python-pytest)
+       ("python-pytest-forked" ,python-pytest-forked)
+       ("python-pytest-xdist" ,python-pytest-xdist)
+       ("python-webtest" ,python-webtest)))
+    ;; lle-bout suggests avoiding propagated-inputs and insteading creating
+    ;; wrappers scripts. See:
+    ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47260#44
+    (propagated-inputs
+     `(("python-alembic" ,python-alembic)
+       ("python-babel" ,python-babel)
+       ("python-celery" ,python-celery)
+       ("python-configobj" ,python-configobj)
+       ("python-dateutil" ,python-dateutil)
+       ("python-docutils" ,python-docutils)  ; What for?
+       ("python-email-validator" ,python-email-validator)
+       ("python-exif-read" ,python-exif-read)
+       ("python-feedgenerator" ,python-feedgenerator)
+       ("python-itsdangerous" ,python-itsdangerous)
+       ("python-jinja2" ,python-jinja2)
+       ("python-jsonschema" ,python-jsonschema)
+       ("python-ldap" ,python-ldap)  ; For LDAP plugin
+       ("python-lxml" ,python-lxml)
+       ("python-markdown" ,python-markdown)
+       ("python-oauthlib" ,python-oauthlib)
+       ("python-openid" ,python-openid)  ; For OpenID plugin
+       ("python-pastescript" ,python-pastescript)
+       ("python-pillow" ,python-pillow)
+       ("python-py-bcrypt" ,python-py-bcrypt)
+       ("python-pyld" ,python-pyld)
+       ("python-pytz" ,python-pytz)
+       ("python-requests" ,python-requests)  ; For Persona plugin, batchaddmedia
+       ("python-setuptools" ,python-setuptools)  ; What for?
+       ("python-soundfile" ,python-soundfile)
+       ("python-sphinx" ,python-sphinx)
+       ("python-sqlalchemy" ,python-sqlalchemy)
+       ("python-translitcodec" ,python-translitcodec)
+       ("python-unidecode" ,python-unidecode)
+       ("python-waitress" ,python-waitress)
+       ("python-werkzeug" ,python-werkzeug)
+       ("python-wtforms" ,python-wtforms)
+
+       ;; Audio/video media
+       ("gobject-introspection" ,gobject-introspection)
+       ("gst-libav" ,gst-libav)
+       ("gst-plugins-bad" ,gst-plugins-bad)
+       ("gst-plugins-base" ,gst-plugins-base)
+       ("gst-plugins-good" ,gst-plugins-good)
+       ("gst-plugins-ugly" ,gst-plugins-ugly)
+       ("gstreamer" ,gstreamer)
+       ("openh264" ,openh264)
+       ("python-gst" ,python-gst)  ; For tests to pass
+       ("python-numpy" ,python-numpy)  ; Audio spectrograms
+       ("python-pygobject" ,python-pygobject)
+
+       ;; PDF media.
+       ("poppler" ,poppler)
+
+       ))
+    (home-page "https://mediagoblin.org/")
+    (synopsis "Web application for media publishing")
+    (description "MediaGoblin is a web application for publishing all kinds of
+media.")
+    (license agpl3+)))
-- 
2.31.1


^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-04-05 15:50       ` Léo Le Bouter via Bug reports for GNU Guix
@ 2021-04-06 12:05         ` Ben Sturmfels via Bug reports for GNU Guix
  0 siblings, 0 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-04-06 12:05 UTC (permalink / raw)
  To: Léo Le Bouter; +Cc: cwebber, 47260, jgart

On Mon, 05 Apr 2021, Léo Le Bouter wrote:

> On Tue, 2021-04-06 at 00:17 +1000, Ben Sturmfels wrote:
>> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
>> 
>> > 7. Work out why H264 support is missing.
>> 
>> This is now fixed MediaGoblin's master branch guix-env.scm by adding
>> gst-libav to propagated inputs.
>
> Hello!
>
> I suggest not using propagated-inputs because they are likely to cause
> conflicts in profiles when used. I suggest intead creating wrapper
> scripts that append to the PATH variable.
>
> Examples of such wrapper scripts can be found:
>
> - 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=373e5fc96724fd38bb1263e4af90932ea36f596b
> (PYTHONPATH more so)
> - 
> https://git.savannah.gnu.org/cgit/guix.git/commit/?id=00c1793ce8e2210e48b18422ea3e76da10541874
> (Append xdg-utils to PATH)
>
> Let me know if there's any questions about creating such wrappers.

Thanks Léo, I'll look into this!

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-04-06 12:01     ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-04-07 13:15       ` Ben Sturmfels via Bug reports for GNU Guix
  0 siblings, 0 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-04-07 13:15 UTC (permalink / raw)
  To: 47260; +Cc: cwebber, Léo Le Bouter, jgart

On Tue, 06 Apr 2021, Ben Sturmfels wrote:

> On Thu, 01 Apr 2021, Ben Sturmfels wrote:
>
>> 5. Get a basic Guix service working, with sqlite3 and without the
>> offloaded media transcoding currently using Celery task queue with a
>> Redis broker.
>
> Woo! After a lot of trial and error, I finally have a basic MediaGoblin
> running entirely under Guix with no virtualenv trickery!
>
> After applying the attached patch to my guix repo...

Even simpler, I've now created a Guix channel for our work-in-progress
MediaGoblin package/service. See the README for instructions on enabling
the channel and installing MediaGoblin:

https://gitlab.com/BenSturmfels/mediagoblin-guix/-/blob/master/README.md

Once the channel is enabled, you can `guix install mediagoblin` and run
`gmg` commands and the web interface. All documented in the above README.

Regards,
Ben




^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-05 14:17     ` Ben Sturmfels via Bug reports for GNU Guix
  2021-04-06 12:01     ` Ben Sturmfels via Bug reports for GNU Guix
@ 2021-05-04 20:58     ` Dr. Arne Babenhauserheide
  2021-05-06  0:49       ` Ben Sturmfels via Bug reports for GNU Guix
  2 siblings, 1 reply; 17+ messages in thread
From: Dr. Arne Babenhauserheide @ 2021-05-04 20:58 UTC (permalink / raw)
  To: Ben Sturmfels; +Cc: 47260, jgart


[-- Attachment #1.1: Type: text/plain, Size: 156 bytes --]

Hi,

I just added non-flickering video-change to the m3u-player. Attaching
the file. I thought that could be useful for MediaGoblin. The file is
attached.


[-- Attachment #1.2: m3u-player.js --]
[-- Type: application/octet-stream, Size: 11050 bytes --]

// [[file:m3u-player.org::*The script][The script:1]]
// @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2-or-Later
const nodes = document.querySelectorAll("audio,video");
const playlists = {};
const prefetchedTracks = new Map(); // use a map for insertion order, so we can just blow away old entries.
// maximum prefetched blobs that are kept.
const MAX_PREFETCH_KEEP = 10;
// maximum allowed number of entries in a playlist to prevent OOM attacks against the browser with self-referencing playlists
const MAX_PLAYLIST_LENGTH = 1000;
const PLAYLIST_MIME_TYPES = ["audio/x-mpegurl", "audio/mpegurl", "application/vnd.apple.mpegurl","application/mpegurl","application/x-mpegurl"];
function stripUrlParameters(link) {
  const url = new URL(link, window.location);
  url.search = "";
  url.hash = "";
  return url.href;
}
function isPlaylist(link) {
  const linkHref = stripUrlParameters(link);
  return linkHref.endsWith(".m3u") || linkHref.endsWith(".m3u8");
}
function isBlob(link) {
  return new URL(link, window.location).protocol == 'blob';
}
function parsePlaylist(textContent) {
  return textContent.match(/^(?!#)(?!\s).*$/mg)
    .filter(s => s); // filter removes empty strings
}
/**
 * Download the given playlist, parse it, and store the tracks in the
 * global playlists object using the url as key.
 *
 * Runs callback once the playlist downloaded successfully.
 */
function fetchPlaylist(url, onload, onerror) {
  const playlistFetcher = new XMLHttpRequest();
  playlistFetcher.open("GET", url, true);
  playlistFetcher.responseType = "blob"; // to get a mime type
  playlistFetcher.onload = () => {
    if (PLAYLIST_MIME_TYPES.includes(playlistFetcher.response.type)) { // security check to ensure that filters have run
      const reader = new FileReader();
      const load = onload; // propagate to inner scope
      reader.addEventListener("loadend", e => {
        playlists[url] = parsePlaylist(reader.result);
        onload();
      });
      reader.readAsText(playlistFetcher.response);
    } else {
      console.error("playlist must have one of the playlist MIME type '" + PLAYLIST_MIME_TYPES + "' but it had MIME type '" + playlistFetcher.response.type + "'.");
      onerror();
    }
  };
  playlistFetcher.onerror = onerror;
  playlistFetcher.abort = onerror;
  playlistFetcher.send();
}
function servedPartialDataAndCanRequestAll (xhr) {
  if (xhr.status === 206) {
    if (xhr.getResponseHeader("content-range").includes("/")) {
      if (!xhr.getResponseHeader("content-range").includes("/*")) {
        return true;
      }
    }
  }
  return false;
}
function prefetchTrack(url, onload) {
  if (prefetchedTracks.has(url)) {
    return;
  }
  // first cleanup: kill the oldest entries until we're back at the allowed size
  while (prefetchedTracks.size > MAX_PREFETCH_KEEP) {
    const key = prefetchedTracks.keys().next().value;
    const track = prefetchedTracks.get(key);
    prefetchedTracks.delete(key);
  }
  // first set the prefetched to the url so we will never request twice
  prefetchedTracks.set(url, url);
  // now start replacing it with a blob
  const xhr = new XMLHttpRequest();
  xhr.open("GET", url, true);
  xhr.responseType = "blob";
  xhr.onload = () => {
    if (servedPartialDataAndCanRequestAll(xhr)) {
      const endRange = Number(xhr.getResponseHeader("content-range").split("/")[1]) - 1;
      const rangeXhr = new XMLHttpRequest();
      rangeXhr.open("GET", url, true);
      rangeXhr.responseType = "blob";
      rangeXhr.setRequestHeader("range", "bytes=0-" + endRange);
      rangeXhr.onload = () => {
        prefetchedTracks.set(url, rangeXhr.response);
        if (onload) {
          onload();
        }
      };
      rangeXhr.send();      
    } else {
      prefetchedTracks.set(url, xhr.response);
      if (onload) {
        onload();
      }
    }
  };
  xhr.send();
}
function updateSrc(mediaTag, callback) {
  const playlistUrl = mediaTag.getAttribute("playlist");
  const trackIndex =  mediaTag.getAttribute("track-index");
  // deepcopy playlists to avoid shared mutation
  let playlist = [...playlists[playlistUrl]];
  let trackUrl = playlist[trackIndex];
  // download and splice in playlists as needed
  if (isPlaylist(trackUrl)) {
    if (playlist.length >= MAX_PLAYLIST_LENGTH) {
      // skip playlist if we already have too many tracks
      changeTrack(mediaTag, +1);
    } else {
      // do not use the cached playlist here, though it is tempting: it might genuinely change to allow for updates
      fetchPlaylist(
        trackUrl,
        () => {
          playlist.splice(trackIndex, 1, ...playlists[trackUrl]);
          playlists[playlistUrl] = playlist;
          updateSrc(mediaTag, callback);
        },
        () => callback());
    }
  } else {
    let url = prefetchedTracks.has(trackUrl)
        ? prefetchedTracks.get(trackUrl) instanceof Blob
        ? URL.createObjectURL(prefetchedTracks.get(trackUrl))
        : trackUrl : trackUrl;
    const oldUrl = mediaTag.getAttribute("src");
    // prevent size flickering by setting height before src change
    const canvas = document.createElement("canvas");
    if (!isNaN(mediaTag.duration)) { // already loaded a valid file so the size should fit
      // fix height to the height of the current video. Re-run after setting the source.
      mediaTag.height = (mediaTag.clientWidth * mediaTag.videoHeight) / mediaTag.videoWidth;
      // take screenshot of video and overlay it to mask flicker
      canvas.width = mediaTag.clientWidth;
      canvas.height = mediaTag.clientHeight;
      const context = canvas.getContext("2d");
      context.scale(mediaTag.clientWidth / mediaTag.videoWidth, mediaTag.clientHeight / mediaTag.videoHeight);
      context.drawImage(mediaTag, 0, 0);
      canvas.hidden = true;
      mediaTag.parentNode.insertBefore(canvas, mediaTag.nextSibling);
      canvas.style.position = "absolute";
      canvas.style.marginLeft = "-" + mediaTag.clientWidth + "px";
      canvas.hidden = false;
    }
    mediaTag.setAttribute("src", url);
    mediaTag.oncanplaythrough = () => {
      if (!isNaN(mediaTag.duration)) { // already loaded a valid file
        // fix height to the height of the current video. Re-run after setting the source.
        mediaTag.height = (mediaTag.clientWidth * mediaTag.videoHeight) / mediaTag.videoWidth;
      }
      // remove overlay
      canvas.hidden = true;
      canvas.remove(); // to allow garbage collection
    };
    setTimeout(() => canvas.remove(), 300); // fallback
    // replace the url when done, because a blob from an xhr request
    // is more reliable in the media tag;
    // the normal URL caused jumping prematurely to the next track.
    if (url == trackUrl) {
      prefetchTrack(trackUrl, () => {
        if (mediaTag.paused) {
          if (url == mediaTag.getAttribute("src")) {
            if (mediaTag.currentTime === 0) {
              mediaTag.setAttribute("src", URL.createObjectURL(
                prefetchedTracks.get(url)));
            }
          }
        }
      });
    }
    // allow releasing memory
    if (isBlob(oldUrl)) {
      URL.revokeObjectURL(oldUrl);
    }
    // update title
    mediaTag.parentElement.querySelector(".m3u-player--title").title = trackUrl;
    mediaTag.parentElement.querySelector(".m3u-player--title").textContent = trackUrl;
    // start prefetching the next three tracks.
    for (const i of [1, 2, 3]) {
      if (playlist.length > Number(trackIndex) + i) {
        prefetchTrack(playlist[Number(trackIndex) + i]);
      }
    }
    callback();
  }
}
function changeTrack(mediaTag, diff) {
  const currentTrackIndex = Number(mediaTag.getAttribute("track-index"));
  const nextTrackIndex = currentTrackIndex + diff;
  const tracks = playlists[mediaTag.getAttribute("playlist")];
  if (nextTrackIndex >= 0) { // do not collapse the if clauses with double-and, that does not survive inlining
    if (tracks.length > nextTrackIndex) {
    mediaTag.setAttribute("track-index", nextTrackIndex);
      updateSrc(mediaTag, () => mediaTag.play());
    }
  }
}

/**
 * Turn a media tag into playlist player.
 */
function initPlayer(mediaTag) {
  mediaTag.setAttribute("playlist", mediaTag.getAttribute("src"));
  mediaTag.setAttribute("track-index", 0);
  const url = mediaTag.getAttribute("playlist");
  const wrapper = mediaTag.parentElement.insertBefore(document.createElement("div"), mediaTag);
  const controls = document.createElement("div");
  const left = document.createElement("span");
  const title = document.createElement("span");
  const right = document.createElement("span");
  controls.appendChild(left);
  controls.appendChild(title);
  controls.appendChild(right);
  left.classList.add("m3u-player--left");
  right.classList.add("m3u-player--right");
  title.classList.add("m3u-player--title");
  title.style.overflow = "hidden";
  title.style.textOverflow = "ellipsis";
  title.style.whiteSpace = "nowrap";
  title.style.opacity = "0.3";
  title.style.direction = "rtl"; // for truncation on the left
  title.style.paddingLeft = "0.5em";
  title.style.paddingRight = "0.5em";
  controls.style.display = "flex";
  controls.style.justifyContent = "space-between";
  const styleTag = document.createElement("style");
  styleTag.innerHTML = ".m3u-player--left:hover, .m3u-player--right:hover {color: wheat; background-color: DarkSlateGray}";
  wrapper.appendChild(styleTag);
  wrapper.appendChild(controls);
  controls.style.width = mediaTag.getBoundingClientRect().width.toString() + "px";
  // appending the media tag to the wrapper removes it from the outer scope but keeps the event listeners
  wrapper.appendChild(mediaTag);
  left.innerHTML = "&lt;"; // not textContent, because we MUST escape
                           // the tag here and textContent shows the
                           // escaped version
  left.onclick = () => changeTrack(mediaTag, -1);
  right.innerHTML = "&gt;";
  right.onclick = () => changeTrack(mediaTag, +1);
  fetchPlaylist(
    url,
    () => {
      updateSrc(mediaTag, () => null);
      mediaTag.addEventListener("ended", event => {
        if (mediaTag.currentTime >= mediaTag.duration) {
          changeTrack(mediaTag, +1);
        }
      });
    },
    () => null);
  // keep the controls aligned to the media tag
  mediaTag.resizeObserver = new ResizeObserver(entries => {
    controls.style.width = entries[0].contentRect.width.toString() + "px";
  });
  mediaTag.resizeObserver.observe(mediaTag);
}
function processTag(mediaTag) {
  const canPlayClaim = mediaTag.canPlayType('audio/x-mpegurl');
  let supportsPlaylists = !!canPlayClaim;
  if (canPlayClaim == 'maybe') { // yes, seriously: specced as you only know when you try
    supportsPlaylists = false;
  }
  if (!supportsPlaylists) {
    if (isPlaylist(mediaTag.getAttribute("src"))) {
      initPlayer(mediaTag);
    }
  }
}
document.addEventListener('DOMContentLoaded', () => {
  const nodes = document.querySelectorAll("audio,video");
  nodes.forEach(processTag);
});
// @license-end
// The script:1 ends here

[-- Attachment #1.3: Type: text/plain, Size: 1850 bytes --]


Best wishes,
Arne


Ben Sturmfels via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> On Tue, 30 Mar 2021, Ben Sturmfels wrote:
>
>> On Fri, 19 Mar 2021, Ben Sturmfels wrote:
>
>>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>>> implicated in many bugs anyway, so there may benefits to the project to
>>> doing this anyway.
>>
>> I learnt that Celery has a Redis backend, so maybe we don't need to
>> rewrite just yet.
>
> It turns out that MediaGoblin's Celery-based media processing backend
> work out of the box by simply configuring:
>
>   [celery]
>   BROKER_URL = "redis://"
>
> (There seems to be an unrelated bug where media is marked as failed after
> restarting Celery, possibly tied to sqlite. We've had reports of this
> with a RabbitMQ broker too though.)
>
>
> This means our shorter to-do list is now:
>
> 1. Upstream our new python-soundfile Guix package from guix-env.scm when
> core-updates is merged.
>
> 2. Upstream our upgraded python-wtforms package.
>
> 6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
> Video and audio are essentially functional without the NPM installed
> players. Some later refinements perhaps.
>
> 4. Package MediaGoblin itself. The build process is ./configure/make
> which is a bit weird for a Python project.
>
> 5. Get a basic Guix service working, with sqlite3 and without the
> offloaded media transcoding currently using Celery task queue with a
> Redis broker.
>
> 7. Work out why H264 support is missing.
>
> 8. Figure out how to deal with translations.
>
> 9. Add a PostgreSQL database to the Guix service instead of sqlite3.
>
> Regards,
> Ben


-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* bug#47260: Package GNU MediaGoblin as a Guix service
  2021-05-04 20:58     ` Dr. Arne Babenhauserheide
@ 2021-05-06  0:49       ` Ben Sturmfels via Bug reports for GNU Guix
  0 siblings, 0 replies; 17+ messages in thread
From: Ben Sturmfels via Bug reports for GNU Guix @ 2021-05-06  0:49 UTC (permalink / raw)
  To: Dr. Arne Babenhauserheide; +Cc: cwebber, 47260, jgart, lle-bout

Thanks for the update Arne. This issue is specifically about Guix
packaging, so to save us losing track of your update, please post it to
the dedicated mediagoblin-devel@gnu.org thread we started a couple of
months back:

https://lists.gnu.org/archive/html/mediagoblin-devel/2021-03/msg00026.html

Thanks again,
Ben

On Wed, 05 May 2021, Arne Babenhauserheide wrote:

> Hi,
>
> I just added non-flickering video-change to the m3u-player. Attaching
> the file. I thought that could be useful for MediaGoblin. The file is
> attached.
>
>
> Best wishes,
> Arne
>
>
> Ben Sturmfels via Bug reports for GNU Guix <bug-guix@gnu.org> writes:
>
>> On Tue, 30 Mar 2021, Ben Sturmfels wrote:
>>
>>> On Fri, 19 Mar 2021, Ben Sturmfels wrote:
>>
>>>> 8. Either package RabbitMQ (probably hard) or rewrite MediaGoblin's
>>>> processing backend from Celery/RabbitMQ to RQ/Redis. Celery has been
>>>> implicated in many bugs anyway, so there may benefits to the project to
>>>> doing this anyway.
>>>
>>> I learnt that Celery has a Redis backend, so maybe we don't need to
>>> rewrite just yet.
>>
>> It turns out that MediaGoblin's Celery-based media processing backend
>> work out of the box by simply configuring:
>>
>>   [celery]
>>   BROKER_URL = "redis://"
>>
>> (There seems to be an unrelated bug where media is marked as failed after
>> restarting Celery, possibly tied to sqlite. We've had reports of this
>> with a RabbitMQ broker too though.)
>>
>>
>> This means our shorter to-do list is now:
>>
>> 1. Upstream our new python-soundfile Guix package from guix-env.scm when
>> core-updates is merged.
>>
>> 2. Upstream our upgraded python-wtforms package.
>>
>> 6. Convert MediaGoblin's jQuery-based JavaScript to use vanilla JS.
>> Video and audio are essentially functional without the NPM installed
>> players. Some later refinements perhaps.
>>
>> 4. Package MediaGoblin itself. The build process is ./configure/make
>> which is a bit weird for a Python project.
>>
>> 5. Get a basic Guix service working, with sqlite3 and without the
>> offloaded media transcoding currently using Celery task queue with a
>> Redis broker.
>>
>> 7. Work out why H264 support is missing.
>>
>> 8. Figure out how to deal with translations.
>>
>> 9. Add a PostgreSQL database to the Guix service instead of sqlite3.
>>
>> Regards,
>> Ben





^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2021-05-06  0:50 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-19 12:20 bug#47260: Package GNU MediaGoblin as a Guix service Ben Sturmfels via Bug reports for GNU Guix
2021-03-19 15:50 ` jgart via Bug reports for GNU Guix
2021-03-21 23:28   ` Ben Sturmfels via Bug reports for GNU Guix
2021-03-22  7:02     ` Dr. Arne Babenhauserheide
2021-03-30  4:02       ` Ben Sturmfels via Bug reports for GNU Guix
2021-03-30  6:40         ` Dr. Arne Babenhauserheide
2021-03-22 17:58     ` Christopher Lemmer Webber
2021-03-30  4:12 ` Ben Sturmfels via Bug reports for GNU Guix
2021-03-30 12:13   ` Ben Sturmfels via Bug reports for GNU Guix
2021-04-01  2:03   ` Ben Sturmfels via Bug reports for GNU Guix
2021-04-05 14:17     ` Ben Sturmfels via Bug reports for GNU Guix
2021-04-05 15:50       ` Léo Le Bouter via Bug reports for GNU Guix
2021-04-06 12:05         ` Ben Sturmfels via Bug reports for GNU Guix
2021-04-06 12:01     ` Ben Sturmfels via Bug reports for GNU Guix
2021-04-07 13:15       ` Ben Sturmfels via Bug reports for GNU Guix
2021-05-04 20:58     ` Dr. Arne Babenhauserheide
2021-05-06  0:49       ` Ben Sturmfels via Bug reports for GNU Guix

all messages for Guix-related lists mirrored at yhetil.org

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix

Example config snippet for mirrors.


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git