Video File Support for Cinematic Cutscenes

[JTDaddy17]

For those who want to make things simpler without using complex Storyboard files, why not add video format support for the intro and story mode cutscenes? GO seems to support something like that!

[ghost]

I like this idea! I got frustrated to capture footage frame by frame.

[ZyonZZ]

Also the loading screen..
have the possibility to add a video..
to be preloaded first... and during playback..
loading of characters, settings, etc...

[BaTung3]

Something like that would be Great
Also does Storybiard framerate limited by Gamespeed / tick per sec ??

Cause If so custom framerate for storybiard would be Fantastic too

[SuperFromND]

Oh hey, this has a discussion already, saves me some time! Writing some notes here since this came up in the Discord tonight:

So one of our developers (Sam) actually did put together a proof of concept branch a while ago that implements MP4 support, and code-wise it doesn't actually seem too hard to update it for our current codebase.

2022-10-25_01-43-18.mp4

The main reason this wasn't finished up and made into a PR it seems (outside of Sam being inactive as far as Ikemen goes), is due to the size of the FFMPEG library that it used to decode video, and the resulting colossal number of codecs it supports. To quote the man himself:

Using the standard FFmpeg library for video playback seems a bit impractical for now. It would have to ship with more than 100 MiB of additional libraries because of the insane number of codecs it supports.
I think we’d need to restrict the number of supported codecs, maybe just H.264 and MP3? What are the most common codecs nowadays?

I haven't built this proof-of-concept myself to see if it's actually that bad, but I wouldn't be surprised. I would definitely be down to look into seeing if I can figure out a way to limit what codecs we support and get this cleaned up and PR'd down the line, because I imagine a lot of creators are gonna find proper video support to be a killer feature. (Speaking as a video editor myself, I certainly agree!)

[potsmugen]

Restricting the number of codecs has only advantages I think. Less code bloat. Less to maintain.

[SuperFromND]

Agreed. I'm in the camp of limiting it to maybe H.264 for video and AAC and MP3 for audio since those are what I typically use in the vast majority of my video files. Just not sure how to actually go about limiting that given how Sam's fork currently implements video decoding with an external lib.

[SuperFromND]

Me from the future here! (season's greetings, I'm writing this on Christmas Day LOL)

Just now tried to build Sam's PoC after updating it to latest nightly and, unfortunately wasn't able to actually get anything to build. Something in Reisen seemed to really not like being compiled under Windows; depending on the version it either complains about a missing pkg-config program it couldn't run mid-compile, or missing FFMpeg library files (despite having already downloaded them via MSYS2 package manager, which is what Reisen's own documentation recommends.)

While I'm here, someone also mentioned this AV1 utilities library (among others; the same repo also contains a similar library for MP4s) that could theoretically be made into a video decoder. Unfortunately, I wasn't able to find any real documentation on this library or any simple examples (the only program that uses it is a web-streaming server, akin to Plex or Jellyfin).

If there happens to be someone who has the know-how to turn this library into something that can spit out RGBA video frames from an input file like what Reisen does, this'd be pretty straightforward to do.