Audio frametime, need to long.

[Blendi87]

After a good 5 minutes of playing, I get more and more of these error messages about the audio until the bot restarts. i mostly play .mkv files.

WARN stream:audio:send Frame takes too long to send (118.50% frametime) {
WARN stream:audio:send "frame_size": 164,
WARN stream:audio:send "duration": 23.699765000026673,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +0ms
WARN stream:audio:send Frame takes too long to send (155.25% frametime) {
WARN stream:audio:send "frame_size": 159,
WARN stream:audio:send "duration": 31.0492090000771,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +26s
WARN stream:audio:send Frame takes too long to send (131.29% frametime) {
WARN stream:audio:send "frame_size": 172,
WARN stream:audio:send "duration": 26.25833599991165,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +35s
WARN stream:audio:send Frame takes too long to send (258.26% frametime) {
WARN stream:audio:send "frame_size": 182,
WARN stream:audio:send "duration": 51.651254999917,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +3s
WARN stream:audio:send Frame takes too long to send (117.26% frametime) {
WARN stream:audio:send "frame_size": 157,
WARN stream:audio:send "duration": 23.451699999859557,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +135ms
WARN stream:audio:send Frame takes too long to send (417.09% frametime) {
WARN stream:audio:send "frame_size": 165,
WARN stream:audio:send "duration": 83.41887400019914,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +2s

and so on

WARN stream:audio:send Frame takes too long to send (660.03% frametime) {
WARN stream:audio:send "frame_size": 180,
WARN stream:audio:send "duration": 132.00662300002296,
WARN stream:audio:send "frametime": 20
WARN stream:audio:send } +575ms

I think the audio is trying to stay in sync with the picture and it's getting worse and worse.

Whether “minimizeLatency” is on or off doesn't matter.

I am using the latest version "4.1.2".

[longnguyen2004]

Weird that it's only happening on audio and not video...
Anyway, that warning message isn't related to syncing, it's measuring the actual time to send the audio frame. What is your system specs?

[Blendi87]

intel n100 on Docker with 32gb ram for 480p .mkv files on a nvme.

it's just really noticeable that only the sound is affected and not the picture (the picture looks good).

btw. any chance of intel quick sync support? :)

[longnguyen2004]

N100 has AES-NI, it shouldn't be struggling like that, weird. Anyway you can try setting forceChacha20Encryption to true, this will force the use of a faster encryption algorithm.

About Quick Sync, this library doesn't care about the specific encoder. You can copy the streamLivestreamVideo function and modify that to your likings. However, we expect the media stream to satisfy some conditions:

• Video codec: H.264/H.265 (currently broken, unknown cause)/VP8 (-c:v h264_qsv)
• Audio codec: Opus (-c:a libopus)
• Container: Matroska (-f matroska)
• No B frames (-bf 0)
• Keyframes every 1s ideally. You can change this, but 1s is good enough (-force_key_frames "expr:gte(t,n_forced*1)")
• Forced IDR frames (-forced_idr)

I'm currently working on a new API that should make this easier to do without leaking too much internal workings of the library out to user code.

[Blendi87]

For the people who also like to have h265.vaapi and don't know how to customize the streamLivestreamVideo.ts,
is not quite perfect but it runs on an intel n100 gpu.

[out#0/matroska @ 0x632af8ff4ac0] sq: send 0 ts 242.75
[out#0/matroska @ 0x632af8ff4ac0] sq: receive 0 ts 242.75 queue head -1 ts N/A
[hevc_vaapi @ 0x632af901b4c0] Input frame: 1920x800 (5825).
[hevc_vaapi @ 0x632af901b4c0] Pick normal P-picture to encode next.
[hevc_vaapi @ 0x632af901b4c0] Issuing encode for pic 5825/5825 as type P.
[hevc_vaapi @ 0x632af901b4c0] L0 refers to 5824/5824.
[hevc_vaapi @ 0x632af901b4c0] Input surface is 0x14.
[hevc_vaapi @ 0x632af901b4c0] Recon surface is 0x1e.
[hevc_vaapi @ 0x632af901b4c0] Output buffer is 0xb.
[hevc @ 0x632af9aecd40] Output frame with POC 21.
[hevc @ 0x632af9aecd40] Param buffer (type 0, 604 bytes) is 0xe.
[hevc @ 0x632af9aecd40] Slice 0 param buffer (264 bytes) is 0x6.
[hevc @ 0x632af9aecd40] Slice 0 data buffer (6490 bytes) is 0xc.
[hevc @ 0x632af9aecd40] Decode to surface 0x13.
[hevc_vaapi @ 0x632af901b4c0] Param buffer (23) is 0x8.
[hevc_vaapi @ 0x632af901b4c0] Slice 0: 0-12 (13 rows), 0-389 (390 blocks).
[hevc_vaapi @ 0x632af901b4c0] RPS for POC 17: (16,1)
[hevc_vaapi @ 0x632af901b4c0] Packed header buffer (3) is 0xa/0x3 (80 bits).
[hevc_vaapi @ 0x632af901b4c0] Param buffer (24) is 0x9.
[hevc_vaapi @ 0x632af901b4c0] Sync to pic 5822/5822 (input surface 0x15).
[hevc_vaapi @ 0x632af901b4c0] Output buffer: 7189 bytes (status

The logs show that the gpu is used 99% of the time, except when creating the POC (Picture Order Count) during final coding,
I also assume that SPS, PPS and VPS are still processed without vaapi and the CPU is used for this.

############### streamLivestreamVideo.ts ###

import ffmpeg from 'fluent-ffmpeg';
import { AudioStream } from "./AudioStream.js";
import { MediaUdp } from '../client/voice/MediaUdp.js';
import { Readable, PassThrough } from 'stream';
import { VideoStream } from './VideoStream.js';
import { normalizeVideoCodec } from '../utils.js';
import PCancelable from 'p-cancelable';
import { demux } from './LibavDemuxer.js';

export function streamLivestreamVideo(
input: string | Readable,
mediaUdp: MediaUdp,
includeAudio = true,
customHeaders?: Record<string, string>
) {
return new PCancelable(async (resolve, reject, onCancel) => {
const streamOpts = mediaUdp.mediaConnection.streamOptions;
const videoCodec = normalizeVideoCodec(streamOpts.videoCodec);

    // ffmpeg setup
    let headers: Record<string, string> = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.3",
        "Connection": "keep-alive"
    }

    headers = { ...headers, ...(customHeaders ?? {}) };

    let isHttpUrl = false;
    let isHls = false;

    if (typeof input === "string") {
        isHttpUrl = input.startsWith('http') || input.startsWith('https');
        isHls = input.includes('m3u');
    }

    const ffmpegOutput = new PassThrough();
    try {
        // command creation
        const command = ffmpeg(input)
			.inputOption('-hwaccel', 'vaapi')
			.inputOption('-vaapi_device', '/dev/dri/renderD128')
			.inputOption('-hwaccel_output_format', 'vaapi')
			.inputOption('-thread_queue_size 1024')
			.addOption('-loglevel', 'debug')
			.on('end', () => {
			resolve("video ended");
            
            })
            .on("error", (err, stdout, stderr) => {
                reject('cannot play video ' + err.message)
            })
            .on('stderr', console.error);

        // general output options
        command
            .output(ffmpegOutput)
            //.size(`${streamOpts.width}x${streamOpts.height}`)
            .fpsOutput(streamOpts.fps)
            .videoBitrate(`${streamOpts.bitrateKbps}k`)
            .outputFormat("matroska");

        // video setup
        command.outputOption('-bf', '0');
        switch (videoCodec) {
            case 'AV1':
                command
                    .videoCodec("libsvtav1")
                break;
            case 'VP8':
                command
                    .videoCodec("libvpx")
                    .outputOption('-deadline', 'realtime');
                break;
            case 'VP9':
                command
                    .videoCodec("libvpx-vp9")
                    .outputOption('-deadline', 'realtime');
                break;
            case 'H264':
                command
                    .videoCodec("h264_vaapi")
                    .outputOptions([
						'-c:v h264_vaapi',
						'-c:a libopus',
                        '-profile:v 578',
                        `-g ${streamOpts.fps}`,
						'-bf 0',
						'-low_power 1',
						'-async_depth 4',
						'-b:a 128k',             
						'-ar 48000',             
						'-ac 2',                
						'-vbr on',             
						'-compression_level 10', 
						`-vf scale_vaapi=w=${streamOpts.width}:h=${streamOpts.height}:format=nv12,hwupload`
	]);

                break;
			case 'H265':
				command
					.videoCodec("hevc_vaapi")
					.outputOptions([
						'-c:v hevc_vaapi',
						'-c:a libopus',
						'-profile:v main',
						`-g ${streamOpts.fps}`, 
						'-bf 0',
						'-low_power 0',
						'-async_depth 2',
						'-b:a 128k',   
						'-ar 48000',
						'-ac 2',
						'-vbr on', 
						'-compression_level 6',
						`-vf scale_vaapi=w=${streamOpts.width}:h=${streamOpts.height}:format=nv12,hwupload`
						
    ]);
                break;
        }

        // audio setup

// command
// .audioChannels(2)
// .audioFrequency(48000)
// .audioCodec("libopus")
// //.audioBitrate('128k')

// if (streamOpts.hardwareAcceleratedDecoding) {
// command.inputOption('-hwaccel', 'vaapi');
// command.inputOption('-vaapi_device', '/dev/dri/renderD128'); // Das VAAPI-Gerät auf deinem System
// command.inputOption('-hwaccel_output_format', 'vaapi'); // Falls du das Output-Format auf VAAPI setzen möchtest
// }

        if (streamOpts.minimizeLatency) {
            command.addOptions([
                '-fflags nobuffer',
                '-analyzeduration 0'
            ])
        }

        if (isHttpUrl) {
            command.inputOption('-headers',
                Object.keys(headers).map(key => key + ": " + headers[key]).join("\r\n")
            );
            if (!isHls) {
                command.inputOptions([
                    '-reconnect 1',
                    '-reconnect_at_eof 1',
                    '-reconnect_streamed 1',
                    '-reconnect_delay_max 4294'
                ]);
            }
        }

        command.run();
        onCancel(() => command.kill("SIGINT"));

        // demuxing
        const { video, audio } = await demux(ffmpegOutput).catch((e) => {
            command.kill("SIGINT");
            throw e;
        });
        const videoStream = new VideoStream(mediaUdp);
        video!.stream.pipe(videoStream)
        if (audio && includeAudio) {
            const audioStream = new AudioStream(mediaUdp);
            audio.stream.pipe(audioStream);
            videoStream.syncStream = audioStream;
            audioStream.syncStream = videoStream;
        }
    } catch (e) {
        //audioStream.end();
        //videoStream.end();
        reject("cannot play video " + (e as Error).message);
    }
})

}

export function getInputMetadata(input: string | Readable): Promise<ffmpeg.FfprobeData> {
return new Promise((resolve, reject) => {
const instance = ffmpeg(input).on('error', (err, stdout, stderr) => reject(err));

    instance.ffprobe((err, metadata) => {
        if (err) reject(err);
        instance.removeAllListeners();
        resolve(metadata);
        instance.kill('SIGINT');
    });
})

}

export function inputHasAudio(metadata: ffmpeg.FfprobeData) {
return metadata.streams.some((value) => value.codec_type === 'audio');
}

export function inputHasVideo(metadata: ffmpeg.FfprobeData) {
return metadata.streams.some((value) => value.codec_type === 'video');
}

#############

[longnguyen2004]

@Blendi87 Please don't comment on unrelated issues, it causes unnecessary notifications on my side. Also, check out the new API, it should be much easier to customize encoder settings without dealing with library internals