Qt
Internal/Contributor docs for the Qt SDK. Note: These are NOT official API docs; those are found at https://doc.qt.io/
Loading...
Searching...
No Matches
backend-notes-gstreamer.qdoc
Go to the documentation of this file.
1// Copyright (C) 2024 The Qt Company Ltd.
2// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
3
4/*!
5\page qtmultimedia-gstreamer.html
6\title Qt Multimedia GStreamer backend
7\brief Platform notes for the GStreamer backend
8
9This page covers limitations and customization points of the GStreamer backend
10of Qt Multimedia, the default media backend on embedded Linux.
11
12\section1 Architectural Considerations
13
14GStreamer is the default media backend on embedded Linux mainly because it is
15the framework most embedded board vendors prioritize for hardware accelerated
16media support. FFmpeg may also be suitable on some platforms, so users are
17encouraged to experiment with both media backends to find out which works
18best for them.
19
20Qt Multimedia is not a general purpose streaming framework and not necessarily
21the architecturally best way to use GStreamer with Qt. Developers, who need a
22high degree of control over the GStreamer pipeline, but only want to show the
23video output Qt, may want to consider using GStreamer's
24\l{https://gstreamer.freedesktop.org/documentation/qml6/index.html}{qml6glsink}.
25
26
27
28
29\section1 Limitations and Known Issues
30
31GStreamer is not bundled with Qt, but it is typically deployed with the Linux distribution.
32
33\list
34\li Certain bugs may be due to the GStreamer version being used. We recommend to use the latest
35 GStreamer bug fix release on your platform.
36\li Certain bugs may also be related to the libraries used by GStreamer, like Pulseaudio. Most
37 notably Pulseaudio v16 has a known bug that causes the GStreamer pipeline to hang and requires
38 backports of these two patches:
39 \list
40 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/745}
41 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/764}
42 \endlist
43 This bug currently affects most mainstream Linux distributions, including Ubuntu 22.04, 23.10 and
44 24.04, Debian 11 and 12, as well as Fedora 39 and 40.
45\li Switching camera devices or cameras during recording is not supported
46\li Seeking, playback rates, loop, switching sinks have known bugs.
47\li Audio features requires PulseAudio. See \l{Qt Multimedia on Linux}{Linux platform notes}
48 for details.
49\li Some embedded boards needs individual configurations to solve pipeline
50negotiation/linking issues and improve performance. We recommend developers to
51experiment with the environment variables listed under \l{Customization points}
52when facing such problems. Qt's Yocto meta layer files contain recommended
53configurations for certain combinations of boards, BSPs and Boot to Qt
54versions. Here are links to the dev branch versions:
55 \list
56 \li NXP i.MX 6/8/9: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/freescale-layer/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
57 \li Raspberry Pi 4/5: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/raspberrypi/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
58 \endlist
59\l Some NXP i.MX boards expose a v4l2 device named imx-capture, which can cause
60errors when used with v4l2src in GStreamer. To prevent Qt Multimedia from
61adding imx-capture to the list of available camera devices, set the environment
62variable QT_GSTREAMER_SKIP_IMXCAPTURE=1. Note that this is considered as
63private API which will be removed when NXP fixes the problems with imx-capture.
64\li Video playback or camera streaming using system memory video frames or
65software conversion can result in dropped frames and unresponsiveness due to
66heavy CPU load. If the GStreamer decoder or camera device outputs DMA-BUF- or
67GLMemory-backed video frames in one of our supported pixel formats, Qt can
68render them without additional conversion or copying. GStreamer selects a
69format and memory type based on the capabilities of its pipeline elements.
70Different ways of controlling which elements are added to the pipeline are
71described under \l{Customization points}. See also
72\l{Specifying I/O mode for v4l2 cameras}.
73\endlist
74
75
76
77
78\section1 Customization points
79
80Qt Multimedia provides certain customization points to allow access to the
81underlying GStreamer pipeline. The entry point is \c{class
82QGStreamerPlatformSpecificInterface}. Additional customizations can be done by
83setting environment variables.
84
85\warning The customization points and Qt specific environment variables listed
86here, are considered as private APIs and may be subject to change. Be aware
87that environment variables are by default inherited by child processes and can
88have unintended consequences.
89
90
91\section2 External OpenGL texture rendering
92
93Qt imports DMA-BUF-backed video frames into the OpenGL graphics pipeline by
94binding the underlying buffer as an EGL image and exposing it to shaders as an
95OpenGL texture. By default, Qt samples these textures using the GL_TEXTURE_2D
96texture target and standard shader code. Some embedded boards performs better
97with the GL_TEXTURE_EXTERNAL_OES texture target instead, together with a
98dedicated fragment shader. This behavior can be enabled by setting the
99following environment variable:
100
101\code
102 QT_MULTIMEDIA_FORCE_GL_TEXTURE_EXTERNAL_OES=1
103\endcode
104
105\section2 Specifying I/O mode for v4l2 cameras
106
107When using a v4l2 capture device with a QCamera or QML Camera, the
108\l{https://gstreamer.freedesktop.org/documentation/video4linux2/v4l2src.html#v4l2src:io-mode}{io-mode}
109property of the underlying v4l2src GStreamer element affects whether or not it
110outputs DMA-BUF-backed video frames that allows zero-copy rendering. In some
111cases the default auto mode chooses mmap mode even though the camera driver
112supports dmabuf mode. The io-mode property int value can be set explicitly via
113the environment variable below. Here's an example showing how to set io-mode to
114dmabuf with the enum GST_V4L2_IO_DMABUF:
115
116\code
117 QT_GSTREAMER_V4L2SRC_IOMODE=4
118\endcode
119
120
121
122\section2 Selecting specific GStreamer elements
123
124A GStreamer distribution can include multiple different plugins that can
125potentially perform the same task in the media pipeline. GStreamer
126automatically selects elements like decoders and demuxers based on their
127feature rank, which can be overridden via a GStreamer environment variable.
128Here's an example that ranks the v4l2h264dec hardware decoder higher than any
129H.264 software decoder, while also making sure the aiurdemux demuxer isn't
130selected:
131
132\code
133 GST_PLUGIN_FEATURE_RANK=v4l2h264dec:MAX,aiurdemux:NONE
134\endcode
135
136To inspect which elements are being created during runtime, enable debug logging
137level 4 for GstElementFactory:
138
139\code
140 GST_DEBUG=GST_ELEMENT_FACTORY:4
141\endcode
142
143Furthermore, you can visually inspect the whole GStreamer pipeline by
144specifying a folder for \c {.dot} graph files that will be dumped at
145various events, for example, changes in playback state:
146
147\code
148 GST_DEBUG_DUMP_DOT_DIR=/root/graphs
149\endcode
150
151Read more about these environment variables in the GStreamer documentation:
152
153\list
154 \li \l{https://gstreamer.freedesktop.org/documentation/gstreamer/running.html?gi-language=c#:~:text=GST_PLUGIN_FEATURE_RANK}{Environment variables - GST_PLUGIN_FEATURE_RANK}
155 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#the-debug-log}{Basic tutorial - The debug log}
156 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#getting-pipeline-graphs}{Basic tutorial - Getting pipeline graphs}
157\endlist
158
159\section3 Specifying hardware conversion elements
160
161If the media source of a GStreamer pipeline cannot provide video buffers in a
162pixel format supported by Qt Multimedia, the pipeline will use software
163conversion unless it contains a suitable hardware accelerated video conversion
164element. To specify a hardware conversion element to be used, set it as a
165\l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html#elements}
166{GStreamer pipeline description} to the following environment variable:
167
168\code
169 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT
170\endcode
171
172Some vendor specific conversion elements (imxvideoconvert_g2d, nvvidconv) are
173added to the pipeline by default if available. This can cause unnecessary
174conversion if they aren't needed, and can be disabled by specifying a GStreamer
175\l{https://gstreamer.freedesktop.org/documentation/coreelements/identity.html}
176{identity} element using the same environment variable:
177
178\code
179 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=identity
180\endcode
181
182GStreamer can also perform format conversion with OpenGL elements, providing Qt
183with GLMemory-backed video frames:
184
185\code
186 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=glupload ! glcolorconvert
187\endcode
188
189
190\section2 Raw pipeline access
191
192The \c{GstPipeline} underlying the \l{QMediaPlayer} and \l{QMediaCaptureSession} can be accessed.
193
194\warning This is an unsafe API, as the pipeline is still managed by the Qt implementation. Great
195care is required when using this API.
196
197\code
198#include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>
199
200[...]
201QMediaMediaPlayer player;
202GstPipeline *pipeline = QGStreamerPlatformSpecificInterface::instance()->gstPipeline(&player);
203[...]
204QMediaCaptureSession session;
205GstPipeline *pipeline = QGSreamerPlatformSpecificInterface::instance()->gstPipeline(&session);
206\endcode
207
208
209\section2 Custom GStreamer elements as sinks and sources
210
211It is possible to create GStreamer elements from a GStreamer pipeline decription and wrap them
212inside a \c{QCamera} or \c{QAudioDevice}:
213
214\code
215#include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>
216
217[...]
218QByteArray pipelineString = "videotestsrc is-live=true ! gamma gamma=2.0";
219
220QMediaCaptureSession session;
221session.setVideoSink(wid.videoSink());
222
223QCamera *cam = QGStreamerPlatformSpecificInterface::instance()->makeCustomGStreamerCamera(
224 pipelineString, &session);
225session.setCamera(cam);
226\endcode
227
228
229\section2 QMediaPlayer: custom sources
230
231The \c{QMediaPlayer} accepts a GStreamer pipeline decription as source URI:
232
233\code
234QMediaPlayer player;
235player.setSource(u"gstreamer-pipeline: videotestsrc name=testsrc"_s);
236\endcode
237
238This will try to compile the pipeline description to use as source in the QMediaPlayer and will be
239automatically connected to the sinks of the QMediaPlayer.
240
241\warning Hic sunt dracones! Custom pipelines are an experimental feature: the custom pipelines do
242not map well to QMediaPlayer APIs, most notably the media status, metadata APIs, and transport state.
243Most calls will directly map to the GStreamer pipeline, which can lead to undefined behavior
244depending on the pipeline. In most cases, the \c{gstreamer-pipeline:} may not be the right choice
245for application code: for arbitrary video sources, the \c{QMediaCaptureSession} with a custom camera
246(see above) is the preferred choice. For arbitrarily complex pipelines that only want to draw into a
247Qt/QML GUI, GStreamer's \c{qml6glsink} (see below) may be a more robust choice.
248
249*/