Qt
Internal/Contributor docs for the Qt SDK. Note: These are NOT official API docs; those are found at https://doc.qt.io/
Loading...
Searching...
No Matches
backend-notes-gstreamer.qdoc
Go to the documentation of this file.
1// Copyright (C) 2024 The Qt Company Ltd.
2// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
3
4/*!
5\page qtmultimedia-gstreamer.html
6\title Qt Multimedia GStreamer backend
7\brief Platform notes for the GStreamer backend
8
9This page covers limitations and customization points of the GStreamer backend
10of Qt Multimedia, the default media backend on embedded Linux.
11
12\section1 Architectural Considerations
13
14GStreamer is the default media backend on embedded Linux mainly because it is
15the framework most embedded board vendors prioritize for hardware accelerated
16media support. FFmpeg may also be suitable on some platforms, so users are
17encouraged to experiment with both media backends to find out which works
18best for them.
19
20Qt Multimedia is not a general purpose streaming framework and not necessarily
21the architecturally best way to use GStreamer with Qt. Developers, who need a
22high degree of control over the GStreamer pipeline, but only want to show the
23video output Qt, may want to consider using GStreamer's
24\l{https://gstreamer.freedesktop.org/documentation/qml6/index.html}{qml6glsink}.
25
26
27
28
29\section1 Limitations and Known Issues
30
31GStreamer is not bundled with Qt, but it is typically deployed with the Linux distribution.
32
33\list
34\li Certain bugs may be due to the GStreamer version being used. We recommend to use the latest
35 GStreamer bug fix release on your platform.
36\li Certain bugs may also be related to the libraries used by GStreamer, like Pulseaudio. Most
37 notably Pulseaudio v16 has a known bug that causes the GStreamer pipeline to hang and requires
38 backports of these two patches:
39 \list
40 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/745}
41 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/764}
42 \endlist
43 This bug currently affects most mainstream Linux distributions, including Ubuntu 22.04, 23.10 and
44 24.04, Debian 11 and 12, as well as Fedora 39 and 40.
45\li Switching camera devices or cameras during recording is not supported
46\li Seeking, playback rates, loop, switching sinks have known bugs.
47\li Audio features requires PulseAudio. See \l{Qt Multimedia on Linux}{Linux platform notes}
48 for details.
49\li Some embedded boards needs individual configurations to solve pipeline
50negotiation/linking issues and improve performance. We recommend developers to
51experiment with the environment variables listed under \l{Customization points}
52when facing such problems. Qt's Yocto meta layer files contain recommended
53configurations for certain combinations of boards, BSPs and Boot to Qt
54versions. Here are links to the dev branch versions:
55 \list
56 \li NXP i.MX 6/8/9: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/freescale-layer/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
57 \li Raspberry Pi 4/5: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/raspberrypi/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
58 \endlist
59\l Some NXP i.MX boards expose a v4l2 device named imx-capture, which can cause
60errors when used with v4l2src in GStreamer. To prevent Qt Multimedia from
61adding imx-capture to the list of available camera devices, set the environment
62variable QT_GSTREAMER_SKIP_IMXCAPTURE=1. Note that this is considered as
63private API which will be removed when NXP fixes the problems with imx-capture.
64\li Video playback or camera streaming using system memory video frames or
65software conversion can result in dropped frames and unresponsiveness due to
66heavy CPU load. If the GStreamer decoder or camera device outputs DMA-BUF- or
67GLMemory-backed video frames in one of our supported pixel formats, Qt can
68render them without additional conversion or copying. GStreamer selects a
69format and memory type based on the capabilities of its pipeline elements.
70Different ways of controlling which elements are added to the pipeline are
71described under \l{Customization points}. See also
72\l{Specifying I/O mode for v4l2 cameras}.
73\endlist
74
75
76
77
78\section1 Customization points
79
80Qt Multimedia provides certain customization points to allow access to the
81underlying GStreamer pipeline. The entry point is \c{class
82QGStreamerPlatformSpecificInterface}. Additional customizations can be done by
83setting environment variables.
84
85\warning The customization points and Qt specific environment variables listed
86here, are considered as private APIs and may be subject to change. Be aware
87that environment variables are by default inherited by child processes and can
88have unintended consequences.
89
90
91\section2 External OpenGL texture rendering
92
93Qt imports DMA-BUF-backed video frames into the OpenGL graphics pipeline by
94binding the underlying buffer as an EGL image and exposing it to shaders as an
95OpenGL texture. By default, Qt samples these textures using the GL_TEXTURE_2D
96texture target and standard shader code. Some embedded boards performs better
97with the GL_TEXTURE_EXTERNAL_OES texture target instead, together with a
98dedicated fragment shader. This behavior can be enabled by setting the
99following environment variable:
100
101\code
102 QT_MULTIMEDIA_FORCE_GL_TEXTURE_EXTERNAL_OES=1
103\endcode
104
105\section2 Specifying I/O mode for v4l2 cameras
106
107When using a v4l2 capture device with a QCamera or QML Camera, the
108\l{https://gstreamer.freedesktop.org/documentation/video4linux2/v4l2src.html#v4l2src:io-mode}{io-mode}
109property of the underlying v4l2src GStreamer element affects whether or not it
110outputs DMA-BUF-backed video frames that allows zero-copy rendering. In some
111cases the default auto mode chooses mmap mode even though the camera driver
112supports dmabuf mode. The io-mode property int value can be set explicitly via
113the environment variable below. Here's an example showing how to set io-mode to
114dmabuf with the enum GST_V4L2_IO_DMABUF:
115
116\code
117 QT_GSTREAMER_V4L2SRC_IOMODE=4
118\endcode
119
120
121\section2 Selecting specific GStreamer elements
122
123A GStreamer distribution can include multiple different plugins that can
124potentially perform the same task in the media pipeline. GStreamer
125automatically selects elements like decoders and demuxers based on their
126feature rank, which can be overridden via a GStreamer environment variable.
127Here's an example that ranks the v4l2h264dec hardware decoder higher than any
128H.264 software decoder, while also making sure the aiurdemux demuxer isn't
129selected:
130
131\code
132 GST_PLUGIN_FEATURE_RANK=v4l2h264dec:MAX,aiurdemux:NONE
133\endcode
134
135To inspect which elements are being created during runtime, enable debug logging
136level 4 for GstElementFactory:
137
138\code
139 GST_DEBUG=GST_ELEMENT_FACTORY:4
140\endcode
141
142Furthermore, you can visually inspect the whole GStreamer pipeline by
143specifying a folder for \c {.dot} graph files that will be dumped at
144various events, for example, changes in playback state:
145
146\code
147 GST_DEBUG_DUMP_DOT_DIR=/root/graphs
148\endcode
149
150Read more about these environment variables in the GStreamer documentation:
151
152\list
153 \li \l{https://gstreamer.freedesktop.org/documentation/gstreamer/running.html?gi-language=c#:~:text=GST_PLUGIN_FEATURE_RANK}{Environment variables - GST_PLUGIN_FEATURE_RANK}
154 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#the-debug-log}{Basic tutorial - The debug log}
155 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#getting-pipeline-graphs}{Basic tutorial - Getting pipeline graphs}
156\endlist
157
158\section3 Specifying hardware conversion elements
159
160If the media source of a GStreamer pipeline cannot provide video buffers in a
161pixel format supported by Qt Multimedia, the pipeline will use software
162conversion unless it contains a suitable hardware accelerated video conversion
163element. To specify a hardware conversion element to be used, set it as a
164\l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html#elements}
165{GStreamer pipeline description} to the following environment variable:
166
167\code
168 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT
169\endcode
170
171Some vendor specific conversion elements (imxvideoconvert_g2d, nvvidconv) are
172added to the pipeline by default if available. This can cause unnecessary
173conversion if they aren't needed, and can be disabled by specifying a GStreamer
174\l{https://gstreamer.freedesktop.org/documentation/coreelements/identity.html}
175{identity} element using the same environment variable:
176
177\code
178 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=identity
179\endcode
180
181GStreamer can also perform format conversion with OpenGL elements, providing Qt
182with GLMemory-backed video frames:
183
184\code
185 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=glupload ! glcolorconvert
186\endcode
187
188
189\section2 Raw pipeline access
190
191The \c{GstPipeline} underlying the \l{QMediaPlayer} and \l{QMediaCaptureSession} can be accessed.
192
193\warning This is an unsafe API, as the pipeline is still managed by the Qt implementation. Great
194care is required when using this API.
195
196\code
197#include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>
198
199[...]
200QMediaPlayer player;
201GstPipeline *pipeline = QGStreamerPlatformSpecificInterface::instance()->gstPipeline(&player);
202[...]
203QMediaCaptureSession session;
204GstPipeline *pipeline = QGStreamerPlatformSpecificInterface::instance()->gstPipeline(&session);
205\endcode
206
207
208\section2 Custom GStreamer elements as sinks and sources
209
210It is possible to create GStreamer elements from a GStreamer pipeline description and wrap them
211inside a \c{QCamera} or \c{QAudioDevice}:
212
213\code
214#include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>
215
216[...]
217QByteArray pipelineString = "videotestsrc is-live=true ! gamma gamma=2.0";
218
219QMediaCaptureSession session;
220session.setVideoSink(wid.videoSink());
221
222QCamera *cam = QGStreamerPlatformSpecificInterface::instance()->makeCustomGStreamerCamera(
223 pipelineString, &session);
224session.setCamera(cam);
225\endcode
226
227
228\section2 QMediaPlayer: custom sources
229
230The \c{QMediaPlayer} accepts a GStreamer pipeline description as source URI:
231
232\code
233QMediaPlayer player;
234player.setSource(u"gstreamer-pipeline: videotestsrc name=testsrc"_s);
235\endcode
236
237This will try to compile the pipeline description to use as source in the QMediaPlayer and will be
238automatically connected to the sinks of the QMediaPlayer.
239
240\warning Hic sunt dracones! Custom pipelines are an experimental feature: the custom pipelines do
241not map well to QMediaPlayer APIs, most notably the media status, metadata APIs, and transport state.
242Most calls will directly map to the GStreamer pipeline, which can lead to undefined behavior
243depending on the pipeline. In most cases, the \c{gstreamer-pipeline:} may not be the right choice
244for application code: for arbitrary video sources, the \c{QMediaCaptureSession} with a custom camera
245(see above) is the preferred choice. For arbitrarily complex pipelines that only want to draw into a
246Qt/QML GUI, GStreamer's \c{qml6glsink} (see below) may be a more robust choice.
247
248*/