Qt
Internal/Contributor docs for the Qt SDK. Note: These are NOT official API docs; those are found at https://doc.qt.io/
Loading...
Searching...
No Matches
backend-notes-gstreamer.qdoc
Go to the documentation of this file.
1// Copyright (C) 2024 The Qt Company Ltd.
2// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
3
4/*!
5\page qtmultimedia-gstreamer.html
6\title Qt Multimedia GStreamer backend
7\brief Platform notes for the GStreamer backend
8
9This page covers limitations and customization points of the GStreamer backend
10of Qt Multimedia, the default media backend on embedded Linux.
11
12\section1 Architectural Considerations
13
14GStreamer is the default media backend on embedded Linux mainly because it is
15the framework most embedded board vendors prioritize for hardware accelerated
16media support. FFmpeg may also be suitable on some platforms, so users are
17encouraged to experiment with both media backends to find out which works
18best for them.
19
20Qt Multimedia is not a general purpose streaming framework and not necessarily
21the architecturally best way to use GStreamer with Qt. Developers, who need a
22high degree of control over the GStreamer pipeline, but only want to show the
23video output Qt, may want to consider using GStreamer's
24\l{https://gstreamer.freedesktop.org/documentation/qml6/index.html}{qml6glsink}.
25
26
27
28
29\section1 Limitations and Known Issues
30
31GStreamer is not bundled with Qt, but it is typically deployed with the Linux distribution.
32
33\list
34\li Certain bugs may be due to the GStreamer version being used. We recommend to use the latest
35 GStreamer bug fix release on your platform.
36\li Certain bugs may also be related to the libraries used by GStreamer, like Pulseaudio. Most
37 notably Pulseaudio v16 has a known bug that causes the GStreamer pipeline to hang and requires
38 backports of these two patches:
39 \list
40 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/745}
41 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/764}
42 \endlist
43 This bug currently affects most mainstream Linux distributions, including Ubuntu 22.04, 23.10 and
44 24.04, Debian 11 and 12, as well as Fedora 39 and 40.
45\li Switching camera devices or cameras during recording is not supported
46\li Seeking, playback rates, loop, switching sinks have known bugs.
47\li Audio features requires PulseAudio. See \l{Qt Multimedia on Linux}{Linux platform notes}
48 for details.
49\li Some embedded boards needs individual configurations to solve pipeline
50negotiation/linking issues and improve performance. We recommend developers to
51experiment with the environment variables listed under \l{Customization points}
52when facing such problems. Qt's Yocto meta layer files contain recommended
53configurations for certain combinations of boards, BSPs and Boot to Qt
54versions. Here are links to the dev branch versions:
55 \list
56 \li NXP i.MX 6/8/9: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/freescale-layer/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
57 \li Raspberry Pi 4/5: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/raspberrypi/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
58 \endlist
59\li Some NXP i.MX boards expose a v4l2 device named imx-capture, which can cause
60errors when used with v4l2src in GStreamer. To prevent Qt Multimedia from
61adding imx-capture to the list of available camera devices, set the environment
62variable QT_GSTREAMER_SKIP_IMXCAPTURE=1. Note that this is considered as
63private API which will be removed when NXP fixes the problems with imx-capture.
64\li Video playback or camera streaming using system memory video frames or
65software conversion can result in dropped frames and unresponsiveness due to
66heavy CPU load. If the GStreamer decoder or camera device outputs DMA-BUF- or
67GLMemory-backed video frames in one of our supported pixel formats, Qt can
68render them without additional conversion or copying. GStreamer selects a
69format and memory type based on the capabilities of its pipeline elements.
70Different ways of controlling which elements are added to the pipeline are
71described under \l{Customization points}. See also
72\l{Specifying I/O mode for v4l2 cameras}.
73\endlist
74
75
76
77
78\section1 Customization points
79
80Qt Multimedia provides certain customization points to allow access to the
81underlying GStreamer pipeline. The entry point is \l QGStreamerInterface.
82Additional customizations can be done by setting environment variables.
83
84\warning The customization points and Qt specific environment variables listed
85here, are considered as private APIs and may be subject to change. Be aware
86that environment variables are by default inherited by child processes and can
87have unintended consequences.
88
89
90\section2 External OpenGL texture rendering
91
92Qt imports DMA-BUF-backed video frames into the OpenGL graphics pipeline by
93binding the underlying buffer as an EGL image and exposing it to shaders as an
94OpenGL texture. By default, Qt samples these textures using the GL_TEXTURE_2D
95texture target and standard shader code. Some embedded boards performs better
96with the GL_TEXTURE_EXTERNAL_OES texture target instead, together with a
97dedicated fragment shader. This behavior can be enabled by setting the
98following environment variable:
99
100\code
101 QT_MULTIMEDIA_FORCE_GL_TEXTURE_EXTERNAL_OES=1
102\endcode
103
104\section2 Specifying I/O mode for v4l2 cameras
105
106When using a v4l2 capture device with a QCamera or QML Camera, the
107\l{https://gstreamer.freedesktop.org/documentation/video4linux2/v4l2src.html#v4l2src:io-mode}{io-mode}
108property of the underlying v4l2src GStreamer element affects whether or not it
109outputs DMA-BUF-backed video frames that allows zero-copy rendering. In some
110cases the default auto mode chooses mmap mode even though the camera driver
111supports dmabuf mode. The io-mode property int value can be set explicitly via
112the environment variable below. Here's an example showing how to set io-mode to
113dmabuf with the enum GST_V4L2_IO_DMABUF:
114
115\code
116 QT_GSTREAMER_V4L2SRC_IOMODE=4
117\endcode
118
119
120\section2 Selecting specific GStreamer elements
121
122A GStreamer distribution can include multiple different plugins that can
123potentially perform the same task in the media pipeline. GStreamer
124automatically selects elements like decoders and demuxers based on their
125feature rank, which can be overridden via a GStreamer environment variable.
126Here's an example that ranks the v4l2h264dec hardware decoder higher than any
127H.264 software decoder, while also making sure the aiurdemux demuxer isn't
128selected:
129
130\code
131 GST_PLUGIN_FEATURE_RANK=v4l2h264dec:MAX,aiurdemux:NONE
132\endcode
133
134To inspect which elements are being created during runtime, enable debug logging
135level 4 for GstElementFactory:
136
137\code
138 GST_DEBUG=GST_ELEMENT_FACTORY:4
139\endcode
140
141Furthermore, you can visually inspect the whole GStreamer pipeline by
142specifying a folder for \c {.dot} graph files that will be dumped at
143various events, for example, changes in playback state:
144
145\code
146 GST_DEBUG_DUMP_DOT_DIR=/root/graphs
147\endcode
148
149Read more about these environment variables in the GStreamer documentation:
150
151\list
152 \li \l{https://gstreamer.freedesktop.org/documentation/gstreamer/running.html?gi-language=c#:~:text=GST_PLUGIN_FEATURE_RANK}{Environment variables - GST_PLUGIN_FEATURE_RANK}
153 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#the-debug-log}{Basic tutorial - The debug log}
154 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#getting-pipeline-graphs}{Basic tutorial - Getting pipeline graphs}
155\endlist
156
157\section3 Specifying hardware conversion elements
158
159If the media source of a GStreamer pipeline cannot provide video buffers in a
160pixel format supported by Qt Multimedia, the pipeline will use software
161conversion unless it contains a suitable hardware accelerated video conversion
162element. To specify a hardware conversion element to be used, set it as a
163\l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html#elements}
164{GStreamer pipeline description} to the following environment variable:
165
166\code
167 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT
168\endcode
169
170Some vendor specific conversion elements (imxvideoconvert_g2d, nvvidconv) are
171added to the pipeline by default if available. This can cause unnecessary
172conversion if they aren't needed, and can be disabled by specifying a GStreamer
173\l{https://gstreamer.freedesktop.org/documentation/coreelements/identity.html}
174{identity} element using the same environment variable:
175
176\code
177 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=identity
178\endcode
179
180GStreamer can also perform format conversion with OpenGL elements, providing Qt
181with GLMemory-backed video frames:
182
183\code
184 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=glupload ! glcolorconvert
185\endcode
186
187
188\section2 Raw pipeline access
189
190The \c{GstPipeline} underlying the \l{QMediaPlayer} and \l{QMediaCaptureSession} can be accessed.
191
192\warning This is an unsafe API, as the pipeline is still managed by the Qt implementation. Great
193care is required when using this API.
194
195\code
196#include <QtMultimedia/spi/qgstreamerinterface.h>
197
198[...]
199QMediaPlayer player;
200GstPipeline *pipeline = QGStreamerInterface::instance()->gstPipeline(&player);
201[...]
202QMediaCaptureSession session;
203GstPipeline *pipeline = QGStreamerInterface::instance()->gstPipeline(&session);
204\endcode
205
206
207\section2 Custom GStreamer elements as sinks and sources
208
209It is possible to create GStreamer elements from a GStreamer pipeline description and wrap them
210inside a \l QGStreamerVideoSource or \l QAudioDevice:
211
212\code
213#include <QtMultimedia/spi/qgstreamerinterface.h>
214#include <QtMultimedia/spi/qgstreamervideosource.h>
215
216[...]
217QMediaCaptureSession session;
218session.setVideoSink(wid.videoSink());
219
220QGStreamerVideoSource videoSource(QStringLiteral("videotestsrc is-live=true ! gamma gamma=2.0"));
221session.setNativeVideoSource(&videoSource);
222videoSource.start();
223\endcode
224
225\section2 QMediaPlayer: custom sources
226
227The \c{QMediaPlayer} accepts a GStreamer pipeline description as source URI:
228
229\code
230QMediaPlayer player;
231player.setSource(u"gstreamer-pipeline: videotestsrc name=testsrc"_s);
232\endcode
233
234This will try to compile the pipeline description to use as source in the QMediaPlayer and will be
235automatically connected to the sinks of the QMediaPlayer.
236
237\warning Hic sunt dracones! Custom pipelines are an experimental feature: the custom pipelines do
238not map well to QMediaPlayer APIs, most notably the media status, metadata APIs, and transport state.
239Most calls will directly map to the GStreamer pipeline, which can lead to undefined behavior
240depending on the pipeline. In most cases, the \c{gstreamer-pipeline:} may not be the right choice
241for application code: for arbitrary video sources, the \c{QMediaCaptureSession} with a custom camera
242(see above) is the preferred choice. For arbitrarily complex pipelines that only want to draw into a
243Qt/QML GUI, GStreamer's \c{qml6glsink} (see below) may be a more robust choice.
244
245*/