Qt
Internal/Contributor docs for the Qt SDK. Note: These are NOT official API docs; those are found at https://doc.qt.io/
Loading...
Searching...
No Matches
backend-notes-gstreamer.qdoc
Go to the documentation of this file.
1// Copyright (C) 2024 The Qt Company Ltd.
2// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
3
4/*!
5\page qtmultimedia-gstreamer.html
6\title Qt Multimedia GStreamer backend
7\brief Platform notes for the GStreamer backend
8
9This page covers limitations and customization points of the GStreamer backend
10of Qt Multimedia, the default media backend on embedded Linux.
11
12\section1 Architectural Considerations
13
14GStreamer is the default media backend on embedded Linux mainly because it is
15the framework most embedded board vendors prioritize for hardware accelerated
16media support. FFmpeg may also be suitable on some platforms, so users are
17encouraged to experiment with both media backends to find out which works
18best for them.
19
20Qt Multimedia is not a general purpose streaming framework and not necessarily
21the architecturally best way to use GStreamer with Qt. Developers, who need a
22high degree of control over the GStreamer pipeline, but only want to show the
23video output Qt, may want to consider using GStreamer's
24\l{https://gstreamer.freedesktop.org/documentation/qml6/index.html}{qml6glsink}.
25
26
27
28
29\section1 Limitations and Known Issues
30
31GStreamer is not bundled with Qt, but it is typically deployed with the Linux distribution.
32
33\list
34\li Certain bugs may be due to the GStreamer version being used. We recommend to use the latest
35 GStreamer bug fix release on your platform.
36\li Certain bugs may also be related to the libraries used by GStreamer, like Pulseaudio. Most
37 notably Pulseaudio v16 has a known bug that causes the GStreamer pipeline to hang and requires
38 backports of these two patches:
39 \list
40 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/745}
41 \li \l{https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/764}
42 \endlist
43 This bug currently affects most mainstream Linux distributions, including Ubuntu 22.04, 23.10 and
44 24.04, Debian 11 and 12, as well as Fedora 39 and 40.
45\li Seeking, playback rates, loop, switching sinks have known bugs.
46\li Audio features requires PulseAudio. See \l{Qt Multimedia on Linux}{Linux platform notes}
47 for details.
48\li Some embedded boards needs individual configurations to solve pipeline
49negotiation/linking issues and improve performance. We recommend developers to
50experiment with the environment variables listed under \l{Customization points}
51when facing such problems. Qt's Yocto meta layer files contain recommended
52configurations for certain combinations of boards, BSPs and Boot to Qt
53versions. Here are links to the dev branch versions:
54 \list
55 \li NXP i.MX 6/8/9: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/freescale-layer/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
56 \li Raspberry Pi 4/5: \l{https://code.qt.io/cgit/yocto/meta-boot2qt.git/tree/meta-boot2qt-distro/dynamic-layers/raspberrypi/recipes-qt/boot2qt-addons/default-qt-envs.bbappend?h=dev}
57 \endlist
58\li Video playback or camera streaming using system memory video frames or
59software conversion can result in dropped frames and unresponsiveness due to
60heavy CPU load. If the GStreamer decoder or camera device outputs DMA-BUF- or
61GLMemory-backed video frames in one of our supported pixel formats, Qt can
62render them without additional conversion or copying. GStreamer selects a
63format and memory type based on the capabilities of its pipeline elements.
64Different ways of controlling which elements are added to the pipeline are
65described under \l{Customization points}.
66\endlist
67
68
69
70
71\section1 Customization points
72
73Qt Multimedia provides certain customization points to allow access to the
74underlying GStreamer pipeline. The entry point is \c{class
75QGStreamerPlatformSpecificInterface}. Additional customizations can be done by
76setting environment variables.
77
78\warning The customization points and Qt specific environment variables listed
79here, are considered as private APIs and may be subject to change. Be aware
80that environment variables are by default inherited by child processes and can
81have unintended consequences.
82
83
84\section2 External OpenGL texture rendering
85
86Qt imports DMA-BUF-backed video frames into the OpenGL graphics pipeline by
87binding the underlying buffer as an EGL image and exposing it to shaders as an
88OpenGL texture. By default, Qt samples these textures using the GL_TEXTURE_2D
89texture target and standard shader code. Some embedded boards performs better
90with the GL_TEXTURE_EXTERNAL_OES texture target instead, together with a
91dedicated fragment shader. This behavior can be enabled by setting the
92following environment variable:
93
94\code
95 QT_MULTIMEDIA_FORCE_GL_TEXTURE_EXTERNAL_OES=1
96\endcode
97
98
99\section2 Selecting specific GStreamer elements
100
101A GStreamer distribution can include multiple different plugins that can
102potentially perform the same task in the media pipeline. GStreamer
103automatically selects elements like decoders and demuxers based on their
104feature rank, which can be overridden via a GStreamer environment variable.
105Here's an example that ranks the v4l2h264dec hardware decoder higher than any
106H.264 software decoder, while also making sure the aiurdemux demuxer isn't
107selected:
108
109\code
110 GST_PLUGIN_FEATURE_RANK=v4l2h264dec:MAX,aiurdemux:NONE
111\endcode
112
113To inspect which elements are being created during runtime, enable debug logging
114level 4 for GstElementFactory:
115
116\code
117 GST_DEBUG=GST_ELEMENT_FACTORY:4
118\endcode
119
120Furthermore, you can visually inspect the whole GStreamer pipeline by
121specifying a folder for \c {.dot} graph files that will be dumped at
122various events, for example, changes in playback state:
123
124\code
125 GST_DEBUG_DUMP_DOT_DIR=/root/graphs
126\endcode
127
128Read more about these environment variables in the GStreamer documentation:
129
130\list
131 \li \l{https://gstreamer.freedesktop.org/documentation/gstreamer/running.html?gi-language=c#:~:text=GST_PLUGIN_FEATURE_RANK}{Environment variables - GST_PLUGIN_FEATURE_RANK}
132 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#the-debug-log}{Basic tutorial - The debug log}
133 \li \l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c#getting-pipeline-graphs}{Basic tutorial - Getting pipeline graphs}
134\endlist
135
136\section3 Specifying hardware conversion elements
137
138If the media source of a GStreamer pipeline cannot provide video buffers in a
139pixel format supported by Qt Multimedia, the pipeline will use software
140conversion unless it contains a suitable hardware accelerated video conversion
141element. To specify a hardware conversion element to be used, set it as a
142\l{https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html#elements}
143{GStreamer pipeline description} to the following environment variable:
144
145\code
146 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT
147\endcode
148
149Some vendor specific conversion elements (imxvideoconvert_g2d, nvvidconv) are
150added to the pipeline by default if available. This can cause unnecessary
151conversion if they aren't needed, and can be disabled by specifying a GStreamer
152\l{https://gstreamer.freedesktop.org/documentation/coreelements/identity.html}
153{identity} element using the same environment variable:
154
155\code
156 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=identity
157\endcode
158
159GStreamer can also perform format conversion with OpenGL elements, providing Qt
160with GLMemory-backed video frames:
161
162\code
163 QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=glupload ! glcolorconvert
164\endcode
165
166
167\section2 Raw pipeline access
168
169The \c{GstPipeline} underlying the \l{QMediaPlayer} and \l{QMediaCaptureSession} can be accessed.
170
171\warning This is an unsafe API, as the pipeline is still managed by the Qt implementation. Great
172care is required when using this API.
173
174\code
175#include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>
176
177[...]
178QMediaMediaPlayer player;
179GstPipeline *pipeline = QGStreamerPlatformSpecificInterface::instance()->gstPipeline(&player);
180[...]
181QMediaCaptureSession session;
182GstPipeline *pipeline = QGSreamerPlatformSpecificInterface::instance()->gstPipeline(&session);
183\endcode
184
185
186\section2 Custom GStreamer elements as sinks and sources
187
188It is possible to create GStreamer elements from a GStreamer pipeline decription and wrap them
189inside a \c{QCamera} or \c{QAudioDevice}:
190
191\code
192#include <QtMultimedia/private/qgstreamer_platformspecificinterface_p.h>
193
194[...]
195QByteArray pipelineString = "videotestsrc is-live=true ! gamma gamma=2.0";
196
197QMediaCaptureSession session;
198session.setVideoSink(wid.videoSink());
199
200QCamera *cam = QGStreamerPlatformSpecificInterface::instance()->makeCustomGStreamerCamera(
201 pipelineString, &session);
202session.setCamera(cam);
203\endcode
204
205
206\section2 QMediaPlayer: custom sources
207
208The \c{QMediaPlayer} accepts a GStreamer pipeline decription as source URI:
209
210\code
211QMediaPlayer player;
212player.setSource(u"gstreamer-pipeline: videotestsrc name=testsrc"_s);
213\endcode
214
215This will try to compile the pipeline description to use as source in the QMediaPlayer and will be
216automatically connected to the sinks of the QMediaPlayer.
217
218\warning Hic sunt dracones! Custom pipelines are an experimental feature: the custom pipelines do
219not map well to QMediaPlayer APIs, most notably the media status, metadata APIs, and transport state.
220Most calls will directly map to the GStreamer pipeline, which can lead to undefined behavior
221depending on the pipeline. In most cases, the \c{gstreamer-pipeline:} may not be the right choice
222for application code: for arbitrary video sources, the \c{QMediaCaptureSession} with a custom camera
223(see above) is the preferred choice. For arbitrarily complex pipelines that only want to draw into a
224Qt/QML GUI, GStreamer's \c{qml6glsink} (see below) may be a more robust choice.
225
226*/