Multimedia with Gstreamer on Brainy Pi

Brainy Pi is an open-source hardware platform that enables developers to build intelligent devices for a wide range of applications. One of the key components of any smart device is the ability to capture, process, and transmit multimedia data. In this blog, we will explore Gstreamer on Brainy Pi

BrainyPi has built-in

1. ISP (Image Signal Processor) dedicated for Capturing Video and images.

2. VPU (Video Processing Unit) dedicated for Video encoding and decoding.

This allows us to do hardware accelerated Video Capture, Encoding and Decoding.

In this tutorial, we will explore how to use GStreamer, a powerful multimedia framework, to build an application on BrainyPi that can capture and display video, encode and decode video streams, and stream video over a network using the VPU and ISP.

Prerequisites:

Before we get started, you will need a Brainy Pi board with a camera module (Raspi cam v1.3) or USB Webcam, HDMI monitor, and a network connection. You will also need to have GStreamer installed on your BrainyPi board. If you have not installed GStreamer, you can do so by running the following command in a terminal window:

sudo apt-get install gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly

Capture and Display:

Capture using CSI camera and Display

gst-launch-1.0 rkisp device=/dev/video0 path-iqf=/etc/iqfiles/ov5647_default_default.xml ! videoconvert ! kmssink

This pipeline captures video from the CSI camera using the rkisp element, converts the video format using the videoconvert element, and display using kmssink.

Capture using USB Webcam and Display

gst-launch-1.0 v4l2src device=/dev/video5 ! videoconvert ! kmssink

This pipeline captures video from the web camera using the v4l2src element, converts the video format using the videoconvert element, and display using kmssink.

Encoding:

The next step is to encode video streams. We can use GStreamer to encode video in a variety of formats, all supported formats are given below,

✔️ says support and ❌ says not supported.

1. H.264

1. Encoding Videotestsrc

gst-launch-1.0 videotestsrc is-live=true ! video/x-raw, format=NV12, width=1920, height=1080 ! mpph264enc ! h264parse ! qtmux ! filesink location=./filename.mp4

2. Encoding CSI camera

gst-launch-1.0 rkisp device=/dev/video0 path-iqf=/etc/iqfiles/ov5647_default_default.xml ! videoconvert ! video/x-raw, format=NV12 ! mpph264enc ! h264parse ! qtmux ! filesink location=./filename.mp4

3. Encoding USB Web camera

gst-launch-1.0 v4l2src device=/dev/video5 ! videoconvert ! video/x-raw, format=NV12 ! mpph264enc ! h264parse ! qtmux ! filesink location=./filename.mp4

2. VP8

1. Encoding Videotestsrc

gst-launch-1.0 videotestsrc is-live=true ! video/x-raw, format=NV12, width=1920, height=1080 ! mppvp8enc ! matroskamux ! filesink location=./filename.mkv

2. Encoding CSI camera

gst-launch-1.0 rkisp device=/dev/video0 path-iqf=/etc/iqfiles/ov5647_default_default.xml ! videoconvert ! video/x-raw, format=NV12 ! mppvp8enc ! matroskamux ! filesink location=./filename.mkv

3. Encoding USB Web camera

gst-launch-1.0 v4l2src device=/dev/video5 ! videoconvert ! video/x-raw, format=NV12 ! mppvp8enc ! matroskamux ! filesink location=./filename.mkv

3. JPEG

1. Encoding Videotestsrc

gst-launch-1.0 videotestsrc num-buffers=1 ! video/x-raw, format=NV12, width=1920, height=1080 ! mppjpegenc ! filesink location=./test.jpg

2. Encoding CSI camera

gst-launch-1.0 rkisp device=/dev/video0 path-iqf=/etc/iqfiles/ov5647_default_default.xml num-buffers=1 ! videoconvert ! video/x-raw, format=NV12 ! mppjpegenc ! filesink location=./test.jpg

3. Encoding USB Web camera

gst-launch-1.0 v4l2src device=/dev/video5 num-buffers=1 ! videoconvert ! video/x-raw, format=NV12 ! mppjpegenc ! filesink location=./test.jpg

Decoding:

The next step is to decode video files. We can use GStreamer to decode video in a variety of formats, all supported formats are given below.

✔️ says support and ❌ says not supported.

Following formats are not tested, – MPEG-1, MPEG-2, MPEG-4, H.263, VC-1, VP8, MVC.

Decode MP4 file

gst-launch-1.0 filesrc location=./filename.mp4 ! qtdemux ! queue ! mppvideodec ! kmssink

This command will decode an .mp4 file encoded in any of the above supported formats

Decode MKV file

gst-launch-1.0 filesrc location=./filename.mkv ! matroskademux ! queue ! mppvideodec ! kmssink

This command will decode an .mkv file encoded in any of the above supported formats

Streaming:

RTP Streaming

1. H.264

1. Encoding Videotestsrc

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 videotestsrc is-live=true ! video/x-raw, format=NV12, width=1920, height=1080 ! mpph264enc ! h264parse ! video/x-h264, stream-format=byte-stream ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

2. Encoding CSI camera

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 rkisp device=/dev/video0 path-iqf=/etc/iqfiles/ov5647_default_default.xml ! videoconvert ! video/x-raw, format=NV12 ! mpph264enc ! h264parse ! video/x-h264, stream-format=byte-stream ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

3. Encoding USB Web camera

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 v4l2src device=/dev/video5 ! videoconvert ! video/x-raw, format=NV12 ! mpph264enc ! h264parse ! video/x-h264, stream-format=byte-stream ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

2. VP8

1. Encoding Videotestsrc

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 videotestsrc is-live=true ! video/x-raw, format=NV12, width=1920, height=1080 ! mppvp8enc ! rtpvp8pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

2. Encoding CSI camera

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 rkisp device=/dev/video0 path-iqf=/etc/iqfiles/ov5647_default_default.xml ! videoconvert ! video/x-raw, format=NV12 ! mppvp8enc ! rtpvp8pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

3. Encoding USB Web camera

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 v4l2src device=/dev/video5 ! videoconvert ! video/x-raw, format=NV12 ! mppvp8enc ! rtpvp8pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

Conclusion

So we have implemented Gstreamer on Brainy Pi by using multiple methods. Stay tuned for more such blogs.