Joint - video streaming framework

Language: Swift 5.6
Compiler: 15.4
Dependencies: MQTTClient, SocketRocket

Sample App: https://github.com/aibo-cora/meliora
MQTTClient v0.15.3 https://github.com/novastone-media/MQTT-Client-Framework
SocketRocket v0.6.0 https://github.com/facebookincubator/SocketRocket
References: https://mqtt.org/

Description

Live streaming is a 1 to many setup where 1 client streams video data to the many watching it. The streamer becomes publisher and the many are subscribers. The MQTT broker that you need to configure is your streaming server that delivers the data to the subscribers.

Steps to configure your app for live streaming

1. Broker
   
   Create a broker cluster with HiveMQ (or any other solution) @ https://console.hivemq.cloud/, there are different tiers for your needs. I'd go with the free tier for development purposes. Declare a Broker instance in your app and use the information provided in your dashboard for arguments.

   Dashboard Example
   
   URL 2457aa93d1444b8cb0dd5d5891a3c3d6.s1.eu.hivemq.cloud
   PORT(TLS) 8883
   
   Client app credentials to authenticate with the server
   Manage Cluster->Access Management->Add User & Password
   
   USERNAME test.admin
   PASSWORD P2ssword
   
   

     let broker = Broker(ip: "2457aa93d1444b8cb0dd5d5891a3c3d6.s1.eu.hivemq.cloud",
                       port: 8883,
                   username: "test.admin",
                   password: "P2ssword")
       

2. Capture Components
   
   

       lazy var components = CaptureComponents(captureSession: CameraManager.shared.session,
                                                     delegate: FrameSupplier.shared)
     

   captureSession Configure an AVCaptureSession instance by adding video input. Joint takes care of the audio input and AV output, so you do not need to do anything here. Do not install taps on the microphone in your app, it might result in unexpected behavior. Adding video output is also not needed. The framework will remove it and add another one that is configured to the required specs.
   
   delegate Conform an object supplying CVPixelBuffer for your video preview to VideoDisplayDelegate. This object will be receiving sample buffers from the AV capture session you configured. If you are using SwiftUI, make sure you setup a chain of modifiers to convert CMSampleBuffer to Image. Refer to the sample app for a way to do it.

   The framework does not configure the AVAudioSession singleton and relies on the OS to make it active and grant the necessary category at runtime. The side effects are unknown at this point if you decide to configure it.

3. Joint Session

  jointSession = JointSession(apiKey: "C!9X5&/WPuU(6pp5",
                              broker: broker,
                   captureComponents: components,
                            delegate: nil,
                         loggingFlag: false)
  


apiKey shown provides basic (free) scopes for live streaming and video conference (future release) applications.
delegate is reserved for future use. Use nil for now.
loggingFlag Set to true to see detailed logs.

Publishers

jointSession?.$sessionStatus Use this to give feedback to users about the status of the session.
jointSession?.$activeStreamers This is a Set of Streamer. Use this to display active streamers on the network.
4. Connect to broker

  await jointSession?.configureClient()
  

configureClient() will verify the API key first, then attempt connecting to the broker that you have configured.

Streamer Is a unique entity on your network. An object of this type should have a name, email and channel associated with it.

Live streaming

jointSession?.startSession(streamer: Streamer()) Start Live session.
jointSession?.stopSession() Stop Live session.

Watching stream

jointSession?.configurePlayer(watching: streamer, using: player)

watching Active streamer on the network.
using An instance of AVQueuePlayer in your view.

Future releases

1.1.0 Video chat, 1 on 1.
1.2.0 Video conference.
1.3.0 Live news type stream where the host can establish a link with 4? other participants and select which one gets the spotlight.
2.0.0 On demand video streaming.

Known Issues

- If a stream does not explicitly end, it does not get removed from the list on the network. It is just a ghost. Resolution: need to run a task on to track whether data is still coming in