Skip to content

Video flow

Video flow в VideoGrace разделяет control lifecycle камеры и media lifecycle видеопотока. Камера создается как server-side device через control channel, а encoded video frames передаются через media transport по ssrc.

Local camera publish

sequenceDiagram
    participant UI
    participant Client as VideograceClient
    participant Control as Control channel
    participant Camera as CameraSession
    participant Media as Media transport
    participant Server

    UI->>Client: toggleCam(true)
    Client->>Camera: start capture
    Camera-->>Client: capture ready(width, height, fps)
    Client->>Control: device_params(device_type=cam, resolution, codec)
    Control->>Server: create video device
    Server-->>Control: device_connect(my=1, device_id, author_ssrc, port)
    Control-->>Client: local video device created
    Client->>Camera: attach media(author_ssrc, port)
    Camera->>Media: register author_ssrc
    Camera->>Media: send RTP/video frames

Remote video receive

sequenceDiagram
    participant Control as Control channel
    participant Client as VideograceClient
    participant Video as Remote VideoSession
    participant Media as Media transport
    participant Renderer

    Control-->>Client: device_connect(my=0, device_type=cam, receiver_ssrc, port)
    Client->>Video: create session(device_id, client_id)
    Video->>Renderer: attach canvas/surface
    Video->>Media: register receiver_ssrc
    Video->>Media: send RTP init
    Video->>Media: send ForceKeyFrame RTCP
    Media-->>Video: RTP/video frames by receiver_ssrc
    Video->>Video: decode frames
    Video->>Renderer: render decoded frame

Pipeline

flowchart LR
    subgraph Publish[Local publish]
        Capture[Camera capture]
        Convert[Format convert / rotate]
        Encode[Video encoder]
        Packetize[RTP packetizer]
        Send[Media transport send]
    end

    subgraph Receive[Remote receive]
        Recv[Media transport receive]
        Route[Demux by SSRC]
        Decode[Video decoder]
        Render[Canvas / surface renderer]
    end

    Capture --> Convert --> Encode --> Packetize --> Send
    Recv --> Route --> Decode --> Render

Keyframe flow

sequenceDiagram
    participant Receiver as Remote VideoSession
    participant Media as Media transport
    participant Server
    participant Sender as Remote publisher

    Receiver->>Media: send RTP init(receiver_ssrc)
    Receiver->>Media: send ForceKeyFrame RTCP(receiver_ssrc)
    Media->>Server: WSM/RTCP frame
    Server->>Sender: forward keyframe request
    Sender-->>Server: next keyframe RTP
    Server-->>Receiver: RTP keyframe frames
    Receiver->>Receiver: decoder starts from keyframe

Keyframe нужен при первом подключении remote video, после media reconnect и после decoder errors. Запросы keyframe должны быть throttled, чтобы ошибка декодера не превращалась в RTCP spam.

Reconnect behavior

stateDiagram-v2
    [*] --> Capturing
    Capturing --> Publishing: device_connect my=1
    Publishing --> TransportLost: media close/error
    TransportLost --> Reattach: media reconnect
    Reattach --> Publishing: register SSRC + continue frames
    Publishing --> Stopping: toggleCam(false) / leave
    Stopping --> [*]

Для remote video после reconnect нужно:

  • заново register receiver ssrc в media transport;
  • повторить RTP init;
  • запросить ForceKeyFrame;
  • сохранить UI session, если server-side device не был отключен.

Invariants

  • Камера публикуется через device_params; media transport не создает device.
  • device_id принадлежит control lifecycle.
  • author_ssrc используется локальной CameraSession для отправки.
  • receiver_ssrc используется remote video session для приема.
  • Один remote video device должен иметь одну renderer/session pair.
  • Reconnect media transport не должен создавать новую плитку UI.
  • toggleCam(false) должен остановить capture, encoder, media attach и отправить disconnect_device.
  • Повторный device_connect должен быть идемпотентным.

Diagnostics

Для video-инцидента в логах нужны:

  • device_id, client_id, device_type;
  • author_ssrc или receiver_ssrc;
  • resolution, fps, codec profile;
  • renderer attach;
  • media transport ready;
  • RTP init sent;
  • ForceKeyFrame sent;
  • first RTP packet received;
  • first keyframe received;
  • decoder configured;
  • first decoded frame;
  • first rendered frame;
  • decoder errors and reconnect reason.

Typical failures

  • Черный экран сразу после подключения: проверить RTP init, ForceKeyFrame, first keyframe, decoder errors.
  • Локальная камера работает, но удаленный участник видит черный экран: проверить local device_connect my=1, encoder start, first RTP sent, server routing, remote decoder.
  • После media reconnect видео не возвращается: проверить re-register receiver SSRC, repeated RTP init, ForceKeyFrame after reconnect.
  • Выключение/включение камеры плодит черные плитки: проверить идемпотентность remote device_connect, очистку old renderer/session и связку device_id + client_id + device_type.
  • Decode errors идут серией: проверить codec config, packet boundaries, keyframe availability и throttling RTCP keyframe requests.