When using WebRTC for multi-server conversations, the native WebRTC statistics only provide latency data for the first hop (i.e., from the client to the first server). This statistic does not comprehensively reflect the actual latency situation. To obtain more accurate end-to-end latency measurements, the following aspects need to be considered:
- Latency from the first client to the first server.
- Latency between each server.
- Latency from the last server to the target client.
This allows for a more comprehensive understanding of the latency across the entire communication link.
Measuring end-to-end latency is much more difficult because it is not automatically provided by the WebRTC stack.
To measure the total latency (including the latency introduced by capture and rendering endpoints), some additional metrics provided by the WebRTC getStats API can be included in the calculations, such as:
- jitterBufferDelay: The delay of the jitter buffer.
- playoutDelay: The playback delay.
- packetSendDelay: The packet sending delay.
These metrics can help to more comprehensively assess the entire latency chain from capture to final rendering.
In some scenarios, the front-end media processing can also introduce additional latency, such as:
- Beauty filters: Beautifying the image after capturing video, adding effects, which requires additional computation and increases processing time, commonly used in live streaming scenarios.
- Virtual backgrounds: Blurring the background of the captured image, commonly used in video conferencing scenarios.
- Merging screens: Stitching multiple captured images together into a single image can also cause some delay.
- AI and CV processing: Computer vision algorithms like facial recognition, posture recognition, and object recognition can significantly increase latency.
- Audio-related: Echo cancellation, noise suppression, and other processing can also introduce some audio latency.
In addition to the above latencies, it may also be necessary to consider that the software executing capture and rendering can also introduce latency.
In most cases, the round-trip time (RTT) is measured and then divided by two as a good estimate of one-way latency. This method is commonly used in practice to estimate the latency from one point to another.
Capture Delay:
The time from when data is captured from an input device (such as a camera or microphone) until it is transmitted to the application.
Encoding Delay:
The time taken to encode data into a compressed format. This includes the encoding process of video or audio streams.
Transmission Delay:
The time taken for data to be transmitted from the sender to the receiver. This includes network transmission time and processing time at intermediate servers.
Jitter Buffer Delay:
At the receiving end, the jitter buffer is used to smooth out latency variations caused by network jitter, and the time it introduces.
Playout Delay:
The time from when the data is received at the receiving end to when playback starts. This includes delays during the receiving and decoding processes.
Decoding Delay:
The time taken to decode data from the encoded format to a playable format.
Rendering Delay:
The time from decoding to when the data is actually rendered or displayed. This includes the time for screen refresh and image rendering.
Round-Trip Time (RTT):
The total time from when data is sent from the sender to when it is received at the receiver, and then returned from the receiver to the sender. It is typically used to estimate half of the one-way latency.
In response to these factors, a series of related optimization measures are proposed as follows:
1. Distributed Service Deployment#
Goal: Optimize global server layout, reduce network latency, and improve service availability and performance
- Proximity access and distribution: Deploy regional servers globally to reduce latency caused by cross-country connections.
- Reduce routing hops between services.
- Choose the best server path.
- Implement load balancing: Reasonably allocate user requests to avoid excessive pressure on a single server.
2. Network Layer Optimization#
Goal: Improve network transmission quality and efficiency
- Dynamically monitor link quality.
- Identify and avoid low-quality links.
- Optimize access services.
3. Application Layer Optimization#
Goal: Improve application processing efficiency and reduce latency
3.1 Performance Optimization#
- Reduce data copying.
- Minimize unnecessary computations and processing time.
- Use assembly or multimedia assembly instructions to enhance data serial and concurrent processing capabilities.
3.2 Jitter Buffer Optimization#
Goal: Balance network jitter and playback smoothness
- Dynamically adjust jitter buffer size: Adjust based on current network jitter conditions to reduce latency caused by network fluctuations.
- Minimize buffer time: Shorten jitter buffer time as much as possible while ensuring smooth playback.
4. Media Processing Optimization#
Goal: Improve media processing efficiency and adapt to different network environments
- Hardware acceleration: Use GPU, FPGA, or dedicated hardware accelerators (hardware encoding/decoding) for media encoding and processing to reduce CPU load.
- Adaptive bitrate: Dynamically adjust video quality (resolution/frame rate) and bitrate based on network conditions.
- Intelligent encoding: Use more efficient encoding algorithms (ROI encoding) or encoders, such as AV1, VP9, or H.265, to reduce data volume while ensuring quality.
5. Transmission Optimization#
Goal: Improve data transmission efficiency and security
- SRTP encryption optimization: Use more efficient encryption algorithms or optimize encryption implementations to reduce latency introduced by encryption and decryption.
- Reduce packet size: Appropriately reduce packet size to lower latency in network transmission.
6. P2P Optimization#
Goal: Reduce server load and improve peer-to-peer communication efficiency
- Direct connection paths: Prefer P2P (peer-to-peer) connections to avoid server relays.
- ICE candidate prioritization: Prioritize the lowest latency paths during ICE candidate negotiation.
7. User Experience Optimization#
Goal: Improve user satisfaction and reduce perceived latency
- Latency hiding techniques: Implement various latency hiding techniques, such as interpolation and prediction, to enhance user experience.
- Optimize UI/UX: Design responsive user interfaces to reduce perceived latency.
- Network access: Prefer wired networks, which are generally more stable than Wi-Fi and 4/5G.
8. Network Security Optimization#
Goal: Minimize performance impact while ensuring security
- Optimize firewall rules: Ensure that security measures do not excessively impact network performance.
- Use efficient encryption algorithms: Choose algorithms that can encrypt and decrypt quickly to reduce latency introduced by security processing.