WebRTC QoS方法十三.2（Jitter延时的计算）

一、背景先容

一些报文在网络传输中，会存在丢包重传和延时的情况。渲染时需要进行适当缓存，等候丢失被重传的报文或者正在路上传输的报文。
jitter延时计算是确认需要缓存的时间
另外，在检测到帧有重传情况时，也可适当在渲染时间内增加RTT延时时间，等候丢失重传的报文
二、jitter实现原理

JitterDelay由两部分耽误造成：传输大帧引起的耽误和网络噪声引起的耽误。计算公式如下：

其中：
estimate[0]：信道传输速率的倒数
MaxFrameSize：表示自会话开始以来所收到的最大帧size
AvgFrameSize：表示平均帧巨细，排除keyframe等超大帧
kNoiseStdDevs：       表示噪声系数2.33
var_noise_ms2_：     表示噪声方差
kNoiseStdDevOffset:  表示噪声扣除常数30 
实现函数：
JitterEstimator::CalculateEstimate

1、传输大帧引起的耽误

传输大帧引起的耽误

这个公式的原理是：[milliseconds] = [1 / bytes per millisecond] * [bytes] 
实现函数：

1. double FrameDelayVariationKalmanFilter::GetFrameDelayVariationEstimateSizeBased(
2.     double frame_size_variation_bytes) const {
3.   // Unit: [1 / bytes per millisecond] * [bytes] = [milliseconds].
4.   return estimate_[0] * frame_size_variation_bytes;
5. }

复制代码

filtered_max_frame_size_bytes

=  std::max<double>(kPsi * max_frame_size_bytes_, frame_size.bytes());
constexpr double kPsi = 0.9999;
filtered_avg_frame_size_bytes

是每一帧的加权平均值，但是需要排除key frame这种超大帧

estimate_[0]参数计算：
利用一个简化卡尔曼滤波算法，在处理帧耽误变革(frame_delay_variation_ms)的估计，考虑了帧巨细变革(frame_size_variation_bytes)和最大帧巨细(max_frame_size_bytes)作为输入参数。

1. void FrameDelayVariationKalmanFilter::PredictAndUpdate(
2.     double frame_delay_variation_ms,
3.     double frame_size_variation_bytes,
4.     double max_frame_size_bytes,
5.     double var_noise) {
6.   // Sanity checks.
7.   if (max_frame_size_bytes < 1) {
8.     return;
9.   }
10.   if (var_noise <= 0.0) {
11.     return;
12.   }
14.   // This member function follows the data flow in
15.   // https://en.wikipedia.org/wiki/Kalman_filter#Details.
17.   // 1) Estimate prediction: `x = F*x`.
18.   // For this model, there is no need to explicitly predict the estimate, since
19.   // the state transition matrix is the identity.
21.   // 2) Estimate covariance prediction: `P = F*P*F' + Q`.
22.   // Again, since the state transition matrix is the identity, this update
23.   // is performed by simply adding the process noise covariance.
24.   estimate_cov_[0][0] += process_noise_cov_diag_[0];
25.   estimate_cov_[1][1] += process_noise_cov_diag_[1];
27.   // 3) Innovation: `y = z - H*x`.
28.   // This is the part of the measurement that cannot be explained by the current
29.   // estimate.
30.   double innovation =
31.       frame_delay_variation_ms -
32.       GetFrameDelayVariationEstimateTotal(frame_size_variation_bytes);
34.   // 4) Innovation variance: `s = H*P*H' + r`.
35.   double estim_cov_times_obs[2];
36.   estim_cov_times_obs[0] =
37.       estimate_cov_[0][0] * frame_size_variation_bytes + estimate_cov_[0][1];
38.   estim_cov_times_obs[1] =
39.       estimate_cov_[1][0] * frame_size_variation_bytes + estimate_cov_[1][1];
40.   double observation_noise_stddev =
41.       (300.0 * exp(-fabs(frame_size_variation_bytes) /
42.                    (1e0 * max_frame_size_bytes)) +
43.        1) *
44.       sqrt(var_noise);
45.   if (observation_noise_stddev < 1.0) {
46.     observation_noise_stddev = 1.0;
47.   }
48.   // TODO(brandtr): Shouldn't we add observation_noise_stddev^2 here? Otherwise,
49.   // the dimensional analysis fails.
50.   double innovation_var = frame_size_variation_bytes * estim_cov_times_obs[0] +
51.                           estim_cov_times_obs[1] + observation_noise_stddev;
52.   if ((innovation_var < 1e-9 && innovation_var >= 0) ||
53.       (innovation_var > -1e-9 && innovation_var <= 0)) {
54.     RTC_DCHECK_NOTREACHED();
55.     return;
56.   }
58.   // 5) Optimal Kalman gain: `K = P*H'/s`.
59.   // How much to trust the model vs. how much to trust the measurement.
60.   double kalman_gain[2];
61.   kalman_gain[0] = estim_cov_times_obs[0] / innovation_var;
62.   kalman_gain[1] = estim_cov_times_obs[1] / innovation_var;
64.   // 6) Estimate update: `x = x + K*y`.
65.   // Optimally weight the new information in the innovation and add it to the
66.   // old estimate.
67.   estimate_[0] += kalman_gain[0] * innovation;
68.   estimate_[1] += kalman_gain[1] * innovation;
70.   // (This clamping is not part of the linear Kalman filter.)
71.   if (estimate_[0] < kMaxBandwidth) {
72.     estimate_[0] = kMaxBandwidth;
73.   }
75.   // 7) Estimate covariance update: `P = (I - K*H)*P`
76.   double t00 = estimate_cov_[0][0];
77.   double t01 = estimate_cov_[0][1];
78.   estimate_cov_[0][0] =
79.       (1 - kalman_gain[0] * frame_size_variation_bytes) * t00 -
80.       kalman_gain[0] * estimate_cov_[1][0];
81.   estimate_cov_[0][1] =
82.       (1 - kalman_gain[0] * frame_size_variation_bytes) * t01 -
83.       kalman_gain[0] * estimate_cov_[1][1];
84.   estimate_cov_[1][0] = estimate_cov_[1][0] * (1 - kalman_gain[1]) -
85.                         kalman_gain[1] * frame_size_variation_bytes * t00;
86.   estimate_cov_[1][1] = estimate_cov_[1][1] * (1 - kalman_gain[1]) -
87.                         kalman_gain[1] * frame_size_variation_bytes * t01;
89.   // Covariance matrix, must be positive semi-definite.
90.   RTC_DCHECK(estimate_cov_[0][0] + estimate_cov_[1][1] >= 0 &&
91.              estimate_cov_[0][0] * estimate_cov_[1][1] -
92.                      estimate_cov_[0][1] * estimate_cov_[1][0] >=
93.                  0 &&
94.              estimate_cov_[0][0] >= 0);
95. }

复制代码

2、网络噪声引起的耽误

网络噪声引起的耽误

constexpr double kNoiseStdDevs = 2.33; //噪声系数
constexpr double kNoiseStdDevOffset = 30.0;//噪声扣除常数
var_noise_ms2_ //噪声方差
实现函数：

噪声方差var_noise_ms2计算

var_noise_ms2 = alpha * var_noise_ms2_ + 
                (1 - alpha) *(d_dT - avg_noise_ms_) *(d_dT - avg_noise_ms_);
实现函数：JitterEstimator::EstimateRandomJitter

其中：
d_dT = 实际FrameDelay - 评估FrameDelay

             在JitterEstimator::UpdateEstimate函数实现
             

实际FrameDelay = （两帧之间实际接收gap - 两帧之间实际发送gap）

             在InterFrameDelayVariationCalculator::Calculate函数实现

1. absl::optional<TimeDelta> InterFrameDelayVariationCalculator::Calculate(
2.     uint32_t rtp_timestamp,
3.     Timestamp now) {
4.   int64_t rtp_timestamp_unwrapped = unwrapper_.Unwrap(rtp_timestamp);
6.   if (!prev_wall_clock_) {
7.     prev_wall_clock_ = now;
8.     prev_rtp_timestamp_unwrapped_ = rtp_timestamp_unwrapped;
9.     // Inter-frame delay variation is undefined for a single frame.
10.     // TODO(brandtr): Should this return absl::nullopt instead?
11.     return TimeDelta::Zero();
12.   }
14.   // Account for reordering in jitter variance estimate in the future?
15.   // Note that this also captures incomplete frames which are grabbed for
16.   // decoding after a later frame has been complete, i.e. real packet losses.
17.   uint32_t cropped_prev = static_cast<uint32_t>(prev_rtp_timestamp_unwrapped_);
18.   if (rtp_timestamp_unwrapped < prev_rtp_timestamp_unwrapped_ ||
19.       !IsNewerTimestamp(rtp_timestamp, cropped_prev)) {
20.     return absl::nullopt;
21.   }
23.   // Compute the compensated timestamp difference.
24.   TimeDelta delta_wall = now - *prev_wall_clock_;
25.   int64_t d_rtp_ticks = rtp_timestamp_unwrapped - prev_rtp_timestamp_unwrapped_;
26.   TimeDelta delta_rtp = d_rtp_ticks / k90kHz;
28.   // The inter-frame delay variation is the second order difference between the
29.   // RTP and wall clocks of the two frames, or in other words, the first order
30.   // difference between `delta_rtp` and `delta_wall`.
31.   TimeDelta inter_frame_delay_variation = delta_wall - delta_rtp;
33.   prev_wall_clock_ = now;
34.   prev_rtp_timestamp_unwrapped_ = rtp_timestamp_unwrapped;
36.   return inter_frame_delay_variation;
37. }

复制代码

评估FrameDelay =  estimate[0] * (FrameSize – PreFrameSize) + estimate[1]

评估FrameDelay实现函数：

1. double FrameDelayVariationKalmanFilter::GetFrameDelayVariationEstimateTotal(
2.     double frame_size_variation_bytes) const {
3.   double frame_transmission_delay_ms =
4.       GetFrameDelayVariationEstimateSizeBased(frame_size_variation_bytes);
5.   double link_queuing_delay_ms = estimate_[1];
6.   return frame_transmission_delay_ms + link_queuing_delay_ms;
7. }

复制代码

3、jitter延时更新流程

三、RTT延时计算 

VideoStreamBufferController::OnFrameReady函数，在判断帧有重传情况时，还会根据实际情况，在渲染帧时间里面增加RTT值。

JitterEstimator::GetJitterEstimate根据实际配置，可以在渲染时间中适当增加一定比例的RTT延时值。 

 
四、参考

WebRTC视频接收缓冲区基于KalmanFilter的耽误模型 - 简书在WebRTC的视频处理流水线中，接收端缓冲区JitterBuffer是关键的构成部分：它负责RTP数据包乱序重排和组帧，RTP丢包重传，请求重传关键帧，估算缓冲区耽误等功能...

https://www.jianshu.com/p/bb34995c549a

免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！更多信息从访问主页：qidao123.com:ToB企服之家，中国第一个企服评测及商务社交产业平台。