webrtc源码分析(5)-Api接口
发布于 2023-05-23 | 作者: woder | 来源: 博客园 | 转载于:
博客园1、前言
本文介绍webrtc的API层整体结构和通话的api流程， 适合作为对webrtc有一定认知但是想研究源码的第一篇文章，推荐piasy的webrtc源码导读13以及webrtc源码导读10，本文实际就是在它们的基础上结合了webrtc源码上的example分析, 如果想要进一步分析整个呼叫过程，可以参考webrtc呼叫建立过程系列写的真的很用心
2、正文
2.1 关键类介绍
2.1.1 PeerConnection
最核心的类，没有之一，所有的功能接口都由它提供，围绕它转，提供了下列类型的接口:
local_streams()
remote_streams()
AddStream()
RemoveStream()
AddTrack()
RemoveTrack()
RemoveTrackNew()
传输相关: transceiver, sender, receiver
一个tranceiver包含了一个sender(编码，发送), receiver(解码接收)。addtrack的时候就是将track添加到transceiver的过程， 一个 sender 至多有一个要发送的 track一个要接收的 track 有一个 receiver，tranceiver使用mid作为标识。
AddTransceiver()
GetTransceivers()
CreateSender()
GetSenders()
GetReceivers()
local_description()
remote_description()
current_local_description()
current_remote_description()
pending_local_description()
pending_remote_description()
RestartIce()
CreateOffer()
CreateAnswer()
SetLocalDescription()
SetRemoteDescription()
AddIceCandidate()
RemoveIceCandidates()
SetBitrate()
SetAudioPlayout()
SetAudioRecording()
LookupDtlsTransportByMid()
GetSctpTransport()
signaling_state()
ice_connection_state()
standardized_ice_connection_state()
peer_connection_state()
ice_gathering_state()
2.1.2 PeerConnectionFactory
提供peerconnection，track，audio/video source的创建接口
CreatePeerConnection()
CreateLocalMediaStream()
CreateAudioSource()
CreateVideoTrack()
CreateAudioTrack()
2.2 通话的流程
2.2.1 流程概览
highlighter- C++
PeerConnectionFactory::CreatePeerConnection() // 创建Pc
PeerConnection::CreateAudioTrack() // 创建音轨
PeerConnection::AddTrack() // 添加音轨
PeerConnection::CreateVideoTrack() // 创建视轨
PeerConnection::AddTrack() // 添加视轨
PeerConnection::CreateOffer() // 创建offer sdp
PeerConnection::SetLocalDescription() // 设置local sdp
send sdp to peer... // 发送给对端
-----------------------------------------------------------
receive remote sdp... // 收到对端的sdp
PeerConnection::SetRemoteDescription() // 设置remote sdp
OnIceCandidate() // ice candidate收集到了
send candidate to peer... // 发送给对端
------------------------------------------------------------
receive remote candidate... // 收到对端的ice candidate
PeerConnection::AddIceCandidate() // 设置到pc中
2.2.2 创建PeerConnection
在InitializePeerConnection()函数中首先调用 webrtc::CreatePeerConnectionFactory()创建PeerConnectionFactory, 注意，创建的参数中需要提供[Audio|Video] [Encoder|Decoder]Factory()以提供视频编解码器
highlighter- C++
bool Conductor::InitializePeerConnection() {
RTC_DCHECK(!peer_connection_factory_);
RTC_DCHECK(!peer_connection_);
// 创建PC factory
peer_connection_factory_ = webrtc::CreatePeerConnectionFactory(
nullptr /* network_thread */, nullptr /* worker_thread */,
nullptr /* signaling_thread */, nullptr /* default_adm */,
webrtc::CreateBuiltinAudioEncoderFactory(),
webrtc::CreateBuiltinAudioDecoderFactory(),
webrtc::CreateBuiltinVideoEncoderFactory(),
webrtc::CreateBuiltinVideoDecoderFactory(), nullptr /* audio_mixer */,
nullptr /* audio_processing */);
if (!peer_connection_factory_) {
main_wnd_->MessageBox("Error", "Failed to initialize PeerConnectionFactory",
true);
DeletePeerConnection();
return false;
}
// 创建PC
if (!CreatePeerConnection(/*dtls=*/true)) {
main_wnd_->MessageBox("Error", "CreatePeerConnection failed", true);
DeletePeerConnection();
}
AddTracks();
return peer_connection_ != nullptr;
}
随后在CreatePeerConnection()中创建PeerConnection, 创建的时候可以放RTCConfiguration 用来做当前PC进行一些选项配置： dtls, audio, video, jitter, ice等等（FEC的配置不在此处，在创建PeerConncetionFactory中）
highlighter- C++
bool Conductor::CreatePeerConnection(bool dtls) {
RTC_DCHECK(peer_connection_factory_);
RTC_DCHECK(!peer_connection_);
webrtc::PeerConnectionInterface::RTCConfiguration config;
config.sdp_semantics = webrtc::SdpSemantics::kUnifiedPlan;
config.enable_dtls_srtp = dtls;
webrtc::PeerConnectionInterface::IceServer server;
server.uri = GetPeerConnectionString();
config.servers.push_back(server);
// 创建pc
peer_connection_ = peer_connection_factory_->CreatePeerConnection(
config, nullptr, nullptr, this);
return peer_connection_ != nullptr;
}
如此，PC就创建完毕了
2.2.3 设置track
创建完PC后就开始添加track，在函数AddTracks()中
highlighter- C++
void Conductor::AddTracks() {
if (!peer_connection_->GetSenders().empty()) {
return; // Already added tracks.
}
// 创建音轨
rtc::scoped_refptr<webrtc::AudioTrackInterface> audio_track(
peer_connection_factory_->CreateAudioTrack(
kAudioLabel, peer_connection_factory_->CreateAudioSource(
cricket::AudioOptions())));
// 添加音轨
auto result_or_error = peer_connection_->AddTrack(audio_track, {kStreamId});
if (!result_or_error.ok()) {
RTC_LOG(LS_ERROR) << "Failed to add audio track to PeerConnection: "
<< result_or_error.error().message();
}
// 创建视频源
rtc::scoped_refptr<CapturerTrackSource> video_device =
CapturerTrackSource::Create();
if (video_device) {
// 创建视轨
rtc::scoped_refptr<webrtc::VideoTrackInterface> video_track_(
peer_connection_factory_->CreateVideoTrack(kVideoLabel, video_device));
main_wnd_->StartLocalRenderer(video_track_);
// 添加视轨
result_or_error = peer_connection_->AddTrack(video_track_, {kStreamId});
if (!result_or_error.ok()) {
RTC_LOG(LS_ERROR) << "Failed to add video track to PeerConnection: "
<< result_or_error.error().message();
}
} else {
RTC_LOG(LS_ERROR) << "OpenVideoCaptureDevice failed";
}
main_wnd_->SwitchToStreamingUI();
}
audio_track(音轨)的创建使用的audio_source(音频源)来自于peer_connection_factory_->CreateAudioSource(options) ，options可以配置是否做增益，噪声消除等，但是audio_source的这种创建方式就导致我们似乎无法自定义一个音频源？ 创建完的````audio_track通过AddTrack()``的方式添加到pc中
而video_track(视轨)的创建使用的video_source来自于自定义的CapturerTrackSource，不妨来看看，这个类的弯弯绕绕比较多, 如下，在Create() 中调用VideoCaptureFactory::CreateDeviceInfo()获取所有的视频捕获设备的信息，然后创建了一个VcmCapturer, 最后使用VcmCapturer创建CapturerTrackSource；CapturerTrackSourceoverride了source()函数， 调用者调用source()获取到video_source源也就是VcmCapturer, VcmCapturer override了OnFrame() 用来提供帧
highlighter- C++
class VcmCapturer : public TestVideoCapturer,
public rtc::VideoSinkInterface<VideoFrame> {
public:
static VcmCapturer* Create(size_t width,
size_t height,
size_t target_fps,
size_t capture_device_index);
virtual ~VcmCapturer();
// 提供帧
void OnFrame(const VideoFrame& frame) override;
private:
VcmCapturer();
bool Init(size_t width,
size_t height,
size_t target_fps,
size_t capture_device_index);
void Destroy();
rtc::scoped_refptr<VideoCaptureModule> vcm_;
VideoCaptureCapability capability_;
};
class CapturerTrackSource : public webrtc::VideoTrackSource {
public:
static rtc::scoped_refptr<CapturerTrackSource> Create() {
const size_t kWidth = 640;
const size_t kHeight = 480;
const size_t kFps = 30;
std::unique_ptr<webrtc::test::VcmCapturer> capturer;
// 获取所有的视频捕获设备
std::unique_ptr<webrtc::VideoCaptureModule::DeviceInfo> info(
webrtc::VideoCaptureFactory::CreateDeviceInfo());
if (!info) {
return nullptr;
}
int num_devices = info->NumberOfDevices();
for (int i = 0; i < num_devices; ++i) {
// 创建一个VcmCapturer， VcmCapturer继承了VideoSinkInterface
capturer = absl::WrapUnique(
webrtc::test::VcmCapturer::Create(kWidth, kHeight, kFps, i));
if (capturer) {
// 使用video capturer创建一个CapturerTrackSource
return new rtc::RefCountedObject<CapturerTrackSource>(
std::move(capturer));
}
}
return nullptr;
}
protected:
explicit CapturerTrackSource(
std::unique_ptr<webrtc::test::VcmCapturer> capturer)
: VideoTrackSource(/*remote=*/false), capturer_(std::move(capturer)) {}
private:
// 提供源
rtc::VideoSourceInterface<webrtc::VideoFrame>* source() override {
return capturer_.get();
}
std::unique_ptr<webrtc::test::VcmCapturer> capturer_;
};
然后通过video_source创建的video_track也通过AddTrack() 的方式添加到peerconnection中
2.2.4 设置SDP
SDP的流程由于涉及到双方，所以放一个图直接说明
作为发起端， PeerConnection执行 CreateOffer()
highlighter- C++
void Conductor::ConnectToPeer(int peer_id) {
RTC_DCHECK(peer_id_ == -1);
RTC_DCHECK(peer_id != -1);
if (peer_connection_.get()) {
main_wnd_->MessageBox(
"Error", "We only support connecting to one peer at a time", true);
return;
}
if (InitializePeerConnection()) {
peer_id_ = peer_id;
// 创建offer
peer_connection_->CreateOffer(
this, webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
} else {
main_wnd_->MessageBox("Error", "Failed to initialize PeerConnection", true);
}
}
offer创建完成后通过异步回调的方式调用Conductor::OnSuccess()；
调用SetLocalDescription()把本地SDP设置到当前的PeerConnection中，根据SDP进行底层相关组件的创建和初始化 ，同时把创建的SDP转成json发送给远端
highlighter- C++
void Conductor::OnSuccess(webrtc::SessionDescriptionInterface* desc) {
//设置本地SDP
peer_connection_->SetLocalDescription(
DummySetSessionDescriptionObserver::Create(), desc);
std::string sdp;
desc->ToString(&sdp);
// For loopback test. To save some connecting delay.
if (loopback_) {
//如果是PeerConnection测试，整个SDP都是一样的，只需要
// 吧offer改成answer即可
// Replace message type from "offer" to "answer"
std::unique_ptr<webrtc::SessionDescriptionInterface> session_description =
webrtc::CreateSessionDescription(webrtc::SdpType::kAnswer, sdp);
peer_connection_->SetRemoteDescription(
DummySetSessionDescriptionObserver::Create(),
session_description.release());
return;
}
Json::StyledWriter writer;
Json::Value jmessage;
jmessage[kSessionDescriptionTypeName] =
webrtc::SdpTypeToString(desc->GetType());
jmessage[kSessionDescriptionSdpName] = sdp;
SendMessage(writer.write(jmessage));
}
过一段时间后，对端会发来remote sdp， 通过SetRemoteDescriptioin()设置好远端的sdp，对于发起端而言，SDP就设置完毕了，这个设置会根据对端发送来的SDP，去创建相应的track和别的东西。
highlighter- C++
void Conductor::OnMessageFromPeer(int peer_id, const std::string& message) {
.....
Json::Reader reader;
Json::Value jmessage;
if (!reader.parse(message, jmessage)) {
RTC_LOG(WARNING) << "Received unknown message. " << message;
return;
}
std::string type_str;
std::string json_object;
rtc::GetStringFromJsonObject(jmessage, kSessionDescriptionTypeName,
&type_str);
if (!type_str.empty()) {
// remote sdp come
if (type_str == "offer-loopback") {
// This is a loopback call.
// Recreate the peerconnection with DTLS disabled.
if (!ReinitializePeerConnectionForLoopback()) {
RTC_LOG(LS_ERROR) << "Failed to initialize our PeerConnection instance";
DeletePeerConnection();
client_->SignOut();
}
return;
}
absl::optional<webrtc::SdpType> type_maybe =
webrtc::SdpTypeFromString(type_str);
if (!type_maybe) {
RTC_LOG(LS_ERROR) << "Unknown SDP type: " << type_str;
return;
}
webrtc::SdpType type = *type_maybe;
std::string sdp;
if (!rtc::GetStringFromJsonObject(jmessage, kSessionDescriptionSdpName,
&sdp)) {
RTC_LOG(WARNING) << "Can't parse received session description message.";
return;
}
webrtc::SdpParseError error;
std::unique_ptr<webrtc::SessionDescriptionInterface> session_description =
webrtc::CreateSessionDescription(type, sdp, &error);
if (!session_description) {
RTC_LOG(WARNING) << "Can't parse received session description message. "
"SdpParseError was: "
<< error.description;
return;
}
RTC_LOG(INFO) << " Received session description :" << message;
// 设置remote sdp
peer_connection_->SetRemoteDescription(
DummySetSessionDescriptionObserver::Create(),
session_description.release());
if (type == webrtc::SdpType::kOffer) {
// 如果是offer类型， 创建一个answer作为回应
peer_connection_->CreateAnswer(
this, webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());
}
}
.....
}
而对于被发起端而言，仍然是通过OnMessageFromPeer( )接收到远端的offer sdp，如上，然后通过SetRemoteDescription()设置好该SDP后，使用CreateAnswer()创建answer，answer创建完毕之后会触发Conductor::OnSuccess()，需要将本地生成的SDP进行SetLocalDescription()，然后发送给对端。
2.2.5 设置 ICE
调用了接口SetLocalDescription()后，底层开始收集ICE Candidate，收集完成后回调Conductor::OnIceCandidate(), 解析之后写成json的形式发送给对方
highlighter- C++
void Conductor::OnIceCandidate(const webrtc::IceCandidateInterface* candidate) {
RTC_LOG(INFO) << __FUNCTION__ << " " << candidate->sdp_mline_index();
// For loopback test. To save some connecting delay.
if (loopback_) {
if (!peer_connection_->AddIceCandidate(candidate)) {
RTC_LOG(WARNING) << "Failed to apply the received candidate";
}
return;
}
Json::StyledWriter writer;
Json::Value jmessage;
// 写成json 发送给对方
jmessage[kCandidateSdpMidName] = candidate->sdp_mid();
jmessage[kCandidateSdpMlineIndexName] = candidate->sdp_mline_index();
std::string sdp;
if (!candidate->ToString(&sdp)) {
RTC_LOG(LS_ERROR) << "Failed to serialize candidate";
return;
}
jmessage[kCandidateSdpName] = sdp;
SendMessage(writer.write(jmessage));
}
对端在收到candidate之后，就执行AddCandidate()将candidate添加到PeerConnection中
highlighter- C++
void Conductor::OnMessageFromPeer(int peer_id, const std::string& message) {
....
std::string type_str;
std::string json_object;
rtc::GetStringFromJsonObject(jmessage, kSessionDescriptionTypeName,
&type_str);
if (!type_str.empty()) {
// handle sdp
.....
} else {
// handle candidate
std::string sdp_mid;
int sdp_mlineindex = 0;
std::string sdp;
if (!rtc::GetStringFromJsonObject(jmessage, kCandidateSdpMidName,
&sdp_mid) ||
!rtc::GetIntFromJsonObject(jmessage, kCandidateSdpMlineIndexName,
&sdp_mlineindex) ||
!rtc::GetStringFromJsonObject(jmessage, kCandidateSdpName, &sdp)) {
RTC_LOG(WARNING) << "Can't parse received message.";
return;
}
webrtc::SdpParseError error;
// 创建candidate
std::unique_ptr<webrtc::IceCandidateInterface> candidate(
webrtc::CreateIceCandidate(sdp_mid, sdp_mlineindex, sdp, &error));
if (!candidate.get()) {
RTC_LOG(WARNING) << "Can't parse received candidate message. "
"SdpParseError was: "
<< error.description;
return;
}
// 添加candidate
if (!peer_connection_->AddIceCandidate(candidate.get())) {
RTC_LOG(WARNING) << "Failed to apply the received candidate";
return;
}
RTC_LOG(INFO) << " Received candidate :" << message;
}
}
3、REF
1. WebRTC Native 源码导读（十四）：API 概览
2. WebRTC Native 源码导读（十）：视频数据 native 层之旅