Use with Pipecat

webrtc.pipecat replaces Pipecat’s SmallWebRTC transport: same classes, same HTTP signalling, same data channel messages, without aiortc, PyAV or OpenCV.

pip install "wrtc[pipecat]"

Tested with Pipecat 1.15.0 on Python 3.11+, and with Pipecat’s JavaScript client (@pipecat-ai/client-js 1.16.0, @pipecat-ai/small-webrtc-transport 1.11.2) unchanged. Python WebRTC isn’t affiliated with Pipecat or Daily.

Why a transport of its own

Pipecat’s own transport also runs on the aiortc layer, with webrtc.aiortc.install_alias(). But there every frame is converted to aiortc’s and PyAV’s objects and back, and Pipecat’s workarounds for aiortc’s limits run although the engine doesn’t need them. webrtc.pipecat talks to the engine directly and keeps Pipecat’s classes and protocol, so bots and clients don’t change. Use the aiortc layer only for code that reaches into aiortc’s objects.

Changing the imports

# before
from pipecat.transports.smallwebrtc.connection import IceServer, SmallWebRTCConnection
from pipecat.transports.smallwebrtc.request_handler import SmallWebRTCRequest, SmallWebRTCRequestHandler
from pipecat.transports.smallwebrtc.transport import SmallWebRTCTransport

# after
from webrtc.pipecat.connection import IceServer, SmallWebRTCConnection
from webrtc.pipecat.request_handler import SmallWebRTCRequest, SmallWebRTCRequestHandler
from webrtc.pipecat.transport import SmallWebRTCTransport

The rest of the bot stays as it is, except what What differs lists. request_handler is the only module that needs FastAPI.

The development runner and unchanged bots

Pipecat’s runner, its WhatsApp support and bots you don’t change import Pipecat’s transport themselves. Call install_alias() first:

import webrtc.pipecat

webrtc.pipecat.install_alias()

from pipecat.runner.run import main  # its WebRTC routes now run on webrtc.pipecat

It raises RuntimeError if Pipecat’s transport is imported already. Call it in every process: one started with spawn imports everything again.

The runner shows its WebRTC routes only when aiortc is installed, though nothing imports it, so add Pipecat’s webrtc extra:

pip install "wrtc[pipecat]" "pipecat-ai[runner,webrtc]"

A bot

from fastapi import BackgroundTasks, FastAPI
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineWorker
from pipecat.transports.base_transport import TransportParams
from pipecat.workers.runner import WorkerRunner

from webrtc.pipecat import IceServer, SmallWebRTCConnection, SmallWebRTCTransport
from webrtc.pipecat.request_handler import SmallWebRTCPatchRequest, SmallWebRTCRequest, SmallWebRTCRequestHandler

app = FastAPI()
handler = SmallWebRTCRequestHandler(ice_servers=[IceServer(urls='stun:stun.l.google.com:19302')])


async def run_bot(connection: SmallWebRTCConnection) -> None:
    transport = SmallWebRTCTransport(
        webrtc_connection=connection,
        params=TransportParams(audio_in_enabled=True, audio_out_enabled=True),
    )
    # stt, llm, tts and the context aggregators, as in any Pipecat bot
    pipeline = Pipeline([transport.input(), stt, user, llm, tts, transport.output(), assistant])
    runner = WorkerRunner(handle_sigint=False)
    await runner.add_workers(PipelineWorker(pipeline))

    @transport.event_handler('on_client_disconnected')
    async def on_client_disconnected(transport, connection):
        await runner.cancel()

    await runner.run()


@app.post('/api/offer')
async def offer(request: SmallWebRTCRequest, background_tasks: BackgroundTasks) -> dict[str, str]:
    async def start(connection: SmallWebRTCConnection) -> None:
        background_tasks.add_task(run_bot, connection)

    return await handler.handle_web_request(request, start)


@app.patch('/api/offer')
async def add_candidates(request: SmallWebRTCPatchRequest) -> dict[str, str]:
    await handler.handle_patch_request(request)
    return {'status': 'success'}

await handler.close() on shutdown disconnects every client. The Pipecat bot example is a complete bot on the runner.

Configuration

  • ICE servers. As in Pipecat: URLs, or IceServer(urls=..., username=..., credential=...).

  • When the answer goes out. The protocol sends the bot’s candidates only in the answer, so it waits for the host candidates of every interface, plus a server-reflexive one with STUN and a relay one with TURN, at most ice_gathering_timeout_secs (5 s). A STUN server on the bot’s own private network can’t give a server-reflexive candidate, so there the answer waits the whole timeout.

  • Port range. port_range=(low, high) keeps the bot’s candidates in a range of ports. In Docker, publish the same range, like -p 40000-40019:40000-40019/udp. The runner makes its own handler, so this needs a server of your own.

  • Codecs. The answer follows the client’s offer. For H.264, see H.264.

What differs

  • Tracks. The *_input_track() methods return a webrtc.MediaStreamTrack, with no SmallWebRTCTrack around it; a source turned off comes with the track-status event. The replace_*_track() methods are coroutines.

  • Leaving. A client that closes its connection or data channel is closed at once. A client that pings, as Pipecat’s JavaScript client does, and stays disconnected for 8 s without a ping, offer or candidate is closed before its connection would fail. connection_timeout_secs counts from the answer.

  • Requests. A restart_pc offer drops the old pc_id from the handler.

  • Not there. force_transceivers_to_send_recv(), PIPECAT_SCTP_MAX_CHUNK_SIZE, SCREEN_VIDEO_JITTER_BUFFER_CAPACITY and the key frame renegotiation on connect: the engine doesn’t need them. capture_participant_video('camera') without a participant id, deprecated in Pipecat 1.13, isn’t supported.

What it fixes

Each checked against Pipecat’s transport on aiortc, with Pipecat’s workaround removed where it has one:

  • Large screen-share key frames, like 1080p ones, arrive whole.

  • A video track the pipeline doesn’t read stays flat in memory (#3066).

  • The data channel works on paths with a small MTU, like VPNs.

  • Lost audio packets are concealed, so audio stays whole on a lossy network.

  • Video adapts to a capped link instead of stalling.

  • The answer goes out as soon as the candidates are enough, not after seconds of gathering (#2426).

  • port_range lets Docker publish only a range of UDP ports (#1808, #1570).

  • A bot uses less CPU per session: media stays native, with no PyAV or OpenCV conversions.

  • A client that vanishes is noticed sooner (#3140).

  • A dead first TURN server falls back to the next.