Freeswitch video conference

Freeswitch video conference DEFAULT

THIS PAGE IS BEING RETIRED, USE Debian 10 Buster INSTEAD. MOST OF THESE STEPS HAVE ALREADY BEEN INTEGRATED INTO THE PACKAGES.

Detailed information on how to properly build and test the new FreeSWITCH™ 1.6 with awesome video conferencing support.

Click here to expand Table of Contents

FreeSWITCH™ 1.6 introduces new video features. See a working demonstration on the VoIP User Conference 539 of 1 May, 2015 putting the FreeSWITCH™ video conference through its paces.

FreeSWITCH™ now features video transcoding.

A few new modules were added with video codec support, such as mod_av, mod_vlc, mod_vpx, and others.  Based on that, mod_conference can now do a video MCU, which introduces the new video layouts and group layouts.  The video layouts feature allows you to set specific locations for the videos participants, floor holder and presenter.

Because  FreeSWITCH™ core now supports video, you can do playback and record with video support, so you can do things like 'record' and stream to a live YouTube event over RTMP, etc. Also, mod_verto now adds the ability to select video settings like resolution, bandwidth, camera selection and desktop sharing, all these features are demonstrated in the Verto HTML Demo page.

For more info, check out mod_conference page. Let's get all these new features configured!

Installation

Setting up the repos

Debian Jessie is required for this setup. A clean installation would make things easier and if you just follow the following steps everything will work smoothly.

A new set of dependencies was added to support the new features. Some of these dependencies were developed by the  FreeSWITCH™ core team and the packages were added to our repository.

Let's configure the Jessie repo:

Double click to select all commands, then copy and paste in your shell session.

echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list wget -O - http://files.freeswitch.org/repo/deb/freeswitch-1.6/key.gpg |apt-key add - apt-get update

Get FreeSWITCH

Choose from the packages listed to complete your FreeSWITCH™ installation.

apt-cache search freeswitch | less

We suggest that you install the Vanilla meta package to get a basic running installation.

apt-get install freeswitch-all

If you desire other FreeSWITCH modules, run the command above and choose from those listed by running

You might get systemd errors. Replace freeswitch.service in /lib/systemd/system/ with this in that case:

[Unit] Description=freeswitch After=syslog.target network.target local-fs.target [Service] ; service Type=forking PIDFile=/run/freeswitch/freeswitch.pid PermissionsStartOnly=true ExecStart=/usr/bin/freeswitch -u freeswitch -g freeswitch -ncwait -nonat -rp TimeoutSec=45s Restart=on-failure ; exec WorkingDirectory=/usr/bin User=root Group=daemon LimitCORE=infinity LimitNOFILE=100000 LimitNPROC=60000 ;LimitSTACK=240 LimitRTPRIO=infinity LimitRTTIME=7000000 IOSchedulingClass=realtime IOSchedulingPriority=2 CPUSchedulingPolicy=rr CPUSchedulingPriority=89 UMask=0007 [Install] WantedBy=multi-user.target
systemctl daemon-reload && systemctl start freeswitch

Installing from Source

Now that we have our repo configured, let's install the deps:

apt-get install -y --force-yes freeswitch-video-deps-most

Compiling

With all the dependencies installed, we can now properly build  FreeSWITCH™ 1.6:

# because we're in a branch that will go through many rebases it's # better to set this one, or you'll get CONFLICTS when pulling (update) git config --global pull.rebase true # then let's get the source cd /usr/src/ git clone https://freeswitch.org/stash/scm/fs/freeswitch.git freeswitch.git cd freeswitch.git ./bootstrap.sh -j ./configure

Enable mod_av:

perl -i -pe 's/#applications\/mod_av/applications\/mod_av/g' modules.conf

Let's make and install the project:

make make install make cd-sounds-install make cd-moh-install make samples

Certificates

It's mandatory that you use valid certificates on this setup. If you're setting up your own CA we can't guarantee that it'll work and we can't help you.

You can set up your certs at http://www.startssl.com/ .

Let's create our directory to store the certificates that will be used by  FreeSWITCH™ and Apache:

mkdir -p /usr/local/freeswitch/certs/

After creating the certs dir, we need to set our wss.pem, agent.pem and cafile.pem:

# NOT NEEDED if you will use following section on SELF SIGNED certificates /usr/local/freeswitch/certs/wss.pem # CERT, KEY AND CHAIN files separated by \n /usr/local/freeswitch/certs/agent.pem # CERT file AND key file separated by \n /usr/local/freeswitch/certs/cafile.pem # CHAIN file or root CA

Make sure you generate those files with proper layout, as described above, otherwise you'll get WS SETUP FAILED messages in FreeSWITCH™ console and things won't work.

If you really really want to use your own self-signed certificates, knowing that we will not support it, follow these steps:

# Create certificates: export DOMAIN=self.bkw.org wget http://files.freeswitch.org/downloads/ssl.ca-0.1.tar.gz tar zxfv ssl.ca-0.1.tar.gz cd ssl.ca-0.1/ perl -i -pe 's/md5/sha256/g' *.sh perl -i -pe 's/1024/4096/g' *.sh ./new-root-ca.sh ./new-server-cert.sh $DOMAIN ./sign-server-cert.sh $DOMAIN mkdir /usr/local/freeswitch/certs cat $DOMAIN.crt $DOMAIN.key > /usr/local/freeswitch/certs/wss.pem # Setup Sofia TLS: cat $DOMAIN.crt $DOMAIN.key > /usr/local/freeswitch/certs/agent.pem cat ca.crt > /usr/local/freeswitch/certs/cafile.pem   # vars.xml: <X-PRE-PROCESS cmd="set" data="internal_ssl_enable=true"/> <X-PRE-PROCESS cmd="set" data="external_ssl_enable=true"/> # Restart FreeSWITCH. ## Now make sure your system has ca.crt imported so it will trust your new found hotness.   # Setup Apache: # default-ssl: SSLCertificateFile /usr/local/freeswitch/certs/wss.pem SSLCertificateKeyFile /usr/local/freeswitch/certs/wss.pem SSLCertificateChainFile /usr/local/freeswitch/certs/wss.pem   # TEST: openssl s_client -connect $DOMAIN:443 openssl s_client -connect $DOMAIN:5061 openssl s_client -connect $DOMAIN:5081 openssl s_client -connect $DOMAIN:8082 # Depending on what you've setup you'll see: subject=/C=US/ST=Oklahoma/L=McAlester/O=Tonka Truck/OU=Secure Web Server/CN=self.bkw.org/[email protected] issuer=/C=US/ST=Oklahoma/L=McAlester/O=Whizzzzzzy Bang Bang/OU=Certification Services Division/CN=WBB Root CA/[email protected] # Or there abouts. # IMPORT THE CA.CRT CERTIFICATE INTO BROWSER ! # BE SURE YOU SET "domain" to self.bkw.org in vars.xml !

You cannot login, and FreeSWITCH is reporting (in debug) WS SETUP FAILED : this is because you have not imported your new self-signed certification authority in the browser. It will fail forever.
Import the self signed CA in browser, or visit with the browser the same address and port you will use as wss. Eg: if you have wss://192.168.1.34 then visit from same browser https://192.168.1.34 , then accept the certificate.

This (accept the certificate from https) is not the optimal solution, and can bring problems and failures to connect.

Optimal solution is to import into your browser your self signed Certification Authority.

Apache with SSL

Note that the HTML does not have to be served from the FreeSWITCH server, it can come from anywhere – the only requirement is that the SSL certificates on the FreeSWITCH server must be trusted.

Enable required Apache modules:

a2enmod ssl a2enmod rewrite a2ensite default-ssl

Copy the CHAIN, CERT and KEY files to /etc/ssl/private:

# NOT NEEDED if you made previous section SELF SIGNED certificates cp chain.pem /etc/ssl/private/chain.pem cp cert.pem /etc/ssl/private/cert.pem cp key.pem /etc/ssl/private/key.pem

Setup Apache default SSL enabled host:

# NOT NEEDED if you made previous section SELF SIGNED certificates perl -i -pe 's/SSLCertificateFile.*/SSLCertificateFile \/etc\/ssl\/private\/cert.pem/g' /etc/apache2/sites-available/default-ssl.conf perl -i -pe 's/SSLCertificateKeyFile.*/SSLCertificateKeyFile \/etc\/ssl\/private\/key.pem/g' /etc/apache2/sites-available/default-ssl.conf perl -i -pe 's/#SSLCertificateChainFile.*/SSLCertificateChainFile \/etc\/ssl\/private\/chain.pem/g' /etc/apache2/sites-available/default-ssl.conf

Rewrite HTTP to HTTPS access:

perl -i -pe 's/AllowOverride None/AllowOverride All/g' /etc/apache2/apache2.conf printf "RewriteEngine On\nRewriteCond %%{SERVER_PORT} 80\nRewriteRule ^(.*)$ https://MY.DOMAIN.NAME/\$1 [R,L]" > /var/www/html/.htaccess printf "\n\nRewriteCond %%{SERVER_PORT} 443\nRewriteRule ^$ https://MY.DOMAIN.NAME/verto [R,L]" >> /var/www/html/.htaccess

Make sure to replace MY.DOMAIN.NAME with your FQDN in the commands above.

Improving Apache SSL Security settings:

perl -i -pe 's/ServerSignature On/ServerSignature Off/g' /etc/apache2/conf-available/security.conf perl -i -pe 's/ServerTokens OS/ServerTokens Prod/g' /etc/apache2/conf-available/security.conf perl -i -pe 's/SSLProtocol(.*)/SSLProtocol All -SSLv2 -SSLv3/g' /etc/apache2/mods-available/ssl.conf perl -i -pe 's/SSLCipherSuite(.*)/SSLCipherSuite AES128+EECDH:AES128+EDH/g' /etc/apache2/mods-available/ssl.conf perl -i -pe 's/#SSLHonorCipherOrder on/SSLHonorCipherOrder on/g' /etc/apache2/mods-available/ssl.conf

Now restart Apache to reflect the new settings:

/etc/init.d/apache2 restart

You can use the following script to test your certificate setup. Create a file called test_tls.sh and paste this content:

#!/bin/bash   DOMAIN=MY.DOMAIN.NAME   echo 'x' | openssl s_client -connect $DOMAIN:443 > /dev/null 2>&1 if [ $? -ne 0 ]; then echo "$DOMAIN:443 TLS FAILED!" fi echo 'x' | openssl s_client -connect $DOMAIN:5061 > /dev/null 2>&1 if [ $? -ne 0 ]; then echo "$DOMAIN:5061 TLS FAILED!" fi echo 'x' | openssl s_client -connect $DOMAIN:5081 > /dev/null 2>&1 if [ $? -ne 0 ]; then echo "$DOMAIN:5081 TLS FAILED!" fi echo 'x' | openssl s_client -connect $DOMAIN:8082 > /dev/null 2>&1 if [ $? -ne 0 ]; then echo "$DOMAIN:8082 TLS FAILED!" fi

Then you can run the script:

chmod +x test_tls.sh ./test_tls.sh

If one of the above tests fail you MUST stop now and try to fix the issue before continuing this guide.

Demo Files

Verto demo pages uses some videos to demonstrate the new video features. Let's get this environment set up:

ln -s /usr/src/freeswitch.git/html5/verto/video_demo /var/www/html/verto mkdir -p /var/www/html/sounds/

Dialplan:

cp -f /usr/src/freeswitch.git/html5/verto/video_demo/dp/dp.xml /usr/local/freeswitch/conf/dialplan/default/0000_dp.xml fs_cli -x 'reloadxml'

FreeSWITCH™ default dialplan checks if you're using the default password and warns you in the console about this security role, this extensions has an sleep of 10 seconds. Let's disable this for now so we don't have to wait 10 seconds each time we make a test call:

perl -i -pe 's/default_password/disabled_default_password_checking/g' /usr/local/freeswitch/conf/dialplan/default.xml

The proper way to fix this is by changing your default password to something more secure. Change it ASAP.

WebRTC is slow to establish media. So we need to provide SDP asap.

We can do this by answering and sending some silent packets, instead of waiting for normal call setup:

Dialplan:

<extension name="itsp_send_call"> <condition field="destination_number" expression="^(\d+)$"> <action application="answer"/> <action application="playback" data="silence_stream://2000"/> <action application="set" data="effective_caller_id_number=333444555"/> <action application="set" data="effective_caller_id_name=ciaociaociao"/> <action application="bridge" data="{absolute_codec_string=pcmu}sofia/gateway/itsp/$1"/> </condition> </extension>

The default and preferred init system on Debian 8 (and CentOS 7) is Systemd. If you didn't revert to system V and you're compiling Freeswitch like you are, the building mechanism now recognises that you're using a Systemd based distribution and it will offer you a script that will create the required user and group and install the Systemd files, so you can just start Freeswitch. You can install it using:

build/startup/install_systemd.sh

If this works out for you, you can skip the following two sections (about owners/permissions and Systemd).

Set Owner and Permissions

Because you most likely built FreeSWITCH™ as root you must now create the user 'freeswitch' in group 'freeswitch' and change the FS files to owner = freeswitch and group = freeswitch and set the file mode appropriately. Execute the following commands as user root:

# create user 'freeswitch' # add it to group 'freeswitch' # change owner and group of the freeswitch installation cd /usr/local groupadd freeswitch adduser --disabled-password --quiet --system --home /usr/local/freeswitch --gecos "FreeSWITCH open source softswitch" --ingroup freeswitch freeswitch chown -R freeswitch:freeswitch /usr/local/freeswitch/ chmod -R ug=rwX,o= /usr/local/freeswitch/ chmod -R u=rwx,g=rx /usr/local/freeswitch/bin/*

systemd

systemd is the replacement service management system that replaces System V init. It is quite thorough and requires much simpler configuration scripts called Unit Files. systemd can start FreeSWITCH™ at boot time, monitor the application, restart it if it fails, and take other useful actions.

At the time of this writing (August 2015) the optimum systemd unit file is not certain, but the following have been proven to work on Debian 8.1 systems. If you have discovered a better approach, please update this section or submit a JIRA ticket so that we can capture this knowledge for the community. If you prefer to use the old sysvinit system, then use the instructions on the Debian 7 page.

The systemd unit file you're going to use should be something similar to this, this works fine when Freeswitch is installed using the methods above. The pre-compiled packages come with their own systemd files and scripts to install them, although you may certainly use this as a guide if you have trouble; but be sure to observe the correct file locations as they differ between compiled installations and package installations.

When troubleshooting systemd difficulties look for messages in and and grep for freeswitch messages.

The unit file start Freeswitch as root, so Freeswitch can setup priority, create some directories if needed, etc. Then it will continue running as user/group Freeswitch.

; This file in installations built from Master can be found in ; /usr/src/freeswitch.git/debian ; or ; /usr/src/freeswitch/debian [Unit] Description=freeswitch After=syslog.target network.target local-fs.target [Service] ; service Type=forking PIDFile=/usr/local/freeswitch/run/freeswitch.pid PermissionsStartOnly=true ExecStart=/usr/local/freeswitch/bin/freeswitch -u freeswitch -g freeswitch -ncwait -nonat -rp TimeoutSec=45s Restart=on-failure ; exec WorkingDirectory=/usr/local/freeswitch/bin User=root Group=daemon LimitCORE=infinity LimitNOFILE=100000 LimitNPROC=60000 ;LimitSTACK=240 LimitRTPRIO=infinity LimitRTTIME=7000000 IOSchedulingClass=realtime IOSchedulingPriority=2 CPUSchedulingPolicy=rr CPUSchedulingPriority=89 UMask=0007 [Install] WantedBy=multi-user.target

Now that the systemd unit file has been set up, you can copy it to the /etc/systemd/system directory where systemd looks for local customized files that won't be overwritten by systemd updates. Linking is possible to, but will give you issues when enabling (https://bugzilla.redhat.com/show_bug.cgi?id=955379)

cp /usr/src/freeswitch.git/debian/freeswitch-systemd.freeswitch.service /etc/systemd/system/freeswitch.service

Next, notify systemd to look for changes in its configuration:

systemctl daemon-reload

Now you are ready to start FreeSWITCH for the first time. Because FS is owned by user freeswitch it should create its databases and log files under that user (instead of root) so that those files will still be accessible to it under future runs.

systemctl start freeswitch

To stop FreeSWITCH:

systemctl stop freeswitch

To install FreeSWITCH to start at boot time:

systemctl enable freeswitch

At this time you have a fully functional  FreeSWITCH™ 1.6 demo system. You can now point your RTC-enabled browser to https://MY.DOMAIN.NAME/verto and you'll see the Verto Demo Page.

You can call for example extension 3500 to enter your first video conference. This is a conference with the video-mcu-stereo profile, that has the video features enabled by default.

After dialing to the conference, you'll hear that you are the only person in the conference and your video will appear on the screen:

When you place your mouse closer to 'Outbound Call' you'll see your user controls:

Now you can invite another caller and your video will be mixed together with the other caller video, and both video swill appear on the screen. The video disposition will behave according to the layout that was configured for this conference room.

FreeSWITCH™ 1.6 has a few layouts configured by default and you can change the video layout on the fly using the conference API, for example:

[email protected]> conference 3200-my.domain vid-layout [ 1up_top_left+5] [ 1up_top_left+7] [ 1up_top_left+9] [ 1x1] [ 1x2] [ 2up_bottom+8] [ 2up_middle+8] [ 2up_top+8] [ 2x1] [ 2x2] [ 3up+4] [ 3up+9] [ 3x3] [ 4x4] [ 5x5] [ 6x6] [ 8x8] [ overlaps] [presenter-dual-horizontal] [presenter-dual-vertical] [presenter-overlap-large-bot-right] [presenter-overlap-large-top-right] [presenter-overlap-small-bot-right] [presenter-overlap-small-top-right] [email protected]> conference 3200-video.evolux.net.br vid-layout 3up+4 Change to layout [3up+4] [email protected]>

All the new APIs and variables of  FreeSWITCH™ 1.6 are already documented on mod_conference page. 

Layouts

Definition

With video layouts you can configure specific video locations for your conference members, video floor holder and presenter.  FreeSWITCH™ ships with many pre-defined examples and so you can start using the layouts right now.

For more information on how layouts work, check the mod_conference documentation about Custom Conference Layouts.

Examples

These are the pre-defined layouts that comes with  FreeSWITCH™:

1up_top_left+5

1up_top_left+7

1up_top_left+9

1x1

1x2

2up_bottom+8

2up_middle+8

2up_top+8

2x1

2x2

3up+4

3up+9

3x3

4x4

5x5

6x6

8x8

overlaps

presenter-dual-horizontal

presenter-dual-vertical

presenter-overlap-large-bot-right

presenter-overlap-large-top-right

presenter-overlap-small-bot-right

presenter-overlap-small-top-right

Sours: https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+1.6+Video

Compile freeSWITCH, support MCU

In WebRTC + JsSIP + freeSWITCH one-to-one video chat we showed how to use WebRTC + JsSIP + freeSWITCH to construct a Web video chat application. This time, on the basis of the last time, we will demonstrate the construction of the video conferencing system.

See the following document:

It can be found that freeSWITCH has supported video MCU since 1.6. You can configure video floor, video mode and other parameters to control the video conference.

You can also configure the layout of multiple videos in a video conference.

I do n’t have time to adjust, I did n’t do related experiments, I did n’t understand how to use ...

In fact, for users of freeSWITCH, the process of calling a conference and calling someone is the same, so we do not need to modify the conference function test based on DEMO implemented by JsSIP, just fill in the conference number on the interface, such as 3000.

The article on compiling freeSWITCH under Windows describes how to compile freeSWITCH, just refer to it.

In WebRTC + JsSIP + freeSWITCH one-to-one video chat , we set freeSWITCH's proxy_media to true and commented it out.

Find the following configuration in internal.xml:

<param name="rtp-timeout-sec" value="300"/> <param name="rtp-hold-timeout-sec" value="1800"/>

Make them bigger, the first one is changed to 30000, and the second one is changed to 180,000. (The RTP establishment process is slow, and the originally set timeout end may not be connected.)

FreeSWITCH reserves some conferences. The extension numbers after 3000 are conference numbers. You can call these numbers to join the conference.

The effect is as follows:

View Image

There are some problems that did not have time to try:

  • How the layout configuration of the video conference takes effect
  • How a layout is related to a meeting
  • how to use video floor

The current understanding is this:

  • freeSWITCH will automatically synthesize video according to the layout you configured, and finally send a video stream to each participant
  • In this video stream, which videos are mixed is related to the video floor (guess, maybe wrong), or to the intensity or volume of the video.

That's it, get here temporarily. Whoever is interested can continue the experiment on this basis.


Related Reading:

Sours: https://titanwolf.org/Network/Articles/Article?AID=d8fab5ae-457a-460d-b6cb-99c36d76b5ac
  1. Real man hyeri
  2. Happy birthday massimo
  3. Ansible kubeadm

About

mod_conference provides both inbound and outbound conference bridge service for FreeSWITCH™. It can process multiple bit rates, load various profiles that specify DTMF controls, play prompt sounds and tones, and many other functions. You can create as many conferences as you like, as long as there still are free system resources (i.e. memory, CPU power, network bandwidth) available.

Conferences created in the dialplan use profiles that are defined in conf/autoload_configs/conference.conf.xml, if you are using the standard configuration files.

Click to expand Table of Contents

Conference configuration settings are stored in

Community member Stanislav Sinyagin has posted a nice personal blog entry on how he configured his conference bridge.

The configuration has the following structure, each "" should be replaced with specific configuration statements as detailed in the sections that follow.

<configuration name="conference.conf" description="Standard Conference configuration"> <advertise> [... config here ...] </advertise> <caller-controls> <group name="default"> [... config here ...] </group> </caller-controls> <chat-permissions> <profile name="default"> [... config here ...] </profile> </chat-permissions> <profiles> <profile name="default"> [... config here ...] </profile> </profiles> </configuration>

advertise

This block specifies whether empty conferences should be advertised in presence, allowing you to see status of empty conferences. Any endpoint that supports presence, such as mod_sofia, can subscribe to these presence notifications.

<advertise> <room name="[email protected]$${subdomain}" status="FreeSWITCH"/> </advertise>

The conference name (888 in this case) should be the profile name that you specified in <profiles> section, $${subdomain} will be replaced with the subdomain that you specified in freeswitch.xml.

"status" is advertised as whatever you pass to it (identifier) or "Available" if none is passed.

Example 'advertise' Event via mod_event_multicast

proto: conf login: [email protected] from: [email protected] status: FreeSWITCH rpid: idle event_type: presence Event-Name: PRESENCE_IN Core-UUID: c76e2d7d-39d7-dc11-93bf-0090fb0792c6 FreeSWITCH-Hostname: example.com FreeSWITCH-IPv4: 192.168.1.5 FreeSWITCH-IPv6: 127.0.0.1 Event-Date-Local: 2008-02-09 13:04:44 Event-Date-GMT: Sat, 09 Feb 2008 18:04:44 GMT Event-Date-timestamp: 1202580284348009 Event-Calling-File: mod_conference.c Event-Calling-Function: send_presence Event-Calling-Line-Number: 5037 Multicast-Sender: example.com

caller-controls

Caller controls are used to modify the state of the conference, such as lowering the volume, mute a participant, and such. Below are the commands that can be assigned to digits and executed during a conference. The "moderator-controls" group provides additional controls for participants who enter the conference with the moderator flag set. See below.

Reserved Control Group Names

NameDescription
noneUse this name to prevent installing caller-controls for callers to a conference.
defaultThis group of settings will be assigned if no "caller-controls" is specified. You can also assign it explicitly. This group is defined in vanilla config, thus removing it from the configurations will make no caller controls at all.

Actions

Each of these actions can be configured in caller-controls to be invoked by a conference member who enters the conference with this caller-controls group. These are instances of the conference API.

ActionDescriptionMin. Version
muteToggle audio from this member into the conference
mute onDisable audio from this member into the conference
mute offEnable audio from this member into the conference
deaf muteBlock audio from conference to this member as well as mute, in one action
energy upIncrease minimum energy threshold by 1 unit above which sound will be admitted into conference (noise gate)
energy equReset minimum energy threshold to default
energy dnDecrease minimum energy threshold by 1 unit
vol talk upIncrease member talk (mic) volume into conference by 1 unit
vol talk zeroReset talk volume to default setting
vol talk dnDecrease talk volume by 1 unit
vol listen upIncrease member receive (earpiece) volume by 1 unit
vol listen zeroReset member receive volume to default setting
vol listen dnDecrease member receive volume by 1 unit
hangupLeave the conference
eventSend the DTMF event via CUSTOM subclass to the event system (even to event socket)
lockToggle the conference lock state (no new members can enter when locked)
transferTransfer member to a given extension in a dialplan context
execute_applicationExecute a dialplan application
floorToggle yourself on and off of talking floor, as long as no one else has floor status.
vid-floorVideo floor. If video floor is currently locked, it will revert to auto; if there is no current holder, you become video floor holder1.6
vid-floor-forceVideo floor. If video floor is currently locked, it will revert to auto, otherwise you become the locked video floor holder1.6
vmuteVideo mute. Toggle video from this member into the conference1.6
vmute onDisable video from this member into the conference1.6
vmute offEnable video from this member into the conference1.6
vmute snapTake a video snapshot for this user to be used when in vmute1.6
vmute snapoffDiscard the vmute video snapshot1.6
<caller-controls> <group name="somekeys"> <control action="vol talk dn" digits="1"/> <control action="vol talk zero" digits="2"/> <control action="vol talk up" digits="3"/> <control action="transfer" digits="5" data="100 XML default"/> <control action="execute_application" digits="0" data="playback conf_help.wav"/> <control action="execute_application" digits="#" data="execute_dialplan conference-menu"/> </group> </caller-controls>

Limitations

Be aware that the caller-controls are applied across the entire conference session. You cannot apply one group of caller-controls to one member and then a second group of caller-controls to a second member in the same conference session.

chat-permissions

The FreeSWITCH conference also provides limited chat capabilities via SIP, XMPP, and Verto.

A conference can be controlled through chat. This profile exists as part of the conference entity so you can have multiple profiles to limit access. When the conference advertises its presence (above) to a chat server, anyone on a federated server can talk to it and issue commands by typing messages via their favorite chat client (even if that is another FreeSWITCH™ box). To send commands to ConferenceA, from your jabber client send a message to [email protected]

As of revision 3789 all commands except "list" have been disabled "until there is security"

Another Note: After spending much time trying to get control of mod_conference via Openfire XMPP server, asked around, and "bougyman" suggested that XMPP muc functionality is very limited as of January 2012.

Here is an example configuration

<configuration name="conference.conf" description="Audio Conference"> <profiles> <profile name="default"> <param name="chat-permissions" value="default"/> </profile> </profiles> <chat-permissions> <profile name="default"> <!-- both of these users have a functionally equivalent capability set --> <user name="[email protected]" commands="all"/> <!-- individually specified commands must be enclosed with the | (pipe) character --> <user name="[email protected]" commands="|deaf|dial|energy|kick|list|lock|mute|norecord|play|record|relate|say|saymember|stop|transfer|undeaf|unlock|unmute|volume_in|volume_out|"/> </profile> </chat-permissions> </configuration>

profiles

You can specify a number of different profiles in the profiles section, these will let you easily apply a number of settings to a conference. Please note that the profiles are not conference rooms, but define settings that are later applied to conference rooms. The dialplan section in this document will describe how you create conference rooms and apply these profile settings.

<profiles> <profile name="default"> <param name="paramName" value="paramValue"/> </profile> </profiles>

You can have any number of <profile> tags, and each <profile> can have any number of <param> tags up to the entire set of parameters.

Conference Profile Parameters

You may specify the conference profile parameters listed below. Keep in mind that if TTS is enabled all audio-file params beginning with 'say:' will be considered text to say with TTS. A TTS module must be loaded by FreeSWITCH for this to work.

NameDescriptionExample valueAllowed valueDefault valuePlayed forMin. Version
announce-countRequires TTS. If the number of members equals or exceeds this value, the conference will speak the count to the conference after a new member joins5

<integer>

0

auto-gain-levelEnables Automatic Gain Control (AGC). If the parameter is set to 'true', then the default AGC value is used. If set to a number it will override the default value900<integer>|true1100

auto-recordSet a filename or stream value for this parameter in order to enable recording every conference call. Within mod_conference there is a special parameter named ${conference_name} that can be used to form the record filename. All channel variables are accessible as well for generating a unique filename. NOTE: auto-record doesn't begin recording until the number of conferees specified in min-required-recording-participants have joined.

Example 1: /var/myNFSshare/${conference_name}_${strftime(%Y-%m-%d-%H-%M-%S)}.wav

Example 2:shout://user:[email protected]/live.mp3

Example 3: /var/myNFSshare/${conference_name}_${strftime(%Y-%m-%d-%H-%M-%S)}.mp4 will give you audio and video recording.

<file-path>


broadcast-chat-messageMessage to send in response to chat messages.Please go away.<string>


caller-controlsName of the caller control group to use for this profile. It must be one of those specified in the <caller-controls> sectionsomekeys<control-name>default

moderator-controlsName of the moderator control group to use for this profile. It must be one of those specified in the <caller-controls> sectionsomekeys<control-name>default

cdr-log-dirTarget directory for conference CDRs to be written. Use "auto" to store in $PREFIX/log/conference_cdr. An absolute path is acceptable as is a relative path. A relative path will yield $PREFIX/log/<value> for the conference CDR directoryauto<path>|auto


cdr-event-modeInclude full cdr or path to file in conference cdr eventcontentcontent|file|none


caller-id-nameDefault Caller ID Name for outbound calls originated by mod_conferenceFreeSWITCH<string>conference

caller-id-numberDefault Caller ID Number for outbound calls originated by mod_conference8777423583<string>

0000000000



channelsThe number of audio channels. Special value "auto" sets this based on the channels of the first member to enter11|2|auto1
1.4
comfort-noiseSets the volume level of background white noise to add to the conference. Special value "true" sets to default value. 0 for no CN (total silence).14000-10000|true1400

conference-flagsCan be any combination of allowed values separated by "|" (pipe character).wait-mod|audio-always

See Table Conference Flags below




descriptionDescription of the conference that is included in some eventsThe main conference<string>


domainThe domain name used for presence events from conferences$${domain}<string>


energy-levelNoise gate. Energy level required for audio to be sent to the other users. The energy level is a minimum threshold of 'voice energy' that must be present before audio is bridged into the conference. Useful if a participant is in a noisy environment, so their background noise is heard only when they speak. 0 disables the noise gate and will bridge all packets even if they are only background noise.3000-1800


intervalNumber of milliseconds per frame. Which may be different from ptime in SIP SDP, or driver with TDM. Higher numbers require less CPU but can degrade conversation quality, so experimentation with your setup is best. The default is good for conversation quality. Special "auto" value sets interval based on interval of the first member who enters.2010-120 (only if divisible by 10) or auto


max-membersSets a maximum number of participants in conferences with this setting in its profile.20Integer > 1


member-flagsCan be any combination of allowed values separated by "|". See table below for descriptions.nomoh|muteSee Table Member Flags below


pinPin code that must be entered before user is allowed to enter the conference.12345<sritng>


moderator-pinPin code that must be entered before moderator is allowed to enter the conference.12345<string>


pin-retriesMax number of times the user or moderator will be asked for PIN3

<integer> >= 0

3

rateAudio sample rate. Special value "auto" sets this based on the rate of the first member to enter.8000

8000, 12000, 16000, 24000, 32000, 44100, 48000, auto




sound-prefixSet a default path here so you can use relative paths in the other sound params./soundfiles<path-string>


suppress-eventsFor use with the event socket. This parameter is a comma delimited string that specifies which events will NOT be sent to the socket when getting CUSTOM conference::maintenance events.del-member,start-talking,stop-talkingSee Table Conference Events below


terminate-on-silenceSpecifies the number of contiguous seconds of silence after which the conference will automatically terminate and disconnect all members.300<integer> > 0


timer-nameName of the timer interface in freeswitch to use for timing the conferencesoft<timer-name>soft

tts-engineText-To-Speech (TTS) Engine to usecepstral<tts-name>


tts-voiceTTS Voice to usedavid<tts-voice>


verbose-eventsEvents related to the conference will send more data. Specifically the events related to members will include all the channel variables on each eventtruetrue|falsefalse

alone-soundFile to play if you are alone in the conferenceyactopitc.wav<sound-path>
User
bad-pin-soundFile to play to when the pin is invalidinvalid-pin.wav<sound-path>
User
enter-soundFile to play when you join the conferencewelcome.wav<sound-path>
All
exit-soundFile to play when you leave the conferenceexit.wav<sound-path>
All
is-locked-soundFile to play when the conference is locked during the call to the members in the conferenceis-locked.wav<sound-path>
All
is-unlocked-soundFile to play when the conference is unlocked during the callis-unlocked.wav<sound-path>
All
join-only-soundFile to play when member with join-only flag tries to create the conference (join as the first).no_reources_try_later.wav<sound-path>
User
kicked-soundFile to play when you are kicked from the conferencekicked.wav<sound-path>
User
locked-soundFile to play when the conference is locked and someone goes to joinlocked.wav<sound-path>
All
max-members-soundIf max-members has been reached, this sound plays instead of allowing new users to the conferencemax-members.wav<sound-path>
Caller
moh-soundthe given sound file/resource will be played only when the conference size is 1 member. It will loop over and over until the member count is 2 or more. When the conference goes back to 1 member it will play againidlemusic.wav<sound-path>
All
muted-soundFile to play when member is mutedmuted.wav<sound-path>
User
mute-detect-soundIf the mute-detect member-flag has been set, this sound plays when the user talks while mutedmute-detect.wav<sound-path>
User
perpetual-soundThe given sound file/resource will be played on a loop forever. This can be used to broadcast sales or emergency messages to callers.announcement.wav<sound-path>
All
pin-soundFile to prompt the user to enter the conference pin code.pin.wav<sound-path>
User
unmuted-soundFile to play when member is unmuted.unmuted.wav<sound-path>
User
ivr-dtmf-timeoutInter-digit timeout between DTMF digits in milliseconds500

<integer> >= 500

500

ivr-input-timeoutTime to wait for the first DTMF in milliseconds, zero = forever0<integer> >= 5000 or 00

endconf-grace-timeDefines how much time all members have before the conference is terminated when the last member with endconf leaves in seconds600<integer> >= 00

min-required-recording-participantsMinimum number of conference participants required for their audio to be heard in a recording and for auto recording to start. This can be either 1 (the default) or 2.11-21

outcall-templTemplate to use for outcall URL
<string>


video-modeThe mode to run video conferencing in. passthrough is non transcoded video follow audio. transcode allows for better switching and multiple codecs. mux allows for multiple parties on the video canvas at the same time

mux

mux

transcode

passthrough

 passthrough
1.6
video-layout-nameThe layout name from conference_layouts.conf.xml or group prefixed by "group:". Setting this setting will enable the video mux. Not setting this will switch video presentation based on floor.

group:grid

2x2

<layout-name> or group:<group-name>

1.6
video-canvas-bgcolorOverall Canvas color for video mux canvas as an HTML HEX color code#333333<color-spec>

#333333


1.6

video-letterbox-bgcolor

Color to use for bars on the caller video if their aspect ratio doesn’t fit the member square in the layout as an HTML HEX color code#FFFFFF<color-spec>

#000000


1.6
video-canvas-sizePixel dimensions of the video mux canvas800x600<integer>x<integer>

1.6
video-fpsFrames per second to run the video mux at15<integer>15
1.6
video-codec-bandwidth

The bandwidth maximum for the conference output video stream, This setting can be auto, our defined in KB (kilobytes), mb (megabits) or MB (megabytes)

1MBauto|<integer>KB|<integer>mb|<integer>MB

1.6
video-no-video-avatarPath to PNG file for member without video to display.



1.6
video-canvas-countNumber of canvases to use in mux mode11-51

1.6 

video-qualityMotion factor of the video, this is used to adjust the video bitrate. From 1 to 4, low motion to high motion video. (ref. Kush Gauge formula for determining bitrate).11-41
1.6
video-border-colorBorder color around each video feed#FFFFFF<color-spec>#000000
1.6
video-border-sizeBorder size (in pixels) around each video feed5<integer> 0-500
1.6
video-mute-bannerSets the video mute banner text, font, font size, font color, and background color for the member. font_scale valid values 5-50, fg/bg hex color code, all settings besides text are optional.


1.6
video-super-canvas-bgcolorBackground color of the supercanvas#FFFFFF<color-spec>#068DF3
1.6
video-super-canvas-label-layersLabel canvases in supercanvastrueBooleanfalse
1.6
video-super-canvas-show-all-layersDisplay all canvases, even empty onestrueBooleanfalse
1.6
video-auto-floor-msecMilliseconds of speaking before a speaker gets the floor800Integer >= 00
1.6
video-kps-debounceMilliseconds between sending packets to the client to request a client bitrate adjustment (video only). Note that FreeSWITCH may decide to force sending more frequently under certain circumstances.30000Integer >= 030000
1.6
auto-record-canvas-idWhich canvas on the supercanvas to auto record, (range is the )2Integer 1 - # of canvases1
1.6
video-layout-confSpecify an alternate conference layout config file.custom_conference_layouts.conf
conference_layouts.conf
1.8

These flags can be applied to individual conferees when entering the conference to control what they can do and how their media is processed.

Member FlagDescriptionMin. Version
muteEnter conference muted
deafEnter conference deafed (can not hear conference); will also mute the mic
mute-detectPlay the mute_detect_sound when talking detected by this conferee while muted
dist-dtmfSend any DTMF from this member to all participants
moderatorFlag member as a moderator
nomohDisable music on hold when this member is the only member in the conference
endconfEnds conference when all members with this flag leave the conference after profile param endconf-grace-time has expired
mintwoEnd conference when it drops below 2 participants after a member enters with this flag
ghostDo not count member in conference tally
join-onlyOnly allow joining a conference that already exists
positionalProcess this member for positional audio on stereo outputs1.4
no-positionalDo not process this member for positional audio on stereo outputs1.4
join-vid-floorLocks member as the video floor holder1.6
no-minimize-encodingBypass the video transcode minimizer and encode the video individually for this member1.6
vmuteEnter conference video muted1.6
second-screenOpen a 'view only' connection to the conference, without impacting the conference count or data.1.6

These flags control the operation of the conference session and apply to all members.

Conference FlagDescriptionMin. Version
wait-mod Members will wait (with music) until a member with the 'moderator' flag set enters the conference1.0
audio-alwaysDo not use energy detection to choose which participants to mix; instead always mix audio from all members1.2
restart-auto-recordIf the auto-record conference param is set, and recording is stopped, auto recording will continue in a new file

1.2.16

rfc-4579Enable support for RFC-4579 SIP–based call control for conferences1.4
livearray-syncAdd support for livearray advertisement for the conference status1.4
auto-3d-positionEnable 3d positioned audio support1.4
json-eventsSend event-channel JSON events when members enter and leave1.4
video-floor-onlyOnly video members can get floor. REMOVED in 1.61.4
minimize-video-encoding

Use a single video encoder per output codec

1.6

livearray-json-status

Machine parseable version of display for conference live array. Example:

{"audio":{"muted":false,"onHold":false,"talking":true,"floor":true,"energyScore":639},

"video":{"avatarPresented":false,"mediaFlow":"sendRecv","muted":false,"floor":true,"reservationID":null,"videoLayerID":0}}

1.6
video-bridge-first-twoIn mux mode, If there are only 2 people in conference, you will see only the other member1.6
video-muxing-personal-canvasIn mux mode, each member will get their own canvas and they will not see themselves1.6
video-required-for-canvasOnly video participants will be shown on the canvas (no avatars)1.6
video-mute-exit-canvasDon't display video muted users on the canvas1.6
manage-inbound-video-bitrateIf calling client supports TMMBR, on each change of layout position FreeSWITCH will instruct the client to increase/reduce the video bit rate appropriately.1.6

These events are available to event consumers.

Event NameDescription
add-memberMember added to a conference

del-member

Member removed from a conference
energy-levelConference default energy level changed
volume-levelConference default volume level changed
gain-levelConference default gain level changed
dtmfKey bound to transfer, event, or execute_application is hit by member
stop-talkingMember stopped talking (as detected by energy level)
start-talkingMember started talking (as detected by energy level)
mute-detectDetected member speaking (as detected by energy level) while muted
mute-memberMember became muted
unmute-memberMember became unmuted
kick-memberMember kicked
dtmf-memberKey bound to DTMF is hit by member
energy-level-memberMember energy level changed
volume-in-memberMember gain level changed
volume-out-memberMember volume level changed
play-fileConference–level play file started
play-file-doneConference–level play file ended
play-file-memberMember–level play file started
lockConference locked, no one else can enter
unlockConference unlocked
transferMember transferred to another conference
bgdial-resultResult from bgdial API command
floor-changeConference floor changed
recordConference recording started or stopped

The conference dialplan application is used to create conferences and to bind a profile to them.

<action application="conference" data="confname[@profile][+[pin][+flags{mute|deaf|...}]"> <action application="conference" data="bridge:confname[@profile]:none|endpoint[+flags{mute|deaf|...}]">

The first time a conference name (confname) is used, it will be created on demand, and the pin will be set to what ever is specified at that time: the pin in the data string if specified, or if not, the "pin" setting in the conference profile, and if that is also unspecified, then there is no pin protection. Any later attempt to join the conference must specify the same pin number, if one existed when it was created.

"profile" should be one of those specified in the conference configuration, or "default".

If the data value starts with "" then it is a bridging conference. The conference name should not be already in use. You can specify the special literal value of "_uuid_" for the conference name, and a a session-specific unique id will be generated for you.

Conferences stay alive until the number of members falls below the minimum. The minimum for bridging conferences is 2, and for other dynamically created conferences is 1.

Dialplan Examples

Action dataDescription
confnameprofile is "default", no flags or pin
confname+1234profile is "default", pin is 1234
[email protected]+1234profile is "profilename", pin=1234, no flags
confname+1234+flags{mute}profile is "default", pin=1234, one flag
confname++flags{endconf|moderator}profile is "default", no p.i.n., multiple flags
bridge:confname:[email protected]${domain_name}a "bridging" conference, you must provide another endpoint, or 'none'.
bridge:_uuid_:nonea "bridging" conference with UUID assigned as conference name
*Note that while some parameters are optional, their order is very important
<action application="conference" data="[email protected]">
<extension name="test_bridging_conference"> <condition field="destination_number" expression="^(3000)$"> <action application="answer"/> <action application="playback" data="connecting_your_call.wav"/> <action application="set" data="ringback=${us-ring}"/> <action application="conference" data="bridge:$1-${domain_name}@default:user/[email protected]${domain_name}"/> </condition> </extension>

Adding Callers From Within The Conference With DTMF

By combining several elements — the dialplan, API calls, bind_digit_action — you can create a simple system for a caller to add another user to the conference. See Conference_Add_Call_Example for the code and explanation.

Announcing Caller Count While In Conference

See Conference Announce Count Inline for a simple example of how to allow a caller to hear an announcement of how many members are in the conference.

Propagate Out-of-Band DTMF to Conference Members

By default, out-of-band DTMF signals (RFC 2833?) are absorbed by the conference. However, there are two ways to accomplish this:

  • Set the dist-dtmf member flag in the conference configuration XML, eg: <param name="member-flags" value="dist-dtmf"/> With this parameter set, all of the caller controls such as modifying volume will be disabled and DTMF will simply pass through to all other members of the conference.
  • There is also an API call that will allow your application to send DTMF to a single conference member or all members:
<confname> dtmf-string <[member_id|all|last]> <digits>

Prompt for Name

Dialplan to prompt callers to speak their names before joining the conference and announce their name to other participants.

<extension name="Record Name and schedule conf announce"> <condition field="destination_number" expression="^55(3\d\d\d)$"> <action application="answer"/> <action application="set" data="namefile=/tmp/${uuid}-name.wav" inline="true"/> <action application="sleep" data="1000"/>> <action application="playback" data="voicemail/vm-record_name1.wav"/> <action application="playback" data="tone_stream://%(1000,0,500)"/> <action application="record" data="${namefile} 1"/> <action application="playback" data="ivr/ivr-call_being_transferred.wav"/> <action application="set" data="res=${sched_api +1 none conference $1-${domain} play file_string://${namefile}!conference/conf-has_joined.wav}"/> <action application="transfer" data="$1 XML default"/> </condition> </extension>
CommandDescriptionSyntaxExamplesMin. Version
agcAdjust conference automatic gain control.conference <confname> agc [on [<level>] | off]Query level: conference testconf agcDisable: conference testconf agc offEnable: conference testconf agc onEnable w/ level: conference testconf agc on 1120
bgdialBackground dial a destination via a specific endpoint conference <confname> bgdial <dial-string> [<callerid_number> [<callerid_name>]]

chkrecordQuery record status of conferenceconference <confname> chkrecord

deafMake a conference member deafconference <confname> deaf <member_id>|all|last|non_moderator

dialDial a destinationconference <confname> dial <dial-string> [<callerid_number> [<callerid_name>]]

dtmf Send DTMF to any member of the conferenceconference <confname> dtmf <member_id>|all|last|non_moderator <digits>conference testconf dtmf all 134
energyAdjusts the conference energy level for a specific member conference <confname> energy <member_id>|all|last|non_moderator <newval>

enter_soundChanges the sound played while entering the conferenceconference <confname> enter_sound on|off|none|file <filename>

exit_soundChanges the sound played while leaving the conferenceconference <confname> exit_sound on|off|none|file <filename>

floorToggle floor status of the member.conference <confname> <member_id>|all|last|non_moderator

file_seekSeek to a point in the currently playing file, in millisecondsconference <confname> file_seek [+|-]<val>

Move forward 1 second: conference testconf file_seek +1000

Move backwards 1 second: conference testconf file_seek -1000

Move to 10 seconds from the start: conference testconf file_seek 10000


file-volChanges the volume of the currently playing sound fileconference <confname> file-vol <val> [async]

get

Get runtime parameter of the given conference

(table of allowed parameters below)

conference <confname> get <parameter_name>

hupKick without the kick soundconference <confname> hup <member_id>|all|last|non_moderator

kickKicks the member or all members from the conferenceconference <confname> kick <member_id>|all|last|non_moderator

listLists conferencesconference list [pretty|summary|count|delim <string>]

lockLock a conference so no new members will be allowed to enterconference <confname> lock

muteMutes a conference memberconference <confname> mute <member_id>|all|last|non_moderator [quiet]

nopinRemoves a pin from a conferenceconference <confname> nopin

norecordRemoves a recording or all recordings from a conferenceconference <confname> norecord <file-path>|all

conference testconf norecord all

conference testconf norecord /tmp/foo.wav


pausePause a conference recordingconference <confname> pause <file-path>

pause_playPause playback of file to a conferenceconference <confname> pause_play

pinSet or change a pin number for a conferenceconference <confname> pin <pin-number>

Note: Members joined from conference dial command

do not get prompted for a pin


playPlay an audio file to a conference or a specific memberconference <confname> play [{vol=<volume>,full-screen=true,png_ms=100}]<file-path> [async|<member_id> [nomux]]full-screen param will play the video in full screen mode in the conference,
png_ms depends on mod_png to be loaded, Specify a PNG file to play and how many milliseconds, -1 for indefinite

recordRecord a conference to a file or streamconference <confname> record <file-path>

recordingStart or control a conference recording

conference <confname> recording start <file-path>

conference <confname> recording check

conference <confname> recording stop <file-path>|all

conference <confname> recording pause <file-path>

conference <confname> recording resume <file-path>



relate

Mute or Deaf a specific member to another member

Control who sees who's video when not using video mux

conference <confname> relate <member_id>[,<member_id>] <other_member_id>[,<other_member_id>] [nospeak|nohear|clear|sendvideo]

See expanded entry below.
resumeResume recordingconference <confname> resume <file-path>

saySpeak text into conference using TTSconference <confname> say <text>

saymemberSpeak text to a member of a conference using TTSconference <confname> saymember <member_id> <text>

setSet runtime parameter of a conferenceconference <confname> set <parameter_name> <value>

stopStops any queued audio from playing into conferenceconference <confname> [current|all] [<member_id>]

tmuteToggle mute on/off for a member of a conferenceconference <confname> tmute <member_id>|all|last|non_moderator [quiet]

transferTransfer a member from one conference to another conferenceconference <confname> transfer <conference_name> <member_id>

undeafAllow conference member to hear againconference <confname> undeaf <member_id>|all|last|non_moderator

unlockUnlock a conference so that new members can enterconference <confname> unlock

unmuteUnmute a conference memberconference <confname> unmute <member_id>|all|last|non_moderator [quiet]

volume_inAdjust the gain on the audio coming from a memberconference <confname> volume_in <member_id>|all|last|non_moderator [<newval>]

volume_outAdjust the volume of audio going to a memberconference <confname> volume_out <member_id>|all|last|non_moderator [<newval>]

xml_listList all or specific conferences in XML format

conference xml_list

conference <confname> xml_list



vid-floorSet or force the video floor for a memberconference <confname> vid-floor <member_id>|last [force]
1.6
vmuteEnable video mute for a memberconference <confname> vmute <member_id>|all|last|non_moderator [quiet]
1.6
tvmuteToggle video mute on/off for a memberconference <confname> tvmute <member_id>|all|last|non_moderator [quiet]
1.6
vmute-snapTake or clear a snapshot image of the member to be used during vmuteconference <confname> vmute-snap <member_id>|all|last|non_moderator [clear]
1.6
unvmuteUnmute video for a memberconference <confname> unvmute <member_id>|all|last|non_moderator [quiet]
1.6
vid-bannerSet a text banner on the members videoconference <confname> vid-banner <member_id>|all|last|non_moderator <text>
1.6
vid-mute-imgCheck, set or clear the file to use as the members video mute imageconference <confname> vid-mute-img <member_id>|all|last|non_moderator [<path>|clear]
1.6
vid-logo-img

Check, set or clear the file to use as the members video logo image

conference <confname> vid-logo-img <member_id>|all|last|non_moderator [<path>|clear]

 conference foo vid-logo-img 1 {options}/path/to/png

 conference foo vid-logo-img 1 clear

*see settable channel var video_logo_path below for all the options you can use with this vid-logo-img api

1.6
vid-res-idSet or clear the video reservation id for this memberconference <confname> vid-res-id<member_id>|all <val>|clear [force]

conference foo vid-res-id 1 res-bar

conference foo vid-res-id 2 res-baz force

conference foo vid-res-id all clear

1.6
clear-vid-floorClear when the video floor is locked to a userconference <confname> clear-vid-floor
1.6
vid-layoutList available video mux layouts or set the layout or group to use in the video muxconference <confname> vid-layout list|<layout-name>|group <group_name> [<canvas_id>]
1.6
vid-fpsSet the frames per second rate of the video mux output for the conferenceconference <confname> vid-fps <fps>
1.6
vid-bandwidthSet the bandwidth used by the output video from the conferenceconference <confname> vid-bandwidth <BW>
1.6
vid-write-pngTake a snapshot of the current video mux output and save it to a PNG fileconference <confname> vid-write-png <path>
1.6
vid-canvasSet the canvas to display this member onconference <confname> vid-canvas <member_id>|all|last|non_moderator [<canvas_id>]
1.6
vid-watching-canvasSet the canvas displayed to this memberconference <confname> vid-watching-canvas <member_id>|all|last|non_moderator [<canvas_id>]
1.6

API Additional Notes

dial and bgdial

If the caller id values are not set, the variables in conference.conf.xml will be used. Specifically, the value for caller-id-number will be used for the number and the value for caller-id-name will be used for the name.

If the conference will be dynamically created as a result of this api call (ie this will be the first participant in the conference) - and the caller id name and number is not provided in the api call - the number and name will be "00000000" and "FreeSWITCH". This appears to be unaffected by the variables in conference.conf.xml.

get and set

ParameterDescriptionUnitsWritable
run_time
Seconds
count
Number
max_members
NumberYes
rate
Number
profile_name
String
sound_prefix
StringYes
caller_id_name
StringYes
caller_id_number
StringYes
is_locked
"locked" or blank
endconference_grace_time
SecondsYes
uuid
String
wait_mod
"true" or blank
<action application="set" data="conf_runtime=${conference(${conference_name} get run_time)}"/> <action application="set" data="conf_sounddir=${conference(${conference_name} get sound_prefix)}"/> <action application="set" data="conf_oldsound=${conference(${conference_name} set sound_prefix ${sound_prefix_pl})}"/> <action application="set" data="void_result=${conference(${conference_name} set endconference_grace_time 300)}"/>

list

Command Output

First line:

  • Conference <conference name> (<member_count> member[s][ locked])
    • “locked” - The lock/unlock status of the conference.

Each following line is a comma-separated list for each conference leg with the following items:

  • id of participant(starts at 1 after FS restart)
  • Register string of participant
  • UUID of participant's call leg
  • Caller id name
  • Caller id number
  • Status (hear|speak|talking|video|floor)
    • “hear” - The mute/unmute status of the member.
    • “speak” - The “deaf /undeaf” status of the member.
    • “talking” - The input channel is currently providing some amount of sound energy into conference.
    • “video” - Providing video?
    • “floor” - This member currently owns the floor.
  • Volume In setting
  • Detected energy
  • Volume Out setting
  • Energy Level setting
<extension name="late entry"> <condition field="destination_number" expression="^(300\d)$" /> <condition field="${conference $1 list count}" expression="^\d+"> <action application="conference" data="[email protected]++flags{nomoh}"/> </condition> </extension>

If the conference is not active, the second "condition" test will fail and bypass this dialplan extension entry. If the conference is active the caller will join it with the 'nomoh' flag set.

relate

conference <conf-name> relate <datum-member1>[,<datum-member2>] <related-member1>[,<related-member2>] <relationship>

<datum> is the conference member that is the subject of the relation; in other words, the relationship will be established with respect to <datum>

<related-member> is the conference member that will be affected by the relate command

<relationship> can be one of

  • nohear — the datum member will no longer hear the related-member; i.e. datum's ear will be muted to related-member's mouth
  • nospeak — the datum member will no longer be able to speak to the related-member
  • clear — reset the relationship between the two conference members

Note that "relate 1 2 nohear" gives the same result as "relate 2 1 nospeak", it is simply a matter of perspective.

Examples

conference 3000 relate 1 2 nospeak

Member 1 may now no longer speak to member 2, i.e. member 2 now cannot hear member 1.

conference 3000 relate 1 2 clear

Member 1 may now speak to member 2 again

conference 3100 relate 1 2 nohear

Member 1 now cannot hear member 2

conference 3100 relate 1 2 clear

Member 1 can now hear member 2 again

Both conference-member arguments can take multiple values, separated by commas (but no spaces!)

conference 3300 relate 1,2 3 nospeak

Member 1 and member 2 may now no longer speak to member 3.

conference 3001 relate 1 2,3 nohear

Member 1 now cannot hear member 2 nor member 3

You can subscribe to the following to receive conference events:

conference::maintenance

The "suppress-events" parameter can be added to the conference profile to prevent events from firing. e.g. if you're not interested in start or stop talking events:

<profile name="default"> ...other options... <param name="suppress-events" value="start-talking,stop-talking"/> </profile>

The events that can be suppressed are:

del-member, energy-level, volume-level, gain-level, dtmf, stop-talking, start-talking, mute-member, unmute-member, kick-member, dtmf-member, energy-level-member, volume-in-member, volume-out-member, play-file, play-file-member, speak-text, speak-text-member, lock, unlock, transfer, bgdial-result and floor-change.

Use conference_set_auto_outcall to have mod_conference call endpoints and join them to a conference bridge. To have it call more than one participant, just repeat the conference_set_auto_outcall action in the dialplan for each destination number or address.

<action application="conference_set_auto_outcall" data="dialstring"/>

Example

Here is an example of using conference_set_auto_outcall with some of the other conference_auto_outcall_* parameters to start a conference when someone dials 12345. Extensions 1000 and 1001 will be dialed when the conference starts and will enter the conference muted.

Note that the internal extensions 1000 and 1001 will provide no ringback to avoid polluting the conference bridge with excessive noise, but the gateway is not under control of FreeSWITCH so it will provide early media.

Alternatively, you can set multiple destinations in one line, just remember to escape your variables if you have more than one or any non-escaped chars in it.

<action application="conference_set_auto_outcall" data="['var1=a,var2=b']user/[email protected]$${domain},['var1=c,var2=d']user/[email protected]$${domain}"/>
VariableDescription
conference_auto_outcall_announceFile name of audio message to play to conference member joining conference via the conference_set_auto_outcall application.
conference_auto_outcall_caller_id_nameCaller ID name to use when dialing endpoints using conference_set_auto_outcall application
conference_auto_outcall_caller_id_numberCaller ID number to use when dialing endpoints using conference_set_auto_outcall application
conference_auto_outcall_delimiterDelimiter between values held in cpstr, default = "," (the comma character)
conference_utils_auto_outcall_flagsConference member flags to set for members joining via conference_set_auto_outcall application (use | as separator for multiple flags)
conference_auto_outcall_maxwaitMaximum time in seconds that the channel that initiated the conference_set_auto_outcall will wait for members to join the conference
conference_auto_outcall_prefixString to prepend to each of the dial-string values set from the conference_auto_outcall application
conference_auto_outcall_profileConference profile to use for members joining the conference via the conference_set_auto_outcall application
conference_auto_outcall_timeoutOriginate timeout to use when joining a member to a conference via conference_set_auto_outcall application
conference_auto_outcall_skip_member_beepWhen true, do not play a beep to moderator when initiating an outbound conference call. Tone is 640Hz for 500msec

Read-Only

These channel variables are set by mod_conference.

VariableDescription
conference_last_matching_digitsContains the last matching digits that the member sent into the conference
conference_member_idContains the member_id of the member for the conference they are in
conference_moderatorContains "true" if the channel is connected to a conference as a moderator
conference_nameThe name of the last conference joined by this channel
conference_recordingContains the file name of the conference recording for the conference the channel is connected to
conference_uuidUnique id of the most recent conference in which the channel was a member
last_transferred_conferenceThe name of the last conference this channel was connected to

Settable Channel Variables

Set these channel variables to control the behavior of mod_conference for the current session.  conference_auto_outcall_ variables are described above in the Outbound Conference section

VariableDescriptionExampleMin. Version
Specify which conference control set to use when transferring a caller into a conference<action application="set" data="conference_controls=moderator"/>

Allows the conference security to be overridden. This applies in two different scenarios, one for inbound and one for outbound.

By default, conference security is always applied to inbound calls and is always skipped for outbound calls.

Inbound

<action application="set" data="conference_enforce_security=false"/> <action application="conference" data="3000"/>

Outbound

originate {conference_enforce_security=true}sofia/internal/1001 &conference(3000)

Overrides the conference parameter for enter sound<action application="set" data="conference_enter_sound=silence_stream://10"/>
Overrides the conference parameter for exit sound<action application="set" data="conference_exit_sound=silence_stream://10"/>
A comma separated list of conference flags, which will be applied when the conference is created. This allows to dynamically set the conference flags from the dial plan as opposed to setting them in the conference profile.<action application="set" data="conference_flags=wait-mod,audio-always"/>
A comma separated list of member flags to be set on the channel, which is joining the conference.<action application="set" data="conference_member_flags=mute"/>

This variable controls how the enter and exit sounds interact with the MOH when the  flag is set. When this variable is set to true, the MOH keeps playing and
the enter and exit sounds are mixed with the MOH. When the variable is false or not set, then any playing MOH is first stopped, then the enter or exit sound is played and the MOH is started again.

This functionality is useful in case the enter and exit sounds are used to announce the name of the caller, who is joining or leaving a conference.


Sets the video banner text, font, font size, font color, background color, min/max font size for the member (must be set before entering conference). font_scale valid values float between 5-50, fg/bg hex color code, min/max font size 5-24, all settings besides text are optional.1.6
video_mute_bannerSets the video mute banner text, font, font size, font color, background color, min/max font size for the member (must be set before entering conference). font_scale valid values float between 5-50, fg/bg hex color code, min/max font size 5-24, all settings besides text are optional.1.6
Path to PNG file to use when an avatar image is needed.1.6
Path to PNG file to use when you video mute, by default it will use your last video frame (overrides ).1.6
Path to PNG file for member without video to display (overrides ).1.6

set video replacement avatar images

1.6
Set to the initial canvas member should be displayed on1.6
Set to the initial canvas member should see

1.6

If set to true, it allows the channel to have a dedicated encoder with its own bandwidth settings1.6
Enable recording for this conference.1.6
video_logo_pathset an image logo, and optional text overlay

<action application="set" data="video_logo_path={position=left-bot,text_x=center,center_offset=190,text=#000000:transparent:/var/ccw/font/AEH.ttf:50:'${toupper ${caller_id_name}}',alt_text_x=center,alt_center_offset=190,alt_text_y=88,alt_text=#ffffff:transparent:/path/to/font/AEH.ttf:40:'${caller_id_number}'}/var/png/freeswitch.png"/>


Parameters separated by a comma (,) are as follows:

position=left-top | left-mid | left-bot | center-top | center-mid | center-bot | right-top | right-mid | right-bot
fit=fit-size | fit-scale | fit-size-and-scale | fit-necessary
center_offset=[ 0 or higher ]
text=foreground:background:font_face:font_size:txt
text_x=[ 0 or higher | center ]
text_y=[ 0 or higher ]
alt_text=foreground:background:font_face:font_size:txt
alt_center_offset=[ 0 or higher ]
alt_text_x=[ 0 or higher | center ]
alt_text_y=[ 0 or higher ]

text=  and alt_text= arguments separated by colon (:) are as follows:

foreground     a hex notation color code i.e. #FF0000 for red
background   a hex notation color code or can be the word transparent
font_face       a path to your font face i.e. /usr/share/font/monospace.ttf
font_size       a decimal value, or percentage value i.e.  10 or 10%
txt                  the literal text to display over the image logo

You could potentially use your name in text=x:x:x:x:${caller_id_name} and email, phone, or whatever in alt_text=x:x:x:x:${caller_id_number}
and give alt_text_y slightly different y value so they're not on top of each other...

Note:

These options prepend the path of the file within { } curly braces, also notice video_logo_path and vid-logo-img achieve the same thing

when using these options in dialplan, utilize video_logo_path like this

<action application="set" data="video_logo_path={OPTIONS}/path/to/my.png"/>

when using these options in fs_cli or esl, utilize vid-logo-img like this

conference CONF vid-logo-img 1 {OPTIONS}/path/to/my.png

Just about any format is supported, but currently it must be at the sample rate of the conference (no resampling is done). Since disk is cheaper than CPU, use a wav.

Layout grid is square, its directly proportional to the canvas resolution and aspect ratio.  These layouts will work on 4:3 or 16:9 aspect ratio.  The values x, y, and scale are all of a range of 0-360, proportional to the actual final resolution.

<configuration name="conference_layouts.conf" description="Audio Conference"> <layout-settings> <layouts> <layout name="presenter-overlap-small-top-right"> <image x="0" y="0" scale="360" floor-only="true"/> <image x="300" y="0" scale="60" overlap="true" reservation_id="presenter"/> </layout> <layout name="1up_top_left+5"> <image x="0" y="0" scale="240" floor="true" audio-position="-.5:0:0"/> <image x="240" y="0" scale="120" audio-position=".25:0:0"/> <image x="240" y="120" scale="120" audio-position=".25:0:-.25"/> <image x="0" y="240" scale="120" audio-position="-.5:0:-1"/> <image x="120" y="240" scale="120" audio-position="0:0:-1"/> <image x="240" y="240" scale="120" audio-position=".25:0:-1"/> </layout> <layout name="1x1"> <image x="0" y="0" scale="360" floor="true"/> </layout> <layout name="2x2"> <image x="0" y="0" scale="180"/> <image x="180" y="0" scale="180"/> <image x="0" y="180" scale="180"/> <image x="180" y="180" scale="180"/> </layout> </layouts> <groups> <group name="grid"> <layout>1x1</layout> <layout>2x2</layout> </group> </groups> </layout-settings> <configuration>
Layout ParamDescription
border

Border size around each video feed (in pixels), values from 0 - 50. Overrides video-border-size conference parameter

bgimgFull path to PNG file to use as a background image for the entire canvas
Layout Image ParamDescription
xValue from 0-360, proportional to the canvas size, amount out of 360 for this images top left corner x position
yValue from 0-360, proportional to the canvas size amount out of 360 for this images top left corner y position
scaleValue from 0-360, proportional to the canvas size amount out of 360 for this images size
floorwith floor="true", this box will prefer the video floor holder and will switch as floor holder changes
reservation_idarbitrary named setting used to specify which location to put someone using the vid-res-id api command
overlapwith overlap="true", this box will be redrawn on every frame. You must set this if box overlaps others to avoid flicker
floor-onlywith floor-only="true" this box will only show the video floor holder, and will be blank if there is none

file-only

with file-only="true" this box will only show video files that are played via the conference play api.
audio-position

Uses OpenAL to position the audio into the location of the square the speaker is in.

Values are x:y:z where x, y and z are values on 3d coordinate system.  These can be positive or negative floating point numbers.

The larger the number, the further the “distance”.  X=left/right, Y=forward/backard, X=up/down.  These are the same values used in the conference position command.

auto-3d-position

Automatically position the conference member based on their location in the current layout.
border

Border size around each video feed (in pixels), values from 0 - 50. Overrides <layout> tag

Check this page for layout examples: FreeSWITCH 1.6 Video#Examples

Conference Add Call Example

Sours: https://freeswitch.org/confluence/display/FREESWITCH/mod_conference
FusionPBX - Free, Open Source, Self Hosted VOIP / PBX based on FreeSwitch.

Freeswitch video conference "standard" solution

This paper is written by FreeSWITCH Du JinFang, founder of Chinese community, is in LiveVideoStack The content of the speech shared online is organized into , In detail FreeSWITCH As an open source video conferencing solution, how to open source 、 On the basis of openness , Docking with all kinds of things that can't be modified “ standard ” Video conference terminal 、WebRTC Browser and wechat applet, etc , Meet all kinds of challenges .

 

writing / Du JinFang

Arrangement / LiveVideoStack

​FreeSWITCH Videoconferencing “ standard ” Solution

 

What we call “ standard ” Solution , Not that the solution is standard . But in the process of video conferencing ,FreeSWITCH As a server , Will face different types of clients and various hardware terminals . Because they use a variety of standard protocols , It's something we can't modify , So they're called standard clients . and FreeSWITCH Videoconferencing “ standard ” The solution is a solution to these non modifiable standard clients .

Video conferencing can be roughly divided into three types . One is the traditional video conference , The traditional video conference is “ standard ” Of , Because they need to communicate with each other . Early video conference protocols were generally H323, Even now, we still often encounter H323 The equipment , But then most of them were SIP The device of the agreement replaces .SIP The protocol is a text protocol , The whole is more flexible .

In recent years, there have been some cloud video conferences , This year can also be regarded as the first year of cloud video conference , Because of the epidemic , People are starting to use video conferencing more . for example Zoom, Tencent Conference 、 Little fish is easy to wait , It is said that Tencent conference was launched within a week 10 10000 servers , Emergency expansion , This is impossible in the traditional video conference era , Only in the era of cloud computing can we achieve rapid expansion , This also reflects the advantages of cloud computing . It is understood that these cloud video conferencing vendors , It's basically a private protocol used , The advantage is that it can be optimized without limitation . But private protocols can't do a good job of Interconnection , However, in order to realize the intercommunication with other devices by various video conference manufacturers , I will also provide some of them SDK.

Open source video conferencing , Yes FreeSWITCH、Jitsi、Kurento、Janus、Medooze etc. , These video conferences are also many years old , At present, most of them have begun to support WebRTC. Some support WebRTC Mainly , for example Kurento and Janus;Janus and Medooze Initially it was to support SIP Of , I haven't paid much attention to ;Jitsi Yes WebRTC Our support is very good .

about FreeSWITCH, There may be some misunderstanding .FreeSWITCH In fact, it was first used for audio communication , namely PBX SPC exchange , But actually FreeSWITCH Video conferencing is also very powerful . Open source video conference because it's open source 、 Open , It's open API, So more open protocols like SIP agreement .

at present WebRTC Comparing the fire , All video conferencing devices are basically supported WebRTC, You can make a phone call in the browser .WebRTC It's a media protocol , No signaling is specified , There is no standard at the signaling level , Therefore, we will be more flexible in implementation .FreeSWITCH Two protocols are implemented in the signaling layer , One is SIP, It is carried in WebSocket On , Because there's only... In the browser WebSocket Can achieve two-way communication . In addition, there are private agreements that we have implemented Verto.

​FreeSWITCH Videoconferencing “ standard ” Solution

 

As we use video conferencing software more and more , There's a problem : There are more and more video conferencing clients on mobile phones and computers .

In fact, we also hope to solve this problem , The way is to interconnect . We hope all terminals can be interconnected , Is it just because the video conferencing clients used are different , Can't we have a meeting together ?

​FreeSWITCH Videoconferencing “ standard ” Solution

 

The ideal is full , But the reality is still very difficult to implement .

In fact, it's more because of business , No one would choose to do this . Of course, from a technical point of view , All using private protocols 、 Servers and terminals , Can better optimize 、 Better security and so on . All in all , There is a long way to go to realize interconnection .

It's a long way to go , But we have always wanted to do this ,FreeSWITCH Also connected to a lot of different clients , I hope to be able to connect with more equipment .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

Speaking of FreeSWITCH, Take a brief look at its history .

2006 year FreeSWITCH Released the 1 A version .FreeSWITCH It started as a PBX On going .PBX It's what we call the switch in the enterprise , For phone calls .

2008 Years issued 1.0 edition — Phoenix version , Like Phoenix Nirvana , After countless collapses 、 Optimize , So it's called Phoenix version .

2012 Released in FreeSWITCH1.2 edition (FreeSWITCH The version numbers of are all even numbers ),1.2 The version is very stable , The audio side is also very mature , There is basically nothing to do on the phone . But with WebRTC Appearance ,FreeSWITCH Decide to support WebRTC.

2014 Launched in 2013 1.4 edition , Start supporting WebRTC. In the early WebRTC It's not very stable , but WebRTC The standard of the standard becomes very fast , therefore FreeSWITCH It's been changing all the time .

In fact as early as 2008 I started to do FreeSWITCH 了 , At that time, I mainly did online education , Early online education didn't have videos , Only audio , Teachers use audio to teach English dialogue . And then I did some other projects , Video required , Then I added some video functions , Including video MCU. because FreeSWITCH Early on, it wasn't very mature in terms of video , I was not particularly satisfied with several projects , So we opened up the video part later .

2014 year -2015 year , We spent a year , Merge what you've done into FreeSWITCH In the main branch of , Take a whole year to standardize your own code 、 Simultaneous adaptation Windows、Linux、Unix And other platforms , Realization FreeSWITCH Compilation and support on various platforms , Released 1.6 edition . At this time FreeSWITCH Start supporting video calls and video conferencing , After from 2017 To begin to 2020 year , I've been repairing for years bug, take FreeSWITCH It's getting better and better .

On this branch of open source video conferencing , We've also done a lot of things , Some have been merged into 1.8 In the version , Some merge in 1.10 In the version . In fact, we have maintained a branch of our own without merging into it , Because it takes a lot of labor to merge your own code into open branches , So it will be gradually completed in the follow-up .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

When it comes to standard protocols ,FreeSWITCH Support these standard protocols in the figure above .

First FreeSWITCH Support SIP Signaling , It's the standard protocol for audio and video calls , Support a variety of clients 、 terminal , At present, many conference equipment on the market is supported SIP Of , It can directly realize intercommunication .

Secondly, we made a H323 Module ,FreeSWITCH There are two of them H323 Module , But they don't support video , Because there are customer needs , Then I made a video module , So you can follow H323 To communicate with each other . With the development of mobile Internet , At present, there are many kinds of mobile devices APP Most of them support SIP Signaling , It can directly realize intercommunication .

With WebRTC The development of , A lot of people started to port it to mobile clients .WebRTC The advantage is that you don't have to write your own media layer , With WebRTC Open source brings a lot of features , for instance JitterBuffer、 Echo cancellation 、 Noise reduction and so on WebRTC Already included in , No need to write it yourself , It's not as good as a variety of privatized manufacturers , Because the privatization of the manufacturer can be more extreme optimization . But for an open source project ,WebRTC Well done enough , because WebRTC Only the media layer, no signaling layer , So everybody started to go WebRTC There are all kinds of signaling .

It is worth mentioning that RTMP, In fact, what I did at the beginning was RTMP In the video . Despite the current Flash Basically no one's using it anymore , but RTMP The agreement is still very good , At present, it is more widely used in live broadcast and streaming .

Several ways to realize video conference :Mesh、MCU、SFU

​FreeSWITCH Videoconferencing “ standard ” Solution

 

Video conferencing, in short, has three ways :Mesh、MCU、SFU.

Mesh It is a net structure formed by simple point-to-point connection and does not need a server , But this method is used very little , Because it's not easy to control .

At present, there are two mainstream ways ,MCU and SFU.

MCU The reason why it is more mainstream , Because the first video conferencing equipment was basically MCU Of .MCU There's a server in the middle , The video client communicates directly with the server , In fact, sending and receiving are all the same . The video server combines all the streams together , That is, video fusion screen . Of course, the audio will also fuse , Simplicity , We're just talking about video . The video server fuses the video streams together to form a picture , Distributed to all terminals , All the terminals see the same picture , This situation is called MCU(Multipoint Control Unit), Multi point control unit .

With WebRTC Appearance , A lot of people start using SFU(Selective Forward in Unit).SFU No decoding 、 It doesn't melt the screen , It was said that MCU I'm going to put together all kinds of pictures 、 Fusion screen , Also need to encode and decode the video . and SFU Only need to receive the video and audio from each client , Send it to different people selectively . The advantage is that it doesn't take up too much CPU, But the disadvantage is that it wastes bandwidth . such as 5 Personal calls , One of them just needs to send all the way , The forwarding unit will be arranged , Copy the stream into multiple copies for distribution , Everyone gets a lot of traffic , The pressure on the terminal will be greater , Because one side of the terminal needs to be on the other side of 4 The path stream decodes . The advantage is that the terminal can freely arrange the display styles of other clients received , Everyone can see different pictures .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

Above, MCU The basic principle diagram of , As shown in the picture 4 A camera , Each camera uploads its own image to MCU,MCU Zoom 、 Splicing , Put together a picture , Then send out . Each terminal displays the same screen .FreeSWITCH This is how it works internally ,FreeSWITCH Internal implementation such as 1、2、3、4 And so on , And a canvas (canvas), We will receive different video decodes , Zoom again , To the canvas , Form a flow , And then it was distributed .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

There are many types of canvas styles ,FreeSWITCH In addition to supporting standard canvases :3×3、4×4、8×8 outside , It also supports the training course model : The picture of the speaker ( Big )+ Audience pictures ( Small ), And more different arrangements .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

We have made a key optimization of this , Namely RTP. As we all know, video streaming depends on RTP transmission ,RTP There is also a sister agreement called RTCP, It's a control protocol that controls RTP, There is a thing in this agreement called TMMBR(Temporary Maximum Media Stream Bit Rate Request). In a video conference , Generally speaking, the HD images you see are 720p or 1080p Of , And in speaker mode , The audience's picture is usually small , There's no need to upload 1080P or 720P The picture of , waste 1 Trillion or 2 Megabytes of bandwidth . At this time ,FreeSWITCH Will send a command -TMMBR, No need to upload HD video , Reduce bandwidth upload , In this way, viewers can go online with low bandwidth consumption , Reduce bandwidth pressure on servers 、 It also reduces the pressure of decoding .

We've applied it to our video conferencing , The effect is very obvious . But only if the terminal supports this protocol , stay WebRTC Is supported by , The standard of this agreement is called RFC 5104.RFC 5104 In addition to norms TMMBR outside , There are other agreements , for example FIR、NACK、PLI, It's all about keyframes . In the case of packet loss ,FIR Is to request a keyframe , Because of decoding failure, there will be mosaic , Ask the other party to send a keyframe , Refresh screen .NACK It's a bag loss , In fact, packet loss involves caching , That's what I'm talking about Jitter Buffer,Jitter Buffer It's between two communication terminals , Whether it's the sender or the receiver , There will be one Buffer, What this buffer sends out , It will be put into the buffer to receive , In case of packet loss , Found a packet loss , Ask the other party to resend . At this time , The other party sees that the packet is still in the buffer , The retransmission can be removed from the buffer , It's called NACK.PLI yes Packet Lost Indication, Tell this end that I lost my bag , This protocol is responsible for the quality of audio and video transmission , Because most video transmission uses UDP The agreement is unreliable , So in case of packet loss, more compensation should be made .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

We did some other optimizations , In large-scale video conferencing , Dozens of people, even hundreds of people participated in , For the server decoding pressure will be greater . And it doesn't make sense to show hundreds of people on one screen at the same time , Because it's too small to see the expression of the audience , So in general, when the number of participants is relatively large , Just show a few or dozens of people , Too much will not show , There is no need to decode the parts that are not shown , No need to waste the processing power of the server . But the audience still needs to show it in turn , So we also did a multi person round show , For example, start looking at this now 10 personal , Next time I'll see another 10 personal , And then make a timer to rotate , Give everyone a chance to be on the stage . Of course, because of keyframes , When you show it , You need to decode , You don't have to have a keyframe coming in , So you need to turn on decoding two seconds in advance , And then when it's shown , Generally, this key frame is here , Because each time you show it, you also ask for keyframes , That's exactly how it shows , No black screen . There are also some of the holes we encountered in the video conferencing process , for example NACK Request too much , Sometimes because many terminals have bad networks , Then they all come and ask for NACk Packet loss , If you find this NACK Request too much , We just send keyframes directly , Ignore packet loss .

And there are restrictions FIR The frequency of requests ,FIR This is what we call keyframe request , Refresh keyframes , When the terminal requests a key frame under bad network conditions , If you have any 10 All terminals request keyframes , If within a second 10 Key frames , The bandwidth will be inflated or the video will be very blurred , Because I can't meet so many keyframes , So we did some algorithms , In fact, this is also some basic algorithms , Limit keyframe requests . For example, you can set two or three seconds , So in two or three seconds , Only two keyframes are generated , That is, the first request and the last request will produce keyframes , The rest is ignored , In this case, the bandwidth will be protected , Of course, the other party may have a short screen , But it doesn't matter , And then it's clear in two seconds , Because after all, its network state is not good, it can be tolerated , Of course, if it is a terminal with good network status , He won't have an impact either .

in addition , Different codes have different encoders , FreeSWITCH Support different encoding , For historical reasons ,Chrome Support vp8, Apple's browser only supports H.264, Can't achieve interoperability , And then at the beginning WebRTC We can't communicate with each other , Of course, in recent years ,Chrome Also support H.264 了 , Other browsers also support vp8 了 , So FreeSWITCH From the very beginning, it supports multiple encodings , And then in the same meeting , Meetings with different codes , You can use different encoders .

There are different terminals , Maybe these are private terminals 、 Private communications don't have this kind of trouble , Because the terminals are all his own , Code what you want to code with , But in the open source world , Need to face a variety of clients , For example, military projects , A lot of their equipment is still H.263 Of , Cannot be replaced , We can only adapt H.263,H.263 I won't support it 720p The resolution of the , It only supports CIF Resolution ,CIF That is not 16:9 Neither 4:3, So we need a separate encoder . And so on and so forth , The more models of terminals you want to support, the more you need to do .

We did some optimizations in the early days , Reduce resolution 、 Reduce the frame rate , Found a terminal to use flash It's something we can't handle , So we'll reduce the resolution , Reduce the resolution by half , And then frame down , The original 30 The frame is reduced to 15 frame . Now there's a better technology called SVC,SVC That is, multiple resolutions and multiple frame rates in one encoder , Yes, of course CPU There is still a price , After it's made up, it can be distributed selectively , Actually FreeSWITCH There's a module called mod_openh264, Using Cisco's open source codec , Support SVC, But we haven't used it yet , It just uses its simple encoding and decoding function , We are FreeSWITCH For internal use , The other is internal libx264, It's in FFmpeg Self contained .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

In the video conference, we often encounter a double stream , Traditional video conference equipment ,H.323 The equipment , Generally supported protocols are called H.239, It can support two streams in a call , One stream is the speaker's video , The other is PPT, Two streams can go up to MCU, You can see , Even the other person can play two TVs or projectors , A projector shows the speaker's video , Another projector display PPT.

SIP The device also supports dual streaming ,SIP The dual stream of the device is called BFCP, The full name is Binary Floor Control Protocol. The speaker will do some control ,H.323 We didn't do it , Just a simple docking , Because there are few docking devices , If you can pass it . about SIP Two streams of agreement , We don't have open source yet , It's also BFCP Of . We are directly in the SIP In the module of SDP, Because in SDP There will be two video streams inside , Hijack to process and generate a new call ( A fake call ),FreeSWITCH When you get a call all the way , I saw that he was a double flow call , And then there are two calls , In this case, the two calls will be transferred to the conference at the same time , The conference code doesn't need to be changed . For the conference, there were two calls , But yes. FreeSWITCH It's a call all the way , In this way, we can support double flow .

In addition to WebRTC in , Shuangliu has a name of Simlcast, It can also be in a SDP There are many streams inside , because Simlcast Early instability , There are many problems , Now we use Simlcast Just doing some experiments , There is no detailed code yet , We're just simple and rough , Make two calls directly in the browser , One call is this video of the speaker , Another call is shared desktop , Because it's launched in the browser WebRTC When you call , You can directly choose whether the video source is a camera or a screen, or share an app , The formation of this double flow . It's the same FreeSWITCH, It's still two-way flow , Enter the meeting as two calls .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

For the above mentioned SFU,FreeSWITCH It can be realized SFU Of , In fact, we have tried a lot , however SFU More complicated , Need to control a lot of calls , future FreeSWITCH At the heart of this is to support multiple streams , But now it's implemented SFU It's not on the agenda yet , Another reason is that there are so many things on the market SFU Products , And they are more mature , for example Jitsi、Kurento、Janus There are SFU The implementation of the , If necessary , Collocation FreeSWITCH To implement .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

When video conferencing is large , You need to cascade , Because one server can't support .FreeSWITCH Video conferencing in the laboratory testing a server can support 400 road 720p Video stream of , According to the specific application scenario, the specification of the server is 32 Nuclear or 64 nucleus , Of course, in the context of the conference , It's not going to show everyone , Just encode and decode the people that are shown , The optimization mentioned above does not decode , In this way, only one stream can be encoded and sent , So for CPU The pressure is not great . In fact, we do it in the lab 400 Road flow , But to the customer online is a server support 100 The end of the road , It should be able to go up to 200 road . But the application scenario needs 800 Road customer terminal , So we'll do a cascade .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

There's a problem with cascading , We call it “ Look at each other ”, It's about being two MCU When cascading , for example 1、2、3 In a MCU On ,5、6、7 On the other side MCU On , When MCU When the composite picture enters another picture , It will bring your picture back , So there is an infinite circle in the middle of the picture , Of course, the dynamic diagram in the following figure is more intuitive .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

When doing desktop sharing, I must be familiar with the above interface .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

Actually Google We can find a lot of similar cases on the website , This is the video we received and sent out as a video source again , This is called eye to eye . Video conference in two MCU This happens when you cascade . So what's the solution ?

​FreeSWITCH Videoconferencing “ standard ” Solution

 

FreeSWITCH It implements a function called multi canvas , As shown in the above application scenario , When a person starts a speech , It will be the main venue , Put it on the canvas 1 On . All the participants need to see the main venue , At the same time, the speaker needs to supervise other participants to learn , You need to see pictures of other participants , We put all the pictures of the participants at the 2 On a canvas , People in the main venue can see the participants in the sub venue .

even to the extent that FreeSWITCH One more Super Canvas—— Super canvas , It's mainly used for monitoring , No matter how many canvases there are in the meeting , You can see at a glance . such , When the backstage manager controls the meeting , You can easily see the whole scene of the meeting . In this way , We solved this “ Look at each other ” The problem of , These two MCU You can cascade , For example, up MCU Always on canvas 2 On , The downside MCU Always on the canvas 1 On , So you can tell them apart .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

The figure above is a diagram of a cascade ,Master Is the primary server , There's a lot of FreeSWITCH From the server , The down screen is always at the 1 On a canvas , The up screen is always in the first 2 On a canvas , And vice versa . In this way, you can switch at any time , Check the uplink , All participants can also see the main venue . Sometimes in the course of a meeting , In order to prevent participants from feeling lonely when they only see the main venue , At this point, we will put the 2 The pictures of the canvases are also pushed down , So everyone can see the other participants . Even in the process of a meeting to communicate , For example, raise your hands to speak , At this time, you can also replace the screen of the main venue . This is the basic principle of cascading , But the actual control is still more complicated .

It is worth mentioning that ,FreeSWITCH It's different from some video conferencing now , For example, some meetings can simply support a meeting of hundreds of people , Push the speaker screen to a streaming media server , But the speaker couldn't see the participants . This is actually a live mode , The streams that participants receive are unidirectional , There's only downstream flow .

In some live scenes , Communication and interaction means live broadcast of Lianmai . Due to the large scale of participating users , For example, a live broadcast of 100000 people , When the host wants to interact with the audience , We need to raise the level of this audience , To the anchor class , Then we can broadcast his picture out . This kind of application scenario is basically the same as that of video conference , however FreeSWITCH The meeting is two-way flow from beginning to end .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

In this way , FreeSWITCH You can talk to other people MCU Cascade . Other units already have MCU 了 , FreeSWITCH There are more canvases , You can distinguish the up and down on different canvases . For example, the two on the left in the picture 1、2 yes FreeSWITCH Users of , On the two mobile clients on the right 3、4 It's something else MCU Users of , After going up, in other MCU After fusing the screen, you can send it out .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

FreeSWITCH Except with other things MCU docking 、 Out of the cascade , And some optimizations .

because FreeSWITCH It's open source. , If you want to connect with other video conferencing systems , You need to integrate other video conferencing SDK. For example, we can connect with Tencent conference ( Now the audio is on , The video hasn't been picked up yet ),TRTC Provide multi platform SDK, We wrote a module to put it in FreeSWITCH in ,FreeSWITCH You can communicate with it . adopt PSTN We can dial in to FreeSWITCH in , You can also access Tencent conference directly ,FreeSWITCH It can be used as a gateway . Of course PSTN Video is not supported yet , Only audio support .

The other is Agora Of SDK, We integrated it many years ago Agora Of SDK, Both audio and video can be connected .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

With Agora For example , We are FreeSWITCH There is a module called mod_agrtc, In this way, it can be realized with Agora Interworking of .Agora And WebRTC similar , Only the media layer SDK There is no signaling layer , Therefore, we need to implement a signaling service by ourselves . Of course, most companies do their own APP, Need to register 、 Authentication etc. , There is already a signaling service , So just call FreeSWITCH Of API, You can control the call 、 Recording, etc , Realize interconnection . But some customers don't want to write their own signaling service , Or it's hard to expand your own signaling service . So we also wrote an article based on WebRTC Signaling service for , Integrate on the mobile end Agora Of SDK,FreeSWITCH Integrated in Linux Of SDK, Then we can realize intercommunication .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

There's a problem with this , Whether it's Agora still TRTC, Because of the early SDK It's for the client , So only one client is supported , That's one SDK Only one way calls are supported . So we did another service — IPC, There's a couple of calls right there FreeSWITCH Create several processes in , In this way, it can be realized with FreeSWITCH Interworking of .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

When it comes to wechat applets , Let's talk about Flash. In fact, for Flash Support for has been around for a long time FreeSWITCH To realize ,FreeSWITCH There's a module called mod_rtmp,RTMP The agreement has been implemented . Because wechat applet uses RTMP The agreement , When we first started making wechat apps , I wanted to be in RTMP On the basis of the extension to achieve wechat applet support , But it's not , The original Flash Just one socket And signaling , You can communicate with audio and video , But wechat apps can't .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

Wechat applet provides two components ,Live Publisher as well as Live Player, It's the concept of push flow and pull flow , use RTMP The stream can receive and send . So we introduced SRS Service for (SRS It's also an open source software , It is mainly used for live broadcast platform ), It's called mod_weixin.FreeSWITCH It can be pushed to SRS On , Pull through a small program , On the contrary, wechat applets can be pushed to SRS On ,FreeSWITCH Pull again . In this way, a two-way communication is realized .

because RTMP No signaling , So we realized Websocket Signaling , The applet still goes through Websocket Connected to the FreeSWITCH, In this way, the establishment and connection of the call can be controlled .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

FreeSWITCH And support ASR/TTS, Not native support, of course , But by calling some third-party SDK Realization , In this way, intelligent control can be realized , Even take automatic minutes 、 Automatic translation and so on .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

Realize video conference on public network , Inevitably it involves NAT Crossing the question . about NAT through ,WebRTC Well done , such as ICE The way .FreeSWITCH Currently supported ICE, stay ICE When you can't get through , It can also be used. Stun/TURN The server gets through .

There are also applications like banking , Due to special circumstances, you can't open too many ports . And based on FreeSWITCH Each call requires more than one RTP The port of . So we can use VPN The way , Connect to the server on the public network , It only needs one UDP The port can be realized .

For large-scale video conferencing , We used iptables. For example, we have servers in Beijing and Shanghai , You can do one on the server in Shanghai iptables, Then all the traffic will be transferred to the server in Beijing , In this way, the client can access nearby . Of course, for some more flexible applications , We need to design the corresponding application program to control how to call these iptables Do the forwarding .

besides , We also tried Tencent cloud's CLB( Load balancing ). In fact, Tencent cloud load balancing , Can directly provide a cloud load balancing IP, And then forward it to the back-end service .

Next, we plan to try to use Ali's LVS.

In the present situation , Plus the conference room cascading technology mentioned earlier , In fact, relatively large-scale video conference can be realized .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

The next step is 4G/5G, We've actually tried a lot . At present, the direct use of mobile phones 4G There may be fewer video calls , But some customer service systems in the industry have begun to use , Some customers can call directly , Use 4G Video call to call center , Information exchange . Believe that with the future 5G The development of , Communication technology and capabilities will also become more powerful .

​FreeSWITCH Videoconferencing “ standard ” Solution

 

FreeSWITCH Community official website :https://www.freeswitch.org.cn Welcome to pay attention 、 Communicate together , In the future, we will open source to the end . in addition , in consideration of FreeSWITCH The coverage may be too narrow , We made one RTS Our community —Real-Time Solutions, In the future, it will be right FreeSWITCH Some of the extended code is open source , Into it . Of course, the ultimate goal is to go straight up the stream FreeSWITCH among , But because it has to go through all kinds of code reviews , Reception may be slow , So it will be stored in this warehouse first , Wait for the opportunity to open source to FreeSWITCH The upstream . If you have the conditions 、 If you have the ability, you are welcome to join us .

版权声明
本文为[rpandora]所创,转载请带上原文链接,感谢

Sours: https://chowdera.com/2020/11/20201106222943064s.html

Video conference freeswitch

And the students, having gathered together with Katya in one of the dorm rooms, came up with a funny plan, well, more precisely, it was a joke, they say, one of the students will specifically behave like a victim everywhere bumping into this professor. Katya herself did not remember how she agreed, she only remembers how they drank to her success and dressed her up like a complete whore.

They came up with a wild plan.

[VoIP] Install and configure your Virtual Conferences, with Freeswitch Embedded WebRTC

Graduation class doesn't have much to do. Until the end of the semester and school in general, there was not much time left and the teenager had a lot of. Free time.

You will also like:

Last time. Hope. This is not the eighth year on the calendar.



357 358 359 360 361