Coturn fails to start (fresh install on Ubuntu 20.04)

[SECtim]

On a fresh Ubuntu 20.04, I ran (with bbb-install from the 2.7.x branch)

sudo ./bbb-install.sh -c turn.<myurl>:<mysecret> -e <myemail>

and it installed all available updates etc., certbot, and coturn (as expected).
However, the output then ends with these lines:

Job for coturn.service failed because a timeout was exceeded.
See "systemctl status coturn.service" and "journalctl -xe" for details.
./bbb-install.sh: line 1735: /etc/bigbluebutton/turn-stun-servers.xml: No such file or directory
chown: invalid group: ‘root:bigbluebutton’
chmod: cannot access '/etc/bigbluebutton/turn-stun-servers.xml': No such file or directory

I of course tried to run the script again after a reboot, with the same result. Manually starting coturn also results in a timeout (after 90s, i.e., plenty of time).

Here's the turnserver log with redacted IP (/var/log/turnserver/turnserver.log), it contains some warnings related to the configuration (which I did not touch):

0: log file opened: /var/log/turnserver/turnserver.log
0: Listener address to use: <serverIP>
0: Relay address to use: <serverIP>
0: Bad configuration format: no-loopback-peers
0: Black listing: 0.0.0.0-255.255.255.255
0: Black listing: ::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
0: White listing: <serverIP>
0: Bad configuration format: no-loopback-peers
0: 
RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
Version Coturn-4.5.1.1 'dan Eider'
0: 
Max number of open files/sockets allowed for this process: 1048576
0: 
Due to the open files/sockets limitation,
max supported number of TURN Sessions possible is: 524000 (approximately)
0: 

==== Show him the instruments, Practical Frost: ====

0: TLS supported
0: DTLS supported
0: DTLS 1.2 supported
0: TURN/STUN ALPN supported
0: Third-party authorization (oAuth) supported
0: GCM (AEAD) supported
0: OpenSSL compile-time version: OpenSSL 1.1.1f  31 Mar 2020 (0x1010106f)
0: 
0: SQLite supported, default database location is /var/lib/turn/turndb
0: Redis supported
0: PostgreSQL supported
0: MySQL supported
0: MongoDB is not supported
0: 
0: Default Net Engine version: 3 (UDP thread per CPU core)

=====================================================

0: Domain name: 
0: Default realm: 
0: 
CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time.
Be aware that you could not mix the username/password and the shared secret based auth methohds. 
Shared secret overrides username/password based auth method. Check your configuration!
0: 
CONFIG ERROR: Empty cli-password, and so telnet cli interface is disabled! Please set a non empty cli-password!
0: 
CONFIGURATION ALERT: you did specify the long-term credentials usage
 but you did not specify the default realm option (-r option).
		Check your configuration.
0: WARNING: cannot find certificate file: turn_server_cert.pem (1)
0: WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
0: WARNING: cannot find private key file: turn_server_pkey.pem (1)
0: WARNING: cannot start TLS and DTLS listeners because private key file is not set properly
0: IO method (main listener thread): epoll (with changelist)
0: WARNING: I cannot support STUN CHANGE_REQUEST functionality because only one IP address is provided
0: Wait for relay ports initialization...
0:   relay <serverIP> initialization...
0:   relay <serverIP> initialization done
0: Relay ports initialization done
0: IO method (general relay thread): epoll (with changelist)
0: turn server id=0 created
0: IPv4. SCTP listener opened on : <serverIP>:3478
0: IPv4. TCP listener opened on : <serverIP>:3478
0: IO method (general relay thread): epoll (with changelist)
0: turn server id=1 created
0: IPv4. SCTP listener opened on : <serverIP>:3478
0: IPv4. TCP listener opened on : <serverIP>:3478
0: IPv4. UDP listener opened on: <serverIP>:3478
0: Total General servers: 2
0: IO method (admin thread): epoll (with changelist)
0: IO method (auth thread): epoll (with changelist)
0: IO method (auth thread): epoll (with changelist)
0: SQLite DB connection success: /var/lib/turn/turndb

I am of course happy to provide additional information if needed.

[SECtim]

I now also tried manually creating the /etc/bigbluebutton directory to get rid of the shell error, ran bbb-install again, and it created the config file; however, chown of course still fails as there is no bigbluebutton group. So I tried changing that to chown root:turnserver "$TURN_XML".
This got rid of all shell errors when running bbb-install, but coturn still refuses to start. However, I think these lines should not be executed at all when setting up a standalone TURN server?

Nevertheless, I was now able to identify the root cause of coturn not starting: /etc/systemd/system/coturn.service.d/override.conf sets an option --pidfile= which had no value, i.e., contained the line

ExecStart=/usr/bin/turnserver -c /etc/turnserver.conf --pidfile= --no-stdout-log --simple-log --log-file /var/log/turnserver/turnserver.log

I manually set the value of pidfile to /run/turnserver/turnserver.pid (as recommended in the BBB docs, and I also added --daemon), and with that, coturn now starts.
I suspect that I now also have to manually adjust some more things to get it to work with BBB.

[SECtim]

I ended up just following the steps in https://docs.bigbluebutton.org/administration/turn-server/, now everything works as expected.

It seems like the bbb-install script in its current form is not suitable to install TURN on a separate server? If so, then the README should be adapted.

[CamZie]

@SECtim I have the same issue installing TURN server on a separate server.

Did you just manually install the coturn server using the guide or did you use the latest bbb-install.sh script?

[SECtim]

@CamZie I installed using the install script, coturn didn't work. So I followed the manual installation instructions to double-check and found that several of the configuration steps noted in the manual instructions are not done (or done differently) by the install script.
I'd recommend not using the install script at all and instead follow the manual instructions (at least based on my experience back in November, maybe the script was fixed).

[ffdixon]

I'm the primary author of bbb-install.sh

Nevertheless, I was now able to identify the root cause of coturn not starting: /etc/systemd/system/coturn.service.d/override.conf sets an option --pidfile= which had no value, i.e., contained the line

ExecStart=/usr/bin/turnserver -c /etc/turnserver.conf --pidfile= --no-stdout-log --simple-log --log-file /var/log/turnserver/turnserver.log
I manually set the value of pidfile to /run/turnserver/turnserver.pid (as recommended in the BBB docs, and I also added --daemon), and with that, coturn now starts.
I suspect that I now also have to manually adjust some more things to get it to work with BBB.

Thanks for this investigation. --pidfile= doesn't look right, but it has been that way since originally added, so maybe something changed in coturn to be more specific.

Nonetheless, I'll fix this shortly and test so it has --pidfile=/run/turnserver/turnserver.pid.

If any of you are deep into the docs and can see items that need updating, take a moment and send in a pull request. It will save us some time as we update them.

[CamZie]

@CamZie I installed using the install script, coturn didn't work. So I followed the manual installation instructions to double-check and found that several of the configuration steps noted in the manual instructions are not done (or done differently) by the install script. I'd recommend not using the install script at all and instead follow the manual instructions (at least based on my experience back in November, maybe the script was fixed).

Thanks for the help! I tried however installing it manually, also tried installing it via script and manually adapted the missing changes as you did however TURN server is somehow not being detected by our BBB installation. Did you install it on the same server as BBB or on a separate server? Would it be alright if you could send how your turnserver.conf and turn-stun-servers.xml looks like, so that I could compare? Thanks a lot in advance!

[SECtim]

First of all, make 100% sure your firewall settings for both the BBB and the TURN server are correct.

That said, turn-stun-servers.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

    <!-- 
         We need turn0 for FireFox to workaround its limited ICE implementation.
         This is UDP connection.  Note that port 3478 must be open on this BigBlueButton
         and reachble by the client.

         Also, in 2.5, we previously defined turn:$HOST:443?transport=tcp (not 'turns') 
         to workaround a bug in Safari's handling of Let's Encrypt. This bug is now fixed
         https://bugs.webkit.org/show_bug.cgi?id=219274, so we omit the 'turn' protocol over
         port 443.
     -->
    <bean id="turn0" class="org.bigbluebutton.web.services.turn.TurnServer">
        <constructor-arg index="0" value="<TURN Secret>"/>
        <constructor-arg index="1" value="turn:<TURN Server Domain>:3478"/>
        <constructor-arg index="2" value="86400"/>
    </bean>
    <bean id="turn1" class="org.bigbluebutton.web.services.turn.TurnServer">
        <constructor-arg index="0" value="<TURN Secret>"/>
        <constructor-arg index="1" value="turns:<TURN Server Domain>:443?transport=tcp"/>
        <constructor-arg index="2" value="86400"/>
    </bean>
    
    <bean id="stunTurnService"
            class="org.bigbluebutton.web.services.turn.StunTurnService">
        <property name="stunServers">
            <set>
            </set>
        </property>
        <property name="turnServers">
            <set>
                <ref bean="turn0"/>
                <ref bean="turn1"/>
            </set>
        </property>
    </bean>
</beans>

turnserver.conf:

listening-port=3478
tls-listening-port=443

listening-ip=<TURN Server IP>
relay-ip=<TURN Server IP>

min-port=32769
max-port=65535
verbose

fingerprint
lt-cred-mech
use-auth-secret
static-auth-secret=<TURN Secret>
realm=<My Realm>

cert=/etc/turnserver/fullchain.pem
pkey=/etc/turnserver/privkey.pem
# From https://ssl-config.mozilla.org/ Intermediate, openssl 1.1.0g, 2020-01
cipher-list="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
dh-file=/etc/turnserver/dhp.pem

keep-address-family

no-cli
no-tlsv1
no-tlsv1_1

# Block connections to IP ranges which shouldn't be reachable
no-loopback-peers
no-multicast-peers


denied-peer-ip=0.0.0.0-255.255.255.255
denied-peer-ip=127.0.0.0-127.255.255.255
denied-peer-ip=::1
# Private (LAN) addresses
# If you are running BigBlueButton within a LAN, you might need to add an "allow" rule for your address range.
# IPv4 Private-Use
denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.168.0.0-192.168.255.255
# Other IPv4 Special-Purpose addresses
denied-peer-ip=100.64.0.0-100.127.255.255
denied-peer-ip=169.254.0.0-169.254.255.255
denied-peer-ip=192.0.0.0-192.0.0.255
denied-peer-ip=192.0.2.0-192.0.2.255
denied-peer-ip=198.18.0.0-198.19.255.255
denied-peer-ip=198.51.100.0-198.51.100.255
denied-peer-ip=203.0.113.0-203.0.113.255
# IPv6 Unique-Local
denied-peer-ip=fc00::-fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
# IPv6 Link-Local Unicast
denied-peer-ip=fe80::-febf:ffff:ffff:ffff:ffff:ffff:ffff:ffff
# Other IPv6 Special-Purpose assignments
denied-peer-ip=::ffff:0:0-::ffff:ffff:ffff
denied-peer-ip=64:ff9b::-64:ff9b::ffff:ffff
denied-peer-ip=64:ff9b:1::-64:ff9b:1:ffff:ffff:ffff:ffff:ffff
denied-peer-ip=2001::-2001:1ff:ffff:ffff:ffff:ffff:ffff:ffff
denied-peer-ip=2001:db8::-2001:db8:ffff:ffff:ffff:ffff:ffff:ffff
denied-peer-ip=2002::-2002:ffff:ffff:ffff:ffff:ffff:ffff:ffff

allowed-peer-ip=<BBB Server IP>

[CamZie]

@SECtim thanks a lot for your help! We confirm that it works now with all your suggestions and manual changes.

[ffdixon]

@CamZie, did you apply these updates to coturn installed on BigBlueButton, or to an external turn server?

Also, I did a clean install of 2.7.6 and I'm not seeing these errors.

Job for coturn.service failed because a timeout was exceeded.
See "systemctl status coturn.service" and "journalctl -xe" for details.
./bbb-install.sh: line 1735: /etc/bigbluebutton/turn-stun-servers.xml: No such file or directory
chown: invalid group: ‘root:bigbluebutton’
chmod: cannot access '/etc/bigbluebutton/turn-stun-servers.xml': No such file or directory

I noticed in your initial post it had version 4.5.1.1

Version Coturn-4.5.1.1 'dan Eider'

The version installed by the latest build is 4.6.2

0: (72209): INFO: Coturn Version Coturn-4.6.2 'Gorst'

[ffdixon]

Regarding

I manually set the value of pidfile to /run/turnserver/turnserver.pid

I checked a clean install and there is no /run/turnserver/turnserver.pid. You might have encountered this error in the previous 4.5.1.1 version: coturn/coturn#683.

[CamZie]

@CamZie, did you apply these updates to coturn installed on BigBlueButton, or to an external turn server?

We did this on an external turn server. These are the steps I did:

1. Created the /etc/bigbluebutton to avoid errors as temporarily as suggested here: issuecomment-1790330434
2. Re-install TURN server using the latest bbb-install.sh script
3. Applied the systemd changes as mentioned here: issuecomment-1790330434
4. Apply the changes in the file turn-stun-servers.xml and turnserver.conf as suggested by SECtim issuecomment-1985635475 or by the offfical BBB TURN server guide
5. Make sure that the firewall ports are correct TURN server ports. I found out that they have changed the UDP relay port range used by coturn to connect to BBB from 16384 - 32768 (used by version 2.3) to 32768 - 65535.