UmTRX Networking Issues

After my initial success getting OpenBTS up and running with a UmTRXv2 hardware Transceiver I seem to have gone backwards while trying to optimise my data network with a bit of Linux tuning and swapping out network hubs and cables!

The OpenBTS transceiver process communicates with UmTRXv2 using UDP over a LAN connection and hence network performance is vital in terms of GSM network stability. One of the most common causes of lost UDP datagrams is an undersized receive buffer on the socket. Hence I implemented some standard Linux tuning of buffers and made the changes permanent in /etc/sysctl.conf:

sysctl -w net.core.rmem_max=50000000
sysctl -w net.core.rmem_max=50000000
sysctl -w net.core.wmem_max=524288
sysctl -w net.core.wmem_max=524288

Image

The Ethernet Chip in UmTRXv2 is LSI 1GbE PHY (ET1011C2-CI-DT) which supports 10Mbps/100Mbps/1Gbps. However, the Ettus UHD framework adopted for the UmTRX only supports GigE(1Gbps) although this may be changing. Hence the UmTRX needs to be connected to a switch that supports 1000baseT.

For my OpenBTS server I am using a Shuttle XS35-703 V2 in which the Ethernet Chip is a JMicron JMC250 and *should* support 1000baseT. After a kernel update and much playing about with JME drivers I could find no way of getting a 1000baseT link established. I even tried with auto negotiation off:

ethtool -s eth1 speed 1000 duplex full autoneg off

After much digging I found out the following: “There are two known 1000baseT link establishment issues with JMC25x. If the full mask revision number of JMC25x controller is less than or equal to 4 and the link partner enabled the IEEE 802.3az Energy Efficient Ethernet feature, the controller will not be able to establish a 1000baseT link. The known workaround for these issues is to force manual link configuration with 100baseTX instead of relying on auto-negotiation. The full mask revision number of controller can be checked with the verbose kernel boot option. Use the lower nibble of the chip revision number to get the full mask revision of the controller.”

And guess what – my Shuttle has a version 3 chip:

>> jme 0000:02:00.5: eth0: JMC250 Gigabit Ethernet chipver:23 pcirev:3

Time to give up (for now) and workout what has changed to the effect that I could not longer get the OpenBTS transceiver to start reliably and except for some short lived success straight after a UmTRX power cycle always got one of two errors:

terminate called after throwing an instance of ‘uhd::runtime_error’
what(): RuntimeError: no control response
EMERG 1101519168 OpenBTS.cpp:134:startTransceiver: Transceiver quit with status
6. Exiting.

Or:

EMERG 1104513344 OpenBTS.cpp:134:startTransceiver: Transceiver quit with status 256. Exiting.

Running ./transceiver on its own always worked fine. I even reset the OpenBTS configuration back to the example configuration just in case there was something if there causing problems (which I have seen) – no success.

Surely this had to be something to do with network configuration?

  • Network cables swapped out – no success
  • Changed from TRENDnet TEG-S8g to TP-Link TL-SG1005D switch – no success
  • Direct cable from server to UmTRX – did not establish link since damned JMicron JMC250 cannot establish a 1000baseT link

Time to dig deeper (and send a call help out on the UmTRX mailing list) …

Looking more closely I notice framing errors showing on the interface:

Image

Not being a networking person it was time to do a bit of research – what does “frame” really mean? The answer is: “RX errors mean that your NIC is receiving malformed frames from the transmitting switchport. Frame errors mean CRC failures on receipt of a frame. The root cause of this could be a bad cable, or a bad interface on either the machine or the switch.”

Duh! – that’s what I suspected and have already swapped over cables and switch. What other tangent can I go off on now … I know “Jumbo Frames”!

Jumbo frames are Ethernet frames with more than 1500 bytes of payload. Many Gigabit Ethernet switches and Gigabit Ethernet network interface cards support jumbo frames. Some Fast Ethernet switches and Fast Ethernet network interface cards also support jumbo frames. Most modern Linux distros support frames larger than 1500 bytes. This can improve the performance.
The JMC250 also supports Jumbo Frames (up to 9216 bytes), which can be configured via the interface MTU setting. Mmm … sounds interesting but the largest packet I can see in a Wireshark trace is 1514 bytes:

Image

Still, seems worth a go:

ifconfig eth1 mtu 9000

;-( – still no success.

UPDATE 1:

Alexander replies (again) on the UmTRX mailing list: “We temporarily solved issues with this JMC chip by putting a switch between it and UmTRX, but not all models of switches worked and even after that stability was not perfect.”

UPDATE 2:

Reconfigured my network and it works (sort of). UmTRX now hanging off TP-Link TL-SG1005D Gigabit switch and Shuttle XS35-703 V2 running OpenBTS hanging off Netgear FS605 10/100Mbps switch.

Root cause: JMicron JMC250 connecting to a Gigabit switch and the link partner enabled the IEEE 802.3az Energy Efficient Ethernet feature

I’m still getting (lots) of framing errors which is not good.

Next step – Since I’m limited on what I can install in the Shuttle I’m waiting for Mr. Amazon to deliver a USB 2.0 to 10/100/1000 Gigabit Ethernet LAN Wired Network Adapter based on ASIX AX88178 Chipset which has Linux drivers. I know USB 2.0 supports theoretical speeds up to 480Mbs which is not Gigabit speeds but lets see …

PS: Thanks again to Alexander and Stephane on the mailing list for their help.

 

Share