More fun on the project this week!
During final acceptance testing we noticed that the voice quality on calls when in GVP was degraded depending upon the DDI number dialled. Voice quality when in-queue (Stream Manager) and when connected to an advisor remained fine.
Diagnosis tracked the difference between DDI numbers to the Avaya MEDPRO card (IP Resources) that the call had landed on. We observed the following:
1) Calls arriving on a TN2302AP card were always OK
2) Calls arriving on a TN2602AP card running firmware 46 were always OK
3) Calls arriving on a TN2602AP card running firmware 49 often has poor voice quality
We found that this issue has also been reported by other customers (http://www.avayausers.com/showthread.php?p=40786).
To further isolate the problem we obtained some Wireshark traces of the RTP stream out of GVP. These traces showed that whilst Jitter was constant at 7-8ms the delta between RTP packets was not. We could also attribute these gaps into the RTP stream to HTTP page fetches between recorded audio segments.
Therefore out first conclusion was that this must be a Genesys problem and MEDPRO firmware 49 must be less tolerant than firmware 46 to gaps and variation in the RTP stream.
Since we use a G.711 codec (PCM) for the call, there should be one packet sent every 20 milliseconds. Therefore, in ideal conditions the Max Delta (ms) value should be pretty close to that. In this case the Jitter value also should be close to 0.
Jitter is a smoothed derivative of the interarrival delta. So it will not get nearly as high as the deltas itself, unless fluctuations of deltas are very frequent and of high amplitude over a longer period of time. According to RFC 3550, Interarrival Jitter is supposed to be: “The interarrival jitter J is defined to be the mean deviation (smoothed absolute value) of the difference D in packet spacing at the receiver compared to the sender for a pair of packets.”
The issue we have is that the RTP packets are not sent at a constant rate (ideally every 20ms). In the real world the delta of 220ms on packet 420 is an issue because it causes the voice quality of the stream as heard on the Avaya end to deteriorate.
But hang on. Why would you want to send RTP packets of silence between recorded announcements and use up additional data network bandwidth just to keep the stream constant and the MEDPRO happy?
This brings us onto the subject of ‘Talkspurts’ and the RTP packet marker bit, the use of which is defined by RTP profiles such as the RFC 3551: RTP Profile for Audio and Video Conferences with Minimal Control (aka RFC 3551) and RFC 3389: RTP Payload for Comfort Noise.
Here is what RFC 3389 has to say about detecting Silence Suppression:
RTP allows discontinuous transmission (silence suppression) on any audio payload format. The receiver can detect silence suppression on the first packet received after the silence by observing that the RTP timestamp is not contiguous with the end of the interval covered by the previous packet even though the RTP sequence number has incremented only by one. The RTP marker bit is also normally set on such a packet.
The market bit is intended to allow significant events such as frame boundaries to be marked in the packet stream. An RTP profile may define additional marker bits or specify that there is no marker bit by changing the number of bits in the payload type field. The Marker bit indicates the beginning of a talkspurt. Beginning of talkspurts are good opportunities to adjust the playout delay at the receiver to compensate for differences between the sender and receiver clock rates as well as changes in the network delay jitter. Packets during a talkspurt need to be played out continuously although listeners generally are not sensitive to slight variations in the durations of a pause.
In our traces we see the marker is set true at start of new recorded segment as we would expect:
So it must be an Avaya problem!
But hang on. Maybe it’s just a standard interoperability between Genesys and whatever telephony platform that we deal with all the time. Let’s re-trace what we know.
In the Genesys world Silence Suppression can be configured both on SIP server and in GVP (IPCS) options.
For SIP server the option is ‘silence-supression’ which can be set to ‘true’, ‘false’ or ‘null’. If the value of the option is ‘null’ or not specified the use of silence suppression is negotiated between the two endpoints:
In IPCS the following options can be set in EMPS and pushed out in the GVP.ini configuration file:
- Enable the new G711 silence detection algorithm [g711sildetecalgorithm] (default 1)
- Energy level for the new G711 silence detection algorithm [energylevel] (default 1)
- Variation level for the new G711 silence detection algorithm [variationlevel] (default 1)
- Min Silence Threshold [minsilencethreshold] (default 6)
- Max Silence Threshold [maxsilencethreshold] (default 32)
- Timeout value in ms before silence filing [silencefilltimeout] (default 100)
In the Avaya world silence suppression is configured on each IP Codec set:
In addition, a review of Avaya release notes for TN2602AP (MEDPRO) cards also reveals some VoIP options that can be configured on the card itself. These parameters seem to be engineering parameters. For example:
- Parameter 17: Echo Canceller Alternate Comfort Noise Option
- Parameter 18: Echo Canceller DTMF Transparency Option
- Parameter 19: Echo Canceller Heavy Double Talk Option
- Parameter 65: Maximum level of Comfort Noise Generator
One thing that jumped out in the release notes was the following statement relating to Firmware 46: “FW46 introduces the ability to limit the maximum level of Comfort Noise Generator (CNG) via a new tunable VOIP parameter, Parameter 65”.
During an active call, a certain amount of comfort noise is generated by the gateway so that the user does not assume that the far-end party has disconnected during silent periods of the conversation. The volume of this comfort noise is based on the ambient noise on the line. When excessive noise at 60 Hz and its harmonics is present, the volume of the comfort noise could be perceived as loud. Since the user can easily distinguish between the comfort noise and the hum, he perceives it as bursts of white noise when the other person is not speaking.
To change the ceiling, parameter 65 can be used to set the maximum to a lower value. For example a value of 100 is approx -50dBm, which is a reasonable compromise that provides adequate comfort noise, yet not so loud to be disconcerting. If hiss is an issue, we recommend initially setting parameter 65 to a value of 100. A low non-zero value is preferred as a value of zero essentially shuts comfort noise off altogether.
These parameters can be changed by Telnet-ing to the card itself and executing the following commands:
setVoipParam 65, 100
So putting all the pieces together I am sure that our problem is related to silence suppression and the maximum level of Comfort Noise Generator (CNG) configured on the Avaya MEDPRO. I’m off to make some changes and my starting point will be to compare the default value of all VoIP parameters on each MEDPRO at both firmware level 46 and firmware level 49.
PS: Whilst I’m on the subject of MEDPRO firmware there we also some interesting notes in firmware 51 which may be related to my DTMF problems that I blogged about earlier!
- Previously, some badly formed incoming DTMF digits would not be detected. FW51 greatly improves DTMF detection.
- The echo canceller has been modified to allow tuning to help eliminate double-talk (both ends “speaking” at the same time) induced corruption of DTMF digits. This improves IVR performance
- Previously, in certain configurations, the RTP-Event DTMF marker bit on TN2602 was not being sent. This now adheres to RFC 2833.