Time for a new car!

I thought it was about time for a new car so goodbye to my trusty old Proton that has gone somewhere to be made into tin cans!

IMG00071-20091127-1452

In all seriousness I will never knock Protons again. This one was 14 years old and eventually covered 80,000 mile. It failed last years MOT on a brake light bulb. That’s all. Cost next to nothing to run and started every single time.

Share

SIP Trunk Optimisation (Revisited

As I mentioned in a previous post we have an ongoing issue with SIP channels remaining in use when a call is transferred back to the Avaya switch from the Genesys switch after in-queue treatments via Stream Manager. This can be seen on the Avaya Trunk utilisation report shown below for the SIP trunks between Avaya and Genesys:

Image

Not only was this causing Avaya SIP channel licenses to be used unnecessarily, there were also knock-on implications during SIP Server failover. We also identified a problem which was causing Verint voice recording to fail because the Trunk ID in CTI events when delivered to an Advisor contained the SIP trunk ID rather than the ISDN trunk ID that the call was actually on and therefore Verint could not initiate recording of the call.

We have at last fixed this problem. There are two parts to the fix:

  • On the SIP trunk from Genesys to Avaya (trunk 82 in our case) set the option ‘oosp-transfer-enabled’ to true. With this option, when set to true, SIP Server puts itself in the Out Of Signaling Path (OOSP) after the single-step transfer or routing to the external destination has been completed:

Image

  • Transfer calls back to External Routing Points (ERP) on the Avaya switch via an intermediate VDN which is not monitored by Genesys. To do this, we modified the access code on each of our Avaya ERPs. In the example shown below, 45100 is an Avaya ERP which is monitored by Genesys and 47100 is an intermediate VDN which is not monitored by Genesys. The access number ’82 47100′ forces the call to be routed to this unmonitored VDN via SIP trunk 82.

Image

  • The intermediate VDN (47100) is associated with a vector that routes the call to the real ERP (45100) using a route-to 45100 step. In this way standard Genesys ISCC functionality works e.g. attached data is preserved.

The diagram below shows the end to end call flow:

Image

Share

Pilot Go Live

Our pilot project eventually went live this Monday (16/11/2009). It has been quite an effort to get this far with many technical issues along the way (mainly SIP interworking with Genesys GVP and Stream Manager and Virtual Hold integration issues).

Everything looks fine so far.

Congratulations to all of the team.

Share

More on IPCS and Silence Supression

Further to my previous post, Genesys have confirmed that it is not possible to disable silence supression with GVP / IPCS 7.6. I had already found this out in the following post (http://www.sggu.com/smf/index.php/topic,4811.0.html) which Genesys support then usefully quoted back to me as the solution!

Regardless, I still think that we have an Avaya MEDPRO problem relating to firmware FW49 and plan to test the latest FW51 as soon as possible.

According to RFC 3389, since the ability to suppress silence is one of the primary motivations for using packets to transmit voice, the RTP header carries both a sequence number and a timestamp to allow a receiver to distinguish between lost packets and periods of time when no data was transmitted. Discontiguous transmission (silence suppression) MAY be used with any audio payload format. Receivers MUST assume that senders may suppress silence unless this is restricted by signaling specified elsewhere. (Even if the transmitter does not suppress silence, the receiver should be prepared to handle periods when no data is present since packets may be lost.)

A Silence suppression attribute is specified in RFC 3108 – the Session Description Protocol (SDP):

Details

RFC 3108 ATM SDP May 2001
5.6.3.2 The ‘silenceSupp’ attribute

When present, the ‘silenceSupp’ attribute is used to indicate the use
or non-use of silence suppression. The format of the ‘silenceSupp’
media attribute line is as follows:

a=silenceSupp: <silenceSuppEnable> <silenceTimer> <suppPref> <sidUse>
<fxnslevel>
If any of the parameters in the silenceSupp media attribute line is
not specified, is inapplicable or is implied, then it is set to “-“.

m=audio 8804 RTP/AVP 18 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=yes
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=silenceSupp:off – – – -\r\n

It is not quite clear from clear from RFC3108 if silenceSupp has to be negotiated for unicast connections or it can be different in each direction? For example, can caller say “silenceSupp:on” and this will turn on silence suppression on called side and then called can say “silenceSupp:off” to receive voice without silence suppression?

Regardless, we do not see silenceSupp being specified in the SIP INVITE message from Avaya CM:

Image

Share

Voice Quality, Jitter, Silence Supression and ‘Talkspurts’

More fun on the project this week!

During final acceptance testing we noticed that the voice quality on calls when in GVP was degraded depending upon the DDI number dialled. Voice quality when in-queue (Stream Manager) and when connected to an advisor remained fine.

Diagnosis tracked the difference between DDI numbers to the Avaya MEDPRO card (IP Resources) that the call had landed on. We observed the following:

1) Calls arriving on a TN2302AP card were always OK
2) Calls arriving on a TN2602AP card running firmware 46 were always OK
3) Calls arriving on a TN2602AP card running firmware 49 often has poor voice quality

We found that this issue has also been reported by other customers (http://www.avayausers.com/showthread.php?p=40786).

To further isolate the problem we obtained some Wireshark traces of the RTP stream out of GVP. These traces showed that whilst Jitter was constant at 7-8ms the delta between RTP packets was not. We could also attribute these gaps into the RTP stream to HTTP page fetches between recorded audio segments.

Image

Image

Therefore out first conclusion was that this must be a Genesys problem and MEDPRO firmware 49 must be less tolerant than firmware 46 to gaps and variation in the RTP stream.

Since we use a G.711 codec (PCM) for the call, there should be one packet sent every 20 milliseconds. Therefore, in ideal conditions the Max Delta (ms) value should be pretty close to that. In this case the Jitter value also should be close to 0.

Jitter is a smoothed derivative of the interarrival delta. So it will not get nearly as high as the deltas itself, unless fluctuations of deltas are very frequent and of high amplitude over a longer period of time. According to RFC 3550, Interarrival Jitter is supposed to be: “The interarrival jitter J is defined to be the mean deviation (smoothed absolute value) of the difference D in packet spacing at the receiver compared to the sender for a pair of packets.”

The issue we have is that the RTP packets are not sent at a constant rate (ideally every 20ms). In the real world the delta of 220ms on packet 420 is an issue because it causes the voice quality of the stream as heard on the Avaya end to deteriorate.

But hang on. Why would you want to send RTP packets of silence between recorded announcements and use up additional data network bandwidth just to keep the stream constant and the MEDPRO happy?

This brings us onto the subject of ‘Talkspurts’ and the RTP packet marker bit, the use of which is defined by RTP profiles such as the RFC 3551: RTP Profile for Audio and Video Conferences with Minimal Control (aka RFC 3551) and RFC 3389: RTP Payload for Comfort Noise.

Here is what RFC 3389 has to say about detecting Silence Suppression:

RTP allows discontinuous transmission (silence suppression) on any audio payload format. The receiver can detect silence suppression on the first packet received after the silence by observing that the RTP timestamp is not contiguous with the end of the interval covered by the previous packet even though the RTP sequence number has incremented only by one. The RTP marker bit is also normally set on such a packet.

The market bit is intended to allow significant events such as frame boundaries to be marked in the packet stream. An RTP profile may define additional marker bits or specify that there is no marker bit by changing the number of bits in the payload type field. The Marker bit indicates the beginning of a talkspurt. Beginning of talkspurts are good opportunities to adjust the playout delay at the receiver to compensate for differences between the sender and receiver clock rates as well as changes in the network delay jitter. Packets during a talkspurt need to be played out continuously although listeners generally are not sensitive to slight variations in the durations of a pause.

In our traces we see the marker is set true at start of new recorded segment as we would expect:

Image

So it must be an Avaya problem!

But hang on. Maybe it’s just a standard interoperability between Genesys and whatever telephony platform that we deal with all the time. Let’s re-trace what we know.

In the Genesys world Silence Suppression can be configured both on SIP server and in GVP (IPCS) options.

For SIP server the option is ‘silence-supression’ which can be set to ‘true’, ‘false’ or ‘null’. If the value of the option is ‘null’ or not specified the use of silence suppression is negotiated between the two endpoints:

Image

In IPCS the following options can be set in EMPS and pushed out in the GVP.ini configuration file:

  • Enable the new G711 silence detection algorithm [g711sildetecalgorithm] (default 1)
  • Energy level for the new G711 silence detection algorithm [energylevel] (default 1)
  • Variation level for the new G711 silence detection algorithm [variationlevel] (default 1)
  • Min Silence Threshold [minsilencethreshold] (default 6)
  • Max Silence Threshold [maxsilencethreshold] (default 32)
  • Timeout value in ms before silence filing [silencefilltimeout] (default 100)

Image

In the Avaya world silence suppression is configured on each IP Codec set:

Image

In addition, a review of Avaya release notes for TN2602AP (MEDPRO) cards also reveals some VoIP options that can be configured on the card itself. These parameters seem to be engineering parameters. For example:

  • Parameter 17: Echo Canceller Alternate Comfort Noise Option
  • Parameter 18: Echo Canceller DTMF Transparency Option
  • Parameter 19: Echo Canceller Heavy Double Talk Option
  • Parameter 65: Maximum level of Comfort Noise Generator

One thing that jumped out in the release notes was the following statement relating to Firmware 46: “FW46 introduces the ability to limit the maximum level of Comfort Noise Generator (CNG) via a new tunable VOIP parameter, Parameter 65”.

During an active call, a certain amount of comfort noise is generated by the gateway so that the user does not assume that the far-end party has disconnected during silent periods of the conversation. The volume of this comfort noise is based on the ambient noise on the line. When excessive noise at 60 Hz and its harmonics is present, the volume of the comfort noise could be perceived as loud. Since the user can easily distinguish between the comfort noise and the hum, he perceives it as bursts of white noise when the other person is not speaking.

To change the ceiling, parameter 65 can be used to set the maximum to a lower value. For example a value of 100 is approx -50dBm, which is a reasonable compromise that provides adequate comfort noise, yet not so loud to be disconcerting. If hiss is an issue, we recommend initially setting parameter 65 to a value of 100. A low non-zero value is preferred as a value of zero essentially shuts comfort noise off altogether.

These parameters can be changed by Telnet-ing to the card itself and executing the following commands:

setVoipParam 65, 100
sendVoipParams
saveVoipParams

So putting all the pieces together I am sure that our problem is related to silence suppression and the maximum level of Comfort Noise Generator (CNG) configured on the Avaya MEDPRO. I’m off to make some changes and my starting point will be to compare the default value of all VoIP parameters on each MEDPRO at both firmware level 46 and firmware level 49.

PS: Whilst I’m on the subject of MEDPRO firmware there we also some interesting notes in firmware 51 which may be related to my DTMF problems that I blogged about earlier!

  • Previously, some badly formed incoming DTMF digits would not be detected. FW51 greatly improves DTMF detection.
  • The echo canceller has been modified to allow tuning to help eliminate double-talk (both ends “speaking” at the same time) induced corruption of DTMF digits. This improves IVR performance
  • Previously, in certain configurations, the RTP-Event DTMF marker bit on TN2602 was not being sent. This now adheres to RFC 2833.

 

 

Share

Wireshark and RTP tracing

Very useful tool is Wireshark! (http://www.wireshark.org/).

During capture (or playback of a capture file) navigate to “Telephony -> VoIP Calls” and then select a VoIP session:

Image

You can then click on “Graph” to get a SIP and RTP call flow chart:

Image

Even nicer you can select “Player -> Decode -> Play” and hear the audio stream:

Image

To analyse jitter select “Tools -> RTP -> All Streams“:

Image

You can then select a stream and click on “Analyse“:

Image

In the example above you can see a large delta of nearly 6 seconds during which a GVP / IPCS page fetch occurs. More on that in a later post ….

Share

Quality of Service Enablement for GVP and Stream Manager on Windows

QoS can be applied to prioritise the delivery of latency-sensitive traffic such as VoIP services and to control the impact of latency-insensitive traffic such as bulk data transfers.

The central problem of defining QoS for TCP/IP networks is how to specify and provide for prioritised delivery of IP traffic. To support QoS, the original RFC 791 for IP defined the Type of Service (ToS) field with the ability to specify precedence, delay, throughput, reliability, and cost characteristics. On CISCO routers, a Differentiated Services Code Point (DSCP) value can be specified in the ToS field of an IP packet. Differentiated Services (DiffServ) is a new model in which traffic is treated by intermediate systems with relative priorities based on the type of services (ToS) field.

Here is how QoS can be enabled on Windows based servers:

  • Firstly, you must have the QoS Packet Scheduler component installed and enabled from the properties of network connections in the Network Connections folder:

Image

  • Secondly, to enable RTP packet marking you must set the value of the following registry key to (DWORD) 1:

HKEY_LOCAL_MACHINE\Software\Microsoft\RTC\Transport\QoSEnabled

  • Microsoft provides APIs for assigning QoS parameters to traffic. You can use Winsock and the IP_TOS socket option to set the DSCP value for outgoing packets for a socket. However, by default the TCP/IP stack ignores the IP_TOS socket option. To use the IP_TOS socket option, you must create and set the value of the following registry key to (DWORD) 1:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DisableUserTOSSetting

  • Phew! That is the basic configuration required on Windows. To test this we can use a PING command using the “-v’ switch set to the ToS value and run a Wire Shark trace. In the example shown below the ToS value is 184 in decimal (HEX 0xB8) or Binary: 1011100 where the first 6 characters represent the DSCP value, which is the Expedited Forwarding (EF) code

Ping -v 184 xx.xx.xx.xx

Image

OK, so far so good. But how can we get Genesys components such as GVP and Stream Manager to set the IP_TOS value in RTP packets?

For Stream Manager we can configure a ToS value in the “rtp-ip-tos” option in the [x-config] section:

Image

We can also define the RTP port range using the “rtp-port” and “max-ports” options in the [contact] section:

Image

However, setting TOS bits is not possible within GVP 7.6 however this can be set within the network through a variety of mechanisms. For example when RTP packets from GVP arrive at the router they can be classified in a variety of ways such as using the source IP or destination IP addresses Layer4 protocol and port numbers, incoming interface, MAC address, etc. Additionally in Cisco environments it is possible to use NBAR (Network Based Application Recognition) to provide prioritisation based upon higher level protocols such as HTTP, RTP, etc. Typically this prioritisation results in setting of specific DiffServ priority levels.

To support such as configuration it is possible to specify the RTP port range by defining custom attributes “PortLow” and “PortHigh” within EMPS. By default the port range used is 1025-65535.

Image

Image

Share

GVP Fail to Record (0x80040003

This one is very strange!

We have a GVP application which captures the Customers name in a record block. All of a sudden this stopped working. The GVP POP gateway log showed the following error:

[2009/11/03 11:46:37.304] 12B0 VxmlPrompt.cpp:1149 C=12:L=8:U=320 Execute::end, Catch All – Rethrow
[2009/11/03 11:46:37.304] 12B0 VxmlRecord.cpp:1231 C=12:L=8:U=0 Record File name , File Ext. :vox
[2009/11/03 11:46:37.304] 12B0 VxmlRecord.cpp:1238 C=12:L=1:U=0:D=E_12_71 CreateFile failed [0x80040003] on [rec612A.vox]
[2009/11/03 11:46:37.304] 12B0 VxmlRedirector.cpp:272 C=8:L=16:U=0 VxmlRedirector::setVxmlSessionVars

A Google search of error code 0x80040003 revealed nothing of interest.

To further diagnose the problem we wrote a small test application which worked fine when provisioned against a test IVR profile but still failed against the original IVR profile. The IVR profile and provisioning data were checked and found to be identical (except for the provisioned DIDs of course). Moved the DIDs over to the test IVR profile and it worked fine!

Then regenerated the IVR profile – still no joy.

Image

Checked that the underlying file system on each IPCS server was the same and that permissions on folder “\GVP\CN\web\upload” were correct:

Image

Finally fixed then problem by deleting the existing IVR profile and then creating a new one – painful!

Share