Empirix Performance Testing

I mentioned in an earlier post that we have been (or trying) to undertake performance testing of the end to end solution at this client prior to rollout of the pilot (which went live in November 2009) to an additional two Contact Centre sites and a couple of 1000 advisors.

The Empirix testing infrastructure consists of 6 G5 Load Generators (2 at each of the 3 Contact Centre sites) and 3 Virtual Agent Simulators (VAS) at each of the Contact Centre sites.

We are injecting calls directly into the centralised Avaya SES server to be as representative of ISDN calls ingress as possible. From a callflow perspective this means that calls are injected in SES which then forwards the SIP INVITE to Avaya Communication Manager (CLAN cards). The call hits a VDN in the same way as for normal ISDN calls and is routed to Genesys via SES over SIP signalling links.

We have been working on 2 major issues for the last few weeks:

  • GVP (IPCS) crashes under load at a rate of 50 calls/minute. Although new calls continue OK we observe “stuck” calls on GVP ports
  • Calls hanging on Avaya stations even though they have been cleared down on the G5 load generators e.g. SIP BYE message sent. We do not want to clear down from the VAS end as this is not representative of the business process whereby advisors must wait for the caller to hangup

This week we have finally resolved both issues and have had an informal test run at moderate load. Here is what we found ….

IPCS Crashes

This turned out to set a JavaScript issue affecting IPCS 7.6.410 (MR1) all the way up to IPCS 7.6.470 (MR7) which is the latest release at the time of writing. The root cause is still under investigation by Genesys Engineering since it is not a good idea to allow a GVP Studio application “bug” which caused a JavaScript exception to crash a whole IPCS.

The error occurred when retrieving configuration data from a custom “config.xml” file in the following JavaScript line:

<assign name=”VOXFILEDIR” expr=”GetData(VOXFILEDIR, ‘VOX_FIlE_PATH’)”/>

And was fixed by changing this line to:

<data name=”VOXFILEDIR” src=”Config.xml”></data>
<assign name=”document.VOXFILEDIR” expr=”VOXFILEDIR.documentElement”/>
<assign name=”VOXFILEDIR” expr=”GetData(VOXFILEDIR, ‘VOX_FIlE_PATH’)”/>

Hung calls on Avaya stations

This turned out to be a SIP interoperability issue (surprise surprise!) and was fixed by a “downgrade” to the Empirix G5 SIP state machine.

We believe that the problem was caused by the SIP routing information (record-route attribute) being updated to include the IP address of the Avaya CLAN card in addition to the Avaya SES server via which the initial INVITE was sent:

Image

Since the Empirix G5 is stateful this updated routing information was being maintained and then re-used on the BYE message at the end of the call:

Image

As a result the BYE was being ignored and the call hanging (the Empirix G5 though that the call had been disconnected event though it never received an ACK back). To fix this problem the Empirix G5 SIP state machine was modified to ignore updated routing information (record-route attribute),

I’m not going to argue who is in the wrong here although I strongly suspect it is Avaya! The reason for saying this is another SIP interoperability issue that has popped up since. This time Avaya SIP interoperability with Kofax which we are using for Fax channel integration (hopefully!)

Please see: http://www.avayausers.com/showthread.php?p=77475

“The problem we always see is when Cisco sends a BYE to Avaya. Avaya sees the BYE but for whatever reason Avaya will never send an OK back to Call Manager. This results in a hung call leg in the Avaya. The hung call leg stays up in Call Manager until my timers expire and then the call is flushed.”

“What we have found is that Avaya fails to honour any SIP method unless record-route is used. If we run our SIP proxy servers in non-stateful mode (record-route off) Avaya fails to honor any method that didn’t come back from the first proxy that routed the call”

In the case of Kofax, Kofax does not include the record-route attribute in any response methods. This can be seen in the Wireshark trace below:

Image

“The only way the SIP stack on Avaya will function with a stateless proxy and respond to all parties that the proxy may send the call to requires the creation of a “dummy” signaling group on the Avaya PBX. Basically, you have to build your main signaling group with trunking to the proxy and then add a “dummy” signaling group into Avaya for each end point IP address that you may see SIP methods come back from. E.g. Kofax”

We have one main signaling group with trunking to the stateless proxy. Since the proxy is stateless it will only be in the call flow until the proxy sends the final OK response back from Cisco Call Manager.

In addition to the signaling on Avaya to the proxy, we also had to build a signaling group on the Avaya that has no trunking but has the IP address of the far end Call Manager server that would be in the call flow after the proxy (SES) sets the call up. This “dummy” signaling group has no trunking in Avaya – we have only defined the far end IP address in the signaling group page on Avaya.

Therefore, from a Kofax perspective I think what they are saying here is create a “Kofax signaling group” with the IP address of the Kofax server specified as the far end IP address.

Will let you know how we get on with this is a future post.

Share

Leave a Reply