Category Archives: UK Utility Company

GVP 7.6 with secure voice using Secure RTP (SRTP)

In a previous post here (http://genesysguru.com/blog/blog/2011/09/14/implementing-secure-voice-using-secure-rtp-srtp/) I mentioned that Genesys GVP 7.6 components do not support voice encryption using Secure RTP (SRTP) and also proposed a solution architecture using Session Border Controllers (SBC) with Back to back User Agent (B2BUA) functionality deployed in front of Genesys Voice Platform (GVP) 7.6 instances to act as a bridge between secure voice traffic (SRTP) and insecure voice traffic (RTP).

Well, I am pleased to confirm that technically this works and at no material cost other than blood, sweat and tears!

I do not have time to describe the full solution here but please feel free to contact me (http://genesysguru.com/blog/contact-me/) if you want some more information.

Before I get into the detail, if you take away one think from this post it is when playing with TLS on Genesys components make sure that your certificates are correct as this took me hours to sort out.

OK, let’s get started.

For my proof of concept I used FreeSWITCH (http://www.freeswitch.org/) as a Session Border Controller. This is an EXCELLENT open-source product that can be used for all sorts of things including resolution of SIP interoperability issues.

The main steps were:

  1. Install FreeSWITCH (5 minutes!)
  2. Configure FreeSWITCH
  3. Generation and installation of TLSv1 self-signed (untrusted) certificates (painful)
  4. Enabling of TLS on Genesys SIP Server
  5. Reconfiguration of GVP / IPCS
  6. Reconfiguration of Voice Treatment Port DNs in CME
  7. Reconfiguration of Avaya (optional)
  8. Testing

Install FreeSWITCH

Just run “freeswitch.msi”!

For testing purposes, I also installed the FreeSWITCH Client which is a softphone which supports secure voice. I also installed CounterPath Bria 3 which is a commercial softphone (http://www.counterpath.com/bria.html).

Configure FreeSWITCH

I needed to modify the Access Control Lists (ACL) to add Local IP network segments into the allowed domains list – you may not need to do this.

I edited the file “C:\Program Files\FreeSWITCH\conf\vars.xml” to enable TLS and set the TLS version to TLSv1 rather than SSLv2.3:

Image

Initially I created SIP profiles which are used to configure gateways (SIP endpoints) for each GVP/IPCS instances. However, after some testing I decided that this was not required as everything could be achieved in the dialplan and out of service (OOS) checks on gateways did not provide any real benefit (see later).

The meat of the configuration is in the FreeSWITCH dialplan.

The XML dialplan is the default dialplan used by FreeSWITCH. Dialplans are used to route calls to endpoints. These endpoints can be traditional extensions, voicemail, IVR or other applications. Dialplans are separated into contexts that allow separate dialplans to be created for different call types.

Within a context are extensions which contain condition rules and associated actions which are performed when the condition rules match. Essentially, extensions can be used in the same way as Avaya Vectored Directory Numbers (VDN). Each rule is processed in order until an action (or anti-action) is reached. There can be multiple conditions and actions (anti-actions) in a single extension.

Condition rules are specified using Perl regular expressions which can be used to test channel variables.

Actions can include applications such as the “bridge” application which is used to bridge two endpoints physically or “set” and “export” which are used to manipulate channel variables.

To enable SRTP the dialplan must check to see if the variable ${sip_has_crypto} contains the data indicating that the calling device supports SRTP. Then, in order to enable SRTP to be used, the dialplan must set the variable sip_secure_media=true. It is as easy as that!

For the proof of concept I wanted to bridge calls for GVP ports in the range 80110xx to the relevant GVP/IPCS server. To do this I created a new file in the “C:\Program Files\FreeSWITCH\conf\dialplan\public” folder which contained the following dialplan:

Certificates

I used OpenSSL to create my own Certificate Authority (CA) and then each user certificate.

The important thing to note is that the name given for Common Name (CN) in the user certificate must be the same as the name used as the registrar (domain) name on SIP endpoints.

For Genesys components, each PEM encoded certificate and key needs to be exported to a single PKCS (Public-Key Cryptography Standards) #12 (PKCS#12) file. To install the certificates on Windows 2003, the Certificates Snap In needs to be added to MMC.

To install the certificates required for FreeSWITCH, copy the root CA certificate file “cafile.pem” to “C:\Program Files\FreeSWITCH\conf\ssl” and the relevant user certificate for the FreeSWITCH server (host) to “C:\Program Files\FreeSWITCH\conf\ssl” and then rename it as “agent.pem

During a test call check the SIP server logs for the following error:

12:52:44.714 Std 08102 Secure connection error, ‘socket 716, InitializeSecurityContext failed, syserror 80090322, The target principal name is incorrect.

The “Target principal name is incorrect” error means that the peer certificate does not contain the name that the server the TLS connection is to. Alternatively, the “Certificate root not trusted” error means that the peer certificate was issued by a remote CA that is not trusted by the local machine.

As I just said, the important thing to note is that the name given for Common Name (CN) in the user certificate must be the same as the name used as the registrar (domain) name on SIP endpoints.

Enabling of TLS on Genesys SIP Server

For TLS, there is a dependency to install Genesys SIP server 8.0.400.xx. I tested with SIP Server 8.0.400.39.
The following options need to be configured on the SIP server application object in CME and then SIP server restarted:

  • sip-port-tls = 5061
  • sip-tls-cert = (see below)
  • sip-tls-mutual = false
  • sip-tls-crl = [no value]
  • sip-tls-target-name-check= no
  • sip-tls-cipher-list= [no value]

Image

I wasted a couple of hours playing with the “sip-tls-cipher-list” option but just leave it blank as it seems to make no difference anyway!

The value for the “sip-tls-cert” option is taken from the thumbprint in the user certificate for the SIP server (host) and not the root CA certificate. The thumbprint can be obtained by double clicking on the “.p12” file and then selecting the details tab.

Image

If you get this wrong you will see the error “cannot find certificate” in SIP server logs.

If this error has occurred it means that either the certificate has not been installed correctly or SYSTEM does not have the correct permissions on folder “C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys“:

Image

If everything is configured correctly, then log file should look as follows:

Image

Reconfiguration of GVP / IPCS

Each GVP / IPCS instance needs to be reconfigured to so that it does not register GVP ports with SIP server since this overrides the “contact” details specified on each GVP voice treatment port DN. This is configured in EMPS and when requires a watchdog restart:

Image

Please note the following important impact of this change:

Normally, in a back of switch configuration with GVP 7.6, each IPCS is configured to register its ports with SIP Server and the option “use-register-for-service-state” is configured on each GVP voice treatment port DN in CME. This enables active out of service (OOS) detection each IPCS instance. When an IPCS instance fails the registration times out (after up to 30 seconds) the status of all associated ports goes unavailable in Stat Server.

Hence, the strategy which routed to the IVR place group will not route calls to OOS GVP ports. Given the registration timeout period there is however still no absolute guarantee that calls are not routed to an OOS IPCS instance.

This configuration is required because GVP 7.6 does not support OOS detection using ping of SIP OPTIONS messages which is the normal mechanism for OOS detection in SIP environments. OOS detection using SIP OPTIONS is only supported on Genesys DNs of type “VoIP Service” and “Trunk”. It is not supported on DNs of type “Voice Treatment Port”.

The effect of the change to “use-register-for-service-state=false” is that GVP ports always remain available in Stat Server even when GVP / IPCS is down. In SIP environments, the standard approach to OOS detection is:

  1. Active OOS detection
  2. Try to route calls and handle any failures. However this introduces some delay e.g. 3 seconds based on the FreeSWITCH dialplan parameter “call_timeout=3”
  3. A combination of both (1) and (2)

By default, FreeSWITCH responds “200 OK” to any SIP OPTIONS request it receives if there is a SIP profile configured and running on the associated port e.g. 5060. In essence, this response just means that FreeSWITCH is available and not that the downstream route end to end e.g. through to GVP / IPCS is available. Therefore, configuration of a Trunk DN in CME for each FreeSWITCH server, enabling OOS and checking the trunk status in the routing strategy would not provide any value.

Within FreeSWITCH SIP profiles (gateway) configuration files there is a parameter “ping” which specifies the interval to send a SIP OPTIONS message to a gateway e.g. GVP / IPCS in order to determine its OOS status e.g. gateway up or down. If a gateway is down any attempt to route calls to it in the dialplan would result immediately in a status of “NETWORK_OUT_OF_ORDER” e.g. no timeout / latency.

FreeSWITCH provides failover on the bridge application by allowing multiple destinations to be specified and dialled sequentially using the “|” separator. Hence, using the ping parameter on the gateway allows FreeSWITCH to determine a gateway has failed which allows the bridge to go to the secondary immediately rather than waiting for a timeout.

However, as previously mentioned, GVP 7.6 does not support OOS detection using SIP OPTIONS. Hence, this option cannot currently be used.

Hence, the only possible solution is the following hybrid approach to out of service (OOS) detection:

Firstly, the IVR routing strategy is modified to check the status of a Third Party application object created in CME for both the FreeSWITCH application and also the Watchdog executable for IPCS. This gives a very good real-time view of the availability of both a GVP/IPCS instance and its associated FreeSWITCH / SBC server. If either is not available the call is not routed any ports hosted on this GVP / IPCS instance.

Secondly, the FreeSWITCH dialplan is configured with the call-timeout set to 3 seconds to catch any errors which occur after the initial check e.g. the call is routed to a GVP port. This will result in an error being returned to SIP server if the end to end route is not available. The IVR routing strategy is modified such that in this circumstance it routes the call to another GVP / IPCS instance. The net effect of this is a 3 second delay in being routed to a GVP port.

Reconfiguration of Voice Treatment Port DNs in CME

As mentioned above, the option “use-register-for-service-state” must be set to false. Also, the contact details much be changed to point at the FreeSWITCH server on port 5061 rather than GVP / IPCS itself on port 5060 and also using “transport=tls”.

Image

Reconfiguration of Avaya (optional)

In my proof of concept environment we have Avaya Communication Manager (ACM) sitting in front of Genesys and using SIP trunking via Avaya SIP Enablement Services (SES) in between. Since I wanted to test full end to end voice encryption I also needed to configure TLS and secure Voice on the Avaya side.

Note: Officially TN2302 IP Media Processors do not support SRTP (they actually do!) so if you are using SRTP use the TN2602. Use the command “display circuit-packs” to confirm.

The main steps are:

  • The option “Media Encryption Over IP” must be set to “y” in System Parameters -> Customer Options.
  • On the associated IP Codec Sets, the “Media Encryption” options and order must be set.The encrypted network region needs to have an unencrypted (none) codec at the end of the media encryption list to enable non secure (RTP) sessions to be negotiated with Genesys Stream Manager which does not support SRTP (this can be changed later when Media Server is used to replace Stream Manager as this component does support SRTP). During the re-INVITE to a GVP port, a new session will be negotiated using SRTP from Avaya to GVP via the SBC / FreeSWITCH.

Image

  • On the route patterns for the SIP signalling groups, the option “Secure SIP” must be set to “y”.
  • On the SIP signalling groups, the “Transport Method” must be set to “tls” and the listening ports set to “5061”.
  • Add the IP address of the FreeSWITCH host into the trusted host list in SES and also into the encrypted network region in ACM.
  • Modify the SES maps to specify “transport=tls” outbound to Genesys SIP server.

Testing

We are done!

Firstly, I did quite a lot of testing without TLS enabled using a Wireshark or FreeSWITCH sip trace (sofia global siptrace on) to see if the “a=crypto” line was present in the SIP messages.

I also ran the FreeSWITCH CLI command “show channels” and looked for the line “srtp:AES_CM_128_HMAC_SHA1_32” in the trace.

Image

I did some testing with Bria 3 on an account with TLS and secure voice configured. Notice the lock symbol next to call established. This shows that the voice traffic is secured:

Image

I made an end to end test call without TLS specified as the transport on GVP ports. A Wireshark RTP trace clearly shows RTP traffic between Avaya (.138), FreeSWITCH SBC (.117) and GVP/IPCS (.111) including RTP DTMF Event payloads from Avaya:

Image

With TLS then re-enabled, a Wireshark RTP trace no longer shows RTP traffic to or from Avaya (.138) including RTP DTMF Event payloads from Avaya. All unencrypted RTP traffic is between FreeSWITCH SBC (.117) and GVP/IPCS (.111) within the same PCI Island:

Image

Job done and thanks for reading!

Share

Implementing secure voice using Secure RTP (SRTP)

Introduction

In a SIP implementation, DTMF information can be transported between SIP endpoints with out-of-band (OOB) or in-band signaling. In-band DTMF transport methods send DTMF tones as either raw tones in the RTP media stream or as signalled tones in the RTP payload with RFC 2833. Among SIP product vendors, RFC 2833 has become the predominant method to send and receive DTMF tones.

From a Payment Card Industry (PCI) perspective if a SIP connected IVR is used to host payment applications, there is an issue in that the DTMF digits (cardholder details) can be intercepted in the RTP payload if the underlying network infrastructure is not secured.

Since, the RTP payload format itself does not have any built-in security mechanisms, confidentiality of the media streams must be achieved by encryption using external mechanisms, such as Secure RTP (SRTP).

Secure RTP (SRTP) is a profile of RTP defined in RFC3711 that provides encryption and authentication of audio (and video) data in a RTP stream. SRTP encryption keys and options are exchanged in SIP INVITE and response messages, preferably using secure SIP (SIPS).

For encryption and decryption of the data flow (and hence for providing confidentiality of the data flow), SRTP utilises the Advanced Encryption Standard (AES) as the default cipher. Besides the AES cipher, SRTP allows the ability to disable encryption outright using the so called “NULL cipher”.

AES specifies three possible key sizes, and by default the Avaya implementation uses AES operating in 128-bit Counter mode (AES-128-CTR) using a 128-bit key.

Almost everything is standardised for secure SIP calls, except for a widely adopted key exchange (derivation) mechanism. The key derivation function is used to derive the different keys used in a crypto context (SRTP encryption keys and salts, SRTP authentication keys) from one single master key in a cryptographically secure way. SRTP relies on an external key management protocol to set up the initial master key.

The most common method to negotiate the SRTP keys is the Security Descriptions for media streams (SDES / sDescriptions) key exchange method as defined in RFC4568. This is the key exchange mechanism used by Avaya Communication Manager.

SDES uses plain text key exchange via the SIP Session Description Protocol (SDP) within SIP messages and ideally requires TLS for enhanced security. However the SDES method, even if coupled with TLS, allows any SIP server that is in the signalling path to see the SRTP Master Key in plain text (but not the session key).

From a PCI perspective, encryption of the SIP signalling traffic is typically not mandated by the PCI QSA since using that master key to deduce the session key is not a simple undertaking, which means that SRTP does come with a lot of added value even if not coupled with TLS.

However, depending on the SIP endpoints there is a risk that if a SIP endpoint is requested to negotiate a secure RTP (SRTP) session but a secure SIP transport is not being used e.g. TLS is not specified as the transport and port 5061 is not being used, it will reject the INVITE message.

The SRTP standard (RFC 3711) defines the SRTP cryptographic parameters. The SRTP master key is passed using the Session Description Protocol (SDP) within SIP signalling messages as the “inline” parameter within SDP packets.

The receiver of an encrypted RTP packet needs to know the encryption cipher and mode, the authentication transform and tag length, the key derivation rate, and other information about the SRTP stream. This information is described with the media stream in SDP using a SRTP SDP attribute, “a=crypto“. An example is shown below:

Image

The diagram below shows how the master key is used in the SRTP Key Derivation process:

Image

A single SRTP master key is input to the Key Derivation Function (KDF). The other input may be the SRTP packet index, derived using the RTP packet sequence number. Thus, SRTP creates the several keys needed for packet encryption at the synchronisation source (SSRC) and authentication from a single master key.

Once the master key is exchanged (or installed) and session keys are derived, SRTP encryption and authentication keys can be periodically refreshed when the key derivation rate is non-zero and is set to some period. A zero key-derivation rate, however, restricts the KDF to one invocation at the start of the session. A non-zero rate means that every time the packet-index modulo key derivation rate is zero, the KDF will be invoked and a new encryption and a new authentication key will be derived. Normally, setting the key derivation rate to zero is recommended.

Genesys support for SRTP

Genesys GVP 7.6 components do not support voice encryption using Secure RTP (SRTP). GVP 8.x supports SRTP as well as SIP over a secured (TLS) transport.

The default behaviour of GVP 8.1 is:

  • If the other side (for example Avaya) ignores SRTP, GVP will fall back to non-SRTP mode
  • If a previously negotiated “m-line” attribute in an SDP is used in a re-offer or if the far end requests an offer and that m-line did not have SRTP negotiated, SRTP will not be added
  • If the far end re-offers and adds SRTP to a previously negotiated m-line, SRTP will be negotiated

GVP 8.x supports the following SRTP modes (srtp.mode):

  • None – No SRTP support. The Media Control Platform will ignore the “crypto” attribute in SDP offers
  • accept_only – SRTP is supported for SDP offers sent to the Media Control Platform, but the platform will not add SRTP to m-lines in outgoing offers that did not previously contain it
  • offer – SRTP is supported for SDP offers sent to the Media Control Platform, and will be included in all outgoing SDP offers
  • offer_strict – The Media Control Platform accepts SRTP received in the offer, and sends a crypto line in its own offer, but will fail if the answer does not contain a valid crypto line

GVP 8.x supports the following SRTP cryptography methods (strp.cryptomethods):

  • AES_CM_128_HMAC_SHA1_80
  • AES_CM_128_HMAC_SHA1_32

Implementation of SRTP between Avaya and GVP 7.6

The diagram below shows a high level overview of a solution architecture using Session Border Controllers (SBC) with Back to back User Agent (B2BUA) functionality deployed in front of Genesys Voice Platform (GVP) 7.6 instances to act as a bridge between secure voice traffic (SRTP) and insecure voice traffic (RTP).

Secure RTP (SRTP) is used to provide encryption and authentication of audio streams between the Session Border Controllers (SBC) and the Media Gateway (Avaya TN2602AP IP Media Resource circuit packs).

A back-to-back user agent (B2BUA) is a logical SIP network element. It resides between both end points of a phone call / SIP session and divides the communication session into two call legs and mediates all SIP signaling between both ends of the call, from call establishment to termination.

In the originating call leg the B2BUA acts as a user agent server (UAS) and processes the request as a user agent client (UAC) to the destination end, handling the signaling between end points back-to-back. A B2BUA maintains complete state for the calls it handles. Each side of a B2BUA operates as a standard SIP network element

Thus, the SBC acts on behalf of caller and creates a second call leg to the GVP port (destination party) and performs specific protocol “normalisation” or “fix-up”. The second call leg therefore does not negotiate any encryption and uses RTP rather than SRTP which is not supported on GVP 7.6.

Image

As shown in the diagram below, a high availably pair of Session Border Controllers are deployed in front of multiple Genesys Voice Platform (GVP) instances. Therefore the B2BUA functionality must support multiple routes allowing SIP requests to be forwarded to different GVP instances. For example, SIP messages received on port 5060 would be forwarded to GVP server 1, SIP messages received on port 5061 would be forwarded to GVP server 2 etc. etc

Image

Share

Virtual Hold Analyser

On this project we have implemented and rolled out out Virtual Hold.

Virtual Hold Concierge is a virtual queuing technology that educates and empowers callers. When customers are faced with hold time, Concierge tells them their estimated wait time (via Queue Speak Settings), and allows them the choice to receive a callback in the same amount of time as if they had waited on hold.

Virtual Hold Rendezvous provides scheduled callback (Appointment setting) capability. When the contact centre is closed, or when it is not convenient to receive a Virtual Queue call at the quoted time, Rendezvous allows customers to schedule a callback at a time that is convenient for them and the contact centre.

Outbound dialling is initiated by the Virtual Hold Queue Manager component using Genesys T-server to make “TMakePredictiveCall” requests. The pacing of dial request is controlled internally within the Virtual Hold application.

Over the last few weeks we have noticed an increasing number of failed callback dial request due to “technical” errors. The suspicion is that this is caused by a lack of Avaya resources (trunks or call classifier ports).

Image

Fundamentally in VHT 6.7.2 there is no way of controlling the pacing of the dial requests other than to limit the number of callback requests offered and hence scheduled.

Looking for concurrent callback dial attempts is like looking for a needle in a haystack so it was time to write yet another analysis tool to take the historical data from the VHT reporting database and then construct a timeline for each callback.

The output from the VHT Analysis tool is an Excel spreadsheet. The workbook contains a worksheet for each hour in the day. Each row represents a callback request and each column represents a 15 second time slice. Each cell is coded as follows:

Image

The figure below shows an example of 4 normal (successful) callbacks and one failed callback. The fail occurs just 1 minute after the callback was originally requested:

Image

The figure below shows examples of callbacks being retried on both busy and no answer:

Image

Here is the output showing callback concurrency and some fails:

Image

Image

Hopefully I can now get to the root cause of the problem quickly.

Share

Nuance Log Analyser

Following on from R2 Performance Testing and subsequent ASR tuning I have been working on further speech recognition analysis in the last week.

The Release 2 solution includes the rollout of Nuance Speech Recognition (ASR) for existing Customer identification. This is based on them saying their postcode and then the first line of the address.

At this client we have a total of 9 Nuance Recognizer servers so pulling of the Nuance log files, analysing them to identify calls with 5 or more utterances, pulling off and listening to each of the individual utterance WAV file and then manually looking up addresses in our customer database was all becoming very time consuming and monotonous!

Therefore I decided to extend the functionality of my custom Nuance Log Analyser tool to do all this at the click of a button! I also did a bit of playing with Microsoft Speech to Text using the dictation grammar to transcribe the audio utterances into text for me automatically!

The output for each call to be analysed further (since the utterance count would indicate retries on both postcode and address line prompts) is 4 files: a WAV file containing the merged utterances separated with a “beep”, a text file containing a possible transcription of the audio, a text file containing the actual ASR interpretations and a text file containing possible addresses returned from the customer database.

Reviewing each call now only takes 10 – 15 seconds!

Here are some screenshots:

Image

Image

Image

Image

Image

Share

More ASR Tuning post Release 2 Go-Live

Another busy week listening to ASR utterances and tuning the IVR application as a result. In order to better understand the 10% of people who never say a postcode I listened to 4526 postcode utterances. I really wish I could share some of these with you!

My findings and recommendations were:

  • Barge-in on the postcode prompt resulted in Customers not expecting to have to say a postcode. This was an issue since Customers then heard silence. The recommendation was to disable barge-in on the postcode prompt and set the continuous recognition timeout to 7 seconds (average time is 3-5 seconds) and silence timeout to 4 seconds
  • A  few people say “YES”, “NO”, “ADVISOR”, “AGENT”, “NOT KNOWN” to try to opt out
  • Some people say an account number instead of a postcode (because they have barged-in and not heard the postcode prompt)
  • A few people did not know where the hash key is on the phone. Recommended changing the initial prompt to “If you haven’t got an account number just press hash on the bottom right of the keypad

The good news is that incremental changes are now having a positive effect on the Customer Experience and overall Customer identification success rates.

Image

Image

Share

ASR Tuning post Release 2 Go-Live

As I posted last week we went live on Monday (27/06/2011) with our Release 2 solution and I am pleased to report that everything went very smoothly (for once!). Of course we had a number of minor issues which the team have worked hard on to resolve this week.

The Release 2 solution includes the rollout of Nuance Speech Recognition (ASR) for existing Customer identification. This is based on them saying their postcode and then the first line of the address. I have been buried in Nuance ASR logs all week and at the same time reviewing the associated recorded utterances. In fact, I analysed at total of 40000 utterances from Monday and 4 hours of utterance audio from 2 of the 9 Nuance Recognizer ASR servers!

As a result the following tuning recommendations have been made:

  • Increase the confidence level on postcode recognition from 0 to 4. This is because we were getting false positives on postcodes and then asking the customer to match against a list of addresses which would never match
  • Change the wording on the address prompt to include house number or name. This is because we observed that Customers were just saying a street name which would never match against a full address line

We have also identified a problem with invalid grammars when the address line contains 4 digits addresses e.g. 1234 SOME ROAD, when house numbers are prefixed with zero e.g. 01 SOME ROAD and when the address line also contains contact details such as telephone number. The result of this is that Customers are transferred directly to an advisor after giving a valid postcode.

Share

Release 2 Go-Live

Some frantic activity over the last few weeks trying to close things down for Release 2 Go-Live which is now scheduled for 27/06/2011. Release 2 has the addition of some core solution functionality including voice self service using speech recognition (ASR), Kofax non-voice channel integration and integration with SAP Web IC.

As usual we have found some “magic” settings at the last minute to fix a couple of critical issues:

IVR interface performance

We have developed a custom C# .NET application which provides the interfaces between the IVR applications (VoiceXML) and back end systems. Although we had performance tested them in isolation we hit a concurrency problem in final testing.

The solution to the problem was to set the .NET option “maxconnection” to enable the .NET runtime to open more than 2 concurrent web service connections (and hence block on subsequent requests):

Image

HTTPS with GVP 7.6

Since we will process payments in Release 2 IVR applications we need to enable HTTPS on the connection between each IPCS (Page Collector) and the IVR (VoiceXML) application servers.

However, enabling HTTPS resulted in intermittent IPCS page fetch errors, especially under load conditions. The solution to this is buried in solution search here:

http://solutionsearch.genesyslab.com/selfservice/dynamickc.do?cmd=show&forward=nonthreadedKC&docType=kc&externalId=15264&sliceId=1

  1. Create the following Registry entry as a DWORD value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort
  2. Set it to the value of 65534 (decimal). The default is 5000.
  3. Create the following Registry entry as a DWORD value: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay
  4. Set it to 60 or lower (decimal). The default is 240.
  5. Reboot the host.

GVP 7.6.470.xx MCU Core Dumps

We had been struggling with IPCS core dumps since Release 1 Go-Live and this was resulting in GVP ports getting stuck and needing to be taken out of service manually. The problem seemed to be related to prompt recording and playback in Virtual Hold (VHT). A long running ticket with Genesys support was eventually resolved this week after a couple of diagnostic builds provided by Genesys and tested by the team.

Release Number 7.6.470.17 [06/24/11] – Hot Fix

The IPCS MCU process no longer terminates unexpectedly at the end of recording. Previously under certain conditions, some internal C++ Standard Template Library (STL) lists would become corrupted at the end of a recording, causing the MCU to terminate unexpectedly. (ER# 269811868)

Well done team – we got there in the end!

Share

R2 Performance Testing

As we get closer to Release 2 go-live, last week saw some performance testing of new solutions components – Nuance Recognizer ASR and Kofax inbound correspondence. For R2 we are taking a predominantly risk based approach which means testing some of the new solution components in isolation.

During each test we monitor the CPU and RAM utilisation on each of the servers which host the relevant components.

Nuance Recognizer

For Nuance ASR testing we created a test application in order to simulate the recognition load on a single ASR server. To analyse the results I created a Nuance log file analyser which exports the raw data to Microsoft Excel for further analysis.

Here is the raw data:

Image

Analysis showing recognition latency as a function of time:

Image

Analysis showing recognition concurrency as a function of time:

Image

Kofax

The approach for Kofax was to create a test harness and then bulk inject emails. The harness monitors the underlying filesystem so that content processing can be tracked from the point of capture until it is released for further processing into SAP via Ceyoniq.

Here is the raw data:

Image

 

 

Share

Kofax System Management

On this project we are using Kofax Capture for the non voice channels i.e. Whitemail scanning, email, webforms and Fax (FoIP).

One of the problems with Kofax is providing integrated system management since the components are distributed over 5 physical servers and multiple Kofax components produce log files in different format and in different locations.

In addition our solution has custom import connectors for webforms, custom KTM validation scripts and custom export connectors to release the content into Ceyoniq for subsequent linking to SAP CRM via SAP ArchiveLink.

We plan to configure the Kofax services as Third Party application objects in CME so that we can monitor the status of the Kofax hosts and services in SCI.

To complete the “end to end” system management perspective I have written a monitoring service. The service:

  • Polls Windows NT application event logs
  • Polls log file folders and then parses the log files looking for standard error phrases
  • Monitors the import and export filesystems looking for index files that have not been processed within the SLA (and hence indicative of an error)

At present the output is via email although it would be very easy to integrate with the wider System Management infrastructure via SNMP traps and/or custom alarm generation into SCI.

Here are some screenshots of the output:

Image

Image

Image

Share

Genesys Strategy Analyser

Right, managed to get some more development time on my “Genesys Strategy Analyser” and now the realtime and debug functionality is up and running.

Here is a screenshot:

Image

When realtime monitoring is enabled, to modify a breakpoint associated with an IRD block and/or to clear the current counter on that shape, right click on the IRD block (Visio shape):

Image

To add a breakpoint click on “File -> Add Breakpoint” or to remove an existing breakpoint click on “File -> Clear Breakpoint”. Objects with a blue border indicate that there is a breakpoint set on the block:

Image

Image

When a breakpoint is hit, the breakpoint will be shown in the “watch” window. The “path” column shows the objects navigated in the strategy:

Image

Selecting a breakpoint in the watch window results in the path being highlighted in green in the Visio diagram:

Image

For those technically minded, the secret of realtime strategy monitoring is message server. Specifically, message types 1000 and 2102 which can be subscribed to using Genesys PSDK 8.0.2.

Message type 1000 contains URS loading information in keys named “RTR”. Message type 2102 contains debug information of type “04” and “05”.

Type “04” data looks something like this:

Image

This message contains executing counts for the object Ids highlighted in yellow. The same object Ids exists in an IRD export of the strategy (note the byte swapping):

Image

Share