Lack of updates!

Sorry for the lack of updates. It has been mad busy over the last few months since I started on a new project. I’ll post something interesting as soon a possible. For now, have a look here: http://www.care4cc.com/

Have a great time over the festive period.

Share

Genesys Hosted Provider Edition (HPE)

I’ve been doing some background research into Genesys hosted / Cloud / SaaS offerings.

In summary, HPE is a Genesys certified reference solution for Hosted Service Providers with wrappers around standard Genesys components intended to simplify deployment and administration in order to support the “No longer need to re-invent the wheel with each deployment” goal.

The key features of HPE are:

  • Service provider / multi-tenant model
  • Pre-defined set of services
  • Pre-defined hardware / software configuration
  • Pre-defined performance levels
  • Pre-defined service availability levels
  • Modular scalability

Image

HPW 8.1 added support for Service Provider License Reporting Manager (LRM) which has reports providing details on tenant utilisation.

HPE is designed to scale in increments called a “Growth Segment” or “Services Segment“. The concept of a segment arises from the requirement to support modular growth. There can be a maximum of 16 segments and each Services Site (see later) can contain up to 8 segments.

Each segment can support:

  • 4000 configured advisors
  • 2000 active advisors
  • Up to 100 tenants
  • 18 interactions per second (70% Inbound / 20% Outbound / 10% Multimedia)

HPE makes extensive use of VMware vSphere 4 to provide virtualisation of hardware resources, to simplify administration, consolidate hardware, and assist with Disaster Recovery. With the exception of Oracle 11g RAC, GIM and Media Server, all components of HPE are deployed on virtualised servers.

HPE uses HP BL 460 G7 blade servers and EMC2 CLARiiON CX4 network storage with a RecoverPoint appliance running VMWare Site Recovery Manager for geographical redundancy and data protection.

HPE 8.1 uses the following VMWare software:

  • VMWare vSphere Hypervisor ESXi v4.1.0
  • VMWare vSphere Enterprise Plus
  • VMWare vCenter Standard server v4.1.0
  • VMWare vCenter SRM v4.1.1
  • VMWare vCenter Heartbeat v6.3
  • Oracle Real Application Clusters 11g Release 2
  • Oracle GoldenGate 11g

HPE components are deployed at Management Sites, Services Sites and Tenant Sites as shown in the diagram below:

Image

The Management Site hosts centralised Configuration, Management and Reporting components including Configuration Server, Solution Control Server, License Reporting Manager (LRM) and Genesys Interactive Insights.

The Management Site can support up to 32,000 active advisors, 64,000 configured advisors, and 1,600 tenants distributed across multiple Services Sites.

Each Services Site hosts the Genesys components associated with providing services such as Inbound Voice, Outbound Voice, E-Mail, Chat and Reporting to tenants. These components include SIP Server, IVR Server, Universal Routing Server, Stat Server, Media Server, Interaction Concentrator, Genesys Info Mart, and associated databases. Each Services Site supports short-term survivability, up to 24 hours in the event of a Management Site outage, or loss of network connectivity to the Management Site.

Tenant Sites host individual tenant (Customer) contact centres e.g. Customer sites. In many cases, no Genesys components are installed at the Tenant Sites. In cases where the tenant has a local switch, its associated T-Server is installed at the tenant site. In addition, some components are also installed at the Tenant Site for tenants that require Genesys Workforce Management.

Each Services Site is composed of several block types.

Each Services site has a single Site Common block. The components in the Site Common block are common to an entire Services Site. These components are deployed at the site level because they can be scaled to support the overall capacity of a single site, and it would be less efficient to deploy them at the segment or tenant level.

Each Services Site contains up to eight “Growth Segments” or “Services Segments”. Each Services Segment supports up to 2000 active advisors. Network-level routing and Disaster Recovery are managed on a per-segment basis. To support Disaster Recovery, each active Services Segment has a designated redundant backup Segment deployed at a different Services Site, with any associated databases replicated to that site.

Each Services Segment contains a single Segment Common block. The components in the Common block are multi-tenant aware, and are shared by all the tenants in the segment.

Each Services Segment can contain up to 100 Tenant blocks. The components in the Tenant block are those that do not currently support multi-tenancy, and must therefore be deployed with one instance per tenant.

Share

GVP 7.6 with secure voice using Secure RTP (SRTP)

In a previous post here (http://genesysguru.com/blog/blog/2011/09/14/implementing-secure-voice-using-secure-rtp-srtp/) I mentioned that Genesys GVP 7.6 components do not support voice encryption using Secure RTP (SRTP) and also proposed a solution architecture using Session Border Controllers (SBC) with Back to back User Agent (B2BUA) functionality deployed in front of Genesys Voice Platform (GVP) 7.6 instances to act as a bridge between secure voice traffic (SRTP) and insecure voice traffic (RTP).

Well, I am pleased to confirm that technically this works and at no material cost other than blood, sweat and tears!

I do not have time to describe the full solution here but please feel free to contact me (http://genesysguru.com/blog/contact-me/) if you want some more information.

Before I get into the detail, if you take away one think from this post it is when playing with TLS on Genesys components make sure that your certificates are correct as this took me hours to sort out.

OK, let’s get started.

For my proof of concept I used FreeSWITCH (http://www.freeswitch.org/) as a Session Border Controller. This is an EXCELLENT open-source product that can be used for all sorts of things including resolution of SIP interoperability issues.

The main steps were:

  1. Install FreeSWITCH (5 minutes!)
  2. Configure FreeSWITCH
  3. Generation and installation of TLSv1 self-signed (untrusted) certificates (painful)
  4. Enabling of TLS on Genesys SIP Server
  5. Reconfiguration of GVP / IPCS
  6. Reconfiguration of Voice Treatment Port DNs in CME
  7. Reconfiguration of Avaya (optional)
  8. Testing

Install FreeSWITCH

Just run “freeswitch.msi”!

For testing purposes, I also installed the FreeSWITCH Client which is a softphone which supports secure voice. I also installed CounterPath Bria 3 which is a commercial softphone (http://www.counterpath.com/bria.html).

Configure FreeSWITCH

I needed to modify the Access Control Lists (ACL) to add Local IP network segments into the allowed domains list – you may not need to do this.

I edited the file “C:\Program Files\FreeSWITCH\conf\vars.xml” to enable TLS and set the TLS version to TLSv1 rather than SSLv2.3:

Image

Initially I created SIP profiles which are used to configure gateways (SIP endpoints) for each GVP/IPCS instances. However, after some testing I decided that this was not required as everything could be achieved in the dialplan and out of service (OOS) checks on gateways did not provide any real benefit (see later).

The meat of the configuration is in the FreeSWITCH dialplan.

The XML dialplan is the default dialplan used by FreeSWITCH. Dialplans are used to route calls to endpoints. These endpoints can be traditional extensions, voicemail, IVR or other applications. Dialplans are separated into contexts that allow separate dialplans to be created for different call types.

Within a context are extensions which contain condition rules and associated actions which are performed when the condition rules match. Essentially, extensions can be used in the same way as Avaya Vectored Directory Numbers (VDN). Each rule is processed in order until an action (or anti-action) is reached. There can be multiple conditions and actions (anti-actions) in a single extension.

Condition rules are specified using Perl regular expressions which can be used to test channel variables.

Actions can include applications such as the “bridge” application which is used to bridge two endpoints physically or “set” and “export” which are used to manipulate channel variables.

To enable SRTP the dialplan must check to see if the variable ${sip_has_crypto} contains the data indicating that the calling device supports SRTP. Then, in order to enable SRTP to be used, the dialplan must set the variable sip_secure_media=true. It is as easy as that!

For the proof of concept I wanted to bridge calls for GVP ports in the range 80110xx to the relevant GVP/IPCS server. To do this I created a new file in the “C:\Program Files\FreeSWITCH\conf\dialplan\public” folder which contained the following dialplan:

Certificates

I used OpenSSL to create my own Certificate Authority (CA) and then each user certificate.

The important thing to note is that the name given for Common Name (CN) in the user certificate must be the same as the name used as the registrar (domain) name on SIP endpoints.

For Genesys components, each PEM encoded certificate and key needs to be exported to a single PKCS (Public-Key Cryptography Standards) #12 (PKCS#12) file. To install the certificates on Windows 2003, the Certificates Snap In needs to be added to MMC.

To install the certificates required for FreeSWITCH, copy the root CA certificate file “cafile.pem” to “C:\Program Files\FreeSWITCH\conf\ssl” and the relevant user certificate for the FreeSWITCH server (host) to “C:\Program Files\FreeSWITCH\conf\ssl” and then rename it as “agent.pem

During a test call check the SIP server logs for the following error:

12:52:44.714 Std 08102 Secure connection error, ‘socket 716, InitializeSecurityContext failed, syserror 80090322, The target principal name is incorrect.

The “Target principal name is incorrect” error means that the peer certificate does not contain the name that the server the TLS connection is to. Alternatively, the “Certificate root not trusted” error means that the peer certificate was issued by a remote CA that is not trusted by the local machine.

As I just said, the important thing to note is that the name given for Common Name (CN) in the user certificate must be the same as the name used as the registrar (domain) name on SIP endpoints.

Enabling of TLS on Genesys SIP Server

For TLS, there is a dependency to install Genesys SIP server 8.0.400.xx. I tested with SIP Server 8.0.400.39.
The following options need to be configured on the SIP server application object in CME and then SIP server restarted:

  • sip-port-tls = 5061
  • sip-tls-cert = (see below)
  • sip-tls-mutual = false
  • sip-tls-crl = [no value]
  • sip-tls-target-name-check= no
  • sip-tls-cipher-list= [no value]

Image

I wasted a couple of hours playing with the “sip-tls-cipher-list” option but just leave it blank as it seems to make no difference anyway!

The value for the “sip-tls-cert” option is taken from the thumbprint in the user certificate for the SIP server (host) and not the root CA certificate. The thumbprint can be obtained by double clicking on the “.p12” file and then selecting the details tab.

Image

If you get this wrong you will see the error “cannot find certificate” in SIP server logs.

If this error has occurred it means that either the certificate has not been installed correctly or SYSTEM does not have the correct permissions on folder “C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys“:

Image

If everything is configured correctly, then log file should look as follows:

Image

Reconfiguration of GVP / IPCS

Each GVP / IPCS instance needs to be reconfigured to so that it does not register GVP ports with SIP server since this overrides the “contact” details specified on each GVP voice treatment port DN. This is configured in EMPS and when requires a watchdog restart:

Image

Please note the following important impact of this change:

Normally, in a back of switch configuration with GVP 7.6, each IPCS is configured to register its ports with SIP Server and the option “use-register-for-service-state” is configured on each GVP voice treatment port DN in CME. This enables active out of service (OOS) detection each IPCS instance. When an IPCS instance fails the registration times out (after up to 30 seconds) the status of all associated ports goes unavailable in Stat Server.

Hence, the strategy which routed to the IVR place group will not route calls to OOS GVP ports. Given the registration timeout period there is however still no absolute guarantee that calls are not routed to an OOS IPCS instance.

This configuration is required because GVP 7.6 does not support OOS detection using ping of SIP OPTIONS messages which is the normal mechanism for OOS detection in SIP environments. OOS detection using SIP OPTIONS is only supported on Genesys DNs of type “VoIP Service” and “Trunk”. It is not supported on DNs of type “Voice Treatment Port”.

The effect of the change to “use-register-for-service-state=false” is that GVP ports always remain available in Stat Server even when GVP / IPCS is down. In SIP environments, the standard approach to OOS detection is:

  1. Active OOS detection
  2. Try to route calls and handle any failures. However this introduces some delay e.g. 3 seconds based on the FreeSWITCH dialplan parameter “call_timeout=3”
  3. A combination of both (1) and (2)

By default, FreeSWITCH responds “200 OK” to any SIP OPTIONS request it receives if there is a SIP profile configured and running on the associated port e.g. 5060. In essence, this response just means that FreeSWITCH is available and not that the downstream route end to end e.g. through to GVP / IPCS is available. Therefore, configuration of a Trunk DN in CME for each FreeSWITCH server, enabling OOS and checking the trunk status in the routing strategy would not provide any value.

Within FreeSWITCH SIP profiles (gateway) configuration files there is a parameter “ping” which specifies the interval to send a SIP OPTIONS message to a gateway e.g. GVP / IPCS in order to determine its OOS status e.g. gateway up or down. If a gateway is down any attempt to route calls to it in the dialplan would result immediately in a status of “NETWORK_OUT_OF_ORDER” e.g. no timeout / latency.

FreeSWITCH provides failover on the bridge application by allowing multiple destinations to be specified and dialled sequentially using the “|” separator. Hence, using the ping parameter on the gateway allows FreeSWITCH to determine a gateway has failed which allows the bridge to go to the secondary immediately rather than waiting for a timeout.

However, as previously mentioned, GVP 7.6 does not support OOS detection using SIP OPTIONS. Hence, this option cannot currently be used.

Hence, the only possible solution is the following hybrid approach to out of service (OOS) detection:

Firstly, the IVR routing strategy is modified to check the status of a Third Party application object created in CME for both the FreeSWITCH application and also the Watchdog executable for IPCS. This gives a very good real-time view of the availability of both a GVP/IPCS instance and its associated FreeSWITCH / SBC server. If either is not available the call is not routed any ports hosted on this GVP / IPCS instance.

Secondly, the FreeSWITCH dialplan is configured with the call-timeout set to 3 seconds to catch any errors which occur after the initial check e.g. the call is routed to a GVP port. This will result in an error being returned to SIP server if the end to end route is not available. The IVR routing strategy is modified such that in this circumstance it routes the call to another GVP / IPCS instance. The net effect of this is a 3 second delay in being routed to a GVP port.

Reconfiguration of Voice Treatment Port DNs in CME

As mentioned above, the option “use-register-for-service-state” must be set to false. Also, the contact details much be changed to point at the FreeSWITCH server on port 5061 rather than GVP / IPCS itself on port 5060 and also using “transport=tls”.

Image

Reconfiguration of Avaya (optional)

In my proof of concept environment we have Avaya Communication Manager (ACM) sitting in front of Genesys and using SIP trunking via Avaya SIP Enablement Services (SES) in between. Since I wanted to test full end to end voice encryption I also needed to configure TLS and secure Voice on the Avaya side.

Note: Officially TN2302 IP Media Processors do not support SRTP (they actually do!) so if you are using SRTP use the TN2602. Use the command “display circuit-packs” to confirm.

The main steps are:

  • The option “Media Encryption Over IP” must be set to “y” in System Parameters -> Customer Options.
  • On the associated IP Codec Sets, the “Media Encryption” options and order must be set.The encrypted network region needs to have an unencrypted (none) codec at the end of the media encryption list to enable non secure (RTP) sessions to be negotiated with Genesys Stream Manager which does not support SRTP (this can be changed later when Media Server is used to replace Stream Manager as this component does support SRTP). During the re-INVITE to a GVP port, a new session will be negotiated using SRTP from Avaya to GVP via the SBC / FreeSWITCH.

Image

  • On the route patterns for the SIP signalling groups, the option “Secure SIP” must be set to “y”.
  • On the SIP signalling groups, the “Transport Method” must be set to “tls” and the listening ports set to “5061”.
  • Add the IP address of the FreeSWITCH host into the trusted host list in SES and also into the encrypted network region in ACM.
  • Modify the SES maps to specify “transport=tls” outbound to Genesys SIP server.

Testing

We are done!

Firstly, I did quite a lot of testing without TLS enabled using a Wireshark or FreeSWITCH sip trace (sofia global siptrace on) to see if the “a=crypto” line was present in the SIP messages.

I also ran the FreeSWITCH CLI command “show channels” and looked for the line “srtp:AES_CM_128_HMAC_SHA1_32” in the trace.

Image

I did some testing with Bria 3 on an account with TLS and secure voice configured. Notice the lock symbol next to call established. This shows that the voice traffic is secured:

Image

I made an end to end test call without TLS specified as the transport on GVP ports. A Wireshark RTP trace clearly shows RTP traffic between Avaya (.138), FreeSWITCH SBC (.117) and GVP/IPCS (.111) including RTP DTMF Event payloads from Avaya:

Image

With TLS then re-enabled, a Wireshark RTP trace no longer shows RTP traffic to or from Avaya (.138) including RTP DTMF Event payloads from Avaya. All unencrypted RTP traffic is between FreeSWITCH SBC (.117) and GVP/IPCS (.111) within the same PCI Island:

Image

Job done and thanks for reading!

Share

Implementing secure voice using Secure RTP (SRTP)

Introduction

In a SIP implementation, DTMF information can be transported between SIP endpoints with out-of-band (OOB) or in-band signaling. In-band DTMF transport methods send DTMF tones as either raw tones in the RTP media stream or as signalled tones in the RTP payload with RFC 2833. Among SIP product vendors, RFC 2833 has become the predominant method to send and receive DTMF tones.

From a Payment Card Industry (PCI) perspective if a SIP connected IVR is used to host payment applications, there is an issue in that the DTMF digits (cardholder details) can be intercepted in the RTP payload if the underlying network infrastructure is not secured.

Since, the RTP payload format itself does not have any built-in security mechanisms, confidentiality of the media streams must be achieved by encryption using external mechanisms, such as Secure RTP (SRTP).

Secure RTP (SRTP) is a profile of RTP defined in RFC3711 that provides encryption and authentication of audio (and video) data in a RTP stream. SRTP encryption keys and options are exchanged in SIP INVITE and response messages, preferably using secure SIP (SIPS).

For encryption and decryption of the data flow (and hence for providing confidentiality of the data flow), SRTP utilises the Advanced Encryption Standard (AES) as the default cipher. Besides the AES cipher, SRTP allows the ability to disable encryption outright using the so called “NULL cipher”.

AES specifies three possible key sizes, and by default the Avaya implementation uses AES operating in 128-bit Counter mode (AES-128-CTR) using a 128-bit key.

Almost everything is standardised for secure SIP calls, except for a widely adopted key exchange (derivation) mechanism. The key derivation function is used to derive the different keys used in a crypto context (SRTP encryption keys and salts, SRTP authentication keys) from one single master key in a cryptographically secure way. SRTP relies on an external key management protocol to set up the initial master key.

The most common method to negotiate the SRTP keys is the Security Descriptions for media streams (SDES / sDescriptions) key exchange method as defined in RFC4568. This is the key exchange mechanism used by Avaya Communication Manager.

SDES uses plain text key exchange via the SIP Session Description Protocol (SDP) within SIP messages and ideally requires TLS for enhanced security. However the SDES method, even if coupled with TLS, allows any SIP server that is in the signalling path to see the SRTP Master Key in plain text (but not the session key).

From a PCI perspective, encryption of the SIP signalling traffic is typically not mandated by the PCI QSA since using that master key to deduce the session key is not a simple undertaking, which means that SRTP does come with a lot of added value even if not coupled with TLS.

However, depending on the SIP endpoints there is a risk that if a SIP endpoint is requested to negotiate a secure RTP (SRTP) session but a secure SIP transport is not being used e.g. TLS is not specified as the transport and port 5061 is not being used, it will reject the INVITE message.

The SRTP standard (RFC 3711) defines the SRTP cryptographic parameters. The SRTP master key is passed using the Session Description Protocol (SDP) within SIP signalling messages as the “inline” parameter within SDP packets.

The receiver of an encrypted RTP packet needs to know the encryption cipher and mode, the authentication transform and tag length, the key derivation rate, and other information about the SRTP stream. This information is described with the media stream in SDP using a SRTP SDP attribute, “a=crypto“. An example is shown below:

Image

The diagram below shows how the master key is used in the SRTP Key Derivation process:

Image

A single SRTP master key is input to the Key Derivation Function (KDF). The other input may be the SRTP packet index, derived using the RTP packet sequence number. Thus, SRTP creates the several keys needed for packet encryption at the synchronisation source (SSRC) and authentication from a single master key.

Once the master key is exchanged (or installed) and session keys are derived, SRTP encryption and authentication keys can be periodically refreshed when the key derivation rate is non-zero and is set to some period. A zero key-derivation rate, however, restricts the KDF to one invocation at the start of the session. A non-zero rate means that every time the packet-index modulo key derivation rate is zero, the KDF will be invoked and a new encryption and a new authentication key will be derived. Normally, setting the key derivation rate to zero is recommended.

Genesys support for SRTP

Genesys GVP 7.6 components do not support voice encryption using Secure RTP (SRTP). GVP 8.x supports SRTP as well as SIP over a secured (TLS) transport.

The default behaviour of GVP 8.1 is:

  • If the other side (for example Avaya) ignores SRTP, GVP will fall back to non-SRTP mode
  • If a previously negotiated “m-line” attribute in an SDP is used in a re-offer or if the far end requests an offer and that m-line did not have SRTP negotiated, SRTP will not be added
  • If the far end re-offers and adds SRTP to a previously negotiated m-line, SRTP will be negotiated

GVP 8.x supports the following SRTP modes (srtp.mode):

  • None – No SRTP support. The Media Control Platform will ignore the “crypto” attribute in SDP offers
  • accept_only – SRTP is supported for SDP offers sent to the Media Control Platform, but the platform will not add SRTP to m-lines in outgoing offers that did not previously contain it
  • offer – SRTP is supported for SDP offers sent to the Media Control Platform, and will be included in all outgoing SDP offers
  • offer_strict – The Media Control Platform accepts SRTP received in the offer, and sends a crypto line in its own offer, but will fail if the answer does not contain a valid crypto line

GVP 8.x supports the following SRTP cryptography methods (strp.cryptomethods):

  • AES_CM_128_HMAC_SHA1_80
  • AES_CM_128_HMAC_SHA1_32

Implementation of SRTP between Avaya and GVP 7.6

The diagram below shows a high level overview of a solution architecture using Session Border Controllers (SBC) with Back to back User Agent (B2BUA) functionality deployed in front of Genesys Voice Platform (GVP) 7.6 instances to act as a bridge between secure voice traffic (SRTP) and insecure voice traffic (RTP).

Secure RTP (SRTP) is used to provide encryption and authentication of audio streams between the Session Border Controllers (SBC) and the Media Gateway (Avaya TN2602AP IP Media Resource circuit packs).

A back-to-back user agent (B2BUA) is a logical SIP network element. It resides between both end points of a phone call / SIP session and divides the communication session into two call legs and mediates all SIP signaling between both ends of the call, from call establishment to termination.

In the originating call leg the B2BUA acts as a user agent server (UAS) and processes the request as a user agent client (UAC) to the destination end, handling the signaling between end points back-to-back. A B2BUA maintains complete state for the calls it handles. Each side of a B2BUA operates as a standard SIP network element

Thus, the SBC acts on behalf of caller and creates a second call leg to the GVP port (destination party) and performs specific protocol “normalisation” or “fix-up”. The second call leg therefore does not negotiate any encryption and uses RTP rather than SRTP which is not supported on GVP 7.6.

Image

As shown in the diagram below, a high availably pair of Session Border Controllers are deployed in front of multiple Genesys Voice Platform (GVP) instances. Therefore the B2BUA functionality must support multiple routes allowing SIP requests to be forwarded to different GVP instances. For example, SIP messages received on port 5060 would be forwarded to GVP server 1, SIP messages received on port 5061 would be forwarded to GVP server 2 etc. etc

Image

Share

Virtual Hold Analyser

On this project we have implemented and rolled out out Virtual Hold.

Virtual Hold Concierge is a virtual queuing technology that educates and empowers callers. When customers are faced with hold time, Concierge tells them their estimated wait time (via Queue Speak Settings), and allows them the choice to receive a callback in the same amount of time as if they had waited on hold.

Virtual Hold Rendezvous provides scheduled callback (Appointment setting) capability. When the contact centre is closed, or when it is not convenient to receive a Virtual Queue call at the quoted time, Rendezvous allows customers to schedule a callback at a time that is convenient for them and the contact centre.

Outbound dialling is initiated by the Virtual Hold Queue Manager component using Genesys T-server to make “TMakePredictiveCall” requests. The pacing of dial request is controlled internally within the Virtual Hold application.

Over the last few weeks we have noticed an increasing number of failed callback dial request due to “technical” errors. The suspicion is that this is caused by a lack of Avaya resources (trunks or call classifier ports).

Image

Fundamentally in VHT 6.7.2 there is no way of controlling the pacing of the dial requests other than to limit the number of callback requests offered and hence scheduled.

Looking for concurrent callback dial attempts is like looking for a needle in a haystack so it was time to write yet another analysis tool to take the historical data from the VHT reporting database and then construct a timeline for each callback.

The output from the VHT Analysis tool is an Excel spreadsheet. The workbook contains a worksheet for each hour in the day. Each row represents a callback request and each column represents a 15 second time slice. Each cell is coded as follows:

Image

The figure below shows an example of 4 normal (successful) callbacks and one failed callback. The fail occurs just 1 minute after the callback was originally requested:

Image

The figure below shows examples of callbacks being retried on both busy and no answer:

Image

Here is the output showing callback concurrency and some fails:

Image

Image

Hopefully I can now get to the root cause of the problem quickly.

Share

Nuance Log Analyser

Following on from R2 Performance Testing and subsequent ASR tuning I have been working on further speech recognition analysis in the last week.

The Release 2 solution includes the rollout of Nuance Speech Recognition (ASR) for existing Customer identification. This is based on them saying their postcode and then the first line of the address.

At this client we have a total of 9 Nuance Recognizer servers so pulling of the Nuance log files, analysing them to identify calls with 5 or more utterances, pulling off and listening to each of the individual utterance WAV file and then manually looking up addresses in our customer database was all becoming very time consuming and monotonous!

Therefore I decided to extend the functionality of my custom Nuance Log Analyser tool to do all this at the click of a button! I also did a bit of playing with Microsoft Speech to Text using the dictation grammar to transcribe the audio utterances into text for me automatically!

The output for each call to be analysed further (since the utterance count would indicate retries on both postcode and address line prompts) is 4 files: a WAV file containing the merged utterances separated with a “beep”, a text file containing a possible transcription of the audio, a text file containing the actual ASR interpretations and a text file containing possible addresses returned from the customer database.

Reviewing each call now only takes 10 – 15 seconds!

Here are some screenshots:

Image

Image

Image

Image

Image

Share

My top 10 posts

I have been reviewing some of my posts from the last 2 years to remember where all my time has gone. Here are my favourites:

Performance Testing and Avaya Overload!
http://genesysguru.com/blog/blog/2010/11/05/performance-testing-and-avaya-overload/

SIP Server failover using Windows Network Load Balancing (NLB)
http://genesysguru.com/blog/blog/2009/10/27/sip-server-failover-using-windows-network-load-balancing-nlb/

Voice Quality, Jitter, Silence Suppression and ‘Talkspurts’
http://genesysguru.com/blog/blog/2009/11/12/voice-quality-jitter-silence-supression-and-talkspurts/

Genesys Strategy Analyser
http://genesysguru.com/blog/blog/2011/03/25/genesys-strategy-analyser/

SAP Gplus Load Balancer
http://genesysguru.com/blog/blog/2011/01/21/sap-gplus-load-balancer/

Who changed that CME object?
http://genesysguru.com/blog/blog/2010/02/19/who-changed-that-cme-object/

Genesys Social Engagement
http://genesysguru.com/blog/blog/2011/04/08/genesys-social-engagement/

DataSift integration with Genesys Social Engagement
http://genesysguru.com/blog/blog/2011/05/27/datasift-integration-with-genesys-social-engagement/

Genesys Test Utility (GTU)
http://genesysguru.com/blog/blog/2010/10/16/genesys-test-utility-gtu/

Virtual Hold Preview Mode
http://genesysguru.com/blog/blog/2010/06/13/vitual-hold-preview-mode/

Share

Genesys Business Driven Testing

Yet another old project which I found when trawling through my archives!

Behaviour-Driven Development (BDD) is an evolution in the thinking behind Test Driven Development and Acceptance Test Driven Planning. The intent of BDD is to focus development on the delivery of prioritised, verifiable business value whilst providing a common vocabulary that spans the divide between Business and Technology.

BDD relies on the use of a very specific (and small) vocabulary to minimise miscommunication and to ensure that everyone – the business, developers, testers etc. are not only on the same page but using the same words.

Although BDD is not truly applicable to packaged application deployment such a Genesys, Behaviour-Driven Testing (BDT) certainly is. BDT requires a very specific, small and common vocabulary to develop and execute tests. The goal is a Business readable and domain specific language that allows you to describe a system’s behaviour without detailing how that behaviour is implemented.

BDT means that the tests (plain text feature descriptions with scenarios) are typically written before anything else and verified by the business e.g. non technical stakeholders.

Back in early 2010 I asked myself the question – how could BDT be applied to Genesys projects?

Cucumber

Image

Cucumber is a tool that can execute plain-text functional (feature) specifications as automated tests. The language that Cucumber understands is called Gherkin. It is a Business Readable Domain Specific Language lets you describe a system’s behaviour without detailing how that behaviour is implemented.

In the context of a Genesys implementation, here is an example of a feature specification:

Image

The nice thing about feature specifications is that we can also use Scenario Outlines. Scenario Outlines are the solution to repetitive Given-When-Then scenarios since they allow us to separate the structure of the test, which doesn’t change, from the data, which does. With Scenario Outlines, Cucumber turns each example-each table row-into a concrete scenario before looking for matching step definitions.

Here is another example using scenario outlines:

Image

From the perspective of a business user or tester that is all there is to it!

For a developer’s perspective, they implement in ‘code’ the step definitions that ultimately get executed. Each of the Given/When/Then calls in the feature description is a step definition. When there’s a matching line in a Cucumber test, the step definition gets executed. Effectively the development methodology is outside-in (the outside being the feature, the inside being the low level code).

Putting it all together here is the end to end process:

  • When the Business decides they want to add a new feature or fix a bug, they (or a tester) start by writing a new feature or scenario that describes how the feature should work. At this point no ‘code’ is written
  • The feature is run in Cucumber and results in a display of yellow (pending steps) – or red (failing) ones. If all steps are green the feature has already been implemented!
  • At this point a developer implements the feature, or more precisely, implements the ‘code’ behind each step definition
  • The feature is run again Cucumber and results should all be green (like a Cucumber!)

Cucumber for Genesys

Rather than developers implementing step definitions I implemented a common library of step definitions related to Genesys in the context of Business Driven testing – I call this “Cucumber for Genesys”.

Even though Cucumber is written in the Ruby language we can use Cuke4Nuke to invoke some Microsoft C# .NET code which wraps the Genesys Platform SDK. Then when Cucumber runs the feature specification, Cuke4Nuke will look for methods marked with Given, When and Then attributes that match steps by regular expression. Any capture groups in the regular expression are passed to the method as arguments (they don’t have to be Strings; you can use other .NET types as well).

Here is a subset of my current Genesys feature step implementations:

Image

Putting all of the above together means I can define feature specifications for test telephony related functionality which allows tests to be automated and regression suites developed.

A test call:

Image

Testing advisor functionality:

Image

Cool as a Cucumber!

Image

Share

More ASR Tuning post Release 2 Go-Live

Another busy week listening to ASR utterances and tuning the IVR application as a result. In order to better understand the 10% of people who never say a postcode I listened to 4526 postcode utterances. I really wish I could share some of these with you!

My findings and recommendations were:

  • Barge-in on the postcode prompt resulted in Customers not expecting to have to say a postcode. This was an issue since Customers then heard silence. The recommendation was to disable barge-in on the postcode prompt and set the continuous recognition timeout to 7 seconds (average time is 3-5 seconds) and silence timeout to 4 seconds
  • A  few people say “YES”, “NO”, “ADVISOR”, “AGENT”, “NOT KNOWN” to try to opt out
  • Some people say an account number instead of a postcode (because they have barged-in and not heard the postcode prompt)
  • A few people did not know where the hash key is on the phone. Recommended changing the initial prompt to “If you haven’t got an account number just press hash on the bottom right of the keypad

The good news is that incremental changes are now having a positive effect on the Customer Experience and overall Customer identification success rates.

Image

Image

Share