We all know that one little application option can make all the difference between a working solution and something that appears to be completely broken. Similarly, we have also all encountered the situation where something was working and then suddenly stops working – often because a standby component has kicked in and we forgot to change an application option on both the primary and the backup components (a sackable offence in my opinion 🙂 )

This situation is often compounded when there are multiple Genesys environments to manage as is the case on my current project were we have the luxury of development, non production and production environments.

Whilst framework 8 has introduced some useful configuration auditing it still leaves a lot to be desired and does not address the issue of multiple environments.

Therefore this week I wrote a Configuration Audit application in C# .NET using the platform SDK to read CME objects as well some other configuration elements such as the actual .EXE version deployed for each application, SQL server database and instance information, Virtual Hold (VHT) options (of which there are many) and GVP / EMPS configuration via LDAP.

The output from each environment is a single text file which I can then compare across environments using WinMerge (http://winmerge.org/). Equally within the same environment the tool can also be used to output the options for high availability pairs and compare them.

The screenshots below show the differences between two configuration snapshots taken in the same environment before and after the weekend.

I have now configured a scheduled task in all environments to take a daily configuration snapshot which should make change and release management much easier in the future.

As I mentioned in an earlier post we are using Virtual Hold (http://www.virtualhold.com/) as part of the solution for this client.

OOTB Virtual Hold does not have any true resilience. Therefore we have deployed VHT on two seperate servers and have configured two Genesys SCI solutions and associated alarm conditions to failover between the primary and backup servers.

VHT uses a number of flat files to store scheduled callbacks and associated attached data and these folders have been configured as network shares (replicated Windows DFS shares actually).

VHT has two associated IVR (GVP) applications – one to schedule the callback and one for callback delivery. As part of the standard callflow the Customer is requested to record their name and this is then replayed on callback delivery as a message “This is a scheduled callback for xxxxx”.

We had a problem today trying to configure the location of the capture folder where the recorded name is stored.

Investigation revealed the following:

• The ASP page ‘CALLBACK_GetName.asp’ returns the filename and path to save the captured name to
• After capture a post back occurs to ‘CALLBACK_CAPTURENAME.asp’ which returned the following error:

2009-10-02 09:28:03 W3SVC1 10.176.201.109 POST /PILOT_IVR/3.5/ASP/CALLBACK_CAPTURENAME.asp
SESSIONID={8A8F9676-9A70-4693-9727-8AED5613AD3B}|203|800a0007|Out_of_memory:_’FS.GetParentFolderName’ 80 – 10.176.201.103 Mozilla/4.0+(compatible;+Telera+_ND_C+7.6.410.02) 500 0 0

• ‘CALLBACK_CAPTURENAME.asp’ includes ‘recordingsupport.inc’ which contains the VBScript in question:-

Sub CreateFolderPath(ByVal Path)
Dim FS
Dim Parent
Set FS = CreateObject(“Scripting.FileSystemObject”)
Parent = FS.GetParentFolderName(Path)
If Not FS.FolderExists(Parent) Then
CreateFolderPath Parent
FS.CreateFolder Parent
End If
End Sub

The problem was caused by a bug in the ASP pages generated by the Genesys Voice Studio application which prevents Universal Naming Convention (UNC) names being used in the capture path. The fix is as follows:

Edit the file “CALLBACK_GetName.asp” and change

strVoxFileName = Replace(strVoxFileName, “\”, “\\\\“)

to

strVoxFileName = Replace(strVoxFileName, “\”, “\\”)

For this client we use trunk access codes to get calls from the Avaya switch to the Genesys (SIP) switch and back again. In our development and non-production environment we have been using ‘1111’ to get calls to the Genesys switch and ‘1112’ to get calls from Genesys back to the Avaya switch. Therefore, on the Genesys SIP side, all our DNs have been prefixed with ‘1111’ and ‘1112’ has been added as part of the access code on each external routing point (ERP) back to the Avaya switch.

This week we discovered an Avaya numbering plan problem / conflict in the production environment which meant that we needed to change our DN numbering plan on the Genesys SIP side.

Since we have quite a few DN’s configured for routing points, external routing points and voice treatment ports, each with associated Annex data and access codes, I did not really want to risk deleting the DN’s and re-creating them again. Also, since we are using Genesys Info Mart in this architecture the impact of changing the DBIDs of these objects was unknown.

Therefore I decided to make the changes at the configuration database level. e.g. stop config server, make the database changes and re-start config server. Everything worked fine and I saved myself hours of re-configuration whilst at the same time minimising the risk of forgetting to add some annex data. Here is the SQL:

You may also notice from the SQL that we are using Virtual Hold (http://www.virtualhold.com/) in the solution and therefore each IVR (voice Treatment Port) needed to be re-configured. I’ll probably post a bit more about my experiences with VHT in the future 🙂

The final part of the numbering plan change involved EMPS provisioning which was done manually. The necessary changes being as follows:

• Add re-numbered IVR ports to “DID Groups -> Default”
• From “Resellers -> GVPOwner -> Admin -> Provision” add new Prov DIDs for each GVP / IVR application
• From “IPCS -> PopGateway -> SIP” change the starting number for PortIds on each IPCS server

Phew!

Following on from my earlier post (http://genesysguru.com/blog/blog/2009/09/26/avaya-sip-interoperability/) this week we upgraded to Genesys SIP Server 8.0.100.17 to test interoperability with CM 5.2 (CM 02.0.947.3 to be precise):

Whilst initial testing was very promising we found a problem when we tried to transfer a call back into a queue using Genesys Stream Manager on the Genesys SIP side. Basically, the second treatment would fail.

With the help of Genesys support and a Wireshark SIP trace the solution was found.

Set the option ‘sip-early-dialog-mode=false’ on the outbound (back to Avaya Trunk DN). On the inbound (from Avaya) Trunk DN leave ‘sip-early-dialog-mode=true’:

Looks like we now have a stable solution to move into production.

I did a bit more work on this yesterday and thought that the problem could be related to byte swapping between the Crypto API’s and the .NET managed code wrappers as discussed here: (http://blogs.msdn.com/shawnfa/archive/2005/12/05/500144.aspx).

It wasn’t! So today I gave up and did an IPCS re-install expecting problems. Guess what – I ran IPCS setup.exe, the license key was re-generated and the existing EMPS configuration data was maintained.

I should have done this in the first place. There again, I now know a bit about cryptology which I’m sure will be of use sometime in the future.

We are using GVP 7.6 as the IVR on this project and last week I managed to break one of our IPCS servers as a result of some network changes. GVP 7.6 is based on a product from Telera and as such the license keys are not based on FLEXlm as is the case for the rest of the Genesys suite.

For GVP 7.6 the license key is generated by the IPCS setup and is based on the MAC address of the NIC card in the server.

In our case the MAC address had changed and therefore the license validation started to fail. Given that this could happen operationally due to a NIC card swap or NIC teaming failure (http://www.howtonetworking.com/Networking/nicteam1.htm) I asked Genesys how to regenerate the license key. “Reinstall” was their only answer (and still is!).

Since the original installation was a complete pain as well as the associated EMPS configuration I thought that I would try to find a solution other than a complete reinstall.

A hunt through the binaries seemed to reveal the answer in some VAR PHP code in the file ‘vwm_start_setup.php’ – the encryption is based on the DES algorithm using ECB as the cipher and a secrey key that can be found in plain text within that file.

A bit of C# .NET code worked both ways on the default VAR password ‘vareports’ which is encrpyted as ‘SaEhZ8yoqDYz02y5HxG8qA’:

However, when I tried the same encryption on a MAC address I did not get the correct results. I also noticed that encrypted VAR passwords are 192 bits whereas encrypted license keys for IPCS are 160 bits. Damn!

Maybe the encrpytion routines are different between VAR and IPCS and were created by different developers at Telera? Back to the drawing board.

Next, I did some disassembly of the IPCS installation code in ‘CNSetup.dll’ and found that standard Microsoft Crypto API’s we being used:

An RSA crypto provider is obtained and an MD5 hash created:

The MAC address from the NIC is added to the hash:

A session key is created based on the MD5 hash of the MAC address. The value ‘26625’ tells me that the algorithm is RC2:

Finally, the data is encryped using the session key:

OK, I now had all the information and tried it in some C# .NET code:

It’s not quite working yet and think the expected result will be a string starting ‘TELERA’. Hopefully I can get it working next week and save myself a painful re-install. Will keep you posted.

Writing a T-Lib client using either the old (unsupported) T-library or newer Platform SDK is a fairly easy task and the complexity of the underlying wire protocol is not an issue.

However, what if you need to write a T-Lib server component?

You may ask why we might need to do this.

As part of this project we need to integrate Verint Impact 360 / Blue Pumpkin workforce management with Genesys to replace the existing integration via CMS. Out of the box Verint provide real-time adherence integration via their CSTI adapter for Genesys. However, the client already has an number of advisor activities defined based on the Avaya event stream and there is no 1:1 mapping for this to the T-server event stream.

As a workaround for the first phase of the project we have developed a custom Advisor desktop which allows us to set ACW codes via a ReasonCode in the extensions attribute of the Genesys CTI events. These are then picked up by some customisation in the Verint CSTI adapter.

However, in the next phase of the project we will be replacing the custom desktop with a vanilla SAP WebIC client and to set the ReasonCode would require major enhancements in SAP (and it not a good idea in terms of future releases).

Therefore, one alternative would be to develop a T-Lib server component that sits between the Verint CSTI adapter and the real T-Server component and ‘manipulates’ the CTI event stream accordingly based on an internal state machine (SCXML). This way we could use the standard Verint and SAP components OOTB.

As a proof of concept I decided to examine the T-library protocol based on some wireshark traces:

After some analysis and with reference to ‘tlibrary.h’ from the old C-based T-lib I think that I have worked out most of the protocol and put together some C# .NET code to test it:

The bottom line is (ignoring support and other commerical issues) that technically such a solution is possible. I’m not sure whether we will need to use this solution but thought that I would share my findings with you anyway.

For this client our target architecture is based on Avaya Communication Manager in an ESS configuration acting as a single logical switch. We are using Genesys 7.6 components for IVR and in-queue treatments. The Genesys side is all SIP based e.g. GVP and Stream Manager. Effectively the Avaya and Genesys sides are seperate switches and we use external routing between them.

We have been having numerous problems with calls being dropped after 32 seconds – the standard SIP session timeout. We have fixed most of these problems but a SIP trace using wireshark (http://www.wireshark.org/) reveals that SIP interoperability is not 100% correct and the biggest problem is that SIP T-Server stays in the signalling path even when the call has been routed to a target back on the Avaya switch. This means that all calls in progress are dropped if we lose SIP T-Server e.g. a single point of failure.

We has been running CM5.1 with SIP Server 7.6 and the recommendation from Avaya and Genesys is to upgrade to CM5.2 and SIP Server 8 respectively. For information to upgrade to SIP Server 8 new license features are required including 8.0 ISCC and HA.

This week we thought that we would start by upgrading to CM5.2. Having done this we could not even get calls onto the SIP trunks – SES logs showed nothing. The upgrade process seemed to work OK and we checked and double checked all the settings. Still no joy.

Eventually we managed to get Avaya support involved and the problem seems to be related to SIP domain names. Basically, the domain name we had been using was not a valid fully qualified domain name (FQDN) e.g. ‘mysipdomain.com’. Whilst it was OK in CM5.1 it looks as though additional validation has been added in CM5.2.

If you are planning to upgrade to CM5.2 make sure you check you current SIP domain name is the places shows below!

We have been doing quite a bit of playing about this week to establish the best settings for auto answer given that we are using TSAPI and therefore ‘route-thru-queue’ is not supported 🙁

For now the best combination seems to be:

• Agent logins – set Auto Answer = ‘all’
• Stations – Answer = ‘acd’

A combination of the above seems to work the best but we still have a problem with a single ring before the call is answered. Looks like we need to modify the IP handset boot configuration to solve this.