addNode timeout error

Context : installing Connections, and federating Cognos Node to Dmgr.


The error is :

ADMU0027E: Une erreur s'est produite au cours de la fédération Read timed out ;
           retour à la configuration d'origine.
ADMU0211I: Les détails de l'erreur peuvent être consultés dans le fichier :
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\addNode.log
ADMU0026I: Une erreur s'est produite au cours de la fédération ; retour à la
           configuration d'origine.
ADMU0113E: Fin du programme avec l'erreur
           com.ibm.websphere.management.exception.AdminException:
           com.ibm.websphere.management.exception.ConnectorException:
           ADMC0009E: Le système n'a pas pu effectuer d'appel RPC SOAP :
           invoke, résultant de [SOAPException: faultCode=SOAP-ENV:Client;
           msg=Read timed out; targetException=java.net.SocketTimeoutException:
           Read timed out]
ADMU1211I: Pour obtenir une trace complète de l'échec, utilisez l'option
           -trace.
ADMU0211I: Les détails de l'erreur peuvent être consultés dans le fichier :
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\addNode.log

In addNode.log :

[12/08/13 10:04:38:867 CEST] 00000000 ProviderTrack I com.ibm.ffdc.osgi.ProviderTracker AddingService FFDC1007I: Fournisseur FFDC installé : com.ibm.ffdc.util.provider.FfdcOnDirProvider@42f47f5a
[12/08/13 10:04:38:913 CEST] 00000000 Ffdc          I com.ibm.ffdc.util.provider.FfdcOnDirProvider logIncident FFDC1003I: Incident FFDC émis sur C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\ffdc\ffdc.1639904841534703718.txt com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate 846
[12/08/13 10:04:38:992 CEST] 00000000 AbstractNodeC E   ADMU0040E: Exception sur appel du MBean WebSphere:name=AdminOperations,process=dmgr,platform=proxy,node=SRVCONNECT1CellManager01,version=8.0.0.5,type=AdminOperations,mbeanIdentifier=AdminOperations,cell=SRVCONNECT1Cell01,spec=1.0 mergeConfigDataOnDmgr com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: Le système n'a pas pu effectuer d'appel RPC SOAP : invoke

[12/08/13 10:04:39:195 CEST] 00000000 AdminTool     A   ADMU0026I: Une erreur s''est produite au cours de la fédération ; retour à la configuration d''origine.
[12/08/13 10:04:48:242 CEST] 00000000 AbstractNodeC E   ADMU0040E: Exception sur appel du MBean WebSphere:name=AdminOperations,process=dmgr,platform=proxy,node=SRVCONNECT1CellManager01,version=8.0.0.5,type=AdminOperations,mbeanIdentifier=AdminOperations,cell=SRVCONNECT1Cell01,spec=1.0 doUnMergeConfigDataOnDmgr javax.management.MBeanException: Exception thrown in RequiredModelMBean while trying to invoke operation doUnMergeConfigDataOnDmgr

Here, it is recommanded to increase timeout.

soap.client.props is here : C:\IBM\WebSphere\AppServer\profiles\AppSrv01\properties

Modified timeout to 600.

#------------------------------------------------------------------------------
# SOAP Request Timeout
#
# - timeout (specified in seconds [default 180], 0 implies no timeout)
#
#------------------------------------------------------------------------------
com.ibm.SOAP.requestTimeout=600

In Dmgr, the node has already been created by the addNode.bat command. In order to-- launch the addNode command again, wa have to remove it.

In Dmgr :

However :

• Removing node from the Dmgr asks the the node agent to be started
• Starting nodeagent on cognos server returns a fileNotFoundException. Server.xml is not found!
• It not possible to removeNode using the command line on Cognos server. Error : the node is not federated to a cell.

So it is not usable but not federated. It already exists in Dmgr, but I'm not able to start the nodeagent.

C:\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>startNode.bat
ADMU0116I: Les informations sur les outils sont journalisées dans le fichier
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\nodeagent\startServ
er.log
ADMU0128I: Démarrage de l'outil avec le profil AppSrv01
ADMU3100I: Lecture de la configuration du serveur : nodeagent
ADMU0111E: Fin du programme avec l'erreur : java.io.FileNotFoundException:
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\config\cells\SRVCONNECT2
Node01Cell\nodes\SRVCONNECT2Node01\servers\nodeagent\server.xml
           (Le chemin d?accès spécifié est introuvable.)
ADMU1211I: Pour obtenir une trace complète de l'échec, utilisez l'option
           -trace.
ADMU0211I: Les détails de l'erreur peuvent être consultés dans le fichier :
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\nodeagent\startServ
er.log
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>syncNode.bat srvconnect1.alteca
.fr 8879
ADMU0116I: Les informations sur les outils sont journalisées dans le fichier
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\syncNode.log
ADMU0128I: Démarrage de l'outil avec le profil AppSrv01
ADMU2026E: Le noeud SRVCONNECT2Node01 n'est pas intégré à une cellule.
C:\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>removeNode.bat
ADMU0116I: Les informations sur les outils sont journalisées dans le fichier
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\removeNode.log
ADMU0128I: Démarrage de l'outil avec le profil AppSrv01
ADMU2001I: Début du retrait du noeud : SRVCONNECT2Node01.
ADMU2026E: Le noeud SRVCONNECT2Node01 n'est pas intégré à une cellule.
ADMU0211I: Les détails de l'erreur peuvent être consultés dans le fichier :
           C:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\removeNode.log

But if you simply force delete the Node in the Dmgr, addnode can run. 

Also uninstall the Cognos application, so you start clean. Without this you won't be able to access Cognos.

Success

-classpath is not recognized as an internal or external command operable program or batch file

During Connections 4.5 installation, at the point of launching cognos-setup.bat, I get this error :

Success to verify the JDBC connection to Cognos Content Store database.
'-classpath' n'est pas reconnu en tant que commande interne
ou externe, un programme exécutable ou un fichier de commandes.
Failed to verify the JDBC connection to Metrics database. Please check the error
 message.
Validation failed, unable to continue setup

Which in English is :

-classpath is not recognized as an internal or external command operable program or batch file

 May this be because of parenthesis in my password ?

-> No I changed every related password on this server and no amelioration.

What annoys me is that it's not even an error message ! That's a bug and that bug was already on version 4. I opened a PMR then but finally abandonned.

Finally solved by modifying the cognos-setup.bat!

1. JDBC Connection for Cognos Content Manager works but not JDBC Connection for Metrics.
2. I bypass the verifying of the JDBC Connection for Cognos Content Manager and now JDBC Connection for Metrics works ok!

That means that everything is good in my file but the ".bat" is not constructed correctly ??

I have some difficulty believing that this batch was not tested... So what is the difference between my environment and everyone else's ?

Here is the modification of cognos-setup.bat (basically I just added a bunch of "REM") :

REM Validate the JDBC connection.
SET was=%was.install.path%
SET java="%was%\java\bin\java.exe"
SET JAR_HOME=%cognosSetupScriptPath_%BI-Customization\JDBC\*
PUSHD %cognosSetupScriptPath_%lib
SET cp=.;"%CD%\*";"%JAR_HOME%"
POPD
REM Validate the JDBC connection to Cognos Content Store database
REM SETLOCAL ENABLEDELAYEDEXPANSION
REM    set "java_PATH=!java!"
REM FOR /F "tokens=*" %%j IN ('"!java_PATH! -classpath %cp% com.ibm.connections.metrics.cognos.install.CognosDBJDBCConnectionVerifer %cognos.db.type% %cognos.db.host% %cognos.db.name% %cognos.db.user% %cognos.db.password%"') DO (
REM ENDLOCAL
REM IF "%%j" == "JDBC Connection Success" SET cognosjdbcconnection=true
REM CALL :log %%j
REM )
REM IF NOT "%cognosjdbcconnection%" == "true" (
REM CALL :log Failed to verify the JDBC connection to Cognos Content Store database. Please check the error message.
REM EXIT /b 1
REM )
REM CALL :log Success to verify the JDBC connection to Cognos Content Store database.
REM Validate the JDBC connection to Metrics database
SETLOCAL ENABLEDELAYEDEXPANSION
    set "java_PATH=!java!"
FOR /F "tokens=*" %%j IN ('"!java_PATH! -classpath %cp% com.ibm.connections.metrics.cognos.install.MetricsDBJDBCConnectionVerifer %metrics.db.type% %metrics.db.host% %metrics.db.name% %metrics.db.user% %metrics.db.password%"') DO (
ECHO %%j
ENDLOCAL
IF "%%j" == "JDBC Connection Success" SET metricsjdbcconnection=true
CALL :log %%j
)
IF NOT "%metricsjdbcconnection%" == "true" (
CALL :log Failed to verify the JDBC connection to Metrics database. Please check the error message.
EXIT /b 1
)
CALL :log Success to verify the JDBC connection to Metrics database.
CALL :log ... performing validation check completed
ENDLOCAL
GOTO :EOF

During cognos-configure.bat, same problem of course. This time I took a different approached. I duplicated a paragraph which is used to set parameters. This one :

REM Validate the JDBC connection.
SET was=%was.install.path%
SET java="%was%\java\bin\java.exe"
SET JAR_HOME=%cognosSetupScriptPath_%BI-Customization\JDBC\*
PUSHD %cognosSetupScriptPath_%lib
SET cp=.;"%CD%\*";"%JAR_HOME%"
POPD

Here is the final result for cognos-configure.bat :

REM Validate the JDBC connection.
SET was=%was.install.path%
SET java="%was%\java\bin\java.exe"
SET JAR_HOME=%cognosSetupScriptPath_%BI-Customization\JDBC\*
PUSHD %cognosSetupScriptPath_%lib
SET cp=.;"%CD%\*";"%JAR_HOME%"
POPD
REM Validate the JDBC connection to Cognos Content Store database
SETLOCAL ENABLEDELAYEDEXPANSION
    set "java_PATH=!java!"
FOR /F "tokens=*" %%j IN ('"!java_PATH! -classpath %cp% com.ibm.connections.metrics.cognos.install.CognosDBJDBCConnectionVerifer %cognos.db.type% %cognos.db.host% %cognos.db.name% %cognos.db.user% %cognos.db.password%"') DO (
ENDLOCAL
IF "%%j" == "JDBC Connection Success" SET cognosjdbcconnection=true
CALL :log %%j
)
IF NOT "%cognosjdbcconnection%" == "true" (
CALL :log Failed to verify the JDBC connection to Cognos Content Store database. Please check the error message.
EXIT /b 1
)
CALL :log Success to verify the JDBC connection to Cognos Content Store database.
REM Validate the JDBC connection.
SET was=%was.install.path%
SET java="%was%\java\bin\java.exe"
SET JAR_HOME=%cognosSetupScriptPath_%BI-Customization\JDBC\*
PUSHD %cognosSetupScriptPath_%lib
SET cp=.;"%CD%\*";"%JAR_HOME%"
POPD
REM Validate the JDBC connection to Metrics database
SETLOCAL ENABLEDELAYEDEXPANSION
    set "java_PATH=!java!"
FOR /F "tokens=*" %%j IN ('"!java_PATH! -classpath %cp% com.ibm.connections.metrics.cognos.install.MetricsDBJDBCConnectionVerifer %metrics.db.type% %metrics.db.host% %metrics.db.name% %metrics.db.user% %metrics.db.password%"') DO (
ENDLOCAL
IF "%%j" == "JDBC Connection Success" SET metricsjdbcconnection=true
CALL :log %%j
)
IF NOT "%metricsjdbcconnection%" == "true" (
CALL :log Failed to verify the JDBC connection to Metrics database. Please check the error message.
EXIT /b 1
)
CALL :log Success to verify the JDBC connection to Metrics database.
CALL :log ... performing validation check completed
ENDLOCAL
GOTO :EOF

This worked also, but I don't like having to modify the batch. I could have opened a PMR but that would take me 3 weeks again and a lot of work

SSO not complete

SSO is only working one way! I login to Connections, then go to webmail, SSO takes care of authentication. But the opposite, when I first login to webmail, it does not log me in to Connections automatically.


Several steps I didn't take, according to Zero To Hero Integration Guide

1. Editing Wimconfig in order to exernalName instead of uniqueName for userUniqueIdMapping
2. importing SSL certificate from Websphere

SSO for Domino official documentation

Here is a interesting Open Mic about SSO

WIM

By the way, what is WIM ? (from What's new in version 6.1? ) :

WebSphere Application Server V6.1 also includes WebSphere Identity Manager (WIM) (also included in WebSphere Portal, which provides basic identity, profile, and user information that can be used by JAAS). Figure 8 illustrates the WIM framework.

So it in linked to authentication.

The wimconfig.xml procedure is explained (because I have to always know why I do what I do!) : here at Kenio blog 

Here is the paragraph I'm referring to :

Please note - if you make subsequent changes to the Global Security Federated Repository area using the ISC - Step 3 may need to be redone as changes may be lost.

What this does -

Step 1.) Insures that the username in the LTPA token created from Domino map to an existing repository in WAS - If there is no match, you get the "user not in defined realm" error in the logs.

Step 2.) Insures that Domino Flat groups can be found for policies

Step 3.) Insures that the username in the LTPA token that WAS generates is resolvable by the Sametime Community Server. In general, Domino does not validate the usernames contained within the LTPA token, it grants the user "default" level access to the database based on the validity of the token.

That seems important! However from that source : http://www-01.ibm.com/support/docview.wss?uid=swg1PM33575 it seems that it is included in 8.0.0.5 having externalName or uniqueName is both fine.

Path of wimconfig.xml : C:\IBM\WebSphere\AppServer\profiles\Dmgr01\config\cells\connectionsCell01\wim\config

However

Wiki : Troubleshooting

In Connections Wiki documentation, they also provide with an action to take on wimconfig file for troubleshooting SSO in the context of using flat group in Domino LDAP.

I applied, and that didn't make any change.

SSO D'bugging

From this Interesting document about SSO in the ICS world (sametime in that case)

D'buggin – It's more an art than science -
● Process of elimination – where is the problem originating?
─ Confirm that basic authentication (username/password) works first
─ Confirm with basic browser based tests before attacking Sametime itself
─ From a browser
– access http://<domino>/names.nsf then go directly to http://<was>/stmeetings
– access http://<was>/stmeetings (login) then go directly to http://<domino>/names.nsf
─ Do both tests – what does this tell you?
– First test – Domino created the LTPA token, Second test WAS created it
– If both tests pass – then continue on to Sametime issues
– If both tests fail – 99% of the time, the LTPA keys are not in synch
– If test(s) fail in one direction only – keys are in synch, but something else is off
– Most common reason for failing from Domino to WAS is “user not in defined realm”

And this point also to "ExternalName" and "UniqueName" modification.

Also (Step 3 = Modifying wimconfig.xml) :

Please note: If you make subsequent changes to the Global Security Federated Repository area using the Integrated Solutions Console (ISC), then Step 3 might need to be redone as changes may be lost.

This is really unfortunate, to have to redo wimconfig modification each time federated repository are changed.

However, I have already ExternalName, so this must be ok.

Getting information

Another source is this paper. What is good about this one, is that, even though this is Websphere portal and not application server, it goes in the direction of getting more information. Debugging by checking parameters is ok at the beginning. But when you checked all the basics, and if you are going in-depth for the first time, my opinion is that you have to know how to get information. I cannot go blind for long.

In fact, this paper is part of a serie on SSO which can be useful if you are interested to have more knowledge on ibm LTPAToken SSO.

Which LTPA

Specifically in my environment, I have :
- several domino in the same domain, SSO ok.
- Connections, SSO works only one way
- Sametime System Console with Proxy Server, SSO ok with domino, and one way with Connections
So basically, Connections side is not OK.
However, is it possible that by adding SSO for Sametime System Console, I erased the LTPA Token for Connections ? How is it possible to check which LTPA Token is in place in domino ?

I reimported the LTPA key from Connections to Domino (restart task HTTP) and Sametime, and now SSO works.

Note: when testing Connections SSO, do not use https://connections.server.com/homepage/login
The "login" at the end puts you at the login page even though SSO will log you if you access https://connections.server.com/homepage

LDAP Bind account and disappearing groups in Sametime

On a production environment, the LDAP Bind account is in LocalDomainAdmins group. That's historical but poses a threat to security. However, when removed, LDAP groups appeared as empty for users in Sametime!

Solving

Testing access to LDAP you have :

- ldapsearch which is delivered with lotus notes

- Ldap Browser from Softerra which i use a lot, and it's free

You can get it here : http://www.ldapbrowser.com/download.htm

Clic there to access the free tool

With LDAP testing I found that some groups had the problem and other not. This is LDAP related.

Finally

Some groups had reader field and I had not made aware of them. I guess you never know what's in a production environment until you break your head on it!

I just added the LDAP Bind account to reader access, restarted LDAP task and it rolled.