Troubleshooting Connections and Backend Issues

Social Connections 11 Chicago, June 1-2 2017
Two wrongs don’t make a right –
Troubleshooting Connections
Terri Warren, panagenda
@TerriLLBean
Nico Meisenzahl, panagenda
@nmeisenzahl

PLATINUM SPONSORS
GOLD SPONSORS
SILVER SPONSORS

Nico Meisenzahl
• Consultant at panagenda
• IBM Connections since version 3.0 / 2010
• IBM Notes / Domino since 2008
• Focusing in ICS
• Deployment & consulting
• Optimization and migration
• “panagendian” since 2016
• IBM Champion
@nmeisenzahl
linkedin.com/in/nicomeisenzahl
meisenzahl.org
nico.meisenzahl
+49 170 7355081
nico.meisenzahl@panagenda.com

Terri Warren
@TerriLLBean
linkedin.com/in/terrillbean
live:TerriLLBean
terri.warren@panagenda.com
• Consultant at panagenda (since 2016)
• In my recent past I was a developer on IBM
Connections, IBM Domino
(Directory/Security) and an SRE on Watson
• Frequent presenter at IBM conferences
focusing in ICS
• Deployment & consulting
• Optimization and migration

Agenda
• Howto: Troubleshooting
• Troubleshooting…
• Connections itself
• Backend like Spectrum CfC, DB2, TDI,
WebSphere, Directory, SSO
• Optional add-ons like Orient Me, Docs, FEB,
Cognos, CCM
• Tools

Howto: Troubleshooting

Reproduce the error
• Reproducible and/or periodically?
• A sequence error?
• Client-side or server-side problem?
• Analyze the root cause

Be aware of the big picture
• Client-side problems
• Debug in different Browsers (IE, FF, Chrome)
• Do not use a server IE
• Server-side: IBM Connections is based on
many components
• debug on “high level” first
• get an overview which backend service is causing
the error

Configuration changes
• Changes in…
• Connections configuration
• Backend (WebSphere, Database, HTTP, CfC)
• Firewall or network
• OS, hardware or VM
Tip: Even the smallest configuration change can
have big consequences!

Analyze log files and browser
• Analyse log files
• Atom.io, Notepad++ or less/tail
• Baretail or tail –f
• kubectl logs
• ELK stack
• Tools for client-side problems
• Firebug or Developer-Tools
• BurpSuite or Fiddler

Analyze root cause
• Find a hint inside the log
• Network timeout or DNS
• SQL errors
• LDAP errors
• Syntax errors in configuration files „xxx-config.xml“
• Error stack
Tip: In a clustered environment, start and analyze only one
Node (if possible)

Find support
• Knowledge Center https://goo.gl/up6cxG
Troubleshooting Tips https://goo.gl/IaVinx
• IBM Connections Forum http://goo.gl/CVvQCU
• Community Blogs and/or Chats
• Fix Central
• PMR

Troubleshooting

WebSphere Application Server logs
• SystemOut.log
• SystemErr.log
• trace.log (if tracing is enabled)
• Log path:
• <wasroot>/profiles/<profilename>/logs/<servername>/

Analyze WAS log files
• Time stap: 24h time stamp with milli-seconds
• Thread id: eight character hexadecimal value
• Short name: typically java class name
• Event type: one character only (E, W, I,…)
• Message identifier: String based on component
• Message: Some information

WAS Event types
• F - Fatal message
• E - Error message
• W - Warning message
• A - Audit message
• I - Informational message
• C - Configuration message
• D - Detail message
• O - Messages that are written directly to System.out by an application
• R - Messages that are written directly to System.err by an application
• Z - Place holder to indicate type was not recognized

WAS Message identifier
• Prefix by Application or Server (CLFRW)
• Specific application code (0042)
• Event Type (I)

Read trace stack
• First line displays key information
• “Caused by” displays root cause

Environment information
• First log lines on server startup
• WebSphere version
• OS version, Process id
• Installation path
• Java version

Enable tracing
• Enable tracing using ISC
• Runtime or configuration
only
• Define tracing based on
• App prefix / error stack
• Must gather (PMR)

Search issues
• http(s)://<fqdn>/search/serverStatus
• Display index, seedlists, log information
• Data is displayed for one node only
• Using node fqdn to access different nodes
Tip: This will create persistent files on your
hard disk!

Debug Search
• Search queries (runtime)
• com.ibm.connections.search.index.searching.*=all
• Crawling & seedlists
• com.ibm.connections.search.index.indexing.*=all:
com.ibm.connections.search.seedlist.*=all:
com.ibm.connections.httpClient.*=all

Recreate Search Index
• SearchService.startBackgroundIndex()
• Crawls seedlists
• Extracts the file content
• Create index
Tip: Use „all_configured“ to index all apps

CLFRW0394E: Search indexing of services
...
• Search index not ready
• interruption at index creation
• CLFRW0283E: Search has encountered a problem
while crawling
• CLFRW0027E: Error Indexing component <app>
for search
• INDEX.READY file not present
• Recreate and enable tracing

Database connections
• Check datasouces
• ISC – Resources – JDBC – Data sources
• Check logs for more information
• DB2 server log
• <instanceroot>/sqllib/db2dump/
• db2diag.log
• db2diag.xxx.log (log rotation, you should enable this!)
Tip: Oracle users have password expiration enabled
by default!

HTTP Server (IHS & Plugins) logs
• IBM HTTP Server
• <installroot>/logs/
• error_log
• access_log
• based on configuration
• WebSphere AppServer Plugins
• <installroot>/logs/<webserver>/http_plugin.log

HTTP 404 not found
• Outdated Plugin configuration
• Restart IHS
• WAS Plugin configuration issue
• http_plugin.log

HTTP 404 not found
• AppServer or App down
• Network issue
• http_plugin.log

HTTP 500 Internal Server Error
• Unexpected error
• http_plugin.log
• Configuration issue
• WAS Root certificate not trusted or missing
• SSL certificate expired

Debug wsadmin
• Enable trace within wsadmin session
• AdminControl.trace(‘com.ibm.*=all’)
• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout

Directory/Troubleshooting/ Tuning
• Waltz and Sonata overview
• Enabling Log parameters…
• Cache Tuning
• SSO Config Glitches
• CCM Troubleshooting

Connections Troubleshooting
• Waltz: "common directory services"
• Directory Service eXtension (DSX)
REST API.
• Sonata: common HttpClient
services through a RESTful service
• configurable authenticators that support
• SSO security tokens and cookies for
secure server-to-server communication.

Troubleshooting:
DSOutOfServiceExceptions
• Symptoms:
• Access Connections not possible
• DSX not working

Troubleshooting: Enable Logging
• Track all basic Waltz configuration settings and transactions.
• com.ibm.connections.directory.services.*=all
• Track all LDAP transactions in between Waltz & LDAP server(s).
• WaltzLDAPUsage=all
• Track all LDAP entry to be cached and hit from cache upon DN of LDAP entries.
• WaltzDNEntryCache=all
• Track all LDAP entry to be cached and hit from cache upon ID of LDAP entries.
• WaltzExactIDMatchCache=all
• Track all group membership (a list of groups) for a given user upon ID.
• WaltzGroupMembershipCache=all
• Track all members (a list of users) for a given
group upon ID.
• WaltzMemberExpansionCache=all
TIP: Save runtime changes to make changes persistent

• WAS VMM/WIM:
• com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all
• WAS SPNEGO:
• com.ibm.ws.security.spnego.*=all
• Apache commons HttpClient:
• org.apache.commons.httpclient.*=all
• WAS security:
• com.imb.ws.security.*=all:com.ibm.ws.security.policy.*=off;

• Basic Sonata configuration settings and transactions:
• com.ibm.connections.httpClient.*=all"
• Track the amount of HTTP(S) transactions.
• SonataHttpUsage=all
• Track headers received for all HTTP(S) transactions.
• SonataHttpHeader=all
• Track all bodies received for all HTTP(S) transactions.
• SonataHttpBody=all

EJPVJ9284E: Unable to get the groups
from the directory for the user…
• User was not able to access Connections
anymore
• WAS LDAP bind user had no read access
to one of the groups the user was member

SSO Configuration Glitches:
• Warning indicating SSO config is missing or
incomplete enabling application security:
• Ensure you filled
in your domain
Tip: Connections CRs sometimes
resets SSO domain

SSO Configuration Glitches:
• LtpaToken Cookies:
• More secure- recommended to use version 2 LTPA cookies
In other words- interoperability mode is left as disabled by
default.
Only enable interoperability mode if
some app servers are still relying on
LTPA version 1 and need to interoperate.
Connections Security supports custom
LTPA cookie names since 4.5 CR1
Sets com.ibm.websphere.security.
customSSOCookieName custom
properties (verify 3rd party products
Portal/Domino can handle it)

Waltz Cache: Changing Group
Membership/permissions remains in Waltz Cache
• Issue: Administrator changes access or Group
membership for a user. That user does not get
immediate access.
• Cache: Connection's Directory Services keeps a “list” of
directory objects it has already searched for performance
reasons
● Improves performance of connections services
● Reduces load on remote directory servers
● Cache is based on a timing mechanism
● Cache is "flushed and renewed" on a 12 hour time schedule

Tuning the Cache: Continued
• How does this affect Group Membership?
● Group membership exists in the cache for 12 hours
● Operations such as renaming, deleting, or updating groups remain
in the cache for that time
• Configuration parameter for cache timing
mechanism
● Enables Connections administrators to flush the cache in
configurable time increments
● Set cache parameter via JVM arguments to flush them (we'll get to
that!)
● Default remains 12 hours

Cache Configuration: Setting the cache
flush parameters
• Setting the cache parameters via JVM configured in
WebSphere
● Google Setting generic JVM arguments in WebSphere Application
Server
• Time Values must be the same for all cache settings
• Adjust with care! Flushing the cache has implications!

IBM Connections Content Manager
(CCM): Troubleshooting
• During library creation, retrieving or modifying content:
• Calls are made between Communities, Profiles, and FileNet
 Issues with Filenet / Connections Profile or Community configuration can
adversely affect access to Libraries and CCM
 Logging from Connections components (previous slides) and FileNet essential
• SystemOut.log & trace.log – these are your friend
• FileNet core logging:
• <wasprofile>/<servername>/p8_server_error.log
• <wasprofile>/<servername>/p8_server_trace.log
• <wasprofile>/<servername>/pesvr_system.log
• <wasprofile>/<servername>/pesvr_trace.log

CCM Troubleshooting: Filenet URLS
• Status about FileNet Collaboration Services:
• check the status page at http://<fqdn>/dm
 Navigator version and configuration
• Status about FileNet Content Engine
• check the status page at http://<fqdn>/FileNet/Engine
 Server status, version, sonata/waltz version
• Status about FileNet Content Engine Domain and Object Store
• check the status page at http://<fqdn>/P8CE/Health
 Health checks for authentication, stores and database

Debug CCM Widget
• Widget issues
• com.ibm.quickr.communitylibrary.*=all:
com.ibm.lconn.widgets.service.*=all:
com.ibm.lconn.widgets.actions.*=all
• Authentication issues (Remember those cool tips a few
slides back? )
• com.ibm.connections.directory.services.*=all:
com.ibm.connections.httpClient.*=all

Debug FileNet using ACCE
• ACCE tool: check if FileNet Content Engine can
authenticate using Connections and inspect other data
for errors:
http(s)://<fqdn>/acce

CCM: Enable logging for Connections and FileNet
using JVM Properties
• Download and copy the sample waltz.sonata.log4j. as log4j.xml:
• Linux: /<absolute_path_for_log4j_file>/log4j.xml
• Windows:<absolute_path_for_log4j_file>log4j.xml
• Add generic JVM properties
• -Dlog4j.configuration=file:<path>/log4j.xml
-DskipTLC=true
• Tells the Content Engine (CE) server to skip it’s tracing configuration
and writes to waltz.sonata.trace.log
Windows:
c:Program FilesIBMWebSphereAppServerprofilesAppSrv01
Linux: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01

CCM: Enable logging for Connections and FileNet
using JVM Properties
• Copy & customize sample log4j.xml
• <ContentEngineRoot>/config/samples
• 20 subsystems (db, engine, security, search,…)

TDI logs
• <tdisol>/log/ibmdi.log
• TDI log file
• <tdisol>/employee.*
• Files include all changed users (adds, update, delete, error, skip)
• <tdisol>/syncupdates/*
• Temporary files within the sync
• Including database dump and ldiff
• sync_updates_clean_temp_files=false (default: true)
• profiles_tdi.properties
Tip: Check the lock file

Analyze TDI logs
• Error code prefix
• CLFRN: Profile & User synchronization
• CTGDIS: TDI itself
• Error code suffix
• I, E, W, …

Debug TDI
• Profile & User synchronization
(<tdisol>/etc/profiles_tdi.properties)
• source_ldap_debug=true
• debug_update_profile=true
• debug_collect=true
• TDI issues (<tdisol>/etc/log4j.properties)
• log4j.rootCategory=DEBUG, Default

Cognos BI logs
• Cognos BI
• SystemOut.log & trace.log
• <installroot>/logs/cogserver.log
• Cognos Transformer
• <installroot>/logs/cogserver.log
• PowerCube build
• <installroot>/metricsmodel/trxschelog.log
• <userhome>/Transformer/Logs/*.log (win only)

BMT-MD-6003 No connection to the
data source …
• PowerCubes not created yet
• Check cronjobs or scheduled jobs
• trxschelog.log

Debug Cognos BI & Metrics
• Communication between Cognos BI and
Connections Metrics
• SonataHttpUsage=all:
SonataHttpHeader=all:
SonataHttpBody=all:
com.ibm.connections.httpClient.*=all:
com.ibm.connections.metrics.*=all
• Connections Metrics Servlet
• com.ibm.connections.metrics.cognos.servlet.*=all

Docs/Viewer logs & urls
• SystemOut.log & trace.log
• http(s):<fqdn>/vsanity/check
• http(s):<fqdn>/sanity/check?app=all&querytype=report
• http(s):<fqdn>/*/version

Debug LTPA between Domino & WAS
• Debug on Domino side (notes.ini)
• Debug_SSO_Trace_Level=2
• Webauth_verbose_trace=1
• WebSess_verbose_trace=1
• Debug_outfile=<logfilepath>
• Debug on WebSphere
• com.ibm.ws.security.ltpa.*=all

Debug Kerberos
• Configuration
• com.ibm.ws.security.spnego.*=all:
com.ibm.ws.security.*=all: com.ibm.issw.spnegoTAI.*=all:
com.ibm.security.krb5.*=all
• Runtime
• com.ibm.connections.httpClient.*=all:
com.ibm.connections.directory.services.*=all:
com.ibm.websphere.wim.*=all:
com.ibm.ws.wim.*=all
• Fiddler & BurpSuite

IBM Spectrum CfC stats
• https://fqdn:8443

Docker & Kubernetes status
• kubectl get pods (--all-namespaces)
• kubectl get services
• kubectl get deployment
• kubectl describe pods xxx
• docker ps
• docker info

Docker & Kubernetes status
• Persistent Volume status
• kubectl get pv,pvc

Orient Me logs
• Display application logs
• kubectl logs xxx
• -f
• --tail=20
• --since=1h
• Special tasks for MongoDB and Solr

MongoDB Troubleshooting
• Check persistent volumes
• Access logs
• kubectl logs xxx [mongo|mongo-sidecar]
• Get replica status
• kubectl exec -it mongo-0 -- mongo mongo-0.mongo:27017 --eval
'rs.status()’
• Check “health” status for all 3
members

Solr logs
• kubectl exec -it xxx -- cat /var/solr/logs/solr.log
• kubectl logs xxx
• only display deploment logs

Pods startup issues
• kubectl describe pod xxx

CfC logs
• docker logs -f kubelet
• docker logs -f calico

Docker tracing
• service docker stop
• docker daemon –debug
• journalctl -u docker.service

What to get more insights on Orient Me?

Tools

Analyze logs
• Analyze logs live
• Baretail
• tail –f
• View logs
• Atom.io, Notepad++
• less, tail
• ELK Stack
• Elasticsearch, Logstash, Kibana
• Small Docker deployment
• Kubectl (local client)

Analyze logs
• Kubectl (local client)

Analyze Client-side
• Browser
• Firebug / Developer Tools
• Intercepting proxies
• Fiddler
• Burp Suite
• VMs with different IE versions
• Without GPO
• https://www.modern.ie/en-us/virtualization-tools

Database Clients
• db2 command
• Dbeaver or IBM Datastudio
• Robomongo (MongoDB)
Tip: Database write access is not supported!

LDAP & Search
• LDAP
• ldapsearch command
• Softerra LDAP Browser
• Apache Directory Studio
• Search Index
• Luke (Lucence Index Toolbox)

Analyze Network
• Wireshark
• tcpdump

Q&A

Troubleshooting Connections and Backend Issues

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à Troubleshooting Connections and Backend Issues

Similaire à Troubleshooting Connections and Backend Issues (20)

Plus de LetsConnect

Plus de LetsConnect (20)

Dernier

Dernier (20)

Troubleshooting Connections and Backend Issues

Notes de l'éditeur