CLM,RTC,RQM,RRC/RDNG: recommendations and “educated guesses” for limitations

September 5, 2013

As part of the Jumpstart team, I help our customers in their CLM adoption and deployments. Customers raise questions on the sizing of their CLM environment, the topologies to adopt, etc. The questions I hear most often are:

  • How many users (total or concurrent) can my CLM environment support ?
  • RTC rollout at our company is close to reach a second milestone (additional teams will use the tool). What planning (HW/SW) should we have wrt. these modifications?
  • We have this huge number of CLM (RTC/RQM/RRC-RDNG) artefacts. Will my CLM environment still handle this without any performance degradation as we continue adding artefacts into our repository ?
  • What are the intrinsic CLM product limitations and – if one is concerning me – what approach should I adopt to keep on working smoothly with my CLM ?
  • etc.

First of all, depending on the CLM version you’re running, central places to check are the CLM 2011 Sizing Guide and the CLM 2012 Sizing Guide which include an “Artifact Sizing Guidelines” section summarizing “the recommendations on artifact sizing that will ensure optimal performance of the repository when the data sizes increase significantly“.

Foreword to the reader:

  • This post follows the “cheat sheet/how to” format I’ve used in earlier posts for CLM Reporting or OSLC-related topics. As a consequence, if you’re already familiar with this post (and know exactly what you’re looking for), you may want to navigate directly to the tables: JTS tableRTC tableRQM tableRRC/RDNG table.
  • If you’re interested by a similar content for Enterprise Modernization products (i.e. RTC EE, RDz, RD&T, RAA, etc.)  by IBM Rational, check this dedicated blog post.

Now back to the core of this post:

Tables

Tables are provided. OK. But what do we call a Limit and an Alert zone ?

  • Quantified data: the (maximum) number of….
  • Limit: a hard limit of the product. Meaning that you cannot go beyond this value.
  • Alert zone: based on experiences with customers, internal tests and development teams, it’s around these values that we start seeing performance issues. If you’re approaching these values, we’d suggest you monitor your system closely to detect any performance degradation before it becomes critical. WARNING: while provided figures are educated guesses and practical rules of thumbs, you could still find that your environment functions perfectly beyond these limits  (e.g. if your environment is particularly fine-tuned). In a similar way,  some intense CLM usage could show that these recommended values are too optimistic…

As a consequence, it’s important to understand this post is NOT an attempt for replacing existing resources (see the References section) that provide extensive views on CLM performance and tuning topics. We encourage administrators and project managers to read them as they both include finer-grain information and insist on the key aspect of not loosing the “bigger picture”.

What’s the use of the following tables then ?

Answer: they’re here to HELP YOU quickly figure out if you’ve reached some known CLM limitation or if you’re getting close to a threshold  (again: on the basis of a typical/average environment) requiring due monitoring of your environment.  To this regard, these tables are COMPLEMENTARY with existing resources and concentrate information ALREADY available but yet disseminated on multiple medias/sites/forum posts/etc.

What if… you can’t find what you’re looking for in the following tables ?

Answer: in such case, there MAY not be soft/hard limit on it. You should check the References section at the bottom of this post and check for the latest information (esp. from the CLM sizing guides).

Ξ

JTS
Quantified data Limit Alert zone Reference(s)
Concurrent user sessions  – 400-2000+ Purple Book
Jazz user id length  – 250 bytes Jazz.net forum

Ξ

RTC 
Quantified data Limit Alert zone Reference(s)
Concurrent user sessions  –  300-500+ (per CCM instance) Purple Book
   Example: “CLM Sizing Strategy” (v4.0.6 – April 2014)  –   – 100-600 concurrent users. See report/environment details.
   Example: “Performance Report” (v5.0 – June 2014) 1200 concurrent users. See report/environment details.
Planning – Work-items
   WIs in a plan (<= v2.0) 2048 Jazz.net article
   WIs in a plan (>=v2.0.0.1)  –  250-500+ (impacts plan display time + questions ability from user to grasp several 100s of WIs in one plan) Jazz.net forumJazz.net forumPurple Book, RTC 4.0.3 Plan performance improvement
   WIs in a project area/repository  – Jazz.net forum, Jazz.net article
   WI attachment size  50 MB  If increasing this value or systematically using large attachments: be aware of the possible impact on DB growth and CLM performance in general. See how to change this value in TechNote, Jazz.net forum
   WI “Estimate” attribute  1 year  – Jazz.net forum. A  presentation-enforced limit.
   WI custom attribute length > Small String (*) 250 bytes Jazz.net article (RTC v4.0)
   WI custom attribute length > Medium String (*) 1000 bytes Jazz.net article (RTC v4.0)
   WI custom attribute length > Large String (*) 32768 bytes Jazz.net article (RTC v4.0), Enhancement 160469
   WI custom attribute length > Medium HTML (*) 1000 bytes Jazz.net article (RTC v4.0)
   WI custom attribute length > Large HTML (*) 32768 bytes Jazz.net article (RTC v4.0)
Queries
   Query results
 1000 results Jazz.net forum. Note: this default value could be increased but be aware of the possible negative impact on usability / server performance.
Planning – Timelines
   Timelines  2048  (see recommended approach in the forum post’s answer) Jazz.net forum
SCM
   Files/folders in a single component (CLM 2011)  50K (split into multiple components if required) Jazz.net forumJazz.net article
   Files/folders in a single component (CLM 2012, RTC 5.0)  100K (split into multiple components if required) Jazz.net forumJazz.net article, Jazz.net forum,
   Suspended change-sets by individual user  300 (for not slowing down operations) Jazz.net article
   Components in workspaces and streams  500 (as tested by IBM) Jazz.net article , Task 176441 (in progress)
Build
  Build definitions associated to a build engine ( < v4.0.3) 2048 TechNote
OSLC
   oslc_cm.pageSize parameter (when querying work-items) 100 Jazz.net RFEJazz.net forum

(*): text-based

Ξ

RQM
Quantified data Limit Recommendation Reference(s)
Concurrent user sessions  –  100-150+ (per QM instance) Purple Book
   Example: “CLM Sizing Strategy” (v4.0.6 – April 2014)  – 350-500 concurrent users. See report/environment details.
   Example: “Performance Report” (v5.0 – June 2014) 1000 concurrent users. See report/environment details.
TER (Test Execution Record) name length  250  – Jazz.net forum
TCERs bulk generated from test plan wizard  500 Work-around article, RQM defect, WAS maxParamPerRequest
TCERs bulk changed/removed at once tbd  tbd Jazz.net forum, Jazz.net forum, WAS maxParamPerRequest
Records in a datapool / test data 2000  – Jazz.net enhancement
Character limit: Description field of a Lab Resource 250 Jazz.net enhancement
Number of categories defined on an artifact type 50 RQM defect, RQM defect
Feed entries per page ( < 4.0.4)  512  – Jazz.net forum, RQM defect
“Large Record Count” (SQL query result set generated by OOTB BIRT reports) ( >= 4.0.5)  –  10K TechNote, Jazz.net defect
Attachment size using UI Jazz.net enhancement
Attachment size using CLI ( Command-Line Interface) ( >= 4.0)  50 Mo Jazz.net article (for how to change this default value, see the Comments section). Note: if increased, be aware of the possible negative impact on usability / server performance.
TCERs runnable off-line and at once (>=4.0) 50 4.0 InfoCenter

Ξ

RRC-DNG/RDNG
Quantified data Limit Recommendation Reference(s)
Concurrent user sessions (< v4.0.1)  200+ Purple Book
 (>= v4.0.1)  400+ Purple Book
   Example: “CLM Sizing Strategy” (v4.0.6 – April 2014) 300-400 concurrent users. See report/environment details.
   Example:  “Performance Report” (v5.0 – June 2014) 400 concurrent users. See report/environment details.
Coexistence with DM (Design Manager) on the same box (in v.4.x and v5.0.x)  Incompatible  – Jazz.net forum (related to the converter component)
Instances of RM application per JTS (<= v4.0.6) 1 4.0.3 InfoCenter, Jazz.net article, Plan Item
RM Projects per RM application / JTS  200+ Jazz.net article
Number of undos in edit mode  20 TechNote
Number of displayable links (>= v4.0.1)
  • 60 (IE7)
  • 100 (other browsers)
Jazz.net forum
Number of artifacts selectable in the Artifact view 50 Enhancement 71080, Jazz.net forum
Using ReqIF
   Imports to DNG from DOORS (>= v4.0.1)
  • 5000 modules
  • 200K objects (total)
Jazz.net article, Jazz.net forum
   max depth supported for import 3 Jazz.net forum

References:

  1. CLM 2012 Sizing Report

  2. CLM 2011 Sizing Guide
  3. Rational Team Concert (RTC) 2.0 sizing guide
  4. Rational Team Concert 4.x sizing report for z/OS

  5. The Deployment wiki
  6. Jazz Performance: A Guide to Better Performance” by D. Toczala (Feb 2013). A.k.a the “Purple Book
  7. Sizing and tuning guide for RDNG (Rational DOORS Next Generation) 5.0

Acknowledment/Credit: thanks to the authors of the cited documents above and more generally to the Jazz community who collaboratively provides accurate information through library articles, forums questions & answer, etc..


Jazz/CLM ETLs performances: a view on release-to-release comparisons

August 1, 2013

Recently, I’ve had a deeper look at ETLs performances. This topic is sensible to our customer as full ETL load could last up to several days !

Based on work-items tracking activity and interactions with our development, support team and performance teams, I came up with a personal summary. I found it interesting to be shared with Jazz/CLM Administrators to get up to speed on performance of such ETLs.

Firstly, let’s start with the existing and customer-oriented information:

ETLs performance comparison between CLM versions 3.0.1.2 and 4.0.

Available in article “Rational solution for CLM 4.0 “Extract, Transform, and Load” Performance Report“.

It’s considering the “D1” topology (note: the article is little bit misleading as the provided URL link points directly… to the “E1” / Enterprise topology. I have notified the authors already).

A simplified summary (which should not prevent you from reading the full outcome of the ETL comparison) follows:

  • RTC ETLs: stable performance
  • RQM ETLs: 20% performance degradation (expected by dev team)
  • RM DM ETLs: significant improvement
  • Star ETL: no major issue

Secondly, let’s continue with the information provided for advanced readers

ETLs performance comparison between CLM versions 4.0.2 and 4.0.3

Available in [Plan Item 248546] “Ensure no ETL performance regressions are found when comparing CLM 2012 Mod 3 ETLs to 4.0.2“.

It’s again considering the “D1” topology.

Status (as for Aug. 1st, 2013) is “Done”. Navigating to the latest comments of the Discussion tab will show you available Excel spreadsheets (Java ETLs for RRDI, DM ETLs for Insight)  .

Finally, mentioning the plans for improving subsequent ETLs performance comparisons:

ETLs performance comparisons for forthcoming 4.x versions

Performance team have plans (be aware that plans are subject to change) to:

  • publish some automated ETL tests for later minor version of CLM 4.x
  • improve the analysis of performance data from the ETLs.

The related WIs are listed below:

Also citing some material of interest (but to a lesser extent from a customer perspective as these resources are part of the Jazz development wiki) as well:

Note: be aware the previous resources are subject to the following statement: “Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly“.

Finally, mentioning some related information (captured in the Deployment wiki)

…. and recalling a basic statement

If you’re concerned that the ETLs performances may have degraded after a particular CLM upgrade, you would need to keep in mind the amount of RTC WIs, RQM test cases, RRC artefacts, etc. has certainly increased since you last run a full ETLs load. This remark for avoiding comparing apples and oranges…


CLM 4.0 Enterprise Deployment tasks: Oracle DB, proxy, IHS, SSO setup and RRDI configuration

June 27, 2013

Sometimes it can get difficult to narrow down all pieces of information you need to perform some specific administrative task on your CLM/RTC/RQM/RQM/DM environment.

As part of the Jumpstart team, I regularly provide guidance to customers in the CLM area covering RTC, RQM, Enterprise deployments, upgrades, customizations, OSLC and Reporting. As questions about CLM Administration and Reporting show up frequently, I use to collect and keep preciously links I can refer to later with customers. Until very recently, my personal “Top 3” included the following resources:

Until… I found out that Rational Support had provided very useful videos in the CLM 4.0.1 Education Assistant

RRDI configuration
Prepare Linux libraries for RRDI (9 min video)
Install RRDI software (4 min video)
Create an Oracle database for the RRDI content store (9 min video)
Install Oracle 32-bit client software for RRDI (6 min video )
Configure Oracle client environment for RRDI (9 min video)
Preparation for RRDI setup process (7 min video)
Run RRDI setup wizard (9 min video)

 []

WebSphere and IBM HTTP Server configuration
Public URI and host file configurations (3 min video)
Install IBM WebSphere Application Server (WAS) Network Deployment (ND) (8 min video)
IBM HTTP Server (IHS) configuration and self-signed SSL certificate (8 min video)
Setup the IBM HTTP Server (IHS) admin server (6 min video )
Application server profile creation (8 min video)
Setup WebSphere Application Server (WAS) global security (11 min video)
Setup HTTP server proxy on RRDI application server instance (20 min video)
Add a second application server profile to the HTTP server proxy (19 min video)
Add a third application server profile to the HTTP server proxy (8 min video)

[]

CLM Solution Configuration
Oracle database user and table space creation (5 min video)
Configure profile JVM properties to support CLM WAR files (9 min video)
Deploy the CLM WAR files to the first server (6 min video)
Update the CLM server plug-in with the profile XML file changes (6 min video )
Virtual frame buffer requirement and creation for Linux Requirements Management Converter (4 min video)
Configure profile JVM properties and deploy CLM WAR files to second server (8 min video)
Configure Single Sign On (SSO) with HTTP server and WebSphere topology (5 min video)
Run Jazz Team Server (JTS) setup wizard (17 min video)

This is my new “Top 4” I wish I had complete a little earlier.

Thanks to Rational support  for providing these videos. Hope this blog will (modestly) provide the additional visibility this useful material deserves.

Note 1: while the title page of the Education assistant states it’s for “IBM Rational Quality Manager“, this content is more largely applicable (to RTC, RRC, etc.)

Note 2: updates for 4.0.1+ versions are not available at this time AFAIK.


Failover recovery and high-availability strategies in the Rational Solution for Collaborative Lifecyle Management (CLM)

June 27, 2012

Recently customers asked me about best practices for disaster/failover recovery and high-availability (HA) implementation for their CLM deployments.
This topic resonates particularly as the Rational Solution for Collaborative Lifecyle Management 2012 provides a new clustering feature. As you feel certainly curious and interested about it, I suggest you have a read at the end of this post for references.

In enterprise organizations, Business Impact Analysis is usually performed, critical functions/activities are identified and requirements in terms of :

are set as well.

But how to make sure your CLM deployment will comply with these “control points” ?
What are the solutions in terms of deployment architecture that could be used ?

To help you along that path, I’ve gathered some considerations and technical aspects you should be aware of :

A backup strategy first !
The need for putting a backup strategy in place shall always be kept in mind.
It’s basically the starting point of any disaster/failover recovery strategy.
As a result, check for the best online backup capability of your enterprise database vendor to produce frequent backups.
Drop the idea of using repotool -import/export commands for backup purpose :

  • they’re not ideal as server must be taken offline (which is not very handy for regularly scheduled backups) and Generally the off-line backup and restore facilities provided by an enterprise database
  • they were not designed for this purpose but instead for database vendor migration scenarios (and more rarely for upgrade scenarios e.g. when upgrading from CLM 2.x to CLM 2011).

The safe path here is to parse CLM 2012 InfoCenter and check for backup techniques : Backing up and restoring other supported databases

I also advise this article from Ralph Schoon: Backup the Rational solution for CLM. It provides valuable insights on what exactly to backup up and which ordering sequence should be respected.

You said high-availability ? Here’s a simple requirement
WebSphere Application Server must be used for any HA configurations, both for manual or automatic failover.

A CLM history : from manual failover to CLM 2012 Clustering…

Act I : in the v2.x times…

A manual failover strategy was available for RTC exclusively. The approach is depicted in Deploying RTC 2.0 on WAS for HA using idle standby :

  • backup server is running but isn’t connected to the database.
  • a couple other tweaks are required to ensure a safe failover.

Act II : in CLM 2011

Manual failover strategy is generalized to all CLM applications. Scott Rich’s article High availability and disaster recovery for Rational’s CALM products details both the cold and idle/hot standby solutions. Unless you really have aggressive RTO, the cold standby :

  • where the backup server is started but
  • where the Jazz applications (JTS/CCM/RM/QM) are stopped

appears as a simpler/safer solution than the hot/idle standby. Because :

  • the backup server needs fewer configuration tweaks and
  • the chances of a request accidentally activating the backup server is reduced

Note that the backup server installation is not intended to be run for extended periods in place of the primary server (which should be put back on-line asap).

For InfoCenter references, check :

Check also this guide for additional failure scenarios (and guidance) associated to virtualization, disk storage and network:

Act III : the clustering feature in CLM 2012

This new feature guarantees a server operating with no manual intervention in case of hardware/software failure on any of the server nodes. In other words, as long as one node is alive, the application is responding.

High Availability with Rational solution for Collaborative Lifecycle Management 2012 Clustering gives an overall picture of the technical implications of adopting the clustering solution.

It shall be noted that this new feature :

  • requires additional key from Rational Support.
  • provides horizontal scalability but primary focus is on high-availability only. In other words, performance scalability is out of its scope. From a practical point of view,  expect a 3 nodes configuration to get similar performances than with a non HA-enabled solution.

(CLM 2012 System Requirements for clustering) Related Information:


(Tip for Jazz Administration) : Hide port numbers from the public server URI

March 30, 2012

This article provides tips for administrators looking for hiding port numbers from the public server URL when configuring the Jazz server initially. It covers both Tomcat and WebSphere server configurations.

Hide port numbers from the public server URI

For eased user access or for later administration purpose, you may want CLM2011 applications to answer with URLs including no port. For example :

https://clm.example.org/ccm rather than https://clm.example.org:9443/ccm
or
https://ccm.example.org/ccm rather than https://ccm.example.org:9443/ccm
https://qm.example.org/qm rather than https://qm.example.org:9443/qm
https://rm.example.org/rm rather than   https://rm.example.org:9443/rm

Reminder

By default :

  • web servers use port 80 for non secure communications (over HTTP) and port 443 for secure communications (over HTTPS).
  • web browsers could access HTTP(S) servers configured this way without having to include the port number in the URL.

Once you have set the Public URI of your applications, you cannot change it afterwards. For more information, see Planning your URIs.

Procedure for a Tomcat server

Following steps guide you for this :

  1. In a text editor, open JazzInstallDir/server/tomcat/conf/server.xml.
  2. Search for the <Connector>element for the non-secure port.
    Note : by default, this <Connector> element is not commented out, has a comment above it that says
    <!– Define a non-SSL HTTP/1.1 Connector on port 9080 –>, and has a redirectPort attribute.
  3. Modify the value of the port attribute with port number 80 to use for non-secure connections.
  4. Modify the value of the redirectPort attribute with port number 443 to use for secure connections.
  5. After you have modified the <Connector> for the non-secure port, search for the <Connector> element for the secureport.
    Note: by default, this <Connector> element is not commented out and has the attribute secure=”true”.
  6. Modify the value of the port attribute with the port number 443 to use for secure connections.
  7. Save the server.xml file.
  8. Start the application server.

As a result, you should have a server.xml file looking like this :

<!– Define a non-SSL HTTP/1.1 Connector on port 80 –>
<Connector port=”80″

redirectPort=”443″
…./>
<!– Define a SSL HTTP/1.1 Connector on port 443 –>
<Connector port=”443″

scheme=”https”
secure=”true”
…./>
….

Procedure for a WebSphere Application Server

Following steps guide you for this :

    1. Start the application server.
    2. Log in to the Integrated Solutions Console as an application server administrator.
    3. Click Servers > Server Types > WebSphere Application Servers > server_name > Ports.
    4. Modify the value for WC_defaulthost with port number 80 to use for non-secure, HTTP connections. Note : make sure this value is not used already.
    5. Modify the value for WC_defaulthost_secure with port number 443 to use for secure, HTTPS connections. Note : make sure this value is not used already.
    6. Click Apply, then click “Save directly to the master configuration”.
As a result, “Ports” should now appear this way :
  1. Stop, and then restart the application server.

Next step : Configure the Jazz Team Server

You can now use the setup wizard to configure the Jazz Team Server. Make sure you use the public URL with no port ! For more information, see Running the setup wizard.

N.B. (for Linux users): all ports below 1024 are “privileged” ports (only root may open a privileged port).

General notes

Here, assumption is made that ports 80 and 443 are available on the machine. This might not be the case :

  • if another server is using one of these ports, or
  • if your CLM applications (JTS, CCM, QM and/or RM) are running on different servers but are at least partly collocated on the same machines.

As mentioned in the Summary, only an initial configuration of the Jazz server was considered in this article.

If you must change the port number your Jazz server is responding on – while also maintaining URL stability -, consider using a reverse proxy to route requests using the old port number to the appropriate new location.

For more information

  1. Moving Jazz Servers and URI Stability with CLM 2011
  2. Change the default port
  3. WebSphere 7 Information Center
  4. Using virtual host names in your topology
  5. Using a reverse proxy in your topology
  6. Configuring Enterprise CLM Reverse Proxies, Part 1: Understanding Reverse Proxy
  7. Configuring Enterprise CLM Reverse Proxies, Part 2: WebSphere and IHS Plugin method

%d bloggers like this: