Recently, I’ve had a deeper look at ETLs performances. This topic is sensible to our customer as full ETL load could last up to several days !
Based on work-items tracking activity and interactions with our development, support team and performance teams, I came up with a personal summary. I found it interesting to be shared with Jazz/CLM Administrators to get up to speed on performance of such ETLs.
Firstly, let’s start with the existing and customer-oriented information:
ETLs performance comparison between CLM versions 18.104.22.168 and 4.0.
Available in article “Rational solution for CLM 4.0 “Extract, Transform, and Load” Performance Report“.
A simplified summary (which should not prevent you from reading the full outcome of the ETL comparison) follows:
- RTC ETLs: stable performance
- RQM ETLs: 20% performance degradation (expected by dev team)
- RM DM ETLs: significant improvement
- Star ETL: no major issue
Secondly, let’s continue with the information provided for advanced readers
ETLs performance comparison between CLM versions 4.0.2 and 4.0.3
Available in [Plan Item 248546] “Ensure no ETL performance regressions are found when comparing CLM 2012 Mod 3 ETLs to 4.0.2“.
It’s again considering the “D1” topology.
Status (as for Aug. 1st, 2013) is “Done”. Navigating to the latest comments of the Discussion tab will show you available Excel spreadsheets (Java ETLs for RRDI, DM ETLs for Insight) .
Finally, mentioning the plans for improving subsequent ETLs performance comparisons:
ETLs performance comparisons for forthcoming 4.x versions
Performance team have plans (be aware that plans are subject to change) to:
- publish some automated ETL tests for later minor version of CLM 4.x
- improve the analysis of performance data from the ETLs.
The related WIs are listed below:
- [Task 265094] “Improve the analysis of ETL performance data“. Status (as for Aug. 1st, 2013) is “Resolved”. Navigating to the Links tab will show you available Excel spreadsheets. It’s again considering the “D1” topology.
- [Task 267133] “Implement additional performance benchmarking builds to integration into the continuous delivery automation“. Status (as for Aug. 1st, 2013) is “New”.
Also citing some material of interest (but to a lesser extent from a customer perspective as these resources are part of the Jazz development wiki) as well:
- [Wiki] https://jazz.net/wiki/bin/view/Main/CLM301Performance
- [Wiki] https://jazz.net/wiki/bin/view/Main/CLMBaselines2012
Note: be aware the previous resources are subject to the following statement: “Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly“.
Finally, mentioning some related information (captured in the Deployment wiki)
- Long-running ETLs with error(s)
- How to read ETL log files
- Why do my OOB ETLs take so long to run?
- Why do my custom ETLs take so long to run?
- Long-running ETLs with error(s)
- Long-running ETLs without error(s)
- ETL and CLM Upgrade
…. and recalling a basic statement
If you’re concerned that the ETLs performances may have degraded after a particular CLM upgrade, you would need to keep in mind the amount of RTC WIs, RQM test cases, RRC artefacts, etc. has certainly increased since you last run a full ETLs load. This remark for avoiding comparing apples and oranges…