Oracle GoldenGate 12c: Troubleshooting and Tuning Training

1. Gathering Evidence

• Acquiring confidence using the standard Oracle GoldenGate tools normally used to interact with the software:
• GGSCI
• Error Logs, Process Reports, Discard File, System Logs
• Competently using the tools mentioned above to gather evidence about alleged issues occurring to an Oracle GoldenGate environment

2. Oracle GoldenGate Troubleshooting Tools

• Accessing knowledge Documents - Mailing lists, Oracle GoldenGate forums, etc.
• Consulting the Documentation
• Getting familiar with the ShowSyntax parameter
• Getting familiar with the logdump utility

3. Basic and Startup Problems

• Getting familiar with basic problems
• Developing the ability to deal with Extract and Replicat startup issues
• Verifying file names, file permissions, consistency between Extract and Replicat group names and their supporting files
• Verifying consistent naming convention when defining trail files and their reference in the Extract/Replicat parameter files
• Verifying that the correct software version has been installed
• Verifying that the Oracle GoldenGate manager is running, that the group name which fails to start exists and is not misspelt
• Verifying that the the parameter file exists, has the same name as the Extract/Replicat group, and is accessible by Oracle GoldenGate
• Introducing the use of CHECKPARAMS to verify that the parameter syntax is correct

4. Data Extraction Problems

• Developing the ability to deal with and troubleshoot data extraction issues:Extract is slow - Extract abends - Data pump abends
• Analyzing various causes of a slow Extract startup
• Tracing may be enabled, activity logging may be enabled, many columns listed in tables being replicated
• Introducing the use of the CacheMgr parameter to control the virtual memory and temporary disk space that are available for caching uncommitted transaction data
• In-depth analysis of abending Extracts: is Extract unable to open the transaction logs? - Checking to see if the disk is full -Is an archive log missing?
• In-depth analysis of abending data pumps: Does the data pump abend with error number 509 ("Incompatible record")?Have the source tables been changed? Is the Extract abending due to data manipulation?
• Was the primary Extract or the data pump stopped, re-added, or restarted without resetting the data pump’s read position in the local trail?

5. Data Replication Problems

• Developing the ability to deal with and troubleshoot data replication issues:
• Analyzing the various causes of a "stuck" Replicat: Does "Stats Replicat" return no statistics? Do successive "Info Replicat" commands indicate that the Replicat read checkpoint is not moving?
• Verifying that the Replicat is reading the trail to which Extract is writing
• Analyzing the reasons why a Replicat abends: Is Replicat unable to locate a trail? Is Replicat unable to open a trail file? Was "Add TranData" issued for each table?

6. Missed Transactions

• Developing the ability to deal with and troubleshoot missed transactions involving issues such as: Tables that are not properly specified - Missing" Add TranData" command - LOB data not replicating
• Analyzing the reasons why transactions are not being processed: Were the tables included in Table and Map statements? Are fully qualified names (owner.table) used for both Table and Map?
• 'Single quotes' on object names imply case-sensitive names for some databases (including the Oracle database)
• Are the transactions coming out of the source system properly?

7. Mapping and Synchronization Problems

• Developing the ability to deal with and troubleshoot mapping problems, such us Fetching from the source data fails - Table and column names not matching between the source and the target
• Analyzing the causes of various mapping problems:Is Extract returning fetch-related errors? Was the row containing the data deleted?
• Has the undo retention expired?
• Is "NoUseLatestVersion" for "FetchOptions" being used?
• Do the source and target tables exist in the databases? Are they specified correctly in the Table or Map statement? Are their names fully qualified (schema.table)?
• Do table and column names contain only supported characters?
• Are WHERE clauses comparing different data types?
• Is KeyCols being used? Are both source and target KeyCols being used? Corresponding source and target columns must match

8. SQLEXEC, File-Maintenance, and Other Problems

• Developing the ability to deal with and troubleshoot
• SQLEXEC problems - File-maintenance problems - Time differences - Too many open cursors
• File-Maintenance Problems: Discard File Is Not Created - Discard File Is Too Big - Trail File Is Too Big
• Miscellaneous Problems: Time Differences - INFO Command Shows Incorrect Status

9. "Classic" Performance Tuning

• Identifying causes of slow startups
• Monitoring lag and statistics
• Tuning Manager
• Identifying network and disk bottlenecks
• Tuning "Classic" Extract
• Tuning "Classic" Replicat

10. "Integrated" Performance Tuning

• Describing architecture and advantages of the new Integrated capture
• Describing architecture and advantages of the new Integrated delivery
• Familiarizing with the new V$ tables holding real time Oracle GoldenGate statistics

11. "Integrated" Capture and Delivery Troubleshooting

• Familiarizing with Troubleshooting Aids and Tools for Integrated Replication
• Describing Oracle GoldenGate Healthcheck for Integrated Replicat
• Describing the Oracle Database 12c GoldenGate AWR report