Posted by: ndandala | July 21, 2009

Transaction Processor Failures


Symptom or Error Message

The Transaction Processor Siebel Remote server tasks may fail under certain conditions which will cause a backlog in the transactions being written out to the SIEBEL_ROOT\DOCKING\TXNPROC directory.

Symptoms of a Transaction Processor failure are that the Task State field reports that the task has Exited; usually an additional error appears in the Status field on the Server Tasks view, such as:

Task State: Exited

Status: TXP-00011: Error: Unable to start APIs

In Siebel applications version 7.7 and higher, the Server Tasks view is Administration – Server Management > Tasks view; in earlier versions, it is Server Administration > Tasks.

NOTE: Exact errors that appear in the Status field may vary with the cause of failure.

To generate more detailed trace information in the Transaction Processor log for Siebel applications versions 5.x and 6.x, run the Transaction Processor task with the following parameters set:

Sql flag = 3

Trace flag = 7

Error flag set = 7

For Siebel applications version 7.x, review FAQ 1944, regarding how to enable component tracing for the Transaction Merger component and read the log files, and FAQ 1894, regarding how to enable SQL-based event logging within the Siebel Remote components.

Cause

At a high level there are three main causes for Transaction Processor failures:

            1. Connectivity and RDBMS issues

            2. Problems with reading from the transaction log and writing to .dx files

            3. Problems accessing cache files and docking directories

Diagnostic Steps

The first step in diagnosing a Transaction Processor failure issue is to try and determine whether the failure was a one off occurrence or a persistent failure. This is easy to achieve by simply trying to restart the Transaction Processor. If the Transaction Processor starts and continues processing without further error then it is likely that the cause may be poor network connectivity, database connectivity or another intermittent system issue at the time of failure. You can then inspect the log file to see why the Transaction Processor failed. If the Transaction Processor fails again after restarting then it is likely that you have a persistent failure which will require some action to be taken before it can restart and continue without failing. Typically the first step is to increase the tracing as detailed above and then restart the Transaction Processor and review the more detailed log for the commonly seen issues as detailed below.

            1. Connectivity and RDBMS Issues

 

Review the Transaction Processor log file for RDBMS and network related errors. The errors below are examples of RDBMS errors that have been seen in Transaction Router log files:

Oracle

            • ORA-03113: end-of-file on communication channel

            • ORA-03114: not connected to ORACLE

            • ORA-01041: internal error. hostdef extension doesn’t exist

            • ORA-04031: unable to allocate 4096 bytes of shared memory (“shared pool”,”unknown object”,”sql area”,”BAMIMA: Bam Buffer”)

            • ORA-00020 maximum number of processes [500] exceeded.

 

DB2

            • [IBM][CLI Driver][DB2/6000] SQL0301N The value of a host variable in the EXECUTE or OPEN statement cannot be used because of its data type. SQLSTATE=07006

            • [IBM][CLI Driver] CLI0125E Function sequence error. SQLSTATE=S1010

 

Microsoft SQL Server

            • [Microsoft][ODBC SQL Server Driver][SQL Server]A time out occurred while waiting for memory resources to execute the query. Rerun the query.

            • [Microsoft][ODBC SQL Server Driver][TCP/IP Sockets]General network error. Check your network documentation.

            • [Microsoft][ODBC SQL Server Driver]String data, right truncation

            • [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid column name ‘X_TEST_NUM’.

            • [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name ‘Siebel.S_APPLICATION_VERSION’.

 

            2. Problems with reading from the transaction log and writing to .dx files

 

Review the Transaction Processor log file for Siebel errors. The errors below are examples of Siebel errors that have been seen in Transaction Processor log files:

            • SBL-DCK-00123: Error opening file Server\Docking\TxnProc\xxxxxxxx.log for read

            • SBL-DCK-00147: Error reading transaction from database

            • SBL-DCK-00148: Error reading operation from database

            • Master Transaction log table (S_DOCK_TXN_LOG) buildup 

            3. Problems accessing cache files and docking directories 

Review the Transaction Processor log file for Siebel errors. The errors below are examples of Siebel errors that have been seen in Transaction Processor log files:

            • SBL-CSC-00119: Block number xxx exceeds file size (yyy blocks)

            • SBL-DCK-00250: A newer version of the dobjinst.dbf must be rebuilt by starting transaction processor with parameter “TSDbRecreate” set to “TRUE”

 Solution

            1. Connectivity and RDBMS issues 

RDBMS errors are not generated by the Siebel application – they are reported in the log files to aid with troubleshooting. The examples given above have all been caused by either underlying network issues or incorrect database configuration or tuning. It is recommended to involve your DBA as many RDBMS errors can be resolved by database tuning alone, this should be done in conjunction with the recommendations made in Siebel Bookshelf version 7,7 > Installation Guide for the appropriate operating system > Guidelines for Configuring the RDBMS. If the RDBMS error points to an underlying network issue then you should also involve appropriate networking resources to investigate the cause further.

In the case of database or network connectivity issues here are some initial checks that should be performed:

            • Verify if the database server is accessible from the Siebel application server machine by executing the ping command from a command prompt on the application server:

> ping %hostname_of_database_server%

• Verify the database server is running.

• Verify ODBC connection strings are configured correctly. Refer to FAQ 1113 for details on how to test connectivity through ODBC data sources created by Siebel.

 

NOTE: Not all RDBMS errors can be solved by database tuning alone. If the error generated indicates a data integrity issue, please raise a Service Request with Siebel Technical Support with an associated Transaction Processor trace file with increased logging enabled. Do not attempt to modify data in Siebel tables via direct SQL as this is not supported unless instructed to do so by Siebel Technical Support.

            2. Problems with reading from the transaction log and writing to .dx files 

The purpose of the Transaction Processor is to pre-process the outstanding transactions present in the S_DOCK_TXN_LOG table (master transaction log) and write these transactions out to xxxxxxxx.dx files in the SIEBEL_ROOT\DOCKING\TXNPROC directory. These transactions will have visibility information written against them indicating the visibility strength of the transaction and whether or not the transaction is classed as a “visibility event”. This information can then be used by the Transaction Routers to make routing decisions for the mobile user base. Under certain conditions the Transaction Processor will encounter errors trying to read or write xxxxxxxx.dx files as part of this process.

 

SBL-DCK-00123: Error opening file Server\Docking\TxnProc\xxxxxxxx.log for read

The above error message may occur in a number of different scenarios and is generally triggered by a corruption of the dock object cache file which the Transaction Processor uses to cache visibility information for dock objects which it is processing. The cache file itself is called dobjinst.dbf (Dock Object Instance.database file) and contains a list of dock objects, visibility strengths, ROW_Ids for primary tables and other information which it caches from the Siebel Repository. While the Transaction Processor is processing transactions it maintains some of this information in the xxxxxxxx.log file for writing to the cache file once it has finished processing the xxxxxxxx.dx file.

The error message is generated when the Transaction Processor is expecting to read a given .log file based on information maintained from a previous iteration. If the Transaction Processor cannot find the expected file, it will shut down with the above error message as it is in a non-stable state.

For a corruption of the dobjinst.dbf file to occur either a fatal error will have occurred causing the Transaction Processor to crash, or more likely, a change will have occurred at the database level. This error is therefore commonly found following a variety of database recovery scenarios or in situations where a Siebel Enterprise has been switched from one database to another.

If a Transaction Processor does fail with the above error message then the first course of action should be to attempt a rebuild of this file. If the dobjinst.dbf file becomes corrupt then restart the Transaction Processor component with the TS DB Recreate parameter set to TRUE. An attempt will therefore be made to rebuild the file and to bring it in line with the data and transactions in the current database. The exact mechanism to set the parameter and restart the component will differ between versions and users should consult the appropriate version of the Siebel Bookshelf for precise instructions. In Siebel Bookshelf version 7.7, refer to Siebel Remote and Replication Manager Administration Guide > Implementing Siebel Remote Server > Starting Siebel Remote Server Components.

If, having worked through the available courses of action, the Transaction Processor still fails repeatedly with the same error message, this indicates that the corruption is occurring regularly rather than at a given point in the past and therefore a simple rebuild of the file will not be sufficient. Customers should log a service request to Siebel Technical Support and provide the information requested in the Service Request Information section at the end of this guide. Additionally, include the following Component Parameter TS Db Recreate = True when generating the detailed Transaction Processor log file.

SBL-DCK-00147: Error reading transaction from database

If the above error message is reported by the Transaction Processor at the time of a failure it is indicative that a problem of some sort was encountered while trying to read transactions from the master transaction log. Unlike a SBL-DCK-00148 error, this error message does not indicate a corruption of a transaction in the table but that a related error has occurred while attempting to read the transaction. When a SBL-DCK-00147 error is returned the application will attempt to provide further clarification of the root cause of the error via the Additional Message tracing which will also be found in the log files. This can often contain errors returned from lower level protocols, drivers, or the database itself.

As mentioned above a SBL-DCK-00147 error is indicative of a problem with the Transaction Processor but one which stems from a third party process or driver. This could be a database access problem, disk or memory resource problem, or an operating system problem. when errors of the above type are encountered it is imperative that additional logging be provided and efforts made to identify a more specific root cause.

 

Efforts should initially be made to identify the root cause via the information provided in the Additional Message tracing. With the extra information that this provides analyze the cause of this error message, and where possible review related postings on Siebel SupportWeb for further information.

If after having completed the above, the root cause of the behavior is still unresolved, log a service request to Siebel Technical Support and provide the information detailed in the Service Request Information section at the end of this guide.

SBL-DCK-00148: Error reading operation from database

If the above error message is ever reported within a Transaction Processor log file, it will result in the process failing. It indicates that the Transaction Processor has encountered a transaction which it is unable to read from the master transaction log. This is probably the most common error message reported by the Transaction Processor and generally requires the intervention of Siebel Technical Support in order to assist with correction of the transaction. The transactions being read by the Transaction Processor are stored in the master transaction log in an encoded format which is beyond the scope of this document to describe. If the Transaction Processor encounters coding which it cannot read or mismatches in the formatting, it will fail and, by default, provide some detail of the failing transaction.

The most common cause of transaction corruption is the introduction of characters which cannot be processed through the codepage in use on the application server that the Transaction Processor is based on. In environments which can span a variety of time zones and languages it is not unusual to find foreign characters within a transaction due to them having been copied from other documents straight into the application by an end user. For further information please refer to Alert 290 on Siebel SupportWeb, regarding the SBL-DCK-00148 error caused by characters from a different code page being present in the transaction log.

Review the information provided in the Transaction Processor log file (TxnProc_xxx.log). In general when transaction based errors occur the process will provide decoded details of the problem transaction within this log file, example:

Trace Trace 3 2003-03-11 10:02:28 Txn Type : S

Trace Trace 3 2003-03-11 10:02:28 Txn Id : 23176452

Trace Trace 3 2003-03-11 10:02:28 Src Node : 1

Trace Trace 3 2003-03-11 10:02:28 Created By : 1-KY39

Trace Trace 3 2003-03-11 10:02:28 DLog Row Id: 1-AVLGF

Trace Trace 3 2003-03-11 10:02:28 OperType : Update

Trace Trace 3 2003-03-11 10:02:28 TableName: S_PARTY

Trace Trace 3 2003-03-11 10:02:28 Txn RowId : J3W-7TZ

Trace Trace 3 2003-03-11 10:02:28 Txn LastUpdBy : 1-KY39

Trace Trace 3 2003-03-11 10:02:28 Txn LastUpd : 2003-03-11 04:26:16

Trace Trace 3 2003-03-11 10:02:28 Txn ModNum : 3

Trace Trace 3 2003-03-11 10:02:28 File Table : No

Trace Trace 3 2003-03-11 10:02:28 unhandled: e2*O10*

Refer to FAQ 1423 on Siebel SupportWeb if the Transaction Processor log does not detail the exact corrupt transaction as shown above. This FAQ this details an alternative way to find the corrupt transaction.

 

Once the corrupt transaction has been determined in order to gather further detail about the transaction record which has caused the Transaction Processor to fail, use a query similar to that shown below and include the referenced transaction details shown from the transaction log above or the TXN_ID found from FAQ 1423:

SELECT *

FROM <table_owner>.S_DOCK_TXN_LOG

WHERE ROW_ID = ‘<Dlog Row Id>’

AND TXN_ID = ‘<Txn Id>’

Example:

SELECT *

FROM SIEBEL.S_DOCK_TXN_LOG

WHERE ROW_ID = ‘1-AVLGF’

AND TXN_ID = ‘23176452’

If the root cause of the corrupt transaction has not been resolved by the supported methods, log a service request to Siebel Technical Support and provide the information detailed in the Service Request Information section at the end of this guide, with the results of the SELECT statement from the transaction log as detailed above. Do not attempt to modify data in Siebel tables via direct SQL as this is not supported unless instructed to do so by Siebel Technical Support.

Master Transaction log table (S_DOCK_TXN_LOG) buildup

The above description is not in itself an error message which will be found at any point in time within the Transaction Processor log file, but more a symptom which will be noticed by Database Administrators when monitoring the server database. Over time the number of rows within the S_DOCK_TXN_LOG table will be seen to be steadily increasing. In some scenarios this can occur for long periods of time and lead to space problems within the database.

There are a number of root causes for the above behavior, but the most common stem from invalid or expired docking status information being held for the Siebel Remote within the S_DOCK_STATUS table. The Transaction Processor will hold on to transactions within this table until it has received confirmation that they have been written out to correctly to xxxxxxxx.dx files for all of the Transaction Processors which require them. This transaction is termed the LOWSCANMARK of a Transaction Processor. If records exist for a non-end-dated Transaction Processor which is not processing transactions this can cause a very low LOWSCANMARK to be maintained, effectively the last transaction that was ever processed by this redundant Transaction Processor. This most often occurs in environments where databases have been migrated from one Enterprise to another, leading to a Transaction Processor reference for the previous Enterprise which will never run in the new one. Until this Transaction Processor is end-dated it will be believed to be Active and will therefore be included in any LOWSCANMARK calculations. In order to confirm that this behavior is taking place you should run the following query at regular intervals. If the TXN_ID value returned does not increase, it is likely that the user has encountered the above behavior and should take action at the earliest opportunity:

SELECT MIN(TXN_ID)

FROM <table_owner>.S_DOCK_TXN_LOG

If a steady increase in the number of transactions within the S_DOCK_TXN_LOG is occurring, ensure that no unnecessary Transaction Processors exist within the current Enterprise. This can be done via a review of the Siebel Remote Administration > Processor Status screen which will list any Transaction Processors currently registered within the Enterprise. If, after end-dating any unnecessary Transaction Processors through this view, the backlog is still increasing, work through the various steps outlined within Troubleshooting Steps 38 on Siebel SupportWeb. This

 

article provides a great detail of information on the above error and outlines a number of other courses of action.

If after having completed the above, the root cause of the behavior is still unresolved, log a service request to Siebel Technical Support and provide the information detailed in the Service Request Information section at the end of this guide.

            3. Problems accessing cache files and docking directories

 

Most of the Siebel application processes running on a Siebel Server will, at some point in time, make reference to cache files as part of their processing. Cache files have been implemented in order to increase the performance of these components, in particular that of the Siebel Remote components which normally run continuously.

It is possible that under certain circumstances these cache files can become corrupt or unreadable and in those situations the error messages which are reported below are likely to be found.

SBL-CSC-00119: Block number xxx exceeds file size (yyy blocks)

The above error is indicative of a corruption of the cache file (dobjinst.dbf) which can be triggered by a wide range of other errors. In particular disk crashes, lack of disk space, network failures, database level errors are all known to cause the Transaction Processor to crash, triggering a corruption of this file. Upon restart the above error message could be reported. In order to resolve this behavior the cache file should be rebuilt using the parameters available when executing the task.

SBL-DCK-00250: A newer version of the dobjinst.dbf must be rebuilt by starting transaction processor with parameter “TSDbRecreate” set to “TRUE”

The above error message is similar to that highlighted earlier but does not necessarily indicate that the cache file is corrupt, merely that it is out of synch with the current repository and therefore it must be rebuilt.

To attempt to resolve both SBL-CSC-00119 and SBL-DCK-00250 perform the following actions:

For Siebel Applications Version 6.x

From the Screens > Server Administration > Component > Component Parameters set the following parameters:

TS Block size = 65536

TS Cache size = 4096

Then navigate to Screens > Server Administration > Servers > Server Tasks and ensure that no Transaction Processor tasks are currently running. Start a new task with the following parameters set:

TS DB Recreate = True

All subsequent Transaction Processor tasks should be started with the TS Db Recreate parameter set to False.

 

For Siebel Applications Version 7.x

            • In Siebel applications version 7.0.x and 7.5.x, navigate to Server Administration > Component > Component Parameters.

            • In Siebel applications version 7.7.x and higher, navigate to Administration – Server Configuration > Servers > Components > Parameters.

 

Set the following parameters:

TS Block size = 65536

TS Cache size = 4096

TS DB Recreate = True

            • In Siebel applications version 7.0.x and 7.5.x, navigate to Server Administration > Servers > Server Components.

            • In Siebel applications version 7.7.x and higher, navigate to Administration – Server Management > Servers > Component Group.

 

Ensure that no Transaction Processor tasks are currently running. Shut down and restart the Transaction Processor component to rebuild the file, and reset the TS DB Recreate parameter to False against the Component Parameters. This will ensure that subsequent Transaction Processors will not attempt to rebuild this cache file.

If after having completed the above, the root cause of the behavior is unresolved, log a service request to Siebel Technical Support and provide the information detailed in the Service Request Information section at the end of this guide.

Service Request Information

If Transaction Processor failures are still occurring after following the recommendations above and the other referenced Alerts, FAQs and Support Web postings, log a service request to Siebel Technical Support providing as much information as possible about the failure. Always include the information below.

            1. All TxnProc_xxx.log files from the SIEBEL_ROOT%\LOG directory. These will provide details of the current status of the Transaction Processor.

            2. The <Enterprise>.<Server>.log

            3. Restart the Transaction Processor ensuring increased tracing is in place as shown below:

 

Siebel Version 6.x

Component Parameters – Error Flags = 7; SQL Trace Flags = 3; Trace Flags = 7;

Siebel 7.x

Component Parameters – Error Flags = 7; SQL Trace Flags = 3; Trace Flags = 7;

Component Event Configuration – Trace = 4; Generic Log = 4; Tall/Skinny = 4; SQL Summary = 4

Advertisements

Responses

  1. Hi,

    This article covers all RDBMS issues. It is excellent and useful.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: