Posted by: ndandala | April 14, 2009

Transaction Router Failures


Symptom or Error Message

The Transaction Router Siebel Remote server task may fail under certain conditions which will cause a backlog in the transactions to be routed to Siebel Remote mobile users and regional nodes. Symptoms of a Transaction Router failure are usually an error on the Status field on the Server task applet, and the Task State field would report that the task has exited with an error:

Task State: Exited with error

Status: ERROR: 1 client(s) failed with errors, 0 client(s) successful

To generate more detailed trace information:

            • In Siebel 7, follow the instructions detailed in FAQ 1944, regarding component tracing and log files, and FAQ 1894, regarding SQL-based event logging.

            • In Siebel applications versions 5.x and 6.x, run the Transaction Router task with the following parameters set:

Sql flag = 3

Trace flag = 7

Error flag set = 7

 Cause

There are three main causes for Transaction Router failures:

            1. Connectivity and RDBMS behaviors

            2. Errors when reading from and writing to .dx files

            3. Errors when accessing cache files and docking directories

 Diagnostic Steps

The first step in diagnosing a Transaction Router failure issue is to try to determine whether the failure was a one-time occurrence or a persistent failure. This is easy to achieve by simply trying to restart the Transaction Router. If the Transaction Router starts and continues processing without further error then it is likely that the cause may be poor network connectivity, database connectivity, or another intermittent system issue at the time of failure. You can then inspect the log file to see why the Transaction Router failed. If the Transaction Router fails again after restarting then it is likely that you have a persistent failure which will require some action to be taken before it can restart and continue without failing. Typically the first step is to increase the tracing as detailed above, then restart the Transaction Router and review the more detailed log for the commonly seen issues as detailed below.

            1. Connectivity and RDBMS Behaviors

Review the Transaction Router log file for RDBMS and network related errors. The errors below are examples of RDBMS errors that have been seen in Transaction Router log files:

Oracle

ORA-01000: maximum open cursors exceeded

ORA-04031: unable to allocate 4096 bytes of shared memory

ORA-03113: end-of-file on communication channel

ORA-03114: not connected to ORACLE

DB2

[IBM][CLI Driver][DB2/SUN] SQL0437W Performance of this complex query may be sub-optimal. Reason code: “6”. SQLSTATE=01602

[IBM][CLI Driver][DB2/6000] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code “68”. SQLSTATE=40001

[IBM][CLI Driver][DB2/6000] SQL0973N Not enough storage is available in the “APP_CTL_HEAP” heap to process the statement. SQLSTATE=57011

MS SQL Server

[MERANT][ODBC Siebel Database driver][SQL Server]ct_results(): network packet layer: internal net library error: Net-Library operation terminated due to disconnect

[MERANT][ODBC Siebel Database driver][SQL Server]ct_cmd_alloc(): user api layer: external error: The connection has been marked dead.

[Microsoft][ODBC SQL Server Driver][SQL Server]SHUTDOWN is in progress. Log off

[Microsoft][ODBC SQL Server Driver][TCP/IP Sockets]ConnectionRead (recv()).

[Microsoft][ODBC SQL Server Driver][TCP/IP Sockets]ConnectionWrite (send())

[Microsoft][ODBC SQL Server Driver][TCP/IP Sockets]General network error             

[Microsoft][ODBC SQL Server Driver]Communication link failure

2. Errors When Reading from and Writing to .dx Files

Review the Transaction Router log file for Siebel errors. The errors below are examples of Siebel application errors that have been seen in Transaction Router log files:

SBL-DCK-00142: Error opening transaction file C:\SIEBEL\docking\TXNPROC0005115.dx for read

Unable to read an item from the User Txn Log file.

SBL-TXR-00026: Error in the main function

NOTE: In Siebel applications prior to version 7.5.3, the error numbers are the same without the SBL prefix, namely DCK-00142 and TXR-00026.

3. Errors When Accessing Cache Files and Docking Directories

Review the Transaction Router log file for Siebel errors. The errors below are examples of Siebel errors that have been seen in Transaction Router log files:

SBL-CSC-00119: Block number xxx exceeds file size (yyy blocks)

SBL-CSC-00213: Invalid visibility database. Shutdown and restart *all* routers to rebuild

SBL-DCK-00123: Error opening file C:\SIEBEL\docking\J_LIBAIRE0000091.log for read

SBL-DCK-00250: A newer version of the dobjinst.dbf must be rebuilt by starting transaction processor with parameter “TSDbRecreate” set to “TRUE”

SBL-CSC-00128: Cannot get exclusive access to Tall/Skinny database for node %NODE%

SBL-CSC-00103: Error reading 204 bytes from dbfile at 0 (0 bytes read)

SBL-CSC-00202: An attempt was made to seek past the end of a data block

SBL-CSC-00301: Invalid value 0 for argument pVisDataInfo->createLatch

SBL-CSC-00120: Not enough space in buffer cache

NOTE: In Siebel applications prior to version 7.5.3, the error numbers are the same without the SBL prefix, namely CSC-00119, CSC-00213, DCK-00123, DCK-00250, CSC-00128, CSC-00103, CSC-00202, CSC-00301, and CSC-00120.

 

 

Solution

            1. Connectivity and RDBMS Behaviors

RDBMS errors are not generated by the Siebel application, but they are reported in the log files to aid with troubleshooting. The examples given above have all been caused by underlying network issues, incorrect database configuration, or incorrect tuning. It is recommended to involve your DBA, because many RDBMS errors can be resolved by database tuning alone. Database tuning should be done in conjunction with the recommendations made in Siebel Bookshelf.

            􀁻 In Siebel Bookshelf version 7.7 > Siebel Installation Guide for (Microsoft Windows or UNIX): Servers, Mobile Web Clients, Tools, refer to the following chapter:

            􀂃 Guidelines for Configuring the RDBMS (Windows)

            􀂃 Guidelines for Configuring the RDBMS (UNIX)

            􀁻 In Siebel Bookshelf version 7.5.3 > Siebel Server Installation Guide for (Microsoft Windows or UNIX), refer to the appropriate chapter for your RDBMS:

Microsoft Windows

            􀂃 Creating the DB2 Universal Database for Windows and UNIX

            􀂃 Creating the Microsoft SQL Server Database

            􀂃 Creating the Oracle Database

            UNIX

            􀂃 Creating the DB2 Universal Database for Windows and UNIX

            􀂃 Creating the Oracle Database

            􀁻 In Siebel Bookshelf version 7.0.x > Siebel Server Installation Guide for (Microsoft Windows or UNIX) review the appropriate chapter for your RDBMS on Creating and Optimizing the database for Siebel eBusiness Applications.

                        If the RDBMS error points to an underlying network issue, you should also involve appropriate networking resources to investigate the cause further.

In case of database or network connectivity issues here are some initial checks that should be performed:

            a. Verify that the database server is accessible from the Siebel application server machine by executing the ping command from a command prompt on the application server:

> ping %hostname_of_database_server%

b. Verify that the database server is running.

c. Verify that ODBC connection strings are configured correctly. Please refer to FAQ 1113 on Siebel SupportWeb for details on how to test connectivity through ODBC data sources created by Siebel

                        NOTE: Not all RDBMS errors can be solved by database tuning alone. If the error generated indicates a data integrity issue, create a Service Request with Siebel Technical Support and provide an associated Transaction Router trace with increased

 

 

                        logging enabled. Do not attempt to modify data in Siebel application tables using direct SQL as this is not supported unless you are explicitly instructed to do so by Siebel Technical Support.

            2. Errors When Reading from and Writing to .dx Files

In normal processing, the Transaction Router reads .dx files that have been created by the Transaction Processor in the SERVER_ROOT\docking\txnproc folder and writes .dx files out to mobile clients or to the \outbox directories of regional nodes. The following errors typically occur when the Transaction Router cannot read .dx files from the \txnproc directory and subsequently continue writing files to the nodes’ outboxes. Effectively no data is being is routed to any node serviced by the application server where a Transaction Router has these errors.

            􀁻 Error Opening Transaction File for Read

SBL-DCK-00142: Error opening transaction file C:\SIEBEL\docking\TXNPROC0005115.dx for read

This error is generated when the Transaction Router attempts to read a corrupt .dx file from the \txnproc directory. This error is usually found paired with SBL-TXR-00026: Error in the main function. The most likely causes of a corrupt .dx file are shortage of disk space or disk failure. To resolve this error without Siebel Technical Support’s assistance, remove all the .dx files from the \txnproc folder and re-extract all mobile clients and regional nodes.

In some occurrences it may be possible with Siebel Technical Support’s assistance to avoid the re-extraction of all nodes. If re-extraction of all nodes has a major business impact, please raise a Service Request with Siebel Technical Support and provide the following:

            􀂃 The Transaction Router log file

            􀂃 The corrupt .dx file; in this example, 00005115.dx from the \txnproc folder

            􀂃 The preceding .dx file; in this example, 00005114.dx from the \txnproc folder

            􀂃 The next .dx file; in this example, 00005116.dx from the \txnproc folder

            􀂃 The results of running txnutl on the corrupt .dx file. This should include the uncompressed .dx file (.dx2 file) and the readable transaction file (txn.log). For details on how to use txnutl please see FAQ 1175 on Siebel SupportWeb.

            􀂃 The results of the following SQL statement run on the server database:

select min(TXN_ID) from S_DOCK_TXN_LOG;

            􀁻 Unable to read an item from the User Txn Log file.

This error normally occurs when there is corrupt transaction in the .dx file being read by the Transaction Router. This is normally followed by some details about the corrupt transaction, example:

Trace Trace 3 2002-12-20 18:00:09 Txn Type : S

 

 

             

Trace Trace 3 2002-12-20 18:00:09 Txn Id : 1615359

Trace Trace 3 2002-12-20 18:00:09 Src Node : 1

Trace Trace 3 2002-12-20 18:00:09 Created By : 1-B7

Trace Trace 3 2002-12-20 18:00:09 DLog Row Id: 1-15DW4

Trace Trace 3 2002-12-20 18:00:09 OperType : Insert

Trace Trace 3 2002-12-20 18:00:09 TableName: CX_EVT_ACT_XM

Trace Trace 3 2002-12-20 18:00:09 Txn RowId : 1-15DW3

Trace Trace 3 2002-12-20 18:00:09 Txn LastUpdBy : 1-B7

Reading from file e:\sea630\siebsrvr\docking\TXNPROC0001580.dx

Trace Trace 3 2001-12-20 18:00:09 Writing to file e:\sea630\siebsrvr\docking\GZTHG0\outbox0000112.dx

If you encounter this behavior, please log a Service Request with Siebel Technical Support and provide the following information, since it may be possible to modify the .dx file:

            􀂃 The Transaction Router log file

            􀂃 The .dx file being read from the \txnproc directory; in the example above, the file would be 00001580.dx

            􀂃 The results of running txnutl on the corrupt .dx file. This should include the uncompressed .dx file (.dx2 file) and the readable transaction file (txn.log). For details on how to use txnutl please refer to FAQ 1175 on Siebel SupportWeb.

            􀁻 SBL-TXR-00026: Error in the main function.

If this error is encountered it is good practice is to check for other DCK or TXR errors that will give better description of the actual error.

            3. Errors When Accessing Cache Files and Docking Directories

The Transaction Router may fail trying to access one of the following:

            􀁻 The visibility database, visdata.dbf file, located in the SERVER_ROOT\docking directory

            􀁻 The Tall\Skinny database for a specific node

            􀁻 The Transaction Processor located in SERVER_ROOT\docking\%node_name%

            􀁻 The SERVER_ROOT\docking\txnproc directory

                        The solutions to these errors are specific to the error message, as follows:

            a. SBL-CSC-00119: Block number xxx exceeds file size (yyy blocks)

If the Transaction Router fails with this error please refer to Alert 233 on Siebel SupportWeb, regarding corruption of the dobjinst.dbf file.

 

 

            b. SBL-CSC-00213: Invalid visibility database. Shutdown and restart *all* routers to rebuild

This error indicates that the visibility database is corrupt. Use the following steps to rebuild visdata.dbf:

            i. Shut down all Transaction Routers.

            ii. Re-start the first Transaction Router with the following parameters set:

Id Db Recreate = TRUE

Id Db Size = 102400

iii. Wait 10-15 minutes to ensure the visdata.dbf is created without interruption.

iv. Restart all remaining Transaction Routers with the Id Db Recreate parameter set to FALSE.

            NOTE: Under normal circumstances, it is neither necessary nor recommended to delete the visdata.dbf file. If the error re-occurs, it is possible that there is another underlying error. In this case, please review the Transaction Router log file for earlier CSC errors in particular, and refer to Troubleshooting Steps 8 as you may be experiencing a Transaction Router performance problem.

            c. SBL-DCK-00123: Error opening file C:\SIEBEL\docking\J_LIBAIRE0000091.log for read

The most probable cause of the above error is that the server database has been restored to a point in time, and the mobile client’s dobjinst.dbf cache file is out of sync with the internal commit number stored in the Siebel database. To resolve this error, it is necessary to re-extract the problem node, J_LIBAIRE in the example above.

d. SBL-DCK-00250: A newer version of the dobjinst.dbf must be rebuilt by starting transaction processor with parameter “TSDbRecreate” set to “TRUE”

This error indicates that the Transaction Processors Tall\Skinny database is corrupt and needs to be rebuilt. It can be rebuilt by starting Transaction Processor with parameter Ts Db Recreate = TRUE. Under normal circumstances, it is neither necessary nor recommended to delete the file. Please note that if the dobjinst.dbf is deleted for a mobile client or regional node, the node has to be re-extracted, because the node’s dobjinst.dbf file can only be recreated by a Database Extract. Refer to FAQ 1264 for additional information on the recreation of dobjinst.dbf.

e. SBL-CSC-00128: Cannot get exclusive access to Tall/Skinny database for node JFIELD

This error indicates that the Transaction Router cannot access a particular node’s dobjinst.dbf file. If you encounter this behavior please try the following steps:

            i. Shut down and restart the Siebel server, and restart the Transaction Router to see if the problem persists.

 

 

            ii. If it does, end date the node in question. In the example above, the node is JFIELD.

            iii. Rename the node’s folder in the docking directory. In the example above, this directory would be SERVER_ROOT\Docking\JFIELD.

            iv. Re-extract the problem node.

            Additionally, please refer to Alert 311, which details another possible cause for the above error.

            f. SBL-CSC-00103: Error reading xxx bytes from dbfile at 0 (0 bytes read)

This error can be caused by a corrupted shared visibility database cache file. To force Transaction Router to recreate the visibility database firstly shutdown all Transaction Routers then restart the first Transaction Router with the Id Db Recreate parameter set to TRUE.

Also if a mobile client or a regional node’s dobjinst.dbf file becomes corrupted this can produce same error. The resolution would be to rename this nodes docking directory on the application server and re-extract this node and then restart the Transaction Router.

g. SBL-CSC-00202: An attempt was made to seek past the end of a data block

SBL-CSC-00301: Invalid value 0 for argument pVisDataInfo->createLatch

Both of the above errors indicate that visibility database has become corrupted. If recreating visdata.dbf as described above does not resolve the error, it is possible that you are experiencing a Transaction Router performance problem. Please refer to Troubleshooting Steps 8 for further information.

 

NOTE: If you are still experiencing Transaction Router failures after following the recommendations above and in the referenced SupportWeb documents, please raise a Service Request providing as much information as possible about the failure, and include the Transaction Router log file with the Service Request.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: