Recover corrupted blocks using Flashback Logs

Friday, May 17, 2013 2 comments
Starting with 11g Oracle can perform block media recovery using flashback logs to get good copies of the blocks.

Flashback database is not enabled by default so first step would be to enable it. When enabled a process RVWR (Recovery Writer) copies modified blocks to flashback buffer. Later this buffer is flushed to disk (flashback logs). It is important to notice that flashback logging is not getting log of a changes but complete block images.


Enable flashback database.

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
PL/SQL Release 11.1.0.7.0 - Production
CORE    11.1.0.7.0      Production
TNS for Linux: Version 11.1.0.7.0 - Production
NLSRTL Version 11.1.0.7.0 - Production

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
SQL>
SQL> startup mount;

ORACLE instance started.

Total System Global Area  467652608 bytes
Fixed Size                  2160752 bytes
Variable Size             180357008 bytes
Database Buffers          281018368 bytes
Redo Buffers                4116480 bytes
Database mounted.
SQL>
SQL> alter database flashback on;

Database altered.

SQL> alter database open;

Database altered.

Create small table and simulate block corruption.
SQL> create table test_corrupt (id number, text varchar2(100)) tablespace users;

Table created.

SQL> insert into test_corrupt values (5,'Marko');

1 row created.

SQL> commit;

Commit complete.

SQL> select id,text from test_corrupt;
 
        ID TEXT
---------- ----------
         5 Marko

Simulate data block corruption.
set heading off
set lines 113
SELECT 'dd of=' || f.file_name || ' bs=8192 conv=notrunc seek=' ||
       to_number(S.HEADER_BLOCK + 1) || ' << EOF',
       'CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt ',
       'EOF'
  FROM DBA_SEGMENTS S, dba_data_files f
 WHERE f.tablespace_name = 'USERS'
   and S.SEGMENT_NAME = 'TEST_CORRUPT';


dd of=/u01/app/oradata/testdb/users01.dbf bs=8192 conv=notrunc seek=523 << EOF
CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt
EOF

-bash-3.2$ dd of=/u01/app/oradata/testdb/users01.dbf bs=8192 conv=notrunc seek=523 << EOF
> CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt
> EOF
0+1 records in
0+1 records out
112 bytes (112 B) copied, 0.0265188 seconds, 4.2 kB/s

Check block corruption.
SQL> alter system flush buffer_cache;

System altered.

SQL> select id,text from test_corrupt;
select id,text from test_corrupt
                    *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 4, block # 523)
ORA-01110: data file 4: '/u01/app/oradata/testdb/users01.dbf'

We can try to perform block media recovery using RECOVER command.
RMAN will search flashback logs for good past image of the block. After good copy of the block is found, it is restored and recovered with information from redo logs.

But this will not work always.
In some cases flashback logs could be used to speed up block media recovery, but you shouldn't rely only on flashback logs.

RMAN> recover datafile 4 block 523;

Starting recover at 17-MAY-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=134 device type=DISK
searching flashback logs for block images
finished flashback log search, restored 0 blocks

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 05/17/2013 10:30:05
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 4 found to restore
RMAN-06023: no backup or copy of datafile 4 found to restore

In this case Oracle tried to find good block image in flashback logs but it failed in that attempt. We need valid backup to perform block media recovery.


Let’s try another scenario.

SQL> create table test_corrupt (id number, text varchar2(100)) tablespace users;

Table created.

SQL>  insert into test_corrupt values (5,'Marko');

1 row created.

SQL> commit;

Commit complete.

SQL> alter system flush buffer_cache;

System altered.

SQL> update test_corrupt set text='Okram' where id=5;

1 row updated.

SQL> commit;

Commit complete.

SQL> alter system flush buffer_cache;

System altered.


set heading off
set lines 113
SELECT 'dd of=' || f.file_name || ' bs=8192 conv=notrunc seek=' ||
       to_number(S.HEADER_BLOCK + 1) || ' << EOF',
       'CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt ',
       'EOF'
  FROM DBA_SEGMENTS S, dba_data_files f
 WHERE f.tablespace_name = 'USERS'
   and S.SEGMENT_NAME = 'TEST_CORRUPT';

dd of=/u01/app/oradata/testdb/users01.dbf bs=8192 conv=notrunc seek=523 << EOF
CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt
EOF

-bash-3.2$ dd of=/u01/app/oradata/testdb/users01.dbf bs=8192 conv=notrunc seek=131 << EOF
> CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt
> EOF

0+1 records in
0+1 records out
112 bytes (112 B) copied, 7.5846e-05 seconds, 1.5 MB/s

In this scenario I have different block corrupted.
SQL> select id, text from test_corrupt;
select id, text from test_corrupt
                     *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 4, block # 131)
ORA-01110: data file 4: '/u01/app/oradata/testdb/users01.dbf'


Perform block media recovery.
RMAN> recover datafile 4 block 131;

Starting recover at 17-MAY-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=129 device type=DISK
searching flashback logs for block images
finished flashback log search, restored 1 blocks

starting media recovery
media recovery complete, elapsed time: 00:00:03

Finished recover at 17-MAY-13

You can notice that this time Oracle found good block image in flashback logs and it completed recovery successfully.

SQL> select id, text from test_corrupt;

ID TEXT
-- --------------------
 5 Okram

Flashback database is nice opinion that enables faster point in time recoveries, or quicker block recoveries, but it cannot completely replace good valid backup.

Read More...

Oracle Restart: srvctl fails to start database with error CRS-5010

Sunday, April 14, 2013 0 comments
I’ve installed Oracle Grid Infrastructure on one of my standalone test servers. This software includes Oracle ASM as volume manager and Oracle Restart for managing Oracle components. As recommended, Grid Infrastructure and RDBMS software are installed under different users.

Grid Infrastructure: 11.2.0.3.0
RDBMS: 11.2.0.3.0
OS: OEL 5.8

After rebooting server I’ve noticed that database is not up so I’ve tried to run startup.
# Display status of the database
[grid@l01test1 ~]$ srvctl status database -d testdb
Database is not running.

# Startup
[grid@l01test1 ~]$ srvctl start database -d testdb
PRCR-1079 : Failed to start resource ora.testdb.db
CRS-5010: Update of configuration file "/u02/app/orcl/product/11.2.0/dbhome_1/dbs/inittestdb.ora" failed: details at "(:CLSN00014:)" in "/u02/app/11.2.0/grid/log/l01test1/agent/ohasd/oraagent_grid/oraagent_grid.log"
CRS-5017: The resource action "ora.testdb.db start" encountered the following error:
CRS-5010: Update of configuration file "/u02/app/orcl/product/11.2.0/dbhome_1/dbs/inittestdb.ora" failed: details at "(:CLSN00014:)" in "/u02/app/11.2.0/grid/log/l01test1/agent/ohasd/oraagent_grid/oraagent_grid.log"
. For details refer to "(:CLSN00107:)" in "/u02/app/11.2.0/grid/log/l01test1/agent/ohasd/oraagent_grid/oraagent_grid.log".

CRS-2674: Start of 'ora.testdb.db' on 'l01test1' failed

Hm, startup procedure failed because inittestdb.ora was missing.

Let’s display configuration of the database and check spfile location.
# Display database configuration
[grid@l01test1 ~]$ srvctl config database -d testdb -a

Database unique name: testdb
Database name: testdb
Oracle home: /u02/app/orcl/product/11.2.0/dbhome_1
Oracle user: orcl
Spfile: +DATA/testdb/spfiletestdb.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: testdb
Disk Groups: DATA,FRA,DISK2
Services:
Database is enabled

[grid@l01test1 ~]$ asmcmd
ASMCMD> cd +DATA/testdb/
ASMCMD> ls
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
mydir/
spfiletestdb.ora

Everything seemed fine to me.

Later I’ve found out why startup sequence failed.
Oracle tried to create "inittestdb.ora" file as grid user in $ORACLE_HOME/dbs folder of another user. It failed due to insufficient privileges.

To overcome this error I’ve changed permissions on $ORACLE_HOME/dbs folder.
[orcl@l01test1 dbhome_1]$ chmod 775 $ORACLE_HOME/dbs

Now when we start database everything works fine.
# Start
[grid@l01test1 tmp]$ srvctl start database -d testdb

# Check status
[grid@l01test1 tmp]$ srvctl status database -d testdb
Database is running.


With changed privileges grid user managed to create “inittestdb.ora” file in $ORACLE_HOME/dbs folder. This “inittestdb.ora” file has only one row - pointer to server parameter file.
[orcl@l01test1 dbs]$ ls -l inittestdb.ora
-rw-r--r-- 1 grid oinstall 62 Apr 14 10:37 inittestdb.ora

[orcl@l01test1 dbs]$ cat inittestdb.ora
SPFILE='+DATA/testdb/spfiletestdb.ora'          # line added by Agent



REFERENCES
Oracle Restart: srvctl fails to start database with error CRS-5010 if RDBMS and Grid under different users [ID 1335607.1]
http://docs.oracle.com/cd/E11882_01/server.112/e25494/restart005.htm#i1009665


Read More...

Change SID of the Oracle database using DBNEWID

Tuesday, April 9, 2013 5 comments
In this post I will describe how to change Oracle SID using utility DBNEWID. As I can see DBNEWID is available from 9i version but I’ve never heard about it. Till now, I have used procedure where I’m manually editing and re-creating control file.

But I think it’s much better to use database utility, then manually editing control file script which is task more prone to errors.

I’m aware that I can perform this task using RMAN DUPLICATE feature, but in this case I don’t want to use active database for duplication. As this is 11gR1 database Backup-Based duplication is not available.

So I will restore production database from backup to development server and change SID using DBNEWID.

I’ve successfully restored and recovered database.

Let’s check DBID and instance name.
SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
crmp

SQL> select dbid,open_mode from v$database;

      DBID OPEN_MODE
---------- ----------
1597268947 READ WRITE

CRMP is production database and I want to change SID from CRMP to CRMD.

First step is to shutdown database and start in mount mode.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area  730714112 bytes
Fixed Size                  2163280 bytes
Variable Size             192941488 bytes
Database Buffers          528482304 bytes
Redo Buffers                7127040 bytes
Database mounted.

Now let’s run nid to change database name.
$ nid target=sys/oracle@test_crmp dbname=crmd setname=YES

DBNEWID: Release 11.1.0.7.0 - Production on Tue Apr 9 11:59:56 2013

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

Connected to database CRMP (DBID=1597268947)

Connected to server version 11.1.0

Control Files in database:
    /u01/app/devcrmd/oradata/control01.ctl
    /u01/app/devcrmd/oradata/control02.ctl

Change database name of database CRMP to CRMD? (Y/[N]) => Y

Proceeding with operation
Changing database name from CRMP to CRMD
    Control File /u01/app/devcrmd/oradata/control01.ctl - modified
    Control File /u01/app/devcrmd/oradata/control02.ctl - modified
    Datafile /u01/app/devcrmd/oradata/datafiles/system01.db - wrote new name
    Datafile /u01/app/devcrmd/oradata/datafiles/sysaux01.db - wrote new name
    Datafile /u01/app/devcrmd/oradata/datafiles/users01.db - wrote new name
    Datafile /u01/app/devcrmd/oradata/datafiles/data01.db - wrote new name
    Datafile /u01/app/devcrmd/oradata/datafiles/indx01.db - wrote new name
    Datafile /u01/app/devcrmd/oradata/datafiles/tools01.db - wrote new name
    Datafile /u01/app/devcrmd/oradata/datafiles/undotbs02.db - wrote new name

NID-00111: Oracle error reported from target database while executing
    begin       dbms_backup_restore.nidprocessdf(:fno, :istemp, :skipped, :idchged,                                        :nmchged);    end;
ORA-01116: error in opening database file /u01/app/crmp/oradata/datafiles/tmp01.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 6694
ORA-06512: at line 1



Change of database name failed.
Must finish change or REVERT changes before attempting any database operation.
DBNEWID - Completed with errors.

If you are changing DBID or database name of your production database it is very important to backup database prior doing anything. As these are very sensitive operations it is very important to think carefully before performing any action. If you are stuck and not sure what to do I advise you to contact Oracle Support immediately.

You can see that I have received error after running nid because my tempfile was missing. I forgot to recreate it after database restore.

To fix this error I will drop tempfile and run nid again.
$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Tue Apr 9 15:59:19 2013

Copyright (c) 1982, 2008, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select name from v$tempfile;

NAME
--------------------------------------------------------------------------------
/u01/app/crmp/oradata/datafiles/tmp01.dbf


SQL> alter database tempfile '/u01/app/crmp/oradata/datafiles/tmp01.dbf' drop including datafiles;

Database altered.

Run nid again.
$ nid target=sys/oracle@test_crmp dbname=crmd setname=YES

DBNEWID: Release 11.1.0.7.0 - Production on Tue Apr 9 16:09:07 2013

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

Connected to database CRMP (DBID=1597268947)
Operation already in progress, continuing

Connected to server version 11.1.0

Control Files in database:
    /u01/app/devcrmd/oradata/control01.ctl
    /u01/app/devcrmd/oradata/control02.ctl

Change database name of database CRMP to CRMD? (Y/[N]) => Y

Proceeding with operation
Changing database name from CRMP to CRMD
    Control File /u01/app/devcrmd/oradata/control01.ctl - modified
    Control File /u01/app/devcrmd/oradata/control02.ctl - modified
    Datafile /u01/app/devcrmd/oradata/datafiles/system01.db - already changed
    Datafile /u01/app/devcrmd/oradata/datafiles/sysaux01.db - already changed
    Datafile /u01/app/devcrmd/oradata/datafiles/users01.db - already changed
    Datafile /u01/app/devcrmd/oradata/datafiles/data01.db - already changed
    Datafile /u01/app/devcrmd/oradata/datafiles/indx01.db - already changed
    Datafile /u01/app/devcrmd/oradata/datafiles/tools01.db - already changed
    Datafile /u01/app/devcrmd/oradata/datafiles/undotbs02.db - already changed
    Control File /u01/app/devcrmd/oradata/control01.ctl - wrote new name
    Control File /u01/app/devcrmd/oradata/control02.ctl - wrote new name
    Instance shut down

Database name changed to CRMD.
Modify parameter file and generate a new password file before restarting.
Succesfully changed database name.
DBNEWID - Completed succesfully.

Now operation completed successfully.

Create parameter file for development database and change database name.
$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Tue Apr 9 16:19:31 2013

Copyright (c) 1982, 2008, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> create spfile from pfile='$ORACLE_HOME/dbs/initcrmp.ora';

File created.


SQL> startup nomount;
ORACLE instance started.

Total System Global Area  730714112 bytes
Fixed Size                  2163280 bytes
Variable Size             192941488 bytes
Database Buffers          528482304 bytes
Redo Buffers                7127040 bytes
SQL>
SQL>
SQL> alter system set db_name=crmd scope=spfile;

System altered.

Startup database in mount mode.
SQL> startup mount force;
ORACLE instance started.

Total System Global Area  730714112 bytes
Fixed Size                  2163280 bytes
Variable Size             192941488 bytes
Database Buffers          528482304 bytes
Redo Buffers                7127040 bytes
Database mounted.


Create new password file.
$ orapwd file=$ORACLE_HOME/dbs/orapwcrmd password=oracle entries=10

Open database.
$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Tue Apr 9 16:23:58 2013

Copyright (c) 1982, 2008, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter database open;

Database altered.

Let’s check DBID and instance name.
SQL> select dbid,open_mode from v$database;

      DBID OPEN_MODE
---------- ----------
1597268947 READ WRITE

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
crmd

Notice that DBID is kept unchanged as I changed only database name. This is the reason why RESETLOGS operation wasn’t needed.

Post operations...

Add missing tempfile for temporary tablespace:
SQL> alter tablespace TMP add tempfile '/u01/app/devcrmd/oradata/datafiles/tmp01.dbf' size 2048M;
Change oratab entry:
$ vi /etc/oratab
...
#### set
...
crmd:/u01/app/devcrmd/product/11.1.0/db_1:N

Change listener.ora and tnsnames.ora files using new SID.


REFERENCES:
http://docs.oracle.com/cd/B28359_01/server.111/b28319/dbnewid.htm


Read More...

Transaction restrictions for Parallel DML on 11g

Saturday, March 30, 2013 0 comments
This will be just short post about restriction on parallel DML that I wasn't aware of.

Last week we had to perform quick update of several million rows in very big table. Parallel DML has proven to be the best tool to speed up large DML operation.

A DML can be parallelized only if you explicitly enable parallel DML in session level. This step is important because parallel and serial DML have different locking, transaction and space requirements.

I'd like to draw attention on one transaction restriction when using parallel DML. You can find other restrictions in Oracle Documentation.

This part was interesting to me:
If any DML statement in a transaction modifies a table in parallel, no subsequent serial or parallel query or DML statement can access the same table again in that transaction. This means that the results of parallel modifications cannot be seen during the transaction.
Even if PL/SQL procedure or block is executed in parallel DML session, this rule applies also.

Test is performed on 11.1.0.7 and 11.2.0.3 databases on Linux machine.
SQL> alter session enable parallel dml;

Session altered.

SQL> update /*+parallel(bt,8) */ big_table bt
  2  set bt.data_object_id=bt.data_object_id*100;

1000000 rows updated.

SQL> select count(*) from big_table;
select count(*) from big_table
*
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel

SQL> commit;

Commit complete.

SQL> select count(*) from big_table;

  COUNT(*)
----------
   1000000

1 row selected.


PL/SQL
SQL> declare
  2     l_cnt number := 0;
  3  begin
  4     execute immediate 'alter session enable parallel dml';
  5     update /*+parallel(bt,8) */ big_table bt
  6     set bt.data_object_id=bt.data_object_id*100;
  7
  8     select count(*) into l_cnt
  9     from big_table;
10  end;
11  /
declare
*
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel
ORA-06512: at line 8


So I cannot query modified table (from my session) until transaction is closed.
Nice to know that.

Parallel DML is a nice feature. It is mostly used in Data Warehouse and for running batch jobs in OLTP databases. But due to it's restrictions and different behavior then serial DML it is advisable to check documentation before using it.

References: http://docs.oracle.com/cd/B28359_01/server.111/b28313/usingpe.htm#CACEJACE

Read More...

Kill stuck Oracle process from OS using DBMS_SCHEDULER

Thursday, February 7, 2013 2 comments
Last month we had annoying problems caused by stuck Oracle process holding locks and not releasing them. Every few days one process hangs with status “ACTIVE” and “SQL*Net message from client” wait event.

In that state it holds locks and never releases them. As this is active OLTP database, new sessions arrive wanting those locked resources but they could not get them. So after some time you get lots of waiters and if you don’t react quickly maximum number of processes could be reached. Then your database refuses new connections and you're system becomes unresponsive.

To complicate a problem little more, when you kill session from Oracle using “disconnect session immediate” or “kill session immediate” - session is only marked as “KILLED”. It is not cleaned and locks are not released even if you wait for several hours. Only opinion is to kill process from OS side and then PMON manages to perform cleaning and release locks.

So we created temporary solution until “stuck process” problem is solved. As this is active OLTP system we are expecting most queries to complete under a minute. Every session active for over 30 minutes is probably stuck process and needs to be killed from OS side.

I will publish scripts we used to temporary solve this problem. You should change them according to your needs.

kill_stuck_process.sh
Main script used to perform OS kill. This script needs to be executable file.
#!/bin/bash

# SET ORACLE ENVIRONMENT #
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db_1
PATH=$ORACLE_HOME/bin:$PATH; export PATH
export ORACLE_SID=orcl

# File which will be used to keep spid
if [ -f /tmp/spid.txt ]; then
/bin/rm -f /tmp/spid.txt
fi

# Find SPID of the hanging process you want to kill
$ORACLE_HOME/bin/sqlplus -s system/password @/oracle/scripts/get_spid.sql > /tmp/spid.txt

spid=`cat /tmp/spid.txt`

# If spid exists kill specified process
if [ -s /tmp/spid.txt ]; then
# Send mail from Oracle about killed process
$ORACLE_HOME/bin/sqlplus -s system/password @/oracle/scripts/send_mail.sql
# Kill process
/bin/kill -9 $spid
fi

get_spid.sql
Query used to retrieve SPID of the stuck process. Modify this query based on your needs.
set serverout on head off ver off feed off
SELECT spid from v$process
where addr = ( SELECT paddr
                        FROM
                        (
                            SELECT paddr, rownum rn
                                from (
                                    SELECT paddr, last_call_et 
                                            FROM v$session
                                            where username='OLTP_USER'
                                            and last_call_et >= 1800
                                            and status='ACTIVE'
                                            order by last_call_et desc
                                       )
                        )
                        where rn=1);
exit

send_mail.sql
Procedure sends detailed information about killed process for better diagnosis later. This is just additional info and it is not essential for the script.
begin
  "ADMINUSR"."P_SEND_PROCESS_DETAILS";
end;
/
exit

Now to create DBMS_SCHEDULER job which will be executed every 30 minutes and kill stuck process.
BEGIN
DBMS_SCHEDULER.CREATE_JOB (
   job_name             => 'KILL_STUCK_PROCESS',
   job_type             => 'EXECUTABLE',
   job_action           => '/oracle/script/kill_stuck_process.sh',
   start_date           => SYSDATE,
   repeat_interval      => 'FREQ=MINUTELY;INTERVAL=30',
   enabled              =>  FALSE,
   comments             =>'Kill long running stuck OLTP_USER session!');
END;
/

Now to test the job:
BEGIN
  DBMS_SCHEDULER.run_job (job_name=> 'KILL_STUCK_PROCESS', use_current_session => TRUE);
END;
/

You will probably receive error:
BEGIN
*
ERROR at line 1:
ORA-27369: job of type EXECUTABLE failed with exit code: Permission denied
ORA-06512: at "SYS.DBMS_ISCHED", line 154
ORA-06512: at "SYS.DBMS_SCHEDULER", line 450
ORA-06512: at line 2

When dbms_scheduler is executing external (operating system) jobs it is using lowly privileged user and group for your platform. In my environment (OEL 5.8) this is “nobody” and “nobody”. As you must kill OS process you must specify more privileged user. There is “externaljob.ora” configuration file in “$ORACLE_HOME/rdbms/admin”. Edit this file as root and specify privileged user and group. In my case:
run_user = orcl11
run_group = oinstall

Everything is working fine now:
BEGIN
  DBMS_SCHEDULER.run_job (job_name=> 'KILL_STUCK_PROCESS', use_current_session => TRUE);
END;
/

PL/SQL procedure successfully completed.


Thoroughly test this scripts in your environment before going to production with them.


Read More...

Listener log missing when Oracle Clusterware starts listener resource script

0 comments
This post could be interesting for those who are running active/passive failover clusters with Oracle Clusterware used for managing resources. In this case owner of the Clusterware installation is not the same as owner of the Oracle RDBMS installation. Among others, Clusterware is assigned for managing listener resource (start,stop,check).

Till 11g version listener logs were by default written in “$ORACLE_HOME/network/log” but with 11g ADR (Automatic Diagnostic Repository) is introduced and location for listener log is changed. From 11g default location is “$ORACLE_BASE/diag/tnslsnr/<hostname>/listener_<dbname>/trace”.

I’ve noticed that when you start listener as owner of Oracle RDBMS everything is working fine. Listener log and traces are written under ADR. But when listener is started from Clusterware environment ADR directory structure is not created and listener log is not written.

In Oracle technical whitepaper Using Oracle Clusterware to Protect A Single Instance Oracle Database 11g is explained how to create active/passive cluster using Oracle Clusterware.

It is said that script for managing listener resource is made generic and you don’t have to modify anything. You only must enter ORACLE_HOME and listener name which will be stored in Oracle Cluster Registry. At the end of the document (page 22) you can find perl script for listener resource so I will not copy it here.


Now lets go back to listener log. As ORACLE_BASE is not specified in the script ADR location could not be found and logging entries are not stored.


There are couple solutions for this problem:

1. You could modify script and specify “ORACLE_BASE” but then this script wouldn’t be generic.

2. Introduce additional parameter “ORACLE_BASE” as part of crs_profile command line to keep script generic and modify script. You should thoroughly test the script afterwards.

3. Disable ADR by setting in listener.ora “DIAG_ADR_ENABLED_<listener_name>=OFF” and restart listener. After disabling ADR listener logs are successfully written in “$ORACLE_HOME/network/log” directory.

4. In listener.ora specify “ADR_BASE_listener = <path>” where <path> should be location stored in “diagnostic_dest” parameter defined in database. Then restart listener and logs should be written in ADR.


I would recommend to avoid first two options and use third or fourth.


Read More...

Workaround for ORA-00600: internal error code, arguments: [kzdlk_zt2 err]

Friday, September 28, 2012 3 comments
To create database link in another schema you must execute “create database link” logged in that schema or you can use workaround noted in this post:

How to Create a Database Link in Another User’s Schema

Neil Johnson wrote nice post on that subject and I’m using his method for creating database links in another user’s schema. So check it out - link is above.


Recently I had situation when I didn’t know password of the user on the local and remote database. But I had to create database link to compile some views and procedures.

In that case my method was:
(execute this steps as user with admin privileges)

-- Grant privilege to create database link to user
grant create database link to locuser;


-- Create temporary procedure to execute create database link as another user
CREATE OR REPLACE PROCEDURE locuser.create_dblink AS
BEGIN
    EXECUTE IMMEDIATE 'CREATE DATABASE LINK TEST_DB CONNECT TO RMTUSER IDENTIFIED BY VALUES ''E977335D83B8468E'' USING ''testdb''';
END create_dblink;
/

-- Execute procedure and create database link in another schema
exec locuser.create_dblink;


-- Drop temporary procedure
drop procedure locuser.create_dblink;


-- Revoke privilege to create database link
revoke create database link from locuser;


But when I tried to test database link I’ve received this nasty error:

ORA-00600: internal error code, arguments: [kzdlk_zt2 err], [18446744073709551603], [], [], [], [], [], [], [], [], [], []


There is MOS note related to this error: [ID 1309705.1]

The error suggests that when the database link was created, the was established using the syntax IDENTIFIED BY VALUES as compared to the document syntax of IDENTIFIED BY

Use of IDENTIFIED BY VALUES is reserved for internal Oracle use only.

While earlier Oracle releases allowed the use of IDENTIFIED BY VALUES, this is not documented as being valid syntax.

From Oracle release 10gR2, database links must be created using the documented syntax.

Solution: Recreate the database link using valid syntax.


So Oracle support says that I cannot use syntax IDENTIFIED BY VALUES while creating database link.

Luckily I’ve found workaround :-)



Environment: Oracle EE 11.1.0.7 on OEL 5.8


To reproduce error I’ve pulled password of another user's schema from "sys.user$" internal table logged as sys user.
Now I will try to create a database link using this password.

-- Create database link
SQL> CREATE DATABASE LINK TEST_DB CONNECT TO RMTUSER IDENTIFIED BY VALUES 'E977335D83B8468E' using 'testdb';

Database link created.

-- Test database link
SQL> select * from dual@TEST_DB;
select * from dual@TEST_DB
                   *
ERROR at line 1:
ORA-00600: internal error code, arguments: [kzdlk_zt2 err],
[18446744073709551603], [], [], [], [], [], [], [], [], [], []


-- Drop
SQL> drop database link TEST_DB;

Database link dropped.



To workaround this problem you could extract user password using “dbms_metadata.get_ddl” package to generate DDL for database link.

-- Use this query to generate DDL
select dbms_metadata.get_ddl('DB_LINK',db_link,owner)||';'
from dba_db_links;

-- Create database link
SQL> CREATE DATABASE LINK "TEST_DB" CONNECT TO RMTUSER IDENTIFIED BY VALUES '05F57DA167252CDAD388952CAE91A1ED33' USING 'testdb';

Database link created.

-- Test database link
SQL> select * from dual@TEST_DB;

D
-
X


-- Drop
SQL> drop database link TEST_DB;

Database link dropped.


In this case, with differently hashed password, using IDENTIFIED BY VALUES is valid syntax.

Although you can create database link using this syntax have in mind that this is not documented as being valid syntax. So use valid and documented syntax when you can.

Read More...

Wrong result after using join predicate push into a view with a GROUP BY

Friday, September 14, 2012 4 comments
Yesterday colleague reported that he’s receiving wrong/odd result from query so we decided to examine closely what's happening. Query had more then 150 lines with inline view, group by clause, several outer joins, function, decodes and cases included. So the first step was to exclude all the stuff that wasn’t important for clearer diagnosis.

Environment: Oracle EE 11.1.0.7 on OEL 5.7


I will post shortened version of the query with fake table names and columns.


SELECT
   p.name p_name,
   r.status r_status
   FROM table_1 r,
   (          
       SELECT  ri.r_id,            
       MAX ( DECODE (ri.c_name,  'SPACE', ri.i_val, NULL)) SPACE,
       MAX ( DECODE (ri.c_name,  'UNIVERSE', ri.i_val, NULL)) UNIVERSE,
       MAX ( DECODE (ri.c_name,  'ALIEN', ri.i_val, NULL)) ALIEN,
       MAX ( DECODE (ri.c_name,  'MUSIC', ri.i_val, NULL)) MUSIC,
       MAX ( DECODE (ri.c_name,  'GUITARS', ri.i_val, NULL)) GUITARS
       FROM table_7 ri                 
                  WHERE ri.r_id IN (SELECT r2.r_id FROM table_1 r2 WHERE  r2.r_id > 1233251 and r2.p_id in (111, 222))
   GROUP BY ri.r_id        
   ) ri,
   table_2 a,
   table_3 cu,
   table_4 c,
   table_5 p,
   table_6 rc,
   table_6 rc2,
   table_6 rc3,
   table_6 rc4,
   table_6 rc5
WHERE     1=1
   AND r.r_id = a.r_id
   AND r.r_id = ri.r_id  
   AND cu.table_3_ID = r.table_3_ID
   AND c.table_4_ID = r.table_4_id
   AND r.p_id = p.p_id
   AND r.p_id in (111, 222)
   AND c.status in ('DA', 'NE', 'A')        
   AND rc.lang(+) = 'BLAH'
   AND rc.domain(+) = 'SPACE'
   AND rc.code(+) = ri.SPACE
   AND rc2.lang(+) = 'BLAH'
   AND rc2.domain(+) = 'MARS'
   AND rc2.code(+) = ri.UNIVERSE
   AND rc3.lang(+) = 'BLAH'
   AND rc3.domain(+) = 'SPEED'
   AND rc3.code(+) = ri.GUITARS
   AND rc4.lang(+) = 'BLAH'
   AND rc4.domain(+) = 'ALIEN'
   AND rc4.code(+) = ri.ALIEN
   AND rc5.lang(+) = 'BLAH'
   AND rc5.domain(+) = 'MUSIC'
   AND rc5.code(+) = ri.MUSIC


Imagine this case.
You execute this query and you receive 114 rows, but when you exclude “p_name” and just ask for one column “r_status” you receive 15 rows.

After few hours of analysis we found possible cause for this behavior - “Bug 8447623 Wrong Results with OUTER joins and tables with Check Constraints”. Workaround for this bug was setting undocumented parameter "_optimizer_join_elimination_enabled" to false. Quick look on Oracle support site showed that “_optimizer_join_elimination_enabled” introduced many bugs so we were pretty sure that this is causing problems.

Altering “_optimizer_join_elimination_enabled” from true to false (on session level) changed result from 15 to 114 rows when asking "r_status" column. We were receiving stable result (114 rows) in any case. But we weren't completely satisfied with receiving this result so we continued with deeper data analysis.

It showed that actually 15 rows was the correct result and with altering “_optimizer_join_elimination_enabled” to false we were continuously receiving wrong result. So we left default value for that parameter and continued with investigation.

After some while of searching and testing I've found note - “Bug 9121708 Wrong result using view pushed predicate and OUTER join”.

This was the root cause. Oracle 11.1.0.7 version was affected and we had query using join predicate push into a view with a GROUP BY. Changing "_push_join_predicate" to false was the correct workaround. We were receiving correct 15 rows in any case.

To avoid similar cases happening in the future we should upgrade database to 11.2.0.2 or disable “_push_join_predicate” on instance level. But for now we decided to do none of that.

We changed our query a little to avoid pushing predicates into a view with group by and everything worked as it should.

Push join predicate is very powerful feature which enables pushing join predicate inside a view. It is not very wise to disable this feature on instance level without further testing because it can significantly affect performance.
As we noticed that there aren’t many bugs associated to this feature, we decided that it is somewhat safe to leave it enabled.


The thing that scares me the most is the amount of "wrong result" bugs in newer Oracle versions. It is very difficult to detect such bug. When developer or application user notices wrong/odd result Oracle is very last to blame. Who knows how many reports or calculations are there with undetected wrong results. It is little scary, isn't it :)

With newer versions Oracle introduces super new features that speed up execution of the query, but with this features new “wrong result” bugs are introduced also.


Don't get me wrong, I'm OK with that. Oracle is software like any other so bugs are expected. Our job (as DBA’s) is to catch this bugs and correct them or at least avoid them.


Read More...

Perform Oracle Block Recovery without having valid backup

Thursday, July 19, 2012 2 comments
My DBA friend Dejan from Baze Podataka database related site wrote post about Oracle Bug 8943287 which reminded me how dangerous bug this is. I was completely unaware about this bug till few months ago.

For this blog post I want to write about “ORA-01578: ORACLE data block corrupted” error and recovery. It is easy to recover from such error if you have RMAN backup - you, of course, have RMAN backup? (if not, create one) Even if you don’t have RMAN backup you can perform block recovery from OS based backups (datafile copies).

But can you perform block recovery without RMAN backups or OS backups?
You can, but only if you have necessary archivelogs available.



OS: OEL5
DB: Oracle 11.1.0.7



Create small tablespace and table for testing.
SQL> create tablespace corrupt_tbs datafile '/u01/app/oradata/aocdb/corrupt_tbs.dbf' size 10M;
Tablespace created.

SQL> create table test_corrupt (id number, text varchar2(100)) tablespace corrupt_tbs;
Table created.

SQL> insert into test_corrupt values(5,'Marko');
1 row created.

SQL> commit;
Commit complete.

SQL> select id,text from test_corrupt;

        ID TEXT
---------- ----------
         5 Marko


Now to simulate data block corruption. I will generate dd command using this small script:
set heading off
set lines 113
SELECT 'dd of=' || f.file_name || ' bs=8192 conv=notrunc seek=' ||
       to_number(S.HEADER_BLOCK + 1) || ' << EOF',
       'CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt ',
       'EOF'
  FROM DBA_SEGMENTS S, dba_data_files f
 WHERE f.tablespace_name = 'CORRUPT_TBS'
   and S.SEGMENT_NAME = 'TEST_CORRUPT';
   
dd of=/u01/app/oradata/aocdb/corrupt_tbs.dbf bs=8192 conv=notrunc seek=131 << EOF
CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt
EOF

Check block corruption after executing dd command.
$ dd of=/u01/app/oradata/aocdb/corrupt_tbs.dbf bs=8192 conv=notrunc seek=131 << EOF
> CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT corrupt CORRUPT CORRUPT CORRUPT CORRUPT corrupt corrupt
> EOF
0+1 records in
0+1 records out
112 bytes (112 B) copied, 7.9169e-05 seconds, 1.4 MB/s

SQL> alter system flush buffer_cache;

System altered.

SQL> select id,text from test_corrupt;
select id,text from test_corrupt
                    *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 8, block # 131)
ORA-01110: data file 8: '/u01/app/oradata/aocdb/corrupt_tbs.dbf'


Block is successfully corrupted - now to recover corrupted block.
As I don’t have RMAN backup or valid copy of datafile I will create copy of corrupted datafile to temporary location using RMAN restore command.
Before performing RMAN restore you should take corrupted datafile offline or you will get error:
Starting restore at 19.07.2012 21:56:13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=139 device type=DISK

creating datafile file number=8 name=/tmp/corrupt_tby_copy.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 07/19/2012 21:56:14
ORA-01182: cannot create database file 8 - file is in use or recovery
ORA-01110: data file 8: '/u01/app/oradata/aocdb/corrupt_tbs.dbf'


Restore datafile 8 to temporary location “/tmp/corrupt_tbs_copy.dbf”.
RMAN> sql 'alter database datafile 8 offline';

using target database control file instead of recovery catalog
sql statement: alter database datafile 8 offline

RMAN> run
2> {
3> set newname for datafile 8 to '/tmp/corrupt_tbs_copy.dbf';
4> restore datafile 8;
5> }

executing command: SET NEWNAME

Starting restore at 19.07.2012 21:27:49
using channel ORA_DISK_1

creating datafile file number=8 name=/tmp/corrupt_tbs_copy.dbf
restore not done; all files read only, offline, or already restored
Finished restore at 19.07.2012 21:27:49


This new file acts as RMAN copy of the corrupted datafile which will be used for RMAN block recovery.
RMAN> list copy of datafile 8;

List of Datafile Copies
=======================

Key     File S Completion Time     Ckp SCN    Ckp Time
------- ---- - ------------------- ---------- -------------------
103     8    A 19.07.2012 21:27:49 12672352727932 19.07.2012 21:11:50
        Name: /tmp/corrupt_tby_copy.dbf


Now to perform RMAN block recovery.
RMAN> blockrecover datafile 8 block 131;

Starting recover at 19.07.2012 21:29:46
using channel ORA_DISK_1

channel ORA_DISK_1: restoring block(s) from datafile copy /tmp/corrupt_tbs_copy.dbf

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 19.07.2012 21:29:48

From alert log:
Thu Jul 19 21:29:47 2012
alter database recover datafile list clear
Completed: alter database recover datafile list clear
Starting block media recovery
Recovery of Online Redo Log: Thread 1 Group 1 Seq 481 Reading mem 0
  Mem# 0: /u01/app/oradata/aocdb/redo01.log
Recovery of Online Redo Log: Thread 1 Group 2 Seq 482 Reading mem 0
  Mem# 0: /u01/app/oradata/aocdb/redo02.log
Completed block media recovery

As you can see RMAN used newly created file to copy “good” block and performed block recovery using information from redo logs.

For the last step we must recover datafile 8 to be consisted. Remember, datafile 8 was made offline before performing RMAN restore command.
RMAN> recover datafile 8;

Starting recover at 19.07.2012 21:32:10
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 19.07.2012 21:32:11


RMAN> sql 'alter database datafile 8 online';

sql statement: alter database datafile 8 online


We successfully recovered corrupted block:
SQL> select id,text from test_corrupt;

        ID TEXT
---------- ----------
         5 Marko



Cleanup.
RMAN> delete copy of datafile 8;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=131 device type=DISK
List of Datafile Copies
=======================

Key     File S Completion Time     Ckp SCN    Ckp Time
------- ---- - ------------------- ---------- -------------------
103     8    A 19.07.2012 21:27:49 12672352727932 19.07.2012 21:11:50
        Name: /tmp/corrupt_tbs_copy.dbf


Do you really want to delete the above objects (enter YES or NO)? yes
deleted datafile copy
datafile copy file name=/tmp/corrupt_tby_copy.dbf RECID=103 STAMP=789082069
Deleted 1 objects

SQL> drop tablespace corrupt_tbs including contents and datafiles;

Tablespace dropped.

Read More...

How to move SYSTEM tablespace with minimum downtime

Friday, May 25, 2012 4 comments
Two weeks ago a colleague DBA asked me what would be the best way to move SYSTEM tablespace with minimum downtime. There are several ways to perform that task and I will note what would be the best way in my opinion.

Messing with SYSTEM tablespace is always tricky so my goal would be minimum risk and minimum downtime.

My testing environment is 11gR1 database on Linux 32bit OS.

After quick search on that subject I’ve noticed that DBA’s are comfortable with moving/renaming other datafiles but SYSTEM or UNDO are making problems. Some tablespaces are essential and cannot be taken offline while database is up and running.





⇒ My database.
RMAN> report schema;

using target database control file instead of recovery catalog
Report of database schema for database with db_unique_name AOCDB

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    710      SYSTEM               ***     /u01/app/oradata/aocdb/system01.dbf
2    750      SYSAUX               ***     /u01/app/oradata/aocdb/sysaux01.dbf
3    1260     UNDOTBS1             ***     /u01/app/oradata/aocdb/undotbs01.dbf
4    2363     USERS                ***     /u01/app/oradata/aocdb/users01.dbf
5    1024     REST                 ***     /u01/app/oradata/aocdb/rest01.dbf
6    1024     INDX_REST            ***     /u01/app/oradata/aocdb/indx_rest01.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    96       TEMP                 32767       /u01/app/oradata/aocdb/temp01.dbf

⇒ While database is up and running I will use RMAN to copy datafile of SYSTEM tablespace to another location.
RMAN> copy datafile 1 to '/u01/app/oradata/loc2/system01.dbf';

Starting backup at 15.05.2012 19:35:23
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=126 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=/u01/app/oradata/aocdb/system01.dbf
output file name=/u01/app/oradata/loc2/system01.dbf tag=TAG20120515T193524 RECID=76 STAMP=783372953
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
Finished backup at 15.05.2012 19:35:59


⇒ Now I will shutdown database and start in mount mode.
RMAN> shutdown immediate;

database closed
database dismounted
Oracle instance shut down

RMAN> startup mount;

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area    1071333376 bytes

Fixed Size                     1318172 bytes
Variable Size                352322276 bytes
Database Buffers             713031680 bytes
Redo Buffers                   4661248 bytes


⇒ Switch datafile to copy.
RMAN> switch datafile 1 to copy;

datafile 1 switched to datafile copy "/u01/app/oradata/loc2/system01.dbf"

⇒ Now to perform quick recover.
RMAN> recover datafile 1;

Starting recover at 15.05.2012 19:38:55
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=154 device type=DISK

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 15.05.2012 19:38:57


⇒ That’s it! Open database.
RMAN> alter database open;

database opened


⇒ Little check that I’ve moved datafile.
RMAN> report schema;

Report of database schema for database with db_unique_name AOCDB

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    710      SYSTEM               ***     /u01/app/oradata/loc2/system01.dbf
2    750      SYSAUX               ***     /u01/app/oradata/aocdb/sysaux01.dbf
3    1260     UNDOTBS1             ***     /u01/app/oradata/aocdb/undotbs01.dbf
…
…

⇒ After switching datafiles, old datafile become copy so it is safe to drop it.
RMAN> list copy of datafile 1;

List of Datafile Copies
=======================

Key     File S Completion Time     Ckp SCN    Ckp Time
------- ---- - ------------------- ---------- -------------------
77      1    A 15.05.2012 19:38:15 5250088    15.05.2012 19:37:03
        Name: /u01/app/oradata/aocdb/system01.dbf

  

RMAN> delete copy of datafile 1;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=154 device type=DISK
List of Datafile Copies
=======================

Key     File S Completion Time     Ckp SCN    Ckp Time
------- ---- - ------------------- ---------- -------------------
77      1    A 15.05.2012 19:38:15 5250088    15.05.2012 19:37:03
        Name: /u01/app/oradata/aocdb/system01.dbf

  
Do you really want to delete the above objects (enter YES or NO)? YES
deleted datafile copy
datafile copy file name=/u01/app/oradata/aocdb/system01.dbf RECID=77 STAMP=783373095
deleted datafile copy



So the only downtime was period during bouncing database and recovering datafile.

Read More...

Tuning query with database link using USE_NL hint

Thursday, March 1, 2012 2 comments
I’ve just realized that I’m never fully satisfied after tuning queries with database links. Yes, I manage to improve performance significantly but I’m always missing some answers.

Case explained in this blog post is not exception also.

Environment:
Source DB (Oracle 11.1.0.7 on Solaris 64bit)
Remote DB (Oracle 10.2.0.4 on Solaris 64bit)
(Table names and values are changed )



One of the developers reported me problem with this simple query:

To get better output click "View Source".
SELECT *
  FROM tab1 cc, 
             tab2@db2 cfp
 WHERE cc.c_id = cfp.b_id 
 AND cc.code = '1234567890';

-------------------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name                | Rows  | Bytes | Cost (%CPU)| Time     | Inst   |IN-OUT|
--------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                     |     1 |   309 |  5160   (4)| 00:01:02 |        |      |
|*  1 |  HASH JOIN                   |                     |     1 |   309 |  5160   (4)| 00:01:02 |        |      |
|   2 |   TABLE ACCESS BY INDEX ROWID| TAB1                |     1 |   125 |     1   (0)| 00:00:01 |        |      |
|*  3 |    INDEX RANGE SCAN          | C_IDX3              |     1 |       |     1   (0)| 00:00:01 |        |      |
|   4 |   REMOTE                     | TAB2                |  2053K|   360M|  5137   (3)| 00:01:02 | CRM_B~ | R->S |
--------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - access("CC"."C_ID"="CFP"."B_ID")
   3 - access("CC"."CODE"='1234567890')


I've executed query and it took over 10 minutes to get results.


Let's break the query:

To get better output click "View Source".
select c_id
from tab1 cc
where cc.code = '1234567890';

1 row selected.

<<<<<< Elapsed: 00:00:00.02 >>>>>>

---------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name                | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |                     |     1 |   125 |     1   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| TAB1                |     1 |   125 |     1   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | C_IDX3              |     1 |       |     1   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("CC"."CODE"='1234567890')



select *
from tab2@db2 cfp
where cfp.b_id = '32165420';

57 rows selected.

<<<<<< Elapsed: 00:00:00.03 >>>>>>

--------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name             | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop | Inst   |
--------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT REMOTE            |                  |    12 |  2208 |    15   (0)| 00:00:01 |       |       |        |
|   1 |  TABLE ACCESS BY GLOBAL INDEX ROWID| TAB2             |    12 |  2208 |    15   (0)| 00:00:01 | ROWID | ROWID |    DB2 |
|*  2 |   INDEX RANGE SCAN                 | B_IDX            |    12 |       |     3   (0)| 00:00:01 |       |       |    DB2 |
--------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("A1"."B_ID"='32165420')

Note
-----
   - fully remote statement

From both queries I've received results instantly.


To get more answers I've decided to gather more details with tracing.
Tkprof output showed why this query behaved poorly.

SELECT *
  FROM tab1 cc, tab2@db2 cfp
 WHERE cc.c_id = cfp.b_id AND cc.code = '1234567890'

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.03       0.16          0          0          1           0
Execute      1      0.00       0.02          0          0          0           0
Fetch        2    131.58     602.67          0          2          0          57
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4    131.61     602.87          0          2          1          57

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 41

Rows     Row Source Operation
-------  ---------------------------------------------------
     57  HASH JOIN  (cr=2 pr=0 pw=0 time=0 us cost=5052 size=309 card=1)
      1   TABLE ACCESS BY INDEX ROWID TAB1 (cr=2 pr=0 pw=0 time=0 us cost=1 size=125 card=1)
      1    INDEX RANGE SCAN C_IDX3 (cr=1 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 66342)
2053552   REMOTE  TAB2 (cr=0 pr=0 pw=0 time=732493 us cost=5029 size=374102912 card=2033168)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  single-task message                             1        0.14          0.14
  SQL*Net message to dblink                  937019        0.00          0.90
  SQL*Net message from dblink                937019        0.86        474.64
  SQL*Net message to client                       2        0.00          0.00
  SQL*Net message from client                     2      452.84        452.85
  SQL*Net more data from dblink                8289        0.55         29.50
  SQL*Net more data to client                     1        0.00          0.00


Main waits were associated to network traffic between databases.
Why Oracle decided to read 2053552 rows (full scan instead index) from remote table?

Maybe there was some problem with stale statistics. I've gathered fresh statistics but that haven't changed anything.

As developer was waiting for me I had to find solution quickly (if I can, of course).

I've solved similar issue using USE_HASH hint (SQL Tuning - using USE_HASH hint - dblink issue) so I've tried with USE_NL hint in this case.
It helped! I've received result under a second.

SELECT /*+ use_nl(cc cfp) */ *
  FROM TAB1 cc, TAB2@DB2 cfp
 WHERE cc.C_ID = cfp.B_ID AND cc.CODE = '1234567890'

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.02       0.13          0          0          1           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2      0.01       0.01          0          2          0          57
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4      0.03       0.15          0          2          1          57

Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 41

Rows     Row Source Operation
-------  ---------------------------------------------------
     57  NESTED LOOPS  (cr=2 pr=0 pw=0 time=0 us cost=5030 size=309 card=1)
      1   TABLE ACCESS BY INDEX ROWID TAB1 (cr=2 pr=0 pw=0 time=0 us cost=1 size=125 card=1)
      1    INDEX RANGE SCAN C_IDX3 (cr=1 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 66342)
     57   REMOTE  TAB2 (cr=0 pr=0 pw=0 time=0 us cost=5029 size=1840 card=10)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  single-task message                             1        0.11          0.11
  SQL*Net message to dblink                      10        0.00          0.00
  SQL*Net message from dblink                    10        0.00          0.02
  SQL*Net message to client                       2        0.00          0.00
  SQL*Net message from client                     1        0.01          0.01
  SQL*Net more data from dblink                   4        0.00          0.00
  SQL*Net more data to client                     1        0.00          0.00

Improvement from 602.87 secs to 0.15 secs.


Developer was very happy with this outcome.
To be honest I wasn't completely satisfied. Hint USE_NL was just a quick guess and
I don't like guesswork to solve problems.

Queries with db links are always little tricky. Oracle simply does not have enough information to take the right decision.

If someone has an idea why USE_NL fixed this problem please leave your comment.
Thanks!

Read More...

Troubleshooting memory usage

Tuesday, January 10, 2012 5 comments
Last few days we had significant load problems with one of our database servers. Server was choking under minor database load which caused applications to be non-responsive.

This is 8-core 32-bit OEL5 (2.6.18-92.el5PAE) server with 4GB RAM running two Oracle 10g (10.2.0.4) SE databases. SGA's of those databases where 1,2 GB and 700MB with pga_aggregate_target 64M and 128M.

As databases were small (under 50G) and not very active I didn't expected any performance problems.


Using vmstat, mpstat, top and sar tools I've noticed that we were having slight issues with swapping, despite that there was enough physical memory.

So I've started troubleshooting...

First I've found that one of top running processes was "kswapd0" process.
This is Kernel Swap Deamon responsible for scanning memory to locate free pages, and scheduling dirty pages to be written to disk. It is OK to occasionally notice kswapd running, but it shouldn't continuously appear in top processes. In some situations you can resolve kswapd issues using huge pages but this wasn't the case here.

I've decided to check several OS parameters related to swapping to find possible cause of our issues.

Parameters below were specified in "/etc/sysctl.conf":
...
vm.overcommit_memory=1
vm.swappiness=0
vm.min_free_kbytes=262144
vm.page-cluster=5
...

Considering that I didn't installed this server, this parameters were little odd to me. All of them were custom parameters not required from install guide.

The swappiness variable bothered me the most. It tells Linux kernel how fast it can move inactive pages to swap area. Default value is 60, but in this case value was changed to 0 which tells the Linux kernel to avoid swapping as much as possible.

As this was production server which was still choking we had to react quickly without prior testing. I haven't seen the purpose of vm custom parameters on this server so I've decided to comment them and apply only default parameters required from install guide. It's easy to add custom parameters later if there is a need.

After I applied parameters running "sysctl -p" load average suddenly started to fall. Several minutes later it fell from 13 to 1. From vmstat I've noticed significant drop in swapping and kswapd0 wasn't in top processes anymore.

With altering swappiness to 60 (default) I've told kernel to find inactive pages more aggressively and swap them out to disk.
Probably that helped OS to use memory more effectively.

I will closely monitor performance of this server in next few days but so far it is working very good.

Considerations for future:
- enable Huge Pages
- migrate os/db to 64bit

When I'm installing database servers I tend to use parameters specified in install guide or Oracle support site. Eventually I will add some custom parameters but only if I have proper reason for that. Even then it is wise to test changes before applying them in production.

So my conclusion for this case would be - do not add custom parameters without proper reason.
Google - "Compulsive Tuning Disorder" :)


Read More...

IMPDP creates indexes with parallel degree 1 during import

Wednesday, August 31, 2011 4 comments
Last few days I’ve been playing with export and import of 750 GB database. During this whole trial and error process I’ve learned several things that were unfamiliar to me. In this blog post I want to share some of the stuff that I learnt.

I have Oracle 10.2.0.4 database on Solaris 64bit which I want to migrate to Oracle 11.1.0.7 on Linux 64bit. It is 750GB large database with mostly partitioned tables (compressed partitions).


Let's get to the subject of the post.

I won't talk much about export operation - just to mention that it is important to specify this bolded line in parameter file:
dumpfile=exportdb_%U.dmp
filesize=30G


This means that multiple 30GB sized dump files will be generated during export. It is important to specify %U expression or to manually define multiple dump files if you want to get the most from parallel processing during export and import.

In my case - after performing full export of the database I've created 20 dump files which is about 600GB of export data.


Now comes the tricky part - importing database to the new environment.

Data import was very fast as it finished in below 2,5 hours. During import server was fully utilized using all available resources in parallel processing. When index creation started utilization of the server dropped to about 10%.

Only one worker performed the entire work, while others were waiting - even with parallel option specified.

Then I found that this was the result of the very nasty bug:
“Bug 8604502 - IMPDP creates indexes with parallel degree 1 during import”

This issue is fixed in:
12.1 (Future Release)
11.2.0.2 (Server Patch Set)
11.2.0.1 Bundle Patch 7 for Exadata Database
11.1.0.7 Patch 24 on Windows Platforms

Great, I don’t have fix for this bug.
With only one worker creating indexes this import could last for days.


For a quick solution I’ve decided to perform import in two steps. Firstly I will import everything but indexes and in next step I will import just indexes.

Data import again lasted below 2,5 hours and it finished without any problems fully utilizing the server.

For index creation I’ve generated CREATE INDEX script using SQLFILE option in IMPDP.

*parameter file*
userid=xxx/yyy
sqlfile=createidx.sql
dumpfile=exportdb_%U.dmp
include=INDEX
directory=export_dir
schemas=(...)


As SQLFILE generates CREATE INDEX scripts with parallel degree set to 1 I’ve decided to manually edit script and replace parallel 1 to parallel 16.

Besides parallel 16 option, I’ve specified NOLOGGING option to restrict generation of large redo log. As I expected creation of large indexes I’ve allocated large enough temporary tablespace and specified higher pga_aggregate_target parameter.

Then executed script from sqlplus and monitored server utilization and waits.

Script finished in about 3 hours which was nice improvement.
Server was fully utilized - mostly using 16 processes for index creation.

Main waits were:
- WAITING direct path read
- WAITING direct path read temp
- WAITING direct path write temp

This waits were expected as temporary space was extensively used.


Later I’ve found nice article from great Oracle expert Randolf Geist - Concurrent Index Creation. As a solution for the same problem he wrote tool for concurrent index creation.
I plan to test this approach in my environment.


Read More...

Wrong results with 'hash group by' aggregation on 10.2.0.2

Thursday, July 7, 2011 0 comments
If you are running 10.2.0.2 in your production environment check document [ID 4604970.8] on Oracle support site.


This week colleague noticed that query, he wanted to use for some reports, returns different results than he expected. He made some additional checks using Microsof Excel and confirmed that something is wrong. He was getting incorrect results.

My first suspicion was that probably something is wrong with query or maybe logical corruption happened.

Example of the queries (I’ve changed names of the table/columns):

-- partitioned_table (date range partitioned table)

create table temp_tab_2011
as
select col, date_to,  sum (col1) col1, 
       sum (col2) f_col1, sum (col3) col3, 
    sum (col4) col4, 
    sum (col5) col5, count(*) cnt
from partitioned_table t
where t.date_to >= to_date ('01.02.2011 00:00:00', 'dd.mm.yyyy HH24:MI:SS')
    and t.date_to < to_date ('01.07.2011 00:00:00', 'dd.mm.yyyy HH24:MI:SS')
group by col, date_to
/

create table temp_tab_2010_2011
as
select col, date_to, sum (col1) col1, 
       sum (f_col1) f_col1, sum (col3) col3, 
    sum (col4) col4, 
    sum (col5) col5, count(*) cnt
from partitioned_table t
where t.date_to >= to_date ('01.02.2010 00:00:00', 'dd.mm.yyyy HH24:MI:SS')
    and t.date_to < to_date ('01.07.2011 00:00:00', 'dd.mm.yyyy HH24:MI:SS')
group by col, date_to
/

-- COMPARASION QUERY
select j.col, j.col1, j.col2, j.col3, j.cnt, d.col, d.col1, d.col2, d.col3, d.cnt, j.col1 - d.col1
from
temp_tab_2011 j, temp_tab_2010_2011 d
where 1=1
and j.col = d.col
and j.date_to = d.date_to
and j.date_to = to_date ('01.06.2011 00:00:00', 'dd.mm.yyyy HH24:MI:SS')
and d.date_to = to_date ('01.06.2011 00:00:00', 'dd.mm.yyyy HH24:MI:SS')
and (j.col1 - d.col1) <> 0
/


We expected 0 rows returned for comparison query but we received some rows - which was serious issue.



I’ve restored that database on another machine. Checked files for logical corruption, rebuilded indexes into another tablespace, rebuilded table into another tablespace and re-created temporary tablespace - but still we were receiving wrong results.


Luckily for us I’ve noticed this blog post during my searches for answer on Google:
http://www.oracloid.com/2006/05/hash-group-by-can-give-wrong-result-in-oracle-102/

Queries were using this plan:
(click "view source" to get better output)
----------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                            | Name                          | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------------------------------------------------------------------------------------
|   0 | CREATE TABLE STATEMENT               |                               |       |       |       | 43069 (100)|          |       |       |
|   1 |  LOAD AS SELECT                      |                               |       |       |       |            |          |       |       |
|   2 |   PARTITION RANGE ITERATOR           |                               |   631K|    50M|       | 39729   (1)| 00:13:08 |    57 |    74 |
|   3 |    HASH GROUP BY                     |                               |   631K|    50M|   219M| 39729   (1)| 00:13:08 |       |       |
|   4 |     TABLE ACCESS BY LOCAL INDEX ROWID| PARTITIONED_TABLE             |  1557K|   124M|       | 20999   (1)| 00:06:56 |    57 |    74 |
|*  5 |      INDEX RANGE SCAN                | IX_DATE                       |  1557K|       |       |  3874   (1)| 00:01:17 |    57 |    74 |
----------------------------------------------------------------------------------------------------------------------------------------------

In our version 10.2.0.2 wrong results are possible from the result of a non-distinct aggregation with a group by when HASH GROUB BY is used. We were hitting that bug.


In session level we changed undocumented parameter "_gby_hash_aggregation_enabled" to FALSE and executed queries again. Result was as expected - 0 rows returned.

It is noted in Metalink note that this issue is fixed in 10.2.0.3 version so I’ve upgraded test instance to 10.2.0.4 version and that solved problems with incorrect results.


This is nasty Oracle bug and if you are running 10.2.0.1 or 10.2.0.2 check your results :)


Read More...

Recover lost datafile without valid backup

Thursday, June 9, 2011 5 comments
I’ve decided to perform this test case and publish blog post after reading this topic on OTN forums - http://bit.ly/mLwaA9

How to recover lost datafile when you don’t have valid backup of your database. Your database is running in archivelog mode and you have all necessary archive logs.

In situation when you don’t have valid backup and your datatabase is running in noarchivelog mode - your datafile is lost forever.

So take regular backups of your database and enable archivelog mode ;-)


Environment for this test:
OS: OEL 5
DB: Oracle EE 10.2.0.4

To check am I running my database in archivelog mode.
SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     70
Next log sequence to archive   72
Current log sequence           72


I will create new tablespace for my test and name it - TS_TEST.

SQL> create tablespace ts_test datafile '/u01/oradata/orcl/ts_test01.dbf' size 256M;

Tablespace created.


Now to create table in TS_TEST tablespace. I will fill this table with 1000000 rows using Tom Kyte's script.

SQL> create table bigtab tablespace ts_test
  2  as
  3  select rownum id, a.*
  from all_objects a
 where 1=0;  4    5
 
 
 
SQL> select tablespace_name, segment_name from 
2         dba_segments where segment_name = 'BIGTAB';

TABLESPACE_NAME                SEGMENT_NAME
--------------- ---------------------------------------
TS_TEST                        BIGTAB
 
--
-- I'VE EXCLUDED SCRIPT LINES CAUSE
-- IT'S NOT IMPORTANT FOR THIS TEST CASE
--
SQL> select count(*) from bigtab;

  COUNT(*)
----------
   1000000



I can see in alert log that there were 5 log switches during insert.

Starting control autobackup
Control autobackup written to DISK device
        handle '/u01/app/oracle/flash_recovery_area/ORCL/autobackup/2011_06_09/o1_mf_s_753372242_6z1folfy_.bkp'
Completed: create tablespace ts_test datafile '/u01/oradata/orcl/ts_test01.dbf' size 256M
Thu Jun  9 14:07:42 2011
Thread 1 advanced to log sequence 73 (LGWR switch)
  Current log# 1 seq# 73 mem# 0: /u01/oradata/orcl/redo01.log
Thu Jun  9 14:07:45 2011
Thread 1 advanced to log sequence 74 (LGWR switch)
  Current log# 2 seq# 74 mem# 0: /u01/oradata/orcl/redo02.log
Thu Jun  9 14:10:55 2011
Thread 1 cannot allocate new log, sequence 75
Checkpoint not complete
  Current log# 2 seq# 74 mem# 0: /u01/oradata/orcl/redo02.log
Thu Jun  9 14:10:56 2011
Thread 1 advanced to log sequence 75 (LGWR switch)
  Current log# 3 seq# 75 mem# 0: /u01/oradata/orcl/redo03.log
Thu Jun  9 14:10:58 2011
Thread 1 advanced to log sequence 76 (LGWR switch)
  Current log# 1 seq# 76 mem# 0: /u01/oradata/orcl/redo01.log
Thread 1 cannot allocate new log, sequence 77
Checkpoint not complete
  Current log# 1 seq# 76 mem# 0: /u01/oradata/orcl/redo01.log
Thu Jun  9 14:11:00 2011
Thread 1 advanced to log sequence 77 (LGWR switch)
  Current log# 2 seq# 77 mem# 0: /u01/oradata/orcl/redo02.log



To simulate disaster I will delete “ts_test01.dbf” datafile using OS command and flush buffer cache several times.

SQL> !rm /u01/oradata/orcl/ts_test01.dbf

SQL> alter system flush buffer_cache;

System altered.

SQL> /

System altered.

SQL> /

System altered.


Now my query throws an error because datafile 8 is missing.

SQL> select count(*) from bigtab;
select count(*) from bigtab
*
ERROR at line 1:
ORA-01116: error in opening database file 8
ORA-01110: data file 8: '/u01/oradata/orcl/ts_test01.dbf'
ORA-27041: unable to open file
Linux Error: 2: No such file or directory
Additional information: 3


How to recover this datafile when you don’t have valid backup to restore it?

Trick is to create new datafile based on the file “ts_test01.dbf” using command:
ALTER DATABASE CREATE DATAFILE 'xxxx.dbf' AS 'xxxx.dbf' and fill it with data from archive logs.


SQL> alter database create datafile '/u01/oradata/orcl/ts_test01.dbf' as '/u01/oradata/orcl/ts_test01.dbf';
alter database create datafile '/u01/oradata/orcl/ts_test01.dbf' as '/u01/oradata/orcl/ts_test01.dbf'
*
ERROR at line 1:
ORA-01182: cannot create database file 8 - file is in use or recovery
ORA-01110: data file 8: '/u01/oradata/orcl/ts_test01.dbf'

To avoid error - before creating new datafile, you must take the existing datafile (or the tablespace) offline.


SQL> alter database datafile 8 offline drop;

Database altered.


SQL> alter database create datafile '/u01/oradata/orcl/ts_test01.dbf' as '/u01/oradata/orcl/ts_test01.dbf';

Database altered.



Now I have new empty datafile which I will fill with records applying archive logs over it during recovery.

SQL> recover datafile 8;
ORA-00279: change 92069000 generated at 06/09/2011 14:03:59 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_72_%u_.arc
ORA-00280: change 92069000 for thread 1 is in sequence #72


Specify log: {=suggested | filename | AUTO | CANCEL}

ORA-00279: change 92069587 generated at 06/09/2011 14:07:42 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_73_%u_.arc
ORA-00280: change 92069587 for thread 1 is in sequence #73
ORA-00278: log file
'/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_72_6z1fw
gvn_.arc' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}

ORA-00279: change 92069713 generated at 06/09/2011 14:07:45 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_74_%u_.arc
ORA-00280: change 92069713 for thread 1 is in sequence #74
ORA-00278: log file
'/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_73_6z1fw
kyj_.arc' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}

ORA-00279: change 92070100 generated at 06/09/2011 14:10:56 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_75_%u_.arc
ORA-00280: change 92070100 for thread 1 is in sequence #75
ORA-00278: log file
'/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_74_6z1g2
jh1_.arc' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}

ORA-00279: change 92071034 generated at 06/09/2011 14:10:58 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_76_%u_.arc
ORA-00280: change 92071034 for thread 1 is in sequence #76
ORA-00278: log file
'/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_75_6z1g2
lbg_.arc' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}

ORA-00279: change 92071751 generated at 06/09/2011 14:11:00 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_77_%u_.arc
ORA-00280: change 92071751 for thread 1 is in sequence #77
ORA-00278: log file
'/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2011_06_09/o1_mf_1_76_6z1g2
nmz_.arc' no longer needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}

Log applied.
Media recovery complete.

Recovery is successfully completed and now to alter datafile 8 to be online again.



SQL> alter database datafile 8 online;

Database altered.


-- CHECK
SQL> select count(*) from bigtab;

  COUNT(*)
----------
   1000000


-- CLEANUP

SQL> drop tablespace ts_test including contents and datafiles;

Tablespace dropped.

Read More...