Validating backups and database files using RMAN

Tuesday, December 29, 2009 0 comments
The most critical tasks for every DBA are database backup and recovery procedures. Knowledge of all Oracle backup and recovery scenarios is practically the most essential skill. For almost every problem there is some solution, but if you can't restore/recover data you are in deep trouble.

To avoid possible problems it is important to regularly monitor backups and to perform validations using various tools. You can't say that you have valid backup until you successfully perform test restore on another machine. There are several methods to validate backups taken with RMAN. Restores on another machine is the best validation method and you should perform it regularly according how critical your database is. Big benefit from performing test restores is valuable practice you'll get.



Another method for validating backups is using RMAN commands.
As a part of my daily tasks I'am using this simple RMAN script for backup validation:

run
{
restore database validate;
restore archivelog from time 'sysdate-1' validate;
restore controlfile validate;
restore spfile validate;
}

In this script I'am doing this checks:
- check most recent RMAN database backup
- check most recent RMAN archivelog(s) backup
- check most recent controlfile backup
- check most recent RMAN spfile backup

To check older backups just set SET UNTIL clause.

Using RESTORE VALIDATION I can verify whether database could be successfully restored or not using existing backups.

It is good practice to automate this tasks and then grep logs for errors. RESTORE VALIDATE operation takes time and it consumes valuable resources so you have to think about when to schedule validation.

You can consider this RMAN validation as addition to your regular trial recoveries.


About using RMAN to check Logical & Physical Database Corruption
During a RMAN backup or RMAN 'backup validate' every block currently used or previously used is read into memory then written to another portion of memory. During this memory to memory write the block is checked for corruption. Therefore RMAN's BACKUP command with the VALIDATE and CHECK LOGICAL clauses allow a Database Adminstrator to quickly check for both physical and logical corruption. If the initialization parameter DB_BLOCK_CHECKSUM=TRUE, specifying CHECK LOGICAL detects all types of corruption that are possible to detect.

Physical corruption can be defined as a damage to the internal data structure and prevents Oracle from being able to read the data in the corrupted blocks.
Logical corruption is situation when Oracle is able to find the data, but data values are incorrect.

To validate all datafiles execute:
RMAN> run
2> {
3> allocate channel c1 type disk;
4> backup check logical validate database;
5> release channel c1;
6> }

After validate is complete query v$database_block_corruption to see if there are any rows listed there. Additionally you can check alert log for corruptions.





REFERENCES:
Metalink notes: [ID 338607.1], [ID 283053.1], [ID 428570.1]

Read More...

New design and best wishes...

Saturday, December 26, 2009 2 comments
First of all to wish Merry Christmas and Happy New Year to all readers of this blog. I wish you even more success in your business and family life in the year 2010.

About new design...

I had some free time to play with my blog so I decided to change design. Implementation of new design took me more time then I expected, because of my humble CSS/HTML knowledge, but at the and I am pleased with new look.

Thank you all for your participation and comments. I wish even more posts, visits and comments for this blog in next year :)

Read More...

Point in Time Recovery fails with ORA-01841 on Oracle 10.2 SE

Wednesday, December 23, 2009 4 comments
Yesterday I've decided to do point in time recovery for Oracle 10.2.0.4 SE database.

I've used the same script that was working before on other databases - but my operation failed with ORA-01841 error.

After spending some time on Google I found that this error was mostly related to illegal year entered.


ORA-01841:
(full) year must be between -4713 and +9999, and not be 0
Cause: Illegal year entered
Action: Input year in the specified range

To test and fix script I've decided to execute it on test Oracle 10.2.0.4 EE database but there everything worked without any error.



Then I've searched little more and found Metalink note 370500.1.

So I was encountering this error because NLS_LANG was not specified.

Little test case:
$ uname -a
Linux testhost2 2.6.18-164.el5 #1 SMP Thu Sep 3 02:16:47 EDT 2009 i686 i686 i386 GNU/Linux

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Release 10.2.0.4.0 - Production
PL/SQL Release 10.2.0.4.0 - Production
CORE    10.2.0.4.0      Production
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production


SQL> select to_char(sysdate,'dd.mm.yyyy hh24:mi:ss')
2  from dual;

TO_CHAR(SYSDATE,'DD
-------------------
23.12.2009 08:51:03



First script execution:

RMAN> shutdown immediate;

using target database control file instead of recovery catalog
database closed
database dismounted
Oracle instance shut down

RMAN> startup mount;

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area    1342177280 bytes

Fixed Size                     1267380 bytes
Variable Size               1191184716 bytes
Database Buffers             134217728 bytes
Redo Buffers                  15507456 bytes

RMAN> run
2> {
3> sql 'alter session set NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS"';
4> set until time='23.12.2009 08:51:03';
5> restore database;
6> recover database;
7> }

sql statement: alter session set NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS"

executing command: SET until clause

Starting restore at 23.12.2009 08:56:19
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=158 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /oratest/oradata/rasapp/datafiles/system01.dbf
restoring datafile 00002 to /oratest/oradata/rasapp/datafiles/undotbs101.dbf
restoring datafile 00003 to /oratest/oradata/rasapp/datafiles/sysaux01.dbf
restoring datafile 00004 to /oratest/oradata/rasapp/datafiles/users01.dbf
channel ORA_DISK_1: reading from backup piece /home/oratest/rmanbkp/hsl1iqaa_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/home/oratest/rmanbkp/hsl1iqaa_1_1 tag=TAG20091222T204706
channel ORA_DISK_1: restore complete, elapsed time: 00:07:27
Finished restore at 23.12.2009 09:03:48

Starting recover at 23.12.2009 09:03:49
using channel ORA_DISK_1

starting media recovery
media recovery failed
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 12/23/2009 09:03:50
RMAN-11003: failure during parse/execution of SQL statement: alter database recover if needed
start until time 'DEC 23 2009 08:51:03'
ORA-01841: (full) year must be between -4713 and +9999, and not be 0



For second execution I've changed script a little:

RMAN> run
2> {
3> set until time="to_date('23.12.2009 08:51:03','DD.MM.YYYY HH24:MI:SS')";
4> restore database;
5> recover database;
6> }
...
...
media recovery failed
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 12/23/2009 09:23:01
RMAN-11003: failure during parse/execution of SQL statement: alter database recover if needed
start until time 'DEC 23 2009 08:51:03'
ORA-01841: (full) year must be between -4713 and +9999, and not be 0



Then I've tried with setting NLS_LANG to .UTF8.
$ export NLS_LANG=.UTF8
$ echo $NLS_LANG
.UTF8
$ rman target /

Recovery Manager: Release 10.2.0.4.0 - Production on Wed Dec 23 09:24:33 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

connected to target database: RASAPP (DBID=1650089695, not open)

RMAN> run
2> {
3> sql 'alter session set NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS"';
4> set until time='23.12.2009 08:51:03';
5> restore database;
6> recover database;
7> }

using target database control file instead of recovery catalog
sql statement: alter session set NLS_DATE_FORMAT="DD.MM.YYYY HH24:MI:SS"

executing command: SET until clause

Starting restore at 23.12.2009 09:25:09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=158 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00001 to /oratest/oradata/rasapp/datafiles/system01.dbf
restoring datafile 00002 to /oratest/oradata/rasapp/datafiles/undotbs101.dbf
restoring datafile 00003 to /oratest/oradata/rasapp/datafiles/sysaux01.dbf
restoring datafile 00004 to /oratest/oradata/rasapp/datafiles/users01.dbf
channel ORA_DISK_1: reading from backup piece /home/oratest/rmanbkp/hsl1iqaa_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/home/oratest/rmanbkp/hsl1iqaa_1_1 tag=TAG20091222T204706
channel ORA_DISK_1: restore complete, elapsed time: 00:07:56
Finished restore at 23.12.2009 09:33:07

Starting recover at 23.12.2009 09:33:08
using channel ORA_DISK_1

starting media recovery

archive log thread 1 sequence 1 is already on disk as file /oratest/oradata/archlogs/1_1_706308053.dbf
archive log thread 1 sequence 2 is already on disk as file /oratest/oradata/archlogs/1_2_706308053.dbf
archive log thread 1 sequence 3 is already on disk as file /oratest/oradata/archlogs/1_3_706308053.dbf
archive log thread 1 sequence 1 is already on disk as file /oratest/oradata/archlogs/1_1_706312614.dbf
archive log filename=/oratest/oradata/archlogs/1_1_706308053.dbf thread=1 sequence=1
archive log filename=/oratest/oradata/archlogs/1_2_706308053.dbf thread=1 sequence=2
archive log filename=/oratest/oradata/archlogs/1_3_706308053.dbf thread=1 sequence=3
media recovery complete, elapsed time: 00:00:34
Finished recover at 23.12.2009 09:33:45


Now everything worked without error.

I must remember that NLS_LANG affects only 10.2 Standard Edition - on Enterprise Edition everything works fine.


After database point-in-time recovery, I must open the database with the RESETLOGS option.

$ sqlplus "/as sysdba"

SQL> alter database open resetlogs;

Database altered.

Read More...

Transportable Tablespace Import Fails With ORA-19721

Saturday, November 28, 2009 2 comments
This blog post is linked with post before : Cross-platform migration - large Oracle 9i db from Solaris 64 to Linux 64

SOURCE: Solaris 64bit, Oracle 10.2.0.4
TARGET: Linux 32bit, Oracle 10.2.0.4

I was playing a little with transportable tablespace imports and experienced transportable tablespace import failure with ORA-19721 error.

After dropping contents of DATA1 tablespace I've decided to import same metadata I've successfully imported before - but import failed.



SQL> alter tablespace DATA1 offline normal;

Tablespace altered.

SQL> drop tablespace DATA1 including contents;

Tablespace dropped.



$ imp userid=\'/ as sysdba\' TRANSPORT_TABLESPACE=Y 
datafiles='/oracle/oradata/rman_convert/data01.dbf' 
file=/oracle/oradata/rman_convert/data1.dmp

...
IMP-00017: following statement failed with ORACLE error 19721:
"BEGIN   sys.dbms_plugts.checkDatafile(NULL,1096888558,12,6400,13,12,0,0,104"
"6791,1047073,162043,50331650,NULL,NULL,NULL); END;"
IMP-00003: ORACLE error 19721 encountered
ORA-19721: Cannot find datafile with absolute file number 12 in tablespace DATA1
ORA-06512: at "SYS.DBMS_PLUGTS", line 2065
...



To investigate problem little more I've created new tablespace DATA2 on source database and moved segments from DATA1 tablespace to DATA2.
SQL> set lines 200
SQL> col segment_name for a25
SQL> col segment_type for a20
SQL> col tablespace_name for a20
SQL>  select segment_name, segment_type, tablespace_name 
from dba_segments where tablespace_name='DATA1';

SEGMENT_NAME              SEGMENT_TYPE         TABLESPACE_NAME
------------------------- -------------------- --------------------
TEST_TABLE                TABLE                DATA1


SQL> create tablespace DATA2 datafile '/oradata1/oradata/data02.dbf' size 50M;

Tablespace created.


SQL> alter table MSUTIC.TEST_TABLE move tablespace DATA2;

Table altered.



Now to convert new tablespace and export metadata:
SQL> alter tablespace DATA2 read only;

Tablespace altered.


RMAN> convert tablespace 'DATA2'
2> to platform="Linux IA (32-bit)"
3> db_file_name_convert='/oradata1/oradata/data02.dbf','/oradata1/rman_convert/data02.dbf';

Starting backup at 28-NOV-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=135 devtype=DISK
channel ORA_DISK_1: starting datafile conversion
input datafile fno=00013 name=/oradata1/oradata/data02.dbf
converted datafile=/oradata1/rman_convert/data02.dbf
channel ORA_DISK_1: datafile conversion complete, elapsed time: 00:00:01
Finished backup at 28-NOV-09


$ exp userid=\'/ as sysdba\' transport_tablespace=y tablespaces=DATA2 file=data2.dmp statistics=none

Export: Release 10.2.0.4.0 - Production on Sat Nov 28 19:39:55 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Export done in UTF8 character set and UTF8 NCHAR character set
server uses AL32UTF8 character set (possible charset conversion)
Note: table data (rows) will not be exported
About to export transportable tablespace metadata...
For tablespace DATA2 ...
. exporting cluster definitions
. exporting table definitions
. . exporting table                     TEST_TABLE
. exporting referential integrity constraints
. exporting triggers
. end transportable tablespace metadata export
Export terminated successfully without warnings.



Copy converted datafile and tablespace metadata to target machine:
$ scp rman_convert/* oracle@10.10.10.18:/oracle/oradata/rman_convert
oracle@10.2.10.18's password:
data02.dbf           100% |******************************************| 51208 KB    00:04
data2.dmp            100% |******************************************|  3072       00:00



First tablespace import to target database:
$ imp userid=\'/ as sysdba\' TRANSPORT_TABLESPACE=Y 
datafiles='/oracle/oradata/rman_convert/data02.dbf' 
file=/oracle/oradata/rman_convert/data2.dmp

Import: Release 10.2.0.4.0 - Production on Sat Nov 28 19:44:42 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Export file created by EXPORT:V10.02.01 via conventional path
About to import transportable tablespace(s) metadata...
import done in UTF8 character set and UTF8 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
. importing SYS's objects into SYS
. importing SYS's objects into SYS
. importing MSUTIC's objects into MSUTIC
. . importing table                   "TEST_TABLE"
. importing SYS's objects into SYS
Import terminated successfully without warnings.

Import execution was successful.


I will drop contents of DATA2 tablespace and try second import:
SQL> drop tablespace DATA2 including contents;

Tablespace dropped.

$ imp userid=\'/ as sysdba\' TRANSPORT_TABLESPACE=Y 
datafiles='/oracle/oradata/rman_convert/data02.dbf' 
file=/oracle/oradata/rman_convert/data2.dmp

Import: Release 10.2.0.4.0 - Production on Sat Nov 28 19:46:23 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Export file created by EXPORT:V10.02.01 via conventional path
About to import transportable tablespace(s) metadata...
import done in UTF8 character set and UTF8 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
. importing SYS's objects into SYS
. importing SYS's objects into SYS
. importing MSUTIC's objects into MSUTIC
. . importing table                   "TEST_TABLE"
. importing SYS's objects into SYS
Import terminated successfully without warnings.


Again successful import.


This time I will change tablespace mode to read/write, drop contents and try import for third time:
SQL> alter tablespace DATA2 read write;

Tablespace altered.

SQL> alter tablespace DATA2 offline normal;

Tablespace altered.

SQL> drop tablespace DATA2 including contents;

Tablespace dropped.


$ imp userid=\'/ as sysdba\' TRANSPORT_TABLESPACE=Y 
datafiles='/oracle/oradata/rman_convert/data02.dbf' 
file=/oracle/oradata/rman_convert/data2.dmp

Import: Release 10.2.0.4.0 - Production on Sat Nov 28 19:48:11 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Export file created by EXPORT:V10.02.01 via conventional path
About to import transportable tablespace(s) metadata...
import done in UTF8 character set and UTF8 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
. importing SYS's objects into SYS
. importing SYS's objects into SYS
IMP-00017: following statement failed with ORACLE error 19721:
"BEGIN   sys.dbms_plugts.checkDatafile(NULL,1096888558,13,6400,14,13,0,0,110"
"4430,1104611,162043,54525954,NULL,NULL,NULL); END;"
IMP-00003: ORACLE error 19721 encountered
ORA-19721: Cannot find datafile with absolute file number 13 in tablespace DATA2
ORA-06512: at "SYS.DBMS_PLUGTS", line 2065
ORA-06512: at line 1
IMP-00000: Import terminated unsuccessfully



This time import fails with error:
ORA-19721: Cannot find datafile with absolute file number 13 in tablespace DATA2

Error probably occurs because "alter tablespace DATA2 read write" statement modified data file header so we can't use exported metadata to plug in modified data file.
Changing mode to read/write also updates control file entry for datafile.

When tablespace is in read only mode, then datafile headers are not updated and import works perfectly with exported metadata.

Read More...

Cross-platform migration - large Oracle 9i db from Solaris 64bit to Linux 32bit

Friday, November 27, 2009 2 comments
Question for this post is - what is the best way to migrate >500GB Oracle 9.2.0.6 database from Solaris 64 to Linux 64 operating system?

As imp/exp or impdp/expdp would probably be very slow option better choice is cross-platform migration.

In this case we have to upgrade db from 9i to 10g and then migrate tablespaces using cross-platform tablespace conversion with transport tablespace feature.



Why upgrading to 10g?
Because we don't have RMAN cross-platform tablespace conversion in Oracle 9i.


This time I will do simple demo case how to migrate Oracle 9.2.0.6 database from Solaris 64bit to Linux 32bit. Similar actions are performed when Linux 64bit is target platform instead of Linux 32bit.


UPGRADE PROCESS:

1. First create new 10g installation using OUI on Solaris machine.

2. Run utlu102i.sql (in 10g ORACLE_HOME) script in Oracle 9i environment while Oracle 9i db is running.

SQL> spool /tmp/preupgrade.log
SQL> @utlu102i.sql
SQL> spool off


Check in log file what actions you have to do before upgrading.


3. Create pfile from current spfile of Oracle 9i database and copy that parameter file to 10g ORACLE_HOME/dbs.

SQL> create pfile from spfile;

File created.


cp initdbtest.ora /oradata1/oratest/product/1020/dbs/


Edit parameter file according to recommendations from pre-upgrade tool.

I've set this parameter values for this little demo:

pga_aggregate_target = 25M
shared_pool_size = 200M
streams_pool_size = 64M
session_max_open_files = 20



4. Few more steps on 9i database.

- Check for invalid objects and recompile:
SQL> SELECT UNIQUE object_name, object_type, owner
FROM dba_objects WHERE status='INVALID';

SQL>@?/rdbms/admin/utlrp



- Set SYSTEM as default tablespace for SYS and SYSTEM users.

SQL> select username, default_tablespace from dba_users
where username in ('SYS','SYSTEM');

To modify use:

SQL> alter user sys default tablespace SYSTEM;
SQL> alter user system default tablespace SYSTEM;



5. Shutdown database and stop listener.


6. Very IMPORTANT setp - create backup!


7. Edit "/var/opt/oracle/oratab" entry and change entry of old 9i ORACLE_HOME to 10g ORACLE_HOME.


8. Login using 10g environment and execute following statements:

-bash-3.00$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Nov 26 21:57:46 2009

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup upgrade pfile='$ORACLE_HOME/dbs/initdbtest.ora';
ORACLE instance started.

Total System Global Area  473956352 bytes
Fixed Size                  2041464 bytes
Variable Size             432019848 bytes
Database Buffers           33554432 bytes
Redo Buffers                6340608 bytes
Database mounted.
Database opened.

SQL>
CREATE TABLESPACE SYSAUX
DATAFILE '/oradata1/oradata/sysaux01.dbf'
SIZE 500M REUSE
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO
ONLINE;



9. Start with upgrade process and check log file for errors:

SQL> set echo on
SQL> SPOOL /tmp/upgrade.log
SQL> @?/rdbms/admin/catupgrd.sql
SQL> spool off




10. Set compatible value to 10.2.0.4.0, create spfile from pfile, restart and recompile.


11. Run post-upgrade status tool - utlu102s.sql:

@?/rdbms/admin/utlu102s.sql


Check status and version of components.


12. Enable listener in new home.


13. I will skip gathering stats.


MIGRATION:

1. List internal names for each platform supporting cross-platform data transport:

SQL> set lines 200
SQL> set pages 999
SQL> select * from V$transportable_platform;

PLATFORM_ID PLATFORM_NAME                                   ENDIAN_FORMAT
----------- ----------------------------------------------- --------------
1 Solaris[tm] OE (32-bit)                             Big
2 Solaris[tm] OE (64-bit)                             Big
7 Microsoft Windows IA (32-bit)                       Little
10 Linux IA (32-bit)                                   Little
6 AIX-Based Systems (64-bit)                          Big
3 HP-UX (64-bit)                                      Big
5 HP Tru64 UNIX                                       Little
4 HP-UX IA (64-bit)                                   Big
11 Linux IA (64-bit)                                   Little
15 HP Open VMS                                         Little
8 Microsoft Windows IA (64-bit)                       Little
9 IBM zSeries Based Linux                             Big
13 Linux x86 64-bit                                    Little
16 Apple Mac OS                                        Big
12 Microsoft Windows x86 64-bit                        Little
17 Solaris Operating System (x86)                      Little
18 IBM Power Based Linux                               Big
20 Solaris Operating System (x86-64)                   Little
19 HP IA Open VMS                                      Little

19 rows selected.



In this case:
source platform = Solaris[tm] OE (64-bit) - ENDIAN_FORMAT = big
target platform = Linux IA (32-bit) - ENDIAN_FORMAT = little


Endian format of the source platform is different from that of the destination platform so we must perform conversion. The endian format of the datafiles in the transportable tablespace set must be converted to match destination platform.


2. Check if tablespace is self contained.
SQL> begin
2     dbms_tts.transport_set_check('TBS_NAME',TRUE);
3  end;
4  /

PL/SQL procedure successfully completed.

SQL> select * from TRANSPORT_SET_VIOLATIONS;

no rows selected


If select query shows any rows we should transport mentioned tablespace also.


3. Put in read only mode tablespaces chosen for migration.
SQL> select 'alter tablespace '||tablespace_name||' read only;'
2  from dba_tablespaces
3  where tablespace_name not in ('SYSTEM','UNDOTBS1','TEMP','SYSAUX','DRSYS',
4                                'XDB','ODM','CWMLITE','USERS');

'ALTERTABLESPACE'||TABLESPACE_NAME||'READONLY;'
----------------------------------------------------------
alter tablespace EXAMPLE read only;
alter tablespace INDX read only;
alter tablespace TOOLS read only;
alter tablespace DATA1 read only;

4 rows selected.


For this test case I've chosen specified tablespaces.


5. Use RMAN to convert datafiles into the endian format of the destination host:

RMAN> convert tablespace 'DATA1'
to platform="Linux IA (32-bit)"
db_file_name_convert='/oradata1/oradata/data01.dbf','/oradata1/rman_convert/data01.dbf';

Starting backup at 27-NOV-09
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile conversion
input datafile fno=00012 name=/oradata1/oradata/data01.dbf
converted datafile=/oradata1/rman_convert/data01.dbf
channel ORA_DISK_1: datafile conversion complete, elapsed time: 00:00:01
Finished backup at 27-NOV-09


Execute this for all tablespaces you would like to migrate.


6. Create transportable tablespace metadata:

-bash-3.00$ exp userid=\'/ as sysdba\' transport_tablespace=y
tablespaces=DATA1 file=data1.dmp statistics=none

Export: Release 10.2.0.4.0 - Production on Fri Nov 27 14:41:28 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Export done in UTF8 character set and UTF8 NCHAR character set
server uses AL32UTF8 character set (possible charset conversion)
Note: table data (rows) will not be exported
About to export transportable tablespace metadata...
For tablespace DATA1 ...
. exporting cluster definitions
. exporting table definitions
. . exporting table                     TEST_TABLE
. exporting referential integrity constraints
. exporting triggers
. end transportable tablespace metadata export
Export terminated successfully without warnings.


Export metadata of all tablespaces you would like to migrate.


7. Create 10.2.0.4 database on target Linux machine.

Notice!
Check database character set on source and target database.
Create users on target database if you are using imp/exp tools.


8. Copy converted datafiles and export output to target machine.

$ scp rman_convert/* oracle@10.10.10.18:/oracle/oradata/rman_convert



9. Import transportable tablespace in target databse:

$ imp userid=\'/ as sysdba\' TRANSPORT_TABLESPACE=Y
datafiles='/oracle/oradata/rman_convert/data01.dbf' 
file=/oracle/oradata/rman_convert/data1.dmp

Import: Release 10.2.0.4.0 - Production on Fri Nov 27 15:02:28 2009

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Export file created by EXPORT:V10.02.01 via conventional path
About to import transportable tablespace(s) metadata...
import done in UTF8 character set and UTF8 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
. importing SYS's objects into SYS
. importing SYS's objects into SYS
. importing MSUTIC's objects into MSUTIC
. . importing table                   "TEST_TABLE"
. importing SYS's objects into SYS
Import terminated successfully without warnings.


Use this script to import all transportable tablespaces.


10. Excerpt from alert log:

...
Fri Nov 27 15:02:28 2009
Plug in tablespace DATA1 with datafile
'/oracle/oradata/rman_convert/data01.dbf'
Plug in tablespace TOOLS with datafile
'/oracle/oradata/rman_convert/tools01.dbf'
...



11. Put imported tablespaces in read/write mode:
SQL> alter tablespace DATA1 read write;

Tablespace altered.

SQL> alter tablespace TOOLS read write;

Tablespace altered.

...


12. Small check on target database:

SQL> conn msutic/pwd
Connected.
SQL> select count(*) from test_table;

COUNT(*)
----------
50327




References:
http://youngcow.net/doc/oracle10g/backup.102/b14191/dbxptrn001.htm

Read More...

Tracing AUTOEXTEND tablespace

Sunday, November 8, 2009 0 comments
In this blog post I will try to examine how AUTOEXTEND operation can affect on performance. Hemant K Chitale already wrote nice article on this subject : AUTOEXTEND ON Next Size, where he wrote about Oracle "default" Autoextend NEXT size.

This topic "Autoextend or not?" from Oracle-L mailing list is great addition to this blog post. You can read there about different usages of AUTOEXTEND option in real life experience of some very good DBA's.



I will perform small test case using 10046 trace to get more information.

For my testing purposes I will use:
SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod
PL/SQL Release 10.2.0.4.0 - Production
CORE    10.2.0.4.0      Production
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production


SQL> select log_mode from v$database;

LOG_MODE
------------
NOARCHIVELOG


$ uname -a
Linux linux2.loc 2.6.18-128.el5 #1 SMP Wed Jan 21 07:58:05 EST 2009 i686 i686 i386 GNU/Linux



First to create three different tablespaces with one table in each tablespace.

CREATE TABLESPACE TESTTBS1 DATAFILE
'/oradata/testtbs1.dbf' SIZE 5M AUTOEXTEND ON NEXT 56K MAXSIZE UNLIMITED
LOGGING
ONLINE
PERMANENT
EXTENT MANAGEMENT LOCAL AUTOALLOCATE
BLOCKSIZE 8K
SEGMENT SPACE MANAGEMENT AUTO
FLASHBACK ON;

create table big_table1 tablespace testtbs1
as
select rownum id, a.*
from all_objects a
where 1=0
/



CREATE TABLESPACE TESTTBS2 DATAFILE
'/oradata/testtbs3.dbf' SIZE 5M AUTOEXTEND ON NEXT 25M MAXSIZE UNLIMITED
LOGGING
ONLINE
PERMANENT
EXTENT MANAGEMENT LOCAL AUTOALLOCATE
BLOCKSIZE 8K
SEGMENT SPACE MANAGEMENT AUTO
FLASHBACK ON;

create table big_table2 tablespace testtbs2
as
select rownum id, a.*
from all_objects a
where 1=0
/


CREATE TABLESPACE TESTTBS3 DATAFILE
'/oradata/testtbs2.dbf' SIZE 500M AUTOEXTEND OFF
LOGGING
ONLINE
PERMANENT
EXTENT MANAGEMENT LOCAL AUTOALLOCATE
BLOCKSIZE 8K
SEGMENT SPACE MANAGEMENT AUTO
FLASHBACK ON;

create table big_table3 tablespace testtbs3
as
select rownum id, a.*
from all_objects a
where 1=0
/


I will execute script below to insert 2000000 rows in every table (about 232 MB). This is script from Tom Kyte but just little modified.

SQL> declare
2          l_cnt number;
3          l_rows number := &1;
4  begin
5          insert
6          into big_table1
7          select rownum, a.*
8          from all_objects a
9          where rownum <= &1;
10
11          l_cnt := sql%rowcount;
12
13          commit;
14
15          while (l_cnt < l_rows)
16          loop
17                  insert into big_table1
18                  select rownum+l_cnt,
19                  OWNER,OBJECT_NAME,SUBOBJECT_NAME,OBJECT_ID, DATA_OBJECT_ID,OBJECT_TYPE,
20                  CREATED,LAST_DDL_TIME,TIMESTAMP,STATUS,TEMPORARY,GENERATED,SECONDARY
21                  from big_table1
22                  where rownum <= l_rows - l_cnt;
23                  l_cnt := l_cnt + sql%rowcount;
24                  commit;
25          end loop;
26  end;
27  /
Enter value for 1: 2000000
old   3:         l_rows number := &1;
new   3:         l_rows number := 2000000;
Enter value for 1: 2000000
old   9:         where rownum <= &1;
new   9:         where rownum <= 2000000;
My intention is to trace every script execution and to extract wait statistics from trace file using tkprof tool. First execution with AUTOEXTEND ON NEXT 56K:
Elapsed: 00:00:51.40

Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait  Total Waited
----------------------------------------   Waited  ----------  ------------
control file sequential read                 2187        0.40         24.20
db file sequential read                       162        0.06          1.29
change tracking file synchronous read         891        0.03          1.72
Data file init write                          390        0.34          5.99
db file single write                           81        0.00          0.02
control file parallel write                   243        0.00          0.17
rdbms ipc reply                                81        0.07          0.86
log file switch completion                      7        0.97          1.28
enq: CF - contention                            1        0.00          0.00
log buffer space                                1        0.04          0.04
log file switch (checkpoint incomplete)         3        0.97          1.09
Second execution with AUTOEXTEND ON NEXT 25M:
Elapsed: 00:00:35.13

Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait  Total Waited
----------------------------------------   Waited  ----------  ------------
db file sequential read                        26        0.03          0.28
control file sequential read                  243        0.23          4.91
change tracking file synchronous read          99        0.01          0.19
Data file init write                          243        0.31          6.52
db file single write                            9        0.00          0.00
control file parallel write                    27        0.00          0.01
rdbms ipc reply                                 9        0.16          0.25
log buffer space                               38        0.41          6.72
log file switch completion                      5        0.97          1.74
log file switch (checkpoint incomplete)         5        0.97          1.16
Third execution with AUTOEXTEND OFF:
Elapsed: 00:00:33.93

Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait  Total Waited
----------------------------------------   Waited  ----------  ------------
log file switch completion                      8        0.97          3.50
log file switch (checkpoint incomplete)        12        0.97          5.38
log buffer space                               42        0.82         13.58
About main wait events in my test cases: "control file sequential read" - process is waiting for blocks to be read from control file. "Data file init write" - occurs when Oracle auto extends datafile and formats the newly created space within the datafile. ... Interesting results if you ask me. I expected "Data file int write" wait event because it is closely related to datafile extensions, but "control file sequential read" event little surprised me. To get some better explanation why "control file sequential read" is related to auto extensions I tried to use Metalink but Metalink is not working very well during weekend because of upgrade process. Maybe I'll find more info later using newly upgraded Metalink. To conclude, different autoextend next sizes can significantly affect on overall performance so it is important to set proper value. References: Ask DBSpecialists: 'Data file init write' wait event

Read More...

Create a Database Link in another user's schema

Monday, October 26, 2009 3 comments
From documentation:

Restriction on Creating Database Links

You cannot create a database link in another user's schema, and you cannot qualify dblink with the name of a schema. Periods are permitted in names of database links, so Oracle Database interprets the entire name, such as ralph.linktosales, as the name of a database link in your schema rather than as a database link named linktosales in the schema ralph.)


CREATE DATABASE LINK

So documentation clearly states that you cannot create database link in another user's schema.



As I am doing all my dba tasks using admin accounts it is little unconvenient for me to search for other schema passwords in my documents or asking developers for this information.
All this is little time consuming so I've found more convenient way to create database links in another user's schema when I am pretty sure that my actions won't make disturbance.

Little demo case...

Task:
- Create database link in MSUTIC schema that connects to APP schema on TEST10 database.

My intention is to temporarily change MSUTIC user password, login as MSUTIC user, quickly create database link and at the end return password as it was before changing.

First to find out encrypted version of password (remember this info).
system@TEST11> select name, password
2  from sys.user$
3  where name='MSUTIC';

NAME                           PASSWORD
------------------------------ ------------------------------
MSUTIC                         66EE6D5F9AB42E0F

1 row selected.


Changed current MSUTIC password to temporary "tmp_pwd" and granted "create database link" privilege to MSUTIC.
system@TEST11> alter user msutic identified by tmp_pwd;

User altered.

system@TEST11> grant create database link to msutic;

Grant succeeded.




Connect as MSUTIC user using temporary password and creat database link.
system@TEST11> connect msutic/tmp_pwd@test11
Connected.

msutic@TEST11> CREATE DATABASE LINK APP_DB
2   CONNECT TO APP
3   IDENTIFIED BY app123
4   USING '(DESCRIPTION =
5         (ADDRESS_LIST =
6           (ADDRESS = (PROTOCOL = TCP)(HOST = linux01)(PORT = 1521))
7         )
8         (CONNECT_DATA =
9           (SID = TEST10)
10         )
11       )';

Database link created.



Simple test:
msutic@TEST11> select count(*) from app.app_log@app_db;

COUNT(*)
----------
355412

1 row selected.


DB Link is functioning perfectly and now I can revoke "create database link privilege" and return password as it was before.

msutic@TEST11> conn system@test11
Enter password:
Connected.

system@TEST11> revoke create database link from msutic;

Revoke succeeded.

system@TEST11> alter user msutic identified by values '66EE6D5F9AB42E0F';

User altered.



The key thing is to have prepared scripts to do this actions as quick as possible because when you change current schema password to temporary other users or apps will not be able to log in.

Read More...

Cursor_sharing=FORCE causing "ORA-01008: not all variables bound" error on Oracle 10.2.0.4

Thursday, October 22, 2009 2 comments
Today we experienced application failure caused by error:


Server Error in '/Complaints' Application.

ORA-01008: not all variables bound
ORA-02063: preceding line from APPDB

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: SyStem.Data.OleDb.OleDbEXception: ORA-01008: not all variables bound
ORA-020632 preceding line from APPDB

Source Error:

Line 615 : 'if (AppConfig. IsMTErrorHandlerUse("critica1"))
Line 616: Trace.TraceError(ea.0perationException.ToString());
Line 617: throw ea.OperationException;
Line 618: }
Line 619: }
Source File: c:\Inetpub\wwwroot\ComplaintS\App_Code\InMotion\Web\Controls\MTDataSourceView.cs Line: 617
Stack Trace:
...
...



It was very hard to find cause of this error because at first sight cause is not easily noticeable even after double checking application code and tracing.


I've searched Metalink and Google for some useful information and found out that ORA-1008 can occur when using cursor_sharing=force/similar.

So I decided to test everything with cursor_sharing=EXACT (it was FORCE before) and suddenly everything worked without any error.

Now I'am searching on Metalink more information related to this issue (maybe some bug note) but till now haven't found anything useful.


If I find something useful I will update this post with additional information.
You can also help me with leaving note/bug number in comments.

Thanks!

Read More...

Datafile Recovery After Deleting Datafile On OS

Thursday, October 8, 2009 2 comments
It is not so rare case that due to the user or hardware error you finish with missing critical database file.
Then you can thank God that you have fresh backup and needed skills to perform datafile recovery.

I'll perform datafile recovery in this small and simple demo case.

OS: Linux 32-bit
DB: Oracle 10.2.0.4
ARCHIVELOG MODE ON




First let's make database backup using RMAN interface:

$ rman target /
RMAN> backup database include current controlfile;

Starting backup at 08-OCT-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=367 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00003 name=/oradata/sysaux01.dbf
input datafile fno=00001 name=/oradata/system01.dbf
input datafile fno=00005 name=/oradata/undotbs02.dbf
input datafile fno=00004 name=/oradata/new_location/users01.dbf
channel ORA_DISK_1: starting piece 1 at 08-OCT-09
channel ORA_DISK_1: finished piece 1 at 08-OCT-09
piece handle=/home/oracle/rmanbkp/full_s9kr9vvd_1_1 tag=TAG20091008T165052 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:55
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current control file in backupset
channel ORA_DISK_1: starting piece 1 at 08-OCT-09
channel ORA_DISK_1: finished piece 1 at 08-OCT-09
piece handle=/home/oracle/rmanbkp/full_sakra030_1_1 tag=TAG20091008T165052 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 08-OCT-09



Create test table in USERS tablespace:
SQL> create table test_recovery (a number) tablespace users;

Table created.

SQL> insert into test_recovery values (5);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from test_recovery;

A
----------
5


SQL> set lines 200
SQL> col file_name for a40
SQL> col tablespace_name for a15
SQL> col status for a15
SQL> select tablespace_name, file_name, status
2  from dba_data_files;

TABLESPACE_NAME FILE_NAME                                STATUS
--------------- ---------------------------------------- ---------------
UNDOTBS2        /oradata/undotbs02.dbf                   AVAILABLE
USERS           /oradata/new_location/users01.dbf        AVAILABLE
SYSAUX          /oradata/sysaux01.dbf                    AVAILABLE
SYSTEM          /oradata/system01.dbf                    AVAILABLE




Now to delete datafile 'users01.dbf' while database is up and running.

SQL> !rm /oradata/new_location/users01.dbf

SQL> alter system flush buffer_cache;

System altered.

SQL> select * from test_recovery;
select * from test_recovery
*
ERROR at line 1:
ORA-00376: file 4 cannot be read at this time
ORA-01110: data file 4: '/oradata/new_location/users01.dbf'


I've received error due to the file missing.


Proceed with datafile recovery.

What will happen when we just execute 'restore datafile 4' from RMAN interface:

RMAN> restore datafile 4;

Starting restore at 08-OCT-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=379 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00004 to /oradata/new_location/users01.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/rmanbkp/full_s5kr9tuh_1_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 10/08/2009 16:24:31
ORA-19870: error reading backup piece /home/oracle/rmanbkp/full_s5kr9tuh_1_1
ORA-19573: cannot obtain exclusive enqueue for datafile 4


To avoid this error we must start database in mount mode, but what will happen when we simple execute 'shutdown immediate':

SQL> shutdown immediate;
ORA-01116: error in opening database file 4
ORA-01110: data file 4: '/oradata/new_location/users01.dbf'
ORA-27041: unable to open file
Linux Error: 2: No such file or directory
Additional information: 3


Oracle is looking for database file I deleted and it cannont find it. So I first must offline and drop datafile in Oracle to be able shutdown database.

SQL> alter database datafile '/oradata/new_location/users01.dbf'
2  offline drop;

Database altered.



Shutdown database and start in mount mode:

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1224736768 bytes
Fixed Size                  1267188 bytes
Variable Size             771754508 bytes
Database Buffers          436207616 bytes
Redo Buffers               15507456 bytes
Database mounted.




Restore and recover datafile from RMAN interface:
RMAN> restore datafile 4;

Starting restore at 08-OCT-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=379 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00004 to /oradata/new_location/users01.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/rmanbkp/full_s9kr9vvd_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/home/oracle/rmanbkp/full_s9kr9vvd_1_1 tag=TAG20091008T165052
channel ORA_DISK_1: restore complete, elapsed time: 00:00:45
Finished restore at 08-OCT-09

RMAN> recover datafile 4;

Starting recover at 08-OCT-09
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:04

Finished recover at 08-OCT-09


First check:
SQL> select * from test_recovery;
select * from test_recovery
*
ERROR at line 1:
ORA-00376: file 4 cannot be read at this time
ORA-01110: data file 4: '/oradata/new_location/users01.dbf'


Let's check state of the datafile:

SQL> set lines 200
SQL> col tablespace_name for a10
SQL> col file_name for a30
SQL> col online_status for a15
SQL> col status for a10
SQL> select tablespace_name, online_status, status
2  from dba_data_files where tablespace_name='USERS';

TABLESPACE ONLINE_STATUS   STATUS
---------- --------------- ----------
USERS      OFFLINE         AVAILABLE



Datafile in tablespace USERS is offline so I must bring datafile online.

SQL> alter tablespace users online;

Tablespace altered.

SQL> select tablespace_name, online_status, status
2  from dba_data_files where tablespace_name='USERS';

TABLESPACE ONLINE_STATUS   STATUS
---------- --------------- ----------
USERS      ONLINE          AVAILABLE



Final check:

SQL> select * from test_recovery;

A
----------
5

Read More...

SQL Tuning - using USE_HASH hint - dblink issue

Wednesday, August 26, 2009 2 comments
Colleague noticed that execution of his job lasted too long so he asked me to check out his query - maybe I could improve performance.

So I'll blog about my steps diagnosing and resolving this issue.
{ I've changed table names because they are not important for this example }

Oracle 9.2.0.6.0
Solaris 64



SQL> select count(*) from user1.table1;

COUNT(*)
----------
597259

SQL> select count(*) from user1.table2;

COUNT(*)
----------
2609503

SQL> select count(*) from user2.table3@rm_db;

COUNT(*)
----------
538512

SQL> select count(*) from user1.table4;

COUNT(*)
----------
93


First to check initial query and what was explain plan telling me.

SQL> explain plan for
2  SELECT
3       DISTINCT col1, 0, 0, TRUNC (SYSDATE - 1), t1.c_id
4       FROM user1.table1 t1,
5              user1.table2 t2,
6              user2.table3@rm_db rt
7       WHERE 1 = 1
8       AND t1.c_t = 'ABC'
9       AND t1.c_id = t2.con_id
10       AND t2.status = 'XY'
11       AND ph NOT IN (SELECT ph
12                                 FROM table4)
13       AND t1.c_nr = rt.s_cr
14       AND TRUNC (rt.fut) = TRUNC (SYSDATE - 1);

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------
| Id  | Operation                     |  Name  | Rows  | Bytes | Cost  | Inst   |IN-OUT|
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |        |     2 |   340 |  8535 |        |      |
|   1 |  SORT UNIQUE                  |        |     2 |   340 |  8523 |        |      |
|   2 |   FILTER                      |        |       |       |       |        |      |
|   3 |    TABLE ACCESS BY INDEX ROWID| TABLE2 |     1 |    12 |     8 |        |      |
|   4 |     NESTED LOOPS              |        |     2 |   340 |  8522 |        |      |
|   5 |      NESTED LOOPS             |        |     2 |   316 |  8507 |        |      |
|   6 |       TABLE ACCESS FULL       | TABLE1 |   157 |  6751 |  8013 |        |      |
|   7 |       REMOTE                  |        |     1 |   115 |     4 | DW_DB  | R->S |
|   8 |      INDEX RANGE SCAN         | IDX_C  |     6 |       |     3 |        |      |
|   9 |    TABLE ACCESS FULL          | TABLE4 |    20 |   540 |     6 |        |      |
----------------------------------------------------------------------------------------


I've executed query and turned tracing on.

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.03       0.12          0          3          1           0
Execute      1      0.00       0.01          0          0          0           0
Fetch        2     88.15     238.90      11396      20206          0         165
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4     88.18     239.04      11396      20209          1         165


Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait  Total Waited
----------------------------------------   Waited  ----------  ------------
SQL*Net message to client                       2        0.00          0.00
db file scattered read                       1502        0.50          2.47
SQL*Net message to dblink                  292103        0.00          0.46
SQL*Net message from dblink                292103        1.33        147.04
db file sequential read                      1384        0.44          1.41
latch free                                     98        0.05          0.23
SQL*Net message from client                     2       28.84         28.95
SQL*Net more data to client                     1        0.00          0.00
*****************************************************

Rows     Row Source Operation
-------  ---------------------------------------------------
165  SORT UNIQUE
166   FILTER
166    TABLE ACCESS BY INDEX ROWID TABLE2
1015     NESTED LOOPS
169      NESTED LOOPS
291934       TABLE ACCESS FULL TABLE1
169       REMOTE
845      INDEX RANGE SCAN IDX_C (object id 30818)
0    TABLE ACCESS FULL TABLE4


From this results I could see that bottleneck was network as biggest waits were related to dblink.
To tune that I decided to try USE_HASH hint and use hash join instead of nested loops on remote table.


SQL> explain plan for
2  SELECT /*+USE_HASH(rt t1) */
3       DISTINCT col1, 0, 0, TRUNC (SYSDATE - 1), t1.c_id
4       FROM user1.table1 t1,
5              user1.table2 t2,
6              user2.table3@rm_db rt
7       WHERE 1 = 1
8       AND t1.c_t = 'ABC'
9       AND t1.c_id = t2.con_id
10       AND t2.status = 'XY'
11       AND ph NOT IN (SELECT ph
12                                 FROM table4)
13       AND t1.c_nr = rt.s_cr
14       AND TRUNC (rt.fut) = TRUNC (SYSDATE - 1);

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------
| Id  | Operation                     |  Name  | Rows  | Bytes | Cost  | Inst   |IN-OUT|
--------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |        |     2 |   340 | 15222 |        |      |
|   1 |  SORT UNIQUE                  |        |     2 |   340 | 15210 |        |      |
|   2 |   FILTER                      |        |       |       |       |        |      |
|   3 |    TABLE ACCESS BY INDEX ROWID| TABLE2 |     1 |    12 |     8 |        |      |
|   4 |     NESTED LOOPS              |        |     2 |   340 | 15209 |        |      |
|   5 |      HASH JOIN                |        |     2 |   316 | 15194 |        |      |
|   6 |       TABLE ACCESS FULL       | TABLE1 |   157 |  6751 |  8013 |        |      |
|   7 |       REMOTE                  |        |  3307 |   371K|  7178 | RM_DB  | R->S |
|   8 |      INDEX RANGE SCAN         | IDX_C  |     6 |       |     3 |        |      |
|   9 |    TABLE ACCESS FULL          | TABLE4 |    20 |   540 |     6 |        |      |
----------------------------------------------------------------------------------------


Hm... this time explain plan shows double cost compare to query without USE_HASH hint.

Trace results:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.03       0.14          0          0          1           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2      2.66       4.76      18417      20195          0         165
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4      2.69       4.90      18417      20195          1         165


Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait  Total Waited
----------------------------------------   Waited  ----------  ------------
SQL*Net message to client                       2        0.00          0.00
db file scattered read                       1669        0.01          0.25
db file sequential read                       341        0.05          0.40
direct path write                             120        0.00          0.11
SQL*Net message to dblink                       7        0.00          0.00
SQL*Net message from dblink                     7        1.65          1.90
direct path read                              120        0.00          0.00
SQL*Net message from client                     2        0.32          0.32
SQL*Net more data to client                     1        0.00          0.00
***************************************************

Rows     Row Source Operation
-------  ---------------------------------------------------
165  SORT UNIQUE
166   FILTER
166    TABLE ACCESS BY INDEX ROWID TABLE2
1015     NESTED LOOPS
169      HASH JOIN
291938       TABLE ACCESS FULL TABLE1
281       REMOTE
845      INDEX RANGE SCAN IDX_C (object id 30818)
0    TABLE ACCESS FULL TABLE4



As you can see second query finished in 4,90 secs compare to first query which lasted 239.04 secs.
Using USE_HASH hint I resolved network bottleneck as much less data was being sent over the network.

My colleague was very pleased with query performance improvement :)

Read More...

Book Review: "Beginning Oracle Database 11G Administration, Iggy Fernandez

Sunday, August 23, 2009 0 comments
Beginning Oracle Database 11G Administration

First of all do not let yourself fooled by the title which says 11G. Knowledge that can be obtained by reading this book is applicable to nearly all current versions of Oracle databases.

The first thing I can say for this book is that this is an excellent book for any beginner Oracle database administrator.



In his chapters Iggy does not enter into too many details explaining various features but in addition to each chapter there is a list of books where you can find more informations about noted topics. Noted book advices for further reading are great for the beginners as they direct them to very useful books and save their time and money. In addition, Iggy often provides his own valuable tips gathered from personal work experience as Oracle database administrator.

Book is well organized so I recommend you read it from cover to cover, without jumping through chapters.

It is divided into four parts:
- Database Concepts
- Database Implementation
- Database Support
- Database Tuning

- Database Concepts
In this chapter Iggy it starts with explaining the most basic, but very important, things related to the database. He explains what is a database or what is actually a RDBMS (Relational Database Management System). Also gives you brief description of the internal mechanisms of Oracle RDBMS.

- Database Implementation
The main question of this chapter is - what is the right process of database implementation? It is important to understand that it is not enough just to follow the basic wizard steps or to execute "create database" command. It is important to use some time in planning physical and logical design as well as many other things that Iggy notes through the whole chapter.

- Database Support
It is very important to understand that primary obligation of a database administrator is to enable the database satisfies all business requirements. Importance of backup and recovery processes is clearly explained also. Iggy shares his own experiences, "Horror stories" as he called them, where you can get bigger picture why backup and recovery are essential for every database administrator.
Most important topic in this chapter is "The Big Picture and the Ten Deliverables" where Iggy states what is the role of database administrators in a business company. Emphasis is placed on the fact that it is very important to understand the operations of the company and to know what is DBA role in all this. In any case this is very interesting chapter even for experienced administrators.

- Database tuning
And finally, this is the least interesting chapter for me, but it still gives beginners a rough picture of how database tuning is performed. My advice for beginners is to execute examples from this chapter on their test environments. The best way to learn is by doing it.


For experienced administrators this book won't be much useful on technical side, but from the business side will provide wealth of useful information. It is well-described importance of documentation and checklists to reduce stress at work and expensive mistakes.

And for the beginners this book may be the first book to read along with Oracle documentation.

Rating 9 / 10

Read More...

How to install Oracle Client 11g on Windows 7?

Saturday, August 15, 2009 10 comments
I've never liked Windows Vista and I thought that Windows 7 is the same operating system just with new popular name. But lately I've read a lot good comments about Windows 7. To be honest Windows XP is still my favorite OS from Microsoft but I decided to try out Windows 7. So I downloaded and installed Windows 7 RC1.

First thing to try out after installing operating system is installation of Oracle Client because this application is essential for my work.

Shortly after initiating installation of Oracle Client 11g (11.1.0.6.0) installation stopped on prerequisite checks with error:

Checking operating system requirements ...
Expected result: One of 5.0,5.1,5.2,6.0
Actual Result: 6.1
Check complete. The overall result of this check is: Failed <<<< Problem: Oracle Database 11g is not certified on the current operating system. Recommendation: Make sure you are installing the software on the correct platform. ========================================================


To workaround this problem I've decided to edit refhost.xml file adding entry for Windows 7.

Location of this file on my system:
c:\unpacked_client_installation\win32_11gR1_client\client\stage\prereq\client\refhost.xml

This is excerpt from my newly edited refhost.xml file:


<CERTIFIED_SYSTEMS>
<OPERATING_SYSTEM>
<!--Microsoft Windows 2000-->
<VERSION VALUE="5.0"/>
<SERVICE_PACK VALUE="1"/>
</OPERATING_SYSTEM>
<OPERATING_SYSTEM>
<!--Microsoft Windows XP-->
<VERSION VALUE="5.1"/>
<SERVICE_PACK VALUE="1"/>
</OPERATING_SYSTEM>
<OPERATING_SYSTEM>
<!--Microsoft Windows 2003-->
<VERSION VALUE="5.2"/>
</OPERATING_SYSTEM>
<!--Microsoft Windows Vista-->
<OPERATING_SYSTEM>
<VERSION VALUE="6.0"/>
</OPERATING_SYSTEM>
<!--Microsoft Windows 7-->
<OPERATING_SYSTEM>
<VERSION VALUE="6.1"/>
</OPERATING_SYSTEM>

</CERTIFIED_SYSTEMS>

New lines are added at the end of this excerpt after <!--Microsoft Windows 7-->. Notice version value=6.1.

With new refhost.xml file all checks passed and I was ready to try out is everything OK after installation.

All tools that I am using in my daily work functioned perfectly.

There are maybe some features that will not function properly on Windows 7 but I won't bother with that features as I don't use them.



O yes, and word or two about Windows 7.

My first impressions are good. It is much better operating system then Windows Vista. It is very stable and fast which is great for beta operating system.
Try it out on test machine or in virtual machine - it won't cost you anything ;)

Read More...

How to edit CRONTAB file on Solaris

Wednesday, July 22, 2009 1 comments
I rarely edit crontab files on our Solaris machines but then when I need to add or edit something I always experience the same problem and always searching for an answer 5-10 minutes. This time I will write solution in blog post for quick reminder.

Problem is, when I execute "crontab -e" to edit current crontab file I don't get editor window.
I get something like this:

bash-2.05$ crontab -e
135




My problem is that I don't have EDITOR environment variable set.

I have to do something like this:

bash-2.05$ export EDITOR=vi
bash-2.05$ crontab -e

#Then I will receive proper output to edit crontab file
30 21 * * * /opt/oracle/product/9.2.0/local/script1
30 22 * * * /opt/oracle/product/9.2.0/local/script2



To avoid this problem happening any more I will add EDITOR environment variable in my ".profile" file:

bash-2.05$ cd ~
bash-2.05$ vi .profile


#!/bin/ksh
set -am -o vi
export ORACLE_BASE=/opt/oracle
export ORACLE_HOME=/opt/oracle/product/9.2.0
export PATH=$PATH:$ORACLE_HOME/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export EDITOR=vi
export ORACLE_SID=db
exec bash


Now, every time I login on this machine environment variable for EDITOR will be automatically set and I won't have problems with "crontab -e" any more.

Read More...

How to avoid ORA-01652 after executing "alter table compress..."

Thursday, July 16, 2009 0 comments
Strange thing happened to me few months ago on one of our biggest databases.
DB Oracle 10.2.0.2.0 on Solaris 9 (64-bit).

This database contains several range partitioned tables where date column is partitioning key. During my regular maintenance tasks I usually compress old partitions to save up disk space and speed up querying.

For some time everything was working smoothly without any ORA errors or such problems.
But several months ago I experienced strange Oracle behavior. I was executing the same commands as I did for a quite time but now my compress operation suddenly failed.



I've checked alert log to see what happened and there I found:

Errors in file /opt/oracle/admin/db/bdump/db_p000_18580.trc:
ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH
Mon Jul 13 14:55:38 2009
Errors in file /opt/oracle/admin/db/bdump/db_p000_18580.trc:
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH
Mon Jul 13 14:55:38 2009
Errors in file /opt/oracle/admin/db/bdump/db_p000_18580.trc:
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH
Mon Jul 13 14:55:38 2009
Errors in file /opt/oracle/admin/db/bdump/db_p000_18580.trc:
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH
Mon Jul 13 14:55:45 2009
Errors in file /opt/oracle/admin/db/bdump/db_p002_18584.trc:
ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH
Mon Jul 13 14:55:46 2009
Errors in file /opt/oracle/admin/db/bdump/db_p002_18584.trc:
ORA-07445: exception encountered: core dump [kghssgdmp()+176] [SIGFPE] [Integer divide by zero] [0x10401C670] [] []
ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH
Mon Jul 13 14:55:46 2009
...
[CUT]
...


"ORA-01652: unable to extend temp segment by 16384 in tablespace DATA_2007_10_ARCH" was the problem.

Tablespace was created with this command:
CREATE TABLESPACE DATA_2007_10_ARCH DATAFILE '/data2/data_2007_10_01_arch.dbf' SIZE 8193M
AUTOEXTEND ON NEXT 128M MAXSIZE 30721M LOGGING ONLINE PERMANENT
EXTENT MANAGEMENT LOCAL UNIFORM SIZE 128M BLOCKSIZE 8K
SEGMENT SPACE MANAGEMENT MANUAL FLASHBACK ON;


Expected size of tablespace DATA_2007_10_ARCH was about 13 GB and autoextend option was set to 30 GB so I thought there should be enough space for temporary segments.
As I said before SAME command worked before. Same sizes, same options, same database version, same parameters and machine - but now I receive error?!

This both commands failed:
-- with parallel option
ALTER TABLE user.data PARTITION DT_2007_10 TABLESPACE 
DATA_2007_10_ARCH COMPRESS PARALLEL 4;

-- without parallel option
ALTER TABLE user.data PARTITION DT_2007_10 TABLESPACE 
DATA_2007_10_ARCH COMPRESS;


As I was receiving errors constantly I created SR and asked support for help. After exchanging several notes with Oracle analyst the answer was that this is bug and I have to apply patch:

Later I asked:
Q: If this is known bug for 10.2.0.2.0 version, why everything worked before few months?
A: not all bugs show themselves right away -- in many cases we see that a given bug first manifests after long time (or never). What triggers a given bug is that we traverse down the Oracle code path where the bug is; we may not have used the buggy part of the code layer until now, but e.g. changes in parameters, increased size of DB, statistics etc. will cause that we begin to use the parts of the code where the bug then 'lurks' ...


So the only solution was to patch this frequent database which wasn't so easy task because it requires quite preparation.


In the meantime one thing came on my mind.

Even if my tablespace was extensible to 30GB Oracle fails when it is needed to extend datafile during compress operation.

So I created tablespace again but this time with initial size of 15GB instead 8 GB to avoid extending datafile.
CREATE TABLESPACE DATA_2007_10_ARCH DATAFILE '/data2/data_2007_10_01_arch.dbf' SIZE 15361M
AUTOEXTEND ON NEXT 128M MAXSIZE 30721M LOGGING ONLINE PERMANENT
EXTENT MANAGEMENT LOCAL UNIFORM SIZE 128M BLOCKSIZE 8K
SEGMENT SPACE MANAGEMENT MANUAL FLASHBACK ON;



And guess what - my compress operation completed successfully.

I don't know why Oracle fails when it is supposed to grab more disk space to extend datafile, but I am happy that problem is solved without applying patch and now I can continue with regular maintenance.

Read More...

How to create Database Link without editing tnsnames.ora file

Tuesday, July 14, 2009 1 comments
A month ago one of our developers needed access to some objects from one schema to another using database link. To enable database link he tried to create entry in tnsnames.ora file but had a problem with insufficient permissions. As a developer he has limited privileges on Unix machines so he can't edit and save tnsnames.ora file.

But there is solution for this little problem.
You can create functional database link without editing tnsnames.ora file.



Little demo case:

system@TEST11> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
PL/SQL Release 11.1.0.7.0 - Production
CORE    11.1.0.7.0      Production
TNS for Linux: Version 11.1.0.7.0 - Production
NLSRTL Version 11.1.0.7.0 - Production

5 rows selected.


system@TEST11> select * from dba_db_links;

1. no rows selected



Create database link testlink_db2 using full tns entry:
system@TEST11> create database link testlink_db2
2  connect to system identified by oracle
3  using
4  '(DESCRIPTION=
5    (ADDRESS=
6     (PROTOCOL=TCP)
7     (HOST=10.2.10.18)
8     (PORT=1525))
9    (CONNECT_DATA=
10     (SID=um)))'
11  /

Database link created.


Now little check and cleanout:
system@TEST11> select * from v$version@testlink_db2;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod
PL/SQL Release 10.2.0.4.0 - Production
CORE    10.2.0.4.0      Production
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production

5 rows selected.

-- cleanout
system@TEST11> drop database link testlink_db2;

Database link dropped.


In this test case I've used system user but this also works with any user.



From documentation:
http://download.oracle.com/docs/html/B13951_01/net.htm#i1153728

server_name = (DESCRIPTION=
(ADDRESS=
(PROTOCOL=TCP)
(PORT=port_number)
(HOST=host_name)
)
(CONNECT_DATA=(SERVICE_NAME=service_name)
)
)

where:

server_name is the name of an Oracle server that matches an entry in the RDB directory. An entry in the RDB directory can be added using the ADDRDBDIRE command.

TCP is the TCP protocol used for TCP/IP connections.

port_number is the port number of the Oracle Net listener. This is usually port number 1521.

host_name is the name that defines the system where the target Oracle server resides. This name must be in the local host definition on the AS/400 or in a name server on your network. The host name can also be entered as an IP address, for example, 161.14.10.12.

service_name is the service name of the Oracle server.

Read More...