About Me

My photo
Bangalore, India
I am an Oracle Certified Professional working in SAP Labs. Everything written on my blog has been tested on my local environment, Please test before implementing or running in production. You can contact me at amit.rath0708@gmail.com.

Friday, January 13, 2017

Could not read EMD_URL from agent configuration files in 12c

Today after OS patching, got below error when trying to check the status of the agent :-

==>./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD_URL is null.
Could not read EMD_URL from agent configuration files

==>./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
EMD_URL is null.
Could not read EMD_URL from agent configuration files

Cause :-

Configuration files for Agent corrupted or accidentally deleted.

Solution :-

Check for the emd.properties file in EM_AGENT_HOME/agent_inst/sysman/config folder
or check for the targets.xml file in EM_AGENT_HOME/agent_inst/sysman/emd folder

In my scenario , my emd.properties got corrupted

==>ls -ltr emd*
-rw------- 1 oracle sigdba 7709 Jan 19  2016 emd.properties.tzbak
-rw-r----- 1 oracle sigdba 7709 Jan 19  2016 emd.properties.2016-01-19-21-51-32
-rw-r----- 1 oracle sigdba 7734 Jan 19  2016 emd.properties.2016_01_19_09_51_36
-rw-r----- 1 oracle sigdba 7861 Oct 29 01:59 emd.properties.bak
-rw-r----- 1 oracle sigdba    0 Dec 16 01:01 emd.properties
[Hostname][oracle][AMIT2]

We can see the size of file is zero , by default Oracle keeps a backup of emd.properties file in the same directory, restore the file from that backup, I did the same

-rw-r----- 1 oracle sigdba  7861 Oct 29 01:59 emd.properties
-rw-r----- 1 oracle sigdba   155 Oct 29 01:59 private.properties
-rw-r----- 1 oracle sigdba   265 Dec 16 01:01 autotune.properties
-rw-r----- 1 oracle sigdba     0 Dec 16 01:01 emd.properties_20170113

Now again tried for a restart of the agent , it worked

==>./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Agent is Not Running
[Hostname][oracle][amit2]
/opt/oracle/product/agent/core/12.1.0.4.0/bin$
==>./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting agent ..................... started.

If your emd.properties file is not corrupted but targets.xml is corrupted , then also Oracle keep a backup of that file in same folder , we can restore from that file.

I hope this article helped you.

Thanks
Amit Rath

Wednesday, January 11, 2017

MRP0: Background Media Recovery terminated with error 1111

Today while troubleshooting a DR database sync scenario issue, I got below error in alert log file :-

MRP0: Background Media Recovery terminated with error 1111
Errors in file /opt/oracle/diag/rdbms/amit/amit1/trace/amit1_pr00_1890.trc:
ORA-01111: name for data file 34 is unknown - rename to correct file
ORA-01110: data file 34: '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034'
ORA-01157: cannot identify/lock data file 34 - see DBWR trace file
ORA-01111: name for data file 34 is unknown - rename to correct file
ORA-01110: data file 34: '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034'

Cause :-

Above issue can occur in DR database in below scenarios :-

1. Space issue in DR database and a datafile is added in primary database.
2. Standby file management is set to manual
3. DB_create_file_dest or DB_file_name_convert parameter is not set in Standby database

Troubleshoot :-

When I crosscheck the primary database and standby database sync details , I saw Redo Apply was stopped.

I crosschecked the alert log files for standby database and got above mentioned error in alert logfile.

Checked the datafile details in DR database :-

File_id       name
34             /opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034

Checked the Primary alert log file and got to know that a datafile addition activity happened in primary database.

Crosschecked all the above mentioned parameters and got to know that space exhausted in BCP database due to that Oracle created an Unknown file in ORACLE_HOME location due to which Redo Apply stopped.

Solution :-

Checked the added file detail in Production database :-

File_id       name
34            +DATA/amit/datafile/data.321.931222251

Login to Standby database server :-

1. As space was exhausted on Diskgroup level, added space by logging in to ASM instance
2. Login to DR database instance :-

a. Change the standby_file_management parameter to Manual

SQL> show parameter standby

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_archive_dest                 string      ?/dbs/arch
standby_file_management              string      AUTO
SQL> alter system set standby_file_management=MANUAL sid='*';

SQL> show parameter standby

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_archive_dest                 string      ?/dbs/arch
standby_file_management              string      MANUAL
SQL>
SQL> select name from v$datafile where file#=34;

NAME
--------------------------------------------------------------------------------
/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034

SQL> alter database create datafile '/opt/oracle/product/database/11.2.0.4/dbs/UNNAMED00034' as '+DATA' size 10g;

Database altered.

After creation of the datafile in DR database , check the file detail in DR database :-

File_id       name
34             +DATA/amit_dr/datafile/data.278.293456881

Check the redo apply status, if not started , start it manually using dgmgrl.

I hope this article helped you.

Thanks
Amit Rath