Posted by John Paul van Helvoort on November 12, 2009
Oracle now provides us with a desktop tool to monitor our environment using the an Oracle Enterprise Grid Control.
Simply install the desktop widget en search your favorite targets to be listed. From here you have a straight link to the target in your monitoring tool in case of any problem.
Here are some examples from after the installation on my desktop.
Target Search & Monitoring
Login screen:
Monitoring targets
High-Load Databases
Load on all databases known to EMGC.
Please visit the Oracle site and try-out there new Tools !
http://www.oracle.com/technology/products/oem/widgets/index.html
Posted in Grid Control | Leave a Comment »
Posted by John Paul van Helvoort on May 19, 2009
After succesfully creating a standby database using RMAN duplicate on a primairy site 11.1.0.6 .
The logfle shipping works ok, but the dg_broker seems to have a problem which i couldn’t pin down.
The Dataguard broker was configured using Enterprise Grid Control using the add standby database option “Manage
existing standby database”. The configuration steps were preformed successfully, however this error is shown for the standby database.
The Data Guard status of stby is Error ORA-16664: unable to receive the result from a database.
Searching this error reveals that there might be a network error.
ORA-16664: unable to receive the result from a database
Cause: During execution of a command, a database in the Data Guard broker configuration failed to return a result.
Action: Check Data Guard broker logs for the details of the failure. Fix any possible network problems and try the command again.
As the RMAN duplicatie is done over TNS using the following command
“rman target sys/xxx@prod AUXILIARY sys/xxx@stby” . I would say that the network setting for TNS are
correct, so what does this error actually mean ?
In the background i could find serveral Dataguard logfiles showing me mixed errors on the subject.
Primary site DG log shows :
….
DG 2009-05-19-08:48:27 1012000 4 686031872 DMON: CTL_GET_STATUS operation completed
DG 2009-05-19-08:48:45 0 2 0 NSV1: Received error ORA-16664 from target remote site stby.
DG 2009-05-19-08:48:45 0 2 686031873 DMON: Database stby returned ORA-16664
…
Standby site DG log shows :
…
DG 2009-05-19-08:51:00 0 2 686031879 Operation CTL_GET_STATUS continuing with warning, status = ORA-16809
DG 2009-05-19-08:51:00 0 2 0 NSV0: Failed to connect to remote database prod. Error is ORA-00604
DG 2009-05-19-08:51:00 0 2 0 NSV0: Failed to send message to site prod.
Error code is ORA-00604.
DG 2009-05-19-08:51:00 0 2 0 DMON: Database prod returned ORA-00604
…
The Data Guard Command-Line Interface ( DGMGRL ) showed the following error.
DGMGRL> show configuration
Configuration
Name: prod
Enabled: YES
Protection Mode: MaxPerformance
Databases:
prod – Primary database
stby – Physical standby database
Fast-Start Failover: DISABLED
Current status for “prod”:
Warning: ORA-16607: one or more databases have failed
All these errors seem very surfear but at the end it all came down to a faulty “logon trigger” which was implemented by another Database Administrator.
As normal connections worked this was not the first thing you would come to , but at the end by dropping this faulty trigger the problem was solved !
Posted in Database, Grid Control | Leave a Comment »
Posted by John Paul van Helvoort on May 18, 2009
When switching to a Grid monitoring environment, standalone dbconsole need to be shutdown and cleared out. Jobs scheduled using the dbconsole on a 11g instance cannot be retreived threw Oracle Enterprise Grid Console.
When your dbconsole is broke and you don’t want to have any jobs running which where submitted through the gui. Here is a way to delete them from you repository manually.
select JOB_OWNER, JOB_NAME from SYSMAN.MGMT_JOB;
SYSMAN ALLOWED FAILED LOGIN ATTEMPTS (CLUSTER DATABASE)
SYSMAN ALLOWED FAILED LOGIN ATTEMPTS (DATABASE INSTANCE)
SYSMAN FIX INVENTORY
BCK XXX_BACKUP_FULL
BCK XXX_BACKUP_INCR
BCK XXX_EXPORT_FULL
SYSMAN PROVISIONING DAEMON
SYSMAN REFRESH_FROM_METALINK_JOB
For each job that you want to remove , run the following statement.
begin
sysman.mgmt_jobs.stop_all_executions('XXX_BACKUP_FULL','BCK');
sysman.mgmt_jobs.delete_job('XXX_BACKUP_FULL','BCK');
end;
/
commit;
Posted in Database, Grid Control | Leave a Comment »
Posted by John Paul van Helvoort on May 5, 2009
These easy steps should do the trick for you :
Logon the monitoring server and change ownership and rights under the root user
$ORACLE_HOME_OMS/Apache/Apache/bin
chown root .apachetcl
chmod 6750 .apachectl
result :
-rwsr-s— 1 root oinstall 1716966 2008-03-05 11:04 .apachectl
Log in under the oracle user and adjust the httpd.conf file located under $ORACLE_HOME_OMS/Apache/Apache/conf
Search for “Listen” or “Port”
# This port is used when starting without SSL
Port 7777
Listen 7778
to
# This port is used when starting without SSL
Port 80
Listen 7778
Listen 80
After this a restart of the HTTP_Server process is enough for changes to take effect :
oracle@xxx: oms10g/opmn/bin> ./opmnctl restartproc process-type=HTTP_Server
opmnctl: restarting opmn managed processes…
oracle@xxx: oms10g/opmn/bin>
You are all set !
Posted in Grid Control | Leave a Comment »
Posted by John Paul van Helvoort on May 2, 2009
When running you runInstaller under another user then the current user logged into the X server screen, you might hit this error.
I ran into this error while being logged on to a vnc session running under the root user. While perfoming tasks under root , switched user to oracle and set the display back to the current screen
su – oracle
export DISPLAY=xxx:1
The installer starts but exits with the following error;
Starting Oracle Universal Installer…
….
Exception in thread “main” java.lang.InternalError: Can’t connect to X11 window
server using ‘xxx:1′ as the value of the DISPLAY variable.
at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
at sun.awt.X11GraphicsEnvironment.(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(Unknown Source)
at java.awt.Window.init(Unknown Source)
at java.awt.Window.(Unknown Source)
at java.awt.Frame.(Unknown Source)
at java.awt.Frame.(Unknown Source)
at oracle.ewt.popup.PopupFrame.(Unknown Source)
at oracle.ewt.lwAWT.BufferedFrame.(Unknown Source)
at oracle.sysman.emgc.oneclick.OneClickWizard.getGuiDefaults(OneClickWizard.java:239)
at oracle.sysman.emgc.oneclick.OneClickWizard.(OneClickWizard.java:205)
at oracle.sysman.emgc.oneclick.OneClick.(OneClick.java:236)
at oracle.sysman.emgc.oneclick.OneClickInstaller.(OneClickInstaller.java:116)
at oracle.sysman.emgc.oneclick.OneClickInstaller.process(OneClickInstaller.java:268)
at oracle.sysman.emgc.oneclick.OneStartup.startup(OneStartup.java:383)
at oracle.sysman.emgc.oneclick.OneArgs.main(OneArgs.java:700)
at oracle.sysman.emgc.oneclick.OneStartup.main(OneStartup.java:391)
oracle@xxx:/u00/agent/linux/agent> ./runInstaller
A solution could be to disable access control under the user from which VNC is running , in my case under root.
xxx:/ # xhost +
access control disabled, clients can connect from any host
xxx:/ #
Now the runInstaller will start fine in this setup !
Posted in Application Server, Database, Grid Control, Linux | Leave a Comment »
Posted by John Paul van Helvoort on May 1, 2009
After a fresh upgrade of the Oracle Enterprise Manager Grid Control 10.2.0.1 to 10.2.0.5 , the error log filled up with “ORA-14400: Inserted partition key does not map to any partition” lines.
( http://host:port/em/console/health/healthSystemError )
The emos.trc file on the host reports :
2009-04-29 18:23:42,180 [XMLLoader0 20000000014.xml] ERROR eml.XMLLoader markFileAsError.909 – Failed to load 20000000014.xml even after 2 attempts; moving to errors directory. Error:
java.sql.SQLException: ORA-14400: inserted partition key does not map to any partition
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:315)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:281)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:638)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:184)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:632)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1160)
at oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement.java:3276)
at oracle.sysman.jdbc.wrapper.OraclePreparedStatementWrapper.sendBatch(OraclePreparedStatementWrapper.java:35)
at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.flushAllStatements(XMLLoaderContext.java:1389)
at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.endElement(XMLDocumentSplitter.java:977)
at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1257)
at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:314)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:281)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:195)
at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.split(XMLDocumentSplitter.java:477)
at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.loadFromStream(XMLLoaderContext.java:1897)
at oracle.sysman.emdrep.dbjava.loader.XMLLoader.LoadFile(XMLLoader.java:607)
at oracle.sysman.emdrep.dbjava.loader.XMLLoader.LoadFiles(XMLLoader.java:740)
at oracle.sysman.emdrep.dbjava.loader.XMLLoader.run(XMLLoader.java:1396)
at java.lang.Thread.run(Thread.java:534)
After some reached it seemed that some objects under the sysman schema where not valid and therefor cause the problem.
To solve this i executed the following command which fixed the problems .
oracle@xxx:/u00/oracle/product/10.2.0/oms10g/sysman/recv/errors> sqlplus “/as sysdba”
SQL*Plus: Release 10.1.0.4.0 – Production on Wed Apr 29 18:25:57 2009
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.1.0.4.0 – Production
With the Partitioning, OLAP and Data Mining options
SQL> conn sysman
Enter password:
Connected.
SQL> exec emd_maintenance.analyze_emd_schema(‘SYSMAN’);
PL/SQL procedure successfully completed.
SQL> exit
This process took atleast 1 hour here , so be patient !
Posted in Grid Control | 1 Comment »
Posted by John Paul van Helvoort on April 15, 2009
When installing a new Enterprise manager Grid Control in an existing 11g Database using the provided reponse file em_using_existing_db.rsp.
You are challenged with a lot of variables which need to match the OS perfectly.
Earlier i provided a solution for adjusting the oraparam.ini file to overcome compatibility problems for OS certification.
However , when using a response file method like here :
./runInstaller -noconfig -ignoreSysPrereqs -silent -responseFile response/em_using_existing_db.rsp use_prereq_checker=false
The silent installer uses the the /etc/SuSE-release file to pass OS information to a file called /usr/bin/lsb_release.
When this SuSE-release file still contains the 10 release version number, your silent install will stop and fail.
In the silent install log ( found under your oraInventory/logs/* ) you might find a very global error warning stating ;
UI-62009:
Some requirement checks failed. You must fulfill these requirements before
continuing with the installation, at which time they will be rechecked.
For solving this we need to backup the /etc/SuSE-release file and adjust the parameter to fake a 9 version OS.
SUSE Linux Enterprise Server 9 (x86_64)
VERSION = 9
PATCHLEVEL = 2
After a restart of the installation , your silent install will pass this test and continue the journey.
Hint : Also be sure to unset ENV as this is the next problem to face when set :)
SEVERE:Install has detected that the environment variable ENV has been set. Please unset this variable and restart the install.
Posted in Database, Grid Control | Leave a Comment »
Posted by John Paul van Helvoort on March 30, 2009
When installing the new Enterprise Manager Grid Control (Agents) version 10.2.0.5. You might find yourself searching why agents automatically start on ever reboot. First you take out the oracle script under /etc/init.d or its symbolic links under rc?.d. But still it will startup automatically. Some research shows that the script “gcstartup” is called upon startup while running in runlevel 3 and 5. This script was already there in previous releases but never used automatically by refering to it from the runlevel rc3.d or rc5.d.
In 10.2.0.5 this is corrected by creating softlinks to the gcstartup script upon installing the agent while running root.sh.
If you take out these (symbolic) softlinks you are again in control of when and how the agents startup.
Background information on it
BUG 7414737
ROOT.SH SHOULD SOFTLINK RC3.D AND RC5.D FILES FOR AUTOSTART
Many other bugs are solved, here is the link.
Posted in Grid Control | 3 Comments »
Posted by John Paul van Helvoort on March 11, 2009
By default the monitoring facillity by oracle does not provide us with a threshold to monitor the synchronization profiles created in Oracle Identity Management.
Several enhancement requests are logged on metalink but still no solution is provided. Know that this option requires the installation of the plugin for identity management on both OMS and Agent ( info )
Here is a quick hack to implement such a feature in your monitoring tool so that a email is sent whenever the status of the syncing profile returns “Agent Execution Successful, Mapping/IMPORT operation Failure”.
We need to change the following files and update the changes so that the agent is now aware of these new monitoring rules.
First we alter this file $AGENT_HOME/sysman/admin/default_collection/oracle_eps_server.xml and add the following to the file :
...
<CollectionItem NAME="ODIIntegrationProfileStatus">
<Schedule>
<IntervalSchedule INTERVAL="2" TIME_UNIT="Min"/>
</Schedule>
<Condition COLUMN_NAME="Status"
CRITICAL="Agent Execution Successful, Mapping/IMPORT operation Failure"
WARNING="N/A"
OPERATOR="EQ"
MESSAGE="The DIP Sync Profile is in broken state"
CLEAR_MESSAGE="The DIP Sync Profile is up"
CLEAR_MESSAGE_NLSID="eps_server_dip_broken_cond_clear"
MESSAGE_NLSID="eps_server_dip_broken_cond"/>
<Condition COLUMN_NAME="ProfileErrors"
CRITICAL="Error"
WARNING="N/A"
OPERATOR="EQ"
MESSAGE="The DIP Sync Profile is in broken state"
CLEAR_MESSAGE="The DIP Sync Profile is up"
CLEAR_MESSAGE_NLSID="eps_server_dip_broken_cond_clear"
MESSAGE_NLSID="eps_server_dip_broken_cond"/>
</CollectionItem>
...
Now update the $AGENT_HOME/sysman/admin/metadata/oracle_eps_server.xml file and increase the release nummer so that the metarepository identifies this change.
<TargetMetadata META_VER="6.7" TYPE="oracle_eps_server" CATEGORY_PROPERTIES="VersionCategory"
RESOURCE_BUNDLE_PACKAGE="oracle.oimcontrol.eps.rsc">
to
<TargetMetadata META_VER="6.8" TYPE="oracle_eps_server" CATEGORY_PROPERTIES="VersionCategory"
RESOURCE_BUNDLE_PACKAGE="oracle.oimcontrol.eps.rsc">
After these changes we need to let the agent know we made the changed by reloading the settings.
$ORACLE_HOME_AGENT\bin\emctl reload
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully
After this you can define the Metric Threshold on the monitoring page of the Directory Integration Platform.
Posted in Grid Control, Identity Manager | Leave a Comment »
Posted by John Paul van Helvoort on March 10, 2009
In February Oracle released there new patchset 10.2.0.5 for the Oracle Enterprise Grid Control server together with a new version Intelligent Agent.
This new patch is now available for linux x86 and Windows 32 bits now.
This release now supports Management Support for Oracle Database 11g Release 1 ! And tons of other new features which can be found
here
You can download this new release here ;
Oracle Enterprise Manager 10g Grid Control Release 5 (10.2.0.5)
Posted in Grid Control | Leave a Comment »
