Failed to start a managed process after using own SSLWallet

After creating a new Oracle Wallet for my apache instance i ran into the error that the opmnctl service would not start when i changed the ssl.conf configuration to pickup the new wallet.

#SSLWallet file:C:\oracle\product\10.1.3\apache\ohs\conf\ssl.wlt\default
SSLWallet file:C:\oracle\WALLETS

Instead i got :

C:\oracle\product\10.1.3\apache\opmn\bin>opmnctl startall
opmnctl: starting opmn and all managed processes…
================================================================================
opmn id=IAS-SRV:6200
0 of 1 processes started.

ias-instance id=IAS-1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
——————————————————————————–
ias-component/process-type/process-set:
HTTP_Server/HTTP_Server/HTTP_Server/

Error
–> Process (index=1,uid=1246497958,pid=2160)
failed to start a managed process after the maximum retry limit
Log:
C:\oracle\product\10.1.3\apache\opmn\logs\\HTTP_Server~1.log

So stop the , if any , processes again.

C:\oracle\product\10.1.3\apache\opmn\bin>opmnctl stopall
opmnctl: stopping opmn and all managed processes…

To solve this , you should check the auto-logon in your wallet to operate under these conditions.

After this start you opmn processes.

C:\oracle\product\10.1.3\apache\opmn\bin>opmnctl startall
opmnctl: starting opmn and all managed processes…

C:\oracle\product\10.1.3\apache\opmn\bin>

Exception in thread “main” java.lang.IndexOutOfBoundsException: No group 9

Exception in thread “main” java.lang.IndexOutOfBoundsException: No group 9

While installing Oracle Webcenter Suite 10.1.3.2.0 i ran into the following error.

Command: /u00/oracle/product/10.1.3/web/jdk/bin/java -jar /u00/oracle/product/10.1.3/web/j2ee/home/admin_client.jar deployer:
oc4j:opmn://xxx.helvoort.nl:6004/OC4J_WebCenter oc4jadmin -script /u00/oracle/product/10.1.3/web/portal/conf/oc4jportal_server_xml.params
Exception in thread “main” java.lang.IndexOutOfBoundsException: No group 9
at java.util.regex.Matcher.group(Matcher.java:463)
at java.util.regex.Matcher.appendReplacement(Matcher.java:730)
at java.util.regex.Matcher.replaceAll(Matcher.java:806)
at java.lang.String.replaceAll(String.java:2000)
at oracle.webdb.config.PortletContainerConfigAssistant.replacePwd(Unknown Source)
at oracle.webdb.config.PortletContainerConfigAssistant.configServerXML(Unknown Source)
at oracle.webdb.config.PortletContainerConfigAssistant.main(Unknown Source)

Please use a password without “$” as this is not escaped during configuration and therefor causing the installation to fail during a replace action.

Install has detected that Oracle Application Server is already installed in this Oracle Home

When trying to install an application server 10.1.3.2.0 i ran into the following error.

The error is thrown based on an earlier attempt on installing an application server. However, the product was removed successfully and should therefor not throw this error. Even when a different ORACLE_HOME is chosen , the error returns.
This behavior is caused by a corrupted Oracle Inventory as a quick peek in the file inventory.xml located in the ContentXML directory under your current oraInventory reveals that there is no such ORACLE_HOME present at this time.

As a workaround i created a dummy file called “oraInst.loc” in the “/tmp” directory containing these lines :

inventory_loc=/tmp/oraInventory
inst_group=dba

After creating this file , start your installer using

./runInstaller -invPtrloc /tmp/oraInst.loc

Your installer will continue now.

Generating FlashChart interactive reports on Apex fails with Flash Security Error

After switching our Embedded Gateway to an Apache front-end using modplsql.
We are faced with an error while loading interactive reports using FlashChart.

As we are using a Reverse Proxy to route the traffic to our internal servers , this is actually the cause of our problem for interactive report to show successfully.

In order to overcome this behavior , we added the following lines to our $ORACLE_APACHE_HOME/ohs/modplsql/dads.conf file in between the “Location” definition:

PlsqlCGIEnvironmentList HTTP_HOST=xxx.yenlo.nl

PlsqlCGIEnvironmentList REQUEST_PROTOCOL=http

xxx.yenlo.nl being our public adres for outside and “http” being our protocol.

Restart your apache services by executing :

[apache] xxx:bin> ./opmnctl stopall
opmnctl: stopping opmn and all managed processes…
[apache] xxx:bin> ./opmnctl startall
opmnctl: starting opmn and all managed processes…

Now your FlashChart should show up successfull again !

Tuning your Oracle HTTP Server (Apache 2.0) to use multi-threading

By default the Apache 2.0 version of Oracle HTTP Server is installed without using the HTTP “worker” method. As this is a performance enhancer , you would like to change your Apache configuration to implement the use of it.
For example , your Apex application will benefit from this as the multi-threated method with mod_plsql is supporting data connection pooling. The old fashion prefork method however is not and there for is not preferable.

Default your Apache process list will look like this after installing the Oracle HTTP Server (Apache 2.0) (10.1.3.3.0);

[apache] xxx:conf> ps -ef | grep httpd
oracle 3775 3738 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 3779 3775 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 3781 3775 0 Oct31 ? 00:03:48 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 3794 3775 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 3801 3775 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 4042 3775 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 4050 3775 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 4327 3775 0 Oct31 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 16187 3775 0 Nov02 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 18433 3775 0 Nov01 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 22394 3775 0 Nov01 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 30316 3775 0 Nov02 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 30439 3775 0 Nov02 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd -d /u00/oracle/product/10.1.3/apache/ohs -DSSL

Here you can clearly see there is no worker process active, but instead the old fashion prefork processes are used.

The following can be done to implement the worker method which will add multi-threading to your httpd configuration.

open your opmn.xml ( located : $ORACLE_HOME/opmn/conf ) and find the following section :

    <ias-component id="HTTP_Server">
        <process-type id="HTTP_Server" module-id="OHS2">
          <module-data>
            <category id="start-parameters">
              <data id="start-mode" value="ssl-enabled"/>
            </category>
          </module-data>
          <process-set id="HTTP_Server" numprocs="1"/>
        </process-type>
      </ias-component>

Change it to ;

    <ias-component id="HTTP_Server">
        <process-type id="HTTP_Server" module-id="OHS2">
          <module-data>
            <category id="start-parameters">
              <data id="start-mode" value="ssl-enabled"/>
              <data id="mpm" value="worker"/>
            </category>
          </module-data>
          <process-set id="HTTP_Server" numprocs="1"/>
        </process-type>
      </ias-component>

After this just restart your opmn services

[apache] xxx:bin> ./opmnctl stopall
opmnctl: stopping opmn and all managed processes…
[apache] xxx:bin> ./opmnctl startall
opmnctl: starting opmn and all managed processes…

Now lets check if the worker method is implemented correctly;

[apache] xxx:bin> ps -ef | grep httpd
oracle 23556 23521 3 14:26 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd.worker -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 23560 23556 0 14:26 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd.worker -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 23562 23556 0 14:26 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd.worker -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 23564 23556 0 14:26 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd.worker -d /u00/oracle/product/10.1.3/apache/ohs -DSSL
oracle 23565 23556 0 14:26 ? 00:00:00 /u00/oracle/product/10.1.3/apache/ohs/bin/httpd.worker -d /u00/oracle/product/10.1.3/apache/ohs -DSSL

When you are getting back the httpd.worker process you are running the multithreaded (MPM) method of the Apache 2.0 HTTP Server.

error while loading shared libraries: libgdbm.so.2 or libdb-3.3.so

While installing the latest HTTP Server (Oracle HTTP Server (Apache 2.0) (10.1.3.3.0) ) from oracle to take advantage of the modplsql module that is provided.
The installation completed succesfully but still the httpd process could not start because of missing library dependencies.

When opening your $ORACLE_ACACHE_HOME/opmn/logs/HTTP_Server~1.log

you could find the following errors there :

——–
09/10/31 18:28:32 Start process
——–

/u00/oracle/product/10.1.3/apache/ohs/bin/apachectl startssl: execing httpd
/u00/oracle/product/10.1.3/apache/ohs/bin/httpd: error while loading shared libraries: libgdbm.so.2: cannot open shared object file: No such file or directory

——–
09/10/31 18:37:31 Start process
——–
/u00/oracle/product/10.1.3/apache/ohs/bin/apachectl startssl: execing httpd
/u00/oracle/product/10.1.3/apache/ohs/bin/httpd: error while loading shared libraries: libdb-3.3.so: cannot open shared object file: No such file or directory

In the same order you could fix this by creating symbolic links to the current shared library for each missing library.

( execute as root user )
cd /usr/lib

ln -s libgdbm.so.3.0.0 libgdbm.so.2

ln -s libdb-4.3.so libdb-3.3.so

These versions are found on “SUSE Linux Enterprise Server 10 SP2 (i586)” , your symbolic links can be different when the problem is faced on a different version operating system.
Please adjust accordingly.

( execute as root user )
cd /usr/lib

ln -s libgdbm.so.X.0.0 libgdbm.so.2

ln -s libdb-4.X.so libdb-3.3.so

java.security.cert.CertificateException: java.io.EOFException

tomcat@xxx $ keytool -import -trustcacerts -file cert_xxxx.pem -alias login.example.nl  -keystore ssl.keystore
Enter keystore password:  xxx
keytool error: java.security.cert.CertificateException: java.io.EOFException

After opening the pem file we saw a strange “enter” after the END CERTIFICATE.

—–END CERTIFICATE—–

~

After removing this obsolete “enter” line the import went well.

—–END CERTIFICATE—–
~

How to renew a Certificate in Oracle Wallet Manager

Open your Wallet Manager from your commandline using $ORACLE_HOME/bin/owm ( set your DISPLAY to the correct value )

Once started open your ewallet.p12 file which most likely can be found at the default location :

/etc/ORACLE/WALLETS/oracle

Enter your Wallet Password.

Select the Certificate which will expire.

Go to Operations -> Export Certificate Request.

Save this file to server in the default location

Send the Certificate request to your CA Authority.
When a new certificate is recieved continue by removing the existing user certificate ;

Select the Certificate which will expire and select OPERATIONS -> REMOVE USER CERTIFICATE

Confirm this with “Yes”

As showed below , your initail request is turning up again and is ready to receive a new certificate.

Select Operations – > Import User Certificate

Select your location

Now the result is again an up to date certificate in the Oracle Wallet.

Remember , for the certificatie to take effect it is required to restart Oracle depending services like Portal !

Prevent “java.lang.OutOfMemoryError: PermGen space failure” on Oracle Application Server 10.1.3 web deployments

Once in a while a re-deployment of a web-app with jdeveloper caused the OC4J server to hang completely. It usually goes right but once in a while the whole OC4J server hangs and a complete restart of opmn is needed.

Logfiles in $ORACLE_HOME/opmn/logs show the following ;

09/06/26 13:39:20 FINER: RmiTransport$DiscreteMessageIn. Receiving message with 22 bytes
09/06/26 13:39:20 FINER: RmiTransport$DiscreteMessageIn. Received 22 bytes of message
09/06/26 13:39:22 WARNING: DeployerRunnable.run java.lang.OutOfMemoryError: PermGen spaceoracle.oc4j.admin.internal.DeployerException:

java.lang.OutOfMemoryError: PermGen space
at oracle.oc4j.admin.internal.DeployerBase.execute(DeployerBase.java:139)
at oracle.oc4j.admin.jmx.server.mbeans.deploy.OC4JDeployerRunnable.doRun(OC4JDeployerRunnable.java:52)
at oracle.oc4j.admin.jmx.server.mbeans.deploy.DeployerRunnable.run(DeployerRunnable.java:81)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.OutOfMemoryError: PermGen space

This behaviour can be altered by adding an option to the Start-parameters in as-control.
By adding “-XX:MaxPermSize=256m” and restarting the OC4J container , the problem did not appear anymore when deploying intensively… By default the MaxPermSize size is set small and differs per version but willl cause the container to crash when used intensively in deployment actions.

In addition to this you should also increase the “Maximum heap size” and “Initial heap size”.

Export Oracle Wallet with unencrypted private key

Today i was asked to export the private key of an Oracle Wallet.
After some searching and trying is was able to export this by using openssl.
Oracle Wallet Manager normally only exports the context of the wallet in encrypted format.
This is an Oracle by design decision not to allow export of unencrypted private keys as of security reasons from within the Oracle Wallet Manager. If you however require to export the unencrypted private key and certificate you should use the following command :

openssl pkcs12 -in ewallet.p12 -passin pass: -out ewallet.txt -nodes