2011 in review

The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.

Here’s an excerpt:

The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 46,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 17 sold-out performances for that many people to see it.

Click here to see the complete report.

Disabling ORACLE Reports to plug SQL Injection Attacks , Don’t do this if you still need Oracle Reports to WORK!

Posting this because I had a hard time disabling all of the reports functionality in Fusion Middleware Server…this will also work for older versions of Oracle Application Server.

Just wanted to alert people to the fact that you may have a major security hole with Oracle Reports Server.

We don’t use it at our site and it is my understanding that it is subject to SQL injection attacks.

http://www.red-database-security.com/wp/sql_injection_reports_us.pdf

First off I would check that Oracle Reports is not available outside your firewall or VPN access.

Once I was made aware of the possible security issue, the next step was figuring out how to disable it.

What I found was this MOS article on disabling the help menu.

How to Disable the Oracle Reports Servlet HELP Command URL? [ID 465454.1]

So…I did it quick and dirty by modifying the httpd.conf adding the code below for all of my application servers (FMW, OAS 10g, etc) and restarting all of the services. It doesn’t seem to take effect if you only restart OHS. Only thing I did different than what the article said was take out the word help so it disables (by not allowing access) to EVERYTHING reports/rwservlet.

<Location /reports/rwservlet/*>
Order deny,allow
Deny from all
</Location>

NOTE: THIS WILL DISABLE ORACLE REPORTS COMPLETELY. Don’t do this if you still want Oracle Reports functionality. Contact your Oracle support team for their best practice on how to make it secure.

See:  Oracle Doc ID 856135.1 How to Deregister Standalone Reports Server 11g From OPMN And Oracle

How to Install FMW 11.1.1.4.0 Standalone Forms/Reports, AKA How to Get to Never Never Land

Hmmm….seems Fusion MW is Peter Panish – a child that doesn’t want to grow up.

Our team is currently using in production the 11.1.1.2.0 FMW (RH 5) install using the Standalone Forms and Reports 32-bit version. Having encountered memory leaks with this version we are now attempting to migrate to 11.1.1.4 64-bit (RH 5).

The documentation to upgrade from 11.1.1.2 to 11.1.1.4 is full of links that take you from one place to another – I am flying over these docs looking for landmarks to get me to where I want to go – Never Never Land. I expect to have lots of fun with little to no consequences when I get there. But alas the following document is giving me fits! Those nasty pirates have been here first.

11.1.1.5.0 seems to be the only ending point (at this writing it doesn’t look to have a standalone Forms Reports install available for 11.1.1.5.0)…what if you need to stop at a different version along the way. AAGH!

http://download.oracle.com/docs/cd/E23104_01/download_readme_ps3/download_readme_ps3.htm#BA BDBHCJ

Following the above documentation, I assume the way to get to Never Never Land is to start with WebLogic 10.1.3.2, install FMW 11.1.1.2.0, verify installation, upgrade Weblogic to 10.1.3.4, install 11.1.1.4.0 patchset. But these steps lead me elsewhere as shown by the following error on starting the WebLogic Node Manager after four successful (no errors reported) product installations.

WARNING: Uncaught exception in server handlerjavax.net.ssl.SSLHandshakeException: FATAL Alert:HANDSHAKE_FAILURE – The handshake handler was unable to negotiate an acceptable set of security parameters. javax.net.ssl.SSLHandshakeException: FATAL Alert:HANDSHAKE_FAILURE – The handshake handler was unable to negotiate an acceptable set of security parameters. at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source) at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertSent(Unknown Source) at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source) at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source) at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessage(Unknown Source) at com.certicom.tls.record.handshake.HandshakeHandler.handleVersion2HandshakeMessages(Unknown Source) at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source) at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source) at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source) at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source) Time to talk to the

Go to the Fairies that live in Pixie Hollow to ask for help (AKA entering a Service Request on MOS).

Tinkerbell comes back with a revised install for the 64-bit Version of 11.1.1.4.0 Standalone Forms/Reports:

Please download the following from http://edelivery.oracle.com

• Oracle WebLogic Server 11gR1 (10.3.4)
• Oracle Portal, Forms, Reports and Discoverer 11g (11.1.1.2.0) (4 parts)
• Oracle Portal, Forms, Reports and Discoverer 11g Patch Set 3 (11.1.1.4.0)

Order of installations

• 1. Install 64 bit JDK – jdk-6u21-linux-x64-rpm.bin (at this writing)
• 2. Install WLS 10.3.4
• 3. Install but do not configure FMW 11.1.1.2
• 4. Install FMW 11.1.1.4
• 5. execute config.sh from the OracleHome/bin folder

I ask Tinkerbell why is her list different from the documentation from MOS Note: How to Install Fusion Middleware 11g Forms and Reports Only (Note:854117.1).

She brightly sighs, shrugs her shoulders and winks at me! ================================

JDK Install

================================

Since we are using a 64-bit OS, we need the Sun JDK 6 64-bit.

The jdk-6u21-linux-x64-rpm.bin can be downloaded from http://java.sun.com/javase/downloads/widget/jdk6.jsp

Verify the correct version of java is installed.

which java

/usr/java/jdk1.6.0_21/bin/java java -version

In my environment there was a symbolic link for an older version of java 1.4 in /usr/bin/java which I removed. This may not be correct for your system, talk to your system administrator about the best way to install a JDK version for system-wide use.

Also you may run out of OS-level file descriptors if left unset. On Linux I edited /etc/security/limits.conf adding the following entries:

* soft nofile 4096

* hard nofile 4096 # End of file

Again, see your System Administrator especially if you want limit the number of open files by user. The example above allows any logon to have those limits.

================================

Install WebLogic 10.1.3.4

================================

This is a typical installation, change the directory if desired.

Install WebLogic. Unzip the file and execute

java -Xmx1024m -jar wls1034_generic.jar

Welcome Screen

Choose Middleware Home Directory

Create a new Middleware Home – /aux/oracle/middleware

Register for Security Updates

Hit Next to bypass (this is not a mandatory step)

Choose Install Type

Select Typical

JDK Selection

Select the 1.6.0_21 version (probably already selected)

Choose Product Installation Directories

WebLogic Server – /aux/oracle/middleware/wlserver_10.3

Installation Summary

================================

Install Forms Reports 11.1.1.2

================================

The important thing is to install the software only at this time.

Install Forms Reports. Unzip the files and execute

cd Disk1

./runInstaller

Specify Inventory Directory

screen Directory: /aux/oracle/oraInventory

Group name: dba (If prompted, run the createCentralInventory.sh as directed.)

Welcome Screen

Select Installation Type Select Install ONLY!..….you will configure using config.sh script later. Doing this method also installs all of the components but the configuration step allows you to choose which ones to configure.

Prerequisite Checks

If you receive an error like Checking for openmotif-2.2.3; not found; failed, Install the missing rpm

Specify Security Updates

================================

Upgrade Forms Reports 11.1.1.4

================================

Install Forms Reports patchset. Unzip the appropriate file and execute

cd Disk1 ./runInstaller

================================

Configure Forms Reports 11.1.1.4

================================

Execute config.sh from the $Oracle_Home/bin folder. This is a configuration tool that can be run as a GUI or command-line.

Other documents that were helpful to our team:

• Upgrading Oracle Middleware 11g; How to Check that the Core Components are Running Successfully? [ID 1086348.1]
• Location Of Different Forms Configuration Files in Fusion Middleware Forms and Reports 11.1.1.1, 11.1.1.2 and 11.1.1.3 Installations [ID 854124.1]
• Maintain FMW 1073776.1

Disable IPV6 for WebCache

We disabled IPV6 because webcache wouldn’t start….see the instructions below.

14.5.3 Disabling IPv6 Support for Oracle Web Cache

By default, IPv6 support is enabled for Oracle Web Cache. You can disable it in the

webcache.xml file, which is located in the following directory:

(UNIX) ORACLE_INSTANCE/config/WebCache/webcache_name
(Windows) ORACLE_INSTANCE\config\WebCache\webcache_name

In the file, change the value of the IPV6 element to “NO”. For example:

<IPV6 ENABLED=”NO”/>

change LD_LIBRARY

If the IPV6 element does not exist in the webcache.xml file, you can add the element to the

file. Add it after the MULTIPORT element, as shown in the following example:

…
<LISTEN IPADDR=”ANY” PORT=”7786″ PORTTYPE=”ADMINISTRATION”/>
<LISTEN IPADDR=”ANY” PORT=”7788″ PORTTYPE=”INVALIDATION”/>
<LISTEN IPADDR=”ANY” PORT=”7787″ PORTTYPE=”STATISTICS”/>
</MULTIPORT>
<IPV6 ENABLED=”NO”/>

100% CPU Utilization, Corrupted Password Files on FMW 11.1.1.2

We are experiencing memory leaks and high CPU utilization for Fusion Middleware (FMW) Version 11.1.1.2  that gradually builds until the server becomes completely unresponsive. The last time this happened rebooting the server did not bring up all of the FMW components.

Errors in the Browser trying to access Oracle Forms:

Failure of server APACHE bridge:
No backend server available for connection: timed out after 10 seconds or idempotent set to OFF.

Errors in the AdminServer.log for Weblogic:

####<May 3, 2011 12:26:38 PM MDT> <Warning> <DeploymentService> <AdminServer> <[ACTIVE] ExecuteThread: ’0′ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<anonymous>> <> <> <BEA-290014> <Invalid user name or password.>

####<May 3, 2011 12:26:38 PM MDT> <Warning> <DeploymentService> <> <AdminServer> <[ACTIVE] ExecuteThread: ’0′ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<anonymous>> <> <> <> <BEA-290014> <Invalid user name or password.>

####<May 3, 2011 12:26:42 PM MDT> <Error> <Configuration Management> <> <AdminServer> <[ACTIVE] ExecuteThread: ’0′ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <> <> <BEA-150035> <An attempt was made to download the configuration for the server WLS_REPORTS by the user with an invalid password.>

####<May 3, 2011 12:26:44 PM MDT> <Error> <Configuration Management> <> <AdminServer> <[ACTIVE] ExecuteThread: ’0′ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<WLS Kernel>> <> <> <> <BEA-150035> <An attempt was made to download the configuration for the server WLS_FORMS by the user with an invalid password.>

####<Apr 1, 2011 10:58:46 AM MDT> <Notice> <Security> <> <AdminServer> <[ACTIVE] ExecuteThread: ’4′ for queue: ‘weblogic.kernel.Default (self-tuning)’> <<anonymous>> <> <> <> <BEA-090078> <User  in security realm myrealm has had 5 invalid login attempts, locking account for 30 minutes.>

Searching on MOS came up with two different articles:

Getting Error “User orcladmin In Security Realm Myrealm Has Had 5 Invalid Login Attempts, Locking Account For 30 Minutes.” [ID 1270253.1]

Oracle Middleware 11g – Troubleshooting the Error “Failure of server APACHE bridge” [ID 1304095.1]

Neither MOS document helped as the user was blank in the logs, what user? Notice the space after user in this section: “User  in security realm myrealm”. This is a standalone Forms/Reports install of FMW so there are very few users.

I changed the password for the WEBLOGIC admin account using the GUI Console which really didn’t help. Aha! We had created boot.properties files for starting all services, somehow those were no longer working (corrupt?) I recreated them and restarted all of the services successfully.

For a more permanent fix, we are upgrading to FMW 11.1.1.4 on 64bit RH 5 in a couple of weeks……I don’t recommend using 32-bit OS for FMW.

Funny Speaker Bio ‘s

Quotes taken from speaker biographies….some of the worse or best. You be the judge. I will add to these….stay tuned.

“He works from the command line and uses vi.”

” He enjoys the latest gadgets and technology.”

” He has also spent many years developing C/C++ compilers and knows more programming languages than he has fingers. “

“____ is now ready to change the world.”

“____ was a particularly driven young man. Having been voted “Most Likely to Succeed” by his high-school graduating class, he promptly moved 1000km away from home to attend the University”

“He is a focal point around the world in ….. circles” (editor note: Think visually on this one)

” Any type of major incident comes to this team.” (editor note: Another visual)

“…..hopes to, one day, complete her graduate degree in Information Systems.”

‘Warning: Subscription For Node Down Event Still Pending’ Can Cause Hanging Listener

Receiving errors for large batch jobs in a production 11.2.0.2 database.

ERROR:
ORA-12541: TNS:no listener

The listener log contained the following event at the same time-  ‘Warning: Subscription For Node Down Event Still Pending’ . I have seen this event before and it hasn’t caused any issues but apparently there are reports of hanging issues in databases >10g.

See the following MOS document :’Warning: Subscription For Node Down Event Still Pending’ In Listener Log [ID 372959.1]

“These messages are related to the Oracle TNS Listener’s default subscription to the Oracle Notification Service (ONS). In a non-RAC environment it is recommended to disable this subscription.   This feature was introduced in Oracle 10g. “

This recommendation was news to me, so this is why I put in a blog post because someone else will come across the same problem…the fix is to add to the listener.ora

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

and then reload or restart the listener. Using the Reload command in lsnrctl will be less disruptive than restarting.

Moving on UP….Now Executive Editor for SELECT Journal

Many thanks to John Kanagaraj, previous Executive Editor, who first spotted a contributing editor from among many reviewers. He is stepping down taking on the role of Associate Editor for a year while I transition into the new leadership role.

Wish both John and I luck…

We are always looking for new articles and reviewers.  The SELECT Journal is published four times a year, with the Best Practices Booklet once a year.

The SELECT Journal magazine focus/themes are primarily driven by IOUG members from an annual poll.

Security Warning jar files expire March 2011 for WebLogic 10.1.3.4 and FMW 11.1.1.2

We had this configuration in production use, and suddenly users received security warning dialog boxes.  Nothing on MOS for this issue and the support engineer associated with my SR didn’t find anything either.   This is just in case anyone else comes across this problem. We used the instructions from the following note on a 11g client install, it didn’t work (malformed certificate error) trying to diagnose using the Fusion Installation Home.

Using Note 1268757.1 I have verified that frmall11.jar and frmall.jar are both expired. It is not the signed jar files from the 3rd party (I verified those as well), it is Oracle.
Certificate[3]:
Owner: CN=”Oracle USA, Inc.”, OU=Products and Technology, OU=Digital ID Class 3 – Java Object Signing, O=”Oracle USA, Inc.”, L=Denver, ST=Colorado, C=US
Issuer: CN=VeriSign Class 3 Code Signing 2004 CA, OU=Terms of use at https://www.verisign.com/rpa (c)04, OU=VeriSign Trust Network, O=”VeriSign, Inc.”, C=US
Serial number: 7c1075295a1865d3633a7b4d852f45a0
Valid from: Sun Mar 02 17:00:00 MST 2008 until: Thu Mar 24 17:59:59 MDT 2011
Certificate fingerprints:
MD5: 4A:CB:89:E0:3F:41:94:CE:41:ED:29:43:B0:93:95:DB
SHA1: 18:FB:97:7F:13:B1:3C:6F:6A:85:41:CA:14:90:7D:97:AC:73:A2:1

WebLogic / OFM Note
——————-
The fix for WebLogic and OFM is to upgrade to OFM 11.1.1.4 with WebLogic 10.3.4
and it should include the updated JAR files.

2010 in review

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads Wow.

Crunchy numbers

About 3 million people visit the Taj Mahal every year. This blog was viewed about 33,000 times in 2010. If it were the Taj Mahal, it would take about 4 days for that many people to see it.

In 2010, there were 20 new posts, growing the total archive of this blog to 41 posts. There were 40 pictures uploaded, taking up a total of 10mb. That’s about 3 pictures per month.

The busiest day of the year was October 20th with 318 views. The most popular post that day was Migrating to 11gR2 – A Case Study on the Step-Ordered Approach.

Where did they come from?

The top referring sites in 2010 were blogs.oracle.com, utm.edu, google.com, google.co.in, and lists.sungardhe.com.

Some visitors came searching, mostly for ora-20011, flashback standby database, ora-29024: certificate validation failure, ora-04043: object xdb_datastore_proc does not exist, and xdb_datastore_proc oracle.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

Migrating to 11gR2 – A Case Study on the Step-Ordered Approach October 2010
6 comments

2

Grid Control 10.2.0.5 – Recommended Method for Installation, Faster and Easier June 2009
10 comments

3

Manual Upgrade to Oracle 11gR2 March 2010

4

Migrating to AL32UTF8 on 10gR2 for BANNER July 2008
3 comments

5

Troubleshooting Oracle Wallet – Hari Kari style August 2009
2 comments

Migrating to 11gR2 – with an Existing Logical Standby

Upgrading Oracle Database with a Logical Standby Database In Place [ID 437276.1]

• Migrating from 10.2.0.4 to 11.2.0.2
• Primary database is approximately 80 GB
• Binary software has already been installed on all nodes
• RMAN catalog upgraded
• Oracle listeners already migrated to newer version.
• Clients (sqlplus, procobol, pro C, odbc) are already at 11gR2.

This is all part of the Step-Ordered Approach to Migrating, see the following page for a case study.

These are not all of the steps, just an overview. Your mileage may vary. Please test in a non-production environment before implementing any information found here.

1.Deferred log shipping to the logical standby before first upgrading the primary database.

PRIMARY>ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=DEFER SCOPE=BOTH;

2. Stop SQL Apply on the standby database:

REPT> ALTER DATABASE STOP LOGICAL STANDBY APPLY;

3. Upgrade the primary database to the new release, these steps came from the case study page.

a.  run /tmp/utlu112i.sql, run /u01/app/oracle/clones/droplinks.sql, shutdown immediate;, exit session.

b. Double check the 11.2.0.2 initSID.ora is in place. New clean session, edit /etc/oratab, set to 11.2.x home, edit locally-modified files with changed environmental variables.

c. SQL>startup upgrade, @?/rdbms/admin/catupgrd, startup @?/rdbms/admin/catuppst, @?/rdbms/admin/utlrp

d. after11gupgrade.sql – this recreated the dropped links, dropped the password-protected roles.

e. Create password file

5. Create Network ACLS

6. Upgrade Oracle Wallet – start owm &, open existing wallet, make sure auto login is checked, save as in the same location.

4. Upgrade the logical standby database.

a. run /tmp/utlu112i.sql, run /u01/app/oracle/clones/droplinks.sql, shutdown immediate, exit session.

b. Double check the 11.2.0.2 initSID.ora is in place. New clean session, edit /etc/oratab, set to 11.2.x home, edit locally-modified files with changed environmental variables. Modified initSID.ora in the 11.2.x home, created spfile.

c. SQL>startup upgrade, @?/rdbms/admin/catupgrd, startup, @?/rdbms/admin/utlu112s.sql @?/rdbms/admin/catuppst, @?/rdbms/admin/utlrp

d. after11gupgrade.sql — this recreated the dropped links, removed any local password-protected roles.

e. Create password file

f. On the upgraded logical standby database, restart the database and restart SQL Apply

LOGICAL>STARTUP

LOGICAL>ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;

5. Enable archiving to the upgraded logical standby database:

PRIMARY>ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE scope=both;

6. Monitor the progress of the logical SQL APPLY process:

LOGICAL> SELECT APPLIED_SCN, APPLIED_TIME, READ_SCN, READ_TIME,  NEWEST_SCN, NEWEST_TIME  FROM DBA_LOGSTDBY_PROGRESS;

LOGICAL>SELECT PID, TYPE, STATUS, HIGH_SCN FROM V$LOGSTDBY;

9. Instantiate any previously skipped tables that have caused the SQL APPLY process to stop in the past. 11gR2 has added additional supported datatypes for the logical standby database. See the following MOS note: Step by Step Approach of How to Instantiate a Table in Logical Standby Database [ID 842160.1]

LOGICAL>alter database stop logical standby apply;

LOGICAL> select statement_opt,owner,name from dba_logstdby_skip where owner=’&SCHEMANAME’;

LOGICAL> exec dbms_logstdby.unskip(‘DML’, ‘SCHEMANAME’, ‘TABLENAME”);

LOGICAL>exec dbms_logstdby.instantiate_table(‘SCHEMANAME’,'TABLENAME’,'LINKNAME”);

LOGICAL>LOGICAL>ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE;