8.54 PreRelease Notes Available

The PeopleSoft Technology Blog has announced the availability of the PreRelease notes for 8.54:

Some observations as I skimmed the document are:

  • Oracle Linux 6, Win 2012, and Win 2012 R2 support added, Win 2008 (R1 not R2) dropped
  • Client OS Windows 8.1 added, Windows 7 (32 bit) dropped
  • WebLogic 12.1.2 added, 10.3.6 dropped
  • Oracle 12, MSSQL 2014 added, Oracle 10.2.0.5, 11.2.0.3, and MSSQL 2008 dropped
  • Current browsers available added (Chrome, Firefox, IE 11, Safari), IE 8 Firefox 17 dropped
  • Tuxedo 12.1.1.0 added, Tuxedo 11gR1 dropped
  • Still uses Java 7
  • SES is 11.2.2.2, SES 11.1.2.2 dropped
  • Excel 32-bit dropped
  • New Fluid User Interface: moves away from ridge page layouts enhancing the use of CSS3, HTML5, and JavaScript.  Supposed to “scale gracefully between devices”.  Fluid page definitions are maintained within App Designer.  Adds Fluid Homepages, Tiles, Notification Framework, PeopleSoft Navigation Bar
  • Mobile Application Platform: Similar to the Fluid User Interface but utilizes RESTful web services
  • 64-bit Development environment!  App Designer, Data Mover, Change Assistant, the whole lot of them all now 64-bit.  Explains the dropped Win 7 (32-bit).
  • App Designer will have improved search functionality (reference, text), code auto-completion for PeopleCode, and new toolbar buttons to improve productivity
  • Enhancements to App Engine tracing: split files, naming convention, program section trace, combined output of PeopleCode and SQL into the AE trace file
  • Portable PS_HOME: Hard-coded paths and sym links within PS_HOME have been removed to further consolidate and allow a single PS_HOME to be shared across multiple environments
  • Two new metaSQL enhancements for Oracle added.  %SqlHint and %SelectDummyTable
  • Also you can now use Oracle Global Temporary Tables, Materialized Views, and the new 12c container/pluggable databases which allow multiple PS databases in the same instance, some may have used GTTs and Materialized Views in the past, but now App Designer will handle them.  Also, App Designer can now be used directly to partition tables and indexes on Oracle.
  • Domain caching changes allowing automatic monitoring and adjusting?  I’ll have to look into that one more.
  • A Push Notification Event Framework, maybe we can finally broadcast a message to all users via the system.
  • Several security enhancements, Oracle Secure Files for the report repository is one that jumped out at me.
  • There’s a lot of other stuff, Just go read it !

Read the foll post here, or find the notes directly on MOS here

PeopleSoft TEMP/TMP Directories

In PeopleTools 8.53 Oracle has changed how the TEMP and TMP environment variables are handled.  They are taking the responsibility of setting these out of the Admins hands.  In the past we set these as environment variables perhaps in the shell profile or psconfig.sh.  When we configured the app server / process scheduler these settings would be inherited into the Tuxedo configuration.  With Windows 2008, some were impacted with it’s default of dynamic TEMP/TMP variables based on session and being deleted on logout.  On Windows a previous co-worker of mine got me into the practice of modifying the psappsrv.ubx and prprcs.ubx files and setting these variables there rather than relying on what the user in Windows might have set or trying to adjust them with scripts.  This proved to be helpful in many ways.  Now Oracle has decided to do the same thing by default.

This may impact Admins who are used to having these variables set to something they specifically wanted.  Depending on your setup you may prefer to change these back.  A default psappsrv.ubx file has the following section:

# ————–

*PS_ENVFILE
TEMP={LOGDIR}{FS}tmp
TMP={LOGDIR}{FS}tmp
TM_BOOTTIMEOUT=120
TM_RESTARTSRVTIMEOUT=120

{LOGDIR} is $PS_CFG_HOME/appserv/<DOMAIN>/LOGS and {FS} is the OS specific path delimiter.  So this will use a temp directory in the LOGS directory of each domain such as /opt/apps/psoft/domains/appserv/HCM92/LOGS/tmp.

I’m not sure how much of a fan I am of having TEMP in the logs directory.  I guess just knowing it’s moving is half the battle.  It’s easy to change back if you like.

See Oracle support document [ID 1486978.1] for the announcement.  I don’t recall seeing this in the release notes, but I might have missed it.  They also said it would be back ported to 8.52.16, but you would need to recreate all your domains, not just reconfigure in order to get the change.  Oracle is adding these settings to the domain templates, once you create the new domain from a template TEMP and TMP will be set in the appropriate ubx file.

Webserver detected with incorrect Version of JDK

Now that PeopleTools 8.53 is out, I’ve seen people that are doing PIA installs posting about this problem a few places.

The only  reason I’ve seen so far for this error is the value for JAVA_HOME is not properly set in $WL_HOME/wlserver_10.3/common/bin/commEnv.sh.  This is set when performing the install of Weblogic.  Fix this and try to install PIA again.

To fix it, look at commEnv file, and find where JAVA_HOME is set.  Double check that it’s the right location to the JDK 7 location.  PeopleTools 8.53 now requires JDK 7 and it is available with the rest of the PeopleTools downloads on Oracle’s Edelivery site.  However, the version provided on edelivery is an rpm and by default will install to /usr/java.  If you are like me, you may want to install this JDK for Weblogic at a specific location.  If that’s the case, I recommend skipping the JDK rpm from edelivery and grab the JDK tarball from the OTN download site.  Download the tar.gz version and do what you want with it.  If you do not have root access this will be helpful too as the rpm installer will want you to be root.  If you do want to use the rpm you can use the – -prefix  (2 dashes without a space) option to install to another location. You’ll still need to be root and it will still place some things in /etc.

I still prefer just extracting the tarball.  There is something to be said about the simplicity of it.  If you:

  1. install the tarball
  2. export JAVA_HOME=/path/to/jdk
  3. install Weblogic with $JAVA_HOME/bin/java -jar /path/to/wls1036_generic.jar -whatever -other -options you want
  4. source psconfig.sh and run $PS_HOME/setup/PsMpPIAInstall/setup.sh

You should have no problems, it worked fine for me on OEL 5.8.  If you used the RPM and are still having problems.  Try running as root

/usr/sbin/alternatives --config java

Which will change system wide the default version of java used.  But you shouldn’t have this problem if you used my method above.

Hope this helps.

PeopleSoft Timeout Problems in 8.50 – 8.52

There have been several PeopleSoft Timeout problems that have impacted various levels of 8.50, 8.51, and 8.52.  Indications are all these problems are fixed now.  According to Oracle, the code has also been fixed in 8.50.25, 8.51.17, 8.52.07, and 8.53.  I’m sure a new problem will pop up at some point though.

The first problem which I consider the most dangerous, is that the timeout warning process breaks down and causes a users browser to repeatedly request the timeout warning pop-up page.  It looks like this particular problem impacts 8.50.22, 8.50.23, and 8.50.24.  It looks like it also impacts 8.51.12 – 8.51.16 and 8.52.00 –  8.52.06.  I don’t have the 8.51 or 8.52 versions running, however looking at the code delivered in the 8.51.16 and 8.52.06 patches, the problem code does exist.  I know 8.52.07 does not have the bad code because I do have access to an environment at that level.  This was actually put in as a fix for another problem, which I’ll mention later.  It was put into 8.50.22, 8.51.12, and 8.52.00.

Users may have their PC’s hang as the browser takes CPU to 100% requesting the timeout warning page. Or if you have access logging turned on and you review the web server access log you will see hundreds to thousands of requests for the warning page from a single IP repeatedly in quick succession.  In some cases this can cause the web server to crash.  In other cases I’ve heard of it maxing out jolt connections on the app server.  Oracle has a work around posted [ID 1405219.1].

It looks like they used the wrong JavaScript method and instead of using setTimeout() used setInterval().  What then occurs is the user will have a small amount of time left after the original setTimeout() was called. There is an if statement in the code that “if (current time – last access time) is less than (the warning time – a small adjustment)”  then call setInterval() for the pop-up with the interval being the remaining time.  This is the fix they put in (the if statement was there earlier without the setInterval() call).  For some reason certain systems are having time left over and entering this If condition, but in older 8.50 code the loop simply returned thus exiting without displaying the time out window.  Well setInterval() runs it’s parameter repeatedly at the specified interval.  In order to just delay the pop-up the remaining amount of time it should be setTimeout().  The work around directs you to correct this.

If you are experiencing this first problem.  After updating the code on the HTML object PT_SAVEWARNINGSCRIPT you may need to force users to clear their browser cache.  I saw this JavaScript being cached on the browser and not being reloaded even though the web servers had been restarted and their own cache cleared.  Since there was 4000 users at the site, they changed the cache directory in the web profile to force a new URL which did the trick.

Another problem is that time outs across windows/tabs are not synched.  This one I think has existed for a long time actually. Oracle indicates that there was a bug in 8.4x that they did not fix which allowed users to not be timed out of idle windows but since it was a bug, it was fixed in the 8.50 release.  I’m not sure what 8.4x releases that bug impacted.  Good news is it’s fixed moving forward.  Oracle says this about the behavior.

In PT8.4, if end user is active in one browser window, but lets the other window remain idle and timeout, the end user could continue in active window. This was a bug: 11558697 in PT8.4 and incorrect behavior. Many customers prefer the incorrect behavior in PT8.4; however, that behavior was a bug. The timeout behavior in PT8.50 is correct and development will not revert back to PT8.4 timeout behavior.

So with 8.50 users start experiencing timeout behavior they may not have seen before.  If your users perform a function that commonly spawns new windows this may impact them.  One particular scenario would be they simply missed the timeout warning pop-up and didn’t acknowledge it, they would now be timed out even if they were active in another window.  This is corrected now in the newer releases indicated above.  They have rewritten the JavaScript to utilize the value of PS_TOKENEXPIRE, which provides a singular Last Access time, across the browsing session.  A nice change if you ask me.  In fact, since PS_TOKENEXPIRE is shared across all SSO shared environments, that actually allows Portal and Content databases to timeout together, another nice addition.  I saw a reference to PeopleBooks 8.53 being updated to specify this, however I wasn’t able to find it yet.

Another more complicated scenario of this problem allows users to be timed out even after clicking OK on the timeout warning window.  This probably doesn’t happen all that often but I heard from a previous co-worker that he has had users impacted by this.  If you open three windows pretty close to one another and remain only active in the third one and leave the visualother two open, you could fall prey to this.  Here’s the details.  Window A gets opened with it’s own timeout. 10 seconds later the user then opens window B, also with it’s own timeout.  In window B they do something to open window C. They continue to work in window C.  The warning will be displayed based on inactivity for window A.  If the user does not acknowledge the timeout by the time window B gets to it’s warning time (10 seconds after window A) window B will initiate a new warning.  However, the warning from window B won’t create a new window, it will update the existing pop-up from window A.  Now we have a problem.  Two windows with separate timers will timeout in a few minutes but there is only one pop-up window.  My testing with IE indicates the JavaScript will actually reset the timer in window A however the browser will make a GET call for window B.  The GET call doesn’t impact the JavaScript timer but it will keep the web server session alive (but your already active in the other window so we are fine there).  What then happens is window B will time out shortly leaving the user wondering what just happened?  They clicked OK on the warning but still got timed out.  As you can see, I tried to present a visual depiction of this process.  I’m not very good with those tools so I hope you can enjoy a laugh at my art work.

I’ve also seen reference to some other problems regarding where the setuptimeout() function is called in the JavaScript.  But I don’t have too many details on that one, it sounds like they were calling setuptimeout() from code associated with key presses.  This did not necessarily make a request to the server.  So you could be typing up your performance review for 2 hours and the window would not time out.  But once you went to save the review, your session had already timed out, and all was gone.  I think even I have been hit by that one before.  This also has been fixed with the new code, you’ll be warned about timing out now to avoid your web server session from vanishing.  This fix was also supposed to go in with the 8.51.17 / 8.50.25 updates.

When PeopleSoft Cache goes Bad

In general Cache is a good thing, but sometimes things go wrong. I usually try to avoid clearing cache unless I have a real reason. In my experience clearing cache without reason usually only adds to the end users perception that PeopleSoft is a slow painful application to work with, and we know PeopleSoft gets plenty of opportunities to prove that daily. In large scale environments if you don’t have a cache building process, clearing cache across multiple domains with a decent amount of processes each could put a significant damper on some users mornings. Say perhaps, your Expense Approval team needs to re-cache 30 – 40 processes, that might ruin their morning. This post isn’t about what makes things go wrong, but how to possibly identify and deal with them in the least impactful manner as possible. Let me show you a recent case I ran into. In this example I’ll go over some basics of tmadmin so if you’ve been doing this a while, you’ll probably already know the stop/start and psr commands, but maybe you’ll learn something new.

Users started reporting intermittent errors, sometimes things worked sometimes they didn’t. That was hint #1. In this case, the errors were around the Query Manager pages, and three different error messages were being reported. Function CheckSec not found in Peoplecode program QRYFUNCTIONS.QRYQUERYFUNCS.FieldFormula. Page load failed for QUERY_MANAGER/GBL. and Data Integrity Error. Usually these error types will show up in the APPSRV log. They would look something like this…


PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) Function CheckSec not found in PeopleCode program QRYFUNCTIONS.QRYQUERYFUNCS.FieldFormula. (2,301)
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) PRMGet failed for component QUERY_MANAGER market GBL
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) Data Integrity Error (124,85)
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) Function CheckSec not found in PeopleCode program QRYFUNCTIONS.QRYQUERYFUNCS.FieldFormula. (2,301)
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) PRMGet failed for component QUERY_MANAGER market GBL
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) An error has occurred which prevents this transaction continuing

If you can find these in your log file then the next step is to determine how many processes they are coming from. Use grep or find (if your on a windows servers) to trim the error messages down to an individual one. For instance, I ran
grep 'PRMGet failed for component QUERY_MANAGER' APPSRV_0117.LOG
You could add something like | awk -F" " '{print $1}' if you had did a similar pattern for grep. Anyway, what we are after is the PSAPPSRV pid, which is the first number after PSAPPSRV. In our example above it’s 7546.

If you look at the entries that come back and find that all the errors are coming from only one pid that is hint #2. Remember that the users were saying that some of the requests work, it’s possible we’ve identified a process that has either gone haywire or it’s cache has gone bad (corrupt). Let’s shut this particular process down and see if our users report the problem stops. There are several ways to identify which appserv process this is, you could use ps, or task manager, but since we want to shut it down properly lets just use our psadmin tools.

run psadmin, pick our domain that has problems, and use option 5
TUXEDO command line (tmadmin)

at the tmadmin prompt “>” enter psr -v -g APPSRV (-v is verbose output, -g limits output to the group name specified)

> psr -v -g APPSRV

the output is paginated, so just page through until you find the process with the process id that matches the one your looking for.


Group ID: APPSRV, Server ID: 2
Machine ID: psapp1.localdomain
Process ID: 7546, Request Qaddr: 229380, Reply Qaddr: 129663006

Here it is, PID 7546 is server id 2. Now if we just run psr server id is the ID column

So now let’s shut down our possible trouble maker.

> shutdown -g APPSRV -i 2

There, we shutdown just ONE of multiple PSAPPSRV processes. Now we can have users test, if the problem goes away, we really did find our culprit. Now it’s safe to clear cache. Are we sure it’s safe? Well let’s take a look. Again, I’m going to describe it as performed on a Linux server, but you could use Process Explorer from sysinternals to do the same verification step here.

Lets make sure no one is using our cache files which might cause us head ache if we were to try to wipe them prematurely. As your PeopleSoft application user on the Linux box run lsof and grep for CACHE

$ /usr/sbin/lsof |grep CACHE

...
.
PSAPPSRV 15803 psoft 178u REG 252,2 0 263319 /opt/apps/psoft/domains/appserv/PA91/CACHE/PSAPPSRV_1/SDEFM.DAT
.
...

Look at that, as an example I can see that PSAPPSRV_1/SDEFM.DAT is open by process PSAPPSRV pid 15803 which is Server ID 1. That builds the correlation that the CACHE/PSAPPSRV_1 directory belongs to the PSAPPSRV process with server id 1. Well that makes sense.

We shut down server ID 2, let see if that directory has anything open.

$ /usr/sbin/lsof |grep PSAPPSRV_2

Crickets, just the prompt returned. Perfect, let’s remove all that cache and give it a fresh start. From inside the domain directory let’s run our rm command,

$ rm -rf CACHE/PSAPPSRV_2

Now let’s start up the process again. Go back into psadmin and run tmadmin again.

> boot -g APPSRV -i 2

The process boots, recreates PSAPPSRV_2 in CACHE and is ready to service requests.

So there we have it. In my case, the problems went away and I inconvenienced a much smaller set of users than I would have if I cleared cache across the board. I’ve seen this pop up a few times in the last 2 years and each time this strategy has worked well. One time there was a problem with process scheduler PSAE cache on a piece of SQL (the SQL statement was actually only partially returned). The job would fail when it ran on PSAE server ID 3, but the same principal applied. In that case it took me a little longer to determine that it was really a caching problem, but that’s what it ended up being. Happy trouble shooting.

Why you should have different local node passwords

I was doing an assessment of a Peoplesoft installation recently when I noticed that the node passwords were the same across all environments of the same application type. While this might not seem like it is a big deal, it can open up a production environment to unauthorized access or accidental data entry. If this sounds like your environment perhaps some of your developers or users may have already realized that this allows them to access production with out a password from a non production environment, especially if they use bookmarks or direct URLs.

The problem here is related to a mechanism of Peoplesoft’s authentication. The PS_TOKEN cookie which is used for authentication is created from among other things the issuing systems default local node and password. If this data is the same across say, all your HCM environments, then a PS_TOKEN created in a development environment can also be valid in the corresponding production environment.

My recommendation is simple enough, force the node passwords to be different for every database in your install via your refresh script or process. This will ensure the PS_TOKEN is not valid across multiple instances of the same application and prevent this rather undesirable “feature” that I know several developers have enjoyed and even relied on in the past.

Environment, Environment, Environment

I’ve seen truck loads of environment problems over the years.  This one was one of the more recent ones.  We start out with reports failing to post with the following error in the PSDSTSRV log.

PostReport](3) PSJNI: Created a Java VM instance
PostReport](1) PSJNI: Java exception thrown: java.lang.UnsupportedClassVersionError: Bad version number in .class file
PostReport](3)     HTTP transfer error.
PostReport](3)      Post Report Elapsed Time: 0.1800

The message log for the process shows the following error:

PSUNX failed to post files to the report repository.  Server scheduled to try again on...  See log
HTTP Status Code is: 905 (63,72)

“Bad version number in .class file” is a mismatch between current java version and version used to compile the source

$ which java
/opt/apps/tuxedo10gR3/jre/bin/java
$ java -version
java version "1.5.0_08"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_08-b03, mixed mode)

looked at .bash_profile, tux.env was being sourced after psconfig.sh which was updating PATH variable with tux info last swapped order of sourcing of files and retested

$ which java
/opt/apps/psoft/fin851/jre/bin/java
$ java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

Resubmit and… Posting success!