Building out a new HCM 9.2 Update Image 9 DEMO

As mentioned here an updated version of  the PeopleSoft HCM media was released in late 2014.  I’ve done a few Demo installs of this on both Linux and Windows now, and here are my notes from a Linux install.  As in the past, these are rough notes I take when I do any install and not a how-to guide.  This install requires PeopleTools 8.54.03 patch as that is the level of PeopleTools within the database.  The media available on Oracle Software Delivery Cloud is only 8.54.00.  This install includes all PeopleSoft HCM patches through Update Image 9, a great thing for those using it to upgrade. Continue reading

PeopleSoft PUM Images on Linux KVM

As we know, with the new PeopleSoft 9.2 applications came a new method of delivering updates via PeopleSoft Update Manager.  We now need to download fully functional VirtualBox images to get our patches.  We can also use these images for Demo environments and specific versions are useable for New Release Demos during application upgrades.  SES is even included in these images.  I’ve been using VirtualBox for years, since the original innotek days.  It works great for desktop use which is it’s intended use.  Sure you can set it up on a server and make it work in a more enterprise environment but I wanted to deploy the images to something I already had hosting some vm’s, Linux KVM.  If you want to run your images on VMWare, I recommend reading this post on running the images on VMWare.  So lets begin setting up a new PUM image for Linux KVM. Continue reading

8.54 PreRelease Notes Available

The PeopleSoft Technology Blog has announced the availability of the PreRelease notes for 8.54:

Some observations as I skimmed the document are:

  • Oracle Linux 6, Win 2012, and Win 2012 R2 support added, Win 2008 (R1 not R2) dropped
  • Client OS Windows 8.1 added, Windows 7 (32 bit) dropped
  • WebLogic 12.1.2 added, 10.3.6 dropped
  • Oracle 12, MSSQL 2014 added, Oracle 10.2.0.5, 11.2.0.3, and MSSQL 2008 dropped
  • Current browsers available added (Chrome, Firefox, IE 11, Safari), IE 8 Firefox 17 dropped
  • Tuxedo 12.1.1.0 added, Tuxedo 11gR1 dropped
  • Still uses Java 7
  • SES is 11.2.2.2, SES 11.1.2.2 dropped
  • Excel 32-bit dropped
  • New Fluid User Interface: moves away from ridge page layouts enhancing the use of CSS3, HTML5, and JavaScript.  Supposed to “scale gracefully between devices”.  Fluid page definitions are maintained within App Designer.  Adds Fluid Homepages, Tiles, Notification Framework, PeopleSoft Navigation Bar
  • Mobile Application Platform: Similar to the Fluid User Interface but utilizes RESTful web services
  • 64-bit Development environment!  App Designer, Data Mover, Change Assistant, the whole lot of them all now 64-bit.  Explains the dropped Win 7 (32-bit).
  • App Designer will have improved search functionality (reference, text), code auto-completion for PeopleCode, and new toolbar buttons to improve productivity
  • Enhancements to App Engine tracing: split files, naming convention, program section trace, combined output of PeopleCode and SQL into the AE trace file
  • Portable PS_HOME: Hard-coded paths and sym links within PS_HOME have been removed to further consolidate and allow a single PS_HOME to be shared across multiple environments
  • Two new metaSQL enhancements for Oracle added.  %SqlHint and %SelectDummyTable
  • Also you can now use Oracle Global Temporary Tables, Materialized Views, and the new 12c container/pluggable databases which allow multiple PS databases in the same instance, some may have used GTTs and Materialized Views in the past, but now App Designer will handle them.  Also, App Designer can now be used directly to partition tables and indexes on Oracle.
  • Domain caching changes allowing automatic monitoring and adjusting?  I’ll have to look into that one more.
  • A Push Notification Event Framework, maybe we can finally broadcast a message to all users via the system.
  • Several security enhancements, Oracle Secure Files for the report repository is one that jumped out at me.
  • There’s a lot of other stuff, Just go read it !

Read the foll post here, or find the notes directly on MOS here

How I forward displays from Linux or other *nix systems to Windows

First off, I usually try to avoid it, I’m old school and like my command line options.  So, often when I’m installing PeopleSoft components, I use the console options if available.  Sometimes it’s not that simple though so when that happens, I’ve got a pretty standard method of doing things.  So here’s how I forward a display from a Linux server to my Windows workstation. Continue reading

PeopleSoft TEMP/TMP Directories

In PeopleTools 8.53 Oracle has changed how the TEMP and TMP environment variables are handled.  They are taking the responsibility of setting these out of the Admins hands.  In the past we set these as environment variables perhaps in the shell profile or psconfig.sh.  When we configured the app server / process scheduler these settings would be inherited into the Tuxedo configuration.  With Windows 2008, some were impacted with it’s default of dynamic TEMP/TMP variables based on session and being deleted on logout.  On Windows a previous co-worker of mine got me into the practice of modifying the psappsrv.ubx and prprcs.ubx files and setting these variables there rather than relying on what the user in Windows might have set or trying to adjust them with scripts.  This proved to be helpful in many ways.  Now Oracle has decided to do the same thing by default.

This may impact Admins who are used to having these variables set to something they specifically wanted.  Depending on your setup you may prefer to change these back.  A default psappsrv.ubx file has the following section:

# ————–

*PS_ENVFILE
TEMP={LOGDIR}{FS}tmp
TMP={LOGDIR}{FS}tmp
TM_BOOTTIMEOUT=120
TM_RESTARTSRVTIMEOUT=120

{LOGDIR} is $PS_CFG_HOME/appserv/<DOMAIN>/LOGS and {FS} is the OS specific path delimiter.  So this will use a temp directory in the LOGS directory of each domain such as /opt/apps/psoft/domains/appserv/HCM92/LOGS/tmp.

I’m not sure how much of a fan I am of having TEMP in the logs directory.  I guess just knowing it’s moving is half the battle.  It’s easy to change back if you like.

See Oracle support document [ID 1486978.1] for the announcement.  I don’t recall seeing this in the release notes, but I might have missed it.  They also said it would be back ported to 8.52.16, but you would need to recreate all your domains, not just reconfigure in order to get the change.  Oracle is adding these settings to the domain templates, once you create the new domain from a template TEMP and TMP will be set in the appropriate ubx file.

When PeopleSoft Cache goes Bad

In general Cache is a good thing, but sometimes things go wrong. I usually try to avoid clearing cache unless I have a real reason. In my experience clearing cache without reason usually only adds to the end users perception that PeopleSoft is a slow painful application to work with, and we know PeopleSoft gets plenty of opportunities to prove that daily. In large scale environments if you don’t have a cache building process, clearing cache across multiple domains with a decent amount of processes each could put a significant damper on some users mornings. Say perhaps, your Expense Approval team needs to re-cache 30 – 40 processes, that might ruin their morning. This post isn’t about what makes things go wrong, but how to possibly identify and deal with them in the least impactful manner as possible. Let me show you a recent case I ran into. In this example I’ll go over some basics of tmadmin so if you’ve been doing this a while, you’ll probably already know the stop/start and psr commands, but maybe you’ll learn something new.

Users started reporting intermittent errors, sometimes things worked sometimes they didn’t. That was hint #1. In this case, the errors were around the Query Manager pages, and three different error messages were being reported. Function CheckSec not found in Peoplecode program QRYFUNCTIONS.QRYQUERYFUNCS.FieldFormula. Page load failed for QUERY_MANAGER/GBL. and Data Integrity Error. Usually these error types will show up in the APPSRV log. They would look something like this…


PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) Function CheckSec not found in PeopleCode program QRYFUNCTIONS.QRYQUERYFUNCS.FieldFormula. (2,301)
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) PRMGet failed for component QUERY_MANAGER market GBL
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) Data Integrity Error (124,85)
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) Function CheckSec not found in PeopleCode program QRYFUNCTIONS.QRYQUERYFUNCS.FieldFormula. (2,301)
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) PRMGet failed for component QUERY_MANAGER market GBL
PSAPPSRV.7546 (1610) [01/17/13 15:34:50 user@client.where.com (IE 8.0; WIN7) ICPanel](0) An error has occurred which prevents this transaction continuing

If you can find these in your log file then the next step is to determine how many processes they are coming from. Use grep or find (if your on a windows servers) to trim the error messages down to an individual one. For instance, I ran
grep 'PRMGet failed for component QUERY_MANAGER' APPSRV_0117.LOG
You could add something like | awk -F" " '{print $1}' if you had did a similar pattern for grep. Anyway, what we are after is the PSAPPSRV pid, which is the first number after PSAPPSRV. In our example above it’s 7546.

If you look at the entries that come back and find that all the errors are coming from only one pid that is hint #2. Remember that the users were saying that some of the requests work, it’s possible we’ve identified a process that has either gone haywire or it’s cache has gone bad (corrupt). Let’s shut this particular process down and see if our users report the problem stops. There are several ways to identify which appserv process this is, you could use ps, or task manager, but since we want to shut it down properly lets just use our psadmin tools.

run psadmin, pick our domain that has problems, and use option 5
TUXEDO command line (tmadmin)

at the tmadmin prompt “>” enter psr -v -g APPSRV (-v is verbose output, -g limits output to the group name specified)

> psr -v -g APPSRV

the output is paginated, so just page through until you find the process with the process id that matches the one your looking for.


Group ID: APPSRV, Server ID: 2
Machine ID: psapp1.localdomain
Process ID: 7546, Request Qaddr: 229380, Reply Qaddr: 129663006

Here it is, PID 7546 is server id 2. Now if we just run psr server id is the ID column

So now let’s shut down our possible trouble maker.

> shutdown -g APPSRV -i 2

There, we shutdown just ONE of multiple PSAPPSRV processes. Now we can have users test, if the problem goes away, we really did find our culprit. Now it’s safe to clear cache. Are we sure it’s safe? Well let’s take a look. Again, I’m going to describe it as performed on a Linux server, but you could use Process Explorer from sysinternals to do the same verification step here.

Lets make sure no one is using our cache files which might cause us head ache if we were to try to wipe them prematurely. As your PeopleSoft application user on the Linux box run lsof and grep for CACHE

$ /usr/sbin/lsof |grep CACHE

...
.
PSAPPSRV 15803 psoft 178u REG 252,2 0 263319 /opt/apps/psoft/domains/appserv/PA91/CACHE/PSAPPSRV_1/SDEFM.DAT
.
...

Look at that, as an example I can see that PSAPPSRV_1/SDEFM.DAT is open by process PSAPPSRV pid 15803 which is Server ID 1. That builds the correlation that the CACHE/PSAPPSRV_1 directory belongs to the PSAPPSRV process with server id 1. Well that makes sense.

We shut down server ID 2, let see if that directory has anything open.

$ /usr/sbin/lsof |grep PSAPPSRV_2

Crickets, just the prompt returned. Perfect, let’s remove all that cache and give it a fresh start. From inside the domain directory let’s run our rm command,

$ rm -rf CACHE/PSAPPSRV_2

Now let’s start up the process again. Go back into psadmin and run tmadmin again.

> boot -g APPSRV -i 2

The process boots, recreates PSAPPSRV_2 in CACHE and is ready to service requests.

So there we have it. In my case, the problems went away and I inconvenienced a much smaller set of users than I would have if I cleared cache across the board. I’ve seen this pop up a few times in the last 2 years and each time this strategy has worked well. One time there was a problem with process scheduler PSAE cache on a piece of SQL (the SQL statement was actually only partially returned). The job would fail when it ran on PSAE server ID 3, but the same principal applied. In that case it took me a little longer to determine that it was really a caching problem, but that’s what it ended up being. Happy trouble shooting.

Ubuntu 10.04 LTS to 12.04 LTS upgrade

I ran a few servers that were Ubuntu 10.04 LTS.  I decided I’d upgrade them to 12.04 LTS now that its out.  Though this doesn’t have anything to do with PeopleSoft, I figured I’d post the how to anyhow.  Here are the steps:

  • Make a backup of what ever you may need for how ever you use your machine.
  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt-get install update-manager-core
  • confirm Prompt=lts set in /etc/update-manager/release-upgrades
  • sudo do-release-upgrade -d ; #follow the prompts during the upgrade process.  There may be several depending on your installed applications
  • Your done

I have two items to note after the upgrade. I had to update my squid config.  squid3 was installed, previously I was running squid2.7.  Well the paths changed so my old config was in /etc/squid and the new config was in /etc/squid3 so I had to update that.

The other had to do with the NFS server not being upgraded properly. I found some bug documented about this but it said it was fixed a while ago.  For whatever reason things didn’t work for me so here is what I did to fix it.

After the upgrade all my clients were getting the following buffer overflow

*** buffer overflow detected ***: mount.nfs terminated
======= Backtrace: =========
/lib64/libc.so.6(__chk_fail+0x2f)[0x7fd66ae77c9f]
mount.nfs[0x7fd66b30cb6c]
mount.nfs[0x7fd66b309dc6]
mount.nfs(main+0x5b8)[0x7fd66b30a468]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x7fd66adae994]
mount.nfs[0x7fd66b309729]

I forced rpcbind to install to get me on track

$sudo apt-get install rpcbind
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
libtirpc1 nfs-common
The following packages will be REMOVED:
portmap
The following NEW packages will be installed:
libtirpc1 rpcbind
The following packages will be upgraded:
nfs-common

$sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
nfs-kernel-server
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

After that, no more problems from NFS.

PeopleSoft Desktop Single Sign-on via Kerberos – Part 3

Welcome to the third installment of PeopleSoft Desktop Single Sign-on via Kerberos.  I hope to wrap up everything in this final post. In Part 1 we configured our Linux servers to talk to our Active Directory server and setup a user/SPN for our Kerberos Authentication. In Part 2 of the PeopleSoft Desktop SSO write up we configured our Linux Weblogic instance to use the Oracle provided servlet filter.  We set filter mapping to /* to force every request through the KerberosSSO filter.  Now it is time to move on to our app server and online configurations to finish this up.

In order for the application server to validate the Kerberos token we need to copy the java class files to $PS_HOME/class/com/peoplesoft/pt/desktopsso/kerberos.  Oracle seems to be delivering these in $PS_HOME/class/com/peoplesoft/PT/desktopsso/kerberos but that doesn’t work! So either recopy these files from our webserver or the “PT” directory.  We need both KerberosSSOValidator$1.class and KerberosSSOValidator.class.

Next lets update our psappsrv.cfg file with the following
-Djava.security.auth.login.config=/home/psoft/krbLogin.conf
-Djava.security.krb5.conf=/etc/krb5.conf

Look familiar? Yep, we did this on the web server. Did you create these files on the app server yet? If not copy them from your web server, don’t forget to copy the keytab file which is referenced in krbLogin.conf.

So the JavaVM Options line will read something like (it’s around line 925 in my config file)
JavaVM Options=-Dxdo.ConfigFile=%PS_HOME%/appserv/xdo.cfg -Djava.security.auth.login.config=/home/psoft/krbLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf

Next lets edit the Peoplecode for FUNCLIB_LDAP.LDAPAUTH. First we want to edit the getWWWAuthConfig function.  Update the username identified in &defaultUserID.

/* Updated for Kerberos Sign On */
&defaultUserId = "PUBUSER";

Next we want to add the following function at the end.

Online create our public user PUBUSER and enable this user for guest login capabilities in the webprofile.  Reload the profile using reloadconfig command or restart the web server. Once reloaded try it out and see if it works.  Don’t load the signon.html page, it’s the signon page and of course will ask for a username/password.  Start by trying to go to http://websrv.testdomain.com/psp/ps/EMPLOYEE/ERP/h/?tab=DEFAULT for Finance for example.

PeopleSoft Desktop Single Sign-on via Kerberos – Part 1

With the release of PeopleTools 8.51 Desktop Single Signon via Kerberos is now a supported reality.  Though this can’t really be considered a delivered solution, I’d call it a reference solution, for those that want to build their own it is at least available and documented.  So here is a walk through of setting up the  “reference” desktop single sign-on solution. (I’m doing this on 8.52 but 8.51 should be very similar).  I’ll also be doing this on Oracle Enterprise Linux servers (Oracle Linux Server release 5.8 to be precise) while Oracle’s documentation is geared towards an all Windows environment.

First lets go over the basic plan of attack:

  1. Need an Active Directory server (instructions not included)
  2. Need our Linux servers to be able to authenticate against said AD
  3. Create required accounts/SPN and keytab file
  4. Need to enable servlet filter on Weblogic to handle Kerberos Token
  5. Need to enable app server to handle Kerberos Token
  6. Need to enable public user and single signon kerberos peoplecode
  7. Need to test and troubleshoot if needed

I’m not going to rehash everything explained in the PeopleBooks article, if you want their explanation of how this actually works you can read it for yourself, it’s a decent writeup.

If you don’t have an Active Directory server already available that’s beyond the scope of this article.

Let’s setup our servers to talk to our Domain Controller via Kerberos.  With Windows, if your servers are in your Domain this is already taken care of but for us Linux users we need to make it happen.  Here’s how I did it and there isn’t a lot to it.

First, lets edit /etc/krb5.conf. It should look something like this.
[libdefaults]
ticket_lifetime = 24000
default_realm = TESTDOMAIN.LOCAL
dns_lookup_realm = yes
dns_lookup_kdc = yes

For this exercise I commented out the realms and domain_realms section. The above settings rely on DNS to look up the kdc, so if for some reason your not using DNS from your kdc you might want to use a config similar to below where you specify an IP or name for your kdc.
[libdefaults]
default_realm = TESTDOMAIN.LOCAL
ticket_lifetime = 24h
forwardable = yes
[realms]
TESTDOMAIN.LOCAL = {
kdc = 192.168.1.30
}
[domain_realm]
.testdomain.local = TESTDOMAIN.LOCAL
testdomain.local = TESTDOMAIN.LOCAL

There are many other options available in this configuration file, see the man page for more details. What I provide here is the basics to get things moving. Once krb5.conf is setup try running kinit to test the ability to acquire a ticket. For this we need to know a user and password in AD.

[root@appsrv ~]# kinit USER@TESTDOMAIN.LOCAL
Password for USER@TESTDOMAIN.LOCAL:
[root@appsrv ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: USER@TESTDOMAIN.LOCAL

Valid starting Expires Service principal
07/12/12 21:32:50 07/13/12 07:34:04 krbtgt/TESTDOMAIN.LOCAL@TESTDOMAIN.LOCAL
renew until 07/13/12 21:32:50

Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached

[root@appsrv ~]# kdestroy

Ok, That works, we got a ticket, listed it, and got rid of it. From the man pages:
kinit - obtain and cache Kerberos ticket-granting ticket
klist - list cached Kerberos tickets
kdestroy - destroy Kerberos tickets

Do this for both the web and app server.

Now let’s create our service account, HTTP service principal, and a keytab file. A side note here, I took a short cut by creating a single account to use for both servers. In a real world environment you might want to consider using Computer Accounts and actually adding the Linux servers as members in the Domain.

  1. Create USERs in AD: add user krbauth/p@ssw0rd
  2. run command prompt as admin
  3. Setspn.exe -S HTTP/websrv.testdomain.com krbauth

webserv.testdomain.com needs to match your server and DNS domain name although my AD Domain is testdomain.local I have testdomain.com as a working DNS entry on this network.
C:\Windows\system32>setspn -S HTTP/websrv.testdomain.com krbauth
Checking domain DC=testdomain,DC=local

Registering ServicePrincipalNames for CN=krbauth,OU=Users,DC=testdomain,DC=local
HTTP/websrv.testdomain.com
Updated object

C:\Windows\system32>
Once we have the account and SPN setup we can create a keytab file for our Linux hosts. On the Windows server run the following command: this command is CASE sensitive as are most Kerberos items including the krb5.conf file.
C:\Windows\system32>ktpass /princ HTTP/websrv.testdomain.com@TESTDOMAIN.LOCAL /mapuser krbauth /pass p@ssw0rd /crypto ALL /ptype KRB5_NT_PRINCIPAL/out c:\users\user\desktop\krb5.keytab
Targeting domain controller: DC1.testdomain.local
Successfully mapped HTTP/websrv.testdomain.com to krbauth.
Password succesfully set!
Key created.
Output keytab to c:\users\user\desktop\krb5.keytab:
Keytab version: 0x502
keysize 98 HTTP/websrv.testdomain.com@testdomain.local ptype 1 (KRB5_NT_PRIN
CIPAL) vno 4 etype 0x12 (AES256-SHA1) keylength 32 (0xc29e40406bb1e6aac2428f7f41
a514c8f832940ca2f0ba2742900503a57c9f15)

Once the keytab file is created copy this file via binary means to the Linux host, I use winscp for these types of things. We can test the keytab file by using the following
kinit -k -t /home/psoft/krb5.keytab HTTP/websrv.testdomain.com@TESTDOMAIN.LOCAL
again, use klist and kdestroy to verify and clean up after yourself.

In Part 2 I’ll continue with setting up the web and app server.