Tuesday 29 March 2011

XML to HTML through XSL

I think it was about 2005 or 2006 that we at Oracle had so-called Mudwrestle sessions. Colleagues organized workshops for each other on the latest technologies. We started at 14:00 and after some introduction we did some exercises to get familiar with the subject. Amongst subjects as Integration (InterConnect), java, linux, we had also XML. And there I got to know XSLT for the first time.

To get it "in the fingers", I thought it might be handy to have all my browser links in an xml file and have an xsl attached to have it transformed into html providing them in pop-lists and a button. With a little java script it is then possible to have the link loaded after choosing a link and pressing the button. So I created my first version of the XML and XSL. It was in the time that Internet Explorer 6 was pretty new or at least the current release. I think I was into Firefox already. IE6 and Firefox were able to load and process the xsl file when it was attached to the xml file. To do so you have to add the following tag to the top of your xml file:

<?xml-stylesheet href="links.xsl" type="text/xsl"?>

Like;
<?xml version="1.0"?>
<?xml-stylesheet href="links.xsl" type="text/xsl"?>
<link-lists
  name="Links"
  title="Internet Links"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">Begin van de Lists!
  <lists>
    <list
      name="DarwinApps"
      title="Darwin Links">
      <link
        description="Darwin-IT"
        link="http://www.darwin-it.nl"
        name="Darwin-IT" />
...

The XSL that I created for it is given at the bottom of this post.
Last week I enhanced it a little to have a button to load all the links of a list at once. That is particularly handy for a list of webmail-urls to load at the start of your day.

I'm very fond of this xml/xsl combi. It is so simple, since it consist of three ascii files (xml, xsl and css). The xml file is easy to extend and after a reload in the browser the link-lists are renewed. The xsl is simple to enhance to add extra functionality. And the css makes it possible to have it a layout that complies with your taste or desktop. Skinning was never so easy. And it works in both Windows and Linux. And probably Mac OS. On Windows XP I even had it on my Active Desktop.  But I can't find how to do it on Windows 7 or Linux.

Later in history, also a few years ago (2009) I created a tool in java to edit the xml file. Also this one is enhanced a few times and last week I posted a blog on this tool. See here. I made it xml-parser-vendor-independent. And it's opensource thus you might take a look at the source.

To try the xml/xsl you can open the xml-links file from here. Of course you can download it to your laptop and edit it to add your own links and remove the ones not needed. The xsl is for download here and the css-file here. Place the xml and the xsl in the same folder, the css is expected in a subfolder called "style".

Below is the source of the XSL for investigation. As you can see I set it up in a modular way. I'm used to have a template for every hierarchical level of XML. And for particular functionalities, such as the generation of the java script for each poplist. You can use the xml-tool set in the the earlier post to perform a transform of the XML with the XSL to see the resulting HTML. Have fun with it.

<?xml version="1.0"?>
<!--  (c) 2008-2011, by Martien van den Akker  
Darwin-IT Professionals
-->

<xsl:stylesheet
  version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output
    method="html" />  <!--  Variables  -->

  <xsl:variable
    name="newline">
    <xsl:text></xsl:text></xsl:variable>  <!--  main  -->

  <xsl:template
    match="/">
    <html>

      <xsl:call-template
        name="heading" />
      <body>
        <xsl:call-template
          name="body" /></body></html></xsl:template>
  <xsl:template
    name="heading">    <!--  Heading  -->

    <head>
      <title>Links</title>
      <LINK
        HREF="style/style.css"
        REL="STYLESHEET"
        TYPE="text/css" /></head>    <!--  Script to load a page  -->

    <script
      language="javascript">
      <xsl:text>function loadPage(link)
{
  newWdw=window.open(link,"_blank","");
}</xsl:text></script></xsl:template>
  <xsl:template
    name="body">    <!--  Body  -->

    <body>
      <xsl:apply-templates
        select="/link-lists" /></body></xsl:template>  <!--  /link-lists  -->

  <xsl:template
    match="link-lists"
    name="link-lists">

    <h1>
      <xsl:value-of
        select="@title" /></h1>
    <table>
      <xsl:apply-templates
        select="lists/list" /></table></xsl:template>  <!--  /link-lists/lists/list  -->

  <xsl:template
    match="lists/list"
    name="lists-list">
    <tr>
      <td>
        <xsl:value-of
          select="@title" /></td>      <!--  generate javascript for loading a link  -->

      <xsl:call-template
        name="loadLink" />      <!--  generate javascript for loading a links  -->

      <xsl:call-template
        name="loadAllLinks" />      <!--  generate a poplist and a button  -->

      <xsl:call-template
        name="writePopList" /></tr></xsl:template>  <!--  loadLink: generate a javascript that loads a link  -->

  <xsl:template
    name="loadLink">
    <xsl:text
      disable-output-escaping="yes">      <!--  script to load a link  -->

</xsl:text>
    <script
      language="javascript">
      <xsl:value-of
        select="concat($newline,'      function load', @name,'Link()')" />
      <xsl:text
        disable-output-escaping="yes">{</xsl:text>
      <xsl:value-of
        select="concat('        var l_idx = document.', @name, 'LinkSelector.select.selectedIndex;',$newline)" />
      <xsl:value-of
        select="concat('        var l_value = document.', @name, 'LinkSelector.select.options[l_idx].value;')" />
      <xsl:text
        disable-output-escaping="yes">if (l_value != "none")
        {
           loadPage(l_value);
        }
      }</xsl:text></script></xsl:template>  <!--  loadAllLinks: generate a javascript that loads all links of a poplist  -->

  <xsl:template
    name="loadAllLinks">
    <xsl:text
      disable-output-escaping="yes">      <!--  script to load a link  -->
</xsl:text> 
    <script
      language="javascript">
      <xsl:value-of
        select="concat($newline,'      function loadAll', @name,'Links()')" />
      <xsl:text
        disable-output-escaping="yes">{
        var l_idx;
        var l_value;</xsl:text>
      <xsl:value-of select="concat('        for(l_idx=0; l_idx&lt;document.', @name, 'LinkSelector.select.length; l_idx++)')"/>
   <xsl:value-of select="concat($newline,'        {',$newline)"/>
      <xsl:value-of select="concat('           l_value=document.', @name, 'LinkSelector.select.options[l_idx].value;')"/> 
      <xsl:text
        disable-output-escaping="yes">

    if (l_value != "none")
    {
            loadPage(l_value);
    }
        }
      }</xsl:text></script></xsl:template>
  <xsl:template
    name="writePopList">
    <td>
      <form
        name="{concat(@name, 'LinkSelector')}">
        <table
          border="0"
          cellspacing="0">
          <tr>
            <td
              width="250px">              <!-- Select  -->

              <select
                name="select">
                <option
                  value="none">Kies een link:</option>
                <xsl:for-each
                  select="link">
                  <option
                    value="{@link}">
                    <xsl:value-of
                      select="@name" /></option></xsl:for-each></select></td>
            <td>              <!--  Button  -->

              <input
                onclick="{concat('load',@name,'Link()')}"
                type="button"
                value="Toon" /></td>

            <td>              <!--  Button Load All -->

              <input
                onclick="{concat('loadAll',@name,'Links()')}"
                type="button"
                value="Toon alles" /></td></tr></table></form></td></xsl:template>
</xsl:stylesheet>

Tuesday 22 March 2011

Vendor independent XML processing

A few years ago, when I was "low in work" at the particular customer, I created a little tool set on XML processing. I called it Darwin XML Editor and XML Tester. The latter name I now find not so well choosen. So I would now call it XML Tools.

The purpose of the tool set was to figure out how to process XML in java. I already had a private xml based solution to gather my frequently used browser links into an xml file and convert it using an attached xslt stylesheet to an html page. This html page provided the links in poplists and using java script it could load the link.
It would be nice to have a hierarchical editor that showed my xml in an explorer and let me edit particular attributes in a table format.
 It turned out handy for me. But the most I had a toolkit that allowed me to easily load XML as a text file and parse it, perform xpath queries and just traverse through nodes, etc. In several projects I used it to read xml-based config files. Many of those cases I probably  could have done using Apache-commons.
But it is handy to have your own toolkit. Especially in cases when you have repetitive lists of entities with properties.

My project originally was written using the Oracle XMLParser. Handy, because I use JDeveloper and that comes with the parser. For some time now I had plans to look if I could make it vendor independent.

The results can be found here. It's open source, whatever that may mean. But it would be nice to leave the credit-references in place or refer to the credits. That would be nice for my ego...

I will provide here some highlights of the changes.
I used the following sites as my references:


Parsing

It starts with parsing. The "Oracle way" I did it was:
public void parse() {

        try {
            String text = super.getText();
            if (text != null) {
                resetError();
                DOMParser parser = new DOMParser();
                InputSource inputStream = new InputSource();
                inputStream.setCharacterStream(new StringReader(text));
                parser.parse(inputStream);
                xmlDoc = parser.getDocument();

                xmlRoot = xmlDoc.getDocumentElement();
                nodeSelected = new NodeSelected((Node)xmlRoot);
                parsed = true;
                log("File: " + super.getFilePath() + " is succesfully parsed");
            } else {
                log("File: " + super.getFilePath() + " empty or not loaded!");
            }

        } catch (SAXException e) {
            setErrorCode(EC_ERROR);
            setError("Error parsing XML: " + e.toString());
            error(e);
        } catch (IOException e) {
            setErrorCode(EC_ERROR);
            setError("Error reading XML: " + e.toString());
            error(e);
        }
    }

With the following oracle imports:
import oracle.xml.parser.v2.DOMParser;
import oracle.xml.parser.v2.XMLDocument;
import oracle.xml.parser.v2.XMLNode;
import oracle.xml.parser.v2.XSLException;

The vendor independent way I used is:
/**
     * Create a new DocumentBuilder.
     * @return
     * @throws ParserConfigurationException
     */
    private DocumentBuilder newDocBuilder() throws ParserConfigurationException {
        DocumentBuilderFactory domFactory = 
            DocumentBuilderFactory.newInstance();
        domFactory.setNamespaceAware(true); // never forget this!
        DocumentBuilder docBuilder;
        docBuilder = domFactory.newDocumentBuilder();
        return docBuilder;
    }

    /**
     * Create an InputSource from the xml-text
     * @return
     */
    public InputSource getInputSource() {
        String text = super.getText();
        InputSource inputStream = null;
        if (text != null) {
            inputStream = new InputSource();
            inputStream.setCharacterStream(new StringReader(text));
        }
        return inputStream;
    }

    /**
     * Parse the XML in the file
     */
    public void parse() {
        try {
            InputSource inputSource = getInputSource();
            if (inputSource != null) {
                resetError();
                DocumentBuilder docBuilder = newDocBuilder();
                Document doc = docBuilder.parse(inputSource);
                setDoc(doc);
                parsed = true;
                log("File: " + super.getFilePath() + " is succesfully parsed");
            } else {
                log("File: " + super.getFilePath() + " empty or not loaded!");
            }
        } catch (ParserConfigurationException e) {
            setErrorCode(EC_ERROR);
            setError("Error creating parser: " + e.toString());
            error(e);
        } catch (SAXException e) {
            setErrorCode(EC_ERROR);
            setError("Error parsing XML: " + e.toString());
            error(e);
        } catch (IOException e) {
            setErrorCode(EC_ERROR);
            setError("Error reading XML: " + e.toString());
            error(e);
        }
    }
with the following imports:
import javax.xml.namespace.QName;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;

Mark that this code is part of a XMLFile class that extends a TextFile class that gives me the methods to load the file into a String attribute (text).
Here I extracted the conversion from the text attribute to an InputSource object in a separate method. Also the creation of the parser (the DocumentBuilder) I put in a seperate method. This because I also need it to be able to create an empty Document.

XPath Expressions

The "Oracle way" is pretty straight forward:

/**
     * Select Nodes using Xpath
     * @param xpath
     * @return NodeList 
     */
    public NodeList selectNodes(String xpath) throws XSLException {
        XMLDocument xmlDoc = getXmlDoc();
        NodeList nl = xmlDoc.selectNodes(xpath);
        return nl;
    }
To have it "Namespace Aware" you'll need a Namespace Resolver, which is a simple class based on a HashMap, implementing an Interface:
package com.darwinit.xmlfiles.xml;
/**
 * Namespace Resolver: Helper class to do namespace aware XPath queries. 
 *
 * @author Martien van den Akker
 * @author Darwin IT Professionals
 *
 * @remark: changed to JSE 1.4 code because of BPEL PM 10.1.2
 */
import java.util.HashMap;

import oracle.xml.parser.v2.NSResolver;

public class XMLNSResolver implements NSResolver{
  private HashMap<String, String> nsMap = new HashMap<String, String>();
    
    public XMLNSResolver() {
    }

   public void addNS(String abbrev, String namespace){
       nsMap.put(abbrev, namespace);
   }
    public String resolveNamespacePrefix(String string) {
        return nsMap.get(string);
    }
}
Then using the namespace resolver the code would be something like:

/**
     * Select nodes using Xpath with Namespace included
     * @param xpath
     * @return Nodelist 
     */
    public NodeList selectNodesNS(String xpath) throws XSLException {
    XMLDocument xmlDoc = getXmlDoc();
    XMLNSResolver nsRes = getNsRes();
        NodeList nl = this.xmlDoc.selectNodes(xpath, nsRes);
        return nl;
    }

The vendor indepent way is a little more complex. But it gives you some more flexibility.
/**
     * Evaluate xpath expression 
     * 
     * @param xpathExpr the xpath expression
     * @param returnType the return type that is expected.
     * http://www.ibm.com/developerworks/library/x-javaxpathapi.html:
     * XPathConstants.NODESET => node-set maps to an org.w3c.dom.NodeList
     * XPathConstants.BOOLEAN => boolean maps to a java.lang.Boolean
     * XPathConstants.NUMBER => number maps to a java.lang.Double
     * XPathConstants.STRING => string maps to a java.lang.String
     * XPathConstants.NODE
     * 
     * @throws XPathExpressionException
     */
    public   Object evaluate(String xpathExpr, 
                    QName returnType) throws XPathExpressionException {
        XPathFactory factory = XPathFactory.newInstance();
        XPath xpath = factory.newXPath();
        XMLNSResolver nsRes = getNsRes();
        if (nsRes != null) {
            xpath.setNamespaceContext(nsRes);
        }
        XPathExpression expr = xpath.compile(xpathExpr);
        Document doc = getDoc();
        Object resultObj = expr.evaluate(doc, returnType);
        return resultObj;
    }

    /**
     * Evaluate xpath expression to a double (when a number is expected from the xpath expression)
     * 
     * @throws XPathExpressionException
     */
    public Double evaluateDouble(String xpathExpr) throws XPathExpressionException {
        Double result = null;
        Object resultObj = evaluate(xpathExpr, XPathConstants.NUMBER);
        if (resultObj instanceof Double) {
            result = (Double)resultObj;
        }
        return result;
    }

    /**
     * Select Nodes using Xpath
     * 
     * @param xpath
     * @return NodeList
     * @throws XPathExpressionException 
     */
    public NodeList selectNodes(String xpath) throws XPathExpressionException {
        NodeList nl = (NodeList)evaluate(xpath, XPathConstants.NODESET);
        return nl;
    }
The Namespace resolver changed slightly. It actually implements another interface:
But the idea is the same:
package com.darwinit.xmlfiles.xml;

import java.util.HashMap;

import java.util.Iterator;

import javax.xml.namespace.NamespaceContext;


/**
 * Namespace Resolver: Helper class to do namespace aware XPath queries. 
 *
 * @author Martien van den Akker
 * @author Darwin IT Professionals
 *
 */
public class XMLNSResolver implements NamespaceContext {
    private HashMap<String, String> nsMap = new HashMap<String, String>();
   /**
     * Constructor
     */
    public XMLNSResolver() {
    }

    /**
     * Add a Namespace
     * @param prefix
     * @param namespace
     */
    public void addNS(String prefix, String namespace) {
        nsMap.put(prefix, namespace);
    }

    /**
     * Resolve a namespace from a prefix
     * @param prefix
     * @return
     */
    public String resolveNamespacePrefix(String prefix) {
        return nsMap.get(prefix);
    }

    /**
     * Resolve a namespace from a prefix
     * @param prefix
     * @return
     */
    public String getNamespaceURI(String prefix) {
        return resolveNamespacePrefix(prefix);
    }

    /**
     * Get the prefix that is registered for a NamespaceURI 
     * However, not necessary for xpath processing
     * @param namespaceURI
     * @return
     */
    public String getPrefix(String namespaceURI) {
        throw new UnsupportedOperationException();
    }

    /**
     * Get an iterator with the prefix registered for a NamespaceURI
     * @param namespaceURI
     * @return
     */
    public Iterator getPrefixes(String namespaceURI) {
        return null;
    }
}


The differences are in the fact that you first have to 'compile' an xpath expression. And that the evaluation of the XPath on a Document expects you to provide the expected result datatype. (see the comments of the evaluate method).
Then it will result an Object that you have to cast to the particular Java Class that corresponds with the Result XML DataType.

XSLT

The last thing is transforming XSLT. The "Oracle Way" I used was:

package com.darwinit.xmlfiles.xml;

import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Hashtable;

import oracle.xml.parser.v2.XMLDocument;
import oracle.xml.parser.v2.XSLException;
import oracle.xml.parser.v2.XSLProcessor;
import oracle.xml.parser.v2.XSLStylesheet;


public class XmlTransformer {
    private void pl(String text) {
        System.out.println(text);
    }

    public XmlTransformer() {
    }

    public String transform(XMLDocument xslDoc, XMLDocument xmlDoc) {     
     return transform (xslDoc, xmlDoc, null);
    }    
    
    /**
     * Transform the xmlDoc using xslDoc into String
     * @param xslDoc
     * @param xmlDoc
     * @param parameters contains parameter that are passed to the xslt
     * @return String output
     */
    public String transform(XMLDocument xslDoc, XMLDocument xmlDoc
                           , Hashtable<String, String> parameters) {
        XSLProcessor xslProcessor = new XSLProcessor();
        XSLStylesheet xslt;
        String result = "";
        try {
            StringWriter sw = new StringWriter();
            PrintWriter pw = new PrintWriter(sw);
            xslt = xslProcessor.newXSLStylesheet(xslDoc);
            
            xslProcessor.setXSLTVersion(XSLProcessor.XSLT20);
            xslProcessor.showWarnings(true);
            xslProcessor.setErrorStream(System.err);
            
            if (parameters != null) {
            
             for (String key : parameters.keySet()) {
              String value = parameters.get(key);
              xslProcessor.setParam("", key, value);
             }
            }                      
            
            xslProcessor.processXSL(xslt, xmlDoc, pw);
            pw.flush();
            pw.close();
            sw.close();
            result = sw.toString();
        } catch (XSLException e) {

            pl(e.toString());
        } catch (IOException e) {
            pl(e.toString());
        }
        return result;
    }   
    /**
     * Clean up text from XML Leftovers
     * @param text
     * @return
     */
    public String cleanText(final String text){
        String result = text;
        result = result.replaceAll("&lt;","<");
        return result;
    }
    /**
     * Transfomr the xmlDoc using xslDoc into String
     * Cleanup XML code leftovers
     * @param xslDoc
     * @param xmlDoc
     * @return String output
     */
    public String transform2Ascii(XMLDocument xslDoc, XMLDocument xmlDoc) {
      String result = transform(xslDoc, xmlDoc);
      result = cleanText(result);
      return result;
    }

}

And the vendor indepent version of the same class:
package com.darwinit.xmlfiles.xml;

import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.Hashtable;

import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;

import org.w3c.dom.Document;

import com.darwinit.xmlfiles.log.LocalStaticLogger;

/**
 * XmlTransformer: Class implementing functionality to Transform XML using XSLT.
 * 
 * See also http://www.ling.helsinki.fi/kit/2004k/ctl257/JavaXSLT/Ch05.html
 * 
 * @author Martien van den Akker
 * @author Darwin IT Professionals
 */

public abstract class XmlTransformer {
 public static final String className = "XmlTransformer";
 private static LocalStaticLogger lgr;

 /**
  * Transform the xmlDoc using xslDoc into String
  * 
  * @param xslDoc
  * @param xmlDoc
  * @return
  */
 public static String transform(Document xslDoc, Document xmlDoc) {
  return transform(xslDoc, xmlDoc, null);
 }

 /**
  * Transform the xmlDoc using xslDoc into String, with parameters
  * 
  * @param xslDoc
  * @param xmlDoc
  * @param parameters
  *            contains parameter that are passed to the xslt
  * @return String output
  */
 @SuppressWarnings("static-access")
 public static String transform(Document xslDoc, Document xmlDoc,
   Hashtable<String, String> parameters) {
  final String methodName = "transform";
  lgr.logStart(className, methodName);
  String result = "";
  try {
   DOMSource xsltSource = new DOMSource(xslDoc);
   DOMSource xmlSource = new DOMSource(xmlDoc);
   StringWriter sw = new StringWriter();
   PrintWriter pw = new PrintWriter(sw);
   StreamResult streamResult = new StreamResult(pw);
   // Get the transformer factory
   TransformerFactory transFact = TransformerFactory.newInstance();
   // Get a transformer for this particular stylesheet
   Transformer trans = transFact.newTransformer(xsltSource);
   // Add parameters
   if (parameters != null) {
    for (String key : parameters.keySet()) {
     String value = parameters.get(key);
     trans.setParameter(key, value);
    }
   }
   // Do the transformation
   trans.transform(xmlSource, streamResult);
   pw.flush();
   pw.close();
   sw.close();
   // Get the result string
   result = sw.toString();
  } catch (IOException e) {
   lgr.error(className, methodName, e);
  } catch (TransformerException e) {
   lgr.error(className, methodName, e);
  }
  lgr.logEnd(className, methodName);
  return result;
 }

 /**
  * Clean up text from XML Leftovers
  * 
  * @param text
  * @return
  */
 public static String cleanText(final String text) {
  String result = text;
  result = result.replaceAll("&lt;", "<");
  return result;
 }

 /**
  * Transform the xmlDoc using xslDoc into String Cleanup XML code leftovers
  * 
  * @param xslDoc
  * @param xmlDoc
  * @return String output
  */
 public static String transform2Ascii(Document xslDoc, Document xmlDoc) {
  String result = transform(xslDoc, xmlDoc);
  result = cleanText(result);
  return result;
 }

}
The code is not much different. Most important is that you have to wrap the input, xslt and result objects into particular interface objects. The examples I found worked with File(s) as Source and Result objects. But I wanted Document objects for the XLST and Input objects. And I want to have the result into a String, to be able to process it in anyway I want.

I added also the possiblity to pass XSLT parameters. But I got it from a project I worked on and see that I haven't put that in the class in my XmlFiles project.

Conclusion

I hope this helps in understanding how to work with XML parsers. Ofcourse all is rudimentary (I have several large books on the subject in the cupboard). But I have pretty much enough with the above. The code in my XmlFiles project is basically around this code. On top of these methods I have several other helper methods to do traverse nodes or to get it in a particular way.

I thought it might be helpfull to put the Oracle Native methods side by side with the vendor-independent (JAXP) code. Doing so I would not state that one way is better than the other. The vendor-independent code have clearly the advantage that you can simply replace the parser, just by changing the class path. The code will work with the Oracle XML Parser just as good as with the Apache Xerces-J parser.
The reason I did it the Oracle way before was just because I ran into these examples first, the time I started this. Probably there still are good reasons to use the native API's.

Oracle also has a Pl/Sql Wrapper around the XML Parser. So the Oracle XML Parsing code has a Pl/Sql counter part. Might be nice to do this in a Pl/Sql way sometime.
But as the code above isn't rocket science, it isn't new also. XML Parsing in Pl/Sql can be done from Oracle 8i onwards. Also roughly ten years already.

Thursday 17 March 2011

Query ClearQuest with Java.

At deployment of a new release of a project it might be required to deliver a list of solved issues in a release notes document. At least that was a requirement that my colleagues on my current project had.
They copied and pasted the issues from the Rational ClearQuest tool, which they felt was error-prone and a tedious job. If I could create a tool for it?

The first thing I did was to look into ClearQuest on how the import and export possibilities were. But although there is an export tool, it is not batch. It'll start a wizard that you have to go through. Within ClearQuest there are query possibilities. But then again: you're in the tool and we wanted to have it batch-wise, integrated with Ant.
So it would be nice to be able to query ClearQuest in Java. And fortunately I found a cqjni.jar in the ClearQuest home. This jar is a Java Native Interface set of API's on CQ.

There are, however, very few java examples. I did find an example on perl here. I could use this with some other examples to come up with my own code. So I'll lay out my steps here.

0. Some helper methods

First I'll give you some helper methods that I use in the following examples. They're in fact shortcuts to "System.out.println".

/**
  * Shortcut for output
  * 
  * @param text
  */
 public static void pl(String text) {
  System.out.println(text);
 }

 public static void pl(String text, Exception e) {
  System.out.println(text);
  e.printStackTrace();
 }

 /**
  * Output a String array.
  * 
  * @param label
  * @param array
  */
 public static void printStringArray(String label, String[] array) {
  for (int idx = 0; idx < array.length; idx++) {
   pl(label + " " + idx + ": " + array[idx]);
  }
 }

1. Create session

The first  step to do is to create a CQSession:
pl("Begin");
CQSession cqSession = new CQSession();

2. List Installed Database Sets and Master Databases

To be able to query on CQ you'll have to find out what databases you can connect to. In CQ apparently databases are grouped in a DatabaseSet. And a DatabaseSet is correlated to a MasterDatabase. You can query them as follows.
CQSession session = getCqSession();
  String[] dbSets = session.GetInstalledDbSets();
  String[] masters = session.GetInstalledMasterDbs();
As I understand it, a DabaseSet is put in a database. In my setup I've two masterdatabases and a DatabaseSet in each. I assumed that the entries in the String Arrays correspond to each other: dbSets[0] correlates to masters[0], etc.

I combined the lot in a some combined methods. These are based on a helper Bean DatabaseSet.
/**
  * Get Installed ClearQuest Database Sets
  * 
  * @return
  * @throws CQException
  */
 public String[] getInstalledDbSetNames() throws CQException {
  CQSession session = getCqSession();
  String[] dbSets;
  dbSets = session.GetInstalledDbSets();
  return dbSets;
 }

 /**
  * Get Installed ClearQuest Master Databases
  * 
  * @return
  * @throws CQException
  */
 public String[] getInstalledMasterDbNames() throws CQException {
  CQSession session = getCqSession();
  String[] masters = session.GetInstalledMasterDbs();
  return masters;
 }

 /**
  * Get Installed ClearQuest Database Sets
  * 
  * @return
  */
 public Vector<DatabaseSet> getInstalledDatabaseSets() {
  final String methodName = "getInstalledDatabaseSets";
  Vector<DatabaseSet> dbSets = new Vector<DatabaseSet>();
  try {
   String[] dbSetNames = this.getInstalledDbSetNames();
   String[] masters = this.getInstalledMasterDbNames();
   for (int i = 0; i < dbSetNames.length; i++) {
    DatabaseSet dbSet = new DatabaseSet(masters[i], dbSetNames[i]);
    dbSets.add(dbSet);
   }
  } catch (CQException e) {
   pl("Exception at " + methodName, e);
  }

  return dbSets;
 }
 /**
  * List Installed ClearQuest Database Sets
  * 
  * @return
  */
 public void listInstalledDbSets() {
  final String methodName = "listInstalledDbSets";
  pl(methodName);
  Vector<DatabaseSet> dbSets;
  dbSets = getInstalledDatabaseSets();
  ListIterator<DatabaseSet> it = dbSets.listIterator();
  while (it.hasNext()) {
   DatabaseSet dbSet = it.next();
   pl(dbSet.toString());
  }
 }

3. Accessible Databases

To list the Accessible Databases I used the code that I found here. In my own code I moved some of the code to the methods above.

/**
  * List Accessible ClearQuest Databases
  * 
  * @return
  */
 public void listAccessibleDatases() {
  final String methodName = "listInstalledDatabaseSets";
  pl(methodName);
  Vector<DatabaseSet> dbSets;
  dbSets = getInstalledDatabaseSets();
  ListIterator<DatabaseSet> it = dbSets.listIterator();
  while (it.hasNext()) {
   try {
    DatabaseSet dbSet = it.next();
    pl("Handling dbSet: " + dbSet.getName());
    CQSession session = getCqSession();
    CQDatabaseDescs dbDescriptors = session.GetAccessibleDatabases(
      dbSet.getMasterDb(), "", dbSet.getName());
    int n = (int) dbDescriptors.Count();
    System.out.println(" Accessible database count: " + n);
    for (int j = 0; j < n; j++) {
     CQDatabaseDesc desc = dbDescriptors.Item(j);

     pl(" Database " + j + ": " + desc.GetDatabaseName());
    }
   } catch (CQException e) {
    pl("Exception at " + methodName, e);
   }
  }
 }


4. Connect to a database

If you know what database you want to connect to, you can logon:
pl("Logon to ClearQuest");
   cqSession.UserLogon("User", "Password", "CQ-database", "CQ-DatabaseSet");
   setCqSession(cqSession)

5. List Record Types

Record types are sort of the logical-entities in ClearQuest. You have to know the entity in which your issues are stored. To list the possible Record Types you'll have to connect to the database (see above). Then you can list the record types as follows:
/**
  * List the possible RecordTypes
  * 
  * @return
  * @throws CQException
  */
 public void listRecordTypes() {
  final String methodName = "listRecordTypes";
  pl(methodName);
  try {
   CQSession session = getCqSession();
   String[] cqRecordTypes = session.GetEntityDefNames();
   printStringArray("Record Type", cqRecordTypes);
  } catch (CQException e) {
   pl("Exception at " + methodName, e);
  }
 }
The same way you can find so-called Family Types. I did not use them further.
/**
  * List the possible Family Types
  */
 public void listFamilyTypes() {
  final String methodName = "listFamilyTypes";
  pl(methodName);
  try {
   CQSession session = getCqSession();
   String[] cqFamilyTypes = session.GetEntityDefFamilyNames();
   printStringArray("Family Type", cqFamilyTypes);
  } catch (CQException e) {
   pl("Exception at " + methodName, e);
  }

 }

6. Build a query

Now it's time to build the query. If you've build a query before in the CQ tool you'll probably be familiar with the concepts. It starts with creating a query and add columns to it:
CQQueryDef cqQueryDef = cqSession.BuildQuery("BPM");
   cqQueryDef.BuildField("id");
   cqQueryDef.BuildField("State");
   cqQueryDef.BuildField("headline");
   cqQueryDef.BuildField("Owner");
   cqQueryDef.BuildField("Fix_By");
The parameter in the BuildQuery method is the RecordType that you can query with the code in previous paragraph. In this example the record type is "BPM". The field-names are those defined in the tool on the record type. So it might be handy to build your query first in the CQ-Tool, to know which attributes/columns are to your disposal.

7. Create a filter

Of course you don't want the whole lot of issues. In a longer running project there can be thousands of them. So we'll need to add a filter. In the ClearQuest Tool you can have a hierarchy of so called filter nodes. A filter node has several expressions that are concatenated using And or Or operators. Such an expression in itself can be a other filter node or what I call a FilterField: a filter on a field.
pl("Query Issues");
   CQQueryFilterNode cqQueryFilterNode = cqQueryDef
     .BuildFilterOperator(BOOL_OP_AND);
   String[] stateValues = { "Closed", "Resolved" };
   cqQueryFilterNode.BuildFilter("State", COMP_OP_EQ, stateValues);
   String[] branchValues = { "R1.%" };
   cqQueryFilterNode.BuildFilter("Fix_by", COMP_OP_LIKE, branchValues);
Here you see that I use some constants. For the FilterOperators the possible values are:
// And operator
 public static final long BOOL_OP_AND = 1;
 // Or operator
 public static final long BOOL_OP_OR = 2;

For the Filter "Comparators" the following values are possible:
// Equality operator (=)
 public static final long COMP_OP_EQ = 1;
 // Inequality operator (<>)
 public static final long COMP_OP_NEQ = 2;
 // Less-than operator (<)
 public static final long COMP_OP_LT = 3;
 // Less-than or Equal operator (<=)
 public static final long COMP_OP_LTE = 4;
 // Greater-than operator (>)
 public static final long COMP_OP_GT = 5;
 // Greater-than or Equal operator (>=)
 public static final long COMP_OP_GTE = 6;
 // Like operator (value is a substring of the string in the given field)
 public static final long COMP_OP_LIKE = 7;
 // Not-like operator (value is not a substring of the string in the given
 // field)
 public static final long COMP_OP_NOT_LIKE = 8;
 // Between operator (value is between the specified delimiter values)
 public static final long COMP_OP_BETWEEN = 9;
 // Not-between operator (value is not between specified delimiter values)
 public static final long COMP_OP_NOT_BETWEEN = 10;
 // Is-NULL operator (field does not contain a value)
 public static final long COMP_OP_IS_NULL = 11;
 // Is-not-NULL operator (field contains a value)
 public static final long COMP_OP_IS_NOT_NULL = 12;
 // In operator (value is in the specified set)
 public static final long COMP_OP_IN = 13;
 // Not-in operator (value is not in the specified set)
 public static final long COMP_OP_NOT_IN = 14;


8. Execute the query and process the rows

Having built the query, it can be executed. First you'll have to create a CQResultSet from the session based on the Query Definition.
The query can then be executed and the result set can be processed using the MoveNext() method. The MoveNext() gives a status. As long as the status equals 1 there is a next record. With GetNumberOfColumns you can count the number of columns in the resultset. And with GetColumnLabel and GetColumnValue you can get the Column name and value at a particular index. Of course you know what columns you asked for and in what order. But traversing over the available columns is handy, because adding a column to the query does not affect this part of the code. Also it enables you to search for the value of a particular column.

CQResultSet cqResultSet = cqSession.BuildResultSet(cqQueryDef);
   long count = cqResultSet.ExecuteAndCountRecords();
   pl("count: " + count);
   long status = cqResultSet.MoveNext();
   while (status == 1) {
    pl("Status: " + status);
    for (int i = 1; i <= cqResultSet.GetNumberOfColumns(); i++) {
     pl(cqResultSet.GetColumnLabel(i) + ": "
       + cqResultSet.GetColumnValue(i));
    }
    status = cqResultSet.MoveNext();
   }
   pl("Status: " + status);
   pl("Done");

Conclusion

That's it. The example-snippets above come from a test class I put together and that can be  downloaded form here. It's a test class that I created from other examples and refactored somewhat. For this blog I anonymised it and rearranged some of the methods. In my application I refactored the class into seperate classes for the dbconnection, a query class and an export class. Of course that's what you do in a project, to make it neat. But in the basics it runs down to this particular example. So I kept it in my project to first pilot a new feature.

Then I created some ant tasks and targets to do the export and run Word with a document and a macro to import the file. But that's another story...

Thursday 10 March 2011

Total Commander for Linux=>Krusader

Oh, how I love Total Commander. I never found a file-manager that is so powerful. And how I missed that under Linux. I work a few years with Linux now, but on my Linux I still have TotalCommander. It can browse through archives, it can FTP and it has tabs to quickly switch locations. And I find copying/moving using to panes much more convenient than the Windows Explorer way. I kind of hate Windows Explorer. KDE-Dolfin is better, but to me still too much Windows Explorer.

Today I looked at the Total Commander website, to see that there is a new version of it. I must admit that I have a new laptop with Windows 7/64bit. And one of the minor things that I miss in TC is that it's still 32 bit. It's not big of a problem, but in the context menu all the 64 bit tools (like svn) are under a sub-menu. That's a little inconvenient.

In the FAQ I saw that a Linux version of TC is postponed indefinitely. But that there is one recommended tool: Krusader. That I missed that until now! I installed it right away and at first glance indeed it is the first tool that I find comparable to TC. Tools like midnight commander and under Windows FreeCommander or other lookalikes aren't for me.

TC works under Wine, but not too optimal. Krusader however is native Linux. So I'm going to check that out. Just another reason to tempt me install OpenSuse over my Windows 7 system...

Thursday 3 March 2011

Connection errors between JDeveloper 11g and OTN SoaBPM Appliance

I' m working with the "Pre-built Virtual Machine for SOA Suite and BPM Suite 11g" for a while now.
I use it to prepare for SoaSuite and BPM Bootcamps and one thing I try to do is to use JDeveloper  11g on my Host  (OpenSuse 11.2) and connect to the application server in my VirtualBox guest. In fact, that would (probably) be the architecture to use in a real developer environment.

So I changed the network adapter of the VirtualBox VM (in Device Settings) from NAT to "host-only". Then I re-activated the network-adapter in the guest (Oracle Enterprise Linux), to have it query for new ip-settings.As root I queried for the ip-address using ' ifconfig':
[root@soabpm-vm ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:1C:4C:37
          inet addr:192.168.56.101  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe1c:4c37/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:219 errors:0 dropped:0 overruns:0 frame:0
          TX packets:192 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:21908 (21.3 KiB)  TX bytes:26686 (26.0 KiB)
          Interrupt:10 Base address:0xd020

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2222 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2222 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2942937 (2.8 MiB)  TX bytes:2942937 (2.8 MiB)

So the ip-address of my vm is '192.168.56.101'.

Then I changed the /etc/hosts file of the VM as:
127.0.0.1       localhost
192.168.56.101  soabpm-vm soapbm-vm.local bpm-vm.local bpm-vm
::1             localhost6.localdomain6 localhost6

And I added the same address in the host file of my OpenSuse host:
 192.168.56.101  soabpm-vm soabpm-vm.darwin-it.local

In Jdeveloper 11g I created an application server connection, named "soabpm-vm', added the credentials (weblogic/welcome1), hostname: soabpm-vm, ports 7001/7002, domain name: domain1. And then we test:

Fine. So in my BPM Bootcamp preperations I came to the point I have to map the abstract roles (swimlanes) to LDAP roles/users. Doing so you have to query the users over the application server connection. Choosing the connection it would populate the realm-poplist. In my case it would not.Testing in the JDeveloper within the VM would give me:
Mark the Realm "jazn.com" automatically populated.

So I tried to deploy a SoaProject, that builds fine, but deploying resulted in:

[03:10:41 PM] Error sending deployment request to server AdminServer [localhost:7001]  
 java.net.ConnectException: Connection refused 
[03:10:41 PM] Error sending deployment request to server AdminServer [localhost:7001]  
 java.net.ConnectException: Connection refused  

After a while I noticed 'AdminServer [localhost:7001]'! That's not where I want to connect to! And it is certainly not the host I applied in the Jdev Applicatioin Server dialog.

Apparently the AdminURL of Weblogic is not correctly set during startup. This is set in  the config.xml.
This can easily be found as follows:
[oracle@soabpm-vm config]$ cd
[oracle@soabpm-vm ~]$ cd bin
[oracle@soabpm-vm bin]$ . ./wls_env.sh
[oracle@soabpm-vm bin]$ cd $DOMAIN_HOME/config
[oracle@soabpm-vm config]$ ls config.xml
config.xml
Edit the config.xml and look for:
<external-dns-name>localhost</external-dns-name>
Then change it in:
<!--<external-dns-name>localhost</external-dns-name>-->
<external-dns-name>soabpm-vm</external-dns-name>

As always it is good practice to backup the config.xml. And after the change of course you'll need to restart the weblogic server.
But then the connection to the AdminServer will work, the realm can be queried and deploying will work.
Important lesson: the hostname in the setting above should not be localhost, but a meaningful host name.that can be resolved on both the SoaSuite server as well as the Client running JDeveloper.

Lightning on Thunderbird 3.x X86_64

I used to use the Mozilla Calander plugin for Thunderbird: Lightning. Haven't used it for long, but today I found that it would be nice to have it back again. I found that it was on my thunderbird plugins but it was not compatible anymore with my version of Thunderbird.

Since Lightning is architecture specific you need the 64 bit version of it. That is: if you use the 64-bit Linux Thunderbird. It is available but for some unknown reason Mozilla did not publish it. Via the Ubuntu site (thanks to the Ubuntu Community) I found the location of the 1.0b2 beta version (for Thunderbird 3.1.x) here. The 1.0b1 version (for Thunderbird 3.0.x) here. Since it is version specific, with new versions you can try it with changing the version id in the url.

Download it, together with the accompanying Google Calendar provider, and install it in Thunderbird.

Tuesday 1 March 2011

VirtualBox on Windows XP-2

I thought it was quite simple to increase the VM-memory beyond the 1500MB limit of the GUI.
However, starting the SOABPM server in the OTN SOABPM appliance would certainly crash the VM. Very nice is the state VirtualBox will bring the VM in: the "Guru Mediation"-Mode.
Well, I would not call myself a VirtualBox-guru, so I wouldn't mediate in why the VM didn't feel as comfortable as it should.  So turn-off the machine.

It turns out that the the 1500MB limit is not as arbitrary as thought. In the Issue Ticket here I read that the VMM process places the complete VM's memory  in the process-memory space. In 32-bit Windows the maximum of the process-address space is 2GB.And 2GB for the kernel. Since there is addresspace needed for the VMM process, the video memory etc., the limit for the VM is set to 1500MB.
I would think that the limit is some further on the "save side", possibly something like 1750MB would do either.

Since I'm a pretty stubborn guy, I'll try 2000MB.  Also to see the "Guru Mediation" mode state again... Then I'll try some saver values.

VirtualBox on Windows XP

Today I installed VirtualBox 3.2.12 on a Windows XP. No not 4.0, since that was the version I happened to have with me on my external HD.

I imported the Technet SOABPM Appaliance, but I found that Virtual Box places the disks and machines in a hidden ".VirtualBox" folder within my "Documents and Settings" area. Since on this machine I have a roaming profile I don't want to have folders there with gig's on virtual disks. Logging on Windows with a roaming profile is already slow without this.

So I moved the ".VirtualBox" folder to a "C:\Data\VirtualBox" folder. Then VirtualBox won't find your machine anymore. To solve that a few things are to be done.

First change the defaults paths. In VB Go to <menu>/File/Preferences:


There you can change the paths to "C:\Data\VirtualMachines\VirtualBox\HardDisks" and "C:\Data\VirtualMachines\VirtualBox\Machines".  I differentiate my machines with a VirtualBox subfolder, since I also use VMWare Player.

Then you can make a new VMware image based on the disks. To get the new machine have the same name as the old one I backed up the existing machine directory ("c:\Data\VirtualMachines\VirtualBox\Machines\vbox-oel5u4-soabpm-11gr1ps2-bp1-otn\").

Then I create the machine with the "do not think for yourself, let's do that later"-default settings.
After creating the machine with the two disks, I did a file compare with the old and new "vbox-oel5u4-soabpm-11gr1ps2-bp1-otn.xml" file in the machine folder.

I copy and paste the diffent settings using the file compare tool of TotalCommander in Edit-mode:
  • Copy the <description> node
  •  <memory pagefusion="false" ramsize="2500"/>
    Under windows XP (with 4GB, effective 3,5 GB memory) VirtualBox does not allow the setting higher then 1500MB. But editing it on file-level allows you to raise it beyond that limit. The appliance was made with 2048. This is pretty little for SoaSuite11g. But not to eat too much from Windows I gave it 2500.
  • Copy the <guestproperties> node
  • Important: change the SATA-controller to:  <storagecontroller name="SCSI Controller" portcount="16" type="LsiLogic" usehostiocache="true"/>
  • I changed some other hardware settings from false to true based on the file compare. Eg. HardwareVirtEx.
Then it turns out that the changes did not reflect in VirtualBox. A refresh did not help. To force  VirtualBox to reread the settings, Killed the VirtualBox service with ProcessExplorer:
That did the trick. Also I have it now running with 2500MB!