The JMS Adapter forJMS documentation can be found here, and the particular part about the properties here.
In essence, in the Assign Values part of the Mediator configuration, custom JMS-properties can be referrered to as
The BPM Process listens to the queue and needed to be adapted to read the JMS-properties. I could not find info about that, but it turns out to be simple: in the ServiceCall activity you have the Service Properties link:
The property name is of the same structure as with the mediator: 'jca.jms.JMSProperty.name'.
The expression must denote an automatically instantiated dataobject of type string. I tried to use a structured business object, but then apparently only the root element is instantiated. On run-time I got errors that the assignment of the JMS-property to the particular element failed. So for each JMS-Property I created a separate Dataobject of type string. And that works perfectly.