ServiceAPI Bulk Loader

The ServiceAPI exposes a bulk loading capabilities for which a C# sample is provided in the help.  This blog post shows the actual JSON that is sent over the wire.

The first step is to create an Origin object in the client, once this is done a batch of Records can be posted to the bulk loader.  The sample below:

  • uses the Origin with Uri 9000000006,
  • does not use background processing (processing is completed within the HTTP post request),
  • creates one Location and one Record,
  • uses the Location created for the Record assignee,
  • assumes that the file "test doc.docx" has already been uploaded (using the UploadFile service), and
  • should be posted to http://MyServer/ServiceAPI/BulkLoader.
{
    "Origin": {
        "Uri":9000000006
    },
    "AutoCommitLocations": true,
    "SQLCheckConstraints": false,
    "SQLServerTableLock": true,
    "UseBulkLoaderRecordNumbering": true,
    "ProcessInBackground":true,
    "LocationsToImport":[{
        "LocationSurname":"Jones", 
        "LocationGivenNames":"Arthur",
        "LocationTypeOfLocation":"Person"
    }],
    "RecordsToImport": [
        {
            "RecordTitle": "My title from BL 5.1 AAAAA",
            "RecordAssignee":{
                "LocationSurname":"Jones", 
                "LocationGivenNames":"Arthur"
            },
            "RecordFilePath":"test doc.docx"
        }
    ]
}

Background processing

If 'ProcessInBackground' is set to true then the ServiceAPI will hand the bulk loading request off to the TRIMServiceAPIBulkLoader service (installed as a windows service).  In this case the bulk loading request will immediately return an OriginHistory object.  The Uri from this object can be used to poll the OriginHistory object to find out when the bulk loading operation has completed.  The Url to fetch the OriginHistory will look like this:

http://localhost/ServiceAPI/OriginHistory?q=uri:9000000019&properties=BulkLoaderIsRunning,OriginHistoryRecordsCreated

Demo

In the video below I demonstrate bulk loading one Record using background processing.

Recursive searching in WebDrawer and ServiceAPI

The CM search syntax allows for recursive searching, for example returning results from folders contained within folders.  The operator that specifies recursion is the plus sign (+), the problem is that in a URL the plus sign is reserved to indicate white space.

To overcome this the ServiceAPI (and WebDrawer) use the tilde (~) instead of the plus sign, so instead of this:

container:[title:asia]+

Do this:

container:[title:asia]~

If you would rather use a character (or sequence of characters) other than the tilde you can customise this in the hptrim.config.  The Web Client, for example, uses _$_ instead of tilde, which is set in the  'searching' element in its hprmServiceAPI.config.

<searching pageSize="30" searchRecursiveOption="_$_"/>

The 'All' property set in the ServiceAPI

Overview

In the ServiceAPI there is a concept of a propertySet, this allows for the retrieval of a pre-defined set of properties, for example:

/Record?q=all&propertySets=Grid&format=json

The problem with 'All'

One of the property sets is 'All' which means that the above URL could look like this:

/Record?pageSize=20&q=all&propertySets=All&format=json

Fetching all properties for many object types (such as Classification, SavedSearch, Activity etc) can be a good idea, being simpler than pre-determining which properties you need.  Using 'All' for Records is problematic, two problems being:

  • some Record properties are very expensive so response times will be unnecessarily slow, and
  • some properties will throw an error if the related feature is not enabled (e.g. NextTaskDue requires the Vital Records feature be enabled).

The solution

The solution is not to use 'All'. The alternatives are:

  • use the 'ViewPane' property set instead, which returns a list of all properties that might be interesting to an end user, or
  • construct a custom property set in hptrim.config which can contain all the properties that are of interest to your application.

Creating a child object (Record 'Alt in' Relationship)

The code

To add a new object to one of the child object collections on a TrimMainObject, for example adding a RecordRelationship to a Record, requires sending an array containing one or more objects to the appropriate collection. For Example, posting the code below to the Record endpoint will cause Record 9000000002 to be 'Alternately contained within' Record 9000008050.

{
    "Uri":9000000002, 
    "ChildRelationships":[
    {
        "RecordRelationshipRelatedRecord":9000008050,
        "RecordRelationshipRelationType": "IsAltIn"
    }]
}

And a video...

Using a search query instead of a Uri in a URL

Recently I wanted to link to a Record using the external ID, not the Uri or the number.  This was from a customised WeDrawer page so I could have done a search to find the Uri and then used that, but I didn't want that extra step.  A little reflection (and poking around in the code) and I realised that I could use the external id in the URL after all.

If it was a search

A search by external id would look like this, problem was I want to open the actual record page, not a list of one record.

/WebDrawer/record?q=recExternal:abc123

As a URL

As long as a search returns only one result then you can substitute the query for the Uri in the URL, as long as you can compose the query without using a colon (colons are reserved characters in URLs).  So below I use equals instead of colon to either open the details page or go directly to the document via the external id.

http://localhost/WebDrawer/record/recExternal=abc123
http://localhost/WebDrawer/record/recExternal=abc123/file/document

Note

As is my habit I use the internal name for the search clause (recExternal) not the name (external) to minimise the possibility of breakages due to either custom captions or use of different languages.

SAML signout

Overview

In a previous post I show how to use Component Space to add SAML support to the Web Client, here I add a sign-out button.

Steps

useADFS == true

Set useADFS to true in hprmServiceAPI.config, for example:

<setup databaseId="J1"  searchAhead="false" advancedSearch="false" workpath="C:\HP Records Manager\ServiceAPIWorkpath\Uploads" useADFS="true"/>

Setup signing certificate

If you plan to sign your logout request then you will need a certificate, either encrypt the password or store the certificate in the Windows certificate store as described in this Component Space document.

Add keep-alive

To allow for notification when the user's session has expired add the keep alive loop.  To do this edit _Initialisation.cshtml (or _Shared.cshtml in later versions) and comment out the RMStayALive function, then add this new version:

var RMStayALive2 = function () {
    var makeRequest = function () {
        $.getJSON(HP.HPTRIM.TrimClient.getServiceAPIUrl() + "/Location/me", function (data, status, xhr) {
                
        }).fail(function () { 
            if (confirm("Your session has expired, do you wish to re-authenticate?")) {
                top.location = HPRMWebConfig.virtualDirectory;
            }
            
        });
    }
    setInterval(makeRequest, (60 * 1000));

}();

Video

The process above is shown in this video.

Java ServiceAPI and additional fields

Previous posts have described the Java client and file upload.  This post details how to fetch the value of additional fields.

Field Names

As described elsewhere the name to use to fetch an additional field is its search clause name, as seen in the client.

adf.PNG

Get field values as text

Once you know the correct names for your fields you need to fetch them.  A full sample is provided in the samples repo, in short you need to:

  1. request the fields by name
  2. request string values be returned
  3. parse the string value as necessary.

Example

See the samples repo for the full sample. 

Records request = new Records();
request.q = "number:rec_222";
// tell the search request to return both string and actual values for the fields and properties
request.setPropertyValue(PropertyType.Both);
request.setStringDisplayType(StringDisplayType.WebService);     

request.Properties = makePropertyList("RoadSurface", "DateOfIssue", "PoliceOfficer");

RecordsResponse response = client.get(request);
Record rec = response.Results.get(0);

ITrimProperty dateOfIssue = rec.Fields.get("DateOfIssue");

SimpleDateFormat parser = new SimpleDateFormat("EEE, d MMM yyyy HH:mm:ss zzz");
Date date = parser.parse(dateOfIssue.getStringValue());
System.out.println(date);

Where to store config, web.config or not...

For those who customise WebDrawer on one machine and then push it to a production machine this may prove handy.  I wanted to change the setup of IIS on my dev machine to not update the web.config when I change the authentication settings.  Here is what I did...

1. Go to Feature delegation in IIS Manager

feature_delegation_icon.PNG

2. Select Custom Sites

custom_sites.PNG

3. Set the authentication settings to Read Only

read_only.PNG

4. Clean modified web.configs

Remove the <security /> section from all web.config files that have already been updated, otherwise you will get an error when you attempt to open these applications.  Also check c:\inetpub\wwwroot\web.config to ensure that authentication has not been set here.

5. Update applications in IIS

Go through each application in IIS and ensure the authentication settings are correct, this will update applicationHost.config.

Summary

Setting these settings to Read Only prevents them being updated in the application specific web.config, instead they will be set in the applicationHost.config

Upload a file and create a new revision

I just added a new sample to the documentation that ships with the ServiceAPI.  In the mean-time here it is, the two step process to first upload a file and then use it to create a new Rendition on a new Record.

You could also use Record.FilePath to use the uploaded file to create the electronic document for the Record itself.

FileInfo fi = new FileInfo("c:\\junk\\test.pdf");

var fileResponse = client.UploadFile(fi, System.Web.MimeMapping.GetMimeMapping(fi.Name));

Record rec = new Record();
rec.RecordType = new RecordTypeRef() { Uri = 2 };
rec.Title = "test";

rec.ChildRenditions = new List();

rec.ChildRenditions.Add(new RecordRendition()
{
    TypeOfRendition = RenditionType.Longevity,
    FromFileName = fileResponse.FilePath,
    Description = "my PDF rendition"
});

FileInfo mainFile = new FileInfo("c:\\junk\\test.docx");

var response = client.PostFileWithRequest(mainFile, rec);

foreach (Record record in response.Results)
{
    Console.WriteLine(record.Title);
}

Custom Authentication

A couple of years I wrote a post describing Google authentication in the ServiceAPI.  This is an update of that.  The details are in the samples repo and here is a video of me applying the sample from the repo to a WebDrawer and Web Client instance.

Other authentication providers

Remember the repo contains a link the ServiceStack documentation which provides assistance for using (and creating) other authentication providers.

Be careful with search grammar items in string searches

I was caught out the other day using hard-coded string searches in a ServiceAPI application.  It can be convenient (or necessary) to store canned searches (e.g. 'extension:docx OR extension:doc') at times.  The thing to remember is that search strings can be localised and your end user may not be using English.

First, the clauses

Typically the English versions of the search clauses should work irrespective of the user's language, so even if the end user has selected French ' extension' should still be OK.  Due to my over cautious nature I still use the internal name (e.g. 'recExtension) which can be found in the list of search clauses in the ServiceAPI.

Then the grammar items

Search grammar items (such as 'or' and 'and') are not language neutral so 'recExtension:doc or recExtensiom:docx' will not produce any results if your end user has selected Dutch as their language, what you will need is 'recExtension:doc of recExtensiom:docx'.

SDK

Getting the caption for a search grammar item in the SDK is simple:

EnumItem item = new EnumItem(AllEnumerations.SearchGrammarItem, (int)SearchGrammarItem.And).Caption

ServiceAPI - .Net

It is nearly as simple to get the grammar items using the ServiceAPI .Net client.

TrimClient client = new TrimClient("http://localhost/ServiceAPI");
client.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;

var response = client.Get<EnumItemDetailsResponse>(new EnumItemDetails() { Enums = new AllEnumerations[] { AllEnumerations.SearchGrammarItem } });
Console.WriteLine(response.EnumItems[AllEnumerations.SearchGrammarItem].Where(ei => ei.Name == "Not").First().Caption);

ServiceAPI

Or query the ServiceAPI directly:

http://localhost/ServiceAPI/EnumItem?Enums=SearchGrammarItem&format=json

In short

Never hard code string searches using things like 'not', 'or', 'and', 'me' or anything else in the SearchGrammarItems enum.  If you do and someone switches languages then everything is broken.

Using Postman to create a record

Previously I experiment with searching using Postman  Today lets create a record and upload a file in the same request.

Some Code

Here is the HTTP I generated using Postman.

POST /ServiceAPI81/record HTTP/1.1
Host: desktop-39dgcn3
Authorization: Basic aXR1X3RhZG1pbjpUcmltQEhQMQ==
Accept: application/json
Cache-Control: no-cache
Postman-Token: b9575896-9e99-ccf0-4803-452e4fba11d4
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="RecordRecordType"

Document
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="Files"; filename=""
Content-Type: 


------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="RecordAuthor"

david
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="AlcoholLevel"

5
------WebKitFormBoundary7MA4YWxkTrZu0gW--

ServiceAPI impersonation

The configuration section of the ServiceAPI help documents a handy little property called 'trustedToImpersonate'.  This allows your ServiceAPI to choose to trust one or more calling services.

Scenario

This can be useful in a server to server scenario where passing the actual user credentials to the ServiceAPI is not practical.

Configuration

To allow a particular account to impersonate others users set their name in the hptrim.config file as seen below.  TrustedToImpersonate is a regular expression so you can list multiple accounts if you need to.

<hptrim
  serviceFeatures="Razor,Html,Json,Xml,PredefinedRoutes" 
  trustedToImpersonate="trim\\davidc"
  ...
</hptrim>       

Usage

To impersonate someone using the .Net client librarys use the SetUserToImpersonate() method.

TrimClient client = new TrimClient("http://MyServer/ServiceAPI");
client.Credentials = new NetworkCredential("davidc", "my password", "trim");
client.SetUserToImpersonate("someone\\else");

To impersonate from a different context simply send an HTTP header named 'userToImpersonate' with the value being the user's name along with each request.

ServiceAPI searching, GET or POST

It had been in the back of my mind that using HTTP GET for searching in a project I was working on would lead to trouble and several hours of re-factoring later I am proved correct.

Background

The default way we do searching in the ServiceAPI is using HTTP GET, for example:

http://localhost/ServiceAPI/Record?pageSize=20&q=all&filter=extension:docx,xlsx&properties=RecordTitle&format=json

There are several advantages to use the GET verb but there is also a trap which is that IIS has limitations on the length of a URL.  Usually this is not a problem and the ServiceAPI has design features to help keep URLs concise (such as custom property sets) however I encountered a scenario where I could not avoid the IIS error resulting from an overlong URL.

The scenario

The scenario that caught me was needing the flexibility for the end user to specify as many properties in the properties query parameter as they desired and I did not want to use the 'all' property set as I did not want the cost of fetching every record property.  If the user chose too many properties their request would fail.

The solution

The solution is that the ServiceAPI allows you to POST the same request that you usually GET, you just have to post it to a different end point.  There are a couple of examples of this using an form post and AJAX in the documentation, here is an example using the .Net client.

TrimClient client = new TrimClient("http://localhost/ServiceAPI");
client.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;


PostSearch request = new PostSearch();
request.TrimType = BaseObjectTypes.Record;
request.q = "all";
request.Properties = new PropertyList(PropertyIds.RecordTitle, PropertyIds.RecordNumber);

var response = client.ServiceClient.Send<RecordsResponse>("POST" , "Search", request);

foreach (Record rec in response.Results)
{
    Console.WriteLine(rec.Title);
}

Java ServiceAPI client

Prior to Content Manager 9.0 there was no supported Java client for the ServiceAPI, although I did publish an experimental project.  As of 9.0 we are able to use the ServiceStack Java client.  This video demonstrates that client. 

The code

The sample project I created in the video is on our samples site. Beware, in the video I omit the call to 'setAlwaysSendBasicAuthHeaders', you should always call this when using basic authentication.