Hexagon Geospatial
MENU

API

M.App Portfolio provides a modern, cloud-based platform for creating and delivering diverse geospatial web applications.
Through the M.App Studio, our partners can design, build, and deploy their own Hexagon Smart M.Apps.
Showing results for 
Search instead for 
Do you mean 

Foundation - Upload API

by Technical Evangelist on ‎11-09-2016 01:15 PM - edited on ‎08-29-2017 09:00 AM by Technical Evangelist (789 Views)

Foundation - Upload API

Provides upload operations of geospatial datasets from the client machine to Smart M.App platform by posting to the /uploads endpoint. The Uploads API, in addition to basic client uploading, is responsible for decoding, provisioning other resources such as catalog availability, thumnails, attachments and making the dataset available for access using the Datasets API.

 

Request all running uploads

Returns a json array of objects, each with detailed information about an upload job that is currently processing.

 

To access all running uploads, a user must make an HTTP GET request to the following endpoint:

GET /api/v1/uploads

 

Successul HTTP Response:
HTTP Status Code Description Payload
200 Successfully retrieved all running uploads
[
  {
    "id": "",
    "displayableStatus": "COMPLETE",
    "progress": "100",
    "state": "COMPLETE",
    "startTime": "2016-9-14T15:28:17Z",
    "endTime": ""
  },
  ...
]

 

Successful Response Parameters:
Title Type Description
id String Unique identifier used to track a specific upload job
displayableStatus String Current status of upload job
progress String Percentage based value to indicate job completion
state String Current state/phase of the upload job
startTime String Timestamp formatted string indicating the date of origin for an upload job
endTime String Timestamp formatted string indicating the date of completion for an upload job

 

Error Response:
HTTP Status Code Description
401 Error, not authorized to access the catalog.
500 Unexpected error while processing the request.

 

 

Initiate dataset upload

Begin the process of uploading files to the catalog.

 

This is the intial step for uploading files to the Smart M.App Platform. This request should use a POST method, and be made to the following endpoint:

 

POST /api/v1/uploads

 

Request Parameters:
Title Required Type Values Description
copyToInternalBucket false Boolean
  • true(Default)
  • false
During an S3 to S3 upload, controls if the uploaded data is copied in to windmill S3 (true) or referenced from the original external S3 source (false).
downloadIsAllowed false Boolean
  • true(Default)
  • false
Controls if the uploaded data is allowed to be downloaded.
exclude false Array[String] A list of file names List of regular expressions that specify files that should be excluded from crawling. The easiest way to use this parameter, if you know the names of the files that you want to exclude is to just list those files (filename only, not full path).
footPrintType false String
  • Extent(Default)
  • ActiveArea
  • None

Controls the type of footprint.

  • Extent - uses a minimum bounding rectangle.
  • ActiveArea - uses the geometric envelope.
  • None - prevents a footprint from being calculated.
generateThumbnail false Boolean
  • true(Default)
  • false
Controls if a thumbnail image is generated.
include false Array[String] A list of file names List of regular expressions to specify files that should be crawled (and only these files). The easiest way to use this parameter, if you know the names of the files that you want to include is to just list those files (filename only, not full path).
parentFolderId true String Any valid catalog id for a folder. The folder the data will be uploaded to. Unless subFolder is also specified, then it will create or use a subFolder in this parentFolder, and put the uploaded resources into the subFolder.
presignRequired false Boolean
  • true(Default)
  • false
Only valid when uploadStoreType = "S3". Defaults to true, controls whether the credentials returned for the S3 upload are for presigned requests. The default is true for backwards compatibility, presigned requests are deprecated and support for them will be removed in a future version. We recommend setting presignRequired = false and using the AWS SDK to perform the uploads.
publishIsAllowed false Boolean
  • true(Default)
  • false
Controls if the resource is allowed to be published to a web service such as WMS, WFS, etc.
publishProperties false Object
  • autoPublish
  • capabilitiesTitle
  • ecwpEnabled
  • geomapserviceEnabled
  • jpipEnabled
  • outputCSList
  • parentVirtualFolder
  • virtualName
  • wmsEnabled
  • wmtsEnabled
This object contains properties to determine if files in the upload collection should be automatically published, and how they are to be published. Files that are not successfully published will not result in an error.
autoPublish false Boolean
  • true(Default)
  • false
Detemines if the files in the upload collection will be automatically published.
capabilitiesTitle false String Any valid string The title a dataset will be registered as with the ImageManagement service.
ecwpEnabled false Boolean
  • true(Default)
  • false
Detemines if the Image Management service will attempt to publish this dataset using the ECW Protocol.
geomapserviceEnabled false Boolean
  • true(Default)
  • false
Detemines if the Image Management service will attempt to publish this dataset using the GeoMap service.
jpipEnabled false Boolean
  • true(Default)
  • false
Detemines if the Image Management service will attempt to publish this dataset using the JPEG 2000 Interactive Protocol.
wmsEnabled false Boolean
  • true(Default)
  • false
Determines if the Image Management service will attempt to publish this image using the Web Map service.
wmtsEnabled false Boolean
  • true(Default)
  • false
Determines if the Image Management service will attempt to publish this image using the Web Map Tile service
outputCSList false Array[String] Any valid coordinate reference system An array of coordinate reference systems for this collection of datasets
parentVirtualFolder false String Any valid parent folder reference The parent virtual folder. Defaults to '/'.
virtualName false String Any valid string The virtual name of the dataset. Defaults to the catalog item id.
s3Keys false Array[String] Valid S3 keys for S3-S3 upload List of S3 keys to upload from the sourceBucket when doing an S3 to S3 upload.
sourceBucket false String Any valid S3 bucket name External S3 bucket name from which to upload when doing an S3 to S3 upload.
subFolder false String Any valid string The name of a subFolder the upload will be placed in.
uploadStoreType false String
  • Direct(Default)
  • S3
  • Direct - "Normal" upload from a file system into windmill. Files are uploaded to a file system in AWS and the crawling is done from there.
  • S3(Preferred) - Data is loaded from an existing S3 bucket (can be external or can be the bucket that M.App Chest uses) into windmill. S3 bucket can be an internal bucket (like M.App Chest uses), a customer bucket, or a third party bucket (like landsat).

       

Request Payload Template:
{
  "copyToInternalBucket": true,
  "downloadIsAllowed": true,
  "exclude": [
    "string"
  ],
  "footPrintType": "Extent",
  "generateThumbnail": true,
  "include": [
    "string"
  ],
"presignRequired": false, "parentFolderId": "string", "publishIsAllowed": true, "publishProperties": { "autoPublish": true, "capabilitiesTitle": "string", "ecwpEnabled": true, "geomapserviceEnabled": true, "jpipEnabled": true, "outputCSList": [ "string" ], "parentVirtualFolder": "string", "virtualName": "string", "wmsEnabled": true, "wmtsEnabled": true }, "s3Keys": [ "string" ], "sourceBucket": "string", "subFolder": "string", "uploadStoreType": "Direct" }

 

The following payload is the minimal recommended payload. The parent folder has the ID of ROOT and the subFolder will be created when the upload is completed if it does not already exist.

{
  "parentFolderId": "146e1167-2508-495d-8cfb-a93cb2852e3c_2c918082559220be0155922a009200dd",
  "subFolder": "My Folder"
}

 

If you are trying to upload to the ROOT folder, you will first need the ROOT catalog id. To obtain the ROOT id use the search.json endpoint with the following payload:

{
  maxresults: 10,
  first: 0,
  template : {
      'class': 'com.erdas.rsp.babel.model.ResourceAggregate',
      'name': 'ROOT'
  }
}

 

 

After receiving a successful response, you have sucessfully started your upload. Next you will need to see the "Upload Client Files" section for instructions on uploading individual files to the upload job you have now created.

 

Successul HTTP Response:
HTTP Status Code Description Payload
201 Successfully initiated your upload
{
  "activeStorageProperties": {
    "activeStoragePath": "string"
  },
  "crawledDatasetIds": {},
  "crawledFolderIds": {},
  "displayableStatus": "string",
  "endTime": "string",
  "errors": "string",
  "id": "string",
  "progress": "string",
  "registerProperties": {
    "copyToInternalBucket": true,
    "downloadIsAllowed": true,
    "footPrintType": "Extent",
    "generateThumbnail": true,
    "parentFolderId": "string",
    "publishIsAllowed": true,
    "publishProperties": {
      "autoPublish": true,
      "capabilitiesTitle": "string",
      "ecwpEnabled": true,
      "geomapserviceEnabled": true,
      "jpipEnabled": true,
      "outputCSList": [
        "string"
      ],
      "parentVirtualFolder": "string",
      "virtualName": "string",
      "wmsEnabled": true,
      "wmtsEnabled": true
    },
    "s3Keys": [
      "string"
    ],
    "sourceBucket": "string",
    "subFolder": "string",
    "uploadStoreType": "Direct"
  },
  "s3Properties": {
    "accessKeyId": "string",
"secretKey": "string",
"sessionToken": "string", "bucket": "string", "encodedPolicy": "string", "keyPrefix": "string", "signature": "string" }, "startTime": "string", "state": "WAITING", "uploadComplete": true, "uploadedFiles": [ "string" ] }

 

Error Response:
HTTP Status Code Description
400 Error with details in the response body.
401 Error, not authorized to access the catalog
403 Error, forbidden to access the item
500 Unexpected Error while processing the request

 

Next Steps

After you have received a successful response, please review the response information to ensure all information regarding your upload is correct. If all of the information in your request is correct, please proceed to the "Uploading Client Files" section and review the instructions for your chosen method of upload. 

 

Uploading Client Files

 

Once you have initiated your upload, the next step is to being uploading files via your chosen method(S3 or Direct). Although we provide both methods of uploading files it is important to note the differences between these methods. Uploading files via S3 is the preferred method because of its stability and performance during uploads, and is what we use internally.

 

Upload files using S3

Handles the process of uploading files using an S3 bucket when the "S3' uploadStoreType method is used. S3 uploads can be presigned or not. The presigned S3 upload type was used before August 2017, but is now deprecated and will be removed at some point in the future. Support for non-presigned uploads was deployed in August 2017 and is now strongly encouraged as it allows the use of the AWS SDK to perform the uploads, which also adds support for uploading single files greater than 5GB in size. Presigned uploads are still documented here for backwards compatibility, but are strongly discouraged.  

 

S3 bucket properties

To use this method, please ensure that you have specified the uploadStoreType parameter in your initial request as "S3". If your initial request uses the S3 option for the uploadStoreType, then your response will include a "s3Properties" object. This object contains all of the information of an s3 bucket made available for you to upload your files.

 

Non-presigned uploads to S3

Perform a non-presigned upload by specifying the "S3" uploadStoreType and presignRequired: false properties in your original POST /uploads request when creating the upload. This will return different information in the s3Properties object in the response, specifically the accessKeyId, secretKey, and sessionToken properties. These properties can be used to perform the file uploads to S3 using the AWS SDK, which will automatically support uploading single files greater than 5GB. 

 

The following code snippet shows an example of how to use the AWS SDK to upload files using the information returned for a non-presigned upload.

 

 

// First, create an AWS S3 object
var uploadResponse = <JSON response from POST /uploads request>;
var s3props = uploadResponse.s3Properties;
var s3 = new AWS.S3({
                apiVersion: '2006-03-01',
                params: {
                    Bucket: s3props.bucket
                },
                credentials: new AWS.Credentials(s3props.accessKeyId, s3props.secretKey, s3props.sessionToken),
                region: "us-west-1"
});

// create the upload
var upload = s3.upload({
                    Key: s3props.keyPrefix + '/' + <filename>,
                    Body: <file object>
});

// to watch the progress of the file upload
upload.on('httpUploadProgress', function(e) {
      // do stuff with 'e'
});

// send the upload and handle success or failure
upload.send(function(err, data) {
      if (err) {
            // error handling
      } else {
            // success handling
      }
});

 

 

Presigned uploads to S3 (Deprecated)

Presigned uploads are deprecated and will be removed in a future version. They are only documented here for backwards compatibility.

 

Once you have access to the necessary S3 bucket information, you will need to make individual HTTP POST request for each of the files you intend to upload. The POST URL will be in the format below:

https://{s3bucket}.{domain}/

 

The value for the s3bucket is the bucket property in your s3Propeties object, and the domain in this case is "s3-us-west-2.amazon.com". With this POST request you will also need to include additional properties to the multipart/form-data containing the file you wish to upload and the remaining s3bucket information.

 

Request Parameters:
Title Required Type Values Description
key true String Any valid AWS key The value for this property is in your s3Properties object. s3Properties.keyPrefix + "/{filename}", is the correct format for this property value, where {filename} is the name of the file you are currently uploading.
AWSAccessKeyId true String Any valid AWS AccesKeyId The value for this property is your s3Properties object. The s3Properties.accessKeyId value should be the value for this property.
policy true String Any valid AWS policy The value for this property is your s3Properties object. The s3Properties.encodedPolicy value should be the value for this property.
signature true String Any valid AWS signature The value for this property is your s3Properties object. The s3Properties.signature value should be the value for this property.
file  true  FileObject Any valid File Object File object for the client file that will be uploaded to S3.

 

 

Upload files using Direct method

Send post request to the following URL with the file as form data in the request payload.

 

After you have initiated your upload request, you will need to begin adding files to this upload job. Using the HTTP Post method, you will need to make request to the following endpoint:

 

POST /api/v1/uploads/{uploadId}/files

 

For each file you will need to make a separate request containing mutltipart/form-data file objects. Continue making these request until all files for this particular upload have been completed. After all files have been added, complete your upload by following the instructions below in the "Complete client upload" section. 

 

Successul HTTP Response:
HTTP Status Code Description Payload
201 Successfully completed file upload There is no response payload for this request

 

Once you have uploaded all of your client files and received successful responses for each file you are ready to complete your upload. To do this please see the "Complete client upload" section to finalize your upload.


Error Response:
HTTP Code Description
401 Error, not authorized to access the catalog
403 Error, forbidden to access the item
404 Error, item not found
500 Unexpected error while processing the request

 

 
Next Steps

After you have successfully uploaded all of your files, please see the "Complete upload" section to finalize your upload.

 

Complete upload

When all files are uploaded, use this request to trigger completion of file upload and cataloging can continue.

 

This is the final request necessary for completing an upload and adding files to the Smart M.App Platform. Using the uploadId that was returned by the initial request, perform a final HTTP request to the following endpoint:

PUT /api/v1/uploads/{uploadId}/complete

 

Successul HTTP Response:
HTTP Status Code Description Payload
201 Successfully completed client upload There is no response payload for this request

Error Response:
HTTP Code Description
401 Error, not authorized to access the catalog
403 Error, forbidden to access the item
500 Unexpected error while processing the request

 

Request status of a running upload

This endpoint will return all information about a specific upload job.

 

While a upload job is running, users will naturally want to check the status of this job. This is done by using the uploadId (returned in the HTTP response of the initial request) to make another request to the following endpoint:

 

GET /api/v1/uploads/{uploadId}

 

Note: When a upload is started in M.App Chest, we periodically make request against this endpoint to update the user on the status of the running upload job. Once the job is complete, we update the user of the progress and inform the user when all processing is finailized.

 

Successul HTTP Response:
HTTP Status Code Description Payload
200 Successfully retrieved status for a running upload
{
  "activeStorageProperties": {
    "activeStoragePath": "string"
  },
  "crawledDatasetIds": {},
  "crawledFolderIds": {},
  "displayableStatus": "string",
  "endTime": "string",
  "errors": "string",
  "id": "string",
  "progress": "string",
  "registerProperties": {
    "copyToInternalBucket": true,
    "downloadIsAllowed": true,
    "footPrintType": "Extent",
    "generateThumbnail": true,
    "parentFolderId": "string",
    "publishIsAllowed": true,
    "publishProperties": {
      "autoPublish": true,
      "capabilitiesTitle": "string",
      "ecwpEnabled": true,
      "geomapserviceEnabled": true,
      "jpipEnabled": true,
      "outputCSList": [
        "string"
      ],
      "parentVirtualFolder": "string",
      "virtualName": "string",
      "wmsEnabled": true,
      "wmtsEnabled": true
    },
    "s3Keys": [
      "string"
    ],
    "sourceBucket": "string",
    "subFolder": "string",
    "uploadStoreType": "Direct"
  },
  "s3Properties": {
    "accessKeyId": "string",
    "bucket": "string",
    "encodedPolicy": "string",
    "keyPrefix": "string",
    "signature": "string"
  },
  "startTime": "string",
  "state": "WAITING",
  "uploadComplete": true,
  "uploadedFiles": [
    "string"
  ]
}

Error Response:
HTTP Status Code Description
401 Error, not authorized to access the catalog
403 Error, forbidden to access the item
404 Error, item not found
500 Unexpected error while processing the request

 

 

Cancel a running upload

Takes a single upload id and stops all effort of uploading/crawling of the files.

 

PUT /api/v1/uploads/{uploadId}/cancel

 

Successul HTTP Response:
HTTP Status Code Description Payload
202 Successfully canceled a running upload There is no payload for this response

 

Error Response:
HTTP Status Code Description
400 Error with details in the response body.
401 Error, not authorized to access the catalog
403 Error, forbidden to access the item
500 Unexpected error while processing the request

 

 

Delete an existing upload

Deletes an existing upload. This will stop all effort to upload the files to the catalog. 

 

DELETE /api/v1/uploads/{uploadId}

 

Successul HTTP Response:
HTTP Status Code Description Payload
204 Successfully removed existing upload There is no payload for this response

 

Error Response:
HTTP Status Code Description
400 Error with details in the response body
401 Error, not authorized to access the catalog
403 Error, forbidden to access the item
404 Error, item not found
500 Unexpected error while processing the request
Contributors