API

Foundation - Upload API

on - edited on by (896 Views)

Foundation - Upload API
Request all running uploads
Initiate dataset upload
Uploading Client Files
Complete upload
Request status of a running upload
Cancel a running upload
Delete an existing upload

Foundation - Upload API

Provides upload operations of geospatial datasets from the client machine to Smart M.App platform by posting to the /uploads endpoint. The Uploads API, in addition to basic client uploading, is responsible for decoding, provisioning other resources such as catalog availability, thumnails, attachments and making the dataset available for access using the Datasets API.

Request all running uploads

Returns a json array of objects, each with detailed information about an upload job that is currently processing.

To access all running uploads, a user must make an HTTP GET request to the following endpoint:

GET /api/v1/uploads

Successul HTTP Response:

HTTP Status Code	Description	Payload
200	Successfully retrieved all running uploads	[ { "id": "", "displayableStatus": "COMPLETE", "progress": "100", "state": "COMPLETE", "startTime": "2016-9-14T15:28:17Z", "endTime": "" }, ... ]

Successful Response Parameters:

Title	Type	Description
id	String	Unique identifier used to track a specific upload job
displayableStatus	String	Current status of upload job
progress	String	Percentage based value to indicate job completion
state	String	Current state/phase of the upload job
startTime	String	Timestamp formatted string indicating the date of origin for an upload job
endTime	String	Timestamp formatted string indicating the date of completion for an upload job

Error Response:

HTTP Status Code	Description
401	Error, not authorized to access the catalog.
500	Unexpected error while processing the request.

Initiate dataset upload

Begin the process of uploading files to the catalog.

This is the intial step for uploading files to the Smart M.App Platform. This request should use a POST method, and be made to the following endpoint:

POST /api/v1/uploads

Request Parameters:

Title	Required	Type	Values	Description
copyToInternalBucket	false	Boolean	true(Default) false	During an S3 to S3 upload, controls if the uploaded data is copied in to windmill S3 (true) or referenced from the original external S3 source (false).
downloadIsAllowed	false	Boolean	true(Default) false	Controls if the uploaded data is allowed to be downloaded.
exclude	false	Array[String]	A list of file names	List of regular expressions that specify files that should be excluded from crawling. The easiest way to use this parameter, if you know the names of the files that you want to exclude is to just list those files (filename only, not full path).
footPrintType	false	String	Extent(Default) ActiveArea None	Controls the type of footprint. Extent - uses a minimum bounding rectangle. ActiveArea - uses the geometric envelope. None - prevents a footprint from being calculated.
generateThumbnail	false	Boolean	true(Default) false	Controls if a thumbnail image is generated.
include	false	Array[String]	A list of file names	List of regular expressions to specify files that should be crawled (and only these files). The easiest way to use this parameter, if you know the names of the files that you want to include is to just list those files (filename only, not full path).
parentFolderId	true	String	Any valid catalog id for a folder.	The folder the data will be uploaded to. Unless subFolder is also specified, then it will create or use a subFolder in this parentFolder, and put the uploaded resources into the subFolder.
presignRequired	false	Boolean	true(Default) false	Only valid when uploadStoreType = "S3". Defaults to true, controls whether the credentials returned for the S3 upload are for presigned requests. The default is true for backwards compatibility, presigned requests are deprecated and support for them will be removed in a future version. We recommend setting presignRequired = false and using the AWS SDK to perform the uploads.
publishIsAllowed	false	Boolean	true(Default) false	Controls if the resource is allowed to be published to a web service such as WMS, WFS, etc.
publishProperties	false	Object	autoPublish capabilitiesTitle ecwpEnabled geomapserviceEnabled jpipEnabled outputCSList parentVirtualFolder virtualName wmsEnabled wmtsEnabled	This object contains properties to determine if files in the upload collection should be automatically published, and how they are to be published. Files that are not successfully published will not result in an error.
autoPublish	false	Boolean	true(Default) false	Detemines if the files in the upload collection will be automatically published.
capabilitiesTitle	false	String	Any valid string	The title a dataset will be registered as with the ImageManagement service.
ecwpEnabled	false	Boolean	true(Default) false	Detemines if the Image Management service will attempt to publish this dataset using the ECW Protocol.
geomapserviceEnabled	false	Boolean	true(Default) false	Detemines if the Image Management service will attempt to publish this dataset using the GeoMap service.
jpipEnabled	false	Boolean	true(Default) false	Detemines if the Image Management service will attempt to publish this dataset using the JPEG 2000 Interactive Protocol.
wmsEnabled	false	Boolean	true(Default) false	Determines if the Image Management service will attempt to publish this image using the Web Map service.
wmtsEnabled	false	Boolean	true(Default) false	Determines if the Image Management service will attempt to publish this image using the Web Map Tile service
outputCSList	false	Array[String]	Any valid coordinate reference system	An array of coordinate reference systems for this collection of datasets
parentVirtualFolder	false	String	Any valid parent folder reference	The parent virtual folder. Defaults to '/'.
virtualName	false	String	Any valid string	The virtual name of the dataset. Defaults to the catalog item id.
s3Keys	false	Array[String]	Valid S3 keys for S3-S3 upload	List of S3 keys to upload from the sourceBucket when doing an S3 to S3 upload.
sourceBucket	false	String	Any valid S3 bucket name	External S3 bucket name from which to upload when doing an S3 to S3 upload.
subFolder	false	String	Any valid string	The name of a subFolder the upload will be placed in.
uploadStoreType	false	String	Direct(Default) S3	Direct - "Normal" upload from a file system into windmill. Files are uploaded to a file system in AWS and the crawling is done from there. S3(Preferred) - Data is loaded from an existing S3 bucket (can be external or can be the bucket that M.App Chest uses) into windmill. S3 bucket can be an internal bucket (like M.App Chest uses), a customer bucket, or a third party bucket (like landsat).

Request Payload Template:

{
  "copyToInternalBucket": true,
  "downloadIsAllowed": true,
  "exclude": [
    "string"
  ],
  "footPrintType": "Extent",
  "generateThumbnail": true,
  "include": [
    "string"
  ],
  "presignRequired": false,
  "parentFolderId": "string",
  "publishIsAllowed": true,
  "publishProperties": {
    "autoPublish": true,
    "capabilitiesTitle": "string",
    "ecwpEnabled": true,
    "geomapserviceEnabled": true,
    "jpipEnabled": true,
    "outputCSList": [
      "string"
    ],
    "parentVirtualFolder": "string",
    "virtualName": "string",
    "wmsEnabled": true,
    "wmtsEnabled": true
  },
  "s3Keys": [
    "string"
  ],
  "sourceBucket": "string",
  "subFolder": "string",
  "uploadStoreType": "Direct"
}

The following payload is the minimal recommended payload. The parent folder has the ID of ROOT and the subFolder will be created when the upload is completed if it does not already exist.

{
  "parentFolderId": "146e1167-2508-495d-8cfb-a93cb2852e3c_2c918082559220be0155922a009200dd",
  "subFolder": "My Folder"
}

If you are trying to upload to the ROOT folder, you will first need the ROOT catalog id. To obtain the ROOT id use the search.json endpoint with the following payload:

{
  maxresults: 10,
  first: 0,
  template : {
      'class': 'com.erdas.rsp.babel.model.ResourceAggregate',
      'name': 'ROOT'
  }
}

After receiving a successful response, you have sucessfully started your upload. Next you will need to see the "Upload Client Files" section for instructions on uploading individual files to the upload job you have now created.

Successul HTTP Response:

HTTP Status Code

Description

Payload

201

Successfully initiated your upload

{
  "activeStorageProperties": {
    "activeStoragePath": "string"
  },
  "crawledDatasetIds": {},
  "crawledFolderIds": {},
  "displayableStatus": "string",
  "endTime": "string",
  "errors": "string",
  "id": "string",
  "progress": "string",
  "registerProperties": {
    "copyToInternalBucket": true,
    "downloadIsAllowed": true,
    "footPrintType": "Extent",
    "generateThumbnail": true,
    "parentFolderId": "string",
    "publishIsAllowed": true,
    "publishProperties": {
      "autoPublish": true,
      "capabilitiesTitle": "string",
      "ecwpEnabled": true,
      "geomapserviceEnabled": true,
      "jpipEnabled": true,
      "outputCSList": [
        "string"
      ],
      "parentVirtualFolder": "string",
      "virtualName": "string",
      "wmsEnabled": true,
      "wmtsEnabled": true
    },
    "s3Keys": [
      "string"
    ],
    "sourceBucket": "string",
    "subFolder": "string",
    "uploadStoreType": "Direct"
  },
  "s3Properties": {
    "accessKeyId": "string",
    "secretKey": "string",
    "sessionToken": "string",
    "bucket": "string",
    "encodedPolicy": "string",
    "keyPrefix": "string",
    "signature": "string"
  },
  "startTime": "string",
  "state": "WAITING",
  "uploadComplete": true,
  "uploadedFiles": [
    "string"
  ]
}

Error Response:

HTTP Status Code	Description
400	Error with details in the response body.
401	Error, not authorized to access the catalog
403	Error, forbidden to access the item
500	Unexpected Error while processing the request

Next Steps

After you have received a successful response, please review the response information to ensure all information regarding your upload is correct. If all of the information in your request is correct, please proceed to the "Uploading Client Files" section and review the instructions for your chosen method of upload.

Uploading Client Files

Once you have initiated your upload, the next step is to being uploading files via your chosen method(S3 or Direct). Although we provide both methods of uploading files it is important to note the differences between these methods. Uploading files via S3 is the preferred method because of its stability and performance during uploads, and is what we use internally.

Upload files using S3

Handles the process of uploading files using an S3 bucket when the "S3' uploadStoreType method is used. S3 uploads can be presigned or not. The presigned S3 upload type was used before August 2017, but is now deprecated and will be removed at some point in the future. Support for non-presigned uploads was deployed in August 2017 and is now strongly encouraged as it allows the use of the AWS SDK to perform the uploads, which also adds support for uploading single files greater than 5GB in size. Presigned uploads are still documented here for backwards compatibility, but are strongly discouraged.

S3 bucket properties

To use this method, please ensure that you have specified the uploadStoreType parameter in your initial request as "S3". If your initial request uses the S3 option for the uploadStoreType, then your response will include a "s3Properties" object. This object contains all of the information of an s3 bucket made available for you to upload your files.

Non-presigned uploads to S3

Perform a non-presigned upload by specifying the "S3" uploadStoreType and presignRequired: false properties in your original POST /uploads request when creating the upload. This will return different information in the s3Properties object in the response, specifically the accessKeyId, secretKey, and sessionToken properties. These properties can be used to perform the file uploads to S3 using the AWS SDK, which will automatically support uploading single files greater than 5GB.

The following code snippet shows an example of how to use the AWS SDK to upload files using the information returned for a non-presigned upload.

// First, create an AWS S3 object
var uploadResponse = <JSON response from POST /uploads request>;
var s3props = uploadResponse.s3Properties;
var s3 = new AWS.S3({
                apiVersion: '2006-03-01',
                params: {
                    Bucket: s3props.bucket
                },
                credentials: new AWS.Credentials(s3props.accessKeyId, s3props.secretKey, s3props.sessionToken),
                region: "us-west-1"
});

// create the upload
var upload = s3.upload({
                    Key: s3props.keyPrefix + '/' + <filename>,
                    Body: <file object>
});

// to watch the progress of the file upload
upload.on('httpUploadProgress', function(e) {
      // do stuff with 'e'
});

// send the upload and handle success or failure
upload.send(function(err, data) {
      if (err) {
            // error handling
      } else {
            // success handling
      }
});

Presigned uploads to S3 (Deprecated)

Presigned uploads are deprecated and will be removed in a future version. They are only documented here for backwards compatibility.

Once you have access to the necessary S3 bucket information, you will need to make individual HTTP POST request for each of the files you intend to upload. The POST URL will be in the format below:

https://{s3bucket}.{domain}/

The value for the s3bucket is the bucket property in your s3Propeties object, and the domain in this case is "s3-us-west-2.amazon.com". With this POST request you will also need to include additional properties to the multipart/form-data containing the file you wish to upload and the remaining s3bucket information.

Request Parameters:

Title	Required	Type	Values	Description
key	true	String	Any valid AWS key	The value for this property is in your s3Properties object. s3Properties.keyPrefix + "/{filename}", is the correct format for this property value, where {filename} is the name of the file you are currently uploading.
AWSAccessKeyId	true	String	Any valid AWS AccesKeyId	The value for this property is your s3Properties object. The s3Properties.accessKeyId value should be the value for this property.
policy	true	String	Any valid AWS policy	The value for this property is your s3Properties object. The s3Properties.encodedPolicy value should be the value for this property.
signature	true	String	Any valid AWS signature	The value for this property is your s3Properties object. The s3Properties.signature value should be the value for this property.
file	true	FileObject	Any valid File Object	File object for the client file that will be uploaded to S3.

Upload files using Direct method

Send post request to the following URL with the file as form data in the request payload.

After you have initiated your upload request, you will need to begin adding files to this upload job. Using the HTTP Post method, you will need to make request to the following endpoint:

POST /api/v1/uploads/{uploadId}/files

For each file you will need to make a separate request containing mutltipart/form-data file objects. Continue making these request until all files for this particular upload have been completed. After all files have been added, complete your upload by following the instructions below in the "Complete client upload" section.

Successul HTTP Response:

HTTP Status Code	Description	Payload
201	Successfully completed file upload	There is no response payload for this request

Once you have uploaded all of your client files and received successful responses for each file you are ready to complete your upload. To do this please see the "Complete client upload" section to finalize your upload.

Error Response:

HTTP Code	Description
401	Error, not authorized to access the catalog
403	Error, forbidden to access the item
404	Error, item not found
500	Unexpected error while processing the request

Next Steps

After you have successfully uploaded all of your files, please see the "Complete upload" section to finalize your upload.

Complete upload

When all files are uploaded, use this request to trigger completion of file upload and cataloging can continue.

This is the final request necessary for completing an upload and adding files to the Smart M.App Platform. Using the uploadId that was returned by the initial request, perform a final HTTP request to the following endpoint:

PUT /api/v1/uploads/{uploadId}/complete

Successul HTTP Response:

HTTP Status Code	Description	Payload
201	Successfully completed client upload	There is no response payload for this request

Error Response:

HTTP Code	Description
401	Error, not authorized to access the catalog
403	Error, forbidden to access the item
500	Unexpected error while processing the request

Request status of a running upload

This endpoint will return all information about a specific upload job.

While a upload job is running, users will naturally want to check the status of this job. This is done by using the uploadId (returned in the HTTP response of the initial request) to make another request to the following endpoint:

GET /api/v1/uploads/{uploadId}

Note: When a upload is started in M.App Chest, we periodically make request against this endpoint to update the user on the status of the running upload job. Once the job is complete, we update the user of the progress and inform the user when all processing is finailized.

Successul HTTP Response:

HTTP Status Code

Description

Payload

200

Successfully retrieved status for a running upload

{
  "activeStorageProperties": {
    "activeStoragePath": "string"
  },
  "crawledDatasetIds": {},
  "crawledFolderIds": {},
  "displayableStatus": "string",
  "endTime": "string",
  "errors": "string",
  "id": "string",
  "progress": "string",
  "registerProperties": {
    "copyToInternalBucket": true,
    "downloadIsAllowed": true,
    "footPrintType": "Extent",
    "generateThumbnail": true,
    "parentFolderId": "string",
    "publishIsAllowed": true,
    "publishProperties": {
      "autoPublish": true,
      "capabilitiesTitle": "string",
      "ecwpEnabled": true,
      "geomapserviceEnabled": true,
      "jpipEnabled": true,
      "outputCSList": [
        "string"
      ],
      "parentVirtualFolder": "string",
      "virtualName": "string",
      "wmsEnabled": true,
      "wmtsEnabled": true
    },
    "s3Keys": [
      "string"
    ],
    "sourceBucket": "string",
    "subFolder": "string",
    "uploadStoreType": "Direct"
  },
  "s3Properties": {
    "accessKeyId": "string",
    "bucket": "string",
    "encodedPolicy": "string",
    "keyPrefix": "string",
    "signature": "string"
  },
  "startTime": "string",
  "state": "WAITING",
  "uploadComplete": true,
  "uploadedFiles": [
    "string"
  ]
}

Error Response:

HTTP Status Code	Description
401	Error, not authorized to access the catalog
403	Error, forbidden to access the item
404	Error, item not found
500	Unexpected error while processing the request

Cancel a running upload

Takes a single upload id and stops all effort of uploading/crawling of the files.

PUT /api/v1/uploads/{uploadId}/cancel

Successul HTTP Response:

HTTP Status Code	Description	Payload
202	Successfully canceled a running upload	There is no payload for this response

Error Response:

HTTP Status Code	Description
400	Error with details in the response body.
401	Error, not authorized to access the catalog
403	Error, forbidden to access the item
500	Unexpected error while processing the request

Delete an existing upload

Deletes an existing upload. This will stop all effort to upload the files to the catalog.

DELETE /api/v1/uploads/{uploadId}

Successul HTTP Response:

HTTP Status Code	Description	Payload
204	Successfully removed existing upload	There is no payload for this response

Error Response:

HTTP Status Code	Description
400	Error with details in the response body
401	Error, not authorized to access the catalog
403	Error, forbidden to access the item
404	Error, item not found
500	Unexpected error while processing the request

Rate this article:

Yes No

Contributors