Provides upload operations of geospatial datasets from the client machine to Smart M.App platform by posting to the /uploads endpoint. The Uploads API, in addition to basic client uploading, is responsible for decoding, provisioning other resources such as catalog availability, thumnails, attachments and making the dataset available for access using the Datasets API.
Returns a json array of objects, each with detailed information about an upload job that is currently processing.
To access all running uploads, a user must make an HTTP GET request to the following endpoint:
GET /api/v1/uploads
HTTP Status Code | Description | Payload |
200 | Successfully retrieved all running uploads |
[ { "id": "", "displayableStatus": "COMPLETE", "progress": "100", "state": "COMPLETE", "startTime": "2016-9-14T15:28:17Z", "endTime": "" }, ... ] |
Title | Type | Description |
id | String | Unique identifier used to track a specific upload job |
displayableStatus | String | Current status of upload job |
progress | String | Percentage based value to indicate job completion |
state | String | Current state/phase of the upload job |
startTime | String | Timestamp formatted string indicating the date of origin for an upload job |
endTime | String | Timestamp formatted string indicating the date of completion for an upload job |
HTTP Status Code | Description |
---|---|
401 | Error, not authorized to access the catalog. |
500 | Unexpected error while processing the request. |
Begin the process of uploading files to the catalog.
This is the intial step for uploading files to the Smart M.App Platform. This request should use a POST method, and be made to the following endpoint:
POST /api/v1/uploads
Title | Required | Type | Values | Description |
copyToInternalBucket | false | Boolean |
|
During an S3 to S3 upload, controls if the uploaded data is copied in to windmill S3 (true) or referenced from the original external S3 source (false). |
downloadIsAllowed | false | Boolean |
|
Controls if the uploaded data is allowed to be downloaded. |
exclude | false | Array[String] | A list of file names | List of regular expressions that specify files that should be excluded from crawling. The easiest way to use this parameter, if you know the names of the files that you want to exclude is to just list those files (filename only, not full path). |
footPrintType | false | String |
|
Controls the type of footprint.
|
generateThumbnail | false | Boolean |
|
Controls if a thumbnail image is generated. |
include | false | Array[String] | A list of file names | List of regular expressions to specify files that should be crawled (and only these files). The easiest way to use this parameter, if you know the names of the files that you want to include is to just list those files (filename only, not full path). |
parentFolderId | true | String | Any valid catalog id for a folder. | The folder the data will be uploaded to. Unless subFolder is also specified, then it will create or use a subFolder in this parentFolder, and put the uploaded resources into the subFolder. |
presignRequired | false | Boolean |
|
Only valid when uploadStoreType = "S3". Defaults to true, controls whether the credentials returned for the S3 upload are for presigned requests. The default is true for backwards compatibility, presigned requests are deprecated and support for them will be removed in a future version. We recommend setting presignRequired = false and using the AWS SDK to perform the uploads. |
publishIsAllowed | false | Boolean |
|
Controls if the resource is allowed to be published to a web service such as WMS, WFS, etc. |
publishProperties | false | Object |
|
This object contains properties to determine if files in the upload collection should be automatically published, and how they are to be published. Files that are not successfully published will not result in an error. |
autoPublish | false | Boolean |
|
Detemines if the files in the upload collection will be automatically published. |
capabilitiesTitle | false | String | Any valid string | The title a dataset will be registered as with the ImageManagement service. |
ecwpEnabled | false | Boolean |
|
Detemines if the Image Management service will attempt to publish this dataset using the ECW Protocol. |
geomapserviceEnabled | false | Boolean |
|
Detemines if the Image Management service will attempt to publish this dataset using the GeoMap service. |
jpipEnabled | false | Boolean |
|
Detemines if the Image Management service will attempt to publish this dataset using the JPEG 2000 Interactive Protocol. |
wmsEnabled | false | Boolean |
|
Determines if the Image Management service will attempt to publish this image using the Web Map service. |
wmtsEnabled | false | Boolean |
|
Determines if the Image Management service will attempt to publish this image using the Web Map Tile service |
outputCSList | false | Array[String] | Any valid coordinate reference system | An array of coordinate reference systems for this collection of datasets |
parentVirtualFolder | false | String | Any valid parent folder reference | The parent virtual folder. Defaults to '/'. |
virtualName | false | String | Any valid string | The virtual name of the dataset. Defaults to the catalog item id. |
s3Keys | false | Array[String] | Valid S3 keys for S3-S3 upload | List of S3 keys to upload from the sourceBucket when doing an S3 to S3 upload. |
sourceBucket | false | String | Any valid S3 bucket name | External S3 bucket name from which to upload when doing an S3 to S3 upload. |
subFolder | false | String | Any valid string | The name of a subFolder the upload will be placed in. |
uploadStoreType | false | String |
|
|
{ "copyToInternalBucket": true, "downloadIsAllowed": true, "exclude": [ "string" ], "footPrintType": "Extent", "generateThumbnail": true, "include": [ "string" ],
"presignRequired": false, "parentFolderId": "string", "publishIsAllowed": true, "publishProperties": { "autoPublish": true, "capabilitiesTitle": "string", "ecwpEnabled": true, "geomapserviceEnabled": true, "jpipEnabled": true, "outputCSList": [ "string" ], "parentVirtualFolder": "string", "virtualName": "string", "wmsEnabled": true, "wmtsEnabled": true }, "s3Keys": [ "string" ], "sourceBucket": "string", "subFolder": "string", "uploadStoreType": "Direct" }
The following payload is the minimal recommended payload. The parent folder has the ID of ROOT and the subFolder will be created when the upload is completed if it does not already exist.
{ "parentFolderId": "146e1167-2508-495d-8cfb-a93cb2852e3c_2c918082559220be0155922a009200dd", "subFolder": "My Folder" }
If you are trying to upload to the ROOT folder, you will first need the ROOT catalog id. To obtain the ROOT id use the search.json endpoint with the following payload:
{ maxresults: 10, first: 0, template : { 'class': 'com.erdas.rsp.babel.model.ResourceAggregate', 'name': 'ROOT' } }
After receiving a successful response, you have sucessfully started your upload. Next you will need to see the "Upload Client Files" section for instructions on uploading individual files to the upload job you have now created.
HTTP Status Code | Description | Payload |
201 | Successfully initiated your upload |
{ "activeStorageProperties": { "activeStoragePath": "string" }, "crawledDatasetIds": {}, "crawledFolderIds": {}, "displayableStatus": "string", "endTime": "string", "errors": "string", "id": "string", "progress": "string", "registerProperties": { "copyToInternalBucket": true, "downloadIsAllowed": true, "footPrintType": "Extent", "generateThumbnail": true, "parentFolderId": "string", "publishIsAllowed": true, "publishProperties": { "autoPublish": true, "capabilitiesTitle": "string", "ecwpEnabled": true, "geomapserviceEnabled": true, "jpipEnabled": true, "outputCSList": [ "string" ], "parentVirtualFolder": "string", "virtualName": "string", "wmsEnabled": true, "wmtsEnabled": true }, "s3Keys": [ "string" ], "sourceBucket": "string", "subFolder": "string", "uploadStoreType": "Direct" }, "s3Properties": { "accessKeyId": "string", |
HTTP Status Code | Description |
---|---|
400 | Error with details in the response body. |
401 | Error, not authorized to access the catalog |
403 | Error, forbidden to access the item |
500 | Unexpected Error while processing the request |
After you have received a successful response, please review the response information to ensure all information regarding your upload is correct. If all of the information in your request is correct, please proceed to the "Uploading Client Files" section and review the instructions for your chosen method of upload.
Once you have initiated your upload, the next step is to being uploading files via your chosen method(S3 or Direct). Although we provide both methods of uploading files it is important to note the differences between these methods. Uploading files via S3 is the preferred method because of its stability and performance during uploads, and is what we use internally.
Handles the process of uploading files using an S3 bucket when the "S3' uploadStoreType method is used. S3 uploads can be presigned or not. The presigned S3 upload type was used before August 2017, but is now deprecated and will be removed at some point in the future. Support for non-presigned uploads was deployed in August 2017 and is now strongly encouraged as it allows the use of the AWS SDK to perform the uploads, which also adds support for uploading single files greater than 5GB in size. Presigned uploads are still documented here for backwards compatibility, but are strongly discouraged.
To use this method, please ensure that you have specified the uploadStoreType parameter in your initial request as "S3". If your initial request uses the S3 option for the uploadStoreType, then your response will include a "s3Properties" object. This object contains all of the information of an s3 bucket made available for you to upload your files.
Perform a non-presigned upload by specifying the "S3" uploadStoreType and presignRequired: false properties in your original POST /uploads request when creating the upload. This will return different information in the s3Properties object in the response, specifically the accessKeyId, secretKey, and sessionToken properties. These properties can be used to perform the file uploads to S3 using the AWS SDK, which will automatically support uploading single files greater than 5GB.
The following code snippet shows an example of how to use the AWS SDK to upload files using the information returned for a non-presigned upload.
// First, create an AWS S3 object var uploadResponse = <JSON response from POST /uploads request>; var s3props = uploadResponse.s3Properties; var s3 = new AWS.S3({ apiVersion: '2006-03-01', params: { Bucket: s3props.bucket }, credentials: new AWS.Credentials(s3props.accessKeyId, s3props.secretKey, s3props.sessionToken), region: "us-west-1" }); // create the upload var upload = s3.upload({ Key: s3props.keyPrefix + '/' + <filename>, Body: <file object> }); // to watch the progress of the file upload upload.on('httpUploadProgress', function(e) { // do stuff with 'e' }); // send the upload and handle success or failure upload.send(function(err, data) { if (err) { // error handling } else { // success handling } });
Presigned uploads are deprecated and will be removed in a future version. They are only documented here for backwards compatibility.
Once you have access to the necessary S3 bucket information, you will need to make individual HTTP POST request for each of the files you intend to upload. The POST URL will be in the format below:
https://{s3bucket}.{domain}/
The value for the s3bucket is the bucket property in your s3Propeties object, and the domain in this case is "s3-us-west-2.amazon.com". With this POST request you will also need to include additional properties to the multipart/form-data containing the file you wish to upload and the remaining s3bucket information.
Title | Required | Type | Values | Description |
key | true | String | Any valid AWS key | The value for this property is in your s3Properties object. s3Properties.keyPrefix + "/{filename}", is the correct format for this property value, where {filename} is the name of the file you are currently uploading. |
AWSAccessKeyId | true | String | Any valid AWS AccesKeyId | The value for this property is your s3Properties object. The s3Properties.accessKeyId value should be the value for this property. |
policy | true | String | Any valid AWS policy | The value for this property is your s3Properties object. The s3Properties.encodedPolicy value should be the value for this property. |
signature | true | String | Any valid AWS signature | The value for this property is your s3Properties object. The s3Properties.signature value should be the value for this property. |
file | true | FileObject | Any valid File Object | File object for the client file that will be uploaded to S3. |
Send post request to the following URL with the file as form data in the request payload.
After you have initiated your upload request, you will need to begin adding files to this upload job. Using the HTTP Post method, you will need to make request to the following endpoint:
POST /api/v1/uploads/{uploadId}/files
For each file you will need to make a separate request containing mutltipart/form-data file objects. Continue making these request until all files for this particular upload have been completed. After all files have been added, complete your upload by following the instructions below in the "Complete client upload" section.
HTTP Status Code | Description | Payload |
201 | Successfully completed file upload | There is no response payload for this request |
Once you have uploaded all of your client files and received successful responses for each file you are ready to complete your upload. To do this please see the "Complete client upload" section to finalize your upload.
HTTP Code | Description |
---|---|
401 | Error, not authorized to access the catalog |
403 | Error, forbidden to access the item |
404 | Error, item not found |
500 | Unexpected error while processing the request |
After you have successfully uploaded all of your files, please see the "Complete upload" section to finalize your upload.
When all files are uploaded, use this request to trigger completion of file upload and cataloging can continue.
This is the final request necessary for completing an upload and adding files to the Smart M.App Platform. Using the uploadId that was returned by the initial request, perform a final HTTP request to the following endpoint:
PUT /api/v1/uploads/{uploadId}/complete
HTTP Status Code | Description | Payload |
201 | Successfully completed client upload | There is no response payload for this request |
HTTP Code | Description |
---|---|
401 | Error, not authorized to access the catalog |
403 | Error, forbidden to access the item |
500 | Unexpected error while processing the request |
This endpoint will return all information about a specific upload job.
While a upload job is running, users will naturally want to check the status of this job. This is done by using the uploadId (returned in the HTTP response of the initial request) to make another request to the following endpoint:
GET /api/v1/uploads/{uploadId}
Note: When a upload is started in M.App Chest, we periodically make request against this endpoint to update the user on the status of the running upload job. Once the job is complete, we update the user of the progress and inform the user when all processing is finailized.
HTTP Status Code | Description | Payload |
200 | Successfully retrieved status for a running upload |
{ "activeStorageProperties": { "activeStoragePath": "string" }, "crawledDatasetIds": {}, "crawledFolderIds": {}, "displayableStatus": "string", "endTime": "string", "errors": "string", "id": "string", "progress": "string", "registerProperties": { "copyToInternalBucket": true, "downloadIsAllowed": true, "footPrintType": "Extent", "generateThumbnail": true, "parentFolderId": "string", "publishIsAllowed": true, "publishProperties": { "autoPublish": true, "capabilitiesTitle": "string", "ecwpEnabled": true, "geomapserviceEnabled": true, "jpipEnabled": true, "outputCSList": [ "string" ], "parentVirtualFolder": "string", "virtualName": "string", "wmsEnabled": true, "wmtsEnabled": true }, "s3Keys": [ "string" ], "sourceBucket": "string", "subFolder": "string", "uploadStoreType": "Direct" }, "s3Properties": { "accessKeyId": "string", "bucket": "string", "encodedPolicy": "string", "keyPrefix": "string", "signature": "string" }, "startTime": "string", "state": "WAITING", "uploadComplete": true, "uploadedFiles": [ "string" ] } |
HTTP Status Code | Description |
---|---|
401 | Error, not authorized to access the catalog |
403 | Error, forbidden to access the item |
404 | Error, item not found |
500 | Unexpected error while processing the request |
Takes a single upload id and stops all effort of uploading/crawling of the files.
PUT /api/v1/uploads/{uploadId}/cancel
HTTP Status Code | Description | Payload |
202 | Successfully canceled a running upload | There is no payload for this response |
HTTP Status Code | Description |
---|---|
400 | Error with details in the response body. |
401 | Error, not authorized to access the catalog |
403 | Error, forbidden to access the item |
500 | Unexpected error while processing the request |
Deletes an existing upload. This will stop all effort to upload the files to the catalog.
DELETE /api/v1/uploads/{uploadId}
HTTP Status Code | Description | Payload |
204 | Successfully removed existing upload | There is no payload for this response |
HTTP Status Code | Description |
---|---|
400 | Error with details in the response body |
401 | Error, not authorized to access the catalog |
403 | Error, forbidden to access the item |
404 | Error, item not found |
500 | Unexpected error while processing the request |