11-15-2017 11:49 AM - edited 11-15-2017 11:50 AM
I think there is some confusion of the purpose of the Define Processing Area operator.
It's there to manipulate the parameters of the working window of the Spatial Model. It is not intended to create/add new information.
The classic example is when you have two rasters being input to the Model. Do you want the working window to be the Intersection of the two spatial footprints, or the Union? Or a completely different boundary which you specify? You can do this using Define Processing Area because you are manipulatign the existing default boundary.
Same with the Coordinate Reference System (CRS). If there are two input rasters with differing CRS', you can specify which of the two inputs is used to define the projection of the working window (processing area). Or you can even specify a completely different CRS and the data will be reprojected on the fly to that CRS. But to do the latter, the two input images must have CRS' in order for the reprojection to work.
What Define Processing Area is not intended to do is set a CRS if none exists. The input data must have existing CRS' (and extents, and pixel sizes, etc)
If your input data lacks a CRS, you should correct it before using it in Spatial Modeler using the Image Command Tool that has previously been mentioned. In theory you may also be able to use the Command Line operator to run the Image Command Tool as part of the model and set the CRS if you wanted. But there's no existing Operator specifically designed to add a CRS to a raster stream if none already exists.
So the enhancement request here isn't to take the CRS definition off of Define Processing Area and make it available on its own - that still wouldn't do what you are asking for. You need a brand new Operator (or try to use the Command Line operator).
11-15-2017 12:29 PM
My original approach was using the Imagine imgcopy and imagecommand to convert png to geotif and add the CRS. Combined with the batch capability in Imagine that seemed good. However I found it did not seem stable running on clients server for 10's of thousands of images - both imgcopy and imagecommand would ocassionally crash.
Switching to modeler seems stable - though I have not given it a real good test run or reviewed comparitive performance.
While we have imagecommand working within the model, given previous expericence with imagecommand I'm not sure how stable that will be so will probably avoid it.
Will give the alternative model provided a try out - thanks Marina.
Will also add seperate operator to ideation board later. I guess another alternative would be an ideation post for Imagine to recognise an additional source for the SRS, such as an .PRJ file. e.g. source raster could be .png, pgw, .prj giving raster, georeference info and SRS.
I also remembered last night that the old ISRU has ability to add GeoTiff tags to an image so will play with that as well.
Our final target is to get all the PNG files into Apollo Core as GeoTiffs to serve up as a thematic basemap along with existing aerial ECWs etc. The PNGs will be periodically regenerated. Looking positive so far.
11-15-2017 02:13 PM
I think Johnnie already added the Idea for you:
11-16-2017 12:49 AM
Hiya. Yup, aware (and frequently use) the Define Processing Area operators as you describe. However, users (me!) frequently repurpose functionality to suite their needs. So, even though the CRS options in the operator weren't meant to assign a CRS to date without projection information, it does do the job rather nicely.
I certainly don't want to remove this functionality from this operator, but a seperate operator would be good.
11-16-2017 01:31 AM - edited 11-16-2017 01:32 AM
I stuck in two details here:
"My original approach was using the Imagine imgcopy and imagecommand to convert png to geotif and add the CRS. Combined with the batch capability in Imagine that seemed good. However I found it did not seem stable running on clients server for 10's of thousands of images - both imgcopy and imagecommand would ocassionally crash."
This I disagree - imagecommand will work stable the problems come if input data has some variation. In nutshell one broken input file etc. will lead to troubles. If you do things in coordinated way it will work. My approach for this task would be
1) Import all png to img
2) add map model to all img if not existing - basically unit=meters is typically needed before normal coordinates can be batched to files
3) Set projection in batch
4) export to geotif
I am pretty sure that this approach works. It just requires 4 separate Image Command Tool batch instead of single batch. Based on my experience amount of odd issues decreases dramatically if you limit your work in .img files.
"Will also add seperate operator to ideation board later. I guess another alternative would be an ideation post for Imagine to recognise an additional source for the SRS, such as an .PRJ file. e.g. source raster could be .png, pgw, .prj giving raster, georeference info and SRS"
This has been under discussion in past. My opinion and wish is this. Whatever defines coordinates it should be trusted as corrrect one. Soif EPSG code is seen in some file or what ever information is get that should automatically transfer to Imagine file metadata. Now current situation is that first is image headers read, then world file etc. and when ever first information is found that is trusted. This comes ahead in cases where TIF has both world file and GeoTIFF header based coordinates. Sometimes these two conflict and only another is used for Imagine. I think all information found should be used - I just have no idea how to do it :-)
One of my cases is such that user wants to set a default projection as they normally work just with one projection. So setting like "if projection is not given assume it always be EPSG 1234" - that would be usable sometimes too.
So world of formats and coordinates is definetly not complete yet
11-16-2017 08:22 AM
I'm absolutely not opposed to repurposing operators for other purposes. That's what Spatial Model Editor is all about after all.
But I would very much caution against using Define Processing Area in this way. It is not designed to add a CRS where none (or incomplete information) previously existed. I'm very surprised that this is working for you at all.
11-17-2017 09:14 PM
Had a play with PNG -> IMG -> Tif with a small data sample.
Initially avoided that process as it increases processing time. Turns out by about 50% based on initial sample (not enough to be truly conclusive).
However as part of that trial we noticed that the IMG file sizes were smaller than Tif with LZW compression.
My original assumption about using TIF with compression was that they were significantly smaller than IMG.
At least for our initial test data that turns out not to be the case.
Given the target is making thematic basemaps available in Apollo Core, my revised thinking is to ditch using Tif and just stick to IMG format giving smaller file size, improved conversion stability and (hopefully) equiv or better performance in Apollo Core.
Off course we need to take that to a bigger data sample to validate, but seems promising.
11-17-2017 12:24 AM - edited 11-17-2017 01:03 AM
I am pretty suprised that .img is smaller than -lzw .tif. THat is the case VERY rarely
Your case key thing was thematic and it happened to be just correct style thematic. So that compresses pretty well with ancient RLE etc. compressions available in .img and in some cases they can be equal or nearby the LZW you used in .tif. LZW is in generally quite good compression and typically the smallest loseless compression available. Wavelet used in ECW/JPG2000 works well only with natural imagery and typically increases filesize in thematic files compared to lzw tif due wawelenght forces single band files to RGB. My rules of thumb for compression are:
For natural images smalles files can be get by (data originated from some sort of camera) ECW/JPG2000 and in some cases LZW TIF
Thematic data typically the best is LZW TIF and in some cases Run Lenght Encoding (RLE) used in .img or .tif might do the same thing. The key thing in thematic data is to keep that thematic and not turn to RGB as RGB means single band conversion to three bands and in same time spoils datas information. So thematic to RGB is an extremely bad move - that you unfortunately must sometimes do if colortables will not follow files properly between different tools.
I do my work always as long as I can in .img and only input&output is put in different formats depending on case. It costs some time but typically wins it back as there is less hassling. Of course if environment is mixed tools you may have to stick in tif that works in all softwares but if your work happens totally in Hexagon tools then .img might be the best. TIF should be nearly as good format to Erdas as .img and it depends case by case what that nearly in each time means. In Apollo side I am not sure is .tif even better than .img but in desktop Imagine .img is definetly better.
I think two format conversion is not a major time consumer as you can always let your computer do the work overnight in batch etc. computer is working in fairly decent hourly salary and union does not order it to keep breaks. The major issue comes in conversion if data starts to grow really huge and yo run out of disk space constantly but as long as you can fit your work to one disk and processes run through in few hours then this conversion does not cost really that much time. If processes start to last weeks then 20-30% improvements in speed start to matter - in one hour process it is just one more cup of coffee which is not a major timing issue.
11-17-2017 06:20 AM
We're off topic here and perhaps should start a new thread if we wish to discuss compression techniques.
But just a note of correction - JPEG 2000's compression has a lossless mode which works very nicely with single band / thematic data. I use it on worldwide DEMs (especially those with sea level being flat - compresses a treat!). It does not need to convert the data to three-band RGB.