Hexagon Geospatial
MENU

Spatial Modeler Tutorials

Learn more about our products, find answers, get the latest updates, and connect with other Hexagon Geospatial product users, or get support from our professional service team.
Showing results for 
Search instead for 
Do you mean 

Handling NoData in Spatial Modeler

by Technical Evangelist ‎12-15-2016 05:42 AM - edited ‎08-23-2017 10:23 AM (4,122 Views)

Description:

 

This article isn't about a specific Spatial Model. It's a general discussion on NoData and how NoData can be handled within Spatial Modeler. It arose out of a conversation which started in the ERDAS IMAGINE Support community on the best practices for stretching data to 8bit. One of the topics that thread didn't get into was the effect NoData might have on such tasks and so I wanted to explore that topic more in the form of an Article.

 

First off, what is NoData? Fundamentally, it means geospatial locations where information represented by the particular layer is unknown, or does not exist - i.e. there is "no data". There's a basic definition here. But that page really only discusses value-based NoData. Value-based NoData is when a value within the information layer itself is reserved to denote the locations where information does not exist (also sometimes known as a Null Value). For Unsigned 8-bit Integer raster layers the DN value 0 is often used for this purpose. A disadvantage of value-based NoData is that it consumes one of what may be a very limited number of values available in the data set - for example, using the value 0 as NoData in u8 data reduces the number of valid values to 255. It's also very difficult to use if the raster format has been lossily compressed (so that there is no longer a single value that can represent locations with no data).

 

The NoData Help page also makes the incomplete statement that "Areas of NoData are not automatically identified as such by the operations that create them". Spatial Modeler has an advantage in internally using mask-based NoData (more on that later) and can automatically create and add to that mask if warranted by the operations of the Spatial Model. For example, if you reproject a raster layer as part of a Spatial Model, NoData will be automatically injected into the resulting raster at the locations where there was no available input data to create valid information for the output.

 

NoData.PNG


As a Spatial Model executes, NoData information is retained, combined and used in the form of a mask. Think of this as a 1bit raster layer which denotes locations that have information and locations that don't. The mask is essentially a separate layer associated with each data layer in the raster stream and therefore does not reduce the data range of those rasters. Processing will only occur at locations of Data (no NoData). At locations where the mask specifies NoData an Operator will, in general, not execute. 

 

But that rule can vary depending on the needs of the specific Operator being used. For example, if using a Convolution (filtering) operation with a 3x3 input kernel window, the convolution will not be applied at pixel locations which fall under the NoData mask. However when evaluating a location at a valid pixel location, the 3x3 moving window may include neighboring pixel cells that do fall under NoData locations. In this instance the operation of the Focal function is modified to take into account the more limited number of valid pixel values present in the moving window and a valid value is created at the output center location.

 

But how does this NoData mask get created, updated and (quite importantly) output, when running a Spatial Model?

 

I think there’s a lot of confusion over how NoData works in Spatial Modeler, so let me see if I can explain how it works using a simple Spatial Model created to take 3 individual TIFF band files of Landsat 8 data and create a stretched 8-bit, 3-band output file suitable for display as a backdrop (in a WMS service, for example):

 

 Model to Stack and Stretch to 8-bit for a True-Color Display
 NoData_Model.png

 

 

  1. The input Landsat TIFF files do not have a NoData value set, but it is generally accepted that the USGS processes them so that DNs of 0 can be assumed to be NoData. However as inputs to the Raster Input operators, the three TIFF files do not have a NoData value specified. So the Spatial Model will build initial NoData masks (one for each raster stream), but those masks will be empty - i.e. all locations will be considered valid locations at the beginning of this Spatial Model.

    If the input files had included definitions of NoData locations, those locations would be identified in the initial masks the Spatial Model constructs.

     Band 4 of a Landsat 8 image, displayed with the background canvas color set to red 
     B4.png


  2. The Set To NoData operators take the input rasters and adds each pixel location that has the specified value (0 in the case of this Model) to the NoData mask. It does not make that value NoData. Spatial Modeler uses a NoData mask concept, so it is just the locations of the identified pixels which are added to the NoData mask. At that point there’s no longer any concept of a value being used – just locations where there is no data and which Spatial Modeler will therefore not bother processing.

  3. Then Stack Layers is used to combine the individual raster streams into a three-band raster (with the desired order so you don’t have to re-order them later). Combining the raster streams does not combine the three separate NoData masks into a single one. The NoData masks remain as per-layer (or per-band) masks.
  4. The Statistics operator does not need to have an Ignore Value specified in this instance (unless you wanted to ignore other values, of course). Pixels falling under the NoData Mask locations will not be considered when accumulating the statistics and therefore “zeros” effectively wont be included.

  5. The Stretch operator then rescales the input data using the specified percentage points. But remember that this is going to stretch your input range to an output range of 0 – 255. Any pixel that, on input to the Model, had a DN value of 0 already falls under the NoData mask so the input range for the Left Right Clip Stretch wont have any 0s. The range might be, for example, from 5,635 to 20,484 and the percentage points you specify might work out to 6,000 and 20,000. The data between 6,000 and 20,000 will be linearly stretched to an output range of 0 to 255. So all input values between 5,635 and 6,000 will become DN value 0 on output from that operator. These pixels with DN 0 are going to be valid pixels – they do not fall under the NoData mask location and 0 does not become NoData (at this point). I.e. these new pixels with DN 0 do not add to the NoData mask.

  6. But, they may be a problem later, depending on what formats we write the results out to (see point 8 below). So the next step of the model checks every pixel (that doesn’t fall under the NoData Mask) to see if it currently has a value of 0 and, if it does, it makes it 1 instead.

  7. Otherwise it retains the input value. Note that this has no effect on pixels at the NoData locations. There’s no data at those locations and so no processing occurs in the Equals test or the Either/Or operator.

  8. Then we write the results out to an image file. Here’s where the big NoData problem raises its head. Different file formats have different abilities to handle the concept of NoData (and some have no concept of it at all).

    A format such as ECW has full NoData mask support (which is vital when dealing with lossy compression techniques) and so Spatial Modeler simply converts its NoData mask into the ECW file’s NoData mask.

    But a format such as IMG doesn’t have support for a mask – just a value which can be identified as specifying the NoData locations (known as Value-based NoData). Since our output data is now Unsigned 8-bit because of the stretch we don’t have a lot of “spare” values to play with. Normally a value of 0 is used to identify NoData in Unsigned 8-bit data. Hence the need for what we did at step 6 – our “valid” dark pixels have all been converted to a DN of 1 or higher, so there’s no confusion when the NoData Mask locations get assigned an output file DN value of 0 (and the file header denotes that 0 should be considered NoData). If we had not performed the recode at step 6 our output IMG file would have pixels that fell under the NoData mask assigned value 0 and we would have “valid” image pixels of value 0 and, since there’s no way to differentiate, all those locations now get considered NoData (and you end up with “holes” in your data when displayed with transparent background over other data). So make sure you think about the potential data ranges of your models output and the capabilities of your likely output formats when designing your models – you may need to do something like step 6 to make sure there’s no confusion between “real” data locations and true NoData locations.

  9. In the Raster Output operator I have not specified a NoData value. This isn’t needed because the default value for the NoData Mask to be converted to (for u8 data) is 0 (and can be changed in Preferences if desired). So 0 already becomes the output NoData value for value-based NoData in the output IMG. You really only need to specify this value if you don’t want to use the Preference-driven default value for the data type you are producing. For example, if you left this model producing Signed 32-bit data (which would be the default) the default NoData value will be -2147483648, which is actually a good value to use because it’s unlikely to be confused with “valid” pixel values. If you instead specified a NoData value in the Raster Output of 0 you would run a much higher risk of confusion between locations of no data and real image locations (which happen to have been processed to a value of 0).

    Think carefully before specifying a NoData Value in the Raster Output.

 

True Color 8-bit RGB IMG which did not remap 0s (step 6 & 7) - note the holes showing red background through
 Landsat_w_Holes.png
 
True Color 8-bit RGB IMG which did remap 0s to 1s (step 6 & 7) - no holes
 Landsat_wo_Holes.png
 
True Color 8-bit RGB ECW which did not remap 0s (step 6 & 7) - no holes because the ECW retained a NoData Mask independant of valid pixels with value 0
Landsat_ECW.png

 

Comments
by lyseng
on ‎07-25-2018 08:52 AM

Hi Ian,

 

Thanks ever so much, I will give this a try ! I'm afraid I cannot upgrade as I am an undergraduate student using the software available on uni computers and they only have the 2015 version.

 

Will let you know how it goes.

 

 

by Technical Evangelist
‎07-25-2018 10:51 AM - edited ‎07-25-2018 12:26 PM

Something like this would do it (model also attached). Note that the data I tested with had a valid range of 0 - 20,000, so you'll need to adjust the numbers to match what level pre-processing data you have.

 

scale_nodata.PNG

 

Moved the solution and attached model to this Support thread:

 

http://community.hexagongeospatial.com/t5/Support-ERDAS-IMAGINE/Re-Handling-NoData-in-Spatial-Modele...

 

by kansy
‎02-19-2019 01:37 AM - edited ‎02-19-2019 01:38 AM

Hello Anderson,

 

I just read this insightful articlel of yours. Thank you very much for providing the tips and energy to solving geospatial problems.

 

About 85% of the scenes in my AOI are SLC-off

 

Can I use this tip with Handling NoData in Spatial Modeler to deal with  the gaps?

or is there a simpler way of filling these gaps more professionally with the latest version 16.5?

 

Thank you in adavnce

 

P.S.  I would be grateful if you could list as many possibilities as there are for filling SLC-off gaps

 

Kansy

 
by Technical Evangelist
on ‎02-19-2019 06:19 AM

Hi Kansy,

 

Well, the world is your oyster with Spatial Modeler. Anything from a simple Focal Mean operation applied at locations with 0 value (and ignoring zeros as values in the calculation), to something like htis:

 

https://community.hexagongeospatial.com/t5/Spatial-Modeler-Tutorials/Cloud-Replacement-using-Linear-...

 

Cheers

 

 

Courses
Contributors