Hexagon Geospatial
MENU

Spatial Modeler Tutorials

Learn more about our products, find answers, get the latest updates, and connect with other Hexagon Geospatial product users, or get support from our professional service team.
Showing results for 
Search instead for 
Do you mean 

Handling NoData in Spatial Modeler

by Technical Evangelist ‎12-15-2016 05:42 AM - edited ‎08-23-2017 10:23 AM (3,552 Views)

Description:

 

This article isn't about a specific Spatial Model. It's a general discussion on NoData and how NoData can be handled within Spatial Modeler. It arose out of a conversation which started in the ERDAS IMAGINE Support community on the best practices for stretching data to 8bit. One of the topics that thread didn't get into was the effect NoData might have on such tasks and so I wanted to explore that topic more in the form of an Article.

 

First off, what is NoData? Fundamentally, it means geospatial locations where information represented by the particular layer is unknown, or does not exist - i.e. there is "no data". There's a basic definition here. But that page really only discusses value-based NoData. Value-based NoData is when a value within the information layer itself is reserved to denote the locations where information does not exist (also sometimes known as a Null Value). For Unsigned 8-bit Integer raster layers the DN value 0 is often used for this purpose. A disadvantage of value-based NoData is that it consumes one of what may be a very limited number of values available in the data set - for example, using the value 0 as NoData in u8 data reduces the number of valid values to 255. It's also very difficult to use if the raster format has been lossily compressed (so that there is no longer a single value that can represent locations with no data).

 

The NoData Help page also makes the incomplete statement that "Areas of NoData are not automatically identified as such by the operations that create them". Spatial Modeler has an advantage in internally using mask-based NoData (more on that later) and can automatically create and add to that mask if warranted by the operations of the Spatial Model. For example, if you reproject a raster layer as part of a Spatial Model, NoData will be automatically injected into the resulting raster at the locations where there was no available input data to create valid information for the output.

 

NoData.PNG


As a Spatial Model executes, NoData information is retained, combined and used in the form of a mask. Think of this as a 1bit raster layer which denotes locations that have information and locations that don't. The mask is essentially a separate layer associated with each data layer in the raster stream and therefore does not reduce the data range of those rasters. Processing will only occur at locations of Data (no NoData). At locations where the mask specifies NoData an Operator will, in general, not execute. 

 

But that rule can vary depending on the needs of the specific Operator being used. For example, if using a Convolution (filtering) operation with a 3x3 input kernel window, the convolution will not be applied at pixel locations which fall under the NoData mask. However when evaluating a location at a valid pixel location, the 3x3 moving window may include neighboring pixel cells that do fall under NoData locations. In this instance the operation of the Focal function is modified to take into account the more limited number of valid pixel values present in the moving window and a valid value is created at the output center location.

 

But how does this NoData mask get created, updated and (quite importantly) output, when running a Spatial Model?

 

I think there’s a lot of confusion over how NoData works in Spatial Modeler, so let me see if I can explain how it works using a simple Spatial Model created to take 3 individual TIFF band files of Landsat 8 data and create a stretched 8-bit, 3-band output file suitable for display as a backdrop (in a WMS service, for example):

 

 Model to Stack and Stretch to 8-bit for a True-Color Display
 NoData_Model.png

 

 

  1. The input Landsat TIFF files do not have a NoData value set, but it is generally accepted that the USGS processes them so that DNs of 0 can be assumed to be NoData. However as inputs to the Raster Input operators, the three TIFF files do not have a NoData value specified. So the Spatial Model will build initial NoData masks (one for each raster stream), but those masks will be empty - i.e. all locations will be considered valid locations at the beginning of this Spatial Model.

    If the input files had included definitions of NoData locations, those locations would be identified in the initial masks the Spatial Model constructs.

     Band 4 of a Landsat 8 image, displayed with the background canvas color set to red 
     B4.png


  2. The Set To NoData operators take the input rasters and adds each pixel location that has the specified value (0 in the case of this Model) to the NoData mask. It does not make that value NoData. Spatial Modeler uses a NoData mask concept, so it is just the locations of the identified pixels which are added to the NoData mask. At that point there’s no longer any concept of a value being used – just locations where there is no data and which Spatial Modeler will therefore not bother processing.

  3. Then Stack Layers is used to combine the individual raster streams into a three-band raster (with the desired order so you don’t have to re-order them later). Combining the raster streams does not combine the three separate NoData masks into a single one. The NoData masks remain as per-layer (or per-band) masks.
  4. The Statistics operator does not need to have an Ignore Value specified in this instance (unless you wanted to ignore other values, of course). Pixels falling under the NoData Mask locations will not be considered when accumulating the statistics and therefore “zeros” effectively wont be included.

  5. The Stretch operator then rescales the input data using the specified percentage points. But remember that this is going to stretch your input range to an output range of 0 – 255. Any pixel that, on input to the Model, had a DN value of 0 already falls under the NoData mask so the input range for the Left Right Clip Stretch wont have any 0s. The range might be, for example, from 5,635 to 20,484 and the percentage points you specify might work out to 6,000 and 20,000. The data between 6,000 and 20,000 will be linearly stretched to an output range of 0 to 255. So all input values between 5,635 and 6,000 will become DN value 0 on output from that operator. These pixels with DN 0 are going to be valid pixels – they do not fall under the NoData mask location and 0 does not become NoData (at this point). I.e. these new pixels with DN 0 do not add to the NoData mask.

  6. But, they may be a problem later, depending on what formats we write the results out to (see point 8 below). So the next step of the model checks every pixel (that doesn’t fall under the NoData Mask) to see if it currently has a value of 0 and, if it does, it makes it 1 instead.

  7. Otherwise it retains the input value. Note that this has no effect on pixels at the NoData locations. There’s no data at those locations and so no processing occurs in the Equals test or the Either/Or operator.

  8. Then we write the results out to an image file. Here’s where the big NoData problem raises its head. Different file formats have different abilities to handle the concept of NoData (and some have no concept of it at all).

    A format such as ECW has full NoData mask support (which is vital when dealing with lossy compression techniques) and so Spatial Modeler simply converts its NoData mask into the ECW file’s NoData mask.

    But a format such as IMG doesn’t have support for a mask – just a value which can be identified as specifying the NoData locations (known as Value-based NoData). Since our output data is now Unsigned 8-bit because of the stretch we don’t have a lot of “spare” values to play with. Normally a value of 0 is used to identify NoData in Unsigned 8-bit data. Hence the need for what we did at step 6 – our “valid” dark pixels have all been converted to a DN of 1 or higher, so there’s no confusion when the NoData Mask locations get assigned an output file DN value of 0 (and the file header denotes that 0 should be considered NoData). If we had not performed the recode at step 6 our output IMG file would have pixels that fell under the NoData mask assigned value 0 and we would have “valid” image pixels of value 0 and, since there’s no way to differentiate, all those locations now get considered NoData (and you end up with “holes” in your data when displayed with transparent background over other data). So make sure you think about the potential data ranges of your models output and the capabilities of your likely output formats when designing your models – you may need to do something like step 6 to make sure there’s no confusion between “real” data locations and true NoData locations.

  9. In the Raster Output operator I have not specified a NoData value. This isn’t needed because the default value for the NoData Mask to be converted to (for u8 data) is 0 (and can be changed in Preferences if desired). So 0 already becomes the output NoData value for value-based NoData in the output IMG. You really only need to specify this value if you don’t want to use the Preference-driven default value for the data type you are producing. For example, if you left this model producing Signed 32-bit data (which would be the default) the default NoData value will be -2147483648, which is actually a good value to use because it’s unlikely to be confused with “valid” pixel values. If you instead specified a NoData value in the Raster Output of 0 you would run a much higher risk of confusion between locations of no data and real image locations (which happen to have been processed to a value of 0).

    Think carefully before specifying a NoData Value in the Raster Output.

 

True Color 8-bit RGB IMG which did not remap 0s (step 6 & 7) - note the holes showing red background through
 Landsat_w_Holes.png
 
True Color 8-bit RGB IMG which did remap 0s to 1s (step 6 & 7) - no holes
 Landsat_wo_Holes.png
 
True Color 8-bit RGB ECW which did not remap 0s (step 6 & 7) - no holes because the ECW retained a NoData Mask independant of valid pixels with value 0
Landsat_ECW.png

 

Comments
by Technical Evangelist
on ‎05-09-2017 05:51 AM
by Technical Evangelist
on ‎08-23-2017 07:48 AM

Here's another point worth considering with regard to NoData. Once a file has been produced you cannot (easily) change the DN value of NoData locations - you can only alter the DN value to be treated as NoData locations.

 

When using the "Set/Clear NoData Value...." option in View/Edit Image Metadata / Image Info, or the Edit Image Metadata utility, you are only changing (or setting for the first time) the DN value to be considered as NoData. You are not “raster editing” the image to change locations of NoData from one value to another.

 

Consider an 8-bit image made entirely of 0s on the left half of the image and 254s on the right half. Initially the value-based NoData might be defined as 0. So the left half of the image would be considered NoData and would show up as transparent (a big hole) and would not be considered in (most) processing applied using Spatial Modeler.

 

If you then use View/Edit Image Metadata to change the value to be considered as NoData to 255, none of the image would be considered NoData (since none of the pixels have a value of 255). The entire thing would be “solid” and considered in any Spatial Modeler processing. You have not changed pixel values of 0 to a value of 255 - you have simply said that existing values of 255 should be treated as NoData.

 

If you instead change the value to be considered as NoData to 254, the right hand half of the image (not the left) would now be considered to be the locations of NoData. So the right half of the image would show up as transparent, etc.

 

As discussed in the article above, this is why it’s very important to get the locations you want to be NoData set to the value you want to use to represent NoData correctly when creating the image. You also need to make sure you do so in a way that is not going to accidentally get “real image” locations lumped in with the value-based NoData when output to a file format. Using 0 as the NoData value on 11-bit imagery increases the likelihood of confusing NoData locations with “real” image locations that are dark. That's why the default when outputting 16-bit integer image from Spatial Model is to use the maximum value of the possible data range, not 0. But you can control this in your Model by defining in the Raster Output operator what value you want the NoData mask to be converted to in the output file (using the NoDataValue port).

 

If you have an existing file where the NoData locations are using a value-based NoData value you consider “wrong”, the only option is to literally perform a raster edit on the existing file to recode the existing value to the value you want and then set the new value as the value-based NoData value. Or you can run the image through a Spatial Model to do something similar, but that will produce a new output file. But make sure you do not recode to a value that is already in use for non-NoData locations. In the example above, you would not want to recode the 0s on the left side of the image to 254 and then set 254 as the NoData value!

by Technical Evangelist
on ‎03-30-2018 11:06 AM
by Technical Evangelist
on ‎07-25-2018 03:54 AM
Question moved to Support thread so I can attach a file to my response.
by Technical Evangelist
on ‎07-25-2018 07:59 AM

Hi Kristina,

 

First step would be to upgrade to ERDAS IMAGINE 2018! You can probably do this in 2015, but I cant guarantee it.

 

Next, fire up the Spatial Model Editor. Use a Raster Input operator to access your Landsat data and remember to declare it as Float (since the scaling step will require floating point calculations).

 

Create a "outside valid range" mask by selecting all pixels <0 or > 10000 and setting those locations to 10001. Bear in mind that this will be a per-band selection, so you might want to get more complex with the selection criteria if you want to exclude an entire pixel if any of the bands at that location are outside the range.

 

Use Set to NoData operator to set 10001 to NoData.

 

Multiply by 0.0001 to scale 0 - 1

 

Send to a Raster Output.

 

That should do it.

Courses
Contributors