applier¶
Basic tools for setting up a function to be applied over
a raster processing chain. The rios.applier.apply()
function is the main
point of entry in this module.
- class rios.applier.ApplierControls[source]¶
Controls for the operation of rios, for use with the
rios.applier.apply()
function.This object starts with default values for all controls, and has methods for setting each of them to something else.
- Attributes are:
windowxsize X size of rios block (pixels)
windowysize Y size of rios block (pixels)
overlap Number of pixels in margin for block overlaps
footprint
rios.applier.INTERSECTION
orrios.applier.UNION
orrios.applier.BOUNDS_FROM_REFERENCE
drivername GDAL driver short name for output
creationoptions GDAL creation options for output
thematic True/False for thematic outputs
layernames List of layer names for outputs
referenceImage Image for reference projection and grid that inputs will be resampled to.
referencePixgrid pixelGrid for reference projection and grid
loggingstream file-like for logging of messages
progress progress object
statsIgnore global stats ignore value for output (i.e. null value)
calcStats True/False to signal calculate statistics and pyramids
omitPyramids True/False to omit pyramids when doing stats
overviewLevels List of level factors used when calculating output image overviews
overviewMinDim Minimum dimension of highest overview level
overviewAggType Aggregation type for calculating overviews
tempdir Name of directory for temp files (resampling, etc.)
resampleMethod String for resample method, when required (as per GDAL)
numThreads Number of parallel threads used for processing each image block
jobManagerType Which
rios.parallel.jobmanager.JobManager
sub-class to use for parallel processing (by name)autoColorTableType Type of color table to be automatically added to thematic output rasters
allowOverviewsGdalwarp Allow use of overviews in input resample (dangerous, do not use)
approxStats Allow approx stats (much faster)
- Options relating to vector input files
burnvalue Value to burn into raster from vector
filtersql SQL where clause used to filter vector features
alltouched Boolean. If True, all pixels touched are included in vector.
burnattribute Name of vector attribute used to supply burnvalue
vectorlayer Number (or name) of vector layer
vectordatatype Numpy datatype to use for raster created from vector
vectornull Rasterised vector is initialised to this value, before burning
Default values are provided for all attributes, and can then be over-ridden with the ‘set’ methods given.
Some ‘set’ methods take an optional imagename argument. If given, this should be the same internal name used for a given image as in the
rios.applier.FilenameAssociations
objects. This is the internal name for that image, and the method will set the parameter in question for that specific image, which will over-ride the global value set when no imagename is given.- getOptionForImagename(option, imagename)[source]¶
Returns the value of a particular option for the given imagename. If only the global option has been set, then that is returned, but if a specific value has been set for the given imagename, then use that.
The imagename is the same internal name as used for the image in the
rios.applier.FilenameAssociations
objects.
- makeResampleDict(imageDict)[source]¶
Make a dictionary of resample methods, one for every image name in the given dictionary. This method is for internal use only.
- selectInputImageLayers(layerselection, imagename=None)[source]¶
Set which layers are to be read from the input image(s). Default will read all layers. If imagename is given, selection will be for that image only. The layerselection parameter should be a list of layer numbers. Layer numbers follow GDAL conventions, i.e. a layer number of 1 refers to the first layer in the file. Can be much more efficient when only using a small subset of layers from the inputs.
- setAllowOverviewsGdalwarp(allowOverviewsGdalwarp)[source]¶
This option is provided purely for testing purposes, and it is recommended that this never be used operationally.
In GDAL >= 2.0, the default behaviour of gdalwarp was modified so that it will use overviews during a resample to a lower resolution. By default, RIOS now switches this off again (by giving gdalwarp the ‘-ovr NONE’ switch), as this is very unreliable behaviour. Overviews can be calculated by many different methods, and the user of the file cannot tell how they were done.
In order to allow users to assess the damage done by this, we provide this option to allow resampling to use overviews. This also allows compatibility with versions of RIOS which did not switch it off, before we discovered that it was happening. To allow this, set this parameter to True, otherwise it defaults to False.
We strongly recommend against allowing gdalwarp to use overviews.
- setAlltouched(alltouched, vectorname=None)[source]¶
Set boolean value of alltouched attribute. If alltouched is True, then pixels will count as “inside” a vector polygon if they touch the polygon, rather than only if their centre is inside. If vectornmame given, then set only for that vector.
- setApproxStats(approxStats)[source]¶
Set boolean value of approxStats attribute. This modifies the behaviour of calcStats by forcing it to use the pyramid layers during stats generation (much faster but only provides approximate values, not recommended for thematic rasters)
- setAutoColorTableType(autoColorTableType, imagename=None)[source]¶
If this option is set, then thematic raster outputs will have a color table automatically generated and attached to them. The type is passed to
rios.rat.genColorTable()
to determine what type of automatic color table is generated.The default type will be taken from $RIOS_DFLT_AUTOCOLORTABLETYPE if it is set. If that is not set, then the default is not to automatically attached any color table to thematic output rasters.
In practise, it is probably simpler to explicitly set the color table using the
rios.rat.setColorTable()
function, after creating the file, but this is an alternative.Note that the imagename parameter can be given, in which case the autoColorTableType will only be applied to that raster.
None of this has any impact on athematic outputs.
- setBurnAttribute(burnattribute, vectorname=None)[source]¶
Set the vector attribute name from which to get the burn value for each vector feature. If vectorname is given, set only for that vector input. Default is to use burnvalue instead of burnattribute.
- setBurnValue(burnvalue, vectorname=None)[source]¶
Set the burn value to be used when rasterizing the input vector(s). If vectorname given, set only for that vector. Default is 1.
- setCalcStats(calcStats, imagename=None)[source]¶
Set True to calc stats, False otherwise. If True, then statistics and pyramid layers are calculated (if supported by the driver
- setCreationOptions(creationoptions, imagename=None)[source]¶
Set a list of GDAL creation options (should match with the driver). Each list element is a string of the form “NAME=VALUE”.
Defaults are suitable for the default driver, and need to be changed if that is changed. However, if an appropriate driver-specific default environment variable ($RIOS_DFLT_CREOPT_<driver>) is given, this will be used.
- setFilterSQL(filtersql, vectorname=None)[source]¶
Set an SQL WHERE clause which will be used to filter vector features. If vectorname is given, then set only for that vector
- setFootprintType(footprint)[source]¶
Set type of footprint, one of INTERSECTION, UNION or BOUNDS_FROM_REFERENCE from this module
The footprint type controls the extent of the pixel grid used for calculation within the user function, and of the output files.
Using INTERSECTION will result in the maximum extent which is wholly included in all of the input images. Using UNION results in the minimum extent which wholly includes all of the input images. If BOUNDS_FROM_REFERENCE is used, then the extent will be the same as that of the reference image or pixgrid, regardless of the extents of the various other inputs.
For both UNION and BOUNDS_FROM_REFERENCE, it is possible to have pixels which are within the extent, but outside one or more of the input files. The input data for such pixels are filled with the null value for that file. If no null value is set for that file, then zero is used.
- setJobManagerType(jobMgrType)[source]¶
Set which type of JobManager is to be used for parallel processing. See
rios.parallel.jobmanager
for details. Default is taken from $RIOS_DFLT_JOBMGRTYPE.
- setLayerNames(layerNames, imagename=None)[source]¶
Set list of layernames to be given to the output file(s)
- setNumThreads(numThreads)[source]¶
Set the number of ‘threads’ to be used when processing each block of imagery. Note that these are not threads in the technical sense, but are handled by the JobManager class, and are some form of cooperating parallel processes, depending on the type of job manager sub-class selected. See
rios.parallel.jobmanager
for full details. Note that this is only worth using on very computationally-intensive tasks. Default is 1, i.e. no parallel processing.
- setOmitPyramids(omitPyramids, imagename=None)[source]¶
Set True to omit pyramid layers, False otherwise. If True, then when statistics are being calculated, pyramid layers will be omitted, otherwise they will be created at the same time. Usual default is False.
- setOptionForImagename(option, imagename, value)[source]¶
Set the given option specifically for the given imagename. This method is for internal use only. If you wish to set a particular attribute, use the corresponding ‘set’ method.
- setOutputDriverName(drivername, imagename=None)[source]¶
Set the output driver name to the given GDAL shortname.
Note that the GDAL creation options have defaults suitable only for the default driver, so if one sets the output driver, then the creation options should be reviewed too.
In more recent versions of RIOS, the addition of driver-specific default creation options ($RIOS_DFLT_CREOPT_<driver>) allows for multiple default creation options to be set up.
- setOverlap(overlap)[source]¶
Set the overlap to the given value.
Overlap is a number of pixels, and is somewhat mis-named. It refers to the amount of margin added to each block of input, so that the blocks will overlap, hence the actual amount of overlap is really more like double this value (allowing for odd and even numbers, etc).
The margin can result in pixels which are outside the extent of the given input images. These pixels will be filled with the null value for that input file, or zero if no null value is set on that file.
- setOverviewAggregationType(overviewAggType, imagename=None)[source]¶
Set the type of aggregation used when computing overview images (i.e. pyramid layers). Normally a thematic image should be aggregated using “NEAREST”, while a continuous image should be aggregated using “AVERAGE”. When the setting is given as None, then a default is used. If using an output format which supports LAYER_TYPE, the default is based on this, but if not, it comes from the value of the environment variable $RIOS_DEFAULT_OVERVIEWAGGREGATIONTYPE.
This method should usually be used to set when writing an output to a format which does not support LAYER_TYPE, and which is not appropriate for the setting given by the environment default.
- setOverviewLevels(overviewLevels, imagename=None)[source]¶
Set the overview levels to be used on output images (i.e. pyramid layers). Levels are specified as a list of integer factors, with the same meanings as given to the gdaladdo command.
- setOverviewMinDim(overviewMinDim, imagename=None)[source]¶
Set minimum dimension allowed for output overview. Overview levels (i.e. pyramid layers) will be calculated as per the overviewLevels list of factors, but only until the minimum dimension falls below the value of overviewMinDim
- setReferenceImage(referenceImage)[source]¶
Set the name of the image to use for the reference pixel grid and projection. If neither referenceImage nor referencePixgrid are set, then no resampling will be allowed. Only set one of referenceImage or referencePixgrid.
Note that this is the external filename, not the internal name (which unfortunately is a bit inconsistent with everything else).
- setReferencePixgrid(referencePixgrid)[source]¶
Set the reference pixel grid. If neither referenceImage nor referencePixgrid are set, then no resampling will be allowed. Only set one of referenceImage or referencePixgrid. The referencePixgrid argument is of type
rios.pixelgrid.PixelGridDefn
.
- setResampleMethod(resampleMethod, imagename=None)[source]¶
Set resample method to be used for all resampling. Possible options are those defined by gdalwarp, i.e. ‘near’, ‘bilinear’, ‘cubic’, ‘cubicspline’, ‘lanczos’.
- setStatsIgnore(statsIgnore, imagename=None)[source]¶
Set the global default value to use as the null value when calculating stats. Setting this to None means there will be no null value in the stats calculations.
- setThematic(thematicFlag, imagename=None)[source]¶
Set boolean value of thematic flag (may not be supported by the GDAL driver)
- setVectorDatatype(vectordatatype, vectorname=None)[source]¶
Set numpy datatype to use for rasterized vectors If vectorname given, set only for that vector
- setVectorNull(vectornull, vectorname=None)[source]¶
Set the vector null value. This is used to initialise the rasterised vector, before burning in the burn value. This is of most importance when burning values from a vector attribute column, as this should be a distinct value from any of the values in the column. If this is not so, then polygons can end up blending with the background, resulting in incorrect answers.
- setVectorlayer(vectorlayer, vectorname=None)[source]¶
Set number/name of vector layer, for vector formats which have multiple layers. Not required for plain shapefiles. Can be either a layer number (start at zero) or a layer name. If vectorname given, set only for that vector.
- setWindowSize(windowxsize, windowysize)[source]¶
Sets the X and Y size of the blocks used in one call. Images are processed in blocks (windows) of ‘windowxsize’ columns, and ‘windowysize’ rows.
- class rios.applier.BlockAssociations[source]¶
Generic object to store the image blocks used within the applied function. The attributes are named the same way as in the corresponding FilenameAssociations object, but are blocks of image data, instead of filenames. Where lists of filenames were used, lists of image blocks are used here.
- class rios.applier.FilenameAssociations[source]¶
Class for associating external image filenames with internal names, which are then the same names used inside a function given to the
rios.applier.apply()
function.Each attribute created on this object should be a filename, or a list of filenames. The corresponding attribute names will appear on the ‘inputs’ or ‘outputs’ objects inside the applied function. Each such attribute will be an image data block or a list of image data blocks, accordingly.
- class rios.applier.OtherInputs[source]¶
Generic object to store any extra inputs and outputs used inside the function being applied. This class was originally named for inputs, but in fact works just as well for outputs, too. Any items stored on this will be persistent between iterations of the block loop.
- class rios.applier.RIOSJobInfo(info, inputs, otherargs=None)[source]¶
Class that contains information for parameters to a RIOS function
- rios.applier.apply(userFunction, infiles, outfiles, otherArgs=None, controls=None)[source]¶
Apply the given ‘userFunction’ to the given input and output files.
infiles and outfiles are
rios.applier.FilenameAssociations
objects to define associations between internal variable names and external filenames, for the raster file inputs and outputs.otherArgs is an object of extra arguments to be passed to the userFunction, each with a sensible name on the object. These can be either input or output arguments, entirely at the discretion of userFunction(). otherArgs should be in instance of
rios.applier.OtherInputs
The userFunction has the following call sequence:
userFunction(info, inputs, outputs)
or:
userFunction(info, inputs, outputs, otherArgs)
if otherArgs is not None.
inputs and outputs are objects in which there are named attributes with the same names as those given in the infiles and outfiles objects. In the inputs and outputs objects, available inside userFunction, these attributes contain numpy arrays of data read from/written to the corresponding image file.
If the attributes given in the infiles or outfiles objects are lists of filenames, the the corresponding attributes of the inputs and outputs objects inside the applied function will be lists of image data blocks instead of single blocks.
The numpy arrays are always 3-d arrays, with shape:
(numBands, numRows, numCols)
The datatype of the output image(s) is determined directly from the datatype of the numpy arrays in the outputs object.
The info object contains many useful details about the processing, and will always be passed to the userFunction. It can, of course, be ignored. It is an instance of the
rios.readerinfo.ReaderInfo
class.The controls argument, if given, is an instance of the
rios.applier.ApplierControls
class, which allows control of various aspects of the reading and writing of images. See the class documentation for further details.There is a page dedicated to Applier Examples.
- rios.applier.closeOutputImages(writerdict, outfiles, controls)[source]¶
Called by
rios.applier.apply()
to close all output image files.
- rios.applier.handleInputResampling(infiles, controls, reader)[source]¶
Called by
rios.applier.apply()
to handle automatic resampling of input rasters. Most of the work is done by the read.allowResample() method.
- rios.applier.makeInputImageLayerSelection(imagefiles, controls)[source]¶
Make a dictionary with the same image name keys as imagefiles, but with layerselection lists for each entry, as per the controls object. If only some images have a layerselection set, then the remaining entries are None.
- rios.applier.makeVectorObjects(vectorfiles, controls)[source]¶
Returns a dictionary of
rios.vectorreader.Vector
objects, with the keys being the attribute names used on the vectorfiles object. This is then ready to go into therios.vectorreader.VectorReader
constructor.
- rios.applier.opensAsRaster(filename)[source]¶
Return True if filename opens as a GDAL raster, False otherwise
- rios.applier.opensAsVector(filename)[source]¶
Return True if filename opens as an OGR vector, False otherwise
- rios.applier.separateVectors(infiles)[source]¶
Given a
rios.applier.FilenameAssociations
object, separate out the files which are raster, and the files which are vectors. Returns tworios.applier.FilenameAssociations
objects, carrying the same attribute names, but each has only the raster or the vectors.
- rios.applier.updateProgress(controls, info, lastpercent)[source]¶
Called by
rios.applier.apply()
to update progress
- rios.applier.writeOutputBlocks(writerdict, outfiles, outputBlocks, controls, info)[source]¶
Called by
rios.applier.apply()
, to write the output blocks, after they have been created by the user function. For internal use only.For all names given in outfiles object, look for a data block of the same name in the outputBlocks object. If the given name is a list, then the corresponding name should be a list of blocks.
- rios.applier.BOUNDS_FROM_REFERENCE = 2¶
Use the spatial extent of the reference file
- rios.applier.DEFAULT_MINOVERVIEWDIM = 33¶
Default minimum dimension of highest overview level calculated
- rios.applier.DEFAULT_OVERVIEWAGGREGRATIONTYPE = 'AVERAGE'¶
Default agregation type when using formats without LAYER_TYPE
- rios.applier.DEFAULT_OVERVIEWLEVELS = [4, 8, 16, 32, 64, 128, 256, 512]¶
Default overview levels on output images
- rios.applier.INTERSECTION = 0¶
Use the spatial intersection of inputs
- rios.applier.NO_VRT_FOR_RESAMPLING = False¶
Whether to use VRTs for resampling inputs. Set by RIOS_NO_VRT_FOR_RESAMPLING environment variable. Default is True.
- rios.applier.UNION = 1¶
Use the spatial union of inputs