Cris’ Image Analysis Blog

Robust estimation of shift, scale and rotation

2026-02-10T13:27:00-07:00

A frequent problem in image processing is aligning two or more images together. Maybe you have a series of images that partially overlap and need to stitch them into a single, larger image. Maybe you have a video sequence and need to estimate and remove the movement. Maybe you have images of the same subject but in different modalities that you need to combine. In all these cases, the transformation you need to apply to one image to match it to the other is only translation (or shift), rotation, and maybe scaling. (You might also need to apply a correction for lense distortion, but that is outside the scope of this post.)

The most common methods for finding the alignment are all based on markers: First you find a set of distinct points (or markers) in the two images (corners, dots, etc.), then you find the correspondences (figure out which marker in image A corresponds to each marker in image B), then you compute the transformation from those correspondences. And this often works well. The hardest part here is finding the correspondences. When fitting the transformation, some number of mismatches can be accounted for, but most matches need to be correct.

Here I wanted to explain a simpler method to align two images, that is not as widely taught in image processing courses as it should be. It is an efficient, accurate and robust method that directly compares the full images, rather than individual markers. It is based on the cross-correlation, which can be used to find the shift between two images, and the Fourier-Mellin transform, which disregards translation, and converts scaling and rotation to shifts. This method is due to Reddy and Chatterji (1996).

Let’s explore the method step by step, and explain concepts as we go along.

The Fourier Transform

The first step in the method is to compute the magnitude of the Fourier Transform (FT). We’re actually computing the Discrete Fourier Transform (DFT), which is distinct but strongly related to the Fourier Transform, which is defined in the continuous domain. The DFT is computed efficiently using the Fast Fourier Transform algorithm (FFT). But we’ll refer to it simply as the FT.

I’ll assume we all know what the FT is, but I’ll summarize really quickly here: The FT is a reversible transformation of the image that is useful to study linear systems. The FT maps a complex-valued function to another complex-valued function. Complex values can be interpreted as having a real and an imaginary component, or, equivalently, a magnitude and a phase.

A shift in the image is encoded solely in the phase of the FT (images on the right are the magnitude of the FT of the images on their left):

So, taking the magnitude of the FT, we’ve removed the shift from the problem. But what happened to the scaling and the rotation? A scaling of the image results in an inverse scaling of the FT:

And a rotation of the image results in an equal rotation of the FT:

However, because the input image was real-valued, the FT is symmetric about the origin, and therefore we conflated the rotation with φ and the rotation with φ+π. We are no longer able to distinguish a rotation of 180°.

After this first transformation, we have simplified the problem of estimating shift, rotation and scaling to a problem of estimating only rotation and scaling. Next, we apply a log-polar mapping. This is a mapping where the image is deformed in a particular way, resulting in the x-axis of the output being the logarithm of the distance to the center of the input image, and the y-axis being the angle around this center:

A rotation around the center of the image corresponds to a vertical shift of the log-polar mapped image. And a scaling will correspond to a horizontal shift of the log-polar mapped image. We thus have converted the rotation and scaling to a simple translation. This is the Mellin transform part of the Fourier-Mellin transform.

Cross-Correlation

Finding a shift between two images is fairly straight-forward using the cross-correlation. In short, we multiply the two images together and compute the sum of the result, yielding a single value. We do this for every possible shift of one image w.r.t. the other. This can be computed efficiently using the FFT. The shift for which the cross-correlation function is largest is the shift that makes the two images most similar. By fitting a parabola to the 3×3 neighborhood of the pixel with the largest value, we can determine its location with sub-pixel precision.

There are different ways to normalize the cross-correlation to increase the sharpness of the peak, and thus improve the localization accuracy. To learn more about these normalization methods, see the DIPlib documentation for dip::CrossCorrelationFT()

Once we found the shift in the log-polar mapped FT image, we can calculate the rotation and scaling for the one image to match the other. Except that we still need to disambiguate the 180° rotation!

What about the shift?

So now we can transform one of the two images to match the other with only a shift and a possible 180° rotation left. We can compute this partially-aligned image, then use the cross-correlation again to find this missing shift. We can apply this cross-correlation twice: once with the transformed image, and once with the transformed image rotated by 180°. One of these two will yield a higher cross-correlation peak, because it will match the other image better.

Example

An example will make all of the above more clear. I’ll be using DIPlib in Python.

Let’s say we have these two input images, img_a and img_b:

import diplib as dip
import numpy as np

img_a = dip.ImageRead("image A")
img_b = dip.ImageRead("image B")

img_b (right) is a crop of some original image. img_a (left) is a different crop obtained after rotating the original image clockwise by π/8 radian and scaling it by a factor of 0.7.

For simplicity, we’ll consider only the green channel, which has good contrast across the image:

gray_a = img_a(1)
gray_b = img_b(1)

We can apply the method to the color images, but we won’t gain anything. After finding the transformation needed to align the gray-scale images, we can apply that transformation to the color image.

We’ll start by computing the FTs of the two images, after applying a windowing function. We discard the phase, and apply a log mapping to improve local contrast:

ft_a = dip.FourierTransform(dip.ApplyWindow(gray_a, "GaussianTukey", 10))
ft_b = dip.FourierTransform(dip.ApplyWindow(gray_b, "GaussianTukey", 10))
lm_a = dip.Ln(dip.Abs(ft_a))
lm_b = dip.Ln(dip.Abs(ft_b))

The windowing function is important because without it, the FT will be corrupted by the edge effects. Image edges usually show up as a sharp horizontal and vertical line through the origin. These two lines will cause a peak in the cross-correlation at the origin, which will be identical for the two images, giving us a chance to erroneously find a shift of 0.

Note also that dip.FourierTransform() computes the full DFT with the origin in the middle of the output image. When using a different FFT implementation, you’ll need to apply a function such as np.fft.fftshift() to get the origin to be in the middle of the image.

Next, we apply the log-polar transformation to the two images:

lpt_a = dip.LogPolarTransform2D(lm_a, "linear")
lpt_b = dip.LogPolarTransform2D(lm_b, "linear")

The cross-correlation between these two images gives a sharp peak, whose location gives the shift:

find_shift_method = "NCC"  # or "PC" or "CC", depending on which normalization you want to use
shift = dip.FindShift(lpt_a, lpt_b, find_shift_method)
print(f"LPT shift = {shift}")

We get a shift of -32.67 along the x-axis and 224.05 along the y-axis. From that we can calculate the rotation and scaling, simply by inverting the computations applied in the log-polar transform:

maxr = np.min(gray_a.Sizes()) // 2
size = lpt_a.Size(0)
scale = maxr ** (shift[0] / (size - 1))
theta = -shift[1] * 2 * dip.pi / size  # 180 degree ambiguity!
print(f"scale = {scale}, theta = {theta}")

We get a scale of 0.7015 and a rotation of -2.749 radian. We expected a scale of 0.7000, and a rotation of 0.3927 (π/8) radian. -2.749 is π/8 - π, so we are off by exactly 180°. Remember that with the FT, we cannot distinguish between these two rotations. We happen to have found the wrong rotation, but fret not, we’ll fix this soon.

We can apply the scale and rotation to the image, which will leave us with a shifted (and rotated by 180°) version of the final result:

matrix = [scale * np.cos(theta), -scale * np.sin(theta), scale * np.sin(theta), scale * np.cos(theta), 0, 0]
gray_b_transformed = dip.AffineTransform(gray_b, matrix, "linear")

Using this transformed image, we can figure out what shift to apply using cross-correlation again:

cc_method = {
    "CC": "don't normalize",
    "NCC": "normalize",
    "PC": "phase",
}[find_shift_method]
cross = dip.CrossCorrelationFT(gray_a, gray_b_transformed, normalize=cc_method)
loc_1 = dip.SubpixelLocation(cross, dip.MaximumPixel(cross))
print(f"loc_1: coordinates = {loc_1.coordinates}, value = {loc_1.value}")

But because we have two possible rotations, we should compute the cross-correlation with gray_b_transformed and also with that image rotated by 180°. The height of the peak will tell us which of the two rotations is the correct one (a larger cross-correlation result means the images are more similar).

gray_b_transformed.Rotation90(2)
cross = dip.CrossCorrelationFT(gray_a, gray_b_transformed, normalize=cc_method)
loc_2 = dip.SubpixelLocation(cross, dip.MaximumPixel(cross))
print(f"loc_2: coordinates = {loc_2.coordinates}, value = {loc_2.value}")

loc_1 has a value of 0.0261, and loc_2 of 0.2920. It is clear that the rotation used for loc_2 is the better one. The peak for loc_2 is at (197.78, 344.04). The shift we’re looking for is the distance from the origin (the middle of the image) to this peak:

if loc_1.value >= loc_2.value:
    # We use theta as the rotation
    matrix[4] = loc_1.coordinates[0] - cross.Size(0) // 2
    matrix[5] = loc_1.coordinates[1] - cross.Size(1) // 2
else:
    # We use theta + 180 as the rotation
    matrix[4] = loc_2.coordinates[0] - cross.Size(0) // 2
    matrix[5] = loc_2.coordinates[1] - cross.Size(1) // 2
    matrix[0] = -matrix[0]
    matrix[1] = -matrix[1]
    matrix[2] = -matrix[2]
    matrix[3] = -matrix[3]

Using this transformation matrix, we can finally compute the transformed version of img_b that matches img_a:

final = dip.AffineTransform(img_b, matrix, "linear")

This whole process is implemented in DIPlib as dip::FourierMellinMatch2D().

You can find the complete code I used to generate all the images in this post right here.

DIPlib 3.5.2 released

2024-12-27T00:00:00-07:00

Today we released DIPlib version 3.5.2. This release has quite a lot of changes, see the change log. In my last blog post I discussed the graph cut algorithm. This is the largest addition (effort wise) in this release, together with the changes to the dip::Graph class and the new (but similar) dip::DirectedGraph class.

The other major change is in documentation, both for DIPimage (the MATLAB toolbox) and for PyDIP (the Python bindings).

In DIPimage, all the references to DIPlib functions are now links to the on-line documentation:

>> help gaussf
 gaussf   Gaussian filter

  SYNOPSIS:
   image_out = gaussf(image_in,sigma,method,boundary_condition,truncation)

  PARAMETERS:
   sigma: Gaussian parameter for each dimension
   method: Method used to compute the Gaussian. One of:
     - 'fir':    Finite Impulse Resonse filter (convolution with a kernel).
     - 'iir':    Infinte Impulse Response filter (recursive filtering).
     - 'ft':     Convolution via a multiplication in the Fourier Domain.
     - 'best':   Chooses the best option above for your kernel.
     - 'kernel': The convolution kernel is returned, rather than the result
                 of the convolution.
   boundary_condition: Defines how the boundary of the image is handled.
                       See HELP BOUNDARY_CONDITION
   truncation: Determines the size of the Gaussian filters.

  DEFAULTS:
   sigma = 1
   metod = 'best'
   bounary_condition = 'mirror'
   truncation = 3

  NOTES:
   See DERIVATIVE for an explanation of the option 'best'.

   If SIGMA==0 for a particular dimension, that dimension is not processed.

  SEE ALSO:
   derivative

  DIPlib:
   This function calls the DIPlib function dip::Derivative (which calls dip::GaussFIR,
   dip::GaussIIR and dip::GaussFT) and dip::CreateGauss.

The text “derivative” is automatically turned into a link by MATLAB, as it’s the name of a function. The last section previoulsy just named “dip::Derivative”, now it’s a link that the user can click on to go to the documentation. (A shout-out to Andre Zeug for this idea.)

In PyDIP, we previously only had the help text automatically generated by Pybind11, the C++ library that we use to create the Python bindings. We have now added the first paragraph from the on-line help to this:

>>> help(dip.Gauss)
Help on built-in function Gauss in module diplib.PyDIP_bin:

Gauss(...) method of builtins.PyCapsule instance
Gauss(*args, **kwargs)
Overloaded function.

    1. Gauss(in: diplib.PyDIP_bin.Image, sigmas: list[float] = [1.0], derivativeOrder: list[int] = [0], method: str = 'best', boundaryCondition: list[str] = [], truncation: float = 3.0) -> diplib.PyDIP_bin.Image

    Convolution with a Gaussian kernel and its derivatives

    2. Gauss(in: diplib.PyDIP_bin.Image, *, out: diplib.PyDIP_bin.Image, sigmas: list[float] = [1.0], derivativeOrder: list[int] = [0], method: str = 'best', boundaryCondition: list[str] = [], truncation: float = 3.0) -> None

    Convolution with a Gaussian kernel and its derivatives

This help shows the input and output arguments to the function, and if there is more than one legal signature (as is the case for most functions in DIPlib) shows each of them. The text “Convolution with a Gaussian kernel and its derivatives” above is new, the rest is the same as it was before.

This automatically generated help text has some issues, it references the module diplib.PyDIP_bin, which is an internal name, rather than diplib, which is the external name of the module, or dip, which is the name under which the user imported it. It also references the “builtins.PyCapsule instance”, which is an internal Pybind11 construct.

This first paragraph of the help (the short description) is sometimes enough to know what a function does, but if you want to know more about the meaning of the input arguments, how to use the function, or exactly what algorithm(s) it implements, then you need to see the full documentation. Putting the full documentation into the Python help string is not doable. The source for the documentation is Markdown, we could parse that and render it as pure text into the Python help string, but for some functions we make heavy use of equations, which we cannot render in pure text.

Instead, we can help the user find the on-line documentation, like we did in DIPimage. We created a new function, dip.Doc, which will open your default web browser on the documentation for any specific function. For example,

dip.Doc(dip.Gauss)

will open this link. If a function has multiple overloads with separate documentation, then it opens the first one it finds, and presents the list of all the matches:

>>> dip.Doc(dip.OtsuThreshold)
Found multiple matches for OtsuThreshold (opening the first one):
  - https://diplib.org/diplib-docs/histograms.html#dip-OtsuThreshold-Histogram-CL
  - https://diplib.org/diplib-docs/segmentation.html#dip-OtsuThreshold-Image-CL-Image-CL-Image-L

dip.Doc("watershed") will open the documentation for dip::Watershed, because the string exactly matches a function name (without regard to case). If the string doesn’t match a function exactly, it will a Google search in the DIPlib documentation. For example, dip.Doc("diffusion") opens this Google search.

I’d love to hear your opinion about these solutions to the lack of documentation in Python, and any ideas you might have to improve it further!

Graph cut segmentation

2024-12-01T00:00:00-07:00

For a while I’ve been interested in adding the popular graph cut segmentation algorithm to DIPlib. But I was not able to find an implementation I could adopt (the most commonly used implementation has an incompatible open-source license). So I finally sat down and re-implemented the algorithm myself from scratch. I learned a lot about the algorithm, hopefully with this blog post you’ll learn something too.

The graph cut segmentation algorithm sees the image as a weighted graph, where each pixel is a node (or vertex), and each pixel is connected to its neighbors with an edge. The weight of the edge is given by the difference in intensity of the two pixels it connects. If we now find a way to split the graph into two, by cutting edges with a high weight (i.e. connecting pixels with a large difference in intensity), then we will have identified a region in the image with a large contrast to its surrounding.

This algorithm requires two markers: one for the background and one for the foreground. The algorithm will find a cut of the graph that separates these two markers: each marker will be in a different region. These markers therefore direct the segmentation. Typically, this is used in a situation where the user can draw a set of markers on the image to give a rough indication of what the object is and what the background is.

How is this optimal cut found? And what really is optimized? First we need to learn about the max-flow problem, the min-cut problem, and their relation.

The min-cut problem

Let’s forget about images for a moment, and think of a network of tubes. Each tube has a specific cross-section, which will determine how much water can flow through it (its capacity). The network has a source, where water flows into the network, and a sink, where water flows out. We can represent this network as a graph: each tube is an edge in the graph, its weight is the tube cross-section. Where the tubes are joined together we have a node in the graph. The source and sink are two nodes like any other. One thing to note: the flow of water going into a node must be equal to the flow going out, except for the source and sink nodes. The flow originating in the source must be equal to the flow exiting at the sink.

There is a maximum amount of water that can flow through this network from source to sink. Finding out how much this is, is the max-flow problem.

When this amount of water is flowing, there must be a set of tubes flowing at maximum capacity (they’re saturated). Any path that you can find from source to sink must go through one of these tubes flowing at maximum capacity, otherwise we’d be able to increase the amount of water flowing (it can flow along the non-saturated path we just found). If we were to cut the set of saturated tubes, we would separate the source from the sink: the graph would be cut into two parts, one containing the source and one containing the sink.

The set of edges (tubes) we cut constitute the minimal cut. That is, the sum of their weights (or the cross-sections of the tubes) is smaller than for any other cut. Any other cut will necessarily cut an edge with available capacity, and therefore yield a larger sum. Finding this cut is the min-cut problem. It is clear, I think, from the explanation above that the min-cut problem is equivalent to the max-flow problem: the maximum amount of water that can flow from source to sink is equal to the minimal possible sum of capacities of tubes we need to cut to separate the source from the sink.

Finding a solution to the min-cut problem

To find the maximum amount of water that can flow through our network of tubes, and which tubes will be saturated, we need to find all possible paths through the network. For each path in turn, we find the minimum capacity, and add that amount of flow to each tube along the path. That is, we find a path and have as much water as possible flow along that path. Do that for all possible paths are we’re done.

Here’s a simple example graph:

One path from source (S) to sink (T) is S–A–B–T. The minimal capacity along this path is 0.1, so we subtract that amount from each of the edge weights along that path:

The numbers attached to each edge now is not the total capacity of the edge, but the residual capacity, how much more water can flow through it.

Another path is S–C–B–T. The residual along this path is 0.6, so we again subtract that amount from each of the edge weights along that path:

Another path is S–C–D–T. Residual to subtract is 0.5:

Now we have a set of edges that are saturated (their residual is 0): A–B, C–B, C–D. This is our minimal cut (1.2 was our total flow). By removing these edges, nodes A and C belong to S, and B and D to T.

But there is another path we haven’t yet explored: S–A–B–C–D–T. We didn’t need to consider this path because at least one of its edges is already saturated. But I needed to point this out because:

First, the B–C edge is being used in reverse in this path, compared to the earlier S–C–B–T path. If we were to add water flow along this path, the flow from C to B would be reduced, or even reversed. Thus, when computing flow, we must also consider direction of flow. We either need to remember in which direction the water flows, or (as is commonly done) we must consider a separate residual in each direction. When one residual is increased, the other is decreased.

Second, the number of paths through a graph is deceptively large. For any practical problem, the number is so large that the trivial algorithm described above is impossible to compute. We need to use a more efficient algorithm. Many such algorithms have been described in the literature. The algorithm commonly used in image processing is due to Boykov and Kolmogorov [1].

An efficient algorithm

Boykov and Kolmogorov [1] described an algorithm with a higher time complexity than that of earlier algorithms. But they show that, in practice, it is faster for the case of graphs derived from images (where each node has few edges).

The algorithm is based on building two trees, one rooted in the source and one in the sink. Each node in the tree points to its parent (so that one can navigate from any node in the tree to the root, but not the other way around). As these trees grow, we’ll eventually find one edge that joins the two trees. From this edge, we can travel backwards to the source and forward to the sink, identifying a complete path from source to sink. Crucially, the trees can later continue growing to identify all the other paths.

When a path is identified, we update residuals along it. One edge will become saturated, and this is not necessarily the edge that joins the two trees. In fact, it is possible for multiple edges to become saturated. The branch that is rooted in this saturated edge is cut off from its tree (we’ll mark its root node as an orphan). Next, we attempt to re-attach the orphan nodes by looking at their neighbors. Any non-saturated edge that connects an orphan node back to the same tree it originally belonged to can be used. If no such edge exists, the node becomes a free node (one that the trees can eventually grow into), and its children are marked orphans. When there are no more orphans, the tree growing phase continues.

The algorithm finishes when all the free nodes have become part of one of the two trees, and no more paths can be found. Note that it is possible for there to be more saturated edges than strictly required to cut the graph in two. And the set of saturated nodes can even yield different graph cuts! We cannot simply look at all the saturated nodes to find our solution. We need to traverse the graph from the source (or from the sink) and find all nodes that we can reach without using a saturated edge. This set of nodes will be one of the graph components, all the other nodes will form the other graph component. The graph cut is the set of edges that join a node in each component.

Application to image processing

The above can be directly applied to an image, from which we construct a graph by turning each pixel into a node, and add edges between each pair of neighboring pixels. In a 2D image, using a 4-connected neighborhood, we’d have 4 edges connecting each node. Edge weights have to be computed in some way from the difference in pixel intensity such that for a large intensity difference we obtain a small weight, and for a small intensity difference we obtain a large weight. The cut will then go between pixels with a larger difference in intensity. We’d mark one pixel as the source and one as the sink, or, in the case of many marker pixels, we’d add a new source node connected to all the source marker pixels with a very high weight, and similarly for the sink.

But Boykov and Jolly [1] suggested a more elaborate scheme, where the source and sink nodes connect to all pixels in the image. The weight from source and sink nodes to each pixel are given by the difference between the pixel’s intensity and the expected foreground or background intensities. They actually use the histogram of the pixel values for the foreground and background markers as the expected intensities. What this accomplishes is striking a balance between just propagating a marker to the nearest edges and just selecting pixels based on colors (as you’d do with a threshold).

In any case, this segmentation approach provides a global optimum (as opposed to a local one in algorithms such as level sets) given the constraints (markers) and the cost function (edge weights). This allows for more easyly tuning markers and edge weights to obtain the desired result, without being dependent on algorithm initialization.

The next release of DIPlib will have a graph cut function.

Literature

Y. Boykov and V. Kolmogorov, “An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision”, IEEE Transactions on Pattern Analysis and Machine Intelligence 26(9):1124–1137, 2004.
Y. Boykov M.-P. Jolly, “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images”, Proceedings Eighth IEEE International Conference on Computer Vision (ICCV 2001) 1:105–112, 2001.

ITK's architecture

2024-07-04T00:00:00-06:00

This morning I was reading a chapter about the architecture of ITK in a book called The Architecture of Open Source Applications (Volume 2). ITK is the Insight Toolkit, a large library for image analysis in C++, specifically aimed at medical applications. Most of its functions work on images of any dimensionality, just like in DIPlib. But just about every other choice made in this library is the polar opposite of the choices made in DIPlib. And this is why I was curious to learn about why those choices were made.

The book chapter was written by Luis Ibáñez and Brad King. Luis is one of the main architects of ITK, and Brad is one of the original developers. ITK’s development started in 1999 with lots of funding from the US government. This chapter was written maybe 10 or 15 years ago, ITK was already well established.

Reading this chapter, I saw some statements that I disagreed with, I guess that is the main reason I’m writing this blog post now. I am going to address these statements, and their justification for some of the design choices, out of order from how they appear in the chapter, so that my comments make more sense. Obviously, my comments will make lots of comparisons to DIPlib.

I once tried using ITK. I spent a week writing a simple function to align two images, then gave up and implemented a rigid alignment from scratch in a day. That is to say, ITK has a very steep learning curve. Keep that in mind when you read my comments below.

Object-oriented programming

Now, some of my criticism of ITK comes from my dislike of the modern interpretation of object-oriented programming (OOP): “everything is a class”. The Java programming language for example is designed around this mentality. In Java you cannot even write a function outside a class. ITK’s architecture revolves around the “everything is a class” mentality. This makes my hair stand on edge when I see ITK code. But I will try to separate my feeling for this programming paradigm from all the other comments I make in this blog post.

OOP

OOP is supposed to simplify the construction of large, complex programs by dividing it up into smaller, minimally-dependent portions. Each portion is an object, that communicates with other objects through their shared API. This shared API needs to be minimal, since it is the only thing that creates dependencies between the components. This division reduces the complexity of the application: pieces of code can be read and modified independently of other pieces of code. Nowhere in this definition does it say or imply that the individual components must be made using classes, that classes must inherit behavior or interfaces from other classes, or that the OOP paradigm is even suitable for writing a library.

Functionality

The article starts by explaining things about the field of image analysis, and what an image analysis library needs to deal with.

In this context, a certain level of “redundancy”—for example, offering three different implementations of the Gaussian filter—is not seen as a problem but as a valuable feature, because different implementations can be used interchangeably to satisfy constraints and exploit efficiencies with respect to image size, number of processors, and Gaussian kernel size that might be specific to a given imaging application.

This is absolutely right. Different algorithms that compute the same thing in different ways are not necessarily redundant. The example of the Gaussian filter (DIPlib also has 3 different algorithms) is the most obvious one: The IIR algorithm is most efficient for a very large sigma, but is an approximation. The FIR algorithm is most efficient for the more common sigma values. The FT algorithm is the only way you can compute the filter for a sub-pixel sigma with any semblance of precision.

[…] ITK has a collection of about 700 filters. Given that ITK is implemented in C++, this is a natural level at which every one of those filters is implemented by a C++ Class following object-oriented design patterns.

I’ve already said that making each filter a class is not real OOP, but OK, let’s set that aside. I don’t think that making each filter a class is a good idea because it makes the user’s code so much more complex. Here’s an example program from the ITK documentation:

const auto input = itk::ReadImage<ImageType>(inputFileName);

using FilterType = itk::SmoothingRecursiveGaussianImageFilter<ImageType, ImageType>;
auto smoothFilter = FilterType::New();
smoothFilter->SetSigma(sigmaValue);
smoothFilter->SetInput(input);

itk::WriteImage(smoothFilter->GetOutput(), outputFileName);

I’ve left out all the include statements, input parsing, exception handling, and the definition of ImageType (which we’ll talk about later). Also, I’m honestly surprised that itk::ReadImage and itk::WriteImage are not classes. Apparently they are convenience functions introduced in release 5.2.0 (May 2021). They just encapsulate the creation and use of the image reading and writing objects.

Anyway, the above is a very simple program, it reads an image, applies a Gaussian filter, and writes the result to file. The same program in DIPlib:

auto input = dip::ImageRead(inputFileName);
auto output = dip::Gauss(input, {sigmaValue});
dip::ImageWrite(output, outputFileName);

As a library user, which would you prefer to write?

Sure, it is more explicit to do smoothFilter->SetSigma(sigmaValue) than having sigmaValue be the second input argument to a function. But when being explicit causes you to have to write four lines of code instead of one… sigh. For functions with lots of parameters, the C++ call syntax can be quite obtuse. We’re reliant on our IDE to show the name of the parameters for such function calls. I really like the Python syntax there, where you can explicitly name the parameters as you fill in their value in the function call. Will C++ ever adopt that syntax?

Pipeline

ITK is designed around the so-called “Data Pipeline architecture”:

The staged nature of most image analysis tasks led naturally to the selection of a Data Pipeline architecture as the backbone infrastructure for data processing. The Data Pipeline enables:

Filter Concatenation: A set of image filters can be concatenated one after another, composing a processing chain in which a sequence of operations are applied to the input images.

Parameter Exploration: Once a processing chain is put together, it is easy to change the parameters of any filter in the chain, and to explore the effects that such change will have on the final output image.

Memory Streaming: Large images can be managed by processing only sub-blocks of the image at a time. In this way, it becomes possible to process large images that otherwise would not have fit into main memory.

This core principle informed every other decision in the library. Regarding the second point, they say:

Image filters typically have numeric parameters that are used to regulate the behavior of the filter. Every time one of the numeric parameters is modified, the data pipeline marks its output as “dirty” and knows that this particular filter, and all the downstream ones that use its output, should be executed again. This feature of the pipeline facilitates the exploration of parameter space while using a minimum amount of processing power for each instance of an experiment.

This is a neat idea, but to change the parameters, I’d have to recompile my program (meaning I don’t get to reuse any of its working data), unless I write code to let the user modify parameters. If I write code to let the user modify parameters, I can also write code to re-process the relevant steps of the pipeline. I don’t understand why this use case needs to be embedded in the library. But it’s cool.

The main problem with this pipeline idea then is that all intermediate images are stored in memory. If I have a pipeline with 100 individual processing steps (this is not out of the ordinary at all!), then I’ll have 100 intermediate results being stored. I cannot instruct a step to overwrite its input to save memory, and I cannot discard intermediate images when I don’t need them anymore.

So this one use case, of allowing a program to do some interactive parameter exploration, makes the (IMO) more common use case, of a program with fixed parameters needing to be as fast and memory-efficient as possible, harder.

The third point, memory streaming, is also very interesting. Indeed, sometimes images are so large they don’t fit in memory. Back in 1999 when ITK was designed, this might have been more common than it is today, but it certainly still happens, for example in digital pathology.

But this point is contradicted by the design decision of keeping all intermediate images in memory. Or maybe that choice makes it more important to be able to process the image in smaller tiles, since it wastes so much memory…

But it is true, being able to write a program as if you’re processing the image as a whole, but have it automatically split the image up into tiles for you and process those tiles independently (and possibly in parallel) is great. Each of the steps in the pipeline know how much margin they need (each filter will read data from outside its output window, so its input needs to be a bit larger), the margin gets computed automatically. This is one of the more difficult things to get right when doing tile-wise processing.

This margin, also called overlap, is processed multiple times, and is necessary for the result on two neighboring tiles to match up at the edge between them. The smaller the margin, the more efficient the algorithm is, but if the margin is too small, then the two tiles don’t agree at the edge that joins them, and the final result will show seams. Hence, finding the minimal and sufficient overlap is important.

However,

Streaming, unfortunately, can not be applied to all types of algorithms. Specific cases that are not suitable for streaming are: Iterative algorithms. […] Algorithms that require the full set of input pixel values […]. Region propagation or front propagation algorithms […]. Image registration algorithms […].

This is a large fraction of the image analysis toolbox. Most algorithms are not trivial to apply in tiles. For example, you need to understand the content of the image to decide how large the margin must be for the Watershed algorithm to produce correct results when applied tile-wise. How far information gets propagated across the image by such an algorithm depends on the image content. It is only simple neighborhood algorithms where you can determine the margin needed without understanding the application.

And then I ask myself: is it really worth it, creating this complex “Data Pipeline architecture” so that some programs can automatically be applied tile-wise to large images?

Obviously in DIPlib we took the approach that an algorithm works on a complete image in memory. The programmer can use these functions to process image tiles, by manually determining how large these margins must be, writing the code to read the input image tile-wise, and writing code to merge the results at the end. ITK writes the result tile-wise to file, but most image analysis programs do not produce an output image, they produce measurement results. So combining the results from file-wise processing is application-dependent. Maybe you need to decide which of the cells that intersect the tile “belong” to the tile, maybe you need to merge region outlines produced in tiles, maybe you need to sum the individual tile measurements, …

I have often thought about adding a tile-wise processing function to DIPlib. This function would take as input a user-written function that processes a single tile. But it is the final merging step that I can’t seem to define generically enough.

And there’s another issue with the pipeline approach.

In a typical image analysis problem, a researcher or an engineer will take an input image, improve some characteristics of the image by, let’s say, reducing noise or increasing contrast, and then proceed to identify some features in the image, such as corners and strong edges This type of processing is naturally well-suited for a data pipeline architecture, as shown in Figure 9.1.

So, I think this is rather naive. A typical problem has many more steps, and they’re not in a simple linear fashion like this. This arbitrary graph is a more realistic pipeline:

In this pipeline, the blocks marked B, C and D read the output of block A, each likely with a different margin in the tile-wise processing. Does block A process the same tile multiple times? How complex does the internal logic have to be for A to see that these different output requests are very similar, and have it process the field of view only once, with the largest of the requested margins? Does ITK implement such logic?

Types

Despite the assistance that the file reader and writer facades provide, it is still up to the application developer to be aware of the pixel type that the application needs to process. In the context of medical imaging, it is reasonable to expect that the application developer will know whether the input image will contain a MRI, a mammogram or a CT scan, and therefore be mindful of selecting the appropriate pixel type and image dimensionality for each one of these different image modalities. This specificity of image type might not be convenient for application settings where users wants to read any image type, which are most commonly found in the scenarios of rapid prototyping and teaching. In the context of deploying a medical image application for production in a clinical setting, however, it is expected that the pixel type and dimension of the images will be clearly defined and specified based on the image modality to be processed.

Sure, this is true. When developing a program for a specific application, you know what file type you’ll be loading, what the pixel type is, and how many dimensions the image has. But when you develop a more generic program, then allowing for run-time type and dimensionality determination is very useful. SimpleITK was developed specifically for this purpose (I actually see it as an admission that ITK is unusable!). DIPlib and OpenCV both allow for run-time type determination. Why? Because it makes writing code easier. And because it makes creating bindings for other languages easier. And because everything can be pre-compiled instead of being offered as templated code in header files that must be compiled over and over again every time you build your application. And because it makes generic programs much, much simpler.

Vigra is another C++ library that has the pixel type and image dimensionality as template parameters. Vigra attempts to create something more akin to the Standard Template Library for image processing. The idea is that algorithms are generic, and the compiler builds an efficient implementation of the algorithm for your particular use case. Maybe you can apply one of the Vigra image processing algorithms on a graph data structure. This is a neat idea in principle, but you end up with the same issues I criticized above.

IO Factories

The Factory pattern in ITK uses class names as keys to a registry of class constructors. The registration of factories happens at run time, and can be done by simply placing dynamic libraries in specific directories that ITK applications search at start-up time. This last feature provides a natural mechanism for implementing a plugin architecture in a clean and transparent way. The outcome is to facilitate the development of extensible image analysis applications, satisfying the need to provide an ever-growing set of image analysis capabilities.

So, instead of allowing the user of the library to write a new image reading function (or class I guess) for a new file format, and call that function directly (or register it with the library before using the library’s image reading functionality), they want this new functionality to be compiled into a separate dynamic library, and installed in a specific directory on the system where the application will run. The itk::ImageFileReader class will then find it and be able to read the new file type.

This seems to be designed specifically so that applications written with ITK can be extended without recompiling those applications. I think this adds a lot of complexity to the library, and to its use. Still, plugins are quite neat.

The loadable IO factories has been one of the most successful features in the architectural design of ITK. It has made it possible to easily manage a challenging situation without placing a burden on the code or obscuring its implementation.

This is surprising to me. Plugins are neat, yes, but are they that important? How many people have built such a plugin? Here is an ITK ImageIO plugin for Bio-Formats. I haven’t found any other plugins so far. Do you know of any others?

Some comments not related to ITK’s design

There were a couple of other comments in this chapter that I’d like to highlight and comment on.

maintainers […] account for more than 75% of the cost of software development over the lifetime of a project.

(Later they have a more elaborate description of this finding, with citations.) I did not know this. An interesting point to keep in mind. For every hour we spend writing an algorithm, we need to spend 3 hours maintaining that code!

In a last section called “Reproducibility”, the authors say:

One of the early lessons learned in ITK was that the many papers published in the field were not as easy to implement as we were led to believe. The computational field tends to over-celebrate algorithms and to dismiss the practical work of writing software as “just an implementation detail”.

Well, the interesting things to discuss about an algorithm are its properties, not its implementation. Then it says:

The outcome is that most published papers are simply not reproducible, and when researchers and students attempt to use such techniques they end up spending a lot of time in the process and deliver variations of the original work. It is actually quite difficult, in practice, to verify if an implementation matches what was described in a paper.

That is completely true. Unless the authors publish their code, reproducing their work is nearly impossible. Often paper authors are ashamed about the quality of their code, and do not want to share it without a complete rewrite. But of course there never is time for that. And so the code will, unfortunately, remain unpublished.

Next, they claim:

ITK disrupted, for the good, that environment and restored a culture of DIY to a field that had grown accustomed to theoretical reasoning, and that had learned to dismiss experimental work. The new culture brought by ITK is a practical and pragmatic one in which the virtues of the software are judged by its practical results and not by the appearance of complexity that is celebrated in some scientific publications.

I have not seen any evidence of this. I imagine they would have liked this outcome. I certainly would have welcomed it. But, 25 years on, papers are, more often than not, still not reproducible. We still don’t get to see the code used to make them.

It turns out that in practice the most effective processing methods are those that would appear to be too simple to be accepted for a scientific paper.

This is also true. I have seen it hundreds of times. It is unfortunate that most journals will only accept a paper if the method described beats the “state of the art”. I have written about this before. This leads to awkward experimental sections with bad comparisons, in an attempt to appear to beat the state of the art. But journals should be publishing papers about new ways of thinking, new approaches, not tiny incremental, arbitrary changes that improve results in one particular test from 91.4% to 91.6% (and probably make results worse in other tests that didn’t get included in the paper).

All the quotes in this blog post are from the chapter ITK, by Luis Ibáñez and Brad King, 9th chapter in the book The Architecture of Open Source Applications (Volume 2), edited by Amy Brown and Greg Wilson, and published under the Creative Commons Attribution 3.0 Unported license.

In closing

And now I’d like to hear from you: Have you used ITK? How usable do you find it? Are any of the design decisions discussed here better than in DIPlib?

DIPlib 3.5.0 released

2024-06-19T00:00:00-06:00

This week we released DIPlib version 3.5.0. In my previous blog post I talked about new syntax for selecting labeled objects. There are quite a few more quality-of-life improvements like that in this release; changes that, rather than adding algorithms, make existing ones easier to use:

dip::Measurement::IteratorFeature::AsImage() converts a measurement column into an image, making it easy to apply a lot of statistics functions to measurement features.
There’s a new overload of dip::GaussianMixtureModel() that takes a 1D histogram as input.
dip::GaussFT() has a new parameter boundaryCondition, so you don’t need to manually pad and crop the image when doing Gaussian convolution through the FFT.
dip::Quartiles() computes the minimum, maximum and three percentiles much more efficiently than it could be computed when calling dip::Percentile() three times.

Of course there are many more changes and bug fixes, see the change log.

There is only one new algorithm (though it’s fairly trivial): dip::Histogram now can use the Freedman–Diaconis rule to choose the bin size. Use the new function dip::Histogram::OptimalConfiguration() to create the configuration object that enables this rule.

This change has the biggest impact of all the changes in this release, because we’re now using it in all the functions that use a histogram to determine a threshold. In some situations, these algorithms weren’t correct because the histogram that they relied on did not have enough resolution.

For example, consider the following example image:

import diplib as dip

img = dip.Image((1024, 1024))
img.Fill(20)
dip.GaussianNoise(img, out=img, variance=16)
dip.DrawBandlimitedPoint(img, (500, 400), value=1000000, sigmas=10)
dip.DrawBandlimitedPoint(img, (800, 200), value=50000, sigmas=3)
dip.DrawBandlimitedPoint(img, (700, 800), value=120000, sigmas=2)
dip.DrawBandlimitedPoint(img, (400, 600), value=400000, sigmas=4)
dip.DrawBandlimitedPoint(img, (200, 300), value=80000, sigmas=3)

The background has a value of 20.0 with normally distributed noise with a variance of 16.0 (standard deviation of 4.0). The foreground is a series of relatively small dots, the brightest one has a peak value of around 4800. This is a challenging image to generate a histogram for, but it’s not out of the ordinary, there are microscope systems producing 12 or 14-bit images like this.

The default histogram generated for this image has 256 bins, stretching from the minimum to the maximum value. This makes the bin size equal to 18.74. All of the background pixels will fall in the same bottom bin:

dip.Histogram(img).Show()

Using such a histogram to estimate the background distribution is simply not possible, the estimate will be wrong by definition.

Using the new histogram configuration, the bin size is computed to be 0.1069. The Freedman–Diaconis rule takes the interquartile range (IRQ, the distance from the 25th to the 75th percentile), and the number of pixels in the image, , and determines the bin size as

Given that such a small bin size could potentially lead to a humongous histogram, pixels 50 IQR above the upper quartile or below the lower quartile are ignored. Thus, in our case, the histogram will be cropped on the top side, and has 2751 bins:

dip.Histogram(img, configuration=dip.Histogram.OptimalConfiguration()).Show()

The pixels that are ignored are not expected to be meaningful, but of course they could be. The alternative configuration dip.Histogram.OptimalConfigurationWithFullRange() does not crop the histogram, leading to 146284 bins.

All the threshold functions that use the histogram to determine a threshold, now use the new OptimalConfiguration(). The functions impacted are:

These functions will no longer always produce the same output, because the threshold value previously was not always correct. Of course, it is possible that an image has a very strange histogram and the algorithm still won’t compute the correct threshold, but it is now much more likely for it to be correct.

For example, applying dip::BackgroundThreshold() to the image above previously yielded a threshold of 58.51, now it yields a threshold of 29.08. We expected a threshold of 29.42, computed as the mean plus two times the half-width at half maximum (HWHM) of the background Gaussian peak. The HWHM is approximately 1.1775 times the stdandard deviation for a Gaussian. So we compute 20 + 2 · 1.1775 · 4.0 = 29.42. The difference in this example is not as extreme as the case that prompted this change, see issue #161 on GitHub.

To reproduce the old behavior, compute a default histogram, then call the equivalent threshold function that takes a histogram as input. This function will compute the threshold, which can then be applied to the image using a comparison operator:

// Existing C++ code:
dip::Image bin = dip::OtsuThreshold( img );
// Code that reproduces the old behavior of the existing code:
double threshold = dip::OtsuThreshold( dip::Histogram( img ));
dip::Image bin = img >= threshold;

Or, in Python:

# Existing Python code:
bin = dip.OtsuThreshold( img )
# Code that reproduces the old behavior of the existing code:
threshold = dip.OtsuThreshold( dip.Histogram( img ))
bin = img >= threshold

Let us know what you think of this new functionality!

Improving the selection of labeled objects

2024-05-30T00:00:00-06:00

Say, you have a labeled image, like this:

And say, some of those labeled regions are not objects of interest. You want to erase those labels from the image. This is a really simple concept, and very generic. It should be easy to do.

import diplib as dip
lab = dip.ImageRead("labels.png")  # the labeled image above
lab.SetPixelSize(0.1, "μm")
msr = dip.MeasurementTool.Measure(lab, features=["Size", "Circularity"])

We decided on a threshold of 2.0 μm for the size, anything smaller is not of interest. How do we go about erasing those objects?

The simple method is to paint each object with the measurement value, then threshold the resulting image and use that to mask the original label image:

mask = dip.ObjectToMeasurement(lab, msr["Size"]) >= 2.0
new_lab = lab.Copy()
new_lab *= mask

This is, however, not at all efficient. And doing this for multiple conditions on multiple features it quickly becomes comical.

Another way to erase these objects is to iterate over the rows (objects) of the measurement table, and for each row (object) we make a decision whether to keep the object or not. To make erasing objects efficient, we create a lookup table that maps all labels to themselves, except the labels we want to erase, which should map to 0. Finally, we apply the lookup table to the labeled image, and produce the new image with only the selected objects.

In Python, we can use NumPy to avoid loops. For example,

import numpy as np
ids = np.asarray(msr.Objects())
size = np.asarray(msr["Size"]).squeeze()
ids = ids[size >= 2.0]

ids now contains the list of labels we want to keep. We create and apply the lookup table:

lut = np.zeros(msr.NumberOfObjects() + 1, dtype=np.uint32)
lut[ids] = ids
new_lab = dip.LookupTable(lut).Apply(lab)

In C++, the code would be something like this:

#include <vector>
#include <diplib.h>
#include <diplib/lookup_table.h>
#include <diplib/measurement.h>
#include <diplib/regions.h>
#include <diplib/simple_file_io.h>

int main() {
   dip::Image lab = dip::ImageRead("labels.png");  // the labeled image above
   lab.SetPixelSize(0.1 * dip::Units::Micrometer());
   dip::MeasurementTool measurementTool;
   dip::Measurement msr = measurementTool.Measure(lab, {}, {"Size", "Circularity"});

   std::vector<dip::uint32> lut(msr.NumberOfObjects() + 1, 0);
   auto it = msr["Size"].FirstObject();
   do {
      if (*it >= 2.0) {
         lut[it.ObjectID()] = it.ObjectID();
      }
   } while(++it);
   dip::Image new_lab = dip::LookupTable(lut.begin(), lut.end()).Apply(lab);
   // ...
}

This is not any more complicated than in Python with NumPy, it’s just… different.

Either way, the result is the following, identical to the first image but with the smaller objects removed:

A simple API

Usually, when a concept is this simple, and it takes me more than a minute to work out how to do it, I get miffed. And when it’s a highly generic concept, I want to add functionality in DIPlib to make it easy to do.

I’ll spare you the messy details of the design processes. What I finally came up with is this:

The comparison operators are overloaded to compare a measurement object column (e.g. the result of msr["Size"]) to a value. The result is a new object, dip::LabelMap.
The new object dip::LabelMap maps labels to either themselves or to 0. It’s a specialized version of dip::LookupTable, specifically for object labels and label images.
dip::LabelMap::Apply() applies the lookup table to a label image.

With this new functionality, that last section of C++ code can be written as:

dip::LabelMap map = msr["Size"] >= 2.0;
dip::Image new_lab = map.Apply(lab);

Neat, eh?

But this is not all:

Bitwise logical operators (&, |, ^ and ~) are overloaded to combine multiple dip::LabelMap objects. This means we can be more refined in our selection:
```
dip::LabelMap map = (msr["Size"] >= 2.0) & (msr["Circularity"] < 0.1);
```
The map object can also map labels to other labels. map.Relabel() has the same functionality as dip::Relabel(lab), and we can further modify the mapping by manually assigning labels with map[5] = 4. This should allow for quite interesting workflows. For example relabeling labeled tiles of a larger image before stitching them back together. Or reassigning labels of 2D slices of a 3D image, or of frames in a video, so that they match from slice to slice or frame to frame.
map.Apply(msr) returns a new dip::Measurement object with rows deleted and the remaining object IDs mapped according to the dip::LabelMap.

Implementation details

If you’re interested in seeing the implementation details, you’ll find them here:

All this functionality is identically available in Python, so the Python code becomes similar to the C++ code:

new_lab = (msr["Size"] >= 2.0).Apply(lab)

I’m sure this functionality will expand in the future. A few notes:

I have not overloaded arithmetic operators for measurement columns, though that would be quite neat to have. It would require a new object class, distinct from the current one, to hold the result of the operation. The current column object references data in the dip::Measurement object, and cannot be updated to hold its own data. If we create such a new class, we’d have to overload all functions that currently take a measurement column to also work with this new object – quite a lot of work!
But, it would be fairly simple to overload comparison operators to compare two measurement columns. Would this be generally useful?
I overloaded the bit-wise logical operators (&) rather than the regular logical operators (&&) because the latter, when overloaded, don’t do short-circuiting, which I think might lead to programming errors. But because we don’t really do a bit-wise operation, this could also be misleading. So don’t think of these operators as bit-wise logical operators. Instead, think of them as element-wise logical operators. For an integer, the elements are the bits. For a dip::LabelMap, the elements are the labels.
Note that in Python, the bit-wise logical operators have higher precedence than the comparison operators. So you do need the parentheses in the expression (msr["Size"] >= 2.0) & (msr["Circularity"] < 0.1). In C++ this is not the case (though I like to be explicit and put in the parentheses any way).

Edit 2024-05-31

A reader suggested that I warn Python users about operator chaining. You cannot use operator chaining with these dip.LabelMap objects in Python. In Python, 1 < x < 5 evaluates as 1 < x and x < 5. But because we cannot overload and, this does not do what the user was expecting. This is true also for NumPy arrays and DIPlib images, as well as, I imagine, many objects across many packages. So don’t use operator chaining!

Please let me know what you think!

Implementing the convolution

2023-12-28T00:00:00-07:00

I find myself regularly explaining how to efficiently implement a convolution (a linear filter). This blog post will combine all my tips and tricks regarding convolution. I’ll start with the 1D case, then will expand into the multi-dimensional case.

In the 1D case, I’ll use as the variable, x(t) is a discrete input signal of finite length (a set of samples) and only defined for integer . We’ll consider , where is the kernel and is the result of convolving the signal and the kernel. We care here only for the case where has the same number of samples as .

1D convolution

There are two different ways that a 1D convolution can be defined. There’s the Infinite Impulse Response (IIR) and the Finite Impulse Response (FIR) filters.

An IIR filter uses previous output values to compute the current output value. That is, it is recursive. For example,

To see what the impulse resonse of this filter is, we’d have to apply it to an impulse signal (a Dirac delta). If we were to do this, we’d find that the impulse resonse (or kernel) is zero for , and non-zero for . That is, it is causal (it doesn’t depend on any future data), and it is infinite (it never really ends).

An IIR filter can also be anti-causal (does not depend on any past data), such a filter would be computed backwards in time.

A FIR filer uses only a finite set of input values to compute the current output value. For example,

A FIR filter can also be causal or anti-causal, but it can also be neither, if it uses past and future input samples.

Implementation

Both the FIR and IIR filters can be implemented quite trivially, except for a few details we’ll discuss below. But most output samples we can compute simply by applying the equation that defines the filter. In the FIR case, each output value can be computed independently of others, and so is trivial to parallelize. The FIR filter, having a recursive definition, must be computed left to right (for the causal case) or right to left (for anti-causal case).

In some cases, there are some tricks worth thinking about. For example, if the FIR kernel is symmetric, the computations can be rearranged to reduce cost:

For longer kernels the savings can be significant.

Boundary condition

For both types of filter, we need to decide what happens when we need to read values outside the domain. In the FIR case, for we need to fill in a value for , which we do not know (there are many ways to choose these values, what is best depends on the application, we won’t go into that here). In the IIR case, we need to initialize for the set of values of that we will need to read.

But what matters is how we determine whether we can read a given value or not. A conditional expression is very expensive in modern hardware. Code like

if t >= 0 {
   sum += x[t] * h[k];
}

sum += x[max(t, 0)] * h[k];

will significantly slow down execution. We want to avoid these tests for the bulk of the signal that is sufficiently far from the boundary, where we know for sure we can read all required input values. There are two ways of accomplishing this:

Copy the input signal into a larger buffer (i.e. add padding), and fill in the samples outside the known signal in our desired way. We can now safely read x[-1] for example. In the IIR case this translates to creating a larger output buffer to write in and read from.
Write three separate loops: the first one applies the filter for the first few output values where the boundary plays a role, a second loop processes the bulk of the signal and has no conditional statements, and a third loop produces the last few output values where the boundary again plays a role (in the skeleton code below, N is the number of samples in x and y, K is the number of samples in h, we assume K is odd and the origin of the kernel is in the center):
```
int i = 0;
int margin = K / 2; // floor division!
for (; i < margin; ++i) {
   // compute convolution applying boundary condition
   // on the left side of the input
}
for (; i < N - margin; ++i) {
   // compute convolution without any conditional statements
}
for (; i < N; ++i) {
   // compute convolution applying boundary condition
   // on the right side of the input
}
```

[I would typically not use int for indexing or loops, but wanted to keep the code above simple.]

Fast Fourier Transform

A FIR filter can be implemented efficiently using the FFT only if the kernel is large. The FFT is relatively expensive, and not worth using for smaller kernels.

To use this method, pad the kernel h with zeros to the same size as the input x, apply the FFT to both, multiply them (remember the output of the FFT is complex, we must properly multiply the complex numbers), then compute the inverse transform of the result. The inverse transform should be real-valued, but the imaginary component will likely have some very small values in it due to floating-point rounding errors. Simply ignore the imaginary component, do not take the magnitude (absolute value) of the complex numbers.

When using the FFT, remember that the FFT takes the first sample as the origin. For the signal it doesn’t matter where the origin is, but for the kernel this is indeed very important. If we define the kernel with the origin in the middle, a careless transformation of the kernel will result in the output signal being shifted.

Assuming again a kernel h with K samples and the origin in the middle (at K/2, using floor division), we must pad the kernel as follows:

padded_h = cmalloc(N * sizeof(*padded_h)); // initialized to 0
memcopy(padded_h, h + K / 2, K / 2 + 1);
memcopy(padded_h + N - K / 2, h, K / 2);

The convolution is now applied with (pseudocode):

X = fft(x, N);
H = fft(padded_h, N);
Y = malloc(N * sizeof(*Y));
for (int i = 0; i < N; ++i) {
   Y[i] = X[i] * H[i];
}
y = ifft(Y, N); // if y is complex, keep the real part

But note that the boundary condition imposed by the FFT is periodic: . This is usually not a desireable boundary condition. So we must pad the input before applying the FFT. We can pad x on both sides by K/2, using our desired method (h must be padded to this new length!), and crop the final result back to the original size. Since we’re padding anyway, we can choose a slightly larger size that will be cheaper to compute the FFT for. The FFT is most efficient when the length N can be decomposed into a product of small integers (typically 2, 3 and 5, maybe also 7 depending on the implementation we’re using).

nD convolution

The nD case is more or less the same as the 1D case. I don’t know of any nD IIR filters, so I’ll discuss FIR filters only. Applying the convolution requires looping over multiple dimensions, and we need to take care of boundary conditions in all dimensions. Again, the simple way is to copy the image into a larger buffer, adding pixels outside the original boundary. The more complex way would require 3ⁿ loops (for n dimensions): each of the three loops discussed in the 1D case would have three loops inside it for the second dimension, etc. Each inner loop handles a corner, and edge or the central part of the image. For the 2D case this might still be doable, but it becomes increasingly difficult for higher dimensions. The copy required to pad the image doesn’t sound so bad after all…

But there is an additional trick that comes into play in the nD case: some kernels can be decomposed into a set of 1D kernels.

Kernel decomposition

We know that the convolution is associative. That is, (x * h_1) * h_2) = x * (h_1 * h_2). This can be used in two ways:

When applying multiple convolutions in sequence, we can combine the kernels together, and apply them all as a single convolution. This saves in memory accesses, since we’re looping over the image only once. In 1D, if one kernel has size K and the other has size M, then the combined kernel has size K + M - 1, so there is not really a computational advantage other than the more efficient memory access. When using the FFT, the core advantage is the fewer FFTs one needs to compute.
If we can decompose a kernel into a set of 1D kernels, we can significantly reduce the computational cost. For example, a kernel of pixels takes multiplications and additions to compute, but if we can decompose it into two 1D kernels of length , then the cost is reduced to multiplications and additions. The larger , the larger the savings. Kernels that can be decomposed like this are the square kernel with uniform weights, the Gaussian kernel and all its derivaties, the Sobel kernels, etc. These kernels are called separable.

However, applying two 1D kernels in succession does require iterating over the image twice, which adds to the cost. So very small kernels like Sobel are not worth decomposing.

The Gaussian filter

But the Gaussian kernel certainly is worth decomposing. Decomposing the Gaussian kernel makes it trivial to implement a Gaussian filter for an image with an arbitrary number of dimensions. We only need to implement the 1D Gaussian, then just loop over the lines along each of the image dimensions. The 1D Gaussian can be implemented as a FIR filter, as an IIR filter, and through the FFT. Which one to pick depends on the sigma: The FIR filter increases in cost with the sigma. The IIR filter is quite expensive, but independent of sigma. And so for sigma above some value, the IIR filter is the most efficient method. The FFT method is always the most expensive, but is useful when the sigma is so small that it cannot be implemented correctly using a FIR filter (the lower limit for sampling a Gaussian kernel is a sigma of about 0.8 pixels).

In closing

To know which implementation is faster, which tricks to apply, you must always measure. There are rules of thumb you can use, but each machine is different, and each situation is unique. Try different things and time them all!

This is a graph comparing the speed of the three methods to compute the Gaussian filter, as implemented in DIPlib, on my machine. We apply the filter to image of 2000x2000 pixels, for different sigma:

On my machine, for a sigma of 10 pixels, the FIR and IIR filters are equally fast. It is never faster to use the FFT to compute a Gaussian filter. Would these conclusions be different on your machine?

Color maps for image display

2023-10-22T00:00:00-06:00

Today’s post discusses different possible color maps you can use to display a scalar image (i.e. an image with a single channel, often referred to as a grayscale image). A color map is necessary to translate the pixel values of the image into colors to show on the screen. When using a grayscale color map, each pixel is represented by its own gray value. More colorful color maps are often employed to either make the display more attractive, to enhance our ability to distinguish specific values, or to highlight specific transitions. Unfortunately, many color maps in frequent use also distort the image we perceive.

There are four general categories that I’ll cover in turn: the grayscale color map, rainbow color maps, linear (or sequential) color maps, and special-purpose color maps.

The grayscale color map

This is an obvious choice, considering we’re displaying a grayscale image. And it usually is the choice I make when working with arbitrary images. It gives me the best ability to judge the shape encoded by the image data. For example, the first image here is a Gaussian blob; I can recognize it as such from the image. Next is a Gaussian with a clipped peak; again I can tell the peak is flat. Finally, a blob with a different shape; it is immediately apparent to me that this is not a Gaussian, that its tail is heavier than that of a Gaussian and its peak is sharper.

Some image display tools will use the grayscale color map by default for grayscale images. But some prefer a more colorful color map as the default. For example Matplotlib uses “viridis” by default. This is a perceptually uniform linear color map (more on that below), so not a terrible choice, but I think it causes more confusion among novices than the grayscale color map.

Note that a properly configured display is required for the grayscale color map (or any other color map) to produce a useful image on the screen. Modern displays, straight from the factory, are much more likely to show the expected intensities than the old cathode ray tubes. We don’t need to calibrate display gamma anymore. But for proper color perception one still needs to use the display in a properly illuminated environment, and calibrate the display’s colors to that environmental illumination. This means that when we limit ourselves to grayscale, the image will likely look the same on any modern display, but when using color, different uncalibrated displays might show different things.

Rainbow color maps

Much has been written about the problems with the rainbow color maps. See for example Borland and Taylor [1], and the references therein. The main arguments are:

No natural ordering to the colors: does red represent a larger value than green or a smaller one?
Not perceptually uniform: at some points, the color map has too much contrast, at others it doesn’t have much at all. This means that some image features can be hidden, and some artificial features can appear.
Some colors, such as yellow or cyan, appear brighter and therefore draw more attention than other colors.

Here is the Gaussian from before with MATLAB’s “jet” color map applied:

Besides the main points argued above, the peak of the Gaussian is a lot darker than other parts, to me the shape for this blob is a donut, not a simple peak.

Still, this has not stopped people from using them. Rainbow color maps are still used frequently (though luckily much less frequently than in the past). And because of that, some people have attempted to improve on them to reduce some of their worst properties. In particular, I’m aware of these modern rainbow color maps:

ColorCET rainbow color maps by Peter Kovesi (2015),
batlow color map by Fabio Crameri (2018), and
turbo color map by Anton Mikhailov (2019).

ColorCET is a large collection of color maps, produced with methods described in a paper [2]. There is a lot we know about perception of color, which can be translated into perceptually-uniform color maps. Now, a rainbow color map can never be perceptually uniform, but the ones in ColorCET are certainly much better approximations.

Likewise, “batlow” comes from a larger collection of color maps also produced according to scientific principles [3]. In contrast, “turbo” was created by eye-balling something that looked good to its author. Here is the Gaussian again, with the three color maps applied:

None of these is an ideal representation of the data. “Batlow” comes closest, though it also is the least like a rainbow color map. In fact, I think it belongs more to the next section, since its intensity increases from dark to light. But it is sold as a rainbow color map by its author.

We see that the ColorCET rainbow color map solves some of the problems of older rainbow color maps, though it still produces rings where none exist in the data. It’s author refers to it as “the least worst rainbow colour map I can devise.”

“Turbo” is more uniform than “jet”, but it still keeps the problem of high values having dark colors. It is the only one of the three that makes the peak look like a donut. To me, this is by far the worst of the batch. Guess which one of these color maps is built into MATLAB and Matplotlib and a whole lot of other tools? That’s right, “turbo” is! Its author works at Google, and published it on the Google Research blog, and that is of course much more important than understanding the science, or the color map actually being any good.

The advantage of rainbow color maps over grayscale or linear color maps is that one can name the colors, which makes it easier to look them up in the reference or compare them across images. On the downside, these clearly identifiable colors introduce cognitive plateaus (even if perceptually they are not).

Linear (or sequential) color maps

To add color to a grayscale image in a proper way, one needs to devise a color map that doesn’t fall in any of the pitfalls of the rainbow color map. Having the lightness (intensity) of the colors increase linearly from dark to bright does several things. First of all, it introduces a unique, intuitive ordering to the colors. Additionally, it allows the figure to be printed in black-and-white, and still maintain its properties. Next, we want to change the hue from one color to another, sometimes through a third color. Doing so in a perceptually uniform manner is not trivial, but we understand enough about color perception that such a transition can be computed.

The main reason to add the hue in this case is to improve the intensity resolution; color improves our ability to see small differences in intensity without reducing the range of intensities displayed.

Many authors have proposed manually-generated color maps approximating perceptual linearity. But, as far as I know, the first such color maps generated algorithmically using color perception theory were Peter Kovesi’s ColorCET [2]. Matplotlib also introduced several such color maps, its default is “viridis”. Some years later, MATLAB introduced the “parula” color map as its new default.

Here is the Gaussian with these three color maps (I picked my favorite from the ColorCET collection):

It is clear that “parula” is the inferior of the three here, with quite a strong orange band. But none of them allow me to recognize the shape as well as the grayscale color map. Granted, I have a lot of experience looking at images (and in particular Gaussian blobs) displayed with a grayscale color map. Would I be able to see the Gaussian shape equally clearly with one of these color maps if I had the same level of experience looking at images displayed with that color map? Maybe!

Special-purpose color maps

Color can do a lot more than improving out ability to discern small differences in intensities. We can also use it to label distinct intensity ranges. Below are three examples.

Cyclic color maps

If the image’s intensities represent angles, we want the color for 360° and 0° to be the same. This requires a cyclic color map (i.e. it must be periodic).

Let’s take as an example the angle of the gradient of the Gaussian image used above. When using a grayscale color map, it has a sharp transition at 180° (the angle computed is in the range of -180° to 180°). Using one of the cyclic color maps from ColorCET, the sharp transition is no longer there.

We can now identify green as 0°, yellow as 90°, magenta as 180°, and blue as -90°.

Divergent color maps

Similarly, when the sign of the image intensities is relevant, we can represent negative values with a different color from positive values. In this case, our color map is two linear color maps, both start at the same color (usually gray, black or white) for 0, and have increasingly bright and/or saturated colors for larger magnitudes.

I experimented with these divergent color maps in a post many years ago. This was before the ColorCET color maps were published, and before I learned the little bit I know now about color perception.

For example, this is the difference between the Gaussian blob and the heavy-tailed blob we displayed earlier:

The neutral gray when using the diverging color map indicates where zero is.

Sharp transitions

Sometimes one specific gray value is important to highlight. For example, this satellite image represents surface height. It is really hard to see what part of the world is represented, unless we know what we’re looking for:

Using a color map with a jump at height of 0, coloring negative values with different colors from positive values, the outline of land mass becomes immediately obvious, and anyone will be able to recognize, without difficulty, what part of the world was imaged:

Another example is the color map often used on microscopes (especially fluorescent microscopes) where pixels with a value of 0 are shown in blue, and pixels with a value of 255 are shown in red. The blue pixels thus indicate underexposed pixels, and red ones overexposed pixels. The operator can then adjust gain and offset to reduce these blue and red pixels as much as possible. Here’s an example of an image with this color map applied:

Finally, this is a nice example from this page, where a grayscale color map is enhanced to highlight regions where oxygen saturation is too high or too low. The color map has yellow and red colors for these regions:

Conclusion

So, for me, the go-to color map will always be the grayscale color map. It is what best allows me to recognize the shape formed by the image intensities. To see smaller intensity differences, the first thing I do is change the intensity range that is mapped to the black-white range on the screen. Of course this cannot be done in a publication, where interaction with the display is typically not possible.

Adding color can improve the resolution of the color map, and it can also highlight specific values. For signed difference images I typically use a diverging color map, where 0 is recognizable and the hue encodes the sign. And for angles I typically use a 4-color cyclic color map, where each cardinal direction has a specific color we can name. But it is important, in all these cases, that the color map be perceptually uniform, otherwise we will distort our perception of the data (of course sometimes we might want to lie about our data, which is possible to do with non-uniform color maps).

Finally, in some very specific cases, one might want to build a custom color map specifically designed to highlight some chosen intensities. But this is something one would do for publication, not for data exploration.

Literature

David Borland and Russel M. Taylor II, “Rainbow Color Map (Still) Considered Harmful”, IEEE Computer Graphics and Applications 27(2), 2007 [DOI] [PDF]
Peter Kovesi, “Good Colour Maps: How to Design Them”, arXiv:1509.03700 [cs.GR], 2015
Fabio Crameri, “Scientific colour maps”, Zenodo, 2018 [DOI]

OpenCV is not designed for quantification

2023-01-02T00:00:00-07:00

OpenCV is undoubtedly the most popular library for image processing and computer vision. According to its website,

OpenCV is a highly optimized library with focus on real-time applications.

That is, it is built to do real-time computer vision, not precise measurement. This means that OpenCV prioritizes speed over precision. There are many applications where this is the right choice. But there are applications where it isn’t: in my line of work, precision is important.

Over the years I have collected links to questions on Stack Overflow that were prompted by OpenCV’s lack of precision. In today’s blog post I wanted to revisit some of these unexpected results and their cause. This is not meant as an OpenCV-bashing post (though I do enjoy a good bashing now and again), but rather an exploration of the limitations of OpenCV, and maybe a demonstration of why a library like DIPlib has a place in a world dominated by OpenCV.

What I am criticizing here is people trying to use OpenCV for applications where a library such as DIPlib would be a more appropriate choice. If you need precise results or accurate measurements use DIPlib, not OpenCV.

Rounding in interpolation

In this answer, Andras Deak shows how OpenCV’s interpolation output is quantized. It looks like they use fixed-point arithmetic, with a precision of 1/32. I don’t doubt this speeds up computations, but it did produce unexpected results at least once.

This reproduces the findings. It applies linear interpolation to an image with some pixels set to 0 and some pixels set to 1. It generates 100 new samples in between the last 0 pixel and its neighboring 1 pixel. One would expect the result to be a ramp, a straight line going from 0 to 1. Instead, we see a staircases with 33 steps. Each step is exactly 1/32 high.

import numpy as np
import cv2
import matplotlib.pyplot as plt

input = np.zeros((1, 100), dtype=np.float32)
input[:, 50:] = 1

grid_x = np.linspace(49, 50, 100, dtype=np.float32)[None, :]
grid_y = np.zeros(grid_x.shape, dtype=np.float32)
output = cv2.remap(input, grid_x, grid_y, interpolation=cv2.INTER_LINEAR)

plt.plot(output[0, :], '-')
plt.show()

Color space conversion is lossy

This question is from someone surprised at the round trip RGB (or rather BGR, why reverse the order?) to HSV and then back to RGB causes visible color banding.

import cv2
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(256)
x, y = np.meshgrid(x, x)
z = np.zeros(x.shape)
img = np.stack((x / 3 + 165, y / 3 + 100, z), axis=2)
img = img.astype(np.uint8)

imghsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
imgback = cv2.cvtColor(imghsv, cv2.COLOR_HSV2BGR)

_, ax = plt.subplots(1, 2)
ax[0].imshow(img)
ax[1].imshow(imgback)
plt.show()

The problem is caused by rounding, which apparently happens mostly in the hue channel. OpenCV rounds the 360 degrees of hue to 180 integer values, meaning that 30% of the dynamic range on the hue channel is not used. That said, I’m not sure if mapping the 360 degrees to all 256 values of the 8-bit integer would be sufficient to make the round trip not introduce visible artifacts. DIPlib outputs floating-point images after color conversion by default, as does scikit-image. Using floating-point images is the only way to guarantee a lossless roundtrip.

Perimeter measurement is biased and imprecise

The more problematic precision issues are the ones where it is not done on purpose to save time. For example, OpenCV’s object perimeter measurement is imprecise and biased, even though computing it properly is not a significantly more expensive. I wrote about computing the perimeter many years ago. OpenCV measures the length of the polygon that goes through each boundary pixel. This measurement is correct only for axis-aligned rectangles and rectangles rotated by 45 degrees. It overestimates the length of any other perimeter.

This experiment computes the perimeter of a square at many orientations, averaging over different sub-pixel shifts to get a better estimate of the average measurement error:

import cv2
import diplib as dip
import numpy as np
import matplotlib.pyplot as plt

R = 80
N = 16
angles = np.linspace(0, np.pi / 2, 41)
perimeter = []
xx = dip.CreateXCoordinate((256, 256))
yy = dip.CreateYCoordinate((256, 256))
rng = np.random.default_rng()
for phi in angles:
    cv_perimeter = 0
    dip_perimeter = 0
    for jj in range(N):
        offset = rng.random(2)
        x = (xx + offset[0]) * np.cos(phi) - (yy + offset[1]) * np.sin(phi)
        y = (xx + offset[0]) * np.sin(phi) + (yy + offset[1]) * np.cos(phi)
        img = (dip.Abs(x) <= R) & (dip.Abs(y) <= R)
        arr = np.asarray(img, dtype=np.uint8)
        cnt, _ = cv2.findContours(arr, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
        cv_perimeter += cv2.arcLength(cnt[0], True)
        msr = dip.MeasurementTool.Measure(+img, features=['Perimeter'])
        dip_perimeter += msr[1]['Perimeter'][0]
    perimeter.append((cv_perimeter / N, dip_perimeter / N))

plt.plot(angles, perimeter)
plt.plot([0, np.pi / 2], [8 * (R - 0.5), 8 * (R - 0.5)], 'k:')
plt.xlim((0, np.pi / 2))
plt.xticks(np.linspace(0, np.pi / 2, 5), ['0', 'π/8', 'π/4', '3π/4', 'π/2'])
plt.legend(['OpenCV', 'DIPlib'])
plt.xlabel('angle')
plt.ylabel('estimated perimeter')
plt.title('Perimeter of rotated square')
plt.show()

As I mentioned earlier, OpenCV gets it exactly right for 0 and 45 degree angles, and overestimates the perimeter for all other angles. In contrast, the corner counting method used by DIPlib is unbiased (averaged over all angles, it gets the right value), and the maximum error is much smaller than for the naive method.

The poor perimeter estimate recently stumped two people, posting questions on Stack Overflow about it less than a week apart (#1, #2). Both were trying to compute the circularity measure, (with the area and the perimeter). This is supposed to be 1 for a circle, and smaller for other shapes. But because OpenCV overestimates the perimeter of the circle, this measure never reaches 1 (except for very small circles, where it severely underestimates the perimeter).

This experiment draws circles of different radii, with different sub-pixel shifts, and measures their perimeter using OpenCV and DIPlib:

import cv2
import diplib as dip
import numpy as np
import matplotlib.pyplot as plt

radii = [2, 3, 5, 10, 20, 30, 50, 100, 200]
sz = 10 + 2 * radii[-1]
error = []
N = 16
xx = dip.CreateXCoordinate((sz, sz))
yy = dip.CreateYCoordinate((sz, sz))
rng = np.random.default_rng()
for R in radii:
    cv_perimeter = 0
    dip_perimeter = 0
    for jj in range(N):
        offset = rng.random(2)
        img = ( (xx + offset[0])**2 + (yy + offset[1])**2 ) < R**2
        arr = np.asarray(img, dtype=np.uint8)
        cnt, _ = cv2.findContours(arr, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
        cv_perimeter += cv2.arcLength(cnt[0], True)
        msr = dip.MeasurementTool.Measure(+img, features=['Perimeter'])
        dip_perimeter += msr[1]['Perimeter'][0]
    true_perimeter = 2 * np.pi * R
    error.append((cv_perimeter / N / true_perimeter, dip_perimeter / N / true_perimeter))

plt.plot(radii, error)
plt.plot([2, 200], [1, 1], 'k:')
plt.xscale('log')
plt.legend(['OpenCV', 'DIPlib'])
plt.xlabel('radius (px)')
plt.ylabel('measured perimeter as fraction of true perimeter')
plt.title('Error in circle perimeter measurement')
plt.show()

This shows that OpenCV tends to overestimate the circle perimeter by a factor of 6% for larger circles, whereas the unbiased estimator that DIPlib uses tends towards the right value for larger circles. For very small circles OpenCV underestimates the perimeter. This is because it measures the length of the polygon formed by the centers of the border pixels of the object. This polygon is slightly smaller than the object itself, as it doesn’t go around all the object pixels, it goes through the outermost pixels of the object. For circles this introduces a bias in the perimeter length of (the radius is 0.5 pixels too small, and is ). This bias is small in comparison to the bias due to the length overestimation for larger circles, but for very small circles it dominates.

Non-symmetric circular structuring elements

My PhD thesis was about using morphological openings and closings to measure grain sizes, using what is called a granulometry. If I had used OpenCV for my thesis, I would have had to roll my own circular structuring elements. Because OpenCV’s are not circularly symmetric. If you create any circular structuring element, then rotate it 90 degrees (or equivalently, transpose the array), then you get a different shape:

import cv2
import numpy as np
import matplotlib.pyplot as plt

kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (45, 45))
comparison = np.dstack((kernel, kernel.transpose(), kernel.transpose())) * 255
plt.imshow(comparison)
plt.show()

The white and red parts together is the generated kernel, the transposed kernel is the white and cyan parts.

This has already been filed as a bug. And as I’ve said before, we shouldn’t fault a large, complex library like OpenCV for having bugs. But that this type of bug exists does illustrate that the library’s focus is not on precise results; if it were, this bug would never have gotten though testing.

Non-idempotent morphological opening

Another strange issue with the morphological opening and closing is shown in this question. The image boundary is not treated correctly, causing repeated applications of the same filter to continue affecting the image. The opening and closing are supposed to be idempotent, meaning that applying the same filter multiple times in sequence should have the same effect as applying it only once.

I wouldn’t fault anyone for a bug, what prompts me to include it here is that this was reported as a bug, but the ticket was dismissed as a “question or other no-action item”, which I interpret as nobody being interested in fixing it. This is an issue that matters in many cases, but none of those cases are intended use cases of OpenCV. Typical users of OpenCV wouldn’t care about the difference.

Pillow

OpenCV is not the only image processing package with issues. For example, Pillow’s convolution mirrors the kernel along only one dimension, which is just a simple bug that I wouldn’t fault anyone for. Except that it exists since the early days of PIL, and the maintainer says it cannot be fixed!

One point of criticism

All of that wasn’t criticism, OpenCV developers made choices favouring speed over precision. Those choices obviously will produce surprises if one wants to use OpenCV for applications that require precision. You just need to use the right tool for the job!

However, given that OpenCV is designed around speed, it is indeed very strange that it doesn’t always implement the fastest known algorithms.

Take for example the Gaussian filter, arguably the most important basic filter in any image processing toolbox. This experiment times the computation of the Gaussian filter with a sigma of 5 along both axes, repeated 1000 times. Both libraries implement the filter quite similarly: They compute the kernel size differently based on the sigma parameter, but in this case they both produce a Gaussian kernel with 31 samples along each axis. Both libraries compute the Gaussian as a separable filter, as one would expect.

import numpy as np
import diplib as dip
import cv2
import timeit

rng = np.random.default_rng()
img = rng.integers(256, size=(1024, 1280)).astype(np.float32)

print(timeit.timeit(
    lambda: cv2.GaussianBlur(img, ksize=(0, 0), sigmaX=5),
    number=1000))
print(timeit.timeit(
    lambda: dip.Gauss(img, sigmas=5),
    number=1000))

On my machine, DIPlib’s filter is about twice as fast (6.87 s for OpenCV, 3.17 s for DIPlib).

As I understand it, OpenCV does not use the symmetry of the 1D Gaussian kernel when computing the convolution. For a general 1D kernel with samples, applied to an image line at , we compute

But if we know that is symmetric, with and so on, then we can instead compute (assuming is odd)

saving us multiplications. If OpenCV is serious about speed, this seems an obvious optimization to make. DIPlib has a dedicated convolution also for 1D antisymmetric kernels, such as the first and third derivatives of the Gaussian.

Likewise, OpenCV’s dilation, the most basic morphological operator, is the naive algorithm, although implemented very efficiently. The naive algorithm computes the maximum over the pixels under the kernel, shifts the kernel, computes the maximum over the new set of pixels, etc. It does not attempt to use the result for the previous pixel to make the current pixel’s computation lighter. OpenCV does recognize that the square kernel is separable, and so has an implementation that is linear in the length of the side of the square. A circular kernel is not separable, which means that the naive algorithm is quadratic in the circle’s diameter (it scales with the area).

Here’s an experiment:

import numpy as np
import diplib as dip
import cv2
import timeit
import matplotlib.pyplot as plt

rng = np.random.default_rng()
img = rng.integers(256, size=(1024, 1280)).astype(np.float32)

times = []
sizes = [7, 15, 31, 63, 127]
for size in sizes:
    kernel = cv2.getStructuringElement(cv2.MORPH_RECT, ksize=(size, size))
    t1 = timeit.timeit(
        lambda: cv2.dilate(img, kernel, iterations=1),
        number=1000) / 1000
    t2 = timeit.timeit(
        lambda: dip.Dilation(img, dip.SE(size, "rectangular")),
        number=1000) / 1000
    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, ksize=(size, size))
    t3 = timeit.timeit(
        lambda: cv2.dilate(img, kernel, iterations=1),
        number=100) / 100
    t4 = timeit.timeit(
        lambda: dip.Dilation(img, dip.SE(size, "elliptic")),
        number=100) / 100
    times.append((t1, t2, t3, t4))

times = np.array(times)

plt.plot(sizes, times[:,0], ".:r", label="Square, OpenCV")
plt.plot(sizes, times[:,1], ".:b", label="Square, DIPlib")
plt.plot(sizes, times[:,2], ".-r", label="Circle, OpenCV")
plt.plot(sizes, times[:,3], ".-b", label="Circle, DIPlib")
plt.yscale("log")
plt.xlabel("kernel diameter (pixels)")
plt.ylabel("time for a single dilation (s)")
plt.legend()
plt.show()

For smaller kernels, OpenCV is much faster than DIPlib. They really put effort into the quality of the implementation. But it is obvious from the graph that DIPlib implements the better algorithms. For both kernel shapes, DIPlib scales better than OpenCV: the square kernel is independent of the size, and the circular kernel scales with the diameter (not the area). I have written before about optimizing the wrong algorithm, MATLAB has a similar issue.

By the way, the reason that OpenCV can be so much faster than DIPlib for small kernels is because OpenCV’s implementation is explicitly 2D, and requires pixels within an image line to be consecutive in memory. This allows the implementation to optimize memory access patterns. DIPlib prefers more generic algorithms. DIPlib’s implementation works for an arbitrary number of dimensions, with an arbitrary memory layout. There are some optimizations that are just not possible without those additional constraints.

DIPlib 3.4.0 released

2022-12-09T00:00:00-07:00

Yesterday we released DIPlib version 3.4.0. The change log is quite extensive. The improved median filter I discussed in my previous blog post is in this release, as well as a much faster built-in FFT implementation (used when not linking against FFTW), a bunch of new functionality including various iterative deconvolution algorithms, and some bug fixes.

New function forms in Python bindings

DIPib has always had two signatures for each function, one where the output image is passed as an input argument to be modified, one that returns the output image. For example, the dip::Gauss function can be called in these two ways:

dip::Image out;
dip::Gauss(img, out, {3}, {0}, "FIR");

dip::Image out = dip::Gauss(img, {3}, {0}, "FIR");

(I’m purposefully specifying more parameters here than necessary, bear with me.)

When I first created the Python bindings, PyDIP, I only created bindings for the second form, with the output image as return value. It is the more user-friendly form, and I might have been subconsciously imitating the MATLAB syntax of DIPimage. In this latest release, we’re adding bindings for the other form as well.

In Python it is not possible to have multiple signatures for one function. We use Pybind11 to create the Python bindings for DIPlib, which does allow multiple overloaded functions to be called from Python. It has an overload resolution algorithm built-in, where it attempts to convert input arguments to the types needed by the first overload, if it fails it tries the second overload, etc. But Python is not as strongly typed as C++, so it can be difficult to avoid certain type conversions from happening, making the two overloads difficult to distinguish for some functions. To avoid ambiguity, we made the output image argument a keyword-only argument. All arguments that come after must be keyword-only as well, as per Python syntax. So, whenever there is an out=xxx argument specified, we mean the one function form, and whenever that argument is not specified, we mean the other function form. This has the added benefit of making the output image argument clearly recognizable.

For example, where previously you could call dip.Gauss as

out = dip.Gauss(img, 3, 0, "FIR")

out = dip.Gauss(img, sigmas=3, method="FIR")

you can now also call it as

out = dip.Image()
dip.Gauss(img, out=out, sigmas=3, method="FIR")

(I mean to say sigmas and method must be given as keyword arguments in this new form).

Note, in that last call, that we didn’t need to allocate space for out, we didn’t need to worry about how large the output image would be, or what data type it would have. The DIPlib function will reallocate the output image to match its needs.

Why bother?

The reason that I wanted to add this alternate, less user-friendly form of each function, is because of the possibilities it opens up. The obvious advantage is that it allows re-using already allocated images, reducing memory requirements, and possibly speed up some computations by working in-place. For example,

dip.Abs(img, out=img)

works faster than

img = dip.Abs(img)

But the more important advantage is being able to determine the data type of the output. For this to work we need the “protect” flag (this works the same as in the DIPlib C++ library of course). The “protect” flag protects an image from being reallocated. That is, if an image is protected, its data segment cannot be freed or changed. This makes the image’s sizes and data type fixed. So one could create, for example, complex-valued image, and use that as the output:

out = img.Similar("SCOMPLEX")
out.Protect()
dip.Gauss(img, out=out, sigmas=3, method="FIR")

The image being protected, the dip.Gauss function must use it as-is for its output, and so will produce, in this case, a complex-valued result even if it would otherwise have produced a real-valued floating-point result. Note that if the protected image were to have the wrong sizes or number of tensor elements, an exception would be raised.

To make this process simpler (i.e. letting the filter function do the allocation), DIPlib has another trick: it is possible to protect a raw image (an image without a data segment). For such an image it is still possible to change its sizes and number of tensor elements, and then allocate its data segment. But all DIPlib functions will see the “protect” flag and not change the data type. This allows the following code to work as expected:

out = dip.Image()
out.SetDataType("SCOMPLEX")
out.Protect()
dip.Gauss(img, out=out, sigmas=3, method="FIR")

In-place processing

The following code might not work in place:

dip.Gauss(img, out=img, sigmas=3, method="FIR")

If img is of an integer type, the output will be adjusted to be a floating-point type, and so will still create a new output data segment. To force a function to work in place, protect the image:

img.Protect()
dip.Gauss(img, out=img, sigmas=3, method="FIR")

Some functions cannot work in place, they will always create an intermediate image to do the computation, maybe by copying the input image so that it can be used as a separate output image. Some of the functions that cannot work in place will raise an exception if the input and output images are the same. There will always be things that could be improved!

Let us know what you think of this new functionality!