Cris’ Image Analysis Blog - algorithms

Robust estimation of shift, scale and rotation

2026-02-10T13:27:00-07:00

A frequent problem in image processing is aligning two or more images together. Maybe you have a series of images that partially overlap and need to stitch them into a single, larger image. Maybe you have a video sequence and need to estimate and remove the movement. Maybe you have images of the same subject but in different modalities that you need to combine. In all these cases, the transformation you need to apply to one image to match it to the other is only translation (or shift), rotation, and maybe scaling. (You might also need to apply a correction for lense distortion, but that is outside the scope of this post.)

The most common methods for finding the alignment are all based on markers: First you find a set of distinct points (or markers) in the two images (corners, dots, etc.), then you find the correspondences (figure out which marker in image A corresponds to each marker in image B), then you compute the transformation from those correspondences. And this often works well. The hardest part here is finding the correspondences. When fitting the transformation, some number of mismatches can be accounted for, but most matches need to be correct.

Here I wanted to explain a simpler method to align two images, that is not as widely taught in image processing courses as it should be. It is an efficient, accurate and robust method that directly compares the full images, rather than individual markers. It is based on the cross-correlation, which can be used to find the shift between two images, and the Fourier-Mellin transform, which disregards translation, and converts scaling and rotation to shifts. This method is due to Reddy and Chatterji (1996).

Let’s explore the method step by step, and explain concepts as we go along.

The Fourier Transform

The first step in the method is to compute the magnitude of the Fourier Transform (FT). We’re actually computing the Discrete Fourier Transform (DFT), which is distinct but strongly related to the Fourier Transform, which is defined in the continuous domain. The DFT is computed efficiently using the Fast Fourier Transform algorithm (FFT). But we’ll refer to it simply as the FT.

I’ll assume we all know what the FT is, but I’ll summarize really quickly here: The FT is a reversible transformation of the image that is useful to study linear systems. The FT maps a complex-valued function to another complex-valued function. Complex values can be interpreted as having a real and an imaginary component, or, equivalently, a magnitude and a phase.

A shift in the image is encoded solely in the phase of the FT (images on the right are the magnitude of the FT of the images on their left):

So, taking the magnitude of the FT, we’ve removed the shift from the problem. But what happened to the scaling and the rotation? A scaling of the image results in an inverse scaling of the FT:

And a rotation of the image results in an equal rotation of the FT:

However, because the input image was real-valued, the FT is symmetric about the origin, and therefore we conflated the rotation with φ and the rotation with φ+π. We are no longer able to distinguish a rotation of 180°.

After this first transformation, we have simplified the problem of estimating shift, rotation and scaling to a problem of estimating only rotation and scaling. Next, we apply a log-polar mapping. This is a mapping where the image is deformed in a particular way, resulting in the x-axis of the output being the logarithm of the distance to the center of the input image, and the y-axis being the angle around this center:

A rotation around the center of the image corresponds to a vertical shift of the log-polar mapped image. And a scaling will correspond to a horizontal shift of the log-polar mapped image. We thus have converted the rotation and scaling to a simple translation. This is the Mellin transform part of the Fourier-Mellin transform.

Cross-Correlation

Finding a shift between two images is fairly straight-forward using the cross-correlation. In short, we multiply the two images together and compute the sum of the result, yielding a single value. We do this for every possible shift of one image w.r.t. the other. This can be computed efficiently using the FFT. The shift for which the cross-correlation function is largest is the shift that makes the two images most similar. By fitting a parabola to the 3×3 neighborhood of the pixel with the largest value, we can determine its location with sub-pixel precision.

There are different ways to normalize the cross-correlation to increase the sharpness of the peak, and thus improve the localization accuracy. To learn more about these normalization methods, see the DIPlib documentation for dip::CrossCorrelationFT()

Once we found the shift in the log-polar mapped FT image, we can calculate the rotation and scaling for the one image to match the other. Except that we still need to disambiguate the 180° rotation!

What about the shift?

So now we can transform one of the two images to match the other with only a shift and a possible 180° rotation left. We can compute this partially-aligned image, then use the cross-correlation again to find this missing shift. We can apply this cross-correlation twice: once with the transformed image, and once with the transformed image rotated by 180°. One of these two will yield a higher cross-correlation peak, because it will match the other image better.

Example

An example will make all of the above more clear. I’ll be using DIPlib in Python.

Let’s say we have these two input images, img_a and img_b:

import diplib as dip
import numpy as np

img_a = dip.ImageRead("image A")
img_b = dip.ImageRead("image B")

img_b (right) is a crop of some original image. img_a (left) is a different crop obtained after rotating the original image clockwise by π/8 radian and scaling it by a factor of 0.7.

For simplicity, we’ll consider only the green channel, which has good contrast across the image:

gray_a = img_a(1)
gray_b = img_b(1)

We can apply the method to the color images, but we won’t gain anything. After finding the transformation needed to align the gray-scale images, we can apply that transformation to the color image.

We’ll start by computing the FTs of the two images, after applying a windowing function. We discard the phase, and apply a log mapping to improve local contrast:

ft_a = dip.FourierTransform(dip.ApplyWindow(gray_a, "GaussianTukey", 10))
ft_b = dip.FourierTransform(dip.ApplyWindow(gray_b, "GaussianTukey", 10))
lm_a = dip.Ln(dip.Abs(ft_a))
lm_b = dip.Ln(dip.Abs(ft_b))

The windowing function is important because without it, the FT will be corrupted by the edge effects. Image edges usually show up as a sharp horizontal and vertical line through the origin. These two lines will cause a peak in the cross-correlation at the origin, which will be identical for the two images, giving us a chance to erroneously find a shift of 0.

Note also that dip.FourierTransform() computes the full DFT with the origin in the middle of the output image. When using a different FFT implementation, you’ll need to apply a function such as np.fft.fftshift() to get the origin to be in the middle of the image.

Next, we apply the log-polar transformation to the two images:

lpt_a = dip.LogPolarTransform2D(lm_a, "linear")
lpt_b = dip.LogPolarTransform2D(lm_b, "linear")

The cross-correlation between these two images gives a sharp peak, whose location gives the shift:

find_shift_method = "NCC"  # or "PC" or "CC", depending on which normalization you want to use
shift = dip.FindShift(lpt_a, lpt_b, find_shift_method)
print(f"LPT shift = {shift}")

We get a shift of -32.67 along the x-axis and 224.05 along the y-axis. From that we can calculate the rotation and scaling, simply by inverting the computations applied in the log-polar transform:

maxr = np.min(gray_a.Sizes()) // 2
size = lpt_a.Size(0)
scale = maxr ** (shift[0] / (size - 1))
theta = -shift[1] * 2 * dip.pi / size  # 180 degree ambiguity!
print(f"scale = {scale}, theta = {theta}")

We get a scale of 0.7015 and a rotation of -2.749 radian. We expected a scale of 0.7000, and a rotation of 0.3927 (π/8) radian. -2.749 is π/8 - π, so we are off by exactly 180°. Remember that with the FT, we cannot distinguish between these two rotations. We happen to have found the wrong rotation, but fret not, we’ll fix this soon.

We can apply the scale and rotation to the image, which will leave us with a shifted (and rotated by 180°) version of the final result:

matrix = [scale * np.cos(theta), -scale * np.sin(theta), scale * np.sin(theta), scale * np.cos(theta), 0, 0]
gray_b_transformed = dip.AffineTransform(gray_b, matrix, "linear")

Using this transformed image, we can figure out what shift to apply using cross-correlation again:

cc_method = {
    "CC": "don't normalize",
    "NCC": "normalize",
    "PC": "phase",
}[find_shift_method]
cross = dip.CrossCorrelationFT(gray_a, gray_b_transformed, normalize=cc_method)
loc_1 = dip.SubpixelLocation(cross, dip.MaximumPixel(cross))
print(f"loc_1: coordinates = {loc_1.coordinates}, value = {loc_1.value}")

But because we have two possible rotations, we should compute the cross-correlation with gray_b_transformed and also with that image rotated by 180°. The height of the peak will tell us which of the two rotations is the correct one (a larger cross-correlation result means the images are more similar).

gray_b_transformed.Rotation90(2)
cross = dip.CrossCorrelationFT(gray_a, gray_b_transformed, normalize=cc_method)
loc_2 = dip.SubpixelLocation(cross, dip.MaximumPixel(cross))
print(f"loc_2: coordinates = {loc_2.coordinates}, value = {loc_2.value}")

loc_1 has a value of 0.0261, and loc_2 of 0.2920. It is clear that the rotation used for loc_2 is the better one. The peak for loc_2 is at (197.78, 344.04). The shift we’re looking for is the distance from the origin (the middle of the image) to this peak:

if loc_1.value >= loc_2.value:
    # We use theta as the rotation
    matrix[4] = loc_1.coordinates[0] - cross.Size(0) // 2
    matrix[5] = loc_1.coordinates[1] - cross.Size(1) // 2
else:
    # We use theta + 180 as the rotation
    matrix[4] = loc_2.coordinates[0] - cross.Size(0) // 2
    matrix[5] = loc_2.coordinates[1] - cross.Size(1) // 2
    matrix[0] = -matrix[0]
    matrix[1] = -matrix[1]
    matrix[2] = -matrix[2]
    matrix[3] = -matrix[3]

Using this transformation matrix, we can finally compute the transformed version of img_b that matches img_a:

final = dip.AffineTransform(img_b, matrix, "linear")

This whole process is implemented in DIPlib as dip::FourierMellinMatch2D().

You can find the complete code I used to generate all the images in this post right here.

Median filtering

2022-06-27T00:00:00-06:00

The median filter is a non-liner smoothing (blurring) filter where an output pixel is the median value of its neighborhood in the input image. It can be generalized to a percentile filter, where the median is the 50^th percentile.

Noisy “trui” image filtered with a percentile filter, from left to right: 100^th percentile (dilation), 95^th percentile, 75^th percentile, 50^th percentile (median), 25^th percentile, 5^th percentile, 0^th percentile (erosion).

In Mathematical Morphology one talks about a rank filter. Mathematical Morphology is based on the maximum and minimum values of sets (actually the supremum and infimum values, but the difference is just a technicality). For noisy images, it sometimes is advantageous to compute the maximum or minimum ignoring a few values, in fact computing a rank order statistic. So a rank 3 filter is like a maximum filter (a dilation), but ignoring the two largest values in the neighborhood. The equivalence between the rank filter and the percentile filter is obvious.

In this blog post I want to discuss implementations of the median filter (and, by extension, the percentile and rank filters). Because computing the median is a non-linear operation, we cannot use the Fourier transform to aid us. There is also no way to split up the median computation. We can compute the mean of a set by dividing up the set into smaller sets, compute the mean of each subset, then compute the mean of the means. We can do the same for the maximum or minimum computation. But for the median this is not possible. This means that reducing the algorithmic complexity of the median filter is not a trivial exercise.

Naive algorithm

The basic algorithm to compute a median filter would be to compute the median over the neighborhood independently for each output pixel. This is the algorithm implemented in scikit-image (which actually uses the implementation in SciPy’s ndimage), as well as the one in DIPlib up to version 3.3.0 (which is the current release, the next release of DIPlib will include a more efficient algorithm, discussed later).

Computing the median cannot be done in-place. One needs to sort the values, so the simplest approach is to copy them over into a temporary buffer, sort the buffer, and then find the median value. Sorting values takes . We can speed up the process by not sorting the array fully, just enough to get the median value in its place. For example, using the Quicksort algorithm for sorting, we can always skip one of the two sub-arrays. This is called partial sorting, and in C++ is implemented as std::nth_element. Partial sorting has a complexity of on average.

For integer values, sorting complexity is , but there is no reduction in complexity for partial sort in this case.

For a square 2D kernel of pixels, this implementation thus costs per pixel. This is actually a quite efficient algorithm for small kernels, but because of the poor scaling, gets quite expensive for larger ones.

Histogram-based algorithm

Because the median computation cannot be split up, as I mentioned earlier, the only way to reduce the algorithmic complexity of the filter is to somehow update the data in the temporary buffer (remove the pixels exiting the kernel as it is shifted over one pixel, and add the pixels entering), and do so in a manner that makes finding the median cheaper, in a manner that allows re-using the earlier work.

In the case of 8-bit images, sorting can be done efficiently by computing a histogram of the values to be sorted. Finding the median in the histogram involves summing up the bins of the histogram, the bin where this sum reaches is the median. Updating the histogram as we shift the kernel by one pixel costs operations, and finding the median is a fixed number of operations. Thus, for these images, the median filter has a complexity of , much better than the naive implementation.

Note that this algorithm works for kernels of any shape; is then the number of pixels that exit and enter the kernel when shifting it by one pixel. For a compact 2D kernel this is the height of the kernel, for a non-compact kernel this can be much larger. In 3D it is the height times the depth of the kernel (for example, for a cubic kernel, the complexity is for the naive algorithm, and for the histogram-based algorithm).

This algorithm is due to T.S. Huang and colleagues [1].

But even for 16-bit images, a histogram becomes unwieldy, because finding the median now is not additions on average, but . The constant cost becomes significant compared to the linear cost for reasonable values of .

It has been proposed to augment the 16-bit histogram with smaller histograms with wider bins. For example, each subsequent scale combining 16 bins into one. Updating such a multi-scale histogram is a bit more expensive, but finding the median becomes much cheaper. Still, this does not allow for an algorithm that can handle floating-point images efficiently.

Binary tree algorithm

The histogram here was used as a data structure that makes it easy to compute the median of a set of values, and allows this set to be updated (removing and adding values). A binary search tree can perform the same function but does not rely on values being small integers. A binary search tree is a binary tree where each node contains a value, and the nodes are sorted such that a node’s left child (and all of its descendants) has a smaller value than the node, and the node’s right child has a larger value. Inserting a new value into the tree takes , and so does removing a value. But computing the median still requires to access half the nodes in order, which takes .

An order statistic tree is a binary search tree where each node additionally records its size, which is given by the number of nodes in the subtree rooted at that node. That is, a node’s size is the sum of the sizes of the two children plus one. Armed with this additional information, finding the median (or the node at any given index) takes only . It is no longer necessary to traverse all nodes in order from the lowest value to the given index. The node’s size allows the algorithm to skip most nodes along that path.

So, again assuming a square kernel, we have values in the tree, and for each pixel we need to remove values, insert values, then find the median. This has a complexity of

The complexity analysis here assumes that the binary tree is balanced. Values in an image are locally correlated, meaning that the tree is likely to become unbalanced quickly unless we specifically rebalance the tree when needed. Binary trees typically store some additional value to enable this balancing act (e.g. red-black trees). One form of self-balancing tree is the weight-balanced tree, which keeps the weight of the two subtrees within some factor of each other. Each node stores the weight of the subtree rooted at that node. The weight is defined as one plus the size of the subtree, and we’re already storing the size to compute the order statistic. This means that we can keep the order statistic tree balanced without any additional information or bookkeeping. I followed Hirai and Yamamoto [2] to implement the balancing scheme, using their proposed (3,2) factors: the weight of the two subtrees are kept within a factor of 3 from each other, and the 2 is used when the balance is broken, to decide on whether to do a single or a double rotation at the node to restore balance.

Because pixel values are locally correlated, it is likely that there are repeated values in a neighborhood. A simple modification to the order statistic tree gives it a speed boost when this happens: repeated values are stored in the same node, which now additionally stores this count. This change means there are fewer nodes to traverse when inserting or removing values, or when indexing, and so the computational cost is reduced.

This implementation is in DIPlib, here. It will be available in the next release (whichever comes after the current 3.3.0 release), or you can build it yourself.

The binary search tree is a bit more expensive than the naive algorithm for small kernels. I ran a small experiment to find out how to decide which of the two algorithms to use for any particular call to the dip::MedianFilter() function. Using a rectangular kernel, I varied the width and height independently, from 3 to 39, and determined the running time on some sufficiently large image. This is the result (blue is the naive algorithm, yellow the new order statistic tree algorithm):

It can clearly be seen that the naive implementation has a running time that is linear in both the width () and the height () of the kernel (). In contrast, the binary tree implementation has a running time that is independent of the width, and linear in the height (). It is actually , but the range is too small to discern the difference.

It can also clearly be seen that the two implementations take the same time for equal to 11. This threshold is likely different on different computers, but I hard-coded this threshold into the dip::MedianFilter() function anyway as it should give some meaningful algorithm selection on any machine.

O(1) algorithm

Perreault and Hérbert [3] figured out how to compute a median filter in constant time. This algorithm is based on Huang’s histogram algorithm, but they store a histogram for each image column, updating these when moving the kernel by one pixel only requires removing one value and adding one value to one of these histograms, then subtracting one histogram and adding another histogram to the main histogram that covers all values under the kernel. OpenCV implements this algorithm, which only works for 8-bit images and square kernels.

Perreault and Hérbert’s paper discusses extension to octagonal kernels (as an approximation to round kernels), which require more intermediate memory (more histograms stored). They claim that for these kernels the algorithm is still , but the increased cost would make this algorithm impractical for all but the largest of kernels.

This algorithm is incredibly fast for any kernel size, as long as the kernel is square and the image is 8-bit. This lack of generality makes it uninteresting to include in DIPlib, which quickly switches to single-precision floating point pixels to avoid rounding errors, and which uses isotropic (round) kernels by default everywhere.

Timing comparisons

The graph below shows the computational time for a square median filter. The new DIPlib implementation is in blue, the constant-time OpenCV implementation is in red, and scikit-image’s implementation (which actually calls SciPy ndimage’s implementation) in green.

For each implementation, I used a single-precision floating-point image (continuous line), an 8-bit unsigned integer version of the same image (dashed line), and that same image divided by 25, meaning that it only has integer values in the range 0-10 (dotted line). This latter case should have many more repeated values within the kernel than the other cases.

One can see the kink in the blue lines for DIPlib between 11 and 13 pixels, where a different algorithm is chosen. To the left, the curve is quadratic (), to the right it is . Scikit-image’s lines are quadratic, the difference with DIPlib’s line is partially due to quality of implementation, but mostly because DIPlib uses all cores whereas scikit-image is single-threaded. For both implementations, reducing the number of values in the input image strongly reduces the computational cost (dotted lines). Quickselect is faster when many values are identical, and our binary tree implementation has strongly reduced number of nodes when each node represents many pixels with the same value.

The constant-time implementation in OpenCV is not actually constant time. It seems that a different algorithm is used for kernels up to 15 pixels, the constant-time algorithm kicks in at 19 pixels. It also looks like the selection of which algorithm to use was tuned on a machine with different characteristics to mine, meaning that a different algorithm was chosen for some sizes where the constant-time implementation would be more efficient. Because of this, the filter is significantly slower for a kernel of size 15 than for a kernel of size 19.

We can see the same issue in the DIPlib implementation for the dotted line, where the binary tree implementation is faster than what I measured earlier because of the reduced number of different values in the image. Picking the most efficient algorithm under all circumstances is an unattainable dream, I am willing to live with small imperfections like this.

Literature

T.S. Huang, G.J. Yang, G.Y. Tang, “A fast two-dimensional median filtering algorithm”, IEEE Transactions on Acoustics, Speech, and Signal Processing 27(1):13–18, 1979.
Y. Hirai, K. Yamamoto, “Balancing weight-balanced trees”, Journal of Functional Programming 21(3):287–307, 2011.
S. Perreault and P. Hérbert, “Median Filtering in Constant Time”, IEEE Transactions on Image Processing 16(9):2389–2394, 2007.

Morphological reconstruction, or don't trust old timings

2021-09-27T00:00:00-06:00

DIPlib 3.1.0 was released last week (see the change log). One of the changes in this release is a very significant speed improvement in the morphological reconstruction (dip::MorphologicalReconstruction). This is quite an interesting story, so I decided to write it down.

The morphological reconstruction algorithm can be described as an iterative dilation or erosion of an image f conditioned by an image g. That is, we iterate the following statement until the image f doesn’t change any more:

f = min(dilation(f, 3), g);

Of course this can be implemented much more efficiently. In 1993, Luc Vincent (on a roll publishing efficient algorithms for quite a few morphological operators, including the watershed) published a paper describing four possible implementations of the reconstruction operator [1]. The fourth method, the most efficient one, is a combination of two other methods: first it does a raster and an anti-raster scan (see box below for an explanation), then it uses a FIFO queue to complete the propagation. In 2004, Robinson and Whelan published a paper describing a much more efficient algorithm, based on a priority queue [2]. This table from the paper shows the results of experiments on 2D images (there’s a similar one for 3D):

Here, A, B, C and D are the algorithms described by Vincent, with D the most efficient one. E is the priority-queue algorithm by Robinson and Whelan. These experiments clearly show that the priority-queue algorithm is significantly more efficient. The reasoning also makes sense: the priority queue ensures that each pixel gets updated only once, whereas the FIFO queue potentially updates some pixels many times. And so this is the algorithm that I implemented in DIPlib many years ago.

Raster scan

A raster scan is a scan of the image in the order in which it is stored in memory. Typically, this is row by row. The scan starts at the top-left pixel, visits each pixel on that top row in order, then continues with the next row, etc. This is the scanning order for most acquisition devices, and therefore also the order in which images are typically stored in memory and on disk.

During a raster scan, one propagates values to the right and down. Information from the top-left of the image can thus make its way all the way to the right and bottom of the image. An anti-raster scan is a scan in the opposite order, starting at the bottom-right pixel, scanning left along the row, and then up to process each row. Here, information is propagated to the left and up.

The combination of a raster scan and an anti-raster scan is sufficient for some algorithms to propagate all the information in the image to where it needs to go. For example, one can compute a distance transform in 2D using just one raster scan and one anti-raster scan.

About two months ago, I was replacing an old, no longer supported mathematical morphology library (used in some code I maintain at work) with DIPlib. To do so, I compared each function in the old library with the equivalent DIPlib function to make sure the results were identical. At the same time I compared running times. Some functions were a bit faster in one library, some a bit faster in the other, and some, like basic dilations with larger structuring elements, were significantly faster in DIPlib. Of course, I took the opportunity to improve the implementation of some of these functions that were a little slower in DIPlib, where obvious gains could be made. But the morphological reconstruction function really surprised me. The old library’s implementation was ten times faster than DIPlib’s!

Examining the DIPlib implementation, I noticed that I had been lazy when making the original implementation. Marking border pixels in a flag image allows the algorithm to skip testing coordinates before accessing neighbors for most pixels in the image, speeding up processing by about 20%. Ensuring consistent strides so that the same offsets can be used in the f, g and intermediate flag images sped up processing by another 20%. And a few small things here and there rounded out improvements to a total of about 50%. But this meant that the implementation was still five times slower than that in the old library.

So I took a look at the code in the old library, and found that it implemented the algorithm proposed by Vincent (method D in the table above), which was supposed to be anywhere from two times to 14 times slower than the priority queue algorithm.

I was doing my timing comparison on quite a large image. Robinson and Whelan did their experiments on quite small images, a much larger fraction of which would have fit in their cache. And a priority-queue algorithm processes pixels in the image in a rather disorganized way. Putting these points together, it becomes obvious that I was seeing a cache miss for just about every pixel processed, whereas Robinson and Whelan were seeing a lot fewer cache misses, and my cache misses were likely more expensive too. I believe that this is why I was seeing such slow execution in the DIPlib implementation. In contrast, Vincent’s algorithm uses a raster scan and an anti-raster scan to do a large chunk of the propagation. These scans are very cache-friendly.

So I rewrote the DIPlib implementation to be a combination of the two algorithms: it first applies a raster scan and an anti-raster scan to do a large chunk of the work. The second scan, in anti-raster order, also initializes a priority queue, enqueueing all pixels from which it is possible to propagate right and down (since this scan is propagating left and up). Next, the algorithm processes the pixels in the priority queue, enqueuing neighbors from which further propagation is possible, until the queue is empty. Each pixel is thus updated no more than three times. This implementation was, in my tests, a bit faster than Vincent’s, which uses a FIFO queue and therefore each pixel can potentially be updated many times.

So, lessons learned:

One cannot make assumptions about which algorithm or implementation is faster without actually measuring times.
Timings from 15 years ago are likely no longer relevant.
Sometimes it’s good to take another look at older, superseded algorithms.
What is true for small data is not necessarily true for large data.

Literature

L. Vincent, “Morphological grayscale reconstruction in image analysis: applications and efficient algorithms”, IEEE Transactions on Image Processing 2(2):176–201, 1993
K. Robinson and P.F. Whelan, “Efficient morphological reconstruction: a downhill filter”, Pattern Recognition Letters 25:1759–1767, 2004

Filling an image with grid points

2019-01-22T23:09:00-07:00

Several algorithms require a set of uniformly distributed points across the image. For example, superpixel algorithms typically start with a regular grid. A rectangular grid of points is trivial to draw into an image. One simply generates the set of points of a grid that covers the image, rounds those points to integer coordinates, and sets those pixels. A rotated grid is a bit more challenging–one needs to compute bounds of the rotated grid such that the full image is covered. With other than rectangular grids (for example a hexagonal grid) the math gets a bit more complicated, but this can still be computed. But how to generalize such an algorithm to three dimensions? And to an arbitrary number of dimensions?

I played around with this task for a bit and came up with a very simple, general solution.

First, let’s review how points on a grid can be generated in general. A grid is defined by a set of basis vectors. These vectors do not need to be orthogonal. For example, for an hexagonal grid, the basis vectors are v1 = [1,0] and v2 = [cos(60°),sin(60°)] (using degrees, not radian, for simplicity of notation). Given these two vectors, a*v1 + b*v2, with a and b integers, yields the grid points.

This notion generalizes trivially to any number of dimensions. For example, in 3D the FCC (face-centered cubic) grid is given by vectors v1 = [1,1,0], v2 = [1,0,1] and v3 = [0,1,1]. There are always as many vectors as number of dimensions. These vectors form the columns of a matrix M, such that M*s is a grid point, with s a column vector of integer values. This s can be thought of as the grid index.

So the difficulty is to determine which integer-valued vectors p lead to grid points that intersect the image. For a rectangular grid that is aligned with the axes, it is also a rectangular domain of s that needs to be considered. When rotating a 2D grid, it is still quite simple to determine the larger rectangular domain that completely covers the image, then discard the points that do not intersect the image. The same can be said for a non-rectangular grid in 2D, it is always possible to find the rectangular domain of s that covers the whole image. But these computations are more complex in the 3D case, and quite difficult to generalize to an arbitrary number of dimensions. For higher-dimensional spaces, these domains of s become increasingly large, making the point discarding solution increasingly inefficient.

Instead, we can start with the image domain itself, and for each pixel find out if it is the pixel nearest a grid point. For this, we first need to find the inverse of the matrix M, let’s call it inv_M. For a given pixel p = [x,y,z,...] the expression s = round(inv_M*p) is an integer grid index. As we saw before, q = M*s is the coordinates of the grid point for grid index s. round(q) is the nearest pixel. If round(q)==p, then the pixel at p should be marked as a grid point.

Using DIPimage 3, we can write as follows (this code will not work in DIPimage 2.9 or earlier without some changes):

% Define image domain
sz = [256,140];
% Define grid
d = 10.2;
M = [[1;0],[cosd(60);sind(60)]] * d;
% Let's rotate the grid too
phi = pi/12;
M = [cos(phi),sin(phi);-sin(phi),cos(phi)] * M;
% The inverse of the grid matrix
inv_M = inv(M);
% Convert M and inv_M to dip_image objects so we can do the multiplication
M = dip_image(M(:),[2,2]);          % Create single-pixel tensor image
inv_M = dip_image(inv_M(:),[2,2]);  % ...yes, this syntax is a bit awkward still, hopefully this will be easier soon
% Do the computation
p = coordinates(sz);                % An image where each pixel's value is it's own coordinate vector
s = round(inv_M * p);
q = round(M * s);
grid = all(q == p,'tensor');

Note that in the code above, we determined the image domain sz as 2D, and defined the grid matrix M as 2D. Other than that, there is no mention of the dimensionality. Changing these two matrices we can instead generate a grid in a 3D image, or a 5D image.

In case you don’t want to bother installing DIPimage, below is a version of the code that uses only standard MATLAB functions. This version, however, is explicitly for 2D.

% Define image domain
sz = [256,140];
% Define grid
d = 10.2;
M = [[1;0],[cosd(60);sind(60)]] * d;
% Let's rotate the grid too
phi = pi/12;
M = [cos(phi),sin(phi);-sin(phi),cos(phi)] * M;
% The inverse of the grid matrix
inv_M = inv(M);
% Do the computation
[x,y] = meshgrid(1:sz(1),1:sz(2));
p = [x(:).'; y(:).'];
s = round(inv_M * p);
q = round(M * s);
grid = all(q == p, 1);
grid = reshape(grid, size(x));

Color quantization, minimizing variance, and k-d trees

2018-05-26T01:22:00-06:00

A recent question on Stack Overflow really stroke my curiosity. The question was about why MATLAB’s rgb2ind function is so much faster at finding a color table than k-means clustering. It turns out that the algorithm in rgb2ind, which they call “Minimum Variance Quantization”, is not documented anywhere. A different page in the MATLAB documentation does give away a little bit about this algorithm: it shows a partitioning of the RGB cube with what looks like a k-d tree.

So I ended up thinking quite a bit about how a k-d tree, minimizing variance, and color quantization could intersect. I ended up devising an algorithm that works quite well. It is implemented in DIPlib 3.0 as dip::MinimumVariancePartitioning (source code). I’d like to describe the algorithm here in a bit of detail.

Partitioning a color histogram

The RGB cube is a 3D space where each pixel can be represented as a point. Put all pixels in an image in there, and you get a point cloud. Clustering of these points leads to a partitioning of the RGB cube into cells (or regions, clusters, tiles, whatever you want to call them) that each represent a group of similar colors present in the image. For each cell one can decide on a single color (e.g. the mean) that will represent all colors within that cell. This leads to a short list of colors, a color palette. Next, for each pixel in the image, one finds the cell it falls into, and replaces it with the color chosen for that cell. This transforms the image such that only the colors in the color palette are used. The colors are now quantized, as there are many fewer colors than in the original image. The more cells the RGB cube is partitioned in, the more the transformed image will look like the original image.

This color quantization was a very important technique in the days of 256-color displays, and GIF images, as the quality of the image displayed depended very much on how well the set of 256 colors was chosen.

But note that it is not necessary to go through a clustering process using all pixels in the image. It is much more efficient (typically) to use a color histogram instead. The color histogram is simply a discretization of the RGB cube, where each element is a count of the number of pixels with that color. That is, a color histogram is a 3D array of, say, 32x32x32 or 64x64x64 values, containing pixel counts corresponding to each combination of RGB values. For a normal photograph (today often around 10 or 15 million pixels) this represents a reduction of the number of data points of 40x to 400x.

K-means clustering is the obvious choice for clustering, except that it is quite expensive. Current common methods are the median cut algorithm and a clustering using octrees. The median cut recursively splits the RGB cube along its longest axis, such that each half represents about half the pixels. This leads to a partitioning where each cell represents the same number of pixels. A k-d tree is the data structure used for this algorithm (see below for details). The other popular method builds an octree over the histogram, and merges groups of 8 cells recursively until the right number of cells remain. The least occupied cells are the first to be merged.

K-d trees

K-d trees are a spatial indexing data structure. It is a binary tree (each node has either zero or two children), where the leaf nodes (the ones with zero children) represent the cells of the partition. Each branch node has a threshold value, which operates along one axis. Thus, each node splits the space in two, and all cells are rectangular (or boxes in 3D). Finding the cell that a point in space belongs to is O(log n), for n cells: starting at the root node, follow the left or right child depending on the threshold value stored in the node, until you reach a leaf node.

The k-d tree thus is very flexible in how the space is partitioned, but it cannot generate splits at arbitrary angles, like you can get with k-means clustering. K-means clustering produces a Voronoi tessellation of the space, induced by the k centroids of the cells. Each cell is composed of the points that are closer to the cell’s centroid than to any other of the centroids.

In contrast, in the octree data structure, each node has either zero or eight children, where the children are all the same size and shape. That is, each cell is divided into eighths by splitting it in half along each dimension. This means that the nodes do not need to store a threshold value and direction, but it also means that meaningful clusters will be split, and thus the octree typically requires more cells than an equivalent k-d tree.

Here are example partitionings of a 2D space into 25 cells using a k-d tree (left), a Voronoi graph (middle), and an octree (actually a quadtree, the 2D equivalent, right):

Minimizing variance

Otsu’s algorithm for finding a global threshold finds a threshold that minimizes the (weighted) sum of the variances of the two classes. This sum is always smaller than the variance of all the data combined. Imagine data with a bimodal distribution, half the data points have a Gaussian distribution with mean and variance , the other half a similar distribution with mean and variance . As long as , then the variance of this data set . Thus, splitting our RGB cube in a meaningful way will reduce the sum of variances along all three axes.

Since the k-d tree always splits along one axis, I figured it makes sense to compute the marginal distributions for a cell (i.e. the projections of the histogram onto each axis), determine the optimal Otsu threshold for each of those projections, and determine which of these three cuts reduces the variance the most. The figure below gives a graphical representation of these marginal distributions and Otsu thresholds in 2D (for the RGB cube you’d add a 3^rd dimension of course). The horizontal cut would reduce the total variance more than the vertical cut, so the horizontal cut is chosen.

This process will allows us to divide the space into two cells. Next, we can repeat this process for each of the resulting two cells, to find the next best cut. But now we also need to choose which of the cells to cut. The more cuts we make, the more cells we will have to choose from. To make this problem tractable, a priority queue is the best approach. Each of the cells is put into the priority queue, with a priority given by how much the sum of variances is reduced by splitting that cell. Now the cell at the top of the queue is always the best one to cut. After cutting it, the two resulting cells are put back into the queue.

Here is an example partitioning of a 2D space with 5 Gaussian blobs:

As shown earlier, the first cut is a horizontal cut that separates the one blob that is a little bit more isolated than the others. The next cut is a vertical cut of the top cell, again separating one blob from the other three. Next we get another horizontal cut, and finally a vertical cut. Note that these cuts do not need to separate one blob from the group, it is a coincidence that it happened that way in this example.

With k-means clustering, using random initialization, you get a different result every time. Sometimes the result is really good, but with data like this you have to get lucky. Many initializations yield poor results, with two or three blobs in one partition, and one blob split into two:

Putting it all together

As an example, I’m using the well-known “mandril” image:

Here are the results of using this minimum variance partitioning algorithm on the color histogram with 100³ bins, for 5 colors (left) and 25 colors (right):

The computation of the 25-color quantized image took 0.05 s on my computer. I find it striking that the image looks so good with only 25 unique colors. For fewer colors, dithering could have improved the results.

For comparison, this is what the results are with k-means clustering. Running the algorithm multiple times gives slightly different results. The computation of the 25-color quantized image took nearly 5 s, 100 times longer than the minimum variance algorithm.

Which results do you prefer? Do you think that MATLAB’s rgb2ind implements a similar algorithm to this?

Union-Find

2018-03-21T01:02:00-06:00

The Union-Find data structure is well known in the image processing community because of its use in efficient connected component labeling algorithms. It is also an important part of Kruskal’s algorithm for the minimum spanning tree. It is used to keep track of equivalences: are these two objects equivalent/connected/joined? You can think of it as a forest of trees. The nodes in the trees are the objects. If two nodes are in the same tree, they are equivalent. It is called Union-Find because it is optimized for those two operations, Union (joining two trees) and Find (determining if two objects are in the same tree). Both operations are performed in (essentially) constant time (actually it is O(α(n)), where α is the inverse Ackermann function, which grows extremely slowly and is always less than 5 for any number you can write down).

Here I’ll describe the data structure and show how its use can significantly speed up certain types of operations.

Implementation

The Union-Find data structure is basically an array A, where an object with ID (label, index) i is located at A[i]. The value at that location is the ID of the parent node. Thus, the tree structure only contains pointers from the children to the parent. A node that points at itself (A[i]==i) is the root of a tree. In C++:

class UnionFind {
   private:
      std::vector<int> A;
};

Three operations are required on this tree:

Create: add a new node.
Find: find the root of a tree.
Union: join two trees.

The Create operation simply extends the array A by one element and sets its value to its index. This creates a new tree, the new node is its own parent:

int UnionFind::Create() {
   int i = A.size();
   A.push_back(i);
   return i;
}

The Find operation needs to follow parent links until it reaches a node that points to itself. This is the root. The trick that makes this operation constant time is that, every time we look for a root, we set all nodes along the path to point directly to the root. This makes each tree completely flat. The next time the same node is queried, its result is obtained directly:

int Find(int i) const {
   if (A[i] == index) {
      return i;
   } else {
      int root = Find(A[i]);
      A[i] = root;
      return root;
   }
}

Finally, the Union operation simply makes one root point to the other. It uses the Find operation on its two operands, so that any two nodes can be used to merge their trees:

int Union(int i1, int i2) {
   i1 = Find(i1);
   i2 = Find(i2);
   if (i1 == i2) {
      return i1;
   } else {
      int root = std::min(i1, i2);
      int leaf = std::max(i1, i2);
      A[leaf] = root;
      return root;
   }
}

Here, I picked the smaller of the indices to be the root. I have seen implementations that pick one node at random, but this is computationally more expensive. Other implementations keep track of a “rank” for each tree, and merge the tree with the lower rank into the one with the higher rank. This keeps the trees flatter. However, because of the flattening in the Find operations, trees are pretty flat anyway. The most efficient approach would be to simply take i1 as root.

Note that it is possible to extend the above and add some additional information to each root, such as the size of the connected component for connected component labeling. The Union then must add the sizes for the two roots, and assign that to the new root.

Watershed region merging

The first time I implemented a Union-Find–based algorithm, I did not know what Union-Find was (I re-invented the wheel) . And consequently I missed the important detail of flattening the trees (my wheel was square). This was the “fast” watershed algorithm in DIPlib and DIPimage. It is an algorithm that takes some shortcuts to speed up the watershed computation. It sorts all pixels, then processes them in this sorted order. Plateaus (regions of constant value in the image) are not handled correctly by this algorithm, but a built-in merging procedure solves that issue. Basically, when two different regions first meet, a decision is made to keep them separate and draw a watershed line, or to merge them. The Union-Find structure is used here to keep track of these merged regions.

Let’s time this watershed in DIPimage 2:

img = repmat(readim,[4,3]);
t1 = timeit(@()watershed(img,1,0,0))  % merge only within plateaus
t2 = timeit(@()watershed(img,1,20,0)) % merge more aggressively

This leads to t1 = 0.4034 and t2 = 1.5193. As you can see, more merging means deeper trees, which means it takes more time to find the root. In DIPlib 3 I implemented a correct Union-Find data structure, and plugged it into the same watershed algorithm. The same test now gives t1 = 0.0819 and t2 = 0.0821. That is, with little merging the algorithm is 5x faster, and no matter how much we merge, it takes the same amount of time. (Do note that some of that 5x speedup is caused by the way that DIPimage 2 handled input argument parsing, I did not eliminate that from these tests.)

Connected component labeling

DIPlib 2 used a fairly efficient connected component labeling algorithm that works for images of any dimensionality, and can be adjusted to use any connectivity. It walks over the image until it finds a foreground pixel that hasn’t been processed yet, assigns a new label to it, then applies a flood filling algorithm to visit all pixels in the connected component and assign the same label to them. It then continues where it left off to search for the next unprocessed pixel. In DIPlib 3 I copied the fastest algorithm from YACCLAB, which is specific to 2D images and 8-connected neighborhood, and wrote a similar algorithm that works for images of any dimensionality and for any connectivity. This algorithm makes two passes over the image. First it labels each pixel it finds, and uses the Union-Find data structure to keep track of which labels are connected (and thus equivalent). In a second pass it paints each output pixel with the index of the corresponding root node. This second step also allows the output labels to be consecutive.

Let’s time these two different labeling algorithms:

img = repmat(readim,[4,3]) > 100; % a large 2D binary image
t1 = timeit(@()label(img))    % 8-connected, DIPlib 3 uses algorithm from YACCLAB
t2 = timeit(@()label(img,1))  % 4-connected, DIPlib 3 uses my own algorithm
img = repmat(readim('chromo3d'),[5,4,10]) > 100; % a large 3D binary image
t3 = timeit(@()label(img))    % DIPlib 3 uses my own algorithm

Under DIPimage 2, this code leads to t1 = 0.0197, t2 = 0.0187 and t3 = 1.1520. Under DIPimage 3 I see t1 = 0.0045, t2 = 0.0051 and t3 = 0.5352. So the difference between the fastest implementation in YACCLAB and my own, generic algorithm is fairly small, but there is a difference of a factor of about 2x with the non–Union-Find algorithm (plus again some overhead for parameter parsing in the DIPimage 2 toolbox). Do note that the old algorithm’s running time depends on the number and size of objects, whereas the Union-Find algorithm is fairly consistent in timing.

Detecting local maxima

In DIPlib 2 I had implemented a local maxima algorithm that used a similar flood-fill method to handle plateaus that are local maxima. I recently implemented a new algorithm that uses Union-Find for these. The new algorithm again walks over the image once, assigns a label to pixels that seem like they are local maxima or part of a local maximum plateau. Other pixels belonging to such a plateau are assigned the same label. At times a pixel is processed that “ties” two differently labeled regions together, these two regions are merged. When a pixel on such a plateau makes it clear that the plateau is not a local maximum (it has a higher-valued neighbor), the region is merged with the “background” (zero) label, identifying it as not a local maximum. Like in the connected component labeling algorithm, a second pass paints all pixels with a consistent label, and resets all pixels that are not local maxima. This algorithm is significantly more efficient for images with many plateaus. For example:

img = repmat(readim,[4,3]);
t1 = timeit(@()maxima(img))
img = round(gaussf(img,2)/16)*16; % image with many plateaus
t2 = timeit(@()maxima(img))

With DIPimage 2 I get t1 = 0.0397 and t2 = 0.0657. The second image needs nearly twice as much time because of the many plateaus. With DIPimage 3 this is t1 = 0.0185 and t2 = 0.0198. That is, the two images run in nearly the same amount of time. The two-fold difference for the first image is in part due to more efficient pixel access. While porting (and rewriting!) algorithms for DIPlib 3 I paid a lot more attention to walking over the image in memory storage order, and avoiding conditional statements within tight loops.

Computing Feret diameters from the convex hull

2012-02-13T17:45:00-07:00

Some time ago I wrote about how to compute the Feret diameters of a 2D object based on the chain code of its boundary. The diameters we computed were the longest and shortest projections of the object. The shortest projection, or smallest Feret diameter, is equivalent to the size measured when physically passing objects through sieves (i.e. sieve analysis, as is often done, e.g., with rocks). The longest projection, or largest Feret diameter, is useful as an estimate of the length of elongated objects.

The algorithm I described then simply rotated the object in two-degree intervals, and computed the projection length at each orientation. The problem with this algorithm is that the width estimated for very elongated objects is not very accurate: the orientation that produces the shortest projection could be up to 1 degree away from the optimal orientation, meaning that the estimated width is length⋅sin(π/180) too large. This doesn’t sound like much, but if the aspect ratio is 100, meaning the length is 100 times the width, we can overestimate the width by up to 175%!

A more accurate algorithm needs to check every possible angle. This sounds infeasible, but can actually be done if one considers the object under examination as a polygon with a finite set of vertices. The smallest diameter is the narrowest calliper measure possible for the object. Imagine the calliper arms tightening around the object from either side. At least one calliper arm will touch the object at two vertices. If it doesn’t, it is possible to rotate the object to reduce the distance between the calliper arms. Knowing this, we can also realize that any of the object’s diameters is equivalent to the corresponding diameter of the object’s convex hull. Any of the vertices of the object that are not part of the convex hull will never be touched by the calliper arms, and are therefore irrelevant for the computation of the Feret diameters. Thus, we can simplify our polygon by computing its convex hull, as we did in this earlier post.

Once we have converted our object boundary into a convex polygon, we can use the so-called rotating callipers algorithm to compute the Feret diameters. Such an algorithm enumerates the antipodal pairs of the convex polygon; that is, it enumerates pairs of points that can be touched at the same time by the two calliper arms. The minimum diameter will always be at an angle such that one of the calliper arms touches a side of the polygon, the other arm might touch a point or a side.

Here I’ll assume we just ran all the code in the post on computing the convex hull, and thus we have a variable D containing the vertices of the convex hull, derived from the polygon coords. Let’s start with defining three simple helper functions we’ll be using:

% Signed area of triangle
area = @(pt1,pt2,pt3) ( (pt2(1)-pt1(1))*(pt3(2)-pt1(2)) - ...
                        (pt2(2)-pt1(2))*(pt3(1)-pt1(1)) ) / 2;
% Height of triangle, pt3 is the top vertex
height = @(pt1,pt2,pt3) ( (pt2(1)-pt1(1))*(pt3(2)-pt1(2)) - ...
                          (pt2(2)-pt1(2))*(pt3(1)-pt1(1)) ) / ...
                                               norm(pt2-pt1);
% Next point along the polygon
next = @(p,N) mod(p,N)+1;

Next comes the rotating calliper algorithm itself. We initialise minD, the minimum diameter, to infinity, and maxD, the maximum diameter, to zero. minV and minV are the vectors that represent the corresponding directions. p and q are a pair of points on opposite sides of the polygon, and we move both around the shape such that they are always antipodal. For a given p, we find all antipodal points, and store them in the list listq. We then check to see if the distance between p and any of its antipodal points is smaller than the current minD or larger than the current maxD. Note that, to check for smallest distance, we cannot use the direct distance between the two points, but need to look at the height of the triangle formed by p, p+1 and q. p+1 is the point next to p, and the line joining the two is the one that the calliper arm is touching. The code below is not very transparent, it is a translation and adaptation of the code on this page (which has several errors). I’ve also added a graphical representation of the calliper:

figure('position',[1000,500,450,450]);
plot([coords(:,1);coords(1,1)],[coords(:,2);coords(1,2)],'b-')
hold on, axis equal, axis off, axis manual
plot([D(:,1);D(1,1)],[D(:,2);D(1,2)],'r-o')
wl = plot([0,0],[0,0],'k-*','markersize',12);
minD = Inf; minV = [];
maxD = 0;   maxV = [];
N = size(D,1);
p0 = N;
p = 1;
q = 2;
while area(D(p,:),D(next(p,N),:),D(next(q,N),:)) > ...
      area(D(p,:),D(next(p,N),:),D(q,:))
   q = next(q,N);
end
q0 = q;
while p~=p0
   p = next(p,N);
   listq = q;                       % p,q is an antipodal pair
   while area(D(p,:),D(next(p,N),:),D(next(q,N),:)) > ...
         area(D(p,:),D(next(p,N),:),D(q,:))
      q = next(q,N);
      listq = [listq,q];            % p,q is an antipodal pair
   end
   if area(D(p,:),D(next(p,N),:),D(next(q,N),:)) == ...
      area(D(p,:),D(next(p,N),:),D(q,:))
      listq = [listq,next(q,N)];    % p,q+1 is an antipodal pair
   end
   for ii=1:length(listq)
      q = mod(listq(ii)-1,N)+1;
      v = D(p,:)-D(q,:);
      d = norm(v);
      if d>maxD
         maxD = d;
         maxV = v;
      end
   end
   pt3 = D(p,:);
   for ii=1:length(listq)-1
      pt1 = D(listq(ii),:);
      pt2 = D(listq(ii+1),:);
      h = height(pt1,pt2,pt3);
      if h<minD
         minD = h;
         minV = pt2-pt1;
         pt4 = pt3 + minV*100;
         pt5 = pt3 - minV*100;
         set(wl,'xdata',...
                [pt1(1),pt2(1),NaN,pt3(1),NaN,pt4(1),pt5(1)],...
                'ydata',...
                [pt1(2),pt2(2),NaN,pt3(2),NaN,pt4(2),pt5(2)]);
         pause(0.5);
      end
   end
end

Now minD and maxD contain the minimum and maximum diameters found, and minV and minV the corresponding vectors. It is simple to derive the orientation of the two diameters. Computing the diameter perpendicular to the minimum diameter involves projecting the coordinates of the vertices onto a normalized minV, then looking for the maximum distance:

maxA = atan2(maxV(2),maxV(1));
minA = atan2(minV(2),minV(1))+pi/2;
minV = minV/norm(minV);
w = D*minV';
minP = max(w)-min(w);
fprintf('Object length: %.2f, at %.2f degreesn',maxD,maxA*180/pi);
fprintf('Minimum bounding box: %.2f by %.2f, at %.2f degreesn',...
        minD,minP,minA*180/pi);

Object length: 37.20, at 53.75 degrees
Minimum bounding box: 21.02 by 36.67, at -26.57 degrees

Note that these distances are between the centres of boundary pixels, and the object itself is typically one pixel wider/longer. The same was true when we computed the Feret diameters from the chain code directly.

The new 'Feret' measurement in DIPimage works essentially this way, as described in the changelist.

The convex hull of a 2D object

2011-09-18T16:32:00-06:00

Last year I wrote about computing the the boundary length and various other measures, given an object’s chain code. The chain code is a simple way of encoding the polygon that represents a 2D object. It is very simple to compute the object’s convex hull given this polygon. Why would I want to do that? Well, the convex hull gives several interesting object properties, such as the convexity (object area divided by convex hull area). Certain other properties, such as the Feret diameters, are identical for an object and its convex hull, and the convex hull thus gives an efficient algorithm to compute these properties.

Avraham Melkman (1987) has published a very efficient algorithm to compute the convex hull given a polygon (find the PDF through Google Scholar). It has been said that this might very well be the most efficient algorithm possible. It visits each polygon vertex only once, and keeps only those vertices that form the convex hull. To accomplish this, it uses a deque (also known as double-ended queue) to represent the convex hull. The deque has 4 basic operations: push and pop add and remove elements from the end of the queue, and insert and remove add and remove elements from the front of the queue. By using such a data structure, Melkman was able to add and remove vertices from his convex polygon at either side. The points in the deque always constitute a convex polygon, and is the convex hull of the input points seen up to that point.

The Melkman algorithm also depends on a function, which I’ve called position here, which returns 1 if pt3 is to the right of the directed line formed by pt1 and pt2, -1 if it is to the left, or 0 if all three points are collinear (form a straight line). This function simply returns the sign of the signed area of the parallelogram given by the vectors pt2-pt1 and pt3-pt1. It we embed these vectors in 3D (adding the value 0 as the z component), and compute the cross product, the norm of this cross product gives the area of the parallelogram, whereas its direction is given by the ordering of the three points on the plane (as per the right-hand rule). The cross product will, however, always have x and y components equal to 0, so we only need to compute the z component. Let’s define an anonymous function position(pt1,pt2,pt3) for this:

position = @(pt1,pt2,pt3) ...
   sign( (pt2(1)-pt1(1))*(pt3(2)-pt1(2)) - ...
         (pt2(2)-pt1(2))*(pt3(1)-pt1(1)) );

Before we can apply the algorithm, we’ll need some data. I like the image “cermet”. Let’s pick object #31, and get its chain code:

img = label(readim('cermet')<120);
cc = dip_imagechaincode(img,2,31);  % for DIPimage 2
% cc = traceobjects(img,31,2,'chain code')  % for DIPimage 3, use cc{1} instead of cc.chain below

The code below, however, takes a polygon represented by a set of points, not a chain code. We can easily convert the chain code cc into a list of coordinates coords (this is the same code as in the post on chain codes):

directions = [ 1, 0
               1,-1
               0,-1
              -1,-1
              -1, 0
              -1, 1
               0, 1
               1, 1];
coords = cumsum(directions(cc.chain+1,:));
N = size(coords,1);
% Draw input polygon:
figure('position',[1000,500,450,450]);
plot([coords(:,1);coords(1,1)],[coords(:,2);coords(1,2)],'b-')
hold on, axis equal, axis off

Melkman’s algorithm starts by taking the first three points in the list and forming them into a start of the convex hull. However, the algorithm assumes these points are not collinear. So let’s examine the first 3 points in the set, and delete the second one if all 3 are collinear:

while N>=3 && position(coords(1,:),coords(2,:),coords(3,:)) == 0
   coords(2,:) = [];
   N = size(coords,1);
end

At this point, we’d of course check to make sure that we have more than 3 points left. If not, we’re done! There is also a special case that could happen if our object is a single line. In this case, all points in the object are collinear, and the code above leaves only 2 points. We’d need to detect this case, and make sure that we keep the two extreme points, rather than the first and last points in the list. Otherwise, length measurements on our convex hull will be wrong.

We start by creating a deque data structure D (in MATLAB we can just use a simple array), and put the first three points in it. We first check the order of these points, and put them in the deque so that the polygon is right-turning. Furthermore, we always want the first and last point in our deque to be the same point:

if position(coords(1,:),coords(2,:),coords(3,:)) > 0
   D = coords([3,1,2,3],:);
else
   D = coords([3,2,1,3],:);
end
% Add convex hull to figure:
h = plot(D(:,1),D(:,2),'ro-');

Next, the algorithm takes one new point v at the time, and checks where it lies with respect to the first two points and the last two points in the deque. This tells us if the point is already inside the convex hull or not. If it is inside, we ignore the point and take the next one. If it is not inside, we’ll be wanting to add this point at the beginning and at the end of the deque (push and insert). But before we add this new point, we need to remove (pop and remove) previous points that will potentially end up inside the convex hull. We start with the end of the deque, and see if the last point is inside the line formed by the second to last point and the new one. If so, we pop that point. This is repeated until we cannot pop the last point any more, and then push the new point. We repeat the same procedure with the front of the deque, removing points and then inserting the new one. Now the algorithm loops back to process the next point. This is the code:

ii = 3;
while 1
   ii = ii+1; if ii>N, break, end
   v = coords(ii,:);
   while position(v,D(1,:),D(2,:)) >= 0 && ...
         position(D(end-1,:),D(end,:),v) >= 0
      ii = ii+1; if ii>N, break, end
      v = coords(ii,:);
   end
   if ii>N,break,end
   while position(D(end-1,:),D(end,:),v) <= 0
      D(end,:) = []; % pop
   end
   D = [D;v];        % push v
   while position(v,D(1,:),D(2,:)) <= 0
      D(1,:) = [];   % remove
   end
   D = [v;D];        % insert v
   % Update convex hull in figure:
   set(h,'xdata',D(:,1),'ydata',D(:,2));
   pause(0.2)
end
D(end,:) = []; % start and end points are the same, remove one.

When running the code, you will see the convex hull being updated as new points are examined by the algorithm (note that the slowness comes from the pause statement in the code!). At the end of the algorithm you’ll see this on your screen:

In a future post we’ll use this convex hull to compute Feret diameters.

Bresenham's line drawing algorithm in any number of dimensions

2011-07-22T10:17:00-06:00

J.E. Bresenham published an algorithm in 1965, which is used to draw 2D digital lines. Many people have extended the algorithm to work in 3D, and I’ve even found a website with code for 4D, 5D and 6D versions of the algorithm. However, all the code I come across is way more complex that it needs to be, giving the impression that this is a complicated algorithm. Nothing is further from the truth! Drawing digital lines is the most simple and straight-forward task. I’m going to deviate from the classical thinking that an efficient algorithm should use only integer values. This will make our task trivial.

The first thing we need to realize is that, when walking from the start pixel to the end pixel, there is always one dimension along which we take integer steps. Along the other dimensions sometimes we’ll have to step, ans sometimes we won’t. This is, to me, the key finding in the Bresenham algorithm. This tells us how many pixels will compose the line. If we use more pixels, the line will be thickened at some points, or we’ll use some pixels more than once. If we use fewer pixels, the line will not appear continuous. Based on this 1-pixel step size in one dimension, we can easily calculate the sub-pixel step sizes we have to take in the other dimensions. We will then take these steps, computing floating-point coordinates, until we reach the end point. Each of the points we compute we’ll round to integer pixel coordinates for output. In the original algorithm, instead of computing a floating-point coordinate, an integer coordinate is computed, and an additional integer value accumulates the fractional error. These two approaches are equivalent, except that, when using floating-point values, everything becomes easier to understand.

Let’s write some MATLAB code. We’ll start with two random points to draw a line between (I’m using 3D points here, but you can use any number of dimensions):

p1 = round(rand(1,3)*20);
p2 = round(rand(1,3)*20);

Next, we take a floating-point vector variable, p, which will contain the sub-pixel coordinates of the line. We also compute the step size, s, which is the total distance travelled in each dimension divided by the longest distance. Thus, the vector s contains one element with value 1 (or -1), and other values that are smaller.

p = p1;
d = p2-p1;
N = max(abs(d));
s = d/N;

We need to take N steps of size s, then round each of the floating-point positions found:

disp(p1)
for ii=1:N
   p = p+s;
   disp(round(p))
end

    14     6    19
    13     6    18
    12     6    17
    11     7    16
    10     7    16
     9     7    15
     8     7    14
     7     8    13
     6     8    12
     5     8    11
     4     8    11
     3     9    10
     2     9     9
     1     9     8

There it is! Bresenham’s algorithm, for any number of dimensions, in 9 lines of code.

Panoramic photograph stitching — again

2011-04-03T12:02:00-06:00

In an earlier post, I described and implemented a method, that was published recently, to stitch together photographs from a panoramic set. In a comment this morning, Panda asked about the parameters that direct the region merging in the watershed that I used. This set me to think about how much region merging the watershed should do. The only limitation that I can think of, is that we need two regions: one touching the left image and one toughing the right. We can easily do this with a seeded watershed: we create two seeds, one at each end of the region where the stitch should be, and run a seeded watershed. This watershed will not create new regions. You should see it as a region growing algorithm, more than a watershed. However, the regions are grown according to the watershed algorithm: low grey values first. That insures that, when the two regions meet, it happens at a line with high grey values (a “ridge” in the grey-value landscape). The graph cut algorithm can now be left out: the region growing algorithm does everything.

Let’s start with a quick recap. We have two images in variables a and b:

I found the coordinates of a common point in the two images, ca and cb. Then we extend (and crop) the two images so that they have a common coordinate system:

This is where the two methods start to differ. We again look at the difference between the two images in the stitching region, taking the maximum of the differences in the three colour channels:

d = max(abs(a-b));
d(0:sz(1)-1+s(1),:) = 0;
d(sz(1):imsize(d,1)-1,:) = 0;

The only difference with the d from the earlier blog post is that we do not look at the inverse of the difference, because the region growing algorithm in DIPimage that I will use can grow into high grey values first (i.e. the opposite of what the watershed would do). The next task is to create a seed image. We set the pixels unique to the first image ( image a) to 1, and the pixels unique to the second image to 2. The common region, where the stitch will be, we leave 0, so that the two seeds can grow into this area:

c = newim(d,'sint8');
c(0:sz(1)+s(1),:) = 1;
c(sz(1)-1:imsize(d,1)-1,:) = 2;

Now we just run the region growing algorithm:

c = dip_growregions(c,d,[],1,0,'high_first');  % For DIPimage 2
% c = waterseed(c,d,1,-1,0,{'high_first','labels','no gaps'});  % For DIPimage 3
w = c==2;

Note that we used the 'high_first' option. Using 'low_first' would yield the more traditional seeded watershed. As before, the final composition is trivial using this mask:

out = a;
out(w) = b(w);

As can be seen below, the two methods choose a different stitching, but both are quite successful. In this example, the region growing method seems better than the graph cut method, but we’d need to do some more extensive testing to know whether this is always the case or not. These two figures show where the images were stitched with each of the two methods, the graph cut method first, the region growing method second:

Finally, a comparison like in the earlier blog post, comparing the blurring, the graph cut, and the region growing methods:

The new method does not duplicate that one rock in the middle of the zoomed area. This might be coincidence, of course. Don’t assume this method is better just because of this one example!

Feel free to download the script I used to generate the images on this page. [Note: This script was written for DIPimage 2, and will not work unmodified with DIPimage 3. See the comments in the code in this post and the previous one for how to change the script.]