Cris’ Image Analysis Blog - criticism

The curse of the big table

2015-06-03T22:51:00-06:00

As an Area Editor for Pattern Recognition Letters, I’m frequently confronted with papers containing big tables of results. It is often the deblurring and denoising papers that (obviously using PSNR as a quality metric!) display lots of large tables comparing the proposed method with the state of the art on a set of images. I’m seriously tired of this. Now I’ve set my foot down, and asked an author to remove the table and provide a plot instead. In this post I will show what is wrong with the tables and propose a good alternative.

I’m just taking a random denoising/deblurring paper as example. It has four tables like this one:

The main problem with this table is that it doesn’t allow comparison of the various methods without looking at each column individually, study the numbers in it, and determine the magnitude of the difference between the largest value and all other values. Very helpfully, the largest value has been bolded. But it is still not a trivial task to determine how significant the advantage of one method over the other is. To further complicate things, PSNR and FSIM values have been mixed in the same column (presumably to save space), meaning that you need to compare odd rows and even rows separately.

In short, the raw dumping of numerical results helps no-one, except maybe the authors, who are able to hide the insignificance of the improvement of their proposed method. The purpose of a scientific paper is to communicate research results, not to let the reader repeat all your calculations. Raw data has no place in a scientific paper (of course there are always exceptions), the author should strive towards explaining his/her findings as simply and compactly as possible. In the same way that long, flowery prose is to be avoided, so are long, flowery displays of data.

For example, how many readers are interested in seeing the value of the PSNR obtained when applying the new algorithm to the image ‘Boats’? Very, very few is my guess. It might be nice to provide the raw results as supplementary data, so that other people can use those values in their own comparisons (but better, of course, is to provide the code so others can replicate the experiments). The majority of the readers will want to compare the results of the new algorithm to the previous state-of-the-art. Why not help them do this?

I would propose to display these results using a box plot:

In this plot we can directly compare the various methods against each other. I have separated the two metrics into separate plots, and the whole display still occupies less space than the one table. If strapped for space, it would be possible to remove two of the x-axes and two of the y-axes, and bring the four boxes close together.

One problem with this plot is that the distribution displayed by the boxes (especially in the case of PSNR) is driven by the images themselves, more than by the method. Thus, overlap between these boxes has no meaning other than saying that PSNR is a poor metric for evaluating this type of method. An alternative solution is to plot the improvement of the method over each of the state-of-the-art methods:

I have simply plotted the difference between the results for the new method and each of the other methods. PSNR is a logarithmic measure, so this makes sense, but for the other measure it might make more sense to plot ratios. In any case, it is now clear that the method improves in most cases, but not in all, and it is easy to see which cases these are.

Edit

I did not discuss here the other issues with this type of analysis: PSNR values depend very heavily on image content and should not be used for comparison across images. Are 10 images representative? Why not use all images in, for example, the Berkeley Segmentation Data Set, a collection of 500 natural images? Why are there not many different levels of blur compared? With how much blur does the method finally break down? Etc. Etc.

Edit 2

If you are curious about how I made these plots, you can find the script here. The boxplot command is one I wrote myself, find it on the MATLAB File Exchange.

No, that's not a Gaussian filter

2015-02-06T13:08:00-07:00

I recently got a question from a reader regarding Gaussian filtering, in which he says:

I have seen some codes use 3x3 Gaussian kernel like

h1 = [1, 2, 1]/4

to do the separate filtering.

The paper by Burt and Adelson (The Laplacian Pyramid as a Compact Image Code, IEEE Transactions on Communication, 31:532-540, 1983) seems to use 5x5 Gaussian kernel like

h1 = [1/4 - a/2, 1/4, a, 1/4, 1/4-a/2],

and a is between 0.3-0.6. A typical value of a may be 0.375, thus the Gaussian kernel is:

h1 = [0.0625, 0.25, 0.375, 0.25, 0.0625]

or

h1 = [1, 4, 6, 4, 1]/16.

I have written previously about Gaussian filtering, but neither of those posts make it clear what a Gaussian filter kernel looks like.

Window size

In a previous post I discussed how the size in pixels of the kernel is given by the chosen σ. Clearly, the Gaussian bell curve has a size given by the σ, and we need a sampling window large enough to contain the whole bell. With the convolution operation, we assume zero values outside the window. The Gaussian has long tails (in fact, it never reaches zero!), but it reaches values close to zero quite quickly. For example, at 3σ the Gaussian has a value of 0.0044, and if we assume it to be equal to zero after that, we ignore less than 0.3% of the total “weight” of the curve. Using a larger window to sample the same Gaussian therefore will not increase the precision by much, but using a smaller window quickly degrades the sampled Gaussian so that it doesn’t look like a Gaussian any more. This is why I always recommend to cut off at 3σ, though in some circumstances it might be better to increase that value. This is a graphical representation of the Gaussian cut off at various values of σ:

Note that the shape in the first two plots is very different to that of a Gaussian bell curve. It is the slow convergence to zero at the tails that give the Gaussian its desirable properties. I mention this here because it is not uncommon to find improperly implemented Gaussians. Because MATLAB’s Image Processing Toolbox implements the Gaussian with two parameters instead of one (i.e. the kernel size and the σ), users get confused and fill in incompatible values. This week I ran into a paper that applied

an 11x11 pixel Gaussian filter with standard deviation σ=25 pixels.

Yes, the authors in this paper used MATLAB to implement their algorithm. I have plotted such a filtering kernel (the one-dimensional version of it):

Does this look like a Gaussian to you? This was not in some obscure paper by some obscure scientists in some obscure university in some obscure country. This came from Duke University, ranked number 18 in the Times Higher Education’s World University Ranking. I mention this to reinforce how common these problems are.

Integer approximations

Next, let’s look at the [1, 2, 1]/4 kernel that is sometimes referred to as a Gaussian kernel. Remember that outside of this 3-pixel window, the kernel has a value of 0. This is thus equivalent to [0, 1, 2, 1, 0]/4, a clearly triangular shape. A kernel with triangular weights is the convolution of a uniform kernel with itself. You can read all about the problems introduced by the uniform kernel. The triangular kernel fixes some of those, but still is not nearly as good at suppressing high frequencies as the Gaussian kernel. If you convolve the uniform kernel with itself many times, you approximate the Gaussian kernel. Thus, a triangular kernel is a very rough approximation to the Gaussian kernel. Here is a plot of a continuous-domain triangular kernel and samples corresponding to the [1, 2, 1]/4 kernel:

The [1, 4, 6, 4, 1]/16 kernel that the reader mentioned above asked about is a different approximation to the Gaussian, using only small integer values. I plotted it here (black x), together with a Gaussian bell curve (σ=1) and a sampled Gaussian (circles):

Back in the day, integer computations were much faster than floating-point operations, and thus there was a value in using this type of approximation. Nowadays, this might still hold in some hardware, but is certainly no longer true in PCs, GPUs or even smartphones. I recommend that you sample a true Gaussian and stick to floating-point–valued kernels.

Proper counting

2014-09-23T11:12:00-06:00

I just came across an editorial in the Journal of the American Society of Nephrology (Kirsten M. Madsen, Am Soc Nephrol 10(5):1124-1125, 1999), which states:

A considerable number of manuscripts submitted to the Journal include quantitative morphologic data based on counts and measurements of profiles observed in tissue sections or projected images. Quite often these so-called morphometric analyses are based on assumptions and approximations that cannot be verified and therefore may be incorrect. Moreover, many manuscripts have insufficient descriptions of the sampling procedures and statistical analyses in the Methods section, or it is apparent that inappropriate (biased) sampling techniques were used. Because of the availability today of many new and some old stereologic methods and tools that are not based on undeterminable assumptions about size, shape, or orientation of structures, the Editors of the Journal believe that it is time to dispense with the old, often biased, model-based stereology and change the way we count and measure.

It then goes on to say that the journal would require appropriate stereological methods be employed for quantitative morphologic studies. I have never read a paper in this journal, but certainly hope that they managed to hold on to this standard during the 15 years since this editorial was written. Plenty of journals have not come this far yet.

Automated image-based diagnosis

2013-08-24T12:52:00-06:00

Nowhere is it as difficult to get a fully automatic image analysis system accepted and used in practice as in the clinic. Not only are physicians sceptical of technology that makes them irrelevant, but an automated system has to produce a perfect result, a correct diagnosis for 100% of the cases, to be trusted without supervision. And of course this is impossible to achieve. In fact, even if the system has a better record than an average (or a good) physician, it is unlikely that it is the same cases where the system and the physician are wrong. Therefore, the combination of machine + physician is better than the machine, and thus the machine should not be used without the physician.

What often happens then is that the system is tuned to yield a near 100% sensitivity (to miss only very few positives), and thus has a very low specificity (that is, it marks a lot of negative tests as positive). The system is heavily biased to the positives. The samples marked by the system as negative are almost surely negative, whereas the samples marked as positive (or, rather, suspect) are reviewed by the physician. This is supposed to lighten the workload of the physician. This seems nice and useful, no? What is the problem?

One example where automated systems are routinely used in the clinic (at least in the rich, western world) is for screening for cervical cancer. This is done with the so-called Pap smear since the 1940’s. It takes about 10 minutes to manually examine one smear, which is made on a microscope glass, stained, and looked at through the microscope. Even before digital computers became common, there have been attempts to automate the analysis of the smear. My colleague Ewert Bengtsson wrote his PhD thesis on the subject in 1977, and is still publishing in the field today. This gives an idea of how hard it is to replicate something that is quite easy, though time consuming, for a trained person. The solution, as is often the case, was to change how the sample is prepared. Instead of smearing the sample on a microscope glass, liquid cytology systems were invented that cleaned the sample (removing slime, blood cells, etc.), and produced a neat deposition of cells on the glass such that they are nicely separated and not likely to overlap each other. Such a preparation makes the automated image analysis much easier. However, these automated systems still do not produce a perfect result, and therefore are only approved to be used together with a trained cytologist. That is, the cytologist still needs to review all the tests. This means that the Pap smear test has become more expensive, rather than cheaper (the liquid-based sample preparation uses expensive consumables).

But the cost is not the real problem, and this is why these new automated Pap test systems are used in practice. The problem is that the probability that a sample is positive is quite low. That means that even a small bias towards the positives will produce a huge amount of false positives. For example, assume one in a 1000 tested women has cervical cancer, and that the system has a specificity of 90% (totally made up figures, I have no idea what these are in practice). This would mean that about 100 perfectly healthy women would be diagnosed with cancer by the system, for every one woman whose cancer is detected!

I recently came across an article in The New York Times reporting on a device, MelaFind, that is designed to recognize melanoma. To me it seems they did a lot of things right: the device takes images at various wavelengths, including some near-infrared, and at various depths under the skin surface (i.e. it obtains information that the dermatologist does not have); they trained the device on a database with 10,000 lesions (with biopsy information, from 40 different sites); etc. Yet look at these three comments from the FDA panel that are reported in the article:

[I am] concerned that a doctor could inadvertently use MelaFind on a non-melanoma skin cancer, receive a score indicating that the spot was not irregular, and erroneously decide not to biopsy it.

My concern with MelaFind is that it just says everything is positive.

There is inadequate data to determine any true value added for MelaFind for use by a dermatologist or other provider.

Why are these FDA panellists so negative? The first comment is about wrong use of the device. Of course, MelaFind is trained on a dataset of “clinically atypical cutaneous pigmented lesions with one or more clinical or historical characteristics of melanoma,” and therefore will produce random output if an image taken is of anything else. If the input is outside of the domain generated by the training set, all bets are off. This is valid concern, but not difficult to avoid by limiting the use of MelaFind to dermatologists, as the FDA did. The second and third comments, on the other hand, are what this blog post is about. As the article puts it: “[…] a biostatistician […] compared the accuracy of MelaFind in distinguishing non-melanomas to a hypothetical pregnancy test which, used on 100 nonpregnant women, would mistakenly conclude that 90 of them were pregnant.” The problem is low specificity!

Further reading on MelaFind’s dermatologist-only web pages (yes, I had to actually claim I’m a dermatologist to get to those pages!), you can see the details of the study that the FDA approval is based on. Apparently they collected a large group of patients, who together had 1632 lesions that were suitable for the device to diagnose. Of these, 175 were melanomas or high-grade lesions (I’ll just call them all melanomas for simplicity). That is, a little over 10% of the test set is positive, most of it is negative. MelaFind detected 173 of 175 melanomas (98.3% sensitivity). This is a very good sensitivity (but of course no mention is made of the dermatologists’ sensitivity in this study, more on that later). However, the specificity was only 10.8%. That is, it recognized only 157 negatives as negatives! MelaFind diagnosed as melanoma 1473 out of 1632 lesions shown to it (90.3%), pretty close to just saying everything as positive. And thus, as the 3rd comment above said, is there are added value to use the device?

We have to compare MelaFind performance with that of dermatologists. In the study, dermatologists had a 5.6% specificity. That is, they marked as melanoma even more of the lesions (though only 65 or so lesions more). Surprisingly, there is no mention of the sensitivity of these dermatologists. I would guess they get 100%, otherwise they would have reported it. Instead, they give some random statistics from another study that shows dermatologists only detect 70–80% of melanomas. There is no way of comparing that study to this one, as the input to the study (the test set) is most likely chosen very differently. And, given the clinical process, dermatologists are very much biased, as is the device, to positives: everything that might be a melanoma is taken out an send to a pathologist; they don’t want to run any risks of missing one. This guess of 100% sensitivity is also supported by the FDA’s approval specifically stating that it is not intended to be used on lesions with a clinical diagnosis of melanoma or likely melanoma, and that biopsy decision on MelaFind negative lesion is up to the dermatologist.

So assuming dermatologists really had a 100% sensitivity, the study says that, if always following the device’s recommendation, 65 fewer samples would have been biopsied, but two melanomas would not have been found. Is that a valid trade-off? Most surely not. I wonder what the dermatologists’ specificity would be if they would be a little bit less cautious (i.e. lowering their sensitivity to that of MelaFind). Or conversely, what the specificity of MelaFind would be if tuned to yield 100% sensitivity on this test set. In either of those cases, it would be possible to directly compare the specificities. I would bet that in such a case the dermatologists would do much better than MelaFind. And thus the comment regarding its added value is legitimate. In the article it says that “patients are paying $25 to $175 for the first mole evaluation and around $25 for subsequent moles.” Is this a waste of money? I don’t think that the device gives the patients a better diagnosis. If it just helps the dermatologist to do his/her job faster, why is the patient paying more for the diagnosis, instead of less?

I hope this does not sound too negative. I know a lot of things can be automated and improved with image analysis, even in clinical settings. I truly believe we will have fully automatic diagnosis at some point in the not too distant future (hey, Dr. McCoy didn’t need to go through med school just to hold that device that diagnosed and cured all his patients, did he?). But we’re not nearly there yet: we need to do more work on our ability to recognize wrong input (in this case the non-cutaneous or non-pigmented lesions, the non-lesions, the patient’s shirt, etc.), and we need to do more on our ability to generalize from examples. This is what sets the human eye–brain combo apart from anything else, natural or engineered: we can take the context into account, we can adapt to environmental changes, and we generalize like there is no tomorrow.

How did this get published?

2012-12-19T11:33:00-07:00

“How did this get published?” is a question I regularly ask myself when reading new papers coming out. I just came across another one of these jewels, and because the topic is that of a previous blog post here, I thought I’d share my frustration with you.

The paper is entitled “On Perceptually Consistent Image Binarization” (IEEE Signal Processing Letters 20(1):3:6, 2013), which leads one to believe that this is an interesting paper related to segmentation or thresholding and visual perception. The abstract, however, helpfully lets us know this is about error diffusion. And it promises a strong theoretical foundation leading to perceptually consistent halftone images. I previously wrote about error diffusion here, so let’s look at what they did.

The introduction doesn’t start very strongly:

Of various image binarization methods, error diffusion is a popular one and has been widely used in image printing and display applications [6]. It was firstly introduced by Floyd and Steinberg and lately some other variations have also been proposed.

And that is it, no review whatsoever of the later variations. This is troubling, because this typically means that they will only compare their new method to that very first method, and ignore all of the literature that came after. If you read my previous blog post, you’ll know that error diffusion was introduced in 1975. That is nearly 40 years of research that these authors are pretending never happened!

The proposed scheme turns out to be the same as Floyd-Steinberg, but they also suggest the use of a larger window, as Jarvis, Judice and Ninke did in 1976. Thus, this is a paper proposing yet another set of weights for the standard algorithm. The promised theory is quite funny too, if you’re only somewhat familiar with diffusion. They start with heat exchange (this is basic diffusion), and somewhere propose that

Then, the amount of heat exchanged between the current pixel and its neighbours in unit time is in proportion to the reciprocal of their distances, that is .

This comes out of nowhere, without references or proof, of course. And we all know that the solution to the heat equation (basic, isotropic diffusion) is a Gaussian. That is, heat exchanged between two points is proportional to . The weights proposed are given only as an equation, and they demonstrate with 4 neighbours (as did Floyd-Steinberg) or with 10 (two neighbours less than Jarvis-Judice-Ninke).

Now comes the kicker! The experimental results is always the place where I get most frustrated. Apparently we don’t teach how to do science any more. This is the most important part of our jobs, but instead most papers in image processing use the experimental results section to show the extent of our ineptitude. Worse even than I feared, the “new” method is compared to thresholding! This is the first paragraph of the results section:

To evaluate the performance of the proposed scheme, experiments are implemented. [wow! how clever!] Partial results on some test images are given in Figs. 3–6, where Fig. 3 are the original test images, Fig. 4 shows the binarized results of traditional fixed-threshold scheme, Figs. 5 and 6 are the results of the proposed scheme with d_s=√2 and d_s=√5 [these are the two neighbourhood sizes] by using raster scan order, respectively. From the results one can see that, compared with the traditional fixed-threshold scheme, the visual quality of the binarized images with our scheme can be greatly improved. [!!!]

They are actually saying that visual quality of error diffusion is better than that of thresholding!

The next paragraph compares, visually, on 4 images, the result of the “new” method with those of Floyd-Steinberg (1975) and Jarvis-Judice-Ninke (1976), leading to the conclusion that it is difficult to tell the difference between the methods (guess why!). Sadly, the authors chose not to show the results of these old methods on two of the images that they showed in the previous paragraph. This probably means that these images looked better with the old methods! Finally, the third paragraph of the experimental section does an “objective” comparison, with actual numbers. They blurred the dithered images and compared them with the original images using the PSNR measure. Of course, as I wrote before, the PSNR measure is pointless. But what is worse is that the differences with Jarvis-Judice-Ninke are mostly insignificant: 28.018 vs 28.079, etc.

To further embarrass themselves to the readers of this blog, they blurred the image by filtering “three times with the 3×3 Gaussian low-pass filter.” As you all know, repeated applications of the Gaussian filter are equivalent to a wider Gaussian filter, and 3×3 pixels is not enough to correctly represent a Gaussian filter. I am afraid that these people fell into the trap put in front of them by MATLAB, which by default generates a 3×3 mask for a Gaussian filter.

I’ve run the algorithms with the proposed weights on the test image I used previously. These are the results:

I’ll copy the Floyd-Steinberg (1975), the Zhou-Fang (2003) and the Chang-Alain-Ostromoukhov (2009) results from the previous post for comparison:

How did this paper get published?

Scientific writing and the pronoun I

2011-10-05T17:08:00-06:00

Why are scientists so scared of writing their statements in the first person? Open any journal, and look for the word “I”. Chances are, you won’t find it. You’ll see article authors jump through hoops just to avoid this word. As if it were dirty, illegal. For example, instead of a normal, complete sentence like “I found existing methods to be insufficiently accurate,” you’ll find the sentence “Existing methods were found to be insufficiently accurate.” This leaves the most important thing out: who found them insufficiently accurate? Is this a generally known thing? Does the whole world share this opinion? Was it the dog that didn’t like the method? Why do they shy away from the word “I”? Does it make science less objective?

In the example above, the author tried to avoid subjectiveness by rewriting the sentence and leaving out the active subject. As if that makes the concept less subjective. It was the author, after all, who evaluated and discarded these methods. The author didn’t like them. This is a subjective evaluation. The only way to avoid subjectiveness is to devise an experiment, and prove that existing methods do not meet the required accuracy requirement. Lacking that, the statement is subjective, and hiding this fact is misleading.

Scientists are subjective, this is no secret. They are human, after all. This means that, no matter what people want to believe, the scientific process is subjective as well. A scientists chooses what to study, and his (or her) opinions influence which of his (her) results are published, for example. This leads to publication bias, of course, but otherwise does not taint the objectiveness of the scientific results. As long as these scientists are honest, and don’t invent data, or misuse statistics, their opinions should not influence the outcome of their experiments.

But if it is subjectiveness that these authors are scared of, why is it less of a problem to use the word “we”? Is it the implied consensus? It is no longer the opinion of a single person, but that of a group of people, and thus more objective in some way. This leads to strange things, like the editors of IEEE replacing all occurrences of “I” with “we” in a recent paper of mine (if there is only one author, who is the “we” in “First we will give a short description of…”?). There’s a famous quote of Mark Twain that is relevant here: “Only kings, presidents, editors, and people with tapeworms have the right to use the editorial ‘we.’“

Interestingly enough, this fear of the pronoun “I” is rather recent. Einstein, at the beginning of the 20th century, wrote things like “It is possible that the movements to be discussed here are identical with the so-called ‘Brownian molecular motion’; however, the information available to me regarding the latter is so lacking in precision, that I can form no judgment in the matter” (in the first paragraph of “On the movement of small particles suspended in stationary liquid…”, a translation of which can be had here). Going back in time to the 17th century, we can find even more extensive use of the first person pronoun. Newton, for example, gives a good example of how experiment-based papers should be written (in my opinion): “[…] I became surprised to see them in an oblong form; which, according to the received laws of Refraction, I expected should have been circular” (from “A new theory of light and colors”). This is so much more vivid and pleasant to read than “they had an oblong form, not circular as predicted by laws of refraction.” But maybe the beginning of that same sentence could have been redacted out, as it certainly does not relate in any way to the science being described: “It was at first a very pleasing divertisement, to view the vivid and intense colours produced thereby […].” Nonetheless, this paper would have been much more boring to read had it been written today.

So I wonder, why did this openness about the subjectivity of the scientific process disappear? When did they start to become afraid of “I”? Why? Please weigh in by adding a comment, I’d love to hear your opinions!

Edit

I was just made aware of this publication (there’s a scanned PDF available here if you don’t have access to the version at Wiley’s online library). It doesn’t talk at all about personal pronouns, but it does criticize the boring tone of scientific writing nowadays. Nice read!

Gaussian filtering with the Image Processing Toolbox

2009-10-06T22:28:00-06:00

Edit May 2018

Since publishing this post, the MATLAB Image Processing Toolbox has added the function imgaussfilt that correctly applies a Gaussian smoothing filter. For Gaussian derivatives, the recommendations here still apply.

Edit July 2024

I just learned that imgaussfilt uses a truncation of 2 by default (i.e. it computes cutoff = ceil(2*sigma)). This is not sufficiently large for most applications, see this other post. There’s a 'FilterSize' name-value argument that should be used with this function to set the correct filter size, computed as shown in this post.

If you don’t use DIPimage, you probably use MATLAB’s Image Processing Toolbox. This toolbox makes it really easy to do convolutions with a Gaussian in the wrong way. On three accounts. The function fspecial is used to create a convolution kernel for a Gaussian filter. This kernel is 2D. That’s the first problem. The other two problems are given by the default values of its parameters. The default value for the kernel size is [3 3]. The default value for the σ (sigma) is 0.5. σ=0.5 is too small for a Gaussian kernel. When a kernel this small is sampled, too much aliasing occurs. Depending on who you listen to, the smallest σ you can use is either 0.8 or 1.0. The value of σ dictates the shape of the kernel, but it also determines its size: truncating the Gaussian kernel at [3 3] pixels creates a sharp border which will cause unwanted artifacts in the output image. All the good properties of the Gaussian filter will be lost. You have to cut off the Gaussian at a point where it is close to zero. This happens around 3σ. This means that the size of the Gaussian kernel should be 6σ (cut off at 3σ on ether size of the origin). Or if you need really high numerical accuracy, you can make this larger, say 10σ. These last two issues are easy to take care of:

img = single(imread('cameraman.tif'));
sigma = 3;
cutoff = ceil(3*sigma);
h = fspecial('gaussian',2*cutoff+1,sigma);
out = conv2(img,h,'same');

However, this does not solve the first issue: the filter h is two-dimensional, but the Gaussian is separable! That it’s separable means that we can save a lot of computation by convolving with a one-dimensional Gaussian along the x-axis and then with another along the y-axis. You can accomplish this with:

h = fspecial('gaussian',[1,2*cutoff+1],sigma); % 1D filter
out = conv2(h,h,img,'same');

The separable Gaussian filter is always faster to compute when implemented as two one-dimensional filters than when implemented as one two-dimensional filter. Taking σ=1, this means we do 7+7=14 multiplications per pixel instead of 7×7=49 multiplications per pixel. The savings are increasingly important for larger kernels. With σ=5 this translates to 62 vs 961 multiplications.

Finally, a Gaussian derivative should not be implemented like so:

h = fspecial('gaussian',[1,2*cutoff+1],sigma);
out = conv2(h,h,img,'same');
out = diff(out,1);

Nor like this:

h = fspecial('gaussian',[1,2*cutoff+1],sigma);
dh = diff(h);
out = conv2(dh,h,img,'same');

Computing the finite difference approximation to the derivative of a smoothed image is the same as convolving with the finite difference approximation to the derivative of the Gaussian. As discussed earlier in this blog, the advantage of computing Gaussian derivatives is lost when implemented this way. Instead, compute the exact derivative of the Gaussian, and convolve with that:

h = fspecial('gaussian',[1,2*cutoff+1],sigma);
dh = h .* (-cutoff:cutoff) / (-sigma^2);
out = conv2(dh,h,img,'same');

Also note that, because the derivative of the Gaussian goes to zero slightly slower than the Gaussian itself, it is worthwhile to increase the value of cutoff to at least ceil(3.5*sigma).

DIPimage issues that I can't solve

2009-08-25T11:26:00-06:00

Edit December 2018

Over the past two years I’ve been working on DIPimage 3, an updated version of DIPimage that is based on a rewritten DIPlib. Just about all of the issues discussed in this blog post have been resolved there, with the exception of the end operator, which will now do the wrong thing if used in curly braces (tensor indexing), and the limitation to the order of spatial and tensor indexing. We incurred a few backwards compatibility breaks, but it was necessary for the long-term health of the toolbox.

There are a few things in DIPimage that I’m annoyed with but can’t solve. Some of these are caused by the limitations of the MATLAB interpreter, and some are caused by my own poor design choices. DIPimage evolved over time to do things that I hadn’t at first even thought about, such as processing color images. Tensor and color images were “tagged on”, if you will, on top of the dip_image object, introducing some oddities and inconsistencies. To fix these up we would need to change existing behavior, which we don’t want to do because it would break too much existing code. This blog entry is about the few weird things with the dip_image object and how to work around them.

The dip_image object was born to contain only a single grey-value object. Pretty soon it was extended to contain multiple, unrelated images (mirroring DIPlib’s dip_ImageArray structure). It seemed logical to me to distinguish a single image and an array of images, even though to MATLAB they are the same object, so I overloaded the isa and class methods to make this distinction artificially. When there is more than one image in the dip_image object, it reports itself as an object of class dip_image_array. This caused a few problems when we introduced tensor (and color) images. As they say, hindsight is always 20-20. Now I wish I had implemented image arrays as a separate class, or even simply as a cell array with dip_image objects.

Indexing into a `dip_image` or a `dip_image_array`

Indexing into a dip_image_array is done using braces: A{1}; indexing into a dip_image is done using brackets: A(20). Because of a limitation of the MATLAB parser, it is only possible to combine these two indices by putting the braces before the brackets: A{1}(20). This is not so bad if you consider the images in the array to be unrelated: first you select image number 1, then you select pixel 20 in that image. But when we introduced the concept of tensor images, this limitation became frustrating. In a tensor image each pixel is a matrix. This is represented as a dip_image_array where each image has the same size, and contains the data for one of the tensor elements. For example, an RGB image is a tensor image with a 3-element vector at each pixel. The first image in the dip_image_array is the red channel, the second is green, the third is blue. In such an image, A(20) is a tensor (in this example an RGB triplet). It would sometimes make more sense to index a tensor element by A(20){1} rather than A{1}(20). But it generates an error when you type it in, and there’s nothing I can do about that.

Methods overloaded for `dip_image` and `dip_image_array` objects

Even worse is the way that the functions size, length, ndims, numel and end work. When the object is of type dip_image, they work on the image, and when the object is of type dip_image_array they work on the array, not the images inside. This made a little bit of sense initially, since it was assumed that the images inside the array were unrelated. But with a color image A:

size(A)
B = colorspace(A,'grey');
size(B)

ans =
     [3,1]

ans =
     [256,256]

In the same way, size(A{1}) would return [256,256] and size(A{1:2}) would return [2,1]. Needless to say, this has lead to some hard-to-find bugs.

It is possible to get around this issue by using only the functions imsize and imarsize for the image size and the image array size, respectively. length(A) should be written as max(imsize(A)), and ndims(A) as length(imsize(A)). numel always returns 1. This was implemented to circumvent a bug in MATLAB 6 that broke the indexing (I don’t know if this was fixed in later versions). numel(A) should be written as prod(imsize(A)).

Finally, the end operator is the worst of the bunch, because I keep using it without thinking about it, and it always takes me a while to find the cause of the bug. Also, the workaround is rather ugly. Like size, end uses the image size if there is only one image in the object, but uses the array size if there is more than one image. Thus, A(0:2:end,:) works perfectly well when A is a grey-value image, but doesn’t do what you’d expect if A is a tensor image. The workaround again is using imsize: A(0:2:imsize(A,1)-1,:). Ugly, no? The reason for this issue is that MATLAB does not report to the overloaded end method whether it is invoked in brackets or braces. The end method existed before cell arrays and braces indexing were introduced in MATLAB. It seems that MATLAB itself also has problems caused by evolution of the software and the desire to keep backwards compatibility! Because the end method does not know the context in which it is called, it has to guess what it is that the user wants, just like size and length have to do. The decision to have all these functions make the same assumptions is sound, but if we would have thought about tensor and color images before implementing these, they might have always looked only at the image dimensions and never at the array dimensions.

Concatenating `dip_image` and `dip_image_array` objects

The last issue I have is with concatenation of images. Again, this was implemented before we even thought of the possibility of using color images, and image arrays usually contained unrelated images. When concatenating an image array, it is treated like one wishes to concatenate the images in the array. Therefore, [A,B], with A an image array, is equivalent to [A{1},A{2},...,A{end},B]. This is very frustrating when you have two color images that you wish to concatenate together. The workaround is, as usual with color images, to use iterate.

iterate repeats one operation on every image in an image array. This is often the only simple way of applying a filter to an image. Bit by bit we are improving functions so that they work on tensor images. For example, gaussf(A) now works as expected on a color image. But many functions do not work yet, and many will never work (for example, mathematical morphology on tensor images can not be defined properly, because there is no universally correct way to decide which of a set of tensors is the largest). For these functions it is possible to use iterate to apply them to each of the components of a color image: iterate('dilation',A,25,'rectangular'). Note that the result contains colors not in the input image: this operation is not a dilation!

iterate is clever enough that, when it is called with multiple image arrays as input, it applies the function with corresponding images from each of the arrays. Thus, to concatenate two color images, one can do iterate('cat',1,A,B).

A better design

If we had thought about color images from the start, we would have created a dip_image object that can hold a grey-value, tensor or color image, but never unrelated images. An image array would be either a cell array with images or a completely separate object. In that way it would be possible to have an array with one image, which now is impossible. The size and related functions would know whether the input is an image (whatever the type) or an image array. Concatenation would be able to do different things on an image array and on a tensor image. However, the end operator would still be incomplete, only usable within brackets but not braces, and the limitation to the order of brackets and braces would still stand. We would also have to introduce methods to return the size, length and number of dimensions of the tensor.

But it is too late for this type of change, I feel. I’m hoping this post will help people understand the design choices made, and help them get around the limitations and oddities of the toolbox as it stands now.

Why do we keep using the word "pixel"?

2009-01-08T16:38:00-07:00

According to Wikipedia, the ultimate source of knowledge, the word pixel comes from “picture element.” This means that a pixel is a part of a picture, and yet everybody I know keeps using that term to refer to a part of an image. Because it is “image analysis” that we do, right? Not “picture analysis”? Of course, this is just an insignificant detail that I’m blowing way out of proportion. No, my real beef with the word pixel is more complex.

The word pixel seems to be used only in the context of 2D images. For 3D images we have a different word: voxel. So if I have a 3D image, and take one 2D slice out of it by selecting a set of voxels, these voxels all of a sudden, magically, become pixels! And what happens when you record a multi-spectral volumetric image? Or a volumetric time series? What do you call the elements of a 4D, 5D or 10D image? Dean et al. use the word imel (for image element) in their ICS file format specification (P. Dean et al., “Proposed standard for image cytometry data files”, Cytometry 11(5):561-569, 1990, DOI:10.1002/cyto.990110502). Yes, it’s more general. Yes, it’s more awkward. And yes, it still hides the fact that the images that we analyse are sampled and discretised representations of some continuous reality. When we sample a function we obtain samples, not pixels. The data sets that we analyse are collections of samples. A digital image is composed of samples, whether it be a 2D image, a 3D image or a 10D image. Only after you paint a little rectangle on the screen with the color of your sample does it become a pixel.

PS: I’m suggesting the word lixel for samples in a 1D signal. You heard it here first!