I woke up awhile back extremely excited to answer the question if I could use a picture and image processing to help isolate nutrient deficiencies or toxicities. All the while thinking if I mentioned this to my family they would say something…”can’t you just look at the leaf?” Yes family. You are indeed right….and yet…I bumble onward in this effort. Still – ever grateful (seriously) that my family cares enough to ask! I crave the thrill of learning how to do something I am clueless about. And who knows…down the road this knowledge might come in handy.
While plants are poor communicators, they do offer visual clues when they are getting not enough or too much of a nutrient. One of the big visual clues is when all or part of a leaf turns yellow.
This is known as chlorosis. The goal of this post is to take an image of a leaf and have my Mac tell me the overall and localized level of chlorosis. The code will work but probably just barely. It will be a sacrificial draft.
Thanks To Those That Went Before
Dr. Mark Chopping, Montclair State University and Dr. Ved Sharma, Albert Einstein College of Medicine. These two wonderful folks answered a question I had on how to create selections based on quadrants of the images. Dr. Chopping led me to understand how getting the bounding rectangle (i.e.: run(“Set Measurements…”, “area bounding redirect=None decimal=2”);) is critical to finding these quadrants. Dr. Sharma pointed out to get only the leaf pixels in the quadrant is a simple task of taking the intersection of the leaf selection with a quadrant.
The ImageJ macro and images used in this post are located at this Github location.
The color analysis algorithm I will use was proposed in the article Smart Sensor for Real-Time Quantification of CommonSymptoms Present in Unhealthy Plants .
My biggest challenge with implementing the algorithm described in the document was a lacking in how to interpret the math language as well as my general fear of math. Ah…but why stop because someone else (Oh alright – several folks) said I can’t do math…
The code on my Mac will:
- Divide an RGB image of a leaf into four quadrants at the leaf’s centroid.
- Figure out how yellow each quadrant is.
- Figure out if the yellow is localized to a quadrant or is all over the leaf.
I will use the definition of Yellow that was proposed in the paper: Yellow = The average of the pixel’s (.5)Red + (.5)Green values.
Figure 13 and the text around it in the paper
gives an example of the chlorosis detection algorithm in action.
Getting an Image
If You Have a Moment…
I’d like to run as many leaf images as possible. It would be very helpful to get images from you. If you are interested, please send the images to email@example.com using the image capturing technique described in this section. I will send you back results values from the analysis.
It took me a few tries at different techniques in order to create an image. I ended up using the white screen of an iPad as background. It is important to get the screen as clean as possible! I opened the Notepad app on the iPad and got most of the iPad’s screen to show a note on white background. Next I put the leaf on the iPad. I put some care into flattening the leaf so that the leaf laid relatively flat on the screen.
After turning off the overhead lights and minimizing any reflections, I took a picture of the leaf with my iPhone.
This technique provided a contrast between the background as well as a backlight to bring out the leaf’s color.
Processing The Image
There is a super amazing open source software tool available to all of us – ImageJ – thanks to the excellent work by Wayne Rasband, the NIH, and the community. I used ImageJ as well as Fiji to get the yellow values needed to figure out the amount of chlorosis. I preferred Fiji for the majority of the programming. Unlike the ImageJ editor, the Fiji editor is language aware which makes editing much easier. On the other hand, ImageJ has a source level debugger that lets me step through the code. Stepping through the code is an immense help in understanding what is going on as well as where an error is lurking. Yet, Fiji has bundled in a lot of great things into the development environment that make it much easier to get stuff done because the additional plug-ins and such are part of Fiji’s installation.
I almost always used Fiji except when debugging. The challenge here is if I modify the script in ImageJ, Fiji won’t read the file. I got around this painful challenge by copy/pasting the code between the two editors. The ideal environment would have ImageJ’s debugger part of the Fiji environment.
I’ll refer to both packages as ImageJ during discussions.
ImageJ Processing Steps
I used ImageJ’s macro language and GUI. The experience is messy since windows with images at various processing states appear and then are closed. I would imagine if I evolve this project that I will move off any GUI to a command line execution.
The steps below are a high level discussion of the ImageJ macro code found at this Github location.
Define Leaf Quadrant Selections
The first part divides the leaf pixels into four quadrants at the centroid.
- Get the GUI focused on processing the leaf image. If there are any windows open showing images or dialog boxes, close them.
- Get the path and filename of a leaf image from the user.
- filepath=File.openDialog(“Select a File”); : openDialog opens the file picker dialog. Go to the folder holding the image and click on the image.
- filename = File.getName(filepath);
- Open the leaf image in a window.
- Create a mask from the leaf image and figure out an outline around the leaf – in ImageJ this is called a selection – in which the pixels inside the selection are part of the leaf. These are the pixels that will be used to figure out the leaf’s yellow amount.
- selectLeafMask() : A mask is created by setting the threshold color. The threshold color defines what HSB (HSB is the same as HSV) a pixel needs to be in order to be included within the mask.
Getting the color threshold “right” for all leaves is an area of further exploration. There is a good chance for false positives – the case when multiple areas are determined to be within the color threshold. In the case of multiple selections, I check to see which has the largest area and assume that is the leaf. I got the macro code by ImageJ’s macro recording feature. I turned on the macro recorder in Fiji, opened the file, went into the threshold color dialog box. I then set the Hue, Saturation, Brightness values. Now that I had the recorded code, I copy/pasted it into the selectLeafMask() function.
- Create four more selections based on a bounding rectangle of the leaf selection and the centroid of the leaf.
This gives four quadrants that includes the parts of the leaf but also some of the white background.
- The pixel calculations should not include any of the white background. To just have leaf pixels in the selection, create four selections that do not include any of the white background. This is done by taking the intersection of the leaf selection with each of the four quadrants.
Whew! Time to take a breath before we dive into calculating the Yellow values…
The second part calculates the Yellow values.
Finally – it is time to figure out a leaf’s relative amount of generalized or local chlorosis.
As noted in the article where these calculations come from:
- Rk is the average yellow pixel value of the k selection (where k = 1-4).
- Rn sums up the four Rks to give a relative value to the amount of chlorosis.
- Rdiff gives a relative value to the difference in chlorosis values between the four leaf quadrants.
Green Basil Leaf
Rn = 207, Rdiff = 15
Unhappy Spearmint Leaf
Rn = 544, Rdiff = 62
First Leaf Gathered on Walk
Rn = 363, Rdiff = 17
Second Leaf Gathered on Walk
Rn = 421, Rdiff = 21
The Green Basil Leaf has the lowest Rn and Rdiff. This visually makes sense. These values are around the range given in the article’s figure 13.a for a low-generalized and low-localized chlorosis.
At an Rn of 544 and Rdiff of 62, the Unhappy Spearmint Leaf analysis gives numbers for a localized chlorosis.
The first leaf gathered on a walk I went on gives numbers for a small amount of generalized chlorosis while the second leaf gathered points to a higher level of generalized chlorosis.
And So On…
That was quite a learning mountain climb for me. Along the way I was delighted to learn about ImageJ, a potential algorithm for analyzing an image of a leaf for chlorosis, and felt a large amount of gratitude for those that took their time to help. Will I continue adding other refinements (such as inter veinal chlorosis detection, necrosis, etc.) or evolve the code? I sure would if there was a practical reason to do so. I definitely want to test what I have on other leaves to evolve the robustness.
THANKS for reading this far. Please find many things to smile about.