Inverse estimation of canopy architecture from 3D images

Abstract

In order to estimate the 3D structure of a sugarcane canopy ...

What is the use of this?

3D canopy modeling; *grab text from proposal*

What has been done before?

*Not sure where to put this or if we’ll need any of it, but here’s a short summary of Cope et al.:*

A review of digital plant morphometrics by (Cope 2012) describes several features that can be extracted from leaf images for the purpose of species identification. Because traditional parameters such as leaf length and area vary significantly with environment and age, even within species, the review focused on parameters related to leaf shape. For example, one method involves elliptical Fourier analysis to describe the leaf outline, which can then be compared with a database of known species to identify the plant in question. Many of the techniques surveyed make use of principal component analysis and morphological image processing to extract the relevant features.

**Soy**

**Sugarcane**

**Agave**

One way to simplify the problem could be to use Agave. because we can get a very good 3D representation of all leaves on an Agave.

We don’t have a geometric model for agave (they existI think we can start from scratch - that could be a ’fun’ task for a math student). One ’first principles’ / unifield scaling theory approach (PRICE 2006).

From (Hunt 1987) \(A = 0.0138V^{0.751\simeq ^3/_4}, r^2 0.98, n = 34\) for *Agave desertii*, which is a biologically significant relationship. They also find that \([A,V]\) increase linearly with time, that root length scales. (As mentioned in (Davis 2014), use of these allometries to estimate biomass (as done in many publications by Nobel) is flawed, but there is no apriori reason to suspect that the underlying data is flawed - this is a great fit, but allometries are good like that \(r^2\) in the high 90’s is common for many allometric relationships (e.g., FIA allometries, http://www.globallometree.org/. Can it be reproudced?)

(Woodhouse 1980) divide leaves into “altitudinal classes”, with bins of 22.5\(^{o}\), a surrogate for leaf age.

First objective is to obtain a representation of a canopy as a triangluar mesh. There are a few approaches. The current approach (Song 2013) is to use the mCanopy (source) model to scale from plant geometric traits to the canopy.

We can also generate a triangular mesh directly using either kinect based 3D camera or reconstruction of a series of images.

In reviewing our options for capturing 3D models of objects, we found there were two major strategies in the field of photogrammetry: structure from motion (SFM), and structured light scanning (SLS). Structure from motion allows the production of 3D models using any off-the-shelf digital camera. It takes a set of photographs that capture the object from many angles and detects features common to multiple pictures, determines how they distort from one image to another, and builds a 3D model based on how the image of the object changes from picture to picture. Structured light scanning employs a more rigid setup of a camera and projector. A known image (typically black and white line patterns) is projected onto the object to be scanned, and software calculates how that projected image is distorted by the object it’s projected on to.

Some software products based on structure form motion include Autodesk 123D Catch, Agisoft, Bundler, VisualSFM, and Sure. Products that employ structured light scanning include the Microsoft Kinect (and various software interfaces, including ReconstructMe and Skanect), DAVID 3D Scanner (projector and software package), and a script produced by the MIT Center for Bits and Atoms called Three Phase.

We’ve tested the Kinect with ReconstructMe, and captured several models with 123D Catch and Agisoft.

**Kinect** based 3D imaging. We first tested a Kinect camera connected to ReconstructMe software using OpenNI drivers. We had promising initial results - with very little setup, it took seconds to produce a mesh that represented our young soy plant. We were able to save this file as a ASCII (plain text) .ply file of 60 MB (Figure \ref{fig:3dimages}a) In subsequent attempts, however, we realized that our first scan likely hit the limit of the detail this particular device (the Kinect) could provide. It’s only a 640x320 resolution camera, and its focal length is designed to capture a person standing several feet away from the video game console, not to get detailed close-ups of an object. Using it to capture 3D models is already a stretch for the technology, it was designed to capture human gestures: gross motor movements, not detailed 3D models. Note: the PrimseSense camera has identical tech specs and capabilities.

**123D Catch** is a freemium (free downloads up to a certain point, then $10/month) cloud service from Adobe Autodesk accessible using a browser and with an iPad app. The iPad app allows user to collect a set of images, which are then stiched together in the cloud (Figure \ref{fig:3dimages}b). This software offers convenience and ease of use (you can log in from any computer to download the resulting models, and view the models on any computer that runs OpenGL in the browser), but has arbitrary limits imposed by Autodesk: maximum 70 photographs, and fixed size of texture map: limited amount of detail to lessen the computational burden on Autodesk’s end (since the 3D model is being generated “in the cloud” i.e. on Autodesk’s dime.) The results of 123D Catch is a binary .obj or .stl, so it takes further processing to convert into a plain text file that can be processed by custom scripts.

**Agisoft** is software that can be purchased and run locally, allowing us to use our own computational resources (instead of waiting for ’the cloud’ to process the result) and giving us much more control of the level of detail we want to capture. There is also no artificially imposed limit on number of photographs, though the more photographs that are processed, the longer the process takes - and it should be noted that this technology is very computationally demanding. Without adaquate processiing power and memory, generating a model will take many, many hours. The software will take advantage of as many processing cores and as much RAM as you can provide it. It can, however, be used on modern consumer machines (high detail models took about 2 hours to produce on my computer equipped with an i7 processor and 8 gigabytes of RAM). The workflow is very simple, and the resulting file can be exported with a variety of options. Most useful is that it can be saved directly as an ASCII (plain text) file with each line simply corresponding to x,y,z coordinates and r g b color values, ready to be processed by custom-built programs.

See the Image protocol (appendix)

We have already simulated Energycane using a reparameterization of the original rice model.

**Soy**

**Sugarcane**

**Agave**

as in (LeBauer 2013))

... please feel free to choose a more appealing set of variables (eg l,w,dh)

What are appropriate distributions? Can start with log-N, Weibull, Gamma for length, width, height; what about angular data?

height \(h\) / internode distance \(\Delta h\);

internodes bigger at the bottom. Is this explained by \(\delta^2 h < 0\)?

to sample: \(\Delta h ~ \textrm{Weibull}(\mu, \sigma); \mu=mx+b, ax^b\)

can fit to data.angle \(\beta \sim ?\) (appropriate angular distribution?)

curvature radius \(\alpha\) \(\sim ?\)

length \(l\) \(\sim ?\)

width \(w\) \(\sim ?\)

Geometric measurements came from ...

**Soy**

**Sugarcane**

**Agave**

We have both data and model output represented as a ’triangular mesh’. The triangular mesh is in the format of an \(n \times 9\) matrix. Each row represents the vertices of a single triangle \(\left(x_1, y_1, z_1, x_2, y_2, z_2, x_3, y_3, z_3\right)\) in each row.

Model output will be an \(m \times n \times 9\) cube, with \(m\) representing the samples from parameter space. For each \(1...m\) sample, we will compute a likelihood as follows.

\(L = f(y-\Theta)\)

Where \(y\) is the data and \(\Theta\) is the model.

Calculating a likelihood, some function of data-model offset given an \(n \times 9\) and a \(n \times n \times 9\) arrays of a triangular mesh is, well, it is not straightforward to subtract one triangle from another. Perhaps we need to identify features.

TODO: Literature review: how do we do this?

TODO: define \(f\)

We can either 1. quasi-random sample from parameters, run model, and emulate (approximate) the likelihood and sample from this likelihood with an MCMC algorithm (Wikle 2007) or 2. use a more standard MCMC directly on the model (Ogle 2008).

## Share on Social Media