1Brain and Behavior Discovery Institute, Georgia Health Sciences University, 2Vision Discovery Institute, Georgia Health Sciences University, 3Department of Opthalmology, Georgia Health Sciences University, 4Intelligent Systems Laboratory, Palo Alto Research Center, 5Pattern Recognition Systems, Palo Alto Research Center, 6Department of Psychology, University of Minnesota
Hauffen, K., Bart, E., Brady, M., Kersten, D., Hegdé, J. Creating Objects and Object Categories for Studying Perception and Perceptual Learning. J. Vis. Exp. (69), e3358, doi:10.3791/3358 (2012).
In order to quantitatively study object perception, be it perception by biological systems or by machines, one needs to create objects and object categories with precisely definable, preferably naturalistic, properties1. Furthermore, for studies on perceptual learning, it is useful to create novel objects and object categories (or object classes) with such properties2.
Many innovative and useful methods currently exist for creating novel objects and object categories3-6 (also see refs. 7,8). However, generally speaking, the existing methods have three broad types of shortcomings.
First, shape variations are generally imposed by the experimenter5,9,10, and may therefore be different from the variability in natural categories, and optimized for a particular recognition algorithm. It would be desirable to have the variations arise independently of the externally imposed constraints.
Second, the existing methods have difficulty capturing the shape complexity of natural objects11-13. If the goal is to study natural object perception, it is desirable for objects and object categories to be naturalistic, so as to avoid possible confounds and special cases.
Third, it is generally hard to quantitatively measure the available information in the stimuli created by conventional methods. It would be desirable to create objects and object categories where the available information can be precisely measured and, where necessary, systematically manipulated (or 'tuned'). This allows one to formulate the underlying object recognition tasks in quantitative terms.
Here we describe a set of algorithms, or methods, that meet all three of the above criteria. Virtual morphogenesis (VM) creates novel, naturalistic virtual 3-D objects called 'digital embryos' by simulating the biological process of embryogenesis14. Virtual phylogenesis (VP) creates novel, naturalistic object categories by simulating the evolutionary process of natural selection9,12,13. Objects and object categories created by these simulations can be further manipulated by various morphing methods to generate systematic variations of shape characteristics15,16. The VP and morphing methods can also be applied, in principle, to novel virtual objects other than digital embryos, or to virtual versions of real-world objects9,13. Virtual objects created in this fashion can be rendered as visual images using a conventional graphical toolkit, with desired manipulations of surface texture, illumination, size, viewpoint and background. The virtual objects can also be 'printed' as haptic objects using a conventional 3-D prototyper.
We also describe some implementations of these computational algorithms to help illustrate the potential utility of the algorithms. It is important to distinguish the algorithms from their implementations. The implementations are demonstrations offered solely as a 'proof of principle' of the underlying algorithms. It is important to note that, in general, an implementation of a computational algorithm often has limitations that the algorithm itself does not have.
Together, these methods represent a set of powerful and flexible tools for studying object recognition and perceptual learning by biological and computational systems alike. With appropriate extensions, these methods may also prove useful in the study of morphogenesis and phylogenesis.
1. Creating Naturalistic Virtual 3-D Objects using VM
2. Creating Naturalistic Object Categories using VP
3. Additional Methods of Creating Shape Variation: Digital Morphing
4. Additional Methods of Creating Shape Variation: Principal Components
5. Creating Haptic Versions of 3-D Objects
6. An Exemplar Application: Bayesian Inference of Image Category
Note that the denominator in the two equations is the same. Therefore, to compare p(C = K | F) and p(C = L | F), it is not necessary to compute the denominator; rather, it is sufficient to compute the quantities
p(C=K|F) ∝ p(C=K)p(F|C=K)
p(C=L|F) ∝ p(C=L)p(F|C=L)
These are sometimes called 'unnormalized probabilities'. The term p(C) is called 'prior', and the term p(F | C) is called 'likelihood'.
VM can be used to generate a virtually limitless supply of novel 3-D shapes. Some exemplar digital embryos generated using the VM algorithm are shown in the bottom panel of Figure 1. Each of these 16 embryos were generated by using the 'growEmbryos.exe' program in the Digital embryo tools for Cygwin (see Table 1) for 40 growth cycles. All other growth parameters were set internally by the program. Most of these parameters were constant (i.e., identical from one embryo to the next). A few parameters such as the location and strength of morphogen sources, were random parameters set internally by the program independently for each run. The shape variations among these 16 embryos arose solely as a result of the variations in these random parameters.
Some examples of surface texturing34,35 using some arbitrarily chosen textures are shown in Figure 2A. Visual scenes of arbitrary complexity can be created using a commercially available 3-D modeling and rendering environment, as shown in Figure 2B.
A representative 'family tree' generated by VP using digital embryos is shown in Figure 3. Comparable trees can also be constructed using objects other than digital embryos, as shown in Figure 4. Note that in either case, the objects that share a common ancestor straightforwardly constitute a category, although the experimenter may also choose to define a category as any other set of objects. It is worth noting from Figure 4 that our current implementation of the VM and VP algorithms tends to produce relatively smooth, curved surfaces, as opposed to jagged or flat objects. It is also worth noting that this is presumably the limitation of our implementation of these algorithms and not the algorithms themselves, since biolological processes can produce objects with flat surfaces and jagged outlines (e.g., rose leaf).
Figures 5 and 6 illustrate the typical results of two methods that can be used in addition to, or instead of, VP for creating principled variations in object shape and object categories.
The top panel of Figure 7 illustrates visual renderings of two digital embryos, and the bottom panel of Figure 7 illustrates the corresponding printouts generated by a commercially available 3-D prototyper.
Figures 8 and 9 illustrate the procedures described in Section 6 for using image fragments to categorize a given visual object.
Figure 1. Virtual morphogenesis. The bottom panel illustrates a type of novel, naturalistic, virtual 3-D objects called "digital embryos"14. Digital embryos can be generated by simulating one or more of some of the key processes of biological embryogenesis: morphogen-mediated cell division, cell growth, cell movement and programmed cell death7,8,36,37. Each run starts with an icosahedron (shown in the top panel), and generates a unique embryo, depending on the VM parameter settings (or the 'genotype') of that embryo. Thus, the 16 embryos in the bottom panel have different shapes, because they all have different genotypes. Note that simpler or more complex shapes can be generated as needed (e.g., to optimally stimulate neurons at a given level of the visual hierarchy) by manipulating the genotype of the embryo. All of the aforementioned embryogenetic processes except programmed cell death were simulated in generating the embryos shown. Simulated programmed cell death is especially useful for creating targeted indentations (not shown).
Figure 2. Creating visual stimuli using digital embryos. Like any virtual 3-D object, digital embryos can be graphically manipulated to create visual scenes of arbitrary complexity using any standard 3-D graphical toolkit. This figure illustrates some common manipulations. (A) The same digital embryo is textured using many different textures, and lit from an invisible light source at top left. (B) A camouflaged scene is created by resizing and re-orienting the digital embryo and digitally placing it against the same background it was textured with. The digital embryo can be found in 'plain view' in the lower right quadrant. For additional examples of visual stimuli created using digital embryos, see refs. 9,10,12-14,38.
Figure 3. Creating digital embryo categories using VP. The VP algorithm emulates biological evolution, in that in both cases, novel objects and object categories emerge as heritable variations accumulate selectively. At each generation Gi , selected embryos procreate, leading to generation Gi+1. The progeny inherit the shape characteristics of their parent, but accrue shape variations of their own (as determined by small variations in their genotype) as they develop. This figure shows a 'family tree' of three generations of descendants starting from a single common ancestor, an icosahedron. Note that, in this case, the shape complexity increases from the icosahedron to generation G1, but not from G1 onward. This is because increase in cell numbers (i.e., cell division) was allowed from the icosahedron to generation G1, but not from G1 onward. In general, cell division tends to increase shape complexity, whereas other morphogenetic processes such as cell movement and cell growth change shape without changing the overall complexity of the shape.
Figure 4. VP using virtual objects other than digital embryos. This figure helps illustrate the general principle that virtual objects other than digital embryos can be used as input to VP. The VP algorithm in its current form can operate on any virtual 3-D object whose surface consists solely of triangles. Generation G1 comprised of (from left to right) a gourd, diamond, face mask, apple, rock, and cactus. Note that the objects in generation G1 in this figure do not have a common ancestor, because VP does not require it. Objects in G2 and G3 represent the descendants of the rock in G1. No cell divisions were allowed in any generation, so that all shape variations arose solely from the movement and/or growth of the individual 'cells' of the given object.
Figure 5. Using morphing to create smooth variations in shape. Morphing involves taking two given objects (far left and far right embryo in this figure) and calculating the intermediate objects (intervening embryos) by interpolating between the corresponding vertices of the two designated objects. In the case shown, all vertices were interpolated using the same scalar factor, resulting in a linear morphing. However, it is also possible to morph the objects non-linearly (not shown). Morphing is computationally straightforward when there is an exact one-to-one correspondence between the vertices of two objects, as in the case shown. However it is possible, in principle, to morph between any two given virtual objects regardless of whether their vertices correspond exactly, although there is no unique principled method for doing so17,18.
Figure 6. Using principal components to create smooth variations in shape. (A) Average embryo. This embryo represents the arithmetic average of 400 embryos (200 each from categories K and L in Figure 3). Principal components were calculated as described in Step 4.3. Note that principal components represent mutually independent, abstract shape dimensions of the 400 embryos (not shown)25,26. 400 embryos yield 399 non-zero principal components25,26, which together account for all the variance, or the shape information, available collectively in the embryos. By convention, principal components are arranged in the decreasing order of their eigenvalues, or the proportion of the overall variance they explain25,26. In this case, the first two principal components respectively accounted for 73% and 19% of the shape information available in the 400 embryos. (B) Embryos that represent different weights (or more precisely, weighted eigenvalues) of Principal Component 1. The weights varied from +2 (far left) to -2 (far right) in equal steps of -0.2. (C) Embryos that represent different weights of Principal Component 2. The weights also varied from +2 (far left) to -2 (far right) in equal steps of -0.2. Note that manipulating principal components does not exclusively manipulate any given specific body part of the embryo (e.g., the wings of the embryo in the case shown). However, if necessary, body parts of virtual 3-D objects can be manipulated in any arbitrary user-defined fashion using most of the commercially available 3-D modeling environments (not shown).
Figure 7. Creating haptic objects. Virtual 3-D objects can be 'printed' as haptic objects using a standard, commercially available 3-D 'printer' or prototyper. This figure shows digital embryos rendered as visual objects (top row) or as the corresponding haptic objects (bottom row). The haptic objects shown in this figure were printed to be about 6 cm wide (scale bar = 1 cm), although the objects can be printed at much smaller or larger sizes.
Figure 8. A template for an example informative fragment. In this example, the template has a threshold of 0.69 associated with it.
Figure 9. A new image for which the object category is not known and needs to be determined.
Usefulness of VM and VP in Cognitive Science Research
We have previously described the usefulness of VM and VP in detaill9,10,12-14. Briefly, VM, especially the digital embryo methodology, is useful because it provides a principled and flexible method for creating novel, but naturalistic 3-D objects14. Similarly, VP provides a principled method of creating naturalistic categories9,10,12,13. It is worth noting that object categories generated by VP share many features with object categories in nature, including the fact that the categories tend to be hierarchical in nature, and the feature variations within and across categories arise independently of the experimenter and the algorithms for classifying them39.
Current Limitations and Future Directions
Three current limitations of our protocol and the directions for future work they suggest are particularly noteworthy: First, both VM and VP simulate biological processes. While we show that non-biological virtual objects can be used as substrates for these processes, the underlying processes are still biologically motivated. However, natural objects - biological and non-biological alike - undergo shape changes due to non-biological forces. For instance, rocks may change in shape due to geological processes such as erosion or sedimentation. New categories of rock may arise from other such geological processes. It should be relatively straightforward to incorporate these processes into the repertoire of available shape change algorithms.
The second major limitation of our protocol is that its current repertoire of dynamic shape changes is rather limited. It is desirable to incorporate a larger array of shape changes, such as biological motion, or motion due to external forces such as wind, water or gravity. We expect that it will be relatively straightforward to bring to bear the known computer animation algorithms to implement such dynamic shape changes.
The third major limitation of our protocol is that VM currently does not include many other known morphogenetic processes including, most notably, gastrulation36. It also fails to incorporate some known constraints, such as the fact that morphogenesis in plants is mediated entirely by growth, with little or no cell movement possible, because of the cell walls36. Similarly, VP does not include other known phylogenetic processes such as genetic drift40. Addressing these limitations would help greatly facilitate the use of our protocol in developmental, ecological and evolutionary simulations.
No conflicts of interest declared.
This work was supported in part by the U.S. Army Research Laboratory and the U. S. Army Research Office grant W911NF1110105 and NSF grant IOS-1147097to Jay Hegdé. Support was also provided by a pilot grant to Jay Hegdé from the Vision Discovery Institute of the Georgia Health Sciences University. Daniel Kersten was supported by grants ONR N00014-05-1-0124 and NIH R01 EY015261 and in part by WCU (World Class University) program funded by the Ministry of Education, Science and Technology through the National Research Foundation of Korea (R31-10008). Karin Hauffen was supported by the Undergraduate Research Apprenticeship Program (URAP) of the U.S. Army.
|Digital Embryo Workshop (DEW)||Mark Brady and Dan Gu||This user-friendly, menu-driven tool can be downloaded free of charge as Download 1 from http://www.hegde.us/DigitalEmbryos. Currently available only for Windows.|
|Digital embryo tools for Cygwin||Jay Hegdé and Karin Hauffen||This is a loose collection of not-so-user-friendly programs. They are designed to be run from the command-line interface of the Cygwin Linux emulator for Windows. These programs can be downloaded as Download 2 from http://www.hegde.us/DigitalEmbryos. The Cygwin interface itself can be downloaded free of charge from www.cygwin.com.|
|Autodesk 3ds Max, Montreal, Quebec, Canada||Autodesk Media and Entertainment||3DS Max||This is a 3-D modeling, animation and rendering toolkit with a flexible plugin architecture and a built-in scripting language. Available for most of the current operating systems.|
|MATLAB||Mathworks Inc., Natick, MA, USA||MATLAB||This is a numerical computing environment and programming language with many useful add-on features. Available for most of the current operating systems.|
|R statistical toolkit||R Project for Statistical Computing||R||Can be downloaded free of charge from http://www.r-project.org/. Available for most of the current operating systems.|
|OpenGL||Khronos Group||OpenGL||This cross-language, cross-platform graphical toolkit can be downloaded free of charge from www.opengl.org.|
|V-Flash Personal Printer||3D Systems Inc., Rock Hill, SC, USA||V-Flash||This is a good value for all 3-D printing applications described in this report. The print materials are also vended by 3D Systems, Inc. Less expensive models are available in open source form from RepRap (rapmanusa.com) and MakerGear. More expensive models (> $30 K) are available from Objet Geometries, 3DS Systems, Z-Corp, Dimension Printing etc.|
|TurboSquid.com||TurboSquid Inc., New York, LA||(various objects)||Various virtual 3-D objects can be downloaded from this site free of charge or for a fee.|
|Table 1. Table Of Specific Toolkits And Equipment.|