We proposed a system that uses inexpensive, noninvasive pseudo-acoustic optical sensors to automatically and accurately detect, count, and classify flying insects based on their flying sound.
An inexpensive, noninvasive system that could accurately classify flying insects would have important implications for entomological research, and allow for the development of many useful applications in vector and pest control for both medical and agricultural entomology. Given this, the last sixty years have seen many research efforts devoted to this task. To date, however, none of this research has had a lasting impact. In this work, we show that pseudo-acoustic optical sensors can produce superior data; that additional features, both intrinsic and extrinsic to the insect’s flight behavior, can be exploited to improve insect classification; that a Bayesian classification approach allows to efficiently learn classification models that are very robust to over-fitting, and a general classification framework allows to easily incorporate arbitrary number of features. We demonstrate the findings with large-scale experiments that dwarf all previous works combined, as measured by the number of insects and the number of species considered.
The idea of automatically classifying insects using the incidental sound of their flight dates back to the earliest days of computers and commercially available audio recording equipment1. However, little progress has been made on this problem in the intervening decades. The lack of progress in this pursuit can be attributed to several related factors.
First, the lack of effective sensors has made data collection difficult. Most efforts to collect data have used acoustic microphones2-5. Such devices are extremely sensitive to wind noise and ambient noise in the environment, resulting in very sparse and low-quality data.
Second, compounding these data quality issues is the fact that many researchers have attempted to learn very complicated classification models, especially neural networks6-8. Attempting to learn complicated classification models, with a mere tens of examples, is a recipe for over-fitting.
Third, the difficultly of obtaining data has meant that many researchers have attempted to build classification models with very limited data, as few as 300 instances9 or less. However, it is known that for building accurate classification models, more data is better10-13.
This work addresses all three issues. Optical (rather than acoustic) sensors can be used to record the “sound” of insect flight from meters away, with complete invariance to wind noise and ambient sounds. These sensors have allowed the recording of millions of labeled training instances, far more data than all previous efforts combined, and thus help avoid the over-fitting that has plagued previous research efforts. A principled method is shown below that allows the incorporation of additional information into the classification model. This additional information can be as quotidian and as easy-to-obtain as the time-of-day, yet still produce significant gains in accuracy of the model. Finally, it is demonstrated that the enormous amounts of data we collected allow us to take advantage of “The unreasonable effectiveness of data”10 to produce simple, accurate and robust classifiers.
In summary, flying insect classification has moved beyond the dubious claims created in the research lab and is now ready for real-world deployment. The sensors and software presented in this work will provide researchers worldwide robust tools to accelerate their research.
1. Insect Colony and Rearing
2. Record Flying Sounds in Experimental Chambers
3. Sensor Data Processing and Detection of Sounds Produced by Flying Insects
4. Insect Classification
Two experiments are presented here. For both experiments, the data used were randomly sampled from a dataset that contains over 100,000 objects.
The first experiment shows the ability of the proposed classifier to accurately classify different species/sexes of insects. As the classification accuracy depends on the insects to be classified, a single absolute value for classification accuracy will not give the reader a good intuition about the performance of the system. Instead, rather than reporting the classifier’s accuracy on a fixed set of insects, the classifier was applied to datasets with an incrementally increasing number of species, and therefore increasing classification difficulty.
The dataset began with just 2 species of insects; then at each step, one more species (or a single sex of a sexually dimorphic species) was added and the classifier was used to classify the increased number of species (the new dataset). A total of ten classes of insects (different sexes from the same species counting as different classes) were considered, with 5,000 exemplars in each class.
The classifier used both insect-sound (frequency spectrum) and time-of-intercept for classification. Table 1 shows the classification accuracy measured at each step and the relevant class added at that step.
According to Table 1, the classifier achieves more than 96% accuracy when classifying no more than 5 species of insects, significantly higher than the default rate of 20% accuracy. Even when the number of classes considered increases to 10, the classification accuracy is never lower than 79%, again significantly higher than the default rate of 10%. Note that the ten classes are not easy to separate, even by human inspection. Among the ten species, eight of them are mosquitoes, with six of them being from the same genus.
The second experiment is to show how accurately the system can sex flying insects, specifically, to distinguish male Ae. aegypti mosquitoes from females. For the first part of the experiment, assume that the “misclassification cost” of misclassifying males as females is the same as the cost of misclassifying females as males. With this assumption, the classification results are shown in Table 2.I. The classification accuracy to sex Ae. aegypti is about 99.4%.
For the second part of experiment, assume the cost is not asymmetric, i.e., misclassification of females as males is much more costly than the reverse. With this assumption, the decision threshold of the classifier was changed to reduce the number of high-cost misclassifications. With the threshold properly adjusted, the classification results in Table 2.II were achieved. Of 2,000 insects in the experiment, twenty-two males, and zero females were misclassified.
Figure 1. (I) One of the experimental cages used to gather the data. (II) A logical version of the sensor setup with the components annotated. Please click here to view a larger version of this figure.
Figure 2. (I) An example of a 1 sec audio clip containing a “flying insect sound” generated by the sensor. The sound was produced by a female Cx. stigmatosoma. The insect sound is highlighted in red/bold. (II) The “insect sound”, which has been cleaned and saved into a 1-sec long audio clip by centering the insect signal and padding with 0s elsewhere. (III) The frequency spectrum of the insect sound, obtained using Discrete Fourier Transform. Please click here to view a larger version of this figure.
Figure 3. A Bayesian network that uses a single feature for classification.
Figure 4. A Bayesian network that uses two independent features for classification
Figure 5. The circadian rhythms of Cx. stigmatosoma (female), Cx. tarsalis (male), and Ae. aegypti (female), learned based on observations generated by the sensor that were collected over 1 month duration. Please click here to view a larger version of this figure.
Figure 6. A Bayesian network that uses three independent features for classification.
Figure 7. The assumptions of the geographic distributions of each insect species and sensor locations in the simulation to demonstrate the effectiveness of using location-of-intercept feature in classification. Please click here to view a larger version of this figure.
Figure 8. The general Bayesian network that uses n features for classification, where n is a positive integer. Please click here to view a larger version of this figure.
Step | Species Added | Classification Accuracy | Step | Species Added | Classification Accuracy | |
1 | Ae. aegypti ♂ | N/A | 6 | Cx. quinquefasciatus ♂ | 92.69% | |
2 | Musca domestica | 98.99% | 7 | Cx. stigmatosoma ♀ | 89.66% | |
3 | Ae. aegypti ♀ | 98.27% | 8 | Cx. tarsalis ♂ | 83.54% | |
4 | Cx. stigmatosoma ♂ | 97.31% | 9 | Cx. quinquefasciatus ♀ | 81.04% | |
5 | Cx. tarsalis ♀ | 96.10% | 10 | Drosophila simulans | 79.44% |
Table 1. Classification accuracy with increasing number of classes.
Predicted class | Predicted class | |||||||
I (Symmetric cost) | female | male | II (Asymmetric cost) | female | male | |||
Actual class | female | 993 | 7 | Actual class | female | 1,000 | 0 | |
male | 5 | 995 | male | 22 | 978 |
Table 2. (I) The confusion matrix for sex discrimination of Ae. aegypti mosquitoes with the decision threshold for females is set at 0.5 (i.e., same cost assumption). (II) The confusion matrix of sex the same mosquitoes, with the decision threshold for females set at 0.1.
The sensor/classification framework described here allows the inexpensive and scalable classification of flying insects. The accuracies achievable by the system are good enough to allow the development of commercial products and could be a useful tool in entomological research.
The ability to use inexpensive, noninvasive sensors to accurately and automatically classify flying insects would have significant implications for entomological research. For example, by deploying the system in the field to count and classify insect vectors, the system can provide real-time counts of the target species, producing real-time information that can be used to plan intervention/suppression programs to combat malaria. Moreover, the system can automatically separate insects by sex, and thus it can be used to free entomologists working on the Sterile Insect Technique15 from the tedious and time-consuming task of manually sexing the insects.
In using this system, the most critical step is to properly set up the sensor for data collection. If the laser and the photo array are not properly aligned, the data will be very noisy. After the insects are placed in the cage, the photo array should always be fine-tuned using the magnets on the outside of the cage. Note that flashing lights, camera flashes and vibrations near the cages will introduce noise to the data. Therefore, to obtain clean data, place the cage in a dark room, and wherever necessary, place dry towels under the cages in order to diminish the level of vibration.
The classifier presented in this work used just two additional features. However, there may be dozens of additional features that could help improve the classification performance. As the potential features are domain and application specific, the users could choose features based on their specific need or applications. The general framework of the classifier allows users to easily add features to the classifier to improve the classification performance.
To encourage the adoption and extension of our ideas, we are making all code, data, and sensor schematics freely available at the UCR Computational Entomology Page16-17. Moreover, within the limits of our budget, we will continue our practice of giving a complete system (as shown in Figure 1) to any research entomologist who requests one.
The authors have nothing to disclose.
We would like to thank the Vodafone Americas Foundation, the Bill and Melinda Gates Foundation, and the São Paulo Research Foundation (FAPESP) for funding this research. We would also like to thank the many faculty members from the Department of Entomology at University of California, Riverside, for their advice on this project.
Name of Material/ Equipment | Company | Catalog Number | Comments/Description |
Audio Recorder: ICD-PX312 | Sony | 4-267-065-11(2) | With a 8 GB microSD extra memory |
Insectary | Lee's Aquarium & Pet Products. | 20088 HerpHaven®, Large Rectangle | 14 1/2" Long x 8 3/4" Wide x 9 3/4" high. Modified to house insects. |
Laser Line Generator, 650nm (red) | Apinex (www.apinex.com) | LN60-650 | 5mW. This is a low powered laser, similar to a teachers lasers pointer |
Photodiode Array | VISHAY SEMICONDUCTOR TEFD4300 PIN PHOTODIODE, 650NM, 20DEG, T1 | TEFD4300 | We made a custom array of 15 of these Photodiodes wired in parallel |
Analogue to Digital Convertor Integrated Circuit | Custom made in our lab | We made this item ourselves, but an easily available commercial product, Gino PCF8591 AD/DA Converter, provides the same functionality. |