Oscar Firschein, Visiting Scholar (previously Manager, ARPA Image Understanding Program)

Seminar abstract

Specialized applications involving image and text databases can sometimes use techniques not available to general image-text database systems. An unusual indexing mechanism, the "site model," for an intelligence application was described. A site model is a stored 3-D model of a region of intelligence interest, such as a port or a military base. If images and reports are keyed to parts of a site model, it is possible to use the site model as an indexing mechanism for both images and text. For example, one might ask for recent intelligence reports related to a feature of the site or for images that show a particular roof or a parking area. The talk described the ARPA-sponsored RADIUS project that uses the site model approach, and indicated the database aspects of the system. RADIUS uses semi-automated and automated image understanding techniques to aid in image exploitation. The ultimate goal is to provide the IA with a seamless system capable of examining images and answering queries in a database. Several novel technical approaches have been used in RADIUS:
  • Model-supported exploitation provides the key to change detection, and also serves as an indexing mechanism to both image and text databases.
  • The context-based approach to image understanding explicitly represents the assumptions made by each algorithm, and uses the context of the current IA task to select the most appropriate algorithms for solving that task.

    Introduction

    Image exploitation is carried out by an intelligence analyst (IA) who reviews images taken from satellites or aircraft using a variety of sensors. The IA determines significant changes at a geographic site of intelligence interest, follows and reports on events such as movement of military forces, follows trends, and monitors and counts objects such as vehicles in a parking area or fuel drums at a loading dock. The IA often uses other information, such as maps, previous IA or intelligence reports, and previous imagery, and other collateral information, to make these determinations.
    While previously image exploitation was carried out by analyzing film products on a light table, there is increasing use of "softcopy," i.e., images in digital form that are analyzed on softcopy workstations using computer-based techniques. Because of increasingly effective sensor platforms, the growing volume of intelligence imagery is overloading the IA, leading to unreviewed images or lack of timely analysis. RADIUS is designed to alleviate this problem.

    A Future Scenario

    Although the current RADIUS system does not have the capability to carry out the following scenario, the scenario serves as a good indication of what the next generation of RADIUS might provide:
    The IA logs onto the system in the morning. Overnight, the system has reviewed the new imagery and provides the IA with the following information ordered in perceived importance:
  • Event. Site 325 armor movement. Location:Radistan
  • Count. Site 5948 amount of petrol exceeded Location: Lallia
  • Change: Site 978 excavation Location: Trufis
  • New: Site 1342 unidentified object Location: Brazistof The first item indicates that an important event, armor movement, has occurred. The second item indicates that a count of petroleum drums that has been carried out over the past 6 months has now exceeded the threshold alarm value. The next item is a change to a site that has been detected, namely an excavation. Finally, the last item indicates an unidentified object at a site. The IA can now take action on these indicators, reviewing the imagery, calling up further collateral, and making new tasking assignments of the system.

    Automatic Selection of Image Exploitation Algorithms

    The analyst must be able to perform the usual image manipulation tasks such image rotation, scaling, and enhancement as well as the registration of two or more images. In addition to these image manipulation capabilities, the analyst must have easy access to both image and textual databases, and the ability to use image understanding algorithms to aid in the exploitation.
    A basic problem in providing computer aid to the IA for image exploitation is the proper selection of an IU algorithm and parameters to carry out a particular task. Without the proper settings, the algorithm may work poorly or not at all. Because the IA cannot be expected to understand the characteristics of each IU algorithm, an interface must be provided that enables the IA to easily specify what he wants the system to do. This specification is then used to select the algorithm and the appropriate parameter settings. The basic approach used is to represent explicitly the assumptions made by each algorithm, and to use the context of the present task to select the most appropriate algorithms for solving that task. The mechanism for selection of algorithms and parameters can be a table of context items to be satisfied versus appropriate algorithms for each row of the table. A more efficient mechanization is to express the table in PROLOG rules and have the PROLOG system solve for the appropriate algorithms and settings.
    This approach to algorithm and parameter selection is of great general interest and can be used whenever an unskilled user must utilize sophisticated algorithms. The basic requirement is that each algorithm be labeled with the context required by the algorithm and the paarameter settings for that context.

    Database Aspects of RADIUS

    The site model can be used as the basic indexing mechanism for both text and imagery. The user can indicate a location in a site model to select an object of interest and can then indicate whether text or imagery related to that object is desired. In addition, the IA can set a trigger on an object in the site model that will automatically alert the analyst when a new image arrives that satisfies that trigger. The triggers are stored in the site model database for each site.
    For example, the IA might set a trigger on a parking lot and task the image understanding system to count vehicles in that parking lot. When a new image arrives for that site, the parking lot is analyzed and vehicles counted. When a threshold number of vehicles is exceeded, the system can notify the IA. Similarly, triggers can be set for new construction, new structures on a roof, etc. Thus, the following sequences of actions occur when a new image arrives:
  • The system first registers the image to the appropriate site model so that each object in the image is identified.
  • Then the site model database triggers assign IU algorithms to analyze the appropriate portion of the new image.
  • The image understanding algorithm carries out the analysis, and detections or counts are reported to the IA.
    Because an important question of interest is "when did a change at a site first take place?" it is important that the database time-stamp all versions of a site.

    Summary

    The RADIUS database ties together both text and imagery so as to provide the IA a seamless system that can manipulate, register, and exploit imagery, and can retrieve text information relevant to the site objects. The site model concept offers a powerful approach for any system that must deal with images and text related to relatively fixed three-dimensional objects.

    Reference

    Papers on RADIUS can be found in the Proceedings of ARPA Image Understanding Workshop, 1995, 1995, and 1996, published by Morgan Kaufmann Publishers, 340 Pine St., San Francisco, CA 94104-3205.