Visualizing Video Streams

Using Sand Glass Metaphor

Edward Chang and Yung-Hsiang Lu
Stanford University
echang@CS.Stanford.EDU
December, 1996

Abstract

Many visualization tools are developed to visualize spatial information. This study examines alternatives for visualizing spatio-temporal relationships, in particular, flows. A flow consists of an object or objects moving in time from one spatial location to another. Multimedia stream delivery is a good example of a flow. In this article, we propose using the sand glass as a visualization metaphor for viewing video streams. Using color and sound in a computer representation of a sand glass, the static and dynamic, as well as macro and micro, aspects of a flow can be easily modeled and vividly presented. Finally we show an implementation of the sand glass metaphor on top of a multimedia storage system simulator to visualize different aspects of video stream delivery. We show that sand glass is indeed a better visualization metaphor that brings forth more insights and intuition about the system.

1. Introduction

The ability to measure time accurately over very long periods and very short intervals has been and remains crucial to the development of modern science. In additional to standardizing three naturally occurring units of time-measurement: the day, month and year, we have invented hour, minute, second, and smaller time scales for scientific measurement. In modern computer science, we measure Input/Output (IO) time, memory access time, and processor speed, in milli, micro, and nano seconds.

Related dimension of scientific measurement is space. Spatial attributes such as distance and volume must be measured with precision. We can also measure space in time or time in space. For example, measuring distance traveled in time by an object, we can compute its average speed. On the other hand, space can be used to record or represent time. The almanac shown in Figure 1, called a clog almanac, is a piece of 3D boxwood. with the times of the season changes carved on it.

Figure 1. A 1896 Clog Almanac [Tuner 90].

Perhaps most important is the ability to discern spatial patterns in time. The discovery of gravity, the shape of earth, and many other natural laws, all resulted from the discovery of certain spatio-temporal relationships. Likewise, many algorithms that assist our daily life, for example, in traffic control, inventory management, investment planning, and the internal operations of computers, involve tradeoffs of space and time. To discover and gain insight into spatio-temporal relationships, we need the assistance of intuitive tools for spatio-temporal visualization.

This study focuses on a particular spatio-temporal relationship - the flow. A flow consists of an object or objects moving in time from one spatial coordinate to another. For instance, river is a flow of water on the ground; clouds are water streams in the sky. In our culture, we have also created traffic flows, data flows, and the like. This study uses multimedia data streams as the flow example. We describe the characteristics of a video stream, and show how sand glass, as a visualization metaphor, provides more insights into the video bit streams.

The rest of this paper is organized as follows. Section 2 describes the characteristics a video data flow. Section 3 examines some possible visualization metaphors for flows. Section 4 proposes using sand glass as the visualization metaphor. Finally, Section 5 shows an implementation that uses sand class metaphor to visualize different video stream delivery algorithms.

2. Video Data Streams

In a multimedia delivery system, data is first read from secondary storage systems such as disks into main memory for each of the streams requested. The data for a stream is read into a buffer for that stream, which must be adequate to sustain the stream until its next segment is read. To analyze the performance of a multimedia delivery system, we typically are given the following parameters:

In addition, a multimedia delivery system has the following tunable parameters. They can be adjusted, within certain bounds, to optimize system throughput.

In a feasible system, the amount of data retrieved in a period, S, must be at least as large as the amount of data displayed. That is, S >= DR * T. However, if we want a stable system, the input rate should equal the output rate to avoid accumulating more and more data in memory with every period. Thus, we have the equation S = DR * T.

Figure 2. Memory Requirement of Stream [Chang 96B].

To visualize this system, we can use a two-dimensional plot. Figure 3 shows that the horizontal axis is time, and the vertical axis is the amount of data in the buffer for a stream. An IO starts shortly before the data staged into memory in the previous period is used up. The data accumulates in memory at the rate of TR - DR until the IO is complete. The data is then consumed at the rate of DR. This periodic behavior repeats until the playback ends, as shown on the right side of the figure.

Although the figure conveys information better than the text, the presentation is static if not dull. In fact, time is treated as another spatial dimension in most presentations. Although this results from the limitations of the presentation media such as papers and terminal screens, we believe there are more intuitive ways to show spatio-temporal relationships. This belief has prompted us to search for visualization metaphors that better represent flows.



3. Visualization Metaphors

This section studies the relationships of volume and time requirements in flows of data and material by representing those flows as the action of clocks measured by water, candles, sticks of incense, and sand.

Figure 3. A Chinese Water Clock [ILin 73].

A Water Clock

Figure 3 shows a water clock. Not so very long ago there were still some of these old-fashioned water clocks to be seen in China. Four big copper kettles were arranged, one below the other, on the steps of a stone staircase. The water ran from one vessel down into the next. Every two hours the guard hung out a little sign board showing what ``tee'' (2-hour unit) had passed. Since water needed to be refilled every now and then, water clocks were considered inconvenient. Nevertheless, as a visual metaphor, the water clock shows not only time, but also the relationship between volume and time.

Using the water clock as a visualization metaphor, we can view memory as a copper kettle that contains data. The narrow vessel that connects the kettles is the the data consumption channel. Adjusting the throttle of the vessel changes the data flow rate. A multi-staged water clock is a good representation of the data flow in the networks. Each kettle can be treated as a router's buffer. The data flows into a kettle and is then distributed to the downstream kettles.

Although the water metaphor is good for representing flows, it is not suitable for presenting a mixture of information. For many simultaneous streams flowing in the multimedia delivery system, for example, it is hard to attach attributes, such as colors and textures, to water to distinguish individual objects. When a yellow data flow is mixed with a blue one, water makes both flows green. Thus, the water metaphor can obscure micro information.

A Candle Clock

Figure 4. A Coil Clock [Tuner 90].

An alternative metaphor to the water clock is the candle clock. A candle also shows the volume and time relationship well. Small candles can be colored to represent each individual object, while a large candle can represent aggregate information. The burning rate of a candle is the flow rate. The candle clock, however, fails to convey the feeling of flowing. Even though a candle is consumed continuously, the sense of its consumption is rather discrete, and constant too. Moreover, the notion of ``replenishment'' does not pertain to candle. We simply replace candle when it is burnt out.

An Incense Clock

The incense clock is a variation on the candle clock. Incense sticks are well suited for time measurement, because they burn constantly, at an even rate, and without flame. Since it is flameless, it is safer than candle. But for the purpose of visualization on a computer screen, safety is not a convincing feature. Therefore, we do not consider incense clock a suitable metaphor.

A Sand Clock

The idea of a sand clock is the same as water clock. Sand is filled into a glass, flows through a narrow channel, and reaches another glass. Sand glass was and is still popular for measuring short time intervals (e.g., cooking eggs). The reason that sand clock is used only for short time interval measurement is because the glass is too fragile to contain a large quantity of sand. Fortunately, a sand glass metaphor never breaks. We find sand glass a good visualization metaphor for flow control applications for the following reasons:

4. A Magic Sand Glass

We have argued that sand glass is a good visualization metaphor to view flows. In this section, we describe in detail how to use this metaphor to represent complex flows.

Figure 5. Sand Clocks

Creating Sand Glass

Input

Let's start simple by using one sand glass to represent one flow. A sand glass a situated in a 3D space. The top chamber represents the source of the flow, the bottom the destination, and the the channel in between the throttle that controls the flow rate. The viewer can assign any functions (i.e., continuous or discrete, deterministic or stochastical) to the source of the flow, which can be refilled via a throttle, and the throttle itself. The user can also assign any color to the sand. The viewer can decide the size of the top and bottom glass based on the capacity. The glass can be of any shape.

Output

Data are collected and statistics are computed at three points: source, throttle, and destination. The viewer may want to associate addition functions and add additional data collection points to the sand glass. Typical information the viewer may be interested in includes average flow rate, maximum flow rate, percentage of loss, minimum source and destination glass size to hold the sand, and so on

Multi-Stage Sand Glass

A sand glass can have more than two stages as illustrated in the picture below. The shape of the glass can be molded to any way the viewer see it fits. The flow can be bi-directional. The viewer is free to zoom in any stage to get a micro view.

Hierachy of Sand Glasses

Aggregated information can be represented by a hierarchy of sand glasses. For instance, the data flow for all streams in a system can be represented with a top level glass mixed with many colors of sand, each represents an individual stream. While viewing the data flow of the entire system, the view can click on any colors to view a group of streams or an particular stream. This is also applicable to viewing a manufacturing shop floor. The supervisor can view the material flows on the entire floor, or select a particular production line to zoom in.

Representing the Flow in A Sand Glass

The viewer should have different ways to view a sand glass. Here we present a few.

Sand Pies/ Grids

To view the composition of sand, the viewer can display a sand pie or sand grid. In addition to showing the percentage of a particular object in the glass, a sand pie shows dynamically the growth and shrink of the object. A sand grid can show the object's location in the glass. For instance, in inventory control or memory management systems, it is often interesting to find out the location of an object or a class of objects, and probably their migration pattern in the 2D or 3D space, to devise better space allocation strategies.

Sand Trees

If the information in a glass has a tree structure, sand tree can be built. This can be a cone-tree, or just a 2D tree. Techniques such as fisheye views can be used to see the objects with greater importance. The viewer can traverse a branch to see smaller view or back out to get a macro perspective. To measure the performance of a particular network protocol, for instance, one can have one color group represent ATM packets, and another color group for TCP/IP. The viewer can see the data flow in the networks as a whole, or take at look at the performance of each protocol. The tree can be rebuilt and colors can be reassigned to see, for instance, the transmission rate in different period of time. Sand tree is a way to categorize information for presentation, and is orthogonal to the topology of the sand glass.

Representing Speed with Sound and Color

The speed of a flow may be difficult to show in the screen. We suggest that the speed of flow can be represented by sound or colors. The user has freedom to associate a piece of music or notes to a particular speed of interest. The viewer can also assign colors to different speed. The throttle can display color or play notes according to the rate of the flow.

5. Implementation

To verify if our sand glass proposal works, we implement a visualization tool on top of a multimedia storage system simulator to compare the memory requirements and the worst startup latency of two video stream delivery algorithms. Without describing the stream delivery algorithms in details (Interested readers please consult references [Chang 96A] and [Chang 96B]), we are interested in getting the following information:

In the macro level, we would like to find out the maximum total memory space required by each algorithm. We are also interested in the variance of the memory requirement, so that we can decide if the memory requirement can be slightly less than the peak and not to jeopardize the real-time delivery constraint of the media data. In addition, we want to find out how different page sizes affect the memory requirement.

In the micro level, we want to ensure that the data supply for each stream is neither inadequate that causes ``jitter'', nor excessive that requires too much memory resource. Another important information is the startup delay of a request. Startup delay, we mean the time between a request arrives at the storage system, and the time the data is available in memory for display.

A Conventional 2D plot

Figure 6. 2D Plot One
Figure 6 shows a conventional 2D plot. To provide rich information in limited space, we aggregate four 2D plots in one canvas. The main plot is the memory requirement in time. The memory requirement increases as requests arrive, until the disk bandwidth saturates. The red line indicates the peak memory of the system under the employeed disk scheduling and memory management policies (in this example, the policy is named Sweep).

At the bottom we plot the average memory requirement. This is used together with the plot on the right hand side, the memory requirement distribution, to determine if the system can be configured with a memory capacity less than the peak memory required. Figure 6 reveals that the peak memory is required, although not constantly, in a regular manner. Therefore, having less memory capacity than the peak requirement can cause regular jitter on the display devices. Finally, the right bottom corner shows a summary including the peak memory requirement and total number of ``jitter''.

Figure 7. 2D Plot Two

Figure 7 shows the memory requirement affected by the page size. Due to the quantization effect and internal fragmentation, the memory requirement increases as the page size increases from 4K to 32KBytes.

This 2D plot successfully displays the memory requirement and the pattern in time. Also, it reveals information such as the memory requirement versus page size. It is also capable of showing if memory or disk bandwidth is the resource bottleneck. However, information pertaining to the flow such as data replenishment and consumption speed, and the profile of the requests are difficult to tell. Attempts to change the 2D plot to incorporate new information such as the profile of the media data in memory would mess up this 2D plot. Therefore, we use a sand glass to compensate for the missing information.

A Macro Sand Glass

Figure 8. Sand Glass Macro View
Figure 9 shows three 2D views of a 3D sand glass. The first view on the left shows the total memory requirement. The level of sand in the glass reveals if the memory requirement exceeds the capacity (the capacity of the glass). The upper right pie chart shows the profile of the media data in memory. In this example we show the media currently being played back consists of three different movies. One can classify the data in other ways, such as by data compression formats (MPEG or JPEG), by the reguests' playback startup time (peak hour, off-peak hour), and so on so forth. These dynamic information is important to determine the hot spots (popular movies, for example) so that proper treatments can be applied. For instance, if a movie is in high demand, and the startup time of the movies are close to each other, batching can be applied to reduce memory requirement. Another example is that if a movie is in high demand, the movie can be either cached in memory or replicated on the idle disks to increase the service rate. These dynamic information is difficult to obtain from the conventional 2D plot.

Finally, the rectangular shape at the right bottom describes the composition of the data flowing through the throttle. The size of the rectangle signifies the data flow rate, and each color can represent the flow rate of each movie, or data into a particular network router. This provides information for congestion control so that problem can be understood and alleviated with promptly.

In short, the macro sand glass complements the 2D plot in providing real-time information to make prompt management decisions related to load balancing and congestion control, while the 2D plot provides a historical prospective to assist the evaluation and comparison of different memory and disk management policies.

Micro Sand Glasses

Figure 9. Sand Glass Macro View
So far we have shown the macro view of the system. However, detailed information such as if and where ``jitter'' occurs, and the startup latency of a request cannot be told from the macro view. Figure 9 shows a snapshot of an animation in progress. The level of the sand in the glass represents the amount of data available for the request to playback. If ``jitter'' occurs, we show red in the sand glass. This particular micro view provides not only the memory requirement information, but also if the memory allocation and data replenishment strategies have been working properly. In addition, it also reveals if memory is over-committed. For instance, when we visualize Sweep algorithm through micro sand glasses, we discover that the data are always plenty in all sandglasses. This means the memory allocated may be larger than necessary, and hence further optimization can be applied.

Another interesting information is the startup delay of a request. From the micro view, we can tell the time between a sand glass is created and the the first time the sand is filled into the glass. This time difference is the startup delay of the request.

Summary

We show that using sand glass metaphor we could see both macro and micro information of a video streams in a more dynamic and intuitive fashion. This is not to say we abandon the conventional 2D plot entirely. On the contrary, both approaches are valuable and can complement each other very well.

Finally, picking cone as the shape of the glass distorts the real volume information from the viewers. When a sand glass is half full, the volume of sand is probably less than a third or a quarter full depends on the angle of the cone shape. However, we feel this is precisely the reason why sand glass is a good visualization metaphors: it makes the viewer feel the urge and urgency of replenishment. This sense of urgency grows more than linearly as the level of sand becomes lower. The actual volume information can be supplemented by the 2D plot.

6. Final Remarks

In this study, we propose using sand glass as a visualization metaphor to view information associated with flows. We show that sand glass is intuitive to represent volume versus time relationship. We verify our idea through implementing a video stream visualization tool that provides much more insightful information into the video data flows in both macro and micro levels.

References

  • [Ilin 73] M. Ilin, What Time Is It, George Routledge & Sons LTD., 1973.
  • [Tuner 90] A.J. Tuner, Time, Tijd voor Tijd Foundation, 1990.
  • [Chang 96A] E. Chang, H. Garcia-Molina, "Reducing Initial Latency for Multimedia Storage Systems," To appear in IEEE Multimedia , A Special Issue on Multimedia Database Systems.
  • [Chang 96B] E. Chang, H. Garcia-Molina, "Minimize Memory Use in Video Servers," Stanford Technical Report, Oct. 1996.
  • [Kimelman 94-2] Kimelman, D., " Reduction of visual complexity in dynamic graphs ," Graph Drawing. DIMACS International Workshop, GD'94. Proceedings.
  • [Liu 96] Liu, Z., "A System for Visualizing and Animating Program Runtime Histories," Proceedings 1996 IEEE Symposium on Visual Languages
  • [Stasko 93] Stasko, J.T., "Three-dimensional computation visualization ," Proceedings 1993 IEEE Symposium on Visual Languages p. 100-7
  • [Tufte 83] Tufte, Edward R., " The Visual display of quantitative information ," Graphics Press, 1983.
  • [Lamping 96] Lamping, J., " The hyperbolic browser: a focus+context technique for visualizing large hierarchies ," Journal of Visual Languages and Computing (March 1996) vol.7, no.1, p. 33-55.
  • [Michail 96] Michail, A., " Teaching Binary Tree Algorithms Through Visual Programming ," Proceedings 1996 IEEE Symposium on Visual Languages