Furthermore, it is useful for multimedia documents to be able to contain information generated by other applications, such as CAD designs, simulation results, etc. There is a need for a mechanism that allows this information to be included in multimedia documents, and its interchange in an open environment.
This paper describes a mechanism that allows the open interchange of structured multimedia documents containing external application specific information (ASI). The mechanism proposed is a general interface which results from the study of the interaction between multimedia document structure and document content in the document processes. Complex multimedia documents consist of a document structure and a document content. The document structure may be more or less complex depending of the type of document. The content of the document may be simple, such as text, or complex, such as video.
This mechanism must consider the following aspects: pre-interchange processing; interchange; and post-interchange processing.
The multimedia documents may also contain an alternative representation of the external application information. This alternative representation should be a simple (more compatible) format that allows other recipients, that do not have this application, to process the multimedia document. For example, the alternative representation of a CAD design may be a raster picture. Using this alternative representation, a minimum of information is guaranteed to any recipient (1).
The document content varies from document type to document type. Multimedia documents contain many different and complex content types, such as video.
The structure and the content types used in a multimedia document may depend on the specific multimedia application, therefore the document format can be vendor-dependent. Each vendor application supports different content types. Some content types are commonly used, but real agreement does not exist regarding the storage format used for these content types.
For example, moving images is a typical content type in multimedia documents. It can be considered as an agreed content type. Nevertheless, the storage format varies from application to application (mpg, avi, etc.). None of these formats seems to be the perfect one, since all of them are still used. All these circumstances decrease the interchangeability of multimedia documents.
Sometimes, it is necessary to include information from other applications (CAD designs, 3-D Images, simulation results, etc.) into multimedia documents. Usually, this information is converted to one of the content types supported by the multimedia application, then it is included into the document.
This paper describes a general interface which allows the inclusion of external application specific information (ASI) into documents in an agreed way. This general interface (ASI interface) results from the study and generalization of the interaction between document structure and document content. This interface is also explained in [1].
For non-external information this interface exists implicitly in the multimedia document application.
The presence of a document structure and a document content, and the interaction between both, is illustrated in an example in Figure 1. This figure shows an example structure of a multimedia paper containing a heading section with the document title and author name, an abstract, several sections and an author information section, that contains a text description, a voice record and a picture.
The interaction interface between document structure and document content is also illustrated in Figure 1.
This paper discusses those aspects of the interaction most often found in multimedia document formats and applications.
These processes are explained in the following sections. Figure 2 shows a scenario where these processes are illustrated. System 1 creates the document, system 2 continues editing, then format it and prints a hard copy (presentation). After formatting, the document is sent to system 3 that presents it on a computer screen.
These processes model the whole document life from its creation to its presentation to the destination user. Furthermore, other actions can be performed on a document, for instance: manipulation of hard copies; re-use of some parts for other documents; references from other documents; destruction of the document.
Actions of these types are not considered in this paper.
The interchange process is described in Section 3.
Changing the order of two sections is a modification of the document structure, but the content is not modified. Replacing a picture by another one is a modification of the document content. Deleting a whole section modifies both structure and content.
Editing actions are performed on the structure and content. These actions can be extended to support editing actions on content with external information. This extension is provided by the general interface explained in Section 2.2.
After the document has been edited (system 1 in Figure 2), the document can be interchanged (as it is) for further editing by another author (system 2 in Figure 2), or it can be formatted for its presentation. After formatting, the document can be also interchanged. In the first case, the document is interchanged for "editing-only", i.e. the recipient continues to edit the document or format it (as illustrated in Figure 2). This document is not still prepared for presentation. Such documents are called processable documents, since they are intended to undergo further processing.
Generation of new content, such as section numbers. This content is created by the document formatting process. Determination of the area required for the presentation of each content element.
Assignment (if necessary) of time slices for the presentation of the content elements. For example, a paragraph explaining the content of a video sequence is to be presented (synchronized) with this video sequence. The time slice assigned to the paragraph is equivalent to the duration of the video sequence of the video content.
Redistribution of the content elements. A content element may be fragmented into several content elements if it cannot be formatted as a whole. For example, if the text of a content element cannot be formatted in only one page, this content element can be broken into two elements which are laid out on different pages.
Generation of the formatted version of the content to be presented. Some content types may require a formatted version for their further presentation. For example, formatting instructions are to be added to text content to present it in bold.
Generation of a layout structure, where content is assigned to different layout elements such as pages or blocks.
Only the document content is formatted. The document structure provides the guidelines for the layout of the document, but finally, only the document content is presented to the user. The document structure is not directly presented, but it has conditioned the content layout. For example, the content elements of Section 1 are presented before the content elements of Section 2 due to the document structure.
When the document formatting takes place, the document content is laid out following the restrictions provided by the document structure. In this way, there is an interaction between document structure and document content in the formatting process, for instance, if a video sequence is to be displayed synchronized with a title text (like a figure caption). In the layout process, the screen may be filled with other content elements, there being enough space to present the video sequence, but not the title. Then the video and title text are presented in the next screen. In this way, the document structure constraints the content layout.
There is a clear interaction between the layout of the document structure and the layout of the document content. This interaction should be extended for external application information.
After the document has been formatted, it can be interchanged as a "presentation-only" document (intended only for presentation), as a "presentation-editing" document (intended for further processing or presentation) or as a processable document. In the first two cases, the information generated in the document formatting is also interchanged.
The presentation of a multimedia document consists in its representation in a human-perceptible form (physical, visual, audible, etc.). This presentation process is to be extended for external application information.
The presentation process is application- and hardware-dependent.
Each document process is to be extended to handle external information. Therefore, the ASI interface provides groups of operations corresponding to these processes. See [1] for more details.
Figure 3 describes the groups of operations supported by the ASI interface.
It is assumed that we are working with a multimedia document format which allows its interchange. The best approach is to use a standard interchange format. The ASI interface extends the interchange features to also allow the interchange of external application information with the document.
The following aspects are to be considered in the interchange of multimedia documents containing external information:
It is supposed that the document recipient can process documents in the format generated by the originator. There must be an interchange format to be used, which also specifies what types of content are supported.
The document originator cannot assume that the recipient is able to process a particular external application information. The recipient is only assumed to be able to process documents following the interchange format, and, optionally, be able to support the ASI interface mechanism.
In an open interchange, the originator is responsible for including the necessary information into the document for processing by the recipient.
To avoid this problem, an alternative representation of the external application information can also be included in the document. This alternative representation must be a representation in a content type supported by the document interchange format.
If the recipient cannot process the application specific information, the alternative representation can be used in the layout and imaging processes. The ASI interface provides the functionality required to generate this alternative representation.
In the first case, the external information is included (directly or indirectly by external references) into the document. In the second case, the external information is converted to a content type supported by the multimedia application. Conversion normally implies information loss. Furthermore, conversion is usually in one direction, i.e. after a conversion, the original data cannot be reconstructed as they were before conversion.
Automatic update of documents when the original external application information is modified can be performed better if the data is included (or referenced), and not converted. Example: Conversion or inclusion of a CAD design.
A CAD design may be converted to a graphics format, such as CGM (Computer Graphics Metafile). CAD designs contain more information than graphics. Therefore, the conversion of CAD information into CGM implies information loss. A CAD design may be reconstructed from a CGM graphic, but it would not be the same CAD design. The conversion process is not reversible.
Furthermore, if the CAD design is included into the document, a document receiver that a good CAD tool can make better use of the CAD design than of the CGM.
Two types of references can be distinguished:
Example: Inclusion of information from a STEP database.
A company stores the manufacturing data in a STEP [2] database. A product report has been generated. This report contains references to the STEP database from the ASI content. The report is to be sent to the sales offices. The database is not accessible for the sales offices, the data referenced from the database are included in the report before it is sent to the offices. The internal report continues using references to the STEP database. In this way, when the database is updated, the document is also automatically updated.
Each document format can provide the adequate mechanisms to guarantee the security aspects in document interchange. However, if external application information is included in multimedia documents, some new aspects are to be considered:
A hospital publishes a study about an illness evolution. The study is created using the patient records stored in the hospital database. The study contains ASI that references the patient records. The study is sent to universities for educational purposes. Some fields of the patient records are confidential and cannot be distributed. Only the non-confidential fields of the patient records can be included into the study before it is sent to the universities.
The external specific application that generates the information should provide security mechanisms. The document interchange format may also incorporate security aspects, but they cannot always be extended to the external application information. In these cases, the ASI interface could be extended to support security extensions.
Using this philosophy it is possible to guarantee the integrity of the external information inside documents in which integrity cannot be guaranteed.
All the security aspects must be solved before the document is interchanged.
The alternative representation is the main mechanism to guarantee an open interchange, since the document originator cannot assume that the document recipient can process this type of external information.
The content types supported by the document in-terchange format and the type of external information both constrain the alternative representation used.
In those cases where security is required, the alternative representation can contain the public version of the external information in a known format.
Document standards, such as ODA[3], SGML[4] or HyTime[5], offer the best possibilities to achieve a real open document interchange. Proprietary formats are used inside organizations and companies, but it is recommended to use a standardized format to achieve an effective open interchange. Multimedia documents may be converted from the proprietary format to a standardized format. Standard formats may not offer so many content types as proprietary formats. Therefore, the ASI interface mechanism proposed in this paper can also be used, considering some content types as application specific information.
The recipient can perform another conversion of the interchanged document, from the interchange format to another proprietary format. In this way, an open interchange is achieved. Conversions may imply information losses, but conversions between document formats is not as difficult as conversion between application information.
The ODA standard defines an architectural model for documents, which consist of a document structure and document content, and an interchange format.
A document may have a logical structure, a layout structure or both. The logical structure defines the document from a logical point of view, structuring the content into logical structure elements such as sections or paragraphs.
The layout structure describes the document from a formatted point of view, defining structure elements like pages, frames and blocks.
The document content is stored in content portions, which are related to the document structure. The base standard allows the inclusion of character, raster graphics, geometric graphics and audio as internal content architectures. ODA does not actually restrict the inclusion of other external data, but it does not provide any processing model for them. Therefore, the ASI interface is defined to extend the ODA processing model for external information.
ASI is contained inside an ASI-specific content portion, where the information is stored as an ASN.1 octet string [6][7], and is therefore transparent for ODA.
The operations described in 2.2.1 are also considered in the ASI interface for ODA documents.
The ASI interface should provide the extension of the layout process to ASI. The operation described in 2.2.2 should be provided by the ASI interface.
The ODA ASI interface should provide the presentation operation to extend the standard ODA presentation process for ASI.
The ODA standard specifies also a mechanism to define alternative representation of content portions. This mechanism can be used for the alternative representation to ASI.
For interchange purposes, the ODA ASI interface specifies the following operations:
The ASI interface for the ODA standard has been presented in EWOS (European Workshop for Open Systems) [9]. EWOS is responsible ,together with other regional workshops, for developing International Standardized Profiles (ISPs) in open systems. The specification of the ASI interface and how to develop profiles based on ASI, has been approved as an EWOS Technical Guide [10] and will be considered in the development of future ISPs.
The ASI interface mechanism will be used to develop profiles for ODA that are related to other applications involving multimedia document interchange, such as medical applications, EDI [11] applications, STEP [2] Applications, etc.
The main objective of this interface is the interchange of multimedia documents containing ASI.
The definition of the ASI interface for ODA documents has been discussed. This work has been considered in EWOS (European Workshop for Open Systems), and will be applied in future document interchange. The ideas presented may be extended to other multimedia document interchange formats, especially for standardized interchange formats.
Table 1: Operations defined for the ODA ASI interface
Process Operation Description ------------------------------------------------------------------------- Create creates a content portion containing ASI Editing Copy copies an ASI content portion Delete deletes an ASI content portion Layout Layout performs the layout process of the ASI content portion Presentation Imaging extends the imaging process of the ODA standard Interchange GenerateInt generates an interchangeable version of the ASI GenerateODA generates an ODA compatible version of the ASI
Jaime Delgado has a Ph.D. in Telecommunications Engineering since 1987. He is Professor at the Computer Architecture Department of the Technical University of Catalunya (UPC). He has led several European R&D; projects in the field of Information Technology, and has also advised the European Commission in this area. He is currently heavily involved in standardization work, being editor of several international standards on distributed applications and document handling and interchange. He is editor of several parts of the ODA standard in ISO/IEC and ITU-T, and of several profiles on ODA, DTAM and DFR (in EWOS, European Workshop for Open Systems). He has also led a Project Team in ETSI (European Telecommunications Standard Institute) on coop-erative document handling applications.