Last update at : Wed May 3 14:37:08 1995

Interchange of multimedia documents containing external information

Interchange of multimedia documents containing external information

Jose Jesus Acebron
Jaime Delgado


There are many multimedia document appli-cations using different proprietary storage and interchange formats. The great variety of content types included in multimedia documents decreases the interchange capabilities between different systems not using the same proprietary document applications and formats.

Furthermore, it is useful for multimedia documents to be able to contain information generated by other applications, such as CAD designs, simulation results, etc. There is a need for a mechanism that allows this information to be included in multimedia documents, and its interchange in an open environment.

This paper describes a mechanism that allows the open interchange of structured multimedia documents containing external application specific information (ASI). The mechanism proposed is a general interface which results from the study of the interaction between multimedia document structure and document content in the document processes. Complex multimedia documents consist of a document structure and a document content. The document structure may be more or less complex depending of the type of document. The content of the document may be simple, such as text, or complex, such as video.

This mechanism must consider the following aspects: pre-interchange processing; interchange; and post-interchange processing.

The multimedia documents may also contain an alternative representation of the external application information. This alternative representation should be a simple (more compatible) format that allows other recipients, that do not have this application, to process the multimedia document. For example, the alternative representation of a CAD design may be a raster picture. Using this alternative representation, a minimum of information is guaranteed to any recipient (1).

1 Introduction

This paper describes a mechanism that allows the open interchange of structured multimedia documents containing external application specific information (ASI).The mechanism (ASI interface) is described as the generalization of the document processes. The interchange of multimedia documents containing ASI is described in detail. The application of these concepts and ideas on a document standard (ODA) is also presented.

2 Including external information into multimedia documents

Multimedia documents consist of a document structure and a document content. The document structure is more or less complex depending on the document type. For instance, complex multimedia documents are much more structured than office documents.

The document content varies from document type to document type. Multimedia documents contain many different and complex content types, such as video.

The structure and the content types used in a multimedia document may depend on the specific multimedia application, therefore the document format can be vendor-dependent. Each vendor application supports different content types. Some content types are commonly used, but real agreement does not exist regarding the storage format used for these content types.

For example, moving images is a typical content type in multimedia documents. It can be considered as an agreed content type. Nevertheless, the storage format varies from application to application (mpg, avi, etc.). None of these formats seems to be the perfect one, since all of them are still used. All these circumstances decrease the interchangeability of multimedia documents.

Sometimes, it is necessary to include information from other applications (CAD designs, 3-D Images, simulation results, etc.) into multimedia documents. Usually, this information is converted to one of the content types supported by the multimedia application, then it is included into the document.

This paper describes a general interface which allows the inclusion of external application specific information (ASI) into documents in an agreed way. This general interface (ASI interface) results from the study and generalization of the interaction between document structure and document content. This interface is also explained in [1].

For non-external information this interface exists implicitly in the multimedia document application.

The presence of a document structure and a document content, and the interaction between both, is illustrated in an example in Figure 1. This figure shows an example structure of a multimedia paper containing a heading section with the document title and author name, an abstract, several sections and an author information section, that contains a text description, a voice record and a picture.

The interaction interface between document structure and document content is also illustrated in Figure 1.

This paper discusses those aspects of the interaction most often found in multimedia document formats and applications.

2.1 Multimedia document processes

If we analyze the actions performed on a multimedia document during its whole life, the following processes are carried out on this document: editing; formatting; interchange; presentation.

These processes are explained in the following sections. Figure 2 shows a scenario where these processes are illustrated. System 1 creates the document, system 2 continues editing, then format it and prints a hard copy (presentation). After formatting, the document is sent to system 3 that presents it on a computer screen.

These processes model the whole document life from its creation to its presentation to the destination user. Furthermore, other actions can be performed on a document, for instance: manipulation of hard copies; re-use of some parts for other documents; references from other documents; destruction of the document.

Actions of these types are not considered in this paper.

The interchange process is described in Section 3.

2.1.1 The editing process

The document editing process creates or modifies the document. This editing can affect the document structure, the document content or both.

Changing the order of two sections is a modification of the document structure, but the content is not modified. Replacing a picture by another one is a modification of the document content. Deleting a whole section modifies both structure and content.

Editing actions are performed on the structure and content. These actions can be extended to support editing actions on content with external information. This extension is provided by the general interface explained in Section 2.2.

After the document has been edited (system 1 in Figure 2), the document can be interchanged (as it is) for further editing by another author (system 2 in Figure 2), or it can be formatted for its presentation. After formatting, the document can be also interchanged. In the first case, the document is interchanged for "editing-only", i.e. the recipient continues to edit the document or format it (as illustrated in Figure 2). This document is not still prepared for presentation. Such documents are called processable documents, since they are intended to undergo further processing.

2.1.2 The formatting process

When the document editing is finished, it should be prepared for presentation. The formatting (or layout) of the multimedia document implies: Figure 2: Document processes in document life (image missing)

Generation of new content, such as section numbers. This content is created by the document formatting process. Determination of the area required for the presentation of each content element.

Assignment (if necessary) of time slices for the presentation of the content elements. For example, a paragraph explaining the content of a video sequence is to be presented (synchronized) with this video sequence. The time slice assigned to the paragraph is equivalent to the duration of the video sequence of the video content.

Redistribution of the content elements. A content element may be fragmented into several content elements if it cannot be formatted as a whole. For example, if the text of a content element cannot be formatted in only one page, this content element can be broken into two elements which are laid out on different pages.

Generation of the formatted version of the content to be presented. Some content types may require a formatted version for their further presentation. For example, formatting instructions are to be added to text content to present it in bold.

Generation of a layout structure, where content is assigned to different layout elements such as pages or blocks.

Only the document content is formatted. The document structure provides the guidelines for the layout of the document, but finally, only the document content is presented to the user. The document structure is not directly presented, but it has conditioned the content layout. For example, the content elements of Section 1 are presented before the content elements of Section 2 due to the document structure.

When the document formatting takes place, the document content is laid out following the restrictions provided by the document structure. In this way, there is an interaction between document structure and document content in the formatting process, for instance, if a video sequence is to be displayed synchronized with a title text (like a figure caption). In the layout process, the screen may be filled with other content elements, there being enough space to present the video sequence, but not the title. Then the video and title text are presented in the next screen. In this way, the document structure constraints the content layout.

There is a clear interaction between the layout of the document structure and the layout of the document content. This interaction should be extended for external application information.

After the document has been formatted, it can be interchanged as a "presentation-only" document (intended only for presentation), as a "presentation-editing" document (intended for further processing or presentation) or as a processable document. In the first two cases, the information generated in the document formatting is also interchanged.

2.1.3 The presentation process

The formatted document which results from the layout process can be presented to the user (systems 2 and 3 in Figure 2).

The presentation of a multimedia document consists in its representation in a human-perceptible form (physical, visual, audible, etc.). This presentation process is to be extended for external application information.

The presentation process is application- and hardware-dependent.

2.2 The external information interface

This section provides a description of the application specific information (ASI) interface which results from the extension of the document processes for external application information.

Each document process is to be extended to handle external information. Therefore, the ASI interface provides groups of operations corresponding to these processes. See [1] for more details.

Figure 3 describes the groups of operations supported by the ASI interface.

2.2.1 Editing operations

Operations provided for extending the editing process:
Create: creates a piece of document content for ASI.
Copy: copies a content piece with ASI.
Delete: deletes an ASI. Dependencies and references are to be considered.
Complex operations can be performed as combinations of these simple operations.

2.2.2 Layout operations

There is one interface operation to perform the layout of the ASI:
Layout: lays out the ASI content constrained to the resources provided by the main document layout process.
If the ASI layout fails, the alternative representation (see 3.1) can be used.

2.2.3 Presentation operations

There is one interface operation to perform the presentation of the ASI:
Presentation: presents the ASI content following the restrictions of the main document presentation. This operation uses the resources of the specific application (to which the data pertains).
If the presentation of the ASI fails, the alternative representation (see 3.1) can be used.

2.2.4 Interchange operations

The interface provides two operations dealing with document interchange:
GenerateInt: generates an interchangeable version of the ASI.
GenerateAlt: generates an alternative version of the ASI (see 3.1).

3 Interchange of multimedia document containing external information

The interchange of multimedia documents containing external information is the most important aim of the ASI interface.

It is assumed that we are working with a multimedia document format which allows its interchange. The best approach is to use a standard interchange format. The ASI interface extends the interchange features to also allow the interchange of external application information with the document.

The following aspects are to be considered in the interchange of multimedia documents containing external information:

It is supposed that the document recipient can process documents in the format generated by the originator. There must be an interchange format to be used, which also specifies what types of content are supported.

The document originator cannot assume that the recipient is able to process a particular external application information. The recipient is only assumed to be able to process documents following the interchange format, and, optionally, be able to support the ASI interface mechanism.

In an open interchange, the originator is responsible for including the necessary information into the document for processing by the recipient.

3.1 Alternative representation to external information

In an open multimedia document interchange, the recipient of the document should be able to process the received document. If documents contain external application information not specified by the interchange format, this open information interchange is no longer possible.

To avoid this problem, an alternative representation of the external application information can also be included in the document. This alternative representation must be a representation in a content type supported by the document interchange format.

If the recipient cannot process the application specific information, the alternative representation can be used in the layout and imaging processes. The ASI interface provides the functionality required to generate this alternative representation.

3.2 Document pre-interchange processing

Before a multimedia document containing external information is interchanged, some actions are to be performed on this document. The following sections discuss some particular aspects of these actions.

3.2.1 Advantages of inclusion versus conversion

External application information can be incorporated into multimedia documents in two ways: information inclusion and information conversion.

In the first case, the external information is included (directly or indirectly by external references) into the document. In the second case, the external information is converted to a content type supported by the multimedia application. Conversion normally implies information loss. Furthermore, conversion is usually in one direction, i.e. after a conversion, the original data cannot be reconstructed as they were before conversion.

Automatic update of documents when the original external application information is modified can be performed better if the data is included (or referenced), and not converted. Example: Conversion or inclusion of a CAD design.

A CAD design may be converted to a graphics format, such as CGM (Computer Graphics Metafile). CAD designs contain more information than graphics. Therefore, the conversion of CAD information into CGM implies information loss. A CAD design may be reconstructed from a CGM graphic, but it would not be the same CAD design. The conversion process is not reversible.

Furthermore, if the CAD design is included into the document, a document receiver that a good CAD tool can make better use of the CAD design than of the CGM.

3.2.2 External references

External application information can also consist of external references to information outside the document.

Two types of references can be distinguished:

Publicly accessible references, which may be accessed by any document recipient.
Specific references, which are accessible for the document originator, but not necessarily for the document recipient.
In the first case, the references can be interchanged as they are within the document. In the second case, the references shall be solved before interchange. Solving references may consist of either including the information into the document or converting this reference to a publicly accessible reference.

Example: Inclusion of information from a STEP database.

A company stores the manufacturing data in a STEP [2] database. A product report has been generated. This report contains references to the STEP database from the ASI content. The report is to be sent to the sales offices. The database is not accessible for the sales offices, the data referenced from the database are included in the report before it is sent to the offices. The internal report continues using references to the STEP database. In this way, when the database is updated, the document is also automatically updated.

3.2.3 Security

Security aspects play an important role in multimedia document interchange, where similar problems to other data interchange arise, such as integrity, confidentiality, certification, etc.

Each document format can provide the adequate mechanisms to guarantee the security aspects in document interchange. However, if external application information is included in multimedia documents, some new aspects are to be considered:

Absolute confidential external information: the external application information included into the multimedia document may be confidential for a certain environment. This information cannot be open interchanged with the document. It should be removed or enciphered.
Partly confidential external information: some parts of the external application information should not be interchanged. The external information is to be transformed (filtered) before it is interchanged. The confidential parts may be removed or enciphered.
Integrity: in some cases, the integrity of the external information must be guaranteed.
Example: Inclusion of patient records in a health care study.

A hospital publishes a study about an illness evolution. The study is created using the patient records stored in the hospital database. The study contains ASI that references the patient records. The study is sent to universities for educational purposes. Some fields of the patient records are confidential and cannot be distributed. Only the non-confidential fields of the patient records can be included into the study before it is sent to the universities.

The external specific application that generates the information should provide security mechanisms. The document interchange format may also incorporate security aspects, but they cannot always be extended to the external application information. In these cases, the ASI interface could be extended to support security extensions.

Using this philosophy it is possible to guarantee the integrity of the external information inside documents in which integrity cannot be guaranteed.

All the security aspects must be solved before the document is interchanged.

3.2.4 The alternative representation

The alternative representation of the external application information must be generated before the document is interchanged.

The alternative representation is the main mechanism to guarantee an open interchange, since the document originator cannot assume that the document recipient can process this type of external information.

The content types supported by the document in-terchange format and the type of external information both constrain the alternative representation used.

In those cases where security is required, the alternative representation can contain the public version of the external information in a known format.

3.3 Multimedia document interchange

To achieve a successful multimedia document interchange there must exist an agreement on the interchange format used between document originator and recipient. This interchange format may be proprietary, publicly available or standardized. None of them are beyond the scope of this paper, since the ideas presented can be applied to any document format.

Document standards, such as ODA[3], SGML[4] or HyTime[5], offer the best possibilities to achieve a real open document interchange. Proprietary formats are used inside organizations and companies, but it is recommended to use a standardized format to achieve an effective open interchange. Multimedia documents may be converted from the proprietary format to a standardized format. Standard formats may not offer so many content types as proprietary formats. Therefore, the ASI interface mechanism proposed in this paper can also be used, considering some content types as application specific information.

The recipient can perform another conversion of the interchanged document, from the interchange format to another proprietary format. In this way, an open interchange is achieved. Conversions may imply information losses, but conversions between document formats is not as difficult as conversion between application information.

3.4 Document post-interchange processing

The received document must be processed to determine aspects such as:
Presence of external information and types of external information. If the contained external information cannot be processed, the alternative representation is to be used.
Security aspects: if enciphered document parts cannot be deciphered, they can be removed.
Resolution of external references: the received document may contain references which are to checked for consistency, and then solved.
The recipient may perform a conversion from the interchange format to the proprietary format used.

4 Inclusion of ASI into ODA documents

The ASI interface presented in this paper has been applied for documents following a standard. ODA (Open Document Architecture) [3] has been chosen. The same ideas may be applied to other document standards such as SGML [4] or HyTime [5].

The ODA standard defines an architectural model for documents, which consist of a document structure and document content, and an interchange format.

A document may have a logical structure, a layout structure or both. The logical structure defines the document from a logical point of view, structuring the content into logical structure elements such as sections or paragraphs.

The layout structure describes the document from a formatted point of view, defining structure elements like pages, frames and blocks.

The document content is stored in content portions, which are related to the document structure. The base standard allows the inclusion of character, raster graphics, geometric graphics and audio as internal content architectures. ODA does not actually restrict the inclusion of other external data, but it does not provide any processing model for them. Therefore, the ASI interface is defined to extend the ODA processing model for external information.

ASI is contained inside an ASI-specific content portion, where the information is stored as an ASN.1 octet string [6][7], and is therefore transparent for ODA.

4.1 ODA document processes

The document processes described in the ODA standard are similar to those described in 2.1. The particular aspects related to the ODA processes are explained in the following subsections.

4.1.1 The ODA editing process

The ODA standard does not provide any strict process model for document editing, since the ODA standard is defined for document description and interchange. Part 3 of standard [8] specifies an interface for the manipulation of ODA documents, which is related to document editing.

The operations described in 2.2.1 are also considered in the ASI interface for ODA documents.

4.1.2 The ODA layout process

The standard defines the layout process in detail. ODA formatted documents may also be interchanged. The layout process is defined for the standardized content types, but it does not provide any specification about the layout of external data.

The ASI interface should provide the extension of the layout process to ASI. The operation described in 2.2.2 should be provided by the ASI interface.

4.1.3 The ODA presentation process

The ODA standard defines a presentation process, which is called the imaging process. Document presentation is not closely related to document interchange. Presentation is application- and hardware-dependent. ODA, as a standard, does not specify any strict description of the imaging process.

The ODA ASI interface should provide the presentation operation to extend the standard ODA presentation process for ASI.

4.1.4 Document interchange in ODA

The ODA standard defines ODIF (Open Document Interchange Format), an interchange format which gives a system-independent encoding of the document in the Abstract Syntax Notation ASN.1[6] [7].

The ODA standard specifies also a mechanism to define alternative representation of content portions. This mechanism can be used for the alternative representation to ASI.

For interchange purposes, the ODA ASI interface specifies the following operations:

4.2 The ODA ASI Interface

The operations defined for this interface are described in Table 1.

The ASI interface for the ODA standard has been presented in EWOS (European Workshop for Open Systems) [9]. EWOS is responsible ,together with other regional workshops, for developing International Standardized Profiles (ISPs) in open systems. The specification of the ASI interface and how to develop profiles based on ASI, has been approved as an EWOS Technical Guide [10] and will be considered in the development of future ISPs.

The ASI interface mechanism will be used to develop profiles for ODA that are related to other applications involving multimedia document interchange, such as medical applications, EDI [11] applications, STEP [2] Applications, etc.

5 Conclusions

This paper has presented a mechanism that allows the inclusion of application specific information into multimedia documents. This mechanism is designed as an interface that can be defined for each structured multimedia document interchange format.

The main objective of this interface is the interchange of multimedia documents containing ASI.

The definition of the ASI interface for ODA documents has been discussed. This work has been considered in EWOS (European Workshop for Open Systems), and will be applied in future document interchange. The ideas presented may be extended to other multimedia document interchange formats, especially for standardized interchange formats.

Table 1: Operations defined for the ODA ASI interface

Process		 Operation		Description
		Create		creates a content portion containing ASI
Editing		Copy		copies an ASI content portion
		Delete		deletes an ASI content portion

Layout		Layout		performs the layout process of the ASI 
				content portion

Presentation	Imaging		extends the imaging process of the ODA standard

Interchange	GenerateInt	generates an interchangeable version of the ASI

		GenerateODA	generates an ODA compatible version of the ASI


  1. J.J. Acebron and J. Delgado, "Handling and interchange of structured multimedia documents containing application-specific information", Proceedings of the 6th Joint European Networking Conference (JENC6), 1995.
  2. ISO 10303. "Standard for the Exchange of Product Model Data (STEP)", 1994.
  3. ISO/IEC 8613 | ITU-T T.410 Series of Recommendations. "Open Document Architecture (ODA) and Interchange Format", 1994.
  4. ISO 8879. Information Processing - Text and Office Systems - "Standard Generalized Markup Language (SGML)", 1986.
  5. ISO/IEC 10744, Information Technology - "Hypermedia/Time-based Structuring Language (HyTime)", 1992.
  6. ISO/IEC 8824 | CCITT Rec. X.208. "Abstract syntax notation one (ASN.1)", 1990.
  7. ISO/IEC 8825 | CCITT Rec. X.209. "Basic encoding rules for abstract syntax notation one (ASN.1)", 1990.
  8. ISO/IEC 8613-3 | ITU-T Recommendation T.413. "Open Document Architecture (ODA) and Interchange Format - Abstract interface for the manipulation of ODA documents", 1995.
  9. J.J. Acebron, J. Delgado, "Including Application Specific Information into ODA Documents: The Application Specific Information Interface", EWOS TA/94/033, 1994.
  10. J.J. Acebron, J. Delgado, "EWOS Technical Guide: Including Application Specific Information into ODA Documents", EWOS SMMI/95/086, EWOS TA/95/092, 1995.
  11. ISO 9735. "Electronic Data Interchange For Administration, Commerce and Trade (EDIFACT)", 1990.

Author Information

Jose Jesus Acebron is a researcher at the Computer Architecture Department of the Technical University of Catalunya (UPC) in the research program of the Generalitat de Catalunya (Programa F.I.). He graduated as a Telecommunications Engineer (UPC, 1992) and is currently working to obtain his Ph.D. His interests are centered on open systems interconnection, open document interchange and multimedia structured documents.

Jaime Delgado has a Ph.D. in Telecommunications Engineering since 1987. He is Professor at the Computer Architecture Department of the Technical University of Catalunya (UPC). He has led several European R&D projects in the field of Information Technology, and has also advised the European Commission in this area. He is currently heavily involved in standardization work, being editor of several international standards on distributed applications and document handling and interchange. He is editor of several parts of the ODA standard in ISO/IEC and ITU-T, and of several profiles on ODA, DTAM and DFR (in EWOS, European Workshop for Open Systems). He has also led a Project Team in ETSI (European Telecommunications Standard Institute) on coop-erative document handling applications.

(1) This work has been partly supported by the Spanish Government (TIC92-1487-CE) and the Generalitat de Catalunya (programa de formació d'investigadors).