Historical Perspective for the Foundation of Internet-Based EDI
Kenneth W. COPELAND <firstname.lastname@example.org>
C. Jinshong HWANG <email@example.com>
This paper continues the work of the authors in addressing Electronic Data Interchange (EDI) and its implementation on the Internet that was started by the publication of two papers, "Electronic Data Interchange: Concepts and Effects" and "Third Generation Web Applications, Full Service Intranets, EDI -- The Fully Integrated Business Model for Electronic Commerce," which were presented at INET'97 and published in the proceedings of that conference.
This paper takes a look at the protocol and specification work done by the first EDI working group of the Internet Engineering Task Force (IETF). The work is significant both from the aspect that a start was made in marrying the two technological areas (EDI and the Internet) and from the aspect that the working group died from the inability to make progress on the second of the two deliverables, the EDI usage document.
The EDI working group of the IETF started their e-mail list in December of 1993 and the discussion of EDI on the Internet began with a bang. The first IETF meeting on EDI was held as a Birds of a Feather (BOF) session at the March 1994 meeting in Seattle, Washington. The EDI BOF decided that the initial work that needed to be accomplished was the Multipurpose Internet Mail Extensions (MIME) Content-Type definitions and an EDI-over-the Internet Usage document. A clear consensus was obtained to pursue a charter and continue work as a formally chartered working group.
The original charter called for two deliverables: "specification for the carriage of various EDI content via MIME-based e-mail, and a discussion document, considering issues in the use of EDI over the Internet. The usage document will cover such issues as addressing and security."
The working group decided that two other items were important: 1) specification of EDI routing information and 2) specification of mappings between Internet-based and X-400-based EDI. These two items would, however, be deferred and recommended to be done at a later time.
Over the course of the next 22 months, the working group did produce a specification for encapsulating EDI within MIME objects. This was the only technical specification produced and although technically trivial, was completely successful and is now a standard. Agreement could not be reached on content for a usage document, however, and this document never was produced.
By April 1995, the work on the usage document had made little progress and this lack of progress eventually doomed the working group to be disbanded. A second informational document entitled "EDI meets the Internet: Frequently Asked Questions about Electronic Data Interchange (EDI) on the Internet " was produced as an informational request for comments (RFC) and final publication occurred in January 1996.
In December of 1993, an e-mail list was started by the Internet Engineering Task Force (IETF) to ascertain if enough interest existed in the Internet community to attempt to design protocols for the carriage of Electronic Data Interchange (EDI) objects over the Internet. Although the general topic of EDI was thought to be arcane, the volume on the e-mail list indicated a strong interest in this topic.
The interested parties met as a Birds of a Feather (BOF) session at the March 1994 meetings of the IETF in Seattle, Washington. It was apparent from the start that there was enough interest to proceed on to working-group status and obtain a charter for the work to be done. It was also apparent that the two communities had collided and that each had its own way of attending to business and even determining what that business entailed.
Two documents were included in the original charter of the EDI working group. The first document was to be a specification for the carriage of EDI by MIME-based e-mail. The second document was to be an informational request for comments (RFC) discussing issues in the use of EDI over the Internet. As will be seen, the working group was successful in detailing the specification of the carriage of EDI objects via MIME. However, the working group floundered badly when attempting to solve the usage issues. The Electronic Data Interchange -- Internet Integration (EDIINT) working group (chartered later) did successfully tackle many of the usage questions and produced a Requirements Document for accomplishing interoperable EDI over the Internet. The differences in the efforts were scope and focus. The scope of the usage document attempted by the EDI working group was broader than what was taken on by the EDIINT working group. Additionally, the EDIINT working group was able to focus more successfully on key issues singly and serially.
EDI is a well-developed and time-tested technological area that has been in existence for approximately the same amount of time as the Internet, both having been born in the middle to late 1960s. The EDI community is large, diverse, and well established. The initial collision of the two communities was loud and mostly unproductive from the standpoint of standards production. Both communities have long-standing ways of doing things and the differences in the viewpoints were initially too great to produce what the working group was chartered to accomplish.
EDI is a set of protocols for conducting highly structured transactions between business entities (trading partners) by sending rigidly structured messages that can be operated upon without the need for human interaction. The initial purpose of the EDI working group that was chartered by IETF was to produce specifications for the use of EDI standards over the Internet. The initial effort was restricted to focus on the transport of EDI via Internet e-mail.
Two deliverables were delineated in the original charter:
It was the second document that was never completed, and the inability of the working group to make sufficient progress on these issues was the reason for the dissolution of the working group before it had completed its charter.
IETF is in the business of producing standards that can be used to enhance and increase the performance and capabilities of communications across the Internet. The IETF standards process is, however, different from traditional standards processes. One of the basic differences is that IETF standards have a very narrow and limited scope. These standards are meant to do a specific function. The implementation of the standard and the interoperability of differing implementations are accomplished before the standard can even be put into draft status. Although the specificity can be broadened or functionality added to existing standards, each function is narrowly defined and handled as a specific goal for the working group that is handling it. Most other standards processes are general in nature, including a wide berth of functions and capabilities, in an attempt to solve all problems that can be envisioned. Then, when the standard is produced, it is up to individual vendors to implement the standard or not, depending on the marketplace and the complexity of the standard involved.
The general idea at IETF is that we need protocols that will allow all users everywhere to communicate and interact with each other. If we build these protocols on open standards that have already been proven, then de facto technology that is already interoperable becomes the basis for all Internet standards.
Requiring interoperability between or amongst two or more participating implementations of the standard before a standards track proposal can even be submitted into draft status makes this process different from other standards processes. This requirement means that every draft standard is already implemented and interoperable across the Internet. Traditional standards processes of many other organizations, where the standards are developed broadly and attempt to solve all problems at once, generally make the standard bulky and difficult to implement. Since the various implementations of the standard do not have to be tested for interoperability, even if there are implementations, they tend to be proprietary.
A BOF mailing list was created in December 1993 with an open invitation to all who were interested in helping to develop a standardized way to use the Internet for EDI.
The general feeling of the initial BOF group was that a working group could be established that had the desire, skill, and time to pursue the topic and that usage of EDI over the Internet could be standardized in a manner beneficial to the entire EDI community.
This was the first meeting of the working group and it was well attended with 42 participants present. The fact that the meeting occurred at all after three months of e-mail list work meant that there was indeed interest in proposing standards for doing electronic commerce over the Internet.
The group had already experienced much head-butting and name-calling as the two different communities tried to merge. The fact that there were members on both sides who had much experience in their own areas of expertise and were unwilling to bend or come to a consensus led to some hard feelings even in the initial encounters.
Four items of work had been identified through the mailing list:
It was finally decided that the first two items would require immediate attention and items three and four would be deferred to a later date.
In assessing the interest of the IETF community in pursuing this initiative and becoming a chartered working group, factors such as the history of the e-mail list, which was extensive, and the attendance and participation in the BOF meeting were considered. It was apparent that even though the topic of EDI might be of no interest to the general Internet community, there was certainly enough interest to continue the work.
It was at this meeting that the agreement was reached and taken forward to the e-mail list to name three MIME content types: EDI-X12, EDIFACT, and EDI-consent. The EDI-consent tag was decided upon to cover all non-X12 and non-EDIFACT EDI sent between trading partners at their agreement and discretion.
The EDI working group held two meetings in Toronto. The first reviewed work to-date on the specification for encapsulating EDI within MIME objects. The focus was agreed upon to be the carriage of EDI objects through simple encapsulation of the EDI objects in MIME. Three MIME objects had been defined: EDI-X12, EDIFACT, and EDI-consent.
In preparation for starting a paper on the use of EDI through the Internet, various requirements and concerns were discussed among the participants. This surfaced a desire for a separate discussion paper concerning use of MIME. Later in the first meeting, Walt Houser gave a brief presentation about the US Electronic Commerce Acquisition Team effort to establish an Electronic Commerce architecture in the federal government.
On the second day, work was begun to formulate the structure of the usage document, and writing assignments were made. Due dates for first drafts were set for August.
The EDI in MIME specification had been technically stable now for over six months. The only real changes to be made in the existing document were the movement of EDIFACT to first position and X12 to the second position. No other national variants of EDI standards would be addressed.
The only discussion about the document involved the question of security mechanisms and the need for user solutions to the security problems. Arguments were raised as to whether security should be implemented by built-in EDI security mechanisms. It was finally decided that no security mechanism would be selected or championed, but that the real security needs would be addressed in the security portion of the document.
The working group did not meet in Danvers. RFC 1767 "MIME Encapsulation of EDI Objects" had already been published as a proposed standard. The lack of progress on the usage document threatened to shut down the working group since the original charter could not be met. It was decided that the working group would be shut down if drafts of the usage and Frequently Asked Questions (FAQ) documents were not published prior to the 33rd IETF, which was to meet in Stockholm in July 1995.
The working group concluded its work and was disbanded prior to this meeting. The usage document was abandoned and a FAQ document was pending publication as an informational RFC.
The working group was already disbanded. The FAQ was published as an RFC the following month.
Much discussion centered on whether or not the transport mechanism should look inside or determine anything based on data inside the EDI object. It was pointed out that, traditionally, in the case of letters and carriers, the carrier does not open mail.
The final document left the EDI object to be what is handed over by the EDI translator. At that point MIME encapsulation according to the specification occurs and the SMTP packaging follows the MIME encapsulation. The mail is then submitted and delivered across the Internet and at the receiving end the SMTP and MIME packaging is stripped off. Then, what is handed to the EDI translator at the receiving end is what came from the EDI translator at the sending site.
This standards track document contains three distinct categories as content-types. EDIFACT is the first content type and is the EDI standard used primarily in Europe. X12 is the second content type and is the EDI standard prominent in the United States. Consent is the third content type and covers all other EDI standards that trading partners agree upon between themselves when they decide to use EDI to do business.
The subject of the EDI usage document was included in the original charter of the EDI working group and the document was targeted as a deliverable that would be forwarded as an informational RFC.
Because the EDI community was already well established and had its own way of doing things, use of the Internet for EDI had to fit within the expectations and conventions of that community. It was also generally well known from the e-mail list and the BOF meeting that most of the EDI community was not very knowledgeable about the Internet. It became general knowledge that a FAQ directed at this community to educate it about the Internet as it relates to the community would be desirable.
As the working group began to do its work via the EDI mailing list, it was apparent that the current users of EDI who were also interested in using the Internet as a transport media were varied and included all types of organizations and all types of transactions. The Automotive Industry Action Group (AIAG) was one of the early voices that spoke often on the mailing list. The AIAG is involved in EDI between automotive manufacturers and their suppliers. The AIAG had already decided to make a recommendation for the use of TCP/IP and public value-added networks (VANs) across the Internet.
It was soon discovered that EDI was not only being used for business transactions but also for information exchange. Educational factions within the working group indicated they had already begun sending electronic student transcripts as an information exchange rather than a business exchange, using the American National Standards Institute (ANSI) X12 130 transaction set.
Early in the work, it was established that the critical issue would soon become e-mail-enabled EDI vs. traditional EDI. It was generally agreed within the working group that any standard e-mail approach is better than the old file-transfer approach. The discussion soon got to the issue of SMTP versus X.400 and the general thinking was that because of the interoperability potential between SMTP/MIME and X.400/X.435 mailers, both approaches should be supported.
Most traditional EDI is done on the store-and-forward model. Transactions are forwarded to a mailbox and stored until the owner of the mailbox clears the mailbox. In fact, mailbox services were the initial business of the VANs. They stored the EDI transactions of the sender in the mailboxes of the recipients until the recipients were ready to access and use the transactions.
Under this model, most VANs offer fax notification if urgent or time-oriented data is received by the VAN. Some VANs will even dial your computer and send the data if it is time-oriented.
One requirement that was voiced by the automobile industry early in the discussion was the need for real-time or interactive EDI. In the case of the automobile manufacturers, Just In Time and Zero inventory manufacturing have already required suppliers to co-locate with the manufacturing plants or to do their own warehousing. A truck may leave a loading dock to arrive at another location in 15 minutes and for EDI to be useful in this environment, the transaction must not only be sent and received, but other computer-based processing must be done after the receipt of the EDI transaction set and before the truck arrives.
The auto industry is requiring that all supplier information be IP-based, but they also have a two-minute time requirement for turn-around time for EDI data. Because of this two-minute turn-around requirement, it did not appear that the e-mail approach would meet the auto industry's needs. Latency times do occur on occasion in the transport of e-mail.
The health care industry also stepped forward and began pushing interactive EDI from the start. This industry indicated a need for real-time, immediate response. Health care providers need immediate access to clinical histories. Health care payers want to manage patient access to services by requiring justification, but at the same time patients expect their insurance coverage to afford them immediate access to the care they require. Patients do not expect questions of coverage and reimbursement to delay the services they need when they already have insurance. So the needs for interactive EDI are already necessary in this industry.
Will the Internet be used for consumer-to-business EDI transactions? For instance, pizza orders are normally made from an individual to a business. Delivery times for the pizza delivery are necessarily short, and the need for at least fast EDI, if not real-time, is apparent for this usage. It appears that as Internet connectivity approaches ubiquity, the uses for EDI across the Internet will increase. What has traditionally been viewed as an enterprise-to-enterprise exchange will be rethought to include lower volume, individual, consumer-to-business transactions.
EDI has been increasingly used by both commercial and governmental entities to conduct routine business transactions through the use of computer-to-computer communications. Commercial VANs have been the traditional carriers of EDI messages between the EDI trading partners. Usage of a VAN provides the EDI trading partner with the ability to send and receive EDI transactions without having to set up communications links with each trading partner on a one-to-one basis. Of course, use of the Internet by trading partners would decrease the telecommunications complexities of having to have separate proprietary communications links with each individual trading partner.
It was pointed out that VANs provide at least two types of services that are not immediately available over the Internet without the use of a middleman. First is the fact that practical applications of EDI did not at that time have authentication and security built into them. The VAN provides the authentication and security to the VAN subscriber. In addition, the VANs provide transaction logging and verification. Traditional implementations of EDI are based on store-and-forward mechanisms, and transaction logging and verification allow the easy resolution of disputes when one party claims to have transmitted an EDI message and the other says it was never received.
The authentication and security problem was seen as being solvable through the use of public key encryption, while the problem of legal receipt could be handled with signed acknowledgements.
Of course another primary reason for the use of a commercial VAN is to enable connections to a single network instead of having to connect to each trading partner directly. The Internet solves this problem inherently.
The VAN model is lacking when real-time EDI is needed. If you were to imagine thousands of simultaneous real-time connections handling responses on demand, then you end up needing only a network provider, not a middleman. It is probable that a third party would be desirable to handle acknowledgments for high-dollar transactions, but this third party would not need to provide all the other traditional VAN services. In general, businesses are not going to see the need to record purchase orders with third parties or log price quotes with a third party. Businesses don't do that now, so why would they do that if the transport mechanism changed? Most businesses deal directly with their suppliers. If the business arrangement is not satisfactory, then the business can deal with another supplier.
It was generally agreed that accountability is required. Audit trails that show exactly what happened and receipts that show dates and times are important. In general, accountability between buyer and seller is not a problem because the seller is trying to keep the buyer as a customer and so problems that arise are easily solved between the parties involved. Private lines are used by telephone or through a VAN. On the Internet it is possible to set up the same concept of a private line by providing an encrypted tunnel between trading partners who have firewalls on the network.
From the standpoint of law, a document trail including dates and times is required by lawyers and accountants to prove time and place of contract formation.
For integrity purposes, checksums of some kind are needed as well as digitally signed receipts.
It is clear that in the case of EDI with no real-time issues and no need for interactive users that e-mail serves the same functionality of a store-and-forward delivery mechanism.
Authentication and encryption RFCs already existed for use for Telnet. Using a Telnet session, the originator establishes an authenticated Telnet TTY connection and starts the EDI transmission by sending a MIME EDI data stream. The originator then waits for a period of time for the return MIME EDI data stream to complete the transaction. No decision was reached on the use of Telnet.
The question of whether or not MIME has the ability to carry binary EDI data was a concern repeated several times over the first few months that the e-mail list was in existence. MIME has a standard solution to the requirement for carrying binary data. Content-Transfer-Encoding:bin64 is the solution. This solution converts the binary data into a form that can be carried over text-only links such as e-mail.
The automobile industry's CAD group did not want to include CAD data inside EDI. They preferred that CAD data be transported separately.
The US Government was in the process of attempting to develop a comprehensive plan for implementing an initial EC procurement capability. Although the federal architecture did have the word Internet on the schematic, the architecture ended up being constructed to use traditional VAN services and connectivity. The traditional connectivity was primarily bisynchronous modem at that time.
The architecture that was designed for the Federal Acquisitions Computer Network (FACNET) was primarily driven and supported by the Department of Defense (DoD). Civilian agencies generally did not support the architecture, and eventually the requirement to use it was changed.
RFC 1865: "EDI Meets the Internet: Frequently Asked Questions About Electronic Data Interchange (EDI) on the Internet"
The document was intended to be informational from its earliest conception and ended up becoming a FAQ that introduces the Internet to the EDI community. This target audience was the existing EDI community, developers, users, managers, and service providers. In contrast the usage document was intended for the Internet community itself.
RFC 1865 was published in January 1996 and is a good beginner's document for orientation to the Internet from the perspective of an EDI practitioner.
The EDI working group of the IETF was at least partially successful in producing a standards track technical document and an informational RFC. It was not, however, able to accomplish its original charter for at least two reasons that have been noted in this paper. First, this was the initial attempt of these two vastly differing communities to work together and establish standards that affect both communities. The people from both sides had different ideas about and approaches to what should be done. The clash of egos and methods was detrimental to the mission of the working group. It has been suggested that the cultural clash between the EDI and the Internet community may also be related to the previous clashes between the Internet community and OSI standardization, in particular X.400. Although some shared work was done (especially through the ISODE consortium; see especially RFC 1006 and X.400/SMTP mappings, etc.), the cultural differences were never really resolved. This background conflict may have influenced the EDI/Internet discussion at the time the EDI working group of IETF initially met.
The second reason the working group had to be disbanded was the failure to produce the usage document that it was originally chartered to produce. The specification for MIME encapsulation of EDI objects was fairly easy to accomplish, because the working group kept a very narrow focus on what that document should accomplish. In addition, the MIME specification was both technically trivial and a version of it already existed when the working group began as an e-mail list. The usage document, in contrast, was very broad in scope and the work was never focused. The document was abandoned for lack of progress and the working group was disbanded.
Work never did begin on the third and fourth deliverables, a specification of EDI routing information (a la X.435) and a specification of mappings between Internet-based and X-400-based EDI. These were defined at the 29th IETF meetings in Seattle in March 1994, but not included in the original charter. The feeling was that the chartered deliverables should receive priority and the other two deliverables could be tackled if the interest was still there after the first two deliverables were accomplished.
It should be remembered that many Internet protocols are based on Dave Clark's "rough consensus and running code" paradigm. One may assume that if EDI over MIME would have solved a "real" problem at that time, the uptake of EDI message formats in MIME would have been swift and completely in parallel to the more "political" discussions. This did not happen so it is probably safe to assume that EDI over the Internet was not in great demand and that no actual problems on the Internet were being solved during these timeframes.