You are here: irt.org | RFCs | RFC713 [ previous next ]
Request for Comments: 713 Jack Haverty (JFH@MIT-DMS) NIC #34739 Apr 1976 I. ABSTRACT A mechanism is defined for use by message servers in transferring data between hosts. The mechanism, called the MSDTP, is defined in terms of a model of the process as a translation between two sets of items, the abstract entities such as 'strings' and 'integers', and the formats used to represent such data as a byte stream. A proposed organization of a general data transfer mechanism is described, and the manner in which the MSDTP would be used in that environment is presented. -1- II. REFERENCES Black, Edward H., "The DMS Message Composer", MIT Project MAC, Programming Technology Division Document SYS.16.02. Burchfiel, Jerry D., Leavitt, Elsie M., Shapiro, Sonya and Strollo, Theodore R., compilers, "Tenex Users' Guide", Bolt Beranek and Newman, Cambridge, Mass., May 1971, revised January 1975, Descriptive sections on the TENEX subsystems: MAlLER, p. 116-11; MAlLSTAT, p. 118-119; READMAIL, p. 137; and SNDMSG, p. 165-170. Haverty, Jack, "Communications System Overview", MIT Project MAC, Programming Technology Division Document SYS.16.00. Haverty, Jack, "Communications System Daemon Manual", MIT Project MAC, Programming Technology Division Document SYS.16.01. ISI Information Automation Project, "Military Message Processing System Design," Internal Project Documentation (Out of Print), Jan. 1975 Message Services Committee, "Interim Report", Jan. 28, 1975 Mooers, Charlotte D., "Mailsys Message System: Manual For Users", Bolt Beranek and Newman, Cambridge, Mass., June 1975 (draft). Myer, Theodore H., "Notes On The BBN Mail System", Bolt Beranek and Newman, November 8, 1974. Myer, Theodore H., and Henderson, D. Austin, "Message Transmission Protocol", Network Working Group RFC 680, NIC 32116, April 30, 1975. Postel, Jon, "The PCPB8 Format", NSW Proposal, June 5, 1975 Tugender, R., and D. R. Oestreicher, "Basic Functional Capabilities for a Military Message Processing Service," ISI?RR-74-23., May 1975 Vezza, Al, "Message Services Committee Minority Report", Jan. 1975 -2- III. OVERVIEW This document describes a mechanism developed for use by message servers communicating over an eight-bit byte-oriented network connection to move data structures and associated data-typing information. It is presented here in the hope that it may be of use to other projects which need to transfer data structures between dissimilar hosts. A set of abstract entities called PRIMITIVE ITEMS is enumerated. These are intended to include traditional data types of general utility, such as integers, strings, and arrays. A mechanism is defined for augmenting the set of abstract data entities handled, to allow the introduction of application-specific data, whose format and semantics are understood by the application programs involved, but which can be transmitted using common coding facilities. An example might be a data structure called a 'file specification', or a 'date'. Abstract data entities defined using this mechanism will be termed SEMANTIC ITEMS, since they are typically used to carry data having semantic content in the application involved. Semantic and primitive items are collectively referred to simply as ITEMS. The protocol next involves the definition of the format of the byte stream used to convey items from machine to machine. These encodings are described in terms of OBJECTS, which are the physical byte streams transmitted. To complete the protocol, the rules for translating between objects and items are presented as each object is defined. An item is transmitted by being translated into an object which is transmitted over the connection as a stream of bytes to the receiver, and reconstructed there as an item. The protocol mechanism may thus be viewed as a simple translator. It enumerates a set of abstract entities, the items, which are known to programmers, a set of entities in byte-stream format, the objects, and the translation rules for conversion between the sets. A site implementing the MSDTP would typically provide a facility to convert between objects and the local representation of the various items handled. Applications using the MSDTP define their interactions using items, without regard to the actual formats in which such items are represented at various machines. This permits programs to handle higher-level concepts such as a character string, without concern for its numerous representational formats. Such detail is handled by the MSDTP. -3- Finally, a discussion of a general data transfer mechanism for communication between programs is presented, and the manner in which the particular byte-oriented protocol defined herein would be used in that environment is discussed. Terminology, as introduced, is defined and highlighted by capitalizing. IV. PRIMITIVE DATA ITEMS The primitive data items include a variety of traditional, well-understood types, such as integers and strings. Primitive data items will be presented using mnemonic names preceded by the character pair "p-", to serve as a reminder that the named object is primitive. These items may be represented in various computer systems in whatever fashion their programmers desire. IV.1 -- Set Of Primitive Items The set of primitive items defined includes p-INT, p-STRING, p-STRUC, p-BITS, p-CHAR, p-BOOL, p-EMPTY, and p-XTRA. Since the protocol was developed primarily for use in message services, items such as p-FLOAT are not included since they were unnecessary. Additional items may be easily added as necessary. A p-INT performs the traditional role of representing integer numbers. A p-BITS (BIT Stream) item represents a bit stream. The two possible p-BOOL (BOOLean) items are used to represent the logical values of *TRUE* and *FALSE*. The single p-EMPTY item is used to, for example, indicate that a given field of a message is empty. It is provided to act as a place-holder, representing 'no data', and appears as *EMPTY*. The p-STRUC (STRUCture) item is used to group together a collection of items as a single value, maintaining the ordering of the elements, such as a p-STRUC of p-INTs. A p-CHAR is a single character. The most common occurrence of character data, however, will be as p-STRINGs. A p-STRING should be considered to be a synonym for a p-STRUC containing only p-CHARs. This concept is important for generality and consistency, especially when considering definitions of permissible operations on structures, such as extracting subsequences of elements, etc. -4- Four p-XTRA items, which can be transmitted in a single byte, are made available for higher level protocols to use when a frequently used datum is handled which can be represented just by its name. An example would be an acknowledgment between two servers. Using p-XTRAs to represent such data permits them to be handled in a single byte. There are four possible p-XTRA items, termed *XTRA0*, *XTRA1*, *XTRA2*, and *XTRA3*. These may be assigned meanings by user protocols as desired. IV.2 -- Printing Conventions The following printing conventions are introduced to facilitate discussion of the primitive items. When a specific instance of a primitive data item is presented, it will be shown in a traditional representation for that kind of data. For example, p-INTs are shown as sequences of digits, e.g. 100, p-STRINGs, as sequences of characters enclosed in double-quote characters, for example "ABCDEF". As shown above, the two possible p-BOOL items are shown as *TRUE* or *FALSE*. The object p-EMPTY appears as *EMPTY*. A bit stream, i.e. p-BITS, appears as a stream of 1s and 0s enclosed in asterisks, for example *100101001*. A p-CHAR will be presented as the character enclosed in single quote characters, e.g., 'A'. P-STRUCs are printed as the representations of their elements, enclosed in parentheses, for example (1 2 3 4) or ("XYZ" "ABC" 1 2) or ((1 2 3) "A" "B"). Note that because p-STRINGs are simply a class of p-STRUCs assigned a special name and printing format for brevity and convenience, the items "ABC" and ('A' 'B' 'C') are identical, and the latter format should not be used. To present a generic p-STRUC, as in specifying formats of the contents of something, the items are presented as a mnemonic name, optionally followed by a colon and the permissible types of values for that datum. When one of several items may appear as the value for some component, the permissible ones appear separated by vertical-bar characters. For example, p-INT|p-STRING represents a single item, which may be either a p-INT or a p-STRING. To represent a succession of items, the Kleene star convention is used. The specification p-INT[*] represents any number of p-INTs. Similarly, p-INT[3,5] represents from 3 to 5 p-INTs, while p-INT[*,5] specifies up to 5 and p-iNT[5,*] specifies at least 5 p-INTs. -5- For example, a p-STRUC which is used to carry names and numbers might be specified as follows. (name:p-STRING number:p-INT) In discussing items in general, when a specific data value is not intended, the name and types representation may be used, e.g., offset:p-INT to discuss an 'offset' which has a numeric value. V. SEMANTIC ITEM MECHANISM The semantic item mechanism provides a means for program designers to use a variety of application-specific data items. This mechanism is implemented using a special tagged structure to carry the data type information as well as the actual components of the particular semantic item. For discussion purposes. Such a special p-STRUC will be termed a p-EDT (Extended Data Type). When p-EDTs are transferred, their identity as a p-EDT is maintained. So that an applications program receives the corresponding semantic item instead of a simple p-STRUC. A p-EDT is identical to a p-STRUC in all other respects. V.1 -- Format of p-EDTs A prototypical p-EDT follows. It is printed as if it were a normal p-STRUC. Since p-EDTs are converted to semantic items for presentation to the user, a p-EDT will never be used except in this protocol definition. (type:p-INT|p-STRING version:p-INT com1:any com2:any ...) The first element, the 'type' is generally a p-INT, and is used to identify the particular type of semantic item. Types are assigned numeric codes in a controlled fashion. The type may alternatively be specified by a p-STRING, to permit development of new data types for possible later assignment of codes. Each type has an equivalent p-STRING name. These may be used interchangeably as 'type' elements, primarily to maintain upward compatibility. The second element of a p-EDT is always an p-INT, the 'version', and specifies the exact format of the particular datum. A semantic item may undergo several revisions of its internal structure. Which would be evident through assigning different versions to each revision. -6- Successive components. The 'com' elements, if any. carry the actual data of the semantic item. As each semantic item is defined, conventions on permissible values and interpretation of these components are presented. Such definitions may use any types of items to specify the format of the semantic item. Use of lower level concepts, such as objects, in these definitions is prohibited. Semantic items will be printed as the mnemonic for the type involved, preceded by the character pair "s-", to signify that the data item is handled by this mechanism. V.2 -- Printing Conventions A semantic item is represented as if it were a p-STRUC containing only the components, if any, but preceded by the semantic type name and a # character. The version number is assumed to be 1 if unspecified. For later versions, the version number is attached to the type name, as in, for example, FILE-2 to represent version 2 of the FILE data type. For example, a semantic item called a 'file specification' might be defined, containing two components, a host number and pathname. A specific instance of such an item might appear as #FILE(69 "DIRECTORY.NAME-OF-FILE"), while a generic s-FILE might be presented as the following. #FILE(host:p-INT|p-STRING pathname:p-STRING) the item, which may be either a p-INT or p-STRING, and 'pathname' is the second component, which must be a p-STRING. The full definition would present interpretation rules for these components. VI. ENCODING OBJECTS This section presents the set of objects which are used to represent items as byte streams for inter-server transmission. Objects will be presented using mnemonic type-names preceded by the character pair "b-", indicating their existence only as byte streams. All servers are required to be capable of decoding the entire set of objects. Servers are not required to transmit certain objects which are available to improve channel efficiency. -7- The encodings are designed to facilitate programming and efficiency of the receiving decoder. In all cases, the type and length in bytes of objects is supplied as the first information sent. This characteristic is important for ease of implementation. The type information permits a decoder to be constructed in a modular fashion. The most important advantage of including size information is that the receiver always knows how many bytes it must read to discover what to do next, and knows when each object terminates. This requirement avoids many potential problems with timing and synchronization of processes. Two varieties of objects are defined. The first will be called ATOMIC, and includes objects used to efficiently encode the most common data. The second variety is termed NON-ATOMIC, and is used to encode larger or less common items. In all cases, a data object begins with a single byte, which will be termed the TYPE-BYTE, a field of which contains the type code of the object. The following bytes, if any, are interpreted according to the type involved. VI.1 -- Presentation Conventations In discussing formats of bytes, the following conventions will be employed. The individual bits of a byte will be referenced by using capital letters from A to H, where A signifies the highest order bit, and H the lowest. The entire eight bit value, for example, could be referred to as ABCDEFGH. Similarly, subfields of the byte will be identified by such sequences. The CDEF field specifies the middle four bits of a byte. In referring to values of fields, binary format will be used, and small letters near the end of the alphabet will be used to identify particular bits for discussion. For example, we might say that the BCD field of a byte contains a specifier for some type, and define its value to be BCD=11z. In discussions of the specifier usage, we could refer to the cases where z=l and where z=0, as shorthand notation to identify BCD=111 and BCD=110, respectively. V1.2 -- Type-Byte Bit Assignment To assist in understanding the assignment of the various type-byte values, the table and graph below are included, showing representations of the eight bits. -8- OXXXXXXX -- CHAR7 (CHARacter, 7 bit) 10XXXXXX -- SINTEGER (Small INTEGER) l10XXXXX -- NON-ATOM (NON-ATOMic objects) 11100XXX -- LINTEGER (Large INTEGER) 11101XXX -- reserved 11110XXX -- SBITSTR (Short BIT STReam) 111110XX -- XTRA (eXTRA single-byte objects) 1111110X -- BOOL (BOOLean) 11111110 -- EMPTY (EMPTY data item) 11111111 -- PADDING (unused byte) In each case, the bits identified by X's are used to contain information specific to the type involved. These are explained when each type is defined. An equivalent tree representation follows, for those who prefer it. start with high order bit | | | 0-----0-----0-----0-----0-----0-----0-----0-----X | | | | | | | | PADDING 0| 0| 0| 0| 0| 0| 0| 0| | | | | | | | | X | X | X | X X CHAR7 | NON-ATOM | BITS | BOOL EMPTY (7) | (5) | (3) | (1) | 0| | | SINTEGER | XTRA (6) | (2) LINTEGER (3) Type-Byte Bit Assignment Scheme This picture is interpreted by entering at the top, and taking the appropriate branch at each node to correspond to the next bit of the type-byte, as it is scanned from left to right. When a type is assigned, the branch terminates with an "X' and the name of the type of the object, with the number of remaining bits in parentheses. The individual object definitions specify how these bits are used for that particular type. V1.3 -- Atomic Objects Atomic objects are identified by specific patterns in a type-byte. Receiving servers must be capable of recognizing -9- and handling all atomic types, since the size of the object is not explicitly present in a uniform fashion. ================================ | Atomic Object: B-CHAR7 | ================================ The b-CHAR7 (CHARacter 7 bit) object is introduced to handle transmission of characters, in 7-bit ASCII format. Since the vast majority of message-related data involves such objects, they are designed to be very efficient in transmission. Other formats, such as eight bit values, can be introduced as non-atomic objects. The format of a b-CHAR7 follows: A=0 identifying the b-CHAR7 data type BCDEFGH=tuvwxyz containing the character code The tuvwxyz objects contain the ASCII code of the character. For example, transmission of a "space' (ASCII code 32, 40 octal) would be accomplished by the following byte. 00100000 ABCDEFGH A=0 to identify this byte as a b-CHAR7. The remaining bits contain the 7 bit code, octal 40, for space. A b-CHAR7 standing alone is presented as a p-CHAR. Such occurrences will probably be rare if they are used at all. The most common use of b-CHAR7's is as elements of b-USTRUCs used to transmit p-STRINGS, as explained later. ============================= | Atomic Object: B-SINTEGER | ============================= The b-SINTEGER (Small INTEGER) object is used to transmit very small positive integers, of values up to 64. It always translates to an p-INT, and any p-INT between 0 and 63 may be encoded as a b-SINTEGER for transmission. The format of an b-SINTEGER follows. AB=10 identifying the object as a b-SINTEGER CDEFGH=uvwxyz containing the actual number For example, to transmit the integer 10 (12 octal), the following byte would be transmitted: 10001010 ABCDEFGH -10- AB=10 to specify a b-SINTEGER. The remaining six bits contain the number 10 expressed in binary. ============================= | Atomic Object: B-SINTEGER | ============================= The b-SINTEGER (Large INTEGER) object is used to transmit p-INTs to any precision up to 64 bits. It is always translated as a p-INT. Sending servers are permitted to choose either b-SINTEGER or b-SINTEGER format for transmission of numbers, as appropriate. When possible, b-SINTEGERs can be used for better channel efficiency. The format of a b-SINTEGER follows: ABCDE=11100 specifying that this is a b-SINTEGER. FGH=xyz containing a count of number of bytes to follow. The xyz bits are interpreted as a number of bytes to follow which contain the actual binary code of the the integer in 2's complement format. Since a zero-byte integer is disallowed, the pattern xyz=000 is interpreted as 1000, specifying that 8 bytes follow. The number is transmitted with high-order bits first. This format permits transmission of integers as large as 64 bits in magnitude. For example, if the number 4096 (10000 octal) is to be transmitted, the following sequence of bytes would be sent: 11100010 00010000 00000000 ABCDEFGH ---actual data--- ABCDE=11100, identifying this as a b-LINTEGER, E=0, specifying a positive number, and FGH=010, specifying that 2 bytes follow, containing the actual binary number. ============================ | Atomic Object: B-SBITSTR | ============================ The b-SBITSTR (Short BIT STReam) object is used to transmit a p-BITS of length 63 or less. For longer bit streams, the non-atomic object b-LBITSTR may be used. The format of a b-SBITSTR follows. ABCDE=11110 specifying the type as b-SBITSTR FGH=xyz specifying the number of bytes following. -11- The xyz value specifies the number of additional bytes to be read to obtain the bit stream values. As in the case of b-SINTEGER, the value xyz=000 is interpreted as 1000, specifying that 8 bytes follow. To avoid requiring specification of exactly the number of bits contained, the following convention is used. The first data byte is scanned from left to right until the first 1 bit is encountered. The bit stream is defined to begin with the immediately following bit, and run through the last bit of the last byte read. In other words, the bit stream is 'right-adjusted' in the collected bytes, with its left end delimited by the first "on' bit. For example, to send the bit stream *001010011* (9 bits), the following bytes are transmitted. 11110010 00000010 01010011 ABCDEhij klmnopqr stuvwxyz The hij=010 value specifies that two bytes follow. The q bit, which is the first 1 bit encountered, identifies the start of the bit stream as being the r bit. The rstuvwxyz bits are the bit stream being handled. ========================= | Atomic Object: b-BOOL | ========================= The b-BOOL (BOOLean) object is used to transmit p-BOOLs. The format of b-BOOL objects follows. ABCDEFG=1111110 specifying the type as b-BOOL H=z specifying the value The two possible translations of a b-BOOL are *FALSE* and *TRUE*. 11111100 represents *FALSE* 11111101 represents *TRUE* ABCDEFGz if z=0, the value is FALSE, otherwise TRUE. ======================================== | Atomic Object: B-EMPTY | ======================================== The b-EMPTY object type is used to transmit a 'null' object, i.e. an *EMPTY*. The format of an b-EMPTY follows. ABCDEFGH=11111110 specifying *EMPTY* -12- ========================= | Atomic Object: B-XTRA | ========================= The b-XTRA objects are used to carry the four possible p-XTRA items, i.e., *XTRA0*, *XTRA1*, *XTRA2*, and *XTRA3*. These four items correspond to the binary coding of the remaining two bits after the b-XTRA type code bits. The format of a b-XTRA follows. ABCDEF=111110 to specify the type b-XTRA GH=yz to identify the particular p-XTRA item carried The GH bits of the byte are decoded to produce a particular p-XTRA item, as follows. GH=00 -- *XTRA0* GH=01 -- *XTRA1* GH=10 -- *XTRA2* GH=11 -- *XTRA3* The b-XTRA object is included to provide the use of several single-byte data items to higher levels. These items may be assigned by individual applications to improve the efficiency of transmission of several very frequent data items. For example, the message services protocols will use these items to convey positive and negative acknowledgments, two very common items in every interaction. ======================================== | Atomic Object: B-PADDING ======================================== This object is anomalous, since it represents really no data at all. Whenever it is encountered in a byte stream in a position where a type-byte is expected, it is completely ignored, and the succeeding byte examined instead. Its purpose is to serve as a filler in byte streams, providing servers with an aid in handling internal problems related to their specific word lengths, etc. The encoders may freely use this object to serve as padding when necessary. All b-PADDING data objects exist only within an encoded byte stream. They never cause any data item whatsoever to be presented externally to the coder module. The format of a b-PADDING follows. ABCDEFGH=11111111 Note that this does not imply that all such 'null' bytes in a stream are to be ignored, since they could be encountered as a byte within some other type, such as b-LINTEGER. Only bytes of this format which, by their position in the stream, appear as a 'type' byte are to be ignored. -13- VI.4 -- Non-Atomic Objects Non-atomic objects are are always transmitted preceded by both a single type byte and some small number of size byte(s). The type byte identifies that the data object concerned is of a non-atomic type, as well as uniquely specifying the particular type involved. All non-atomic objects have type byte values of the following form. ABC=110 specifying that the object is non-atomic DEFGH=vwxyz specifying the particular type of object The vwxyz value is used to specify one of 31 possible non-atomic types. The value vwxyz=00000 is reserved for use in future expansion. In all non-atomic data objects, the byte(s) following the type-byte specify the number of bytes to follow which contain the data object. In all cases, if the number of bytes specified are processed, the next byte to be seen should be another type-byte, the beginning of the next object in the stream. The number of bytes containing the object size information is variable. These bytes will be termed the SIZE-BYTES. The first byte encountered has the following format. A=s specifying the manner in which the size information is encoded BCDEFGH=tuvwxyz specifying the size, or number of bytes containing the size The tuvwxyz values supply a positive binary number. If the s value is a one, the tuvwxyz value specifies the number of bytes to follow which should be read and concatenated as a binary number, which will then specify the size of the object. These bytes will appear with high order bits first. Thus, if s=1, up to 128 bytes may follow, containing the count of the succeeding data bytes, which should certainly be sufficient. Since many non-atomic objects will be fairly short, the s=0 condition is used to indicate that the 7 bits contained in tuvwxyz specify the actual data byte count. This permits objects of sizes up to 128 bytes to be specified using one size-information byte. The case tuvwxyz=0000000 is interpreted as specifying 128 bytes. For example, a data object of some non-atomic type which requires 100 (144 octal) bytes to be transmitted would be sent as follows. -14- 110XXXXX -- identifying a specific non-atomic object 01100100 -- specifying that 100 bytes follow . . data -- the 100 data bytes . . Note that the size count does not include the size-specifier byte(s) themselves, but does include all succeeding bytes in the stream used to encode the object. A data object requiring 20000 (47040 octal) bytes would appear in the stream as follows. 110XXXXX -- identifying a specific non-atomic object 10000010 -- specifying that the next 2 bytes contain the stream length 01001110 -- first byte of number 20000 00100000 -- second byte . . data -- 20,000 bytes . . Interpretation of the contents of the 20000 bytes in the stream can be performed by a module which knows the specific format of the non-atomic type specified by DEFGH in the type-byte. The remainder of this section defines an initial set of non-atomic types, the format of their encoding, and the semantics of their interpretation. ================================ | Non-atomic Object: B-LBITSTR | ================================ The b-LBITSTR (Long BIT Stream) data type is introduced to transmit p-BITS which cannot be handled by a b-SBITSTR. A b-LBITSTR may be used to transmit short p-BITS as well. Its format follows. -15- 11000001 size-bytes data-bytes ABCDEFGH ABC=110 identifies this as a non-atomic object. DEFGH=00001 specifies that it is a b-LBITSTR. The standard sizing information specifies the number of succeeding bytes. Within the data-bytes, the first object encountered must decode to a p-INT. This number conveys the length of the bit stream to follow. The actual bit stream begins with the next byte, and is left-adjusted in the byte stream. For example to encode *101010101010*, the following b-LBITSTR could be used, although a b-SBITSTR would be more compact. 11000001 -- identifies a b-LBITSTR 00000010 -- b-SINTEGER, to specify length 10001100 -- size = 2 10101010 -- first 8 data bits 10100000 -- last 4 data bits ============================== | Non-atomic Object: B-STRUC | ============================== The b-STRUC (STRUCture) data type is used to transmit any p-STRUC. The translation rules for converting a b-STRUC into a primitive item are presented following the discussion of b-REPEATs. The b-STRUC format appears as follows. 11000010 size-bytes data-bytes ABCDEFGH ABC=110 identifies this as a non-atomic type. DEFGH=00010 specifies that the object is a b-STRUC. Within the data-bytes stream, objects simply follow in order. This implies that the b-STRUC encoder and decoder modules can simply make use of recursive calls to a standard encoder/decoder for processing each element of the b-STRUC. Note that any type of object is permitted as an element of a b-STRUC, but the size information of the b-STRUC must include all bytes used to represent the elements. Containment of b-STRUCs within other b-STRUCs is permitted to any reasonable level. That is, a b-STRUC may contain as an element another b-STRUC, which contains another b-STRUC, and so on. All servers are requires to handle such containment to at least a minimum depth of three. Examples of encoded structures appear in a later section. -16- ============================ | Non-atomic Object: B-EDT | ============================ A b-EDT is the object used as the carrier for p-EDTs in transmission of semantic items. It is functionally identical to a b-STRUC, but has a different type code to permit it to be identified and converted to a semantic item instead of a p-STRUC. The format of a b-EDT follows. 11000011 size-bytes data-bytes ABCDEFGH As with all non-atomic types, ABC=110 to identify this as such, and DEFGH=00011 to specify a b-EDT. The objects in the data-bytes are decoded as for b-STRUCs. However, the first object must decode to a p-iNT or p-STRING and the second to a p-INT, to conform to the format of p-EDTs. =============================== | Non-atomic Object: b-REPEAT | =============================== The b-REPEAT object is never translated directly into an item. It is legal only as an component of an enclosing b-STRUC, b-USTRUC, b-EDT, or b-REPEAT. A b-REPEAT is used to concisely specify a set of elements to be treated as if they appeared in the enclosing structure in place of the b-REPEAT. This provides a mechanism for encoding a sequence of identical data items or patterns efficiently for transmission. A common example of this would be in transmission of text, where line images containing long sequences of spaces, or pages containing multiple carriage-return, line-feed pairs, are often encountered. Such sequences could be encoded as an appropriate b-REPEAT to compact the data for transmission. The format of a b-REPEAT is as follows. 11000100 -- identifyIng the object as a b-REPEAT size-bytes -- the standard non-atomic object size information countspec -- an object which translates to a p-INT . . data -- the objects which define the pattern . . The 'countspec' object must translate to an p-INT to specify the number of times that the following data pattern should be repeated in the object enclosing the b-REPEAT. -17- The remaining objects in the b-REPEAT constitute the data pattern which is to be repeated. The decoding of the enclosing structure will be continued as if the data pattern objects appeared 'countspec' times in place of the b-REPEAT. Zero repeat counts are permitted, for generality. They cause no objects to be simulated in the enclosing structure. An encoder does not have to use b-REPEATs at all, if simplicity of coding outweighs the benefits of data compression. In message services, for example, an encoder might limIt itself to only compressing long text strings. It is important for compatibility, however, to have the ability in the decoders to handle b-REPEATs. =============================== | Non-atomic Object: B-USTRUC | =============================== The b-USTRUC (Uniform Structure) object type is provided to enable servers to convey the fact that a p-STRUC being transferred contains items of only a single type. The most common example would involve a b-USTRUC which translates to a p-STRUC of only p-CHARs, and hence may be considered to be a p-STRING. Servers may use this information to assist them in decoding objects efficiently. No server is required to generate b-USTRUCs. The internal construction of a b-USTRUC is identical to that of a b-STRUC, except for the type-byte. The format of a b-USTRUC follows. 11000101 size-bytes data-bytes ABCDEFGH ABC=110 to identify a non-atomic object. DEFGH=00101 specifies the object as a b-USTRUC. =============================== | Non-atomic Object: B-STRING | =============================== The b-STRING object is included to permit explicit specification of a structure as a p-STRING. This information will permit receiving servers to process the incoming structure more efficiently. A b-STRING is formatted similarity to a b-USTRUC, except that its type-byte identifies the object as a b-STRI/NG. The normal sizing information is followed by a stream of bytes which are interpreted as b-CHAR7s, Ignoring the high-order bit. The format of a b-STRING follows. 11000110 size-bytes data-bytes ABCDEFGH ABC=110 to identify a non-atomic object. DEFGH=00110 specifies the object as a b-STRING. -18- VI.5 -- Structure Translation Rules A b-STRUC is translated into a p-STRUC. This is performed by translating each object of the b-STRUC Into its corresponding item, and saving it for inclusion In the p-STRUC being generated. A b-USTRUC is handled similarly, but the coding programs may utilize the information that the resultant p-STRUC will contain items of uniform type. The preferred method of coding p-STRINGS is to use b-USTRUCs. If all of the elements of the resultant p-STRUC are p-CHARs, it is presented to the user of the decoder as a p-STRING. A p-STRING should be considered to be a synonym for a p-STRUC containing only characters. It need not necessarily exist at particular sites which would present p-STRUCs of p-CHARs to their application programs The object b-REPEAT is handled in a special fashion when encountered as an element. When this occurs, the data pattern of the b-REPEAT is translated into a sequence of items, and that sequence is repeated in the next higher level as many times as specified in the b-REPEAT. Therefore, b-REPEATS are legal only as elements of a surrounding b-STRUC, b-USTRUC, b-EDT, or b-REPEAT. In encoding a p-STRUC or p-STRING for transmission, a translator may use b-REPEATs as desired to effect data compression, but their use is not mandatory. Similarly, b-STRINGS may be used, but are not mandatory. A b-EDT is translated into a p-EDT to identify it as a carrier for a semantic item. Otherwise, it is treated identically to a b-STRUC. VI.6 -- Translation Summary The following table summarizes the possible translations between primitive items and objects. p-INT <--> b-LINTEGER, b-SINTEGER p-STRING <--> b-STRING, b-STRUC, b-USTRUC p-STRUC <--> b-STRING, b-STRUC, b-USTRUC p-BITS <--> b=SBITSTR, b-LBITSTR p-CHAR <--> b-CHAR7 p-BOOL <--> b-BOOL p-EMPTY <--> b=EMPTY p-XTRA <--> b-XTRA p-EDT <--> b-EDT (all semantic items) -none- <--> b-PADDING -none- <--> b-REPEAT (only within structure) Note that all semantic items are represented as p-EDTs which always exist as b-EDTs in byte-stream format. -19- V1.7 -- Structure Coding Examples The following stream transmits a b-STRUC containing 3 b-SINTEGERs, with values 1, 2, and 3, representing a p-STRUC containing three p-INTs, i.e. (1 2 3). 11000010 -- b-STRUC 00000011 -- size=3 10000001 -- b-SINTEGER=1 10000010 -- b-SINTEGER=2 10000011 -- b-SINTEGER=3 The next example represents a b-STRUC containing the characters X and Y, followed by the b-LINTEGER 10, representing a p-STRUC of 2 p-CHARs and a p-INT, i.e., ('X' 'Y' 10). Note that the p-INT prevents considering this a p-STRING. 11000010 -- b-STRUC 00000100 -- size=4 01011000 -- b-CHAR7 'X' 01011001 -- b-CHAR7 'Y' 11100001 -- b-LINTEGER 00001010 -- 10 Note that a better way to send this p-STRUC would be to represent the integer as a b-SINTEGER, as shown below. 11000010 -- b-STRUC 00000011 -- size=3 01011000 -- b-CHAR7 'X' 01011001 -- b-CHAR7 'Y' 10001010 -- b-SINTEGER=10 The next example shows a b-STRUC of b-CHAR7s. It is the translation of the b-STRING "HELLO". 11000010 -- b-STRUC 00000101 -- size=5 01001000 -- b-CHAR7 'H' 01000101 -- b-CHAR7 'E' 01001100 -- b-CHAR7 'L' 01001100 -- b-CHAR7 'L' 01001111 -- b-CHAR7 'O' This datum could also be transmitted as a b-STRING. Note that the character bytes are not necessarily b-CHAR7s, since the high-order bit is ignored. 11000110 -- b-STRING 00000101 -- size=5 01001000 -- 'H' 01000101 -- 'E' 01001100 -- 'L' 01001100 -- 'L' 01001111 -- 'O' -20- To encode a p-STRING containing 20 carriage-return line-feed pairs, the following b-STRUC containing a b-REPEAT could be used. 11000010 -- b-STRUC 00000101 -- size=5 11000100 -- b-REPEAT 00000011 -- size=3 10010100 -- count, b-SINTEGER=20 00001101 -- b-CHAR7, "CR' 00001010 -- b-CHAR7, 'IF' To encode a p-STRUC of p-INTs, where the sequence contains a sequence of thirty 0's preceded by a single 1, the following b-STRUC could be used. 11000010 -- b-STRUC 00000110 -- size=6 10000001 -- b-SINTEGER=1 11000100 -- b-REPEAT 00000010 -- size=2 10011110 -- count, b-SINTEGER=30 10000000 -- b-SINTEGER=0 VII. A GENERAL DATA TRANSFER SCHEME This section considers a possible scheme for extending the concept of a data translator into an multi-purpose data transfer mechanism. The proposed environment would provide a set of primitive items, including those enumerated herein but extended as necessary to accommodate a variety of applications. Communication between processes would be defined solely in terms of these items, and would specifically avoid any consideration of the actual formats in which the data is transferred. A repertoire of translators would be provided, one of which is the MSDTP machinery, for use in converting items to any of a number of transmission formats. Borrowing a concept from radio terminology, each translator would be analogous to a different type of modulation scheme, to be used to transfer data through some communications medium. Such media could be an eight-bit byte-oriented connection, 36-bit connection, etc. and conceivably have other distinguishing features, such as bandwidth, cost, and delay. For each media which a site supports, it would provide its programmers with a module for performing the translations required. -21- Certain media or translators might not handle various items. For example, the MSDTP does not handle items which might be termed p-FLOATs, p-COMPLEXs, p-ARRAY, and so on. In addition, the efficiency of various media for transfer of specific items may differ drastically. MSDTP, for example, transfers data frequently used in message handling very efficiently, but is relatively poor at transfer of very large or deep tree structures. Available at each site as a process or subroutine package wouLd be a module responsible for interfacing with its counterpart at the other end of the media. These modules would use a protocol, not yet defined, to match their capabilities, and choose a particular media and translator, when more than one exists, for transfer of data items. Such a facility could totally insulate applications from need to consider encoding formats, machine differences, and so on, as well as eliminate duplication of effort in producing such facilities for every new project which requires them. In addition, as new translators or media are introduced, they would become immediately available to existing users without reprogramming. Implementation of such a protocol should not be very difficult or time-consuming, since it need not be very sophisticated in choosing the most appropriate transfer mechanism in initial implementations. The system is inherently upward-compatible and easily expandable. -22-