Modifications from the last posted version (10 July 2002) are indicated in RED type
3. The syntax of CIF is a proper subset of the syntax of STAR Files as described by Hall (1991) and Hall & Spadaccini (1994). The general structure is described below under the heading General features, and a number of subsections list specific restrictions to the STAR syntax that are in force within CIF. A formal language grammar using computer science notation is included as Appendix A.
4. A CIF consists of data names (tags) and associated values organized into data blocks. A data block may contain data items (associated data names and data values) and/or it may contain save frames.
5. Save frames may only be used in dictionary files.
5a. Implementation note: At a purely syntactic level there is no way to distinguish between dictionary and data files. (It is also to be noted that not all dictionary files contain save frames.) A fully validating parser must therefore be able to detect the start and termination of save frames, the uniqueness of the framecode within a data block, and the uniqueness of data names within a frame code. It is however legitimate for an application-based parser designed to handle only the contents of data files to consider the presence of a save frame as an error.
6. A data block begins with the reserved string data_ followed immediately by the name of the data block, forming a data block header. A save frame has a similar structure to a data block, but may not itself contain further save frames. A save frame begins with the reserved string save_ followed immediately by the name of the save frame, forming a save frame header. Unlike a data block, a save frame also has a marker for the end of the frame in the form of a repetition of the reserved word save_, this time without the name of the frame. Save frames may not nest. Within a single CIF, no two data blocks may have the same name; within a single data block no two save frames may have the same name, although a save frame may have the same name as a data block in the same CIF.
7. A given data name (tag) (see 2.4 and 2.7) may appear no more than once in a given data block or save frame. A tag may be followed by a single value, or a list of one or more tags may be marked by the preceding reserved word loop_ as the headings of the columns of a table of values. White space is used to separate a data block or save frame header from the contents of the data block or save frame, and to separate tags, values and the reserved word loop_.
8. The reserved word stop_ may optionally be used to mark the end of a loop header and/or of a loop body.
9. The word global_, used in STAR Files to introduce a group of data values with a scope extending to the end of the file, is an additional reserved word in CIF (that is, it may not be used as the unquoted value of any data item).
10. If a data value (see 2.5) contains white space or begins with a character string reserved for a special purpose, it must be delimited by one of several sets of special character strings (the choice of which is constrained if the data value contains characters interpretable as marking a new line of text according to the discussion in the following paragraphs). Such a data value will be indicated by the term non-simple data value.
11. A simple data value (i.e. one which does not contain white space or begin with a special character string) may optionally be delimited by any of the same set of delimiting character strings, except for data values that are to be interpreted as numbers.
12. The special character strings in this context are listed in the following table. The term "non-simple data values" in this table refers to data values beginning with these special character strings.
Character or string | Role |
---|---|
_ | identifies data name |
# | identifies comment |
$ | identifies save frame pointer |
' | delimits non-simple data values |
" | delimits non-simple data values |
[ | reserved opening delimiter for non-simple data values (see paragraph 19) |
] | reserved closing delimiter for non-simple data values (see paragraph 19) |
; at beginning of line of text | delimits non-simple data values |
data_ | identifies data block header |
save_ | identifies save frame header or terminator |
In addition the following reserved words may not occur as unquoted data values.
Reserved word | Role |
---|---|
loop_ | identifies looped list of data |
stop_ | terminates looped list of data |
global_ | reserved as STAR global block header |
13. The complete syntactic description of a numeric data value is included in Appendix A (section 59) under the production <Numeric>
14. The base CIF specification distinguishes between character and numeric values (see section 15 of the document "Common semantic features"). Particular CIF applications may make more finely-grained distinctions within these types. The paragraphs immediately above have the corollary that a data value such as 12 that appears within a CIF may be quoted (e.g. '12') if, and only if it is to be interpreted and stored in computer memory as a character string and not a numeric value. For example '12' might legitimately appear as a label for an atomic site, where another alphabetic or alphanumeric string such as 'C12' is also acceptable; but it may not legitimately be used to represent an integer quantity twelve.
15. Matching single or double quote characters (' or ") may be used to bound a string representing a non-simple data value provided the string does not extend over more than one line.
16. Because data values are invariably separated from other tokens in the file by white space, such a quote-delimited character string may contain instances of the character used to delimit the string provided they are not followed by white space. For example, the data item
_example 'a dog's life'is legal; the data value is a dog's life.
17. The special sequence of end-of-line followed immediately by a semicolon in column one (in C-string notation, "\n;") may also be used as a delimiter at the beginning and end of a character string comprising a data value. The complete bounded string is called a text field, and may be used to convey multi-line values. The end-of-line associated with the closing semicolon does not form part of the data value. Within a multi-line text field, leading white space within text lines must be retained as part of the data value; trailing white space on a line may however be elided.
18. A text field delimited by the \n; digraph may not include a semicolon at the start of a line of text as part of its value.19. Matching square bracket characters, '[' and ']', are reserved for possible future introduction as delimiters of multi-line data values. At this revision of the CIF specification a data value may not begin with an unquoted left square bracket character '['. (While not strictly necessary, the right square bracket character ']' is restricted in the same way in recognition of its reserved use as a closing delimiter.)
20. For example, the data value foo may be expressed
equivalently as an unquoted string foo, as a quoted string
'foo'
, as a bracketed string [foo]
or as a text field
;foo ;
By contrast the value of the text field
; foo bar ;is ` foo\n bar' (with the C-string convention of using the \n digraph to represent an end-of-line); the embedded space characters are significant.
[ foo bar]where the whitespace before the leading bracket has no significance except to separate the value from any preceding tokens, but the whitespace immediately following the leading bracket is part of the value.
21. A comment in a CIF begins with an unquoted character "#" and extends to the end of the current line.
22. Characters within a CIF are restricted to certain printable or white-space characters. Specifically, these are the ones located in the ASCII character set at decimal positions 09 (HT or horizontal tab), 10 (LF or line feed), 13 (CR or carriage return) and the letters, numerals and punctuation marks at positions 32-126.
The ASCII characters at decimal positions 11 (VT or vertical tab) and 12 (FF or form feed), often included in library implementations as white space characters, are explicitly excluded from the CIF character set at this revision.
23. The reference to the ASCII character set is specifically to identify characters in an established and widely available standard. It is understood that CIFs may be constructed and maintained on computer platforms that implement other character-set encodings. However, for maximum portability only the characters identified in the section above may be used. Other printable characters, even if available in an accessible character set such as Unicode, must be indicated by some encoding mechanism using only the permitted characters. At this revision, only the encoding convention detailed in sections 29-36 of the document "Common semantic features" is recognised for this purpose.
24. Any of the whitespace characters listed under the heading Character set (i.e. HT, LF, CR) and the visible space character SP (position number 32 in the ASCII encoding) may be used interchangeably to separate tokens, with the exception that the semicolon characters delimiting multiline text fields must be preceded by the whitespace character or characters understood as indicating an end of line (see next section).
25. The way in which a line is terminated is operating-system dependent. The STAR File specification does not address different operating-system conventions for encoding the end of a line of text in a text file. For a file generated and read in the same machine environment, this is rarely a problem, but increasingly applications on a network host may access files on different hosts through protocols designed to present a unified view of a file system. In practice, for current common operating systems many applications may regard the ASCII characters LF or CR or the sequence CR LF as signalling an end-of-line, inasmuch as these represent the end-of-line conventions supported under the common operating systems Unix, MacOS or DOS/Windows. On platforms with record-oriented operating systems applications must understand and implement the appropriate end-of-line convention. Care must be taken when transferring such files to other operating systems to insert the appropriate end-of-line characters for the target operating system. A more complete discussion is given in section 42 below.
26. Data names, block and frame codes, and reserved words are case-insensitive. The case of any characters within data values must be respected.
27. Certain allowed features of STAR File syntax have been expressly excluded or restricted from the CIF implementation.
28. Lines of text may not exceed 2048 characters in length. This count excludes the character or characters used by the operating system to mark the line termination.
29. Data names may not exceed 75 characters in length.
30. Data block codes and save frame codes may not exceed 75 characters in length (and therefore data block headers and save frame headers may not exceed 80 characters in length).
31. Only a single level of looping is permitted.
32. Save frames are permitted in CIFs, but expressly for the purpose of encapsulating data name definitions within data dictionaries. No reference to these save frames is envisaged, and the save frame reference code permitted in STAR is not used. This means that unquoted character strings commencing with the $ character may not be interpreted as save frame codes in CIF. Use of such unquoted character strings is reserved to guard against subsequent relaxation of this constraint.
33. In the full STAR specification, blocks of data headed by the special word global_ are permitted before normal data blocks. They contain data names and associated values which are inherited in subsequent data blocks; the scope of a value extends from its point of declaration in a global block to the end of the file. Because rearrangements of the order of data blocks and concatenation of data blocks from different files are commonplace operations in many CIF applications, and because of the difficulty in properly tracking and implementing values implied by global blocks, use of the global_ feature of STAR is expressly forbidden at this revision. To guard against its future introduction, the special word global_ remains reserved in CIF.
34. As an archival file format, the CIF specification is expected to change infrequently. Revised specifications will be issued to accompany each substantial modification. A CIF may be considered compliant against the most recent version for which in practice it satisfies all syntactic and content rules as detailed in the formal specification document. However, to signal the version against which compliance was claimed at the time of creation, or to signal the file type and version to applications (such as operating-system utilities), it is recommended that a CIF begin with a structured comment that identifies the version of CIF used. For CIFs compliant with the current specification, the first 11 bytes of the file should be the string
#\#CIF_1.1immediately followed by one of the whitespace characters permitted in the section Character set.
Syntactic Unit | Syntax | Case Sensitive? |
---|---|---|
Basic Structure of a CIF | ||
<CIF> | <Comments>? <WhiteSpace>? { <DataBlock> { <WhiteSpace> <DataBlock> }* { <WhiteSpace> }? }? | yes |
<DataBlock> | <DataBlockHeading> {<WhiteSpace> { <DataItems> | <SaveFrame>} }* | yes |
<DataBlockHeading> | <DATA_> { <NonBlankChar> }+ | no |
<SaveFrame> | <SaveFrameHeading> { <WhiteSpace> <DataItems> }+ <WhiteSpace> <SAVE_> | yes |
<SaveFrameHeading> | <SAVE_> { <NonBlankChar> }+ | no |
<DataItems> | <Tag> <WhiteSpace> <Value> | <LoopHeader> <LoopBody> |
yes |
<LoopHeader> | <LOOP_> {<WhiteSpace> <Tag>}+ { <WhiteSpace> <STOP_> }? | no |
<LoopBody> | <Value> { <WhiteSpace> <Value> }* { <WhiteSpace> <STOP_> }? | yes |
Reserved Words | ||
<DATA_> | {'D' | 'd'} {'A' | 'a'} {'T' | 't'} {'A' | 'a'} '_' | no |
<LOOP_> | {'L' | 'l'} {'O' | 'o'} {'O' | 'o'} {'P' | 'p'} '_' | no |
<GLOBAL_> | {'G' | 'g'} {'L' | 'l'} {'O' | 'o'} {'B' | 'b'} {'A' | 'a'} {'L' | 'l'} '_' | no |
<SAVE_> | {'S' | 's'} {'A' | 'a'} {'V' | 'v'} {'E' | 'e'} '_' | no |
<STOP_> | {'S' | 's'} {'T' | 't'} {'O' | 'o'} {'P' | 'p'}'_' | no |
Tags and Values | ||
<Tag> | '_'{ <NonBlankChar>}+ | no |
<Value> | { '.' | '?' | <Numeric> | <CharString> | <TextField> } | yes |
Numeric Values | ||
<Numeric> | { <Number> | <Number> '(' <UnsignedInteger> ')' } | no |
<Number> | {<Integer> | <Float> } | no |
<Integer> | { '+' | '-' }? <UnsignedInteger> | no |
<Float> |
{ <Integer> | { {'+'|'-'} ? { {<Digit>} * '.' <UnsignedInteger> } | { <Digit>} + '.' } } {<Exponent>} ? } } |
no |
<Exponent> | { {'e' | 'E' | 'd' | 'D'} | { '+' | '- ' } | {'e' | 'E' | 'd' | 'D'} { '+' | '- ' } } <UnsignedInteger> | no |
<UnsignedInteger> | { <Digit> }+ | no |
<Digit> | { '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' } | no |
Character Strings and Text Fields | ||
<CharString> | <UnquotedString> | <SingleQuotedString> | <DoubleQuotedString> | yes |
<eol><UnquotedString> | <eol><OrdinaryChar> {<NonBlankChar>}* | yes |
<noteol><UnquotedString> | <noteol>{<OrdinaryChar>|';'} {<NonBlankChar>}* | yes |
<SingleQuotedString> <WhiteSpace> | <single_quote>{<AnyPrintChar>}* <single_quote> <WhiteSpace> | yes |
<DoubleQuotedString> <WhiteSpace> | <double_quote> {<AnyPrintChar>}* <double_quote> <WhiteSpace> | yes |
<TextField> | { <SemiColonTextField>
|
yes |
<eol><SemiColonTextField> | <eol>';' { {<AnyPrintChar>}* <eol> {{<TextLeadChar> {<AnyPrintChar>}*}? <eol>}* } ';' |
yes |
|
|
|
WhiteSpace and Comments | ||
<WhiteSpace> | { <SP> | <HT> | <eol> | <TokenizedComments>}+ | yes |
<Comments> | { '#' {<AnyPrintChar>}* <eol>}+ | yes |
<TokenizedComments> | { <SP> | <HT> | <eol> |}+ <Comments> | yes |
Character Sets | ||
<OrdinaryChar> | { '!' | '%' | '&' | '(' | ')' | '*' | '+' | ',' | '-' | '.' | '/' | '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' | ':' | '<' | '=' | '>' | '?' | '@' | 'A' | 'B' | 'C' | 'D' | 'E' | 'F' | 'G' | 'H' | 'I' | 'J' | 'K' | 'L' | 'M' | 'N' | 'O' | 'P' | 'Q' | 'R' | 'S' | 'T' | 'U' | 'V' | 'W' | 'X' | 'Y' | 'Z' | '\' | '^' | '`' | 'a' | 'b' | 'c' | 'd' | 'e' | 'f' | 'g' | 'h' | 'i' | 'j' | 'k' | 'l' | 'm' | 'n' | 'o' | 'p' | 'q' | 'r' | 's' | 't' | 'u' | 'v' | 'w' | 'x' | 'y' | 'z' | '{' | '|' | '}' | '~' } | yes |
<NonBlankChar> | <OrdinaryChar> | <double_quote> | '#' | '$' | <single_quote> | '_' |';' | '[' | ']' | yes |
<TextLeadChar> | <OrdinaryChar> | <double_quote> | '#' | '$' | <single_quote> | '_' | <SP> | <HT> |'[' | ']' | yes |
<AnyPrintChar> | <OrdinaryChar> | <double_quote> | '#' | '$' | <single_quote> | '_' | <SP> | <HT> | ';' | '[' | ']' | yes |
|
|
|
35. The rows of this table are called "productions". Productions are rules for constructing sentences in a language. They are written in terms of "terminal symbols" and "non-terminal symbols". "Terminal symbols" are what actually appear in a language. For example, 'poodle' might be given as a string of terminal symbols in some language discussing dogs. Non-terminal symbols are the higher-level constructs of the language, e.g. sentences, clauses, etc. For example <DOG> might be given as a non-terminal symbol in some language discussing dogs. Productions may be used to infer rules for parsing the language. For example, <DOG> ::= { 'poodle' | 'terrier' | 'bulldog' | 'greyhound' } might be given as a rule telling us what names of types of dogs we are allowed to write in this language. In this table, terminal symbols (i.e. terminal character strings) are enclosed in single quotes. To avoid confusion, the terminal symbol consisting of a single quote (i.e. an apostrophe) is indicated by <single_quote> and the terminal symbol consisting of a double quote is indicated by <double_quote>. The printable space character is indicated by <SP>, the horizontal tab character by <HT>, and the end of a line by <eol>. To allow for the occurrence of a semicolon as the initial character of an unquoted character string, provided it is not the first character in a line of text, the special symbol <noteol> is used below to indicate any character that is not interpretable as a line terminator. The cases of context sensitivity involving the beginning of text fields and the ends of quoted strings are discussed below, but they are most commonly resolved in a lexical scan.
36. Productions can be used to produce documents, or equivalently to check a document to see if it is valid in this grammar. The angle brackets delimit names for the syntactic units (the "non-terminal symbols") we are defining. The curly braces enclose alternatives separated by vertical bars and/or followed by a plus sign for "one or more", an asterisk for "zero or more", or a question mark for "zero or one".
37. In most cases, each production has a single non-terminal symbol in
the syntactic unit we are defining. However, in some cases, both the
syntactic unit and the syntax begin or end with some common symbol. This
indicates that a specific context is required in order for the rule to be
applied. We do this because the initial semicolon of a semicolon-delimited
text field only has meaning at the beginning of a line, and quoted strings
may contain their initial quoting character provided the embedded quoting
character is not immediately followed by white space.
(Bracket-delimited text fields follow a different
quoting convention; see below).
This "context sensitive" notation is unusual in defining computer languages
(though very common in the full specifications of many computer and
non-computer languages). This context-sensitive notation greatly
simplifies our definitions and is simple to implement. We will elaborate
the formal definitions below.
37a. In the present revision the production for <TextField> is a trivial equivalence to <SemiColonTextField>. The redundancy is retained to permit possible future extensions to text fields, in particular the possible introduction of a bracket-delimited text value.
38. Those not familiar with the conventions used in describing language grammars may wish to consult various lecture notes on the subject available on the web, e.g. http://www.bernstein-plus-sons.com/TMM/Parsing
39. In creating a parser for CIF, the normal process is to first perform a "lexical scan" to identify "tokens" in the CIF. A "token" is a grammatical unit, such as a special character, or a tag or a value, or some major grammatical subunit. In the course of a lexical scan, the input stream is reduced to manageable pieces, so that the rest of the parsing may be done more efficiently. The convention we will follow in this document is to mark the "non-terminal" tokens that we build up out of actual strings of characters or which do not have an immediate representation as printable characters by angle brackets, <>, and to indicate the tokens that are actual strings of characters as quoted strings of characters.
40. The precise division between a lexical scan and a full parse is a matter of convenience. We will present a suggested division. Before getting to that point, however, there are some highly machine-dependent matters that need to be resolved. There must be a clear understanding of the character set to be used, and of how files and lines begin and end. The character set will be specified in terms of printable characters and a few control characters from the venerable 7-bit ASCII (the US national variant of the ISO character set). In addition we will need some means of specifying the end of a line.
41. The character set in CIF is restricted to the ASCII control characters <HT> (horizontal tab, position 09 in the ASCII character set), <NL> (new line, position 10 in the ASCII character set, also named <LF>), and <CR> (carriage return, position 13 in the ASCII character set), and the printable characters in positions 32-126 of the ASCII characters set. These are the characters permitted by STAR with the exception of VT (vertical tab, position 11 in the ASCII character set) and FF (form feed, position 12 in the ASCII character set). In general it is a poor practice to use characters which are not common to all national variants of the ISO character set. On systems or in programming languages which do not "work in ASCII", the characters themselves may have different numeric values and in some cases there is no access to all the control characters.
42. The <eol> token stands for the system-dependent end-of-line.
Implementation note: CIF implementations may follow common HTML and XML practice in handling <eol>:On many modern systems, "lines are typically separated by some combination of the characters carriage-return (#xD) and line-feed (#xA). To simplify the tasks of applications, the characters passed to an application ... must be as if the ... [parser] normalized all line breaks in external parsed entities ... on input, before parsing, [e.g.] by translating both the two-character sequence #xD #xA and any #xD that is not followed by #xA to a single #xA character." [from the XML specification http://www.w3.org/TR/2000/REC-xml-20001006].
Because Unix systems use '\n' (the ASCII LF control character, or #xA), and MS Windows systems use '\r''\n' (the ASCII CR control character, or #xD, followed by the ASCII LF control character, or #xA), and classic MacOS systems use '\r', a parser which covers a wide range of system in a reasonable manner could be constructed using a pseudo-production for <eol> such as
<eol> ::= { <LF> | <CR><LF> | <CR> }provided the supporting infrastructure (such as the lexer) deals with the necessary minor adjustment to ensure that each end-of-line is recognized and that all end-of-line control characters are filtered out from the portions of the text stream that are to be processed by other productions. One case to handle with care is the end-of-document case. It is not uncommon to encounter a last line in a document which is not terminated by any of the above-mentioned control characters. Instead, it may be terminated by the end of the character stream or by a special end-of-text-document control character (e.g. #x4 (control-D) or #x1A (control-Z)). A CIF parser should normalize such unterminated terminal lines to appear to an application as if they had been properly terminated. On the other hand, care should also be taken so that in multiple generations of CIF processing such processing does not result in an ever-growing "tail" of empty lines at the end of a CIF document.
This discussion is not meant to imply that a parser for a system which uses one of these line termination conventions must recognize a CIF written using another of these line termination conventions.
This discussion is not meant to imply that parsers on systems that use other line termination conventions and/or non-ASCII characters sets need to handle these ASCII control characters.
In processing a valid CIF document, it is always sufficient that a parser
be able to recognize the line-termination conventions of text files local
to its system environment, and that it recognize the local translations of
However, when circumstances permit, if a parser is able to recognize "alien" line terminations, it is permissible for the parser to accept and process the CIF in that form without treating it as an error.
In writing CIF documents, the software that emits lines should follow the text file line termination conventions of the target system for which it is writing the CIF documents, and not mix conventions from multiple systems. In transmitting CIF documents from system to system, software should be used that causes the document to conform to the line termination conventions of the target system. In most cases this objective can best be achieved by using "text" or "ascii" transmission modes, rather than "binary" or "image" transmission modes.
43. In order to write the grammar, we need a way to refer to the single-quote characters which we use both to quote within the syntax and to quote within a CIF. To avoid system dependent confusion, we define the following special tokens:
token | meaning |
---|---|
<SP> | ' ', the printable space character |
<HT> | the horizontal tab characters on the system |
<eol> | the machine-dependent end of line |
<noteol> | the complement of the above; any character that does not indicate the machine-dependent end of line |
<single_quote> | the apostrophe, ' |
<double_quote> | the double quote character, " |
44. There are CIF specifications not definable directly in a context-free BNF. Restrictions in record and dataname lengths, and the parsing of text fields and quoted character strings are best handled in the initial lexical scan. A pure BNF can then be used to parse the tokenized input stream.
|
|