[Cif2-encoding] The discussion so far

Thanks, Brian, for creating this list.

Since no one else has had the combination of time, energy, and inclination to do so, I’ll open with a summary of the state of the CIF 2.0 character encoding discussion so far, as it currently stands at its original location on the DDLm group.  Specifically, the previous summary and the discussion that proceeded from it can be found at http://www.iucr.org/__data/iucr/lists/ddlm-group/msg00744.html and http://www.iucr.org/__data/iucr/lists/ddlm-group/msg00690.html, though some of the most recent messages seem not to be available at present via the web interface.

The controversy derives from CIF 2.0's expansion of character set to all of Unicode.  It is magnified by CIF 1.0's explicit self description as an encoding-independent text format, and by the accumulated body of CIF software and author practices that rely on that text orientation.  There has been considerable debate about what it would mean for CIF 2.0 to be a text format vs. a binary format, and the relative advantages and disadvantages of each.  Among the points covered were:

1) A 'text' format implies that CIF content may comply with local, locale-specific conventions for electronic text representation, including details such as line termination conventions and, especially, character encoding.  Such files are suitable input for general-purpose text tools such as text editors, text extraction utilities, and text indexers.  Alternatively, a conformant text CIF might be expressed according to some other convention suitable for a particular application or foreign environment.  Because conventions differ, correctly archiving text or moving it between environments requires accounting for the text conventions in use, and may involve conversions such as line terminator changes and transcoding.  This is the CIF 1.0 position, though CIF1's restricted character set significantly reduces the impact of character encoding considerations relative to CIF2.

2) A 'binary' format is anything else, but in this context, the key characteristic of binary-CIF2 proposals is that they add to the text specification a specification for text serialization to byte-oriented media, such as disks and networks.  In particular, one strongly advocated position in the CIF 2.0 standardization discussion is that CIF 2.0 should require serialization of the underlying CIF text according to the UTF-8 character encoding scheme.  This would be a text-like binary format, in that some text tools can handle UTF-8 encoded text (sometimes requiring a little persuasion), and therefore could be used to read, modify, and write binary-CIF2 files.

3) The many specific issues and arguments that have been raised mostly fall into one or both of two general areas:

3a) reliability, by which we mean that a CIF consumer should have justifiable high confidence that he is interpreting CIF data in the way the CIF producer intended, and

3b) usability, by which we mean that human authors, and to a lesser extent, software, should be able to manipulate CIF2 files as they are accustomed to manipulating CIF1 files, e.g. using the default configurations of their systems’ text editors.  In addition to general usability, arguments in this category include some appealing to respect of scientists of many nationalities, and similar ones appealing to freedom / liberty.

Furthermore, a few of the points have appealed to

3c) practicality, by which we mean that CIF stakeholders should be able to use the specification effectively.  This aspect is subordinate to reliability and usability, but it cuts across both.  For the most part, it relates to the ability to develop software and practices that address the likely real-world usage (and misusage) of the standard.

4) The group as a whole appears to have agreed that UTF-8 is a highly suitable encoding for CIF2.  It can encode the entire Unicode code space, it is a superset of ASCII, it can be recognized heuristically with low probability of error, and it is widely implemented.  These characteristics yield high reliability at the cost of some usability.  The debate is not about whether UTF-8 should be used, but rather about whether the standard should forbid use of other encodings.  Essentially, this is a recasting of the text vs. binary debate.

5) Inasmuch as consensus on the issues described above has not yet been reached and does not appear likely, the group has issued a call for comments from a wider group of stakeholders.  No results of that call have yet been reported back to the group.

6) In the interim, the discussion has moved toward finding middle ground.  In particular, James Hester asked:

>If we consider CIF as text as the overriding priority:
>
>1. How do we then make exchanging and storing files according to text conventions sufficiently reliable for the purposes of CIF?  How far are we prepared to compromise?
>
>If we consider reliable exchange of information as the top priority:
>
>2. How do we then make CIFs sufficiently accessible to text-based tools?  How far are we prepared to compromise?

A short series of proposed schemes for CIF exchange and storage proceeded from that call:

7) In response to question (2), James offered a scheme (A) that primarily would relax the explicit specification of UTF-8 into a set of characteristics that a CIF encoding would need to satisfy.  His characteristics would need to be further pared down or relaxed to in practice permit encodings other than UTF-8.  This scheme will be reproduced in a separate e-mail, as it is not currently available from the DDLm-list web archive.

8) In response to question (1), John Bollinger offered a scheme (B) that retains text character for CIF2, and relies on labeling when wanted or needed to convey text conventions, and on hashing to provide verification and reliability.  This scheme will be reproduced in a separate e-mail, as it is not currently available from the DDLm-list web archive.

A limited amount of additional discussion proceeded from these proposed exchange and archiving schemes, and that’s where we currently stand.

Regards,

John

--

John C. Bollinger, Ph.D.

Department of Structural Biology

St. Jude Children's Research Hospital