Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [ddlm-group] options/text vs binary/end-of-line. .. .. .

Dear Herbert

I have to confess to not entirely understanding your proposed description.

Two questions:

1) "all fully compliant CIF2 processing systems should, at a minimum be able to process
text files as unicode code points represented in UTF-8"

Does this mean that a fully compliant CIF2 processing system need *not* accept text encoded in anything else?

More importantly:

2) What exactly do you mean by

 "it is important to clearly specify the intended mapping to UTF-8" ?



From: Herbert J. Bernstein <yaya@bernstein-plus-sons.com>
To: Group finalising DDLm and associated dictionaries <ddlm-group@iucr.org>
Sent: Tuesday, 22 June, 2010 18:40:47
Subject: Re: [ddlm-group] options/text vs binary/end-of-line. .. .. .

Dear Colleagues,

  Except when I find the time to work with hardware, much of the science
I do ends up involving a great deal of editing of documents -- and it
is a royal waste of time to tell somebody to learn new editing habits
without a very good reason, so it is very much the case the such
mundane issues as encodings and keyboard layouts are a large factor
in how science gets done by many people.

  Most people don't even realize how many different text encodings
they use and how different the text encodings used by their colleagues
may be.  In going from system to system, e.g. by email, the translations
among encodings are close to invisible.

  Instead of focusing on the change document, could we please focus
on what the CIF2 specification as a complete, coherent document should
say.  Taking into account what has been said thus far, here is a
slightly revised version of what I proposed:


CIF2 is a specification for the interchange of text files.  Text files
have many possible system dependent representations and encodings.  To
ensure clarity in the specification of CIF2, this document is written
in terms of a sequence of unicode code points, and all fully compliant
CIF2 processing systems should, at a minimum be able to process
text files as unicode code points represented in UTF-8, subject to the
XML-based restrictions below.  This approach is not meant to prevent
people from preparing valid CIF2 files with non-UTF-8-based text
editors, but, if a non-UTF-8 file format is produced, it is important
to clearly specify the intended mapping to UTF-8.  Almost all modern
systems have available a standard mapping from their internal text
representation to and from UTF-8.

Special care is needed in dealing with end-of-line indicators (see
http://en.wikipedia.org/wiki/Newline).  This document will only
refer to LF (line feed or newline) as the line terminator.  When handling
CIF2 files produced under MS windows, CR-LF sequences should be accepted as
an alternative to LF, and when handling CIF2 files produced under
Mac OS, CR should be accepted as an alternative to LF.  The safest policy
is to accept any of CR-LF or CR or LF and line terminators if possible,
and to map all of them to LF on reading a CIF.  Systems with other,
additional line terminators should avoid introducing them into CIF2
files meant for interchange.

To ensure compatibility with older Fortran text processing software,
lines in CIF2 files should be restricted to no more than 2048
code points in length, not including the line terminator itself.
Not that the UTF-8 encoding of such a line may well be much longer."

At 4:13 PM +0000 6/22/10, SIMON WESTRIP wrote:
>Perhaps John's compromise might be the way forward?
>From: "Bollinger, John C" <John.Bollinger@STJUDE.ORG>
>To: Group finalising DDLm and associated dictionaries <ddlm-group@iucr.org>
>Sent: Tuesday, 22 June, 2010 16:15:36
>Subject: Re: [ddlm-group] options/text vs binary/end-of-line. .. .. .
>I prefer leaving the issue of character encoding entirely out of the
>scope of the CIF format specification (effectively allowing any
>encoding).  On the other hand, I think it's a bit of an
>aggrandizement to characterize UTF-16 / Shift-JIS / etc. as "ways in
>which many of our colleagues get their science done."  In no way do
>I dispute that many of our colleagues indeed use these encodings
>routinely, but I am doubtful that editing Unicode text with a text
>editor constitutes a significant part of many of their research
>programs.  At least, few of my English-speaking colleagues edit flat
>Unicode text files with any frequency, if ever they do at all.
>I think there is already good software, some of it free (both
>senses), for operating systems at least as old as Windows 9x, that
>supports editing UTF-8 encoded text.  Most of it also supports a
>multitude of other encodings.  We would leave no one out by
>requiring UTF-8, and I do not see that respect for our colleagues
>demands that CIF2 be equally convenient to create and edit with
>every text editor in current use.  If that is doubtful, however, and
>respect is our goal, then wouldn't the most respectful thing be to
>*ask* a few of the people about whom we are concerned?
>My issue here is different, and at least partly philosophical.  The
>CIF format can and should be about the structure and meaning of CIF
>text content.  Character encoding is on a different level: it's a
>characteristic of storage and interchange.  Comingling these layers
>is inelegant and unnecessary.
>Moreover, a CIF2 requirement to encode in UTF-8 will be small
>comfort when presented with a file that is not, in fact, encoded
>that way.  What can you then do?  Either reject the file or
>autodetect the encoding.  If CIF2 does not specify a particular
>encoding, and you receive the same file, then what can you do?
>Exactly the same things, but then it's more likely that the file's
>provider will have also specified the encoding by some means.
>(Particularly so if the CIF2 spec calls attention to the need to do
>Perhaps something like this would be an acceptable compromise:
>a) Rewrite change 2 to remove the requirement for UTF-8
>b) Add:
>CHANGE 9 - NEW (CIF Interchange Format)
>Many alternative encodings are available for recording and
>exchanging Unicode character data via byte-oriented media.  The CIF
>format itself is encoding independent, but that allows for
>uncertainty as to how to handle putative CIF data unaccompanied by
>encoding information.  We therefore define a simple, binary CIF
>Interchange Format, consisting of CIF2 text encoded in UTF-8, with
>an optional initial UTF-8 byte-order mark.  CIF Interchange Format
>is intended as a storage and interchange standard for CIF2.  Its use
>is strongly encouraged, but its existence should not be taken as a
>prohibition against use of alternative storage and interchange
>formats among agreeing parties.
>The standard file name extension for CIF Interchange Format files is .cif.
>John C. Bollinger, Ph.D.
>Department of Structural Biology
>St. Jude Children's Research Hospital
>Email Disclaimer:
>ddlm-group mailing list
>ddlm-group mailing list

  Herbert J. Bernstein, Professor of Computer Science
    Dowling College, Kramer Science Center, KSC 121
        Idle Hour Blvd, Oakdale, NY, 11769

ddlm-group mailing list
ddlm-group mailing list

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.