Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [ddlm-group] [THREAD 4] UTF8

Dear Colleagues,

   Brian has asked about UTF-8 and old Fortran systems.  Almost all
Fortran system still in use can deal with UTF-8 becuse most of them
are 8-bit systems, so the extra bytes in the multi-byte sequences
just look like any other printable charcaters.  There are still some
systems around that use 7 bit characters or odd byte sizes.   There
is a UTF-7 that could work, but it is messy and not recommended.
For those systems and entrernal econcoder/decoder would be the best
choice.  I think they wuld rarely be needed.

Most of the world in now on 8-bit character systems, even for Fortran.
Certainly Fortran 95/2003 makes that assumption.

In any case, at the moment, Fortran (old and new) still does a bad job in 
handling trailing blanks on physical lines.  The fix (the Q edit 
descriptor) was simply not widely accepted.  Maybe when we revisit this in 
a few years the world of Fortran will be different, but not yet.

The simple cure (just use C-I/O) is not acceptable to certain major
crytallographic application developers.  On that, too, there are fixes
in the wings, but not solid yet.

For now, I think we need to include trailing blank stripping in the spec.

   Regards,
     Herbert
=====================================================
  Herbert J. Bernstein, Professor of Computer Science
    Dowling College, Kramer Science Center, KSC 121
         Idle Hour Blvd, Oakdale, NY, 11769

                  +1-631-244-3035
                  [email protected]
=====================================================

On Tue, 13 Oct 2009, Brian McMahon wrote:

> Without having had time to analyse it in detail, I like the
> pragmatic feel of much of what Herbert says.
>
> But I wonder about the reference to "old fortran systems":
>
>> Here, past practice with CIF rears its head -- what do we do with trailing
>> white space?  In CIF until now, in order to deal with old fortran systems,
>> we have assumed that we cannot tell the difference between lines that end
>> with one blank or with an arbitrary number of blanks...
>
> We've ascertained that "modern" Fortran systems can accommodate UTF-8
> byte streams - can the "old" ones? In other words, if the principle of
> maximal disruption applies and we accept UTF-8, are we justified at
> the same time in sacrificing compatibility with such "old"
> Fortran-based systems? And if so, does that allow a different
> handling of "physical lines" ?
>
> Regards
> Brian
>
>
>
> On Tue, Oct 13, 2009 at 10:09:18AM -0400, Herbert J. Bernstein wrote:
>> Dear Colleagues,
>>
>>    Let us "zero-base" this dicsussion and consider just the lexical
>> analysis appropriate to some future CIF-like language.  Let us look at
>> some of the lexical issues that python deals with and consider what
>> lessons we may learn there in trying to go from a string of characters
>> to a string of tokens.
>>
>>    First, we need to settle on what characters we will be using.
>> Origincally, python restricted its attention to just 7-bit ascii
>> characters "for program text."  Now (from version 2.3 onwards), python
>> allows "an encoding declaration [to be] used to indicate that string
>> literals and comments use an encoding different from ASCII".
>>
>>    I propose that we do something similar, but with a more modern starting
>> point:
>>
>> new cif character set and encoding:
>>
>>    C1:  that the character set for a "new cif" be unicode, and
>>    C2:  that the default encoding be UTF-8; and
>>    C3:  that other encodings be permitted as an optional
>> system-dependent feature when an explicit encoding
>> has been specified by
>>      C3.1:  a unicode BOM (byte-order-mark) (see
>> http://en.wikipedia.org/wiki/Byte-order_mark) has been introduced
>> into a character stream, or
>>      C3.2.  the first or second line being a comment of the form:
>>        # -*- coding: <encoding-name> -*-
>>      as recognized by GNU Emacs, or
>>      C3.3.  the first or second line being a comment of the form:
>>        # vim:fileencoding=<encoding-name>
>>      as recognized by Bram Moolenaar's VIM
>> (see section 2.1.4 of
>> http://docs.python.org/reference/lexical_analysis.html for a more
>> information).
>>
>> For the rest of this discussion, let us assume unicode conventions
>>
>>
>> Next, we need to decide on the rules for handling lines breaks.  I would
>> suggest we follow the pythn convention of first considering "physical
>> lines" and then introduce rules for joinng those physcial lines into
>> "logcal lines".
>>
>> Here, past practice with CIF rears its head -- what do we do with trailing
>> white space?  In CIF until now, in order to deal with old fortran systems,
>> we have assumed that we cannot tell the difference between lines that end
>> with one blank or with an arbitrary number of blanks. Many fortran
>> implementations do not support an clean way to detect end of line, and,
>> worse, have no way to cope with lines of arbitrary length. We also still
>> have the system-dependent definitions of line termination.  For our
>> "customer-base" I do not see any practical way around this right now, so,
>> with regret, I propose
>>
>>    physical line:
>>
>>    PL1: In describing the lexer, the system-dependent end-of-line will be
>> given a '\n'.  In source files, any of the standard platform line
>> termination sequences can be used - the Unix form using ASCII LF
>> (linefeed), the Windows form using the ASCII sequence CR LF (return
>> followed by linefeed), or the old Macintosh form using the ASCII CR
>> (return) character. All of these forms can be used equally, regardless of
>> platform.  I addition, all space and tab charcaters, '\x20' '\x09',
>> immediately prior to the system-dependent end-of-line will be removed
>> prior to further lexical analysis; and
>>    PL2: There may be a system-dependent limit on the maximal length
>> of the resulting line, but in all cases, lines of up to 2048 charcaters
>> will be accepted.
>>
>>    comments:
>>
>>    LC1:  A comment starts with a hash character (#) that is not part of a
>> string literal, and ends at the end of the physical line. A comment
>> signifies the end of the logical line unless the implicit line joining
>> rules are invoked. Comments are ignored by the syntax; they are not
>> tokens.
>>
>>    logical line:
>>
>>    LL1:  A logical line is constructed from one or more physical lines by
>> following explicit or implicit joining rules
>>    LL2:  Explicit line joining:  Two or more physical lines may be joined
>> into
>> logical lines using reverse solidus characters (\), as follows: when a
>> physical
>> line ends in a reverse solidus that is not part of a string literal or
>> comment,
>> it is joined with the following forming a single logical line, deleting
>> the backslash and the following end-of-line character.
>>    LL2.  Implicit line joining: Expressions in parentheses, square brackets
>> or curly braces can be split over more than one physical line without
>> using backslashes.  Implicitly continued lines can carry comments. Blank
>> continuation lines are allowed. There is no end-of-line token between
>> implicit continuation lines. Implicitly continued lines can also occur
>> within triple-quoted strings (see below); in that case they cannot carry
>> comments.
>>
>> Strings
>>
>>    With the character stream and the lines defined, the next thing we need
>> to define are string.  I propose we adopt a subset of the python
>> convention, but without the string prefixes.  :
>>
>> String literals can be enclosed in matching single quotes (') or double
>> quotes ("). They can also be enclosed in matching groups of three single
>> or double quotes (these are generally referred to as triple-quoted
>> strings). The reverse solidus (\) character is used to escape characters
>> that otherwise have a special meaning, such as newline, backslash itself,
>> or the quote character.
>>
>> In triple-quoted strings, unescaped newlines and quotes are allowed (and
>> are retained), except that three unescaped quotes in a row terminate the
>> string. (A quote is the character used to open the string, i.e. either '
>> or ".)
>>
>> There is more to define, but if we go this far, we should be able to
>> have fairly clean lexical scanners that are able to handle nested
>> quotation marks in a way that most programmers will understand.
>>
>> Regards,
>>     Herbert
>>
>> =====================================================
>>   Herbert J. Bernstein, Professor of Computer Science
>>     Dowling College, Kramer Science Center, KSC 121
>>          Idle Hour Blvd, Oakdale, NY, 11769
>>
>>                   +1-631-244-3035
>>                   [email protected]
>> =====================================================
> _______________________________________________
> ddlm-group mailing list
> [email protected]
> http://scripts.iucr.org/mailman/listinfo/ddlm-group
>
_______________________________________________
ddlm-group mailing list
[email protected]
http://scripts.iucr.org/mailman/listinfo/ddlm-group

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links