Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [ddlm-group] Data-name character restrictions - one last time

FWIW, here is my view.

Whatever the formal standard defines, it is almost certain that many 
CIF2 implementations will allow CIF1 data names for all of the practical 
reasons defined by Herbert. Other people prefer to avoid a "messy" 
implementation, and make a strict conversion to CIF2. As Brian points 
out, there is quite a bit of diversity in CIF and pseudo-CIF 
implementations. Sometimes (often?) it is due to not reading the CIF 
specifications; that is how PDB atom name element-alignment rules got 
mutilated. Other times, it is for practical reasons to get work done.

I think a good analogy is Fortran77. Fortran language development 
stalled after that due to conflicting views on modernization versus 
maintaining traditional Fortran. Meanwhile, people needed to get work 
done, and compiler developers added many non-standard extensions. Most 
Fortran code used these extensions, and many became almost universal. 
Everyone still called it Fortran. So, maybe it really does not matter if 
"standard" CIF2 allows CIF1 data names. Everyone that wants to will do 
it anyhow, and others will run them through dictionary aliases before 
using the non-standard data files.

Fortran developed slowly due to lack of agreement, leading to many 
annoyances that were being solved quickly by newer programming 
languages. That is why Fortran is almost dead. So, the other lesson is 
that CIF needs to avoid annoying the user base, or they will just switch 
to XML, and this work will all be for nothing.

If the consensus is to keep name restrictions to promote proper 
DDLm-compatible names, it might be worth writing a formal CIF2 
extension. This would keep the non-standard naming implementations 
compatible, while still making it clear that they are not proper CIF2.

I always write code to compile flags for strict standards compliance. It 
often avoids problems, but can also prevent me from using some useful 
language extensions. Other people are happy to mandate Gnu compilers, 
and benefit from those features -- Gnu libc uses them a lot.


Herbert J. Bernstein wrote:
> Dear James,
>     With all due respect, I believe are completely, unconditionally wrong about
> this point.  The aliases in the CIF2 dictionaries will allow us to
> continue to accept both CIF1 DDL1 and CIF 1 DDL2 data files and process
> them against DDLm dictionaries, using methods and doing a better job
> at validation.  From a user point of view, we are processing all these
> CIFs with DDLm -- who gains anything by saying that something is illegal in his
> CIF?  Perhaps you are concerned that a user will mix DDL1 and DDL2
> tags in a single core CIF file.  Why is that a problem?  Whether you
> do it in one pass or 2 passes, the translation of the CIF 1 tags into
> the aliases CIF2 tags and a cleanup-up of any funny strings should
> give a valid CIF2 file for code later down the pipeline.
>    I cannot see _any_ user external in the specification of a data file
> where this distinction matters.  This is very different from the
> differences between Fortran and C where the computational model is
> very different
> making, for example, the output of f2c almost unreadably complex.
> Hopefully we are not talking about a similar difference between CIF1
> and CIF2.
>    The better comparison if not Fortran versus C, but C versus Java.
> C includes an explicit preprocessor.  Java does not.  This creates
> a nuisance for Java users, and multiple competing approaches to
> handling macros -- e.g. using m4 versus using the C preprocessor.  Both
> we and the users are better off clearly specifying the operation of
> the aliasing mechanism as part of CIF 2.
>    The alias mechanism is not something external to CIF2.  It is an
> essential part of CIF2.  I propose a very simply rule for aliases:
> that any string beginning with an underscore and not containing any
> whitespace may will be accepted by and handled by the aliasing
> mechanism and may appear as a tag in a CIF data file presented for processing
> if it is aliased to a valid CIF2 tag in the associated dictionary.
> This has zero impact on dREL methods, but it does require a clear
> agreement to provide an alias-translating front-end.
>    Regards,
>      Herbert
> At 4:13 PM +1100 12/10/09, James Hester wrote:
>> Dear Herbert and others:
>> See comments inserted below.
>> On Thu, Dec 10, 2009 at 3:01 PM, Herbert J. Bernstein
>> <<mailto:yaya@bernstein-plus-sons.com>yaya@bernstein-plus-sons.com>
>> wrote:
>> Dear James and Nick,
>>   I find this position very difficult to understand.  Please look at
>> the problem from the external point of view of user of CIF, rather
>> than the point of view of a designer of lexers and parsers, and I
>> think you will see that _externally_ we have to support tag names
>> with at least the square brackets.
>>   We are committed to being able to process existing CIF 1 files
>> using DDLm dictionaries.  It does not matter how we do this --
>> with one pass in an integrated parser, or two passes through a
>> dictionary driver alias applier, or more passes through some other
>> construct -- from the _user's_ point of view, there is a specification
>> of something called DDLm that is going to accept files that
>> conform to the existing DDL1 and DDL2 dictionaries and that
>> also accepts files that conform to the DDLm specification.  We
>> accomplish nothing useful by tell such a user that yes, we will
>> accept CIFs with tags that contain square brackets from the CIF1
>> core dictionary, but no, such tags are not legal in CIF2.  Be the
>> user.  He will think we are insane.
>> You go on to say:
>>     "We are going to accept the old CIF1 tags, no matter whether a
>>     particular parser accepts them, they are a legal part of the overall
>>     CIF2 system -- deprecated perhaps -- but legal. "
>> If I translate from Fortran to C, does that make Fortran part of my
>> C environment?  I think not.  In the same way, just because a route
>> exists to use CIF1 data files in a CIF2 environment, that does not
>> mean that CIF1 'is part of' CIF2.
>> Now, let's think about the 'users' more specifically.  There are the
>> CIF writing contingent (single crystal software authors, for
>> example) who will be told: you can continue to use CIF1 datanames as
>> defined in DDL1/2 dictionaries and from an external point of view
>> these will be processed as before (why is this insane?).
>> Alternatively, you can use these CIF2 datanames as replacements, and
>> the files will still be processed in the same way (this is also not
>> insane).   We do not have to tell the 'users' not to use square
>> brackets, we simply tell them about replacement datanames. You seem
>> to be arguing that having data name aliases is in itself crazy?
>> By the same token, we require of CIF readers nothing more than an
>> acceptance of the aliasing protocol, which has been around for a
>> while.
>>    The user will have
>> no trouble if we say that we are moving away from tags with such
>> characters.  The user would not even have a problem if we said
>> that we would not be able to do as good a job validating files
>> with such tags as we could for ones with the new tags, but if the
>> alias mechanism works properly, even that would be untrue.
>>   From the user point of view, nothing but confusion is results if
>> we tell him that tags in existing official CIF1 dictionaries
>> with square brackets are illegal -- they won't be.
>> Well, I would be confused too if somebody told me that something was
>> illegal in CIF1 when it wasn't.  CIF2 is not part of CIF1, but it
>> does maintain backward compatibility with CIF1.  There is a
>> difference.
>>   No, the only people who need to hear about this character restriction
>> and understand it are CIF2 DDLm dictionary writers, who need to be
>> told:
>>   "If you are creating a new tag, would would be well advised not to
>> include square brackets in the name because CIF2 requires somewhat
>> complex alias mechanisms to handle such tags."
>>    The entire argument you are making sounds like an argument against
>> fully supporting the alias mechanism.  That simply will not fly.
>> The aliases of the old CIF1 tags have to be fully documented
>> and supported in CIF2.  You can say that the valid CIF 2 tags with the
>> restricted character set are the only tags that will appear unquoted
>> in a CIF2 dictionary, and that they are the only tags that will
>> appear in a dREL method, but that is _not_ something the users will care
>> about -- only the dictionary writers.
>>    To be blunt the entire argument about "simplicity" or being
>> "maximally disruptive" is, in my opinion, misguided and uncoupled
>> to the more important objectives to providing something useful
>> and comprehensive to our users.  Issues such as simplicity and
>> zero-based design (which is as much of maximally disruptive
>> design as I can swallow) are important _only_ if reasonable
>> user externals are achieved.  In this case these peripheral issues
>> are getting in the way and must be sacrificed to the real needs
>> of our users to get work done.
>>   Regards,
>>     Herbert
>> At 1:58 PM +1100 12/10/09, James Hester wrote:
>>> Dear All,
>>> I've just had a discussion with Nick around the dataname characterset
>>> issue, and have elicited a strong argument from him against
>>> liberalising the character set.  First, let me state the case for
>>> *not* restricting the character set in CIF2 datanames:
>>> 1. Characters in a dataname such as forward slash, square brackets,
>>> hyphen and most of UTF8 do not break the other CIF2 syntax that we
>>> have agreed on, because:
>>> (a) datanames cannot appear inside lists
>>> (b) Lexers will not misinterpret square brackets etc. inside
>>> datanames, as a dataname must be separated from any succeeding token
>>> by whitespace. All characters up to that whitespace must therefore
>>> either belong to the dataname, or be syntax violations.
>>> 2. If we allow datanames with these 'extra' characters, we can have
>>> all CIF1 names in a CIF2 file, improving backwards compatibility
>>> The argument *against* including these characters involves looking at
>>> the whole picture.  This whole CIF2 effort is motivated by the need to
>>> add list structures and to define a simple dictionary method language
>>> which can manipulate data items, including these lists.  So, an
>>  >argument against including these characters runs as follows:
>>> 1. Almost any liberalisation of the dataname characterset will
>>> necessarily force additional complexity in the dictionaries, because
>>> the datanames are used as identifiers in dREL methods.  This
>>> complexity manifests either through extra aliases, or a new dREL
>>> 'quote' function, or category/object names that do not match the
>>> data name.  This complexity makes it both more difficult to write
>>> dictionaries, as one must keep track of the 'true' name, and more
>>> difficult for a human reader to read and check dictionaries, as it is
>>> not necessarily easy to find out which 'real' dataname a dREL method
>>> is referring to in its derivation.  The dREL 'quote' function idea
>>> makes writing dictionary methods potentially more complex, when the
>>> intention is that these methods should be maximally accessible to
>>> non-programmers for both reading and, in particular, construction.
>>> 2. While it would be possible to simply promulgate a general principle
>>> that DDLm dictionaries cannot define datanames using characters
>>> outside a restricted characterset, a more robust approach is to reduce
>>> the possibility of their appearance by making them syntactically
>>> forbidden.
>>> Which of these lines of argument you favour comes down to how highly
>>> you value simplicity in the overall design compared to how much you
>>> value compatibility at the syntactical level with CIF1, given that
>>> workarounds for compatibility are possible at the dictionary level.
>>> As we are prepared for CIF2 to be a disruptive but clean
>>> change, I would favour simplicity and therefore keeping a restricted
>>> character set.
>>> On Thu, Dec 10, 2009 at 6:29 AM, David Brown
>>  ><<mailto:<mailto:idbrown@mcmaster.ca>idbrown@mcmaster.ca><mailto:idbrown@mcmaster.ca>idbrown@mcmaster.ca>
>> wrote:
>>> I would suggest that we add CIF2 data namea as aliases in the DDL1
>>> and DDL2 dictionaries for those few items where the names differ.
>>> This would mean that any DDL1 dictionary would recognize all the
>>> CIF2 data names that corresponded to items appearing in the DDL1
>>> dictionary.  Of course files with arrays could not be read this way,
>>> and adding arrays to DDL1 dictionaries would violate the DDL1 rules
>>> and would essentially convert the DDL1 dictionaries into
>>> non-conforming DDL2 dictionaries, thus defeating the goal of being
>>> able to read CIF2 data files with DDL1 software.  Adding DDLm
>>> aliases to the DDL1 dictionaries would be easy since we would need
>>> to add less than a dozen aliases (but we would have to know what the
>>> DDLm data name is, or is going to be).  Of course there is also the
>>> _ versus . problem with the data names.  Adding '.' data names as
>>> aliases in the DDL1 dictionaries would get around that problem.  It
>>> would be straightforward to add these, but it would be a larger job
>>> since all the data names would need an added alias.  Even this will
>>> not help with legacy software using hard coded data names, but this
>>> might just encourage people to write a front-end that uses the
>>> dictionaries for input.
>>> There is still the problem of the data names in DDL2 dictionaries
>>> that include [].  Adding an alias name that does not use these
>>> characters may allow DDL2 programs to read CIF2 data files, but CIF
>>> data files containing these [] data names would require a CIF1
>>> parser (and lexer?), so we just need to recognize this fact and live
>>> with it.  In any case a CIF1 lexer should always be an optional
>>> front-end to a DDLm dictionary if it is to read in legacy data files
>>> as required by the specifications.
>>> It would not be straightforward for a DDLm program to output a CIF1
>>> data file, but is this really necessary?  Once one has started to
>>> use the features of CIF2 one would probably wish to output items
>>> that do not even exist in CIF1.  If one needed a fully compliant
>>> CIF1 data file in order to make use of legacy software, it might be
>>> better to write a CIF2 file, but restrict the items to those that
>>> exist in the DDL1 (or 2) dictionaries (this information can be found
>> >from the aliases in the DDLm dictionaries).  The DDLm data names
>>  >that would appear in this data file are either identical to the DDL1
>>> data names or would appear as aliases to the DDL1 dictionary as
>>> described above.
>>> As for recognizing CIF2 data files, isn't that what the magic code
>>> is for?  If someone chooses not to use the magic code their CIF2
>>> data file is non-conforming and they can expect difficulties.  Most
>>> of the CIFs in DDL1 are initially prepared by computer and only the
>>> text being added by hand.  Once the computers start generating CIF2
>>> data files, they will be programmed to add the magic code.
>>> David
>>> Herbert J. Bernstein wrote:
>>>> Personally, I would greatly prefer to allow all data names that do not
>>>> create a major lexer/parser conflict to appear in a data CIF and
>>>> only apply the strong restrictions to data names that appear in CIF2
>>>> dictionaries as defined data names (not as aliases).  -- Herbert
>>>> At 2:40 PM +0000 12/9/09, Brian McMahon wrote:
>>>>> I have one remaining niggle that I'd like to revisit before we put
>>>>> this finally to bed. As has been mentioned a couple of times
>>>>> recently, restricting the data-name character set does invalidate
>>>>> syntactically many existing CIF 1 files (e.g. _refine_ls_shift/esd_max ).
>>>>> We have discussed strategies for handling this, and I think these
>>>>> are workable strategies, but will involve investment and hence expense
>>>>> in workflow management in CIF archives.
>>>>> I understand the rationale behind this restriction is to simplify
>>>>> future processing of data names in areas such as dREL
>>>>> applications. The question really is whether we're choosing the right
>>>>> trade-off in making things cleaner at that end of the processing
>>>>> chain. I would suppose that a dREL or other application could ingest a
>>>>> data name with dangerous characters, convert it internally into a
>>>>> "safe" identifier that's used for all processing, and then restore the
>>>>> original form upon output; but writing that intermediate layer of
>>>>> processing is of course expensive (especially if there aren't readily
>>>>> available libraries that will do this transparently).
>>>>> I suspect that some of the original proposed syntactic changes also
>>>>> had the effect (whether by design or collaterally) of simplifying i/o,
>>>>> data structure management, symbol table processing etc., but those may
>>>>> have suffered in the subsequent revision exercise we've just been
>>>>> practising. Given the consensus we are now approaching, would the code
>>>>> builders now be prepared to incur the addition expense of handling
>>>>> "dangerous" data names?
>>>>> I really don't want to spark off a long discussion on this - if a
>>>>> quick round of response shows that there's no appetite to allow
>>>>> the additional punctuation characters in data names, I'll accept that
>>>>> gracefully.
>>>>> ***
>>>>> One last comment while I have the floor, though it is related in part
>>>>> to the above question. A concern raised in the editorial office was
>>>>> that there would be circumstances where users didn't know if they were
>>>>> dealing with a CIF 1 or 2 ("users" meaning authors, perhaps resorting
>>>>> to the vi editor - and we're imagining most of them are dealing with
>>>>> small-molecule/inorganic CIFs). My supposition is that the IUCr
>>>>> editorial offices would only want to use CIF2 seriously in association
>>>>> with DDLm dictionaries, and that we would expect the revised core
>>>>> dictionaries to use the dot component in data names to signal this
>>>>> further evolution. So even a superficial glimpse of the middle of a
>>>>> CIF would make it clear whether it was CIF1 or CIF2.
>>>>> Does that fit in with how others see this progressing?
>>>>> Cheers
>>>>> Brian
ddlm-group mailing list

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.