Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [ddlm-group] Finalizing DDLm

Herbert, it seems to me that both of your issues are not relevant to this discussion, in that they refer to situations for which DDLm is not used.  First, a clarification.  When I talk about a dictionary being 'available' to a program, I have in mind that it could be available at program writing time (i.e. available to the programmer) and/or at program running time.  I hope this corresponds with other peoples' usages.

On Wed, Apr 14, 2010 at 9:23 PM, Herbert J. Bernstein <yaya@bernstein-plus-sons.com> wrote:
Inasmuch as we appear to be discussing, rather than voting, please allow me to clarify my position:

I am _not_ concerned about whether a DDLm-conformant dictionary does or does not have rules to say that a particular category is or is not allowed to be presented "unrolled".  I am concerned with how to handle two important cases:

 1.  Existing legacy data files that have "rolled" or "unrolled" loops that do not conform to the new dictionary rules; and

Those legacy data files were written with a particular dictionary in mind.  If that dictionary DDL allows loop unrolling (i.e. DDL2) then any application that presumes to read datafiles based on that dictionary will need to support it.  But what we are discussing is how to specify the construction of data files written with respect to a DDLm (not DDL2) based dictionary.  So I don't see how your case (1) is relevant.

 2.  Applications that are confronted with a data file, portions of which are not in dictionaries to which that application has access.

If an application has no access to the dictionary relevant to a given dataname, it cannot be compelled to issue an error or warning when confronted with an unrolled loop, because it has no way of knowing that the loop is unrolled.   In such a situation it would be bizarre to specify any dictionary-derived behaviour, and I am not proposing to do so.  Likewise, if a CIF-writing application has no dictionary information about a dataname that it is writing, we are unable to impose any dictionary-based behaviour.   The latter is a fairly 'Alice in Wonderland' situation: a program writing a dataname that neither it (nor the programmer) knows nothing about...

If an application has a dictionary handy and that dictionary says something relevant about the rolledness or unrolledness of a loop, then I am reluctantly willing to accept the DDLm specification requiring the issuance of a warning or an error message.  Some application writers may decide not to do that, but that is a different discussion.

What I am concerned about is the very practical issues above -- of doing something useful with user data that either does not conform to this stricture as presented in a dictionary or on which the dictionaries available to the application are silent.  I am proposing that, rather than requiring an application to throw up its hands and die, we try to maximize the useful work to be accomplished and try to do something sensible with the data, i.e. roll that which is unrolled or unroll that which is rolled, if it allows the work of the application to get done.

If an application has no access to a dictionary for the datanames, it will not be able to roll up an unrolled loop, as it won't know what datanames should be in the loop.  So I would make a counter-suggestion that (in order to get useful work done), we can help this dictionary-challenged program by making sure all datafiles that it is presented with have their loop structures left intact.

I have yet to hear of a reason not to adopt that approach for the cases listed above.  Once we have those two cases settled, I would be happy to discuss the subtleties of whether the List attribute itself should be modified or not, but first, please, let us deal with this practical issue.


P.S. to Nick:  It is the current DDLm specification that would require every application writer to read the dictionary in order to process a CIF, else we would have no way to tell whether rolled or unrolled presentation was in conformance with the dictionary.  The list attribute is in the dictionary, not the data file.  The discussion we are having is orthogonal to the question of whether the DDLm specification requires the reading of the dictionary.

I think this is the wrong way around.  *If* an application writer wants to see if a key-value dataitem should be instead in a loop, *then* they will need to read the dictionary.  If they can do useful work without knowing this information, then I'm not standing in their way.  A program which claims to validate a data file *cannot* do the work it was designed for unless it reads the dictionary, and must flag unrolled loops as a violation of the standard.  It may then offer to roll up the loops, to create a conformant file.  What is the problem here?

P.S. to James:  I have read Nick's argument and on the DDLm specification
issue and stick to voting for 2.  If we change the specification then
strict adherence will no longer require List categories to be presented
as looped data, and no more or less dictionary reading will be required
than is required by the current specification, but users will be
annoyed by one less warning/error message they are not likely to understand or be able to do anything about.  However, no matter how
that vote comes out, we really do need to deal with the practical issue
above -- there are an awful lot of PDB mmCIF data files.

PDB mmCIF files are not an issue for DDLm *at all*, as the mmCIF data files are written with respect to the DDL2 specification (not DDLm).  If and when a DDLm version of mmCIF appears, conversion of legacy files will involve an order of magnitude more work than just rolling up unrolled loops, so the outcome of the present discussion will be by comparison background noise.

 Herbert J. Bernstein, Professor of Computer Science
  Dowling College, Kramer Science Center, KSC 121
       Idle Hour Blvd, Oakdale, NY, 11769


On Wed, 14 Apr 2010, Nick Spadaccini wrote:

This doesn?t actually make things clearer or easier James. I will repeat it again.

Strict adherence to the formal specification of DDLm REQUIRES List categories to be presented as looped data, even
if there is only one row.

If the IUCr wishes to universally adopt the case where instances of List categories that contain only one row may
be presented as a Set category then it can do so as an accepted extension. This of course requires every
application writer to necessarily read the dictionary to establish if the data is really a Set category or
possibly a List category. Once the IUCr adopts this extension to DDLm within its implementation, I would assume
every application writer would be required to adhere to it.

On 14/04/10 3:29 PM, "James Hester" <jamesrhester@gmail.com> wrote:

     Dear all,

     Both John and Herb have come out in favour of allowing one-row loops to be unrolled.  Nick and I are
     both sceptical about the value of this idea.  We have a few options:

     1.  Disallow loop unrolling altogether (as in DDL1).
     2.  Allow loop unrolling for all DDLm dictionaries
     3.  Add a category-scope DDLm attribute stating that one-row loops in this category and child
     categories may be unrolled.  If it appears in the 'Head' category of the dictionary, it would mean
     that all categories in the dictionary could be unrolled.

     We have not discussed option 3: it basically means deferring the decision on loop unrolling to the
     dictionary writers.  It also means that programmers of generic CIF software will need to be prepared
     for either behaviour, so in that sense it is slightly more burdensome than option 2.

     Unless the silent majority would like to contribute further thoughts on this matter, I suggest that we
     vote and move on.  I discern that the voting so far would be:

     Option 1: James, Nick
     Option 2: John, Herb
     Option 3: ?

     (Some comments on John's post are inserted below).

     On Thu, Apr 1, 2010 at 10:41 AM, John Westbrook <jwest@rcsb.rutgers.edu> wrote:
           Hi all,

           Coming in late on this in support of Herb's position.

           I  have never understood the necessity of marking a category as
           a 'list' type in the dictionary in the early CIF DDL,
           and in DDLm I find this even more confusing.   Given
           that DDLm supports a category key which provides a
           well defined basis for each category, this alone
           would seem to provide the appropriate expression of

     Absolutely agree, my objection is not to the loss of some packet ordering information, this is
     explicitly excluded from the infoset produced by the parser in any case.

           The choice of exporting a category with a single row as
           a collection of keyword-value pairs or  using a table
           format via a loop_  seems like a presentation style
           matter rather than dictionary issue.

     It is more than a presentation issue, as you have lost the information that those key-value pairs
     belong together, and so you need to refer to your dictionary to reconstitute them as a group.  And if
     you allow the possibility of unrolling single-row loops for all categories, then significant extra
     work is done to check, and if necessary, transform the internal representation back to a canonical
     looped form.  This reconstruction of the canonical form is highly desirable in a DDLm context, where
     we often wish to apply dREL operations to all packets in a loop.

           As Herb has observed, the vast majority of DDL2 files opt
           for key-value output for any category with a single row.
           I do not see what additional semantics are conveyed by
           regulating the manner data presentation in these cases.

     See above - some semantic information is lost.



ddlm-group mailing list

T +61 (02) 9717 9907
F +61 (02) 9717 3145
M +61 (04) 0249 4148
ddlm-group mailing list

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.