Re: [ddlm-group] Semantics of whitespace-delimited values
- To: Group finalising DDLm and associated dictionaries <ddlm-group@iucr.org>
- Subject: Re: [ddlm-group] Semantics of whitespace-delimited values
- From: "Bollinger, John C" <John.Bollinger@STJUDE.ORG>
- Date: Mon, 3 Aug 2015 22:17:29 +0000
- Accept-Language: en-US
- authentication-results: spf=none (sender IP is )smtp.mailfrom=John.Bollinger@STJUDE.ORG;
- In-Reply-To: <1866181872.1289826.1438626213348.JavaMail.yahoo@mail.yahoo.com>
- References: <BY2PR0401MB09361817139D43D10235DBB3E0770@BY2PR0401MB0936.namprd04.prod.outlook.com><1866181872.1289826.1438626213348.JavaMail.yahoo@mail.yahoo.com>
An extensive body of practice (substantially all DDL2 applications) indicates that 2.2.7.1 (10) is not to be taken at face value. I think it's
best to interpret it as pertaining to the base 'numb'
type, on which DDL2 does not rely. It is in any event about a relationship between syntax and semantics, thus neither wholly one nor wholly the other. It brings out the fact that it isn’t always possible to define a clean separation between the two. CIF’s
attempts to do so are not wholly successful. I stand by my earlier words, but I should clarify. The key point that I ineffectively attempted to make in my answer to question (2) was that you cannot rely
on syntax *alone* to establish data typing, not even in a DDL1 application. That’s why the distinction does not occur at the syntactic level, the presence of the <Numeric> production in the grammar notwithstanding. A value matching that production does not
necessarily have numeric type, and values not matching that production can and are interpreted as numbers. Syntax does contribute to data type determination in some cases, however, per 2.2.7.1 (within its scope) and 2.2.7.4.8. If that were not so then we would not
be having this conversation. My interpretation of the specs and general practice renders 2.2.7.1 in particular relevant only to DDL1 applications, including checkCIF. As I wrote before, a DDL1 application that accepts quoted values as numeric thereby exercises
an error recovery mechanism. That is useful and appropriate for some, but not for others. I will not quibble too much over
"can be sensitive"
vs. "is sensitive".
Both are true. One calls attention to the fact that some values will be interpreted identically, regardless of quoting, whereas the other emphasizes that some will be interpreted differently when presented quoted than when presented unquoted. I prefer the
former because to me it more clearly describes the situation that the interpretations of some values are sensitive to quoting, but those of others aren’t. Data typing is ultimately a determination to be made at the semantic level, as informed by the syntax and details of the value string. This has always been
the case for CIF 1.1, and it does not change for files presented in CIF 2.0 syntax. Any needed clarification should be made for CIF 1.1. Cheers, John From: ddlm-group [mailto:ddlm-group-bounces@iucr.org]
On Behalf Of SIMON WESTRIP I am confused - paragraph 10 of section 2..2.7 "Formal specification of the Crystallographic Information File" clearly states that numeric data are
not to be quoted: "10. A simple data value (i.e. one which does not
This is part of section 2.2.7.1 Syntax - not 'semantics'.
So I believe the CIF 2.0 specification manuscript is wrong in
its description of CIF1.1, i.e. "Interpretation of a CIF data value 'can be' sensitive to whether it is presented in whitespace-delimited form" - should be *is* sensitive to whether it is presented in whitespace-delimited form.
For better of for worse, the requirement that numbers are always delimited by whitespace is being recognized and enforced (checkCIF
- perceived as the de facto standard in CIF validation - issues level A alerts when a numeric value is delimited by anything other than whitespace).
If this is purely a semantic feature, then for CIF2 we should make it clear - i.e. remove all confusion and any necessity to issue
alerts or worse reject a CIF that is otherwise syntactically correct, or even semantically valid (e.g. with respect to dictionary definitions)?
Cheers
Simon
From: "Bollinger, John C" <John.Bollinger@STJUDE.ORG> 1) The fodder for this debate is for the most part the CIF 1.1 specifications and the behavior of existing CIF 1.1 applications. My
argument therefore applies first and foremost to CIF 1.1. On the other hand, we have not approved or before now even really discussed any change in this area for CIF 2, and I do not think there is any interest in making a change, so whatever is decided for
CIF 1.1 probably will apply to CIF 2.0 as well. Nevertheless, the last sentence of section 2 of the CIF 2.0 specification manuscript already does address this: "Interpretation of a CIF data value can be sensitive to whether it is presented in whitespace-delimited
form." What we are discussing now is essentially whether that should be narrowed (or, perhaps, emphasized). More on that below. 2) We do not agree that the distinction between number and character types is at the syntactic level in CIF 1.1 or above. All text
in ITVG 2.2 that discusses data typing falls under one of two "common semantic features" headings, 2.2.5 and 2.2.7.4. The grammar does contain a production for numbers, but that can and should be interpreted as merely supporting the "common semantic features"
prose. Section 2.2.7.4.7.1 (17) cannot otherwise be effective, and the consensus interpretation is that it *is* effective. CIF 2.0 isn’t changing anything here, but I do hope that omission of a numeric format from the CIF 2.0 EBNF will help reduce confusion
in this regard. 3) None of the existing DDLs has a formal mechanism for saying anything about data value quoting. My understanding is that DDL2 applications
are expected to altogether ignore the character/number typing described in ITVG 2.2, however, and indeed to ignore distinctions between quoted and unquoted values except for '.' and '?'. Effectively, that’s implicit in using a DDL2 dictionary, at least for
the items defined by such a dictionary, and it is inconsistent with syntactic data typing. Practice is less uniform among DDL1 applications, as the behavior survey showed. I’d be inclined to say that ITVG 2.2 data typing is implicit at least in the core
dictionary, and probably in all DDL1 dictionaries. In practice, many -- but not all -- DDL1 applications are more accepting, which I think is best characterized as their choice of error-handling behavior. Overall, then, I’m saying that the applicable DDL
implicitly directs this detail of value interpretation. This is certainly an area that would benefit from clarification. I see only two viable interpretations of ITVG 2.2 as it pertains
to the significance of data value quotation: A) syntactically, CIF 1.1 data values have exactly two properties, which can be characterized as a string of characters and a flag
indicating whether the string is quoted; or B) syntactically, CIF 1.1 data values have several properties (which can be counted various ways), describing at least: a string of
characters, whether that string matches one or more numeric formats, whether the string consists of a single decimal point, whether the string consists of a single question mark, and whether that string is presented quoted. The last can be suppressed for
values that don’t have any of the given specific forms (the advantage to doing so being unclear to me), but it is nevertheless implicit in some other data value forms, such as those matching data names. Interpretation (B) or something close to it is required if one wants the result that quoting can affect value interpretation only for
numbers and the special null values, yet doesn’t necessarily do so for numbers. It is the narrowest interpretation that can work. Interpretation (A) permits more distinctions to be drawn between quoted and unquoted values, but does not inherently require
such a distinction to be made in any particular case. It is the simplest specialization of STAR grammar that can work. I prefer (A). Any way around, all parsers I know about apply at least a bit of semantic interpretation at parse time. In particular, although behavior
varies with respect to parse-time numeric interpretation, I don’t know any CIF parser that fails to provide special handling for the null values. This is not a problem. CIF parsers need only serve the purposes for which they are intended; they are not required
to be completely general. Even those that are designed to serve general purposes may still take into account the shape of the problem space, as reflected by the current DDLs and practices. John --
John C. Bollinger, Ph.D. Computing and X-Ray Scientist Department of Structural Biology St. Jude Children's Research Hospital (901) 595-3166 [office] From: ddlm-group [mailto:ddlm-group-bounces@iucr.org]
On Behalf Of SIMON WESTRIP John asserts that: "I nevertheless maintain, however, that these and other parts
of the specs absolutely make it *permissible* to ascribe different significance to quoted and unquoted values, especially with regard to data typing. Since dictionaries are the primary vehicle for type specification, that means dictionaries are empowered
to distinguish values and their types based on quoting status, whether or not they take any advantage of that power." Three questions: 1) If this is to hold true for CIF2, should we make this clear in the specification of CIF2 - importantly that a robust parser must return the quoting
status (context) or a data value to the calling application? 2) If the distinction between number and character types at the 'syntactical' level (which I think we agree is part of the 'non-semantic' CIF1.1 specification)
is not to be enforced in CIF2, should this be made clear in the specification of CIF2? NB.
Section 2.2.7.4.7.1 (17) is part of the "common semantic features", which on the whole are largely inappropriate for CIF2, and indeed are not entirely ideal for CIF1 (though
have proved 'workable'). To clarify: I currently happen to be working on a new CIF1.1. application and find myself wishing it could use CIF2 - i.e. some of the text-field semantics of CIF1.1
can be a bit of a nightmare for developers and users alike, e.g. (i) representation of a caret is not really possible according to the semantics - i.e. \^ is a combining circumflex, while an unescaped caret ^ signifies
the start of a superscript; (ii) <i> indicates italic, so to represent the same sequence according to the semantics you would have to use
\\langle i>, which isn't strictly the same thing; (iiii) a tilde similarly has to be interpreted according to context...; (iv) the description of line-folding semantics uses C:\foldername\filename as an example - is this C:<phi>oldername<phi>ilename? I'm being pedantic
I know - especially as these 'semantics' are no longer relevant to CIF2.... Main point is the distinction between 'semantics' and 'specification' - semantics as described for CIF1 are very much optional - but the base character/number
type distinction is outside the 'semantic' description so is not optional in CIF1. 3) Lastly, is it possible to specify in a dictionary how a value should be quoted in the syntax? Cheers Simon From: "Bollinger, John C" <John.Bollinger@STJUDE.ORG> The authoritative specification of CIF 1.1 is section 2.2 of ITVG, and ITVG 2.2.7.1.4 contains pretty much the same text. In practice,
however, the quoted provisions yield little practical benefit, and they have been rendered largely moot by other specifications and by widespread practice. In particular, the specifications elsewhere assign primary responsibility for data type determination
to dictionaries. Section 2.2.7.4.7.1 (17) is explicit about dictionaries taking precedence over syntactic type determination: "Where the attributes of a data value are not available in a dictionary listing, it may be assumed that a character string interpretable
as a number should be taken to represent an item of type 'numb'. However, an explicit dictionary declaration of type will override such an assumption. " One could argue that paragraphs 10 and 13 of section 2.2.7.1.4 require that quoted character strings not be considered "interpretable
as a number", and indeed some parsers take that approach, but the prevailing interpretation is different. Few parsers used by prevalent programs interpret the specs as forbidding them to interpret quoted strings as numbers, ITVG 2.2.7.1.4 (10-13) notwithstanding. Furthermore, these days I hold a rather broad interpretation of what a "dictionary" is. Certainly there are the DDL[12m]-format external
dictionary files, but also, the many programs that hard-code usage of specific data items for specific purposes thereby build in a collection of data item definitions that should be considered a local dictionary. From that perspective, ITVG 2.2.7.1.4 is applicable
only to values that are not subjected to any specific interpretation. Such values can be used at all only by programs that perform generic CIF manipulations such as pretty printing, and those must take care to preserve values’ forms exactly because they do
not know which aspects of those forms -- including quoting status -- are significant. I nevertheless maintain, however, that these and other parts of the specs absolutely make it *permissible* to ascribe different significance
to quoted and unquoted values, especially with regard to data typing. Since dictionaries are the primary vehicle for type specification, that means dictionaries are empowered to distinguish values and their types based on quoting status, whether or not they
take any advantage of that power. John From: ddlm-group [mailto:ddlm-group-bounces@iucr.org]
On Behalf Of SIMON WESTRIP I think we need to clarify what CIF1.1 specifies in this respect. From what I can see, the distinction between character types and numeric types is
part of the base specification. appears within a CIF may be quoted (e.g. '12') *if, and only if*
it is to be interpreted and stored in computer memory as a
character string and not a numeric value. For example '12' might Simon From: SIMON WESTRIP <simonwestrip@btinternet.com> In an attempt to reinforce my argument, does anyone know of a programming language that will read 6.6666(6) as a float with an associated s.u.? As far
as I can tell, such 'numbers' will require additional parsing to transform them into a type that is useable by a program? Furthermore, if I want to round-up such a value (as journals do for presentation), I may well need to treat it as a 'string' in order
to avoid floating-point errors (depending on the programming languages and associated libraries I'm using). Cheers Simon From: SIMON WESTRIP <simonwestrip@btinternet.com> I agree with John Westbrook - this is how I would like it to be with CIF2: "All data type interpretation at input is done via the data dictionary" CIF is a data container not a programming language. The data container does not have to be aware of data types - it simply needs to unambiguously tag
(in the case of CIF) and store the data so that it may be retrieved reliably according to the specified syntax. There is really no need these days for the data container (in this case its text-based to all intents and purposes) to concern itself with the type
of value that it stores, as long as it is stored and retrievable unambiguously. The power of CIF lies in its dictionaries, and especially for CIF2 in the methods that they describe. When it comes down to it, CIF2 is not backwards compatible with CIF1 (the use of UTF8 in particular is a potential pitfall), so why not take the opportunity
to address some of those issues that still today propagate a negative attitude to CIF (all too often I've been on the receiving end of criticism regarding CIF and its 'idiosyncrasies', whether justified or not; and only recently I've read some concerns regarding
the PDB's preference for CIF rather than PDB format in the macromolecular community - to my mind unjustified but nonetheless the perception is not favourable in some quarters).
At the risk of sounding as if I'm 'ranting', I am worried that some of the CIF2 changes will not be received that favourably from developers (for example,
the list and table structures are so close to JSON structures that an obvious question is why not just follow JSON - for what its worth my answer would be that lists would simply require the same parsing as loops, and much the same for tables, except for the
colon... :-) I think we have an opportunity with CIF2 to simplify the basic data storage format, while adding power through the enhanced dictionary format. We have
a general precedent in STAR2 for these changes, so why not break a little more from CIF1. Cheers Simon From: "john.westbrook@rcsb.org"
<john.westbrook@rcsb.org>
_______________________________________________ _______________________________________________ _______________________________________________ |
_______________________________________________ ddlm-group mailing list ddlm-group@iucr.org http://mailman.iucr.org/cgi-bin/mailman/listinfo/ddlm-group
Reply to: [list | sender only]
- Follow-Ups:
- Re: [ddlm-group] Semantics of whitespace-delimited values (James Hester)
- References:
- Re: [ddlm-group] Semantics of whitespace-delimited values (Bollinger, John C)
- Re: [ddlm-group] Semantics of whitespace-delimited values (SIMON WESTRIP)
- Prev by Date: Re: [ddlm-group] Semantics of whitespace-delimited values
- Next by Date: Re: [ddlm-group] Semantics of whitespace-delimited values
- Prev by thread: Re: [ddlm-group] Semantics of whitespace-delimited values
- Next by thread: Re: [ddlm-group] Semantics of whitespace-delimited values
- Index(es):