[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Cif2-encoding] Splitting of imgCIF and other sub-topics

> On Fri, 3 Sep 2010, James Hester wrote:
>
>> Thanks Herbert for providing the imgCIF perspective.
>>
>> I am unfortunately severely restricted in my ability to attend
>> overseas meetings at present, for family and work reasons.  I am also
>> keen to have our discussions written down and available for perusal by
>> those that will come later.
>
> How about an e-meeting?

OK, I think we need to try online as my carefully crafted arguments
seem to be misunderstood more often than not.
Let me buy a web cam first!

>> We need to discuss the relationship of imgCIF to CIF2 explicitly, if
>> imgCIF is going to influence our decisionmaking.  Some questions for
>> Herbert to answer for the record:
>>
>> 1. How widely used are non-CBF forms of imgCIF at present?  By "widely
>> used" I mean both
>>  (a) supported by software packages that allow one to do "useful
>> work", most obviously to extract diffraction spots
>
> I assume by "non-CBF" you mean the forms that do the binary sections
> in something that is not pure binary -- all software that uses CBFlib
> supports them automatically for reading.  For writing, most software
> chooses one representation for writing, usually byte-offset or
> packed binary, except when we have to debug -- then the ascii
> forms, esp. the hexdump form are very useful.

You are correct in interpreting what I mean by "non-CBF".

I understand that CBFlib supports everything, but CBFlib on its own is
not useful. Do you know approximately what programs use CBFlib?  I
know only of rasmol, but you presumably know of many more.

>>  (b) provided as an output format (even optionally) by beamlines or
>> detector manufacturers

> See above

I see nothing in your reply on the availability of imgCIF files from
detectors or instruments.

>> 2. What is the advantage of having "pure text" image files?  Why isn't
>> a format like CBF more appropriate?
>
> While I agree, when we deal with people who like XML e.g. the NeXus
> form of imgCIF, then we have no choice -- no binary is allowed, so
> UCS-2 becomes important.  Don't ask me to defend XML.  It is simply a
> fact of life.

I am guessing that this NeXuS-XML requirement is coming from Diamond,
and if this is what they want I can see why you are keen to integrate
imgCIF into HDF5, so that HDF5-XML conversion can be carried out the
standard HDF5 way, rather than encapsulating the entire imgCIF file as
a NeXuS-XML dataset.  OK: so apart from this relatively recent and
frankly crazy-wierd use case, is there any other use-case for
pure-text imgCIF?  Can we regard the "Diamond" case as a
beaurocratically-driven kluge that will be resolved via your HDF5
work, leaving no other reason to create a space-efficient CIF2 version
of imgCIF?

>> 3. What is the problem with a scenario where "pure text" imgCIF
>> remains in its current CIF1 form, and CIF2 advances are incorporated
>> into the CIF sections of CBF?
>
> I don't understand this question, nor the assumptions behind it.

Let me be less obtuse:
I envision a CBF2 format, which is a CBF file with CIF2 instead of
CIF1 syntax.  A corresponding imgCIF2 format exists. We *do not care*
about the space-efficiency of these imgCIF2 files. We recommend that
all new crystallographic image-handling applications should target
CBF2 only, rendering space-efficiency of imgCIF2 files irrelevant.
Legacy applications, of which there are very few, will be restricted
to the original imgCIF, which is very rarely produced in any case
(anticipating your answers to my above questions).

What are your (Herbert's, anybody else's) thoughts on such a plan?

>> Herbert: your work merging a DDL2-based version with DDLm-like
>> features in HDF5 format sounds interesting.  Are you planning to
>> present a motivation and/or discussion of this work at some stage?
>
> This is the subject of some grant applications, so not appropriate for
> detailed open discussion in this forum at this time.  The motivations
> are simple -- to satisfy the demands of several major facilities for
> easy integration of crytallographic synchrotron images into HDF5-based data
> management systems while preserving access to metadata, and to extend HDF5
> with relational meta-data access.  This second aspect is an increasingly
> critical need and will go forward in any case.  If we have
> a meeting or e-meeting, I can explain better.

OK, I think reading between the lines I see where this is coming from
(read your CACM article as well, BTW).  It'd be good to discuss some
of these plans at some stage.

>>
>> On Tue, Aug 24, 2010 at 11:31 PM, Herbert J. Bernstein
>> <yaya@bernstein-plus-sons.com> wrote:
>>>
>>> Dear James,
>>>
>>>  I have not been at all reticent -- imgCIF will be very poorly supported
>>> by CIF2 as currently proposed.  Of necessity, imgCIF changes encodings
>>> internally -- that it why it uses MIME -- same problem as email with
>>> images, same solution.
>>>
>>>  Any purely text version has at least a 7% overhead as compared to
>>> pure binary.  Restricting to UTF-8 increases the overhead to at least
>>> 50%.
>>> We may get away with the 7% (UTF-16).  The 50% version (UTF-8) will be
>>> ignored by the community as unworkable.  The most likely to be used
>>> version
>>> will be the current DDL2-based version with embedded compressed binaries
>>> that I am augmenting with DDLm-like features
>>> and merging in with HDF5.
>>>
>>>  As I noted many months ago, the unfortunate reality is that the
>>> current CIF2 effort will not merge well with imgCIF.  If avoiding
>>> a split is a important -- we need a meeting.  I would suggest
>>> involving Bob Sweet and holding it at BNL in conjunction with
>>> something relevant to NSLS-II.
>>>
>>>  Regards,
>>>    Herbert
>>>
>>> =====================================================
>>>  Herbert J. Bernstein, Professor of Computer Science
>>>   Dowling College, Kramer Science Center, KSC 121
>>>        Idle Hour Blvd, Oakdale, NY, 11769
>>>
>>>                 +1-631-244-3035
>>>                 yaya@dowling.edu
>>> =====================================================
>>>
>>> On Tue, 24 Aug 2010, James Hester wrote:
>>>
>>>> Hi Herbert: regarding imgCIF,  I agree that splitting it off is not a
>>>> desirable outcome.  I would like to get an idea of how well imgCIF can
>>>> be accommodated under the various encoding proposals currently
>>>> floating around, as you have been rather reticent to bring it up.  My
>>>> naive take on things is that a UTF8-only encoding scheme for CIF2
>>>> would not pose significant issues for imgCIF, and a decorated UTF16
>>>> encoding in the style of Scheme B would be even better, and quite
>>>> adequate, so imgCIF is not actually presenting any problems and so was
>>>> a red herring.
>>>>
>>>> I'm not sure that face-to-face or Skype discussions are necessarily
>>>> going to be more productive.  Writing things down, while slower,
>>>> allows me at least to collect my thoughts and those of other
>>>> participants, and hopefully make a reasoned contribution (my apologies
>>>> if I am too long-winded) and as an added bonus those thoughts are
>>>> recorded for later reference.  For example, where would I now find the
>>>> background on why a container format for imgCIF is such a bad idea?
>>>> Presumably that was all thrashed out in face to face discussions, and
>>>> no record now remains.
>>>>
>>>> On Tue, Aug 24, 2010 at 8:56 PM, Herbert J. Bernstein
>>>> <yaya@bernstein-plus-sons.com> wrote:
>>>>>
>>>>> Dear Colleagues,
>>>>>
>>>>>   James' and John's last interchange is so voluminous, I doubt any of
>>>>> us has been able to fully appreciate the rich complexity of ideas
>>>>> contained therein.  For example, one of the suggestions far down in
>>>>> the text is:
>>>>>
>>>>> (James now)  Indeed.  My intent with this specification was to ensure
>>>>> that third parties would be able to recover the encoding. If imgCIF is
>>>>> going to cause us to make such an open-ended specification, it is
>>>>> probably a sign that imgCIF needs to be addressed separately.  For
>>>>> example, should we think about redefining it as a container format,
>>>>> with a CIF header and UTF16 body (but still part of the
>>>>> "Crystallographic Information Framework")?
>>>>>
>>>>> The idea of an imgCIF "header" in CIF format and a image in another is
>>>>> an
>>>>> old, well-established, thoroughly discussed, and mistaken idea,
>>>>> rejected
>>>>> in 1998.  The handling of multiple images in a single file (e.g.
>>>>> a jpeg thumbnail and crystal image and a full-size diffraction image)
>>>>> requires the ability to switch among encodings within the file --
>>>>> something handled by the current DDL2 and MIME-based imgCIF format and
>>>>> which would be a serious problem in CIF2 has currently proposed,
>>>>> increasing the chances that we will have to move imgCIF entirely into
>>>>> HDF5 and abandon the CIF representation entirely, sharing only
>>>>> the dictionary and not the framework.
>>>>>
>>>>> If you look carefully, you will see a similar trend with mmCIF, in
>>>>> which
>>>>> and XML representation sharing the dictionary plays a much more
>>>>> important role than the CIF format.
>>>>>
>>>>> Is it really desirable to make the new CIF format so rigid and
>>>>> unadaptable that major portions of macromolecular crysallography
>>>>> end up migrating to very different formats, as they already are
>>>>> doing?  Yes, there is great value in having a common dictionary,
>>>>> but would there not be additional value in having a sufficiently
>>>>> flexible common format to allow for more software sharing than
>>>>> we now have?  It is really desirable for us to continue in the
>>>>> direction of a single macromolecular experiment having to
>>>>> deal with HDF5 and CIF/DDL2/MIME representations of the image data
>>>>> during collection, CCP4-style CIF representations during processing
>>>>> and deposition and legacy PDB and PDBML representations in subsequent
>>>>> community use?  If we could be a little bit more flexible, we might be
>>>>> able to reduce the data interchange software burdens a little.
>>>>> Right now, this discussion seems headed in the direction of simply
>>>>> adding yet another data representation (DDLm/CIF2) to the mix,
>>>>> increasing the chances of mistranslation and confusion, rather
>>>>> that reducing them.
>>>>>
>>>>> Please, step back a bit from the detailed discussion of UTF8 and
>>>>> look at the work-flow of doing and publishing crystallographic
>>>>> experiments and let us try to make a contribution that simplifies
>>>>> it, not one that makes it more complex than it needs to be.
>>>>>
>>>>> I suggest we need to meet and talk, either face-to-face, or by skype.
>>>>>
>>>>> Regards,
>>>>>   Herbert
>>>>>
>>>>> =====================================================
>>>>>  Herbert J. Bernstein, Professor of Computer Science
>>>>>    Dowling College, Kramer Science Center, KSC 121
>>>>>         Idle Hour Blvd, Oakdale, NY, 11769
>>>>>
>>>>>                  +1-631-244-3035
>>>>>                  yaya@dowling.edu
>>>>> =====================================================
>>>>>
>>>>> _______________________________________________
>>>>> cif2-encoding mailing list
>>>>> cif2-encoding@iucr.org
>>>>> http://scripts.iucr.org/mailman/listinfo/cif2-encoding
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> T +61 (02) 9717 9907
>>>> F +61 (02) 9717 3145
>>>> M +61 (04) 0249 4148
>>>> _______________________________________________
>>>> cif2-encoding mailing list
>>>> cif2-encoding@iucr.org
>>>> http://scripts.iucr.org/mailman/listinfo/cif2-encoding
>>>
>>> _______________________________________________
>>> cif2-encoding mailing list
>>> cif2-encoding@iucr.org
>>> http://scripts.iucr.org/mailman/listinfo/cif2-encoding
>>>
>>>
>>
>>
>>
>> --
>> T +61 (02) 9717 9907
>> F +61 (02) 9717 3145
>> M +61 (04) 0249 4148
>> _______________________________________________
>> cif2-encoding mailing list
>> cif2-encoding@iucr.org
>> http://scripts.iucr.org/mailman/listinfo/cif2-encoding
>
> _______________________________________________
> cif2-encoding mailing list
> cif2-encoding@iucr.org
> http://scripts.iucr.org/mailman/listinfo/cif2-encoding
>
>



-- 
T +61 (02) 9717 9907
F +61 (02) 9717 3145
M +61 (04) 0249 4148
_______________________________________________
cif2-encoding mailing list
cif2-encoding@iucr.org
http://scripts.iucr.org/mailman/listinfo/cif2-encoding


Reply to: [list | sender only]