Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Seattle Discussions

  • To: Multiple recipients of list <imgcif-l@bnl.gov>
  • Subject: Seattle Discussions
  • From: Andy Hammersley <hammersl@esrf.fr>
  • Date: Thu, 12 Sep 1996 11:24:43 -0400 (EDT)

Hello Everyone,

   The Seattle IUCr Congress together with the two CIF workshops and
an informal open discussion on "imageNCIF" is starting to fade in the
memory, but before I forget too many points I thought that I'd try to
summarise some of the main points discussed and in some cases decisions
made. (I should have done this a little sooner, but I've been busy
catching up with basic work.)

Perhaps first it is worth emphasizing the value of meeting "E-mail
correspondants" face to face. Whilst the e-mail discussions have
achieved much, there were areas of slight mis-understanding where
face to face discussions and "brain-storming" were efficient in
quickly finding the real problems and solutions. I held very useful 
discussions throughout the congress with other imageNCIF, COMCIF, mmCIF,
and powderCIF members, and with commercial detector manufacturers.

Summarising the meetings in relation to "imageNCIF"
---------------------------------------------------

CIF I: Data Interchange and Publications Workshop: Provided an introduction
to CIF for myself and others. Here a number of facilities of STAR which
I hadn't realised were introduced which are useful for "imageNCIF" e.g.
The abilities to define dependencies of one data name on another.
Brian Toby introduced powderCIF which has similarities in that it
deals with raw data, and therefore data names for sample-detector
distances, and 2-theta angles are needed. Paula Fitzgerald talked about
the state of mmCIF.

CIF II: New Applications and Definitions: Here amongst a variety of
talks on DDL2, mmCIF, and possible new CIF's, I was given the chance to
introduced the "imageNCIF" inititative. I won't try to comment on my
talk, other than to say that I tried to cover items which have been
discussed by e-mail.

Image File, Open Discussion: Bob Sweet chaired an open meeting dedicated
to discussing image file formats (CIF based or otherwise). This meeting
was well attended with a variety of imageNCIF members, COMCIFS members,
commercial detector manufacturer representative, analyis programmers, 
and interested crystallographers. Some useful decisions were made (see
below).

Decisions
---------

1. A CIF based "imageNCIF" proposal will be pursued for submission to
COMCIFS for eventual IUCr approval.

2. There needs to be a data status data item e.g. Is the data raw,
processed, distorted etc.

3. The line terminator should be carriage-return followed by line-feed
(\cr\lf, ASCII(13, 10). (See discussions.)

4. In addition to testing the readability of the ASCII section on the
common operating systems (Unixes, MS-DOS, VMS), it should be tested
using WWW browsers.

5. Whilst the format is designed to be suitable for blocked I/O
it should also be emphasised that it is suitable for stream I/O.

6. Bob Sweet will try to organise a small workshop to finalise the first
draft of the format.

Discussions
-----------

1. Jim Fait from Siemens provided very useful input given their
experience of the Siemens file format running on a variety of different 
operating systems. They use the \cr\lf "line terminator" combination, 
which allows "readable" files on Unixes, MS-DOS, and VMS. (Thus defined, 
a file of type binary can be viewed on VMS using  "TYPE"). They also 
use a 512 byte "blocked" scheme.

2. Siemens use BOTH a <Control-Z> and a <Control-D> at the end of the
ASCII section. This stops the binary section being accidently
transferred by ftp in ASCII mode. I suggest that we adopt this sensible
approach. (They pad any unused bytes,the binary section starting at
the 512 byte boundary with dots.) It is suggested that we included at
least 1 form-feed character at the end of the ASCII section to separate
the readable text from the binary data when displayed as text.

3. With Peter Keller and Syd Hall the idea of an file identifier 
(magic number) was discussed. (This was a case where face to face
discussions were most useful.) Here I had proposed something like:

###_CRYSTALLOGRAPHIC_BINARY_FILE: VERSION 1.0

but the point was made that his information would be lost (in 
general) if the header were extracted and made into a CIF, and
processed into another CIF, etc. (as some programs strip out
comment lines).

Peter suggested using a specially defined 

data_

opening block.

Syd suggested a similar construct, but using a 

save_

block.

(But since this part of STAR, but not presently part of CIF, I think
Peter's suggestion is probably more appropriate.)

Perhaps I could ask Peter to submit his suggestion to the group.

If we use this to define the start of the ASCII header section, it's
then not clear to me what we should use to signal the end. Previously
I suggested:

###_END_OF_CBF_HEADER
 
to give some sort of \begin \end pair. Any suggestions ?

4. Multi-element Mosaicked detectors (e.g. 2x2, 3x3 fibre-optic/CCD
array detectors) need to be considered, although in initial cases
at least they seem likely to be considered as single 2-D arrays.

5. Care needs to be taken to avoid a legal mine-field if sophisticated
data compression schemes are included e.g. Lempel-Ziv coding has been
the source of various law suits. (I believe the "byte_offsets" scheme
is too simple to present a problem here.)
 
6. Although not discussed within the meeting it seems clear to me that
we should at least cater for DDL-2 with the "imageNCIF" software, and 
probably "imageNCIF" should be defined in DDL-2. (Would someone like
to translate my CBF definition document into DDL-2 ?)


Conclusion
----------

The response to "imageNCIF" was very positive from all directions.
But it seems clear to me that a number of people need a format now
or in the near future. Thus, it is preferable to define the overall
structure of the format as soon as possible, even if a number of
details change. (Final full IUCr approval is presumably a long process,
but this needn't stop a workable format being developed and used, indeed
this is probably a necessary step.)

Parallel to this I have a need for such a format, so propose to start
coding a proto-CBF system. This will be a useful test ground for the 
format and ideas.



       Andy

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links