Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

imageNCIF


Hi Everyone,

This is my first communication to the "bulletin board". As I haven't
received any from the BB I guess it's the first communication at all. I
hope that it prompts some others !

We've just had the ESRF User's meeting so I've had a chance to chat with
Phil Evans, about the "Cambridge PX view" (that's Cambridge, England).
He's keen on something simple, but would like data compression built in
right from the start. This is something which I've been thinking about.
Perhaps the default storage for integer data could be a VERY SIMPLE
lossless byte difference based compression scheme. Such a scheme could be very
simple, whilst giving roughly a factor 2 gain in storage, AND allow
larger dynamic range data. Should we think about adopting such a scheme,
or was I right in thinking that compression was a subject best avoided
in the first instance ?

Here are my answers to the questions I raised in a previous e-mail
concerning the requirements of the data format (so read I instead of "we"):

-------------------------------------------------------------------------------

> Do we want a BINARY based format ? This of course is contrary to
> existing CIF, but in line with almost all image formats.

Yes, images are too big already. ASCII for the image data storage is out.

> Do we want a HEADER / DATA SECTION based format ?

Yes, a self-describing format is necessary to cover variable image
sizes, etc.

> If there are headers must they be in ASCII, or is a binary based header
> information scheme O.K. ?

Yes, ASCII is simplest (see below)

> If ascii headers, is the KEYWORD / VALUE PAIRS scheme required ?

Probably yes, this seems a good simple solution, given the need for
flexibility and extensibility.

> Do we want a SIMPLE system (various people stress a desire for
> simplicity), and if so what is simple and what is too complicated ?

Yes, yes, yes. But what is simple or not simple. The best answer I know
to this question is a quote from Albert Einstein:

"Things should be made as simple as possible, but no simpler."

I think this should be the guiding principle of the file format i.e.
certain complications are necessary to allow flexibility and
functionality, but all unnecessary complication should be avoided.

> Is EFFICIENCY in either storage size, or access time an issue ?

Yes, but probably not the most important issue. Reasonable storage 
efficiency is certainly desirable.

> How FLEXIBLE does the system need to be, in its first level
> implementation ? or in its ultimate limit ? 

We don't know, so (simple) mechanisms should be included to allow future
extension
 
> Are there other important characteristics which should be included in
> the requirements ?

Probably, yes, but I can't think of any at the moment. Any ideas ?

-------------------------------------------------------------------------------

What are the views on the above issues ? 

What are the views on HDF and the APS approach to image formats ?


Awaiting your replies,

       Andy

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links