Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Fwd: Re: DDLm, dREL, images and NeXus

I was under the impression the dREL use case was primarily for calculation and validation methodologies embedded within a relational database. It was a "fit for purpose" language that didn't include any external IO routines, although they were added to the dREL shell interpreter for convenience. Consequently conversion of CIF content to NeXus/HDF5 seems way out of language scope to me. I am sure if the designers intended a general purpose programming language to be used, then they would have embedded python in its totality and been done with it. Personally I see dREL as a promissing technology in the way that fusion reactors have been promissing for the past 50 years, and probably the next 50 too. Consequently I would be hesitant to base a technology around waiting for dREL to come of age. What would be more useful (from a search and archival perspective) would be something like a standardised mapping from CIF names to NeXus NXDetector.NXdata names and vice-versa - for data values that don't map onto already defined NeXus elements. It would be useful also to map those into the sample_parameter, dataset_parameter and datafile_parameters of the STFC's ICAT metadata catalogue. And even the metadata attributes of the TARDIS spec and the RDF of eCrystals. If that mapping was in CIF or some other format, then everyone could write their own conversion tools in their language of choice (and then use NXTranslate? to convert to HDF5). Just for the record, there a couple of non-authoritative, kludged pseudo-schema's here: http://cima.chem.usyd.edu.au:8095/schemaFusion/html/index.html but they are only intelligible if you view them with firefox. Actually, probably it would be nice if all the various diffractometer company frame header parameters could be mapped to a standard as well. Herbert, I believe you have already written an imageCIF to NeXus (XML?) conversion tool and I would be interested to know which CIF items were mapped into which NeXus fields and whether the NeXus files hold multiple images or links to external filesets and what do you do with image parameters that don't fit exist within the imageCIF vocabulary. I understand that HDF5 has one of the most efficient structured file access protocols around, so presumably you would embed an image within the NeXus file? Or are there granularity issues? Should a complete imageCIF file be embedded as a single NX_CHAR/NX_BINARY record? As datasets are becoming larger over time, both in the number and size of images,the simple act of looking at files within directories can be troublesome, especially over remote networks. Pascal Calvat, the author of the JUX SRB browser, was suggesting to archive complete datasets in SRB repositories as a single very large file. I think he was actually refusing to handle more than 500 files in a folder/directory with JUX. But apparently SRB and iRODS have backends that can let you peer into such stored HDF5 files in any case, so I am not sure if his arguments still hold. Doug On Thu, 11 Dec 2008, Herbert J. Bernstein wrote: > Dear Colleagues, > There are many frameworks > to consider, but two that are very close to achieving the goal of becoming > inter-operable in the immediate future are imgCIF and NeXus. What is > missing is a formal language within which to specify how to move between > them. > > We could, of course, just come up with a verbal description of how > to move between imgCIF and NeXus and a couple of example conversion > programs written ad hoc in whatever language might come to mind. However, > the effort being expended on dREL, the supporting language for DDLm, > suggests the possibility of building on dREL as a base to do this job > by extending dREL to have the capability of working with NeXus (dREL > is already capable of dealing with CIF). > James has made the counter > proposal of leaving dREL as just a CIF-specific language and keeping > the CIF-NeXus conversion algorithm specification as a matter for a > different language and/or API. James has also suggested the further > step of stripping out the built-in functions from dREL and dealing with > just a very stable dREL language specification in one instance and a > perhaps evolving API (list of builtin functions available in dREL) on the > other: > "My comment at this stage would be that defining a coupling mechanism > between CIF and a given language is not a large task, due to the > simplicity of the CIF syntax, whereas adding lots of stuff to dREL > would be a serious task and has some important downsides (loss of > simplicity being an important one). Apropros the > simplicity of the coupling mechanism, I (predictably) quite like my > Python model of a CIF file as a hash table of CIF data block objects > indexed by datablock name, and the datablock objects are themselves > hash tables of strings/lists of strings indexed by dataname. This > model would appear to translate pretty easily into most other > languages. What then remains is some syntactic sugar (the use of > square brackets to do key-based lookup is nice in dREL) which can be > replaced in another language by a few standard methods." > > There was a lot more to the discussion, but let us try to settle a > direction: > > Should we be trying to extend dREL to support more than just CIF, > specifically NeXus, making something we might call dREL++, or should the > language for this broader task be something distinct from dREL with a > distinct name. In practice, in either case, I suspect all of this will be > built on a python base, or something similar, as James suggests, but > names do matter, > > Comments please. > > Regards, > Herbert > > ===================================================== > Herbert J. Bernstein, Professor of Computer Science > Dowling College, Kramer Science Center, KSC 121 > Idle Hour Blvd, Oakdale, NY, 11769 > > +1-631-244-3035 > yaya@dowling.edu > ===================================================== > _______________________________________________ > comcifs mailing list > comcifs@iucr.org > http://scripts.iucr.org/mailman/listinfo/comcifs -------------------------------------------------------
Reply to: [list | sender only]