Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: _database.dataset_doi - any problems if this might be a DOI forraw data?

Thanks for the feedback Horst. It was a bit more broad than I needed, but you raise some interesting points.

We are making slow progress in CIF towards having all forms of data properly described and incorporated into the CIF "scheme", regardless of the format that they are preserved in.  One prong of this work has been to ensure that the DDLm language used for data definition in CIF is not format-specific, that is, data names defined in our dictionaries describe data that might be stored in any format - but of course somebody still has to actually do the specification that says "in files of this format you can find CIF data item x_y_z at this location". Another prong of this work is to rigorously allow for multi-data-block data (this is now done). Putting those two prongs together means you can have some zip file or directory full of files in various formats, and have the relationships between the data stored in the various files rigorously specifiable by CIF dictionaries. The missing work here is the specification (and software that uses that specification) of where data name Y is found in format X (and given typical conformance issues "...at facility Z".)

Another prong going in a bit of a different direction has been to define "pointer" data names that describe how to interpret externally-stored image data in CIF terms (see https://github.com/yayahjb/cbflib/doc/cif_img_1.8.6.dic and search for _array_data_external_data). This allows the bulky image data to be stored outside the CIF file while the CIF file itself contains the metadata. A couple of examples of this in action are in preparation, or you can have fun with https://github.com/jamesrhester/ImgCIFHandler.jl which takes such imgCIF files and loads the externally-located HDF5/CBF/ADSC image from optionally tar/gzipped archives.

To circle back to your comments, DOIs that point to files that have no properly (in the sense of CIF data names for the contents of the files) defined roles are less useful than DOIs that can be resolved to an object that has an exact slot in a CIF-based description of a data collection. The latter can be transparently auto-processed with no human intervention. Given that DOIs resolve to landing pages they are currently not that useful for machine-based data processing, but perhaps DOI standards will evolve to allow the actual data files to be specified in a DOI.

And finally, while the tsc files might be too bulky for storing in CIF syntax, the contents can eventually be brought into the CIF data world. Here are the steps:
1. Define CIF data names for the relevant quantities appearing in the files. These would logically go in a new dictionary as NoSpherA2 refinement is different to the refinement model of the core dictionary
2. Define where these data names are located in .tsc files

Step 2 is a brave new world unfortunately, as there are no general machine-readable standards that I know of, or that we have developed, for specifying where "something" is located in a file with arbitrary format - if anyone reading this knows of such a thing, get in touch.  The _array_data_external_data work I linked to above does contain some ideas about how this might work in practice.

anyway, thanks again for your thoughts.

On Fri, 27 May 2022 at 22:59, Horst Puschmann <horst.puschmann@gmail.com> wrote:
Hello James,

I think providing a DOI in the CIF pointing at large amounts of data would be fantastic. I must say that I don't quite understand that wording here -- what does the CSD or PDP have to do with it? Surely, all hkl data is now included in the CIF as standard -- but of course: it would also be excellent if the hkl could be handled via an 'hkl DOI'.

I take 'raw data' to mean the diffraction images (frames) -- possibly together with a recipe for how they were processed. Having easy access to frames 'by default' would be fantastic.

But there are other kinds of files, and some might be required to repeat the actual refinement. I am thinking of our own NoSpherA2 refinement, which requires a '.tsc' file -- a file which is clearly too large to be embedded in the CIF. A DOI for this kind of file would be really useful (right now, we just include the hkl and the exact parameters to repeat the generation of the '.tsc' file -- which is clearly not ideal.

And then there might be a third kind of DOI -- one where people can deposit 'random' files -- like videos, images, descriptions of special setup etc, scripts etc.

I am not sure whether this is the sort of feedback you were looking for, but there you are.


On Fri, 27 May 2022 at 04:09, James H via coreDMG <coredmg@iucr.org> wrote:
(cross-posted to cif-developers and core DMG, apologies for cross-posting)

Dear CIF Developers and core DMG,

IUCr Journals are looking at using _database.dataset_doi to indicate the DOI of a raw data set associated with a data block. The meaning of "dataset" is not clear here, for example, it might have been intended to refer to hkl listings.

So, please give feedback on any problems your software/database might encounter if this DOI might resolve to a raw dataset.

The current definition:

" The digital object identifier (DOI) registered to identify
    a data set publication associated with the structure
    described in the current data block. This should be used
    for a dataset obtained from a curated database such as
    CSD or PDB. "

T +61 (02) 9717 9907
F +61 (02) 9717 3145
M +61 (04) 0249 4148
coreDMG mailing list

T +61 (02) 9717 9907
F +61 (02) 9717 3145
M +61 (04) 0249 4148
cif-developers mailing list

Reply to: [list | sender only]