Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [dddwg] Please comment on the attached for ICSTI's InsightsSeries

Dear John,

   I think this is an extraordinarily useful document.  I have just a 
few minor quibbles
with one paragraph, the first paragraph of section 7 of the third article:

"Costs could be minimised by preserving each raw diffraction data set at 
the local
centre (synchrotron facility, neutron research centre or university) 
thus avoiding
quite large network file-transfer costs. Also costs could be cut by 
preserving only
a proportion of each of the raw data sets or by making some form of data 
set
compression (lossy or lossless; James Holton pers comm) or both."

I believe that, properly interpreted, the second sentence undercuts the 
first:
By proper use of compression we can have useful diffraction datasets that
can be handled in a cost-effective manner, independent of whether they 
need to
be sent over an internal instrument network, an internal institutional 
network
or over the planet-wide internet, and, in order to ensure proper access 
to such
data, we need to move in that direction.  If we combine James Holton's
suggestion of lossy compression with the highly effective lossless imgCIF
Hammersley byte-offset compression used with the Dectris detectors, or
with the slower, but even more effective J. P. Abrahams CCP4 compression
(also supported in imgCIF) we can have our validation and continuous
improvement cake while being able to eat it on current networks.  If we do
lossless compression for, say, 1 of 20 images, and lossy compression for 
the
remaining 19 of 20 images, we should be able to get at least 10:1 
compression,
and more likely better than 20:1, with a sufficient number of "perfect" 
frames
for forensic validation and more than enough useful data for continuous
inprovement.

I would suggest that following rewrite of the paragraph in question:

Costs could be minimised by a combination of
   7.1  Distributing the burden of storage of the raw data among the 
collection sites
(synchrotron facility, neutron research centre or university), the users 
sites (e.g.
universities) and long-term central repositories where appropriate (e.g. 
journal
supplemental data repositories, university archives, and, perhaps, 
commercial
sites such as Google and Amazon); and
   7.2.  Both lossless and lossy compression to achieve a balance 
between retention
of forensic quality data and copies of sufficient scientific validity to 
facilitate
continuous improvement; and
   7.3.  Use of DOIs or similar location-independent identifiers to 
identify the data
as it migrates over the various nets involved among collection sites, 
users sites and
archives.

   Regards,
      Herbert


On 4/9/12 3:21 AM, John Helliwell wrote:
>
> Dear Colleagues,
> Please comment on the attached which we have drafted for ICSTI's 
> Insights Series.
>
> In parallel with inviting your comments we are consulting the ICSTI 
> Executive Secretary (and who in turn may well consult the President of 
> ICSTI) as to its suitability; the biggest challenge being the 
> technical crystallographic terms for the general audience that ICSTI 
> members cover.
>
> As a reminder recall that ICSTI (the International Council for 
> Scientific and Technical Information) embraces numerous national 
> information science agencies, Librarians, agencies like DataCite, the 
> Research Information Network, the ALPSP and as scientist delegates:- 
> from IUPAC (Dr Wendy Warr) and IUPAP (Sir Roger Elliott), besides JRH 
> for IUCr.
>  Current full list:-
> http://www.icsti.org/spip.php?rubrique44
>
>
> We are striving to keep to an end of April 2012 delivery of these 
> three articles to ICSTI if at all possible.
> There are still some polishings on referencing style and adding a 
> movie clip, perhaps most easily via a weblink, that are obviously 
> still to be done.
>
> Please reply with any comments before April 19th 2012.
> Thankyou in anticipation.
>
> Best wishes and regards,
> John, Tom and Brian
>
>
> Prof John R Helliwell DSc FInstP CPhys FRSC CChem F Soc Biol
> Chair School of Chemistry, University of Manchester, Athena Swan Team.
> http://www.chemistry.manchester.ac.uk/aboutus/athena/index.html
>
>
> _______________________________________________
> dddwg mailing list
> dddwg@iucr.org
> http://mailman.iucr.org/mailman/listinfo/dddwg
>    

_______________________________________________
dddwg mailing list
dddwg@iucr.org
http://mailman.iucr.org/mailman/listinfo/dddwg

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links