Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Imgcif-l] High speed image compression

Hi Justin,

An observation - I assume that a 4k x 4k image is acquired from multiple sensors? If so the images from each could be separately compressed then composed by the data processing software - after all CBF is more than capable of storing data from multiple detectors. That way each could be handled by a separate thread which should be able to get the job done in the time available.

As an alternative, I remember something about splitting the byte offset stream to allow parallel reading or writing - which would probably also help but I don't remember any details.

Best wishes,

Graeme


________________________________________
From: imgcif-l-bounces@iucr.org [imgcif-l-bounces@iucr.org] on behalf of Nicholas Sauter [nksauter@lbl.gov]
Sent: 29 July 2011 05:41
To: The Crystallographic Binary File and its imgCIF application to image data
Subject: Re: [Imgcif-l] High speed image compression

Justin,

I'm also getting disappointing results with the code you sent.

Compiled with g++ -O3, on 64-bit Fedora 8 Intel Xeon 2.93 GHz, I'm getting a
compress time of 80-90 ms.  What type of throughput are you aiming for...<
40 ms?

I'll continue testing a bit more tomorrow...

Nick

On Thu, Jul 28, 2011 at 3:36 PM, Justin Anderson <justin@rayonix.com> wrote:

> Thanks Nicholas.
>
> I only made a couple small changes to Graeme's code.  1: to load an image
> from a file and write to file and 2: to pass the data vectors by reference.
>  The last change seems to have sped things up a little but it's still taking
> 110 - 130 ms to compress which is too slow.  We are not as concerned with
> decompression speed as that will not need to occur in real-time.
>
> I put on our FTP here: ftp://ftp.rayonix.com/pub/del_**
> in_30_days/byte_offset.tgz<ftp://ftp.rayonix.com/pub/del_in_30_days/byte_offset.tgz>
> .
>
> Thanks,
>
> Justin
>
>
> On 7/28/11 2:06 PM, Nicholas Sauter wrote:
>
>> Justin,
>>
>> Just some comments based on our experience...first, I haven't tried the
>> compression extensively, just the decompression.  But I've found Graeme's
>> decompression code to be significantly faster than the CBF library, first
>> because it is buffer-based instead of file-based, and also because it
>> hard-codes some assumptions about data depth.
>>
>> I'd be happy to examine this in more detail if there is some way to share
>> your code example...
>>
>> Nick
>>
>> On Thu, Jul 28, 2011 at 11:46 AM, Justin Anderson<justin@rayonix.com>**
>> wrote:
>>
>>  Hello all,
>>>
>>> I have run Graeme's byte offset code on a 4k x 4k (2 byte depth) Gaussian
>>> noise image and found it to compress the image in around 150 ms (64-bit
>>> RHEL, Pentium D 3.46GHz).  Using CBF library with byte offset
>>> compression, I
>>> find the compression takes around 125 ms.
>>>
>>> This will be too slow to keep up with our high speed CCD cameras.  We are
>>> considering parallelizing the byte offset routine by operating on each
>>> line
>>> of the image individually.  Note that this would mean that a given
>>> compressed image would be stored differently than via the whole image
>>> algorithm.
>>>
>>> Has anyone been thinking about this already or does anyone have any
>>> thoughts?
>>>
>>> Regards,
>>>
>>> Justin
>>>
>>> --
>>> Justin Anderson
>>> Software Engineer
>>> Rayonix, LLC
>>> justin@rayonix.com
>>> 1880 Oak Ave. #120
>>> Evanston, IL, USA 60201
>>> PH:+1.847.869.1548
>>> FX:+1.847.869.1587
>>>
>>>
>>> ______________________________**_________________
>>> imgcif-l mailing list
>>> imgcif-l@iucr.org
>>> http://scripts.iucr.org/**mailman/listinfo/imgcif-l<http://scripts.iucr.org/mailman/listinfo/imgcif-l>
>>>
>>>
>>>
>>
>>
> _______________________________________________
> imgcif-l mailing list
> imgcif-l@iucr.org
> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>
>


--
Nicholas K. Sauter, Ph. D.
Computer Staff Scientist/Engineer
Physical BioSciences Division
Lawrence Berkeley National Laboratory
1 Cyclotron Rd., Bldg. 64R0121
Berkeley, CA 94720-8118
(510) 486-5713
_______________________________________________
imgcif-l mailing list
imgcif-l@iucr.org
http://scripts.iucr.org/mailman/listinfo/imgcif-l

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 



_______________________________________________
imgcif-l mailing list
imgcif-l@iucr.org
http://scripts.iucr.org/mailman/listinfo/imgcif-l

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.