Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Imgcif-l] High speed image compression

By the way, attached is the new code.

On 7/29/11 10:41 AM, Justin Anderson wrote:
> Thank you everyone for the great suggestions.
>
> Note: I am not including the time to write the compressed data to disk 
> intentionally.  I want to test only the compression time and not the 
> disk speed.  We will be writing these files to a PCIe solid state 
> drive in production.  These drives can write uncompressed frames in 
> real time.
>
> Our goal is to be decently under 100 ms with the 4K (actually 1920 x 
> 1920), 2 byte images to keep up at 10 fps.
>
> On an Intel Core i7 940 processor the same code runs in 50 - 60 ms.
>
> Some new runtimes (on the Core i7):
>    Reserving the vector space for the compressed data ahead of time:
>       40 - 50 ms
>    Adding compressed data via address instead of push_back:
>       30 - 40 ms
>
> Hopefully with the image correction time and transfer times this will 
> work.
>
> ~Justin
>
> On 7/29/11 9:40 AM, Herbert J. Bernstein wrote:
>> And you can gain a little more speed once you preallocate by
>> switching internally from indexed references to Vectors to
>> indexed references to C pointers to the same Vectors,
>> e.g.
>>
>>       const int16_t * vptr;
>>       char * pptr;
>>       vptr =&values[0];
>>
>> and, after you preallocate packed
>>
>>       pptr =&packed[0];
>>
>>
>> At 6:53 AM -0400 7/29/11, Herbert J. Bernstein wrote:
>>> I agree.  On my Mac, the time also drops sharply with pre-allocation 
>>> and []
>>> instead of push_back.
>>>
>>>
>>> At 10:51 AM +0200 7/29/11, Jonathan WRIGHT wrote:
>>>> Dear Justin,
>>>>
>>>> Your code counts the time compressing, but not the time writing the
>>>> file, which is much longer for me. As it stands, you might gain a 
>>>> little
>>>> by adding "packed.reserve(size*2)" just before the call to compress 
>>>> (54
>>>> to 38 ms here on vista64, 3.3 Ghz). That falls further (28 ms) if you
>>>> stop using "push_back" and instead allocate something which is
>>>> "certainly" large enough to start with and use packed[p++]=c.
>>>>
>>>> Cheers,
>>>>
>>>> Jon
>>>>
>>>> On 29/07/2011 00:36, Justin Anderson wrote:
>>>>>    Thanks Nicholas.
>>>>>
>>>>>    I only made a couple small changes to Graeme's code. 1: to load 
>>>>> an image
>>>>>    from a file and write to file and 2: to pass the data vectors by
>>>>>    reference. The last change seems to have sped things up a 
>>>>> little but
>>>>>    it's still taking 110 - 130 ms to compress which is too slow. 
>>>>> We are not
>>>>>    as concerned with decompression speed as that will not need to 
>>>>> occur in
>>>>>    real-time.
>>>>>
>>>>>    I put on our FTP here:
>>>>>    ftp://ftp.rayonix.com/pub/del_in_30_days/byte_offset.tgz.
>>>>>
>>>>>    Thanks,
>>>>>
>>>>>    Justin
>>>>>
>>>>>    On 7/28/11 2:06 PM, Nicholas Sauter wrote:
>>>>>>    Justin,
>>>>>>
>>>>>>    Just some comments based on our experience...first, I haven't 
>>>>>> tried the
>>>>>>    compression extensively, just the decompression. But I've 
>>>>>> found Graeme's
>>>>>>    decompression code to be significantly faster than the CBF 
>>>>>> library, first
>>>>>>    because it is buffer-based instead of file-based, and also 
>>>>>> because it
>>>>>>    hard-codes some assumptions about data depth.
>>>>>>
>>>>>>    I'd be happy to examine this in more detail if there is some 
>>>>>> way to share
>>>>>>    your code example...
>>>>>>
>>>>>>    Nick
>>>>>>
>>>>>>    On Thu, Jul 28, 2011 at 11:46 AM, Justin
>>>>>>    Anderson<justin@rayonix.com>wrote:
>>>>>>
>>>>>>>    Hello all,
>>>>>>>
>>>>>>>    I have run Graeme's byte offset code on a 4k x 4k (2 byte depth)
>>>>>>>    Gaussian
>>>>>>>    noise image and found it to compress the image in around 150 
>>>>>>> ms (64-bit
>>>>>>>    RHEL, Pentium D 3.46GHz). Using CBF library with byte offset
>>>>>>>    compression, I
>>>>>>>    find the compression takes around 125 ms.
>>>>>>>
>>>>>>>    This will be too slow to keep up with our high speed CCD 
>>>>>>> cameras. We are
>>>>>>>    considering parallelizing the byte offset routine by 
>>>>>>> operating on
>>>>>>>    each line
>>>>>>>    of the image individually. Note that this would mean that a 
>>>>>>> given
>>>>>>>    compressed image would be stored differently than via the 
>>>>>>> whole image
>>>>>>>    algorithm.
>>>>>>>
>>>>>>>    Has anyone been thinking about this already or does anyone 
>>>>>>> have any
>>>>>>>    thoughts?
>>>>>>>
>>>>>>>    Regards,
>>>>>>>
>>>>>>>    Justin
>>>>>>>
>>>>>>>    --
>>>>>>>    Justin Anderson
>>>>>>>    Software Engineer
>>>>>>>    Rayonix, LLC
>>>>>>>    justin@rayonix.com
>>>>>>>    1880 Oak Ave. #120
>>>>>>>    Evanston, IL, USA 60201
>>>>>>>    PH:+1.847.869.1548
>>>>>>>    FX:+1.847.869.1587
>>>>>>>
>>>>>>>
>>>>>>>    _______________________________________________
>>>>>>>    imgcif-l mailing list
>>>>>>>    imgcif-l@iucr.org
>>>>>>>    http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>    _______________________________________________
>>>>>    imgcif-l mailing list
>>>>>    imgcif-l@iucr.org
>>>>>    http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>> _______________________________________________
>>>> imgcif-l mailing list
>>>> imgcif-l@iucr.org
>>>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>
>>> -- 
>>> =====================================================
>>>    Herbert J. Bernstein, Professor of Computer Science
>>>      Dowling College, Kramer Science Center, KSC 121
>>>           Idle Hour Blvd, Oakdale, NY, 11769
>>>
>>>                    +1-631-244-3035
>>>                    yaya@dowling.edu
>>> =====================================================
>>> _______________________________________________
>>> imgcif-l mailing list
>>> imgcif-l@iucr.org
>>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>

/*
 * byte_offset.cpp
 * 
 * An implementation of the byte_offset compression scheme used with CBF 
 * images with the hopeful intention of replacing existing Python code for 
 * doing this with something quicker. Main routines are: 
 *
 * vector<char> compress(vector<int>)
 * vector<int> uncompress(vector<char>)
 * 
 */ 

#include <iostream>
#include <fstream>
#include <vector>
#include <cstdlib>
#include <ctime>
#include <cmath>
#include <stdint.h>

using namespace std;

// unions to assist in byte manipulation

typedef union {
  char b[2];
  short s;
} u_s;

typedef union {
  char b[4];
  int i;
} u_i;

// functions for byte swapping

void byte_swap_short(char * b)
{
  char c;
  c = b[0];
  b[0] = b[1];
  b[1] = c;
  return;
}

void byte_swap_int(char * b)
{
  char c;
  c = b[0];
  b[0] = b[3];
  b[3] = c;
  c = b[1];
  b[1] = b[2];
  b[2] = c;
  return;
}

// helper function: is this machine little endian? CBF files are

bool little_endian()
{
  int i = 0x1;
  char b = ((u_i *) &i)[0].b[0];
  if (b == 0)
    {
      return false;
    }
  else
    {
      return true;
    }
}

// main functions

vector<int16_t> read_from_file() {
  int16_t read_short;
  int num_read = 0;
  vector<int16_t> data;

  ifstream fin("comp_test_image.raw", ios::binary);
  if (! fin) {
    cout << "Error opening image file!" << endl;
    return data;
  }

  while (fin.read((char *) &read_short, sizeof(read_short))) {
    data.push_back(read_short);
    ++num_read;
  }

  cout << "read_from_file: read " << num_read << " pixels." << endl;

  fin.close();

  return data;
}

template <class T>
bool write_to_file(const vector<T> data, const string filename) {
  ofstream fout(filename.c_str());

  if (!fout) {
    cout << "write_to_file: ERROR opening file for writing!" << endl;
    return false;
  }

  for(typename vector<T>::const_iterator it = data.begin(); it != data.end(); ++it) {
    fout.write((char *) &(*it), sizeof(*it));
  }

  fout.close();

  return true;
}

void compress(const vector<int16_t> &values, vector<char> &packed)
{
  int current = 0;
  int delta, i;
  unsigned int j;
  //bool le = little_endian();
  short s;
  char c;
  char * b;

  int size = values.size();
  for (j = 0; j < size; j++)
    {
      delta = values[j] - current;

      if ((-127 <= delta) && (delta <= 127))
	{
	  c = (char) delta;
	  packed.push_back(c);
	  current += delta;
	  continue;
	}

      packed.push_back(-128);

      if ((-32767 <= delta) && (delta <= 32767))
	{
	  s = (short) delta;
	  b = ((u_s *) & s)[0].b;

#if 0
	  if (!le) 
	    {
	      byte_swap_short(b);
	    }
#endif

	  packed.push_back(b[0]);
	  packed.push_back(b[1]);
	  
	  current += delta;
	  continue;
	}

      cout << "compress: Rayonix not expected to get here right now!" << endl;

      s = -32768;
      b = ((u_s *) & s)[0].b;

#if 0
      if (!le) 
	{
	  byte_swap_short(b);
	}
#endif

      packed.push_back(b[0]);
      packed.push_back(b[1]);
      
      if ((-2147483647 <= delta) && (delta <= 2147483647))
	{
	  i = delta;
	  b = ((u_i *) & i)[0].b;

#if 0
	  if (!le) 
	    {
	      byte_swap_int(b);
	    }
#endif

	  packed.push_back(b[0]);
	  packed.push_back(b[1]);
	  packed.push_back(b[2]);
	  packed.push_back(b[3]);
	  current += delta;
	  continue;
	}

      /* FIXME I should not get here */

    }
}

void compress_C(const int16_t *values, const int n_values, char *packed, int *n_packed)
{
  int current = 0;
  int delta, i;
  unsigned int j;
  //bool le = little_endian();
  short s;
  char c;
  char * b;
  char *packed_p;

  int size = n_values;
  packed_p = packed;
  for (j = 0; j < size; j++)
    {
      delta = values[j] - current;

      if ((-127 <= delta) && (delta <= 127))
	{
	  c = (char) delta;
	  *packed_p++ = c;
	  current += delta;
	  continue;
	}

      *packed_p++ = -128;

      if ((-32767 <= delta) && (delta <= 32767))
	{
	  s = (short) delta;
	  b = ((u_s *) & s)[0].b;

#if 0
	  if (!le) 
	    {
	      byte_swap_short(b);
	    }
#endif

	  *packed_p++ = b[0];
	  *packed_p++ = b[1];
	  
	  current += delta;
	  continue;
	}

      cout << "compress: Rayonix not expected to get here right now!" << endl;

      s = -32768;
      b = ((u_s *) & s)[0].b;

#if 0
      if (!le) 
	{
	  byte_swap_short(b);
	}
#endif

      *packed_p++ = b[0];
      *packed_p++ = b[1];
      
      if ((-2147483647 <= delta) && (delta <= 2147483647))
	{
	  i = delta;
	  b = ((u_i *) & i)[0].b;

#if 0
	  if (!le) 
	    {
	      byte_swap_int(b);
	    }
#endif

	  *packed_p++ = b[0];
	  *packed_p++ = b[1];
	  *packed_p++ = b[2];
	  *packed_p++ = b[3];
	  current += delta;
	  continue;
	}

      /* FIXME I should not get here */

    }
    *n_packed = packed_p - packed;
}

void uncompress(const vector<char> &packed, vector<int16_t> &values)
{
  int16_t current = 0;
  unsigned int j = 0;
  short s;
  uint16_t t;
  char c;
  int i;
  //bool le = little_endian();

  int size = packed.size();
  while (j < size)
    {
      c = packed[j];
      j += 1;

      if (c != -128)
	{
	  current += c;
	  values.push_back(current);
	  continue;
	}

      ((u_s *) & s)[0].b[0] = packed[j];
      ((u_s *) & s)[0].b[1] = packed[j + 1];

      j += 2;

#if 0 
      if (!le) 
	{
	  byte_swap_short((char *) &s);
	}
#endif

      if (s != -32768)
	{
	  current += s;
	  values.push_back(current);
	  continue;
	}
	  
      cout << "uncompress: Rayonix not expected to get here right now!" << endl;

      ((u_i *) & i)[0].b[0] = packed[j];
      ((u_i *) & i)[0].b[1] = packed[j + 1];
      ((u_i *) & i)[0].b[2] = packed[j + 2];
      ((u_i *) & i)[0].b[3] = packed[j + 3];
      j += 4;
      
#if 0
      if (!le) 
	{
	  byte_swap_int((char *) &i);
	}
#endif

      current += i;
      values.push_back(current);
    }
} 

// helper for timing tests

double ms(clock_t t1, clock_t t2)
{
  return 1000.0 * (t2 - t1) / CLOCKS_PER_SEC;
}

// demo / test code

int main(int argc,
	 char ** argv)
{
  vector<int16_t> values = read_from_file();

  unsigned int j;
  unsigned int size = 4096 * 4096 * 2;
  clock_t start;

#if 0
  start = clock();

  for (j = 0; j < size; j ++)
    {
      values.push_back((rand() & 0xffff));
    }

  cout << "Generating: " << ms(start, clock()) << endl;
#endif



  start = clock();
#define USE_RESERVE 1
#define USE_COMPRESS_C 1
  vector<char> packed;
#if USE_COMPRESS_C
  int packed_size;
  packed.resize(size, 0);
  compress_C(values.data(), values.size(), packed.data(), &packed_size);
  packed.resize(packed_size);
#else
   #if USE_RESERVE
     packed.reserve(size);
   #endif
  compress(values, packed);
#endif
  
  cout << "Packing:    " << ms(start, clock()) << endl;
  write_to_file<char>(packed, "packed.raw");

  start = clock();
  vector<int16_t> unpacked;
  uncompress(packed, unpacked);
  cout << "Unpacking:  " << ms(start, clock()) << endl;
  write_to_file<int16_t>(unpacked, "unpacked.raw");

#if 1
  cout << "Verifying unpacked data matches original data..." << endl;
  for (j = 0; j < unpacked.size(); j ++)
    {
      if (unpacked[j] != values[j])
	{
	  cout << "Error for index " << j << endl;
	}
    }
    cout << "Done." << endl;
#endif

  return 0;
}
  

_______________________________________________
imgcif-l mailing list
imgcif-l@iucr.org
http://scripts.iucr.org/mailman/listinfo/imgcif-l

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links