Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]

Hi,

Apologies for the noise. Seems things work using 0.9.0 from sourceforge 
with a small patch in the Makefile to add -fPIC to CFLAGS so that python 
can link to libcbf.a directly. Details are below. The main thing which 
seems to be missing is that the array dimensions are getting lost in the 
output as I've coded below. In the input there was:

X-Binary-Size-Fastest-Dimension:    2048
X-Binary-Size-Second-Dimension:    2048

I'm not quite sure how to pick these up when reading the data item, or 
how to send them back again.

All the best

Jon





wget \ 
http://sourceforge.net/projects/cbflib/files/cbflib/CBFlib_0.9.0/CBFlib-0.9.0_14Feb10.tar.gz/download
tar -zxf CBFlib-0.9.0_14Feb10.tar.gz
cd CBFlib-0.9.0
vi Makefile # add -fPIC to CFLAGS
make all
cd pycbf
python setup.py
cp build/lib.linux-x86_64-2.5/_pycbf.so .
wget http://www.ccp4.ac.uk/xia/image.cbf


~/cbflib_jun2010/CBFlib-0.9.0/pycbf % more demo.py
import pycbf, sys, time, numpy as np

start = time.time()
o = pycbf.cbf_handle_struct()
o.read_file(sys.argv[1],pycbf.MSG_NODIGEST)
o.select_category(0)
o.rewind_column()
while 1:
     o.next_column()
     if o.column_name() == "data":
         s=o.get_integerarray_as_string()
         break

data = np.fromstring( s , np.int32)
print "Max",data.max(), "Min",data.min(), "Mean",data.mean()
print "Time",time.time()-start

# modify the data and write out
data = data + 10

# This is somewhat cryptic:
comp, id, elsiz, elsign, elunsign, elem, mine, maxe = \
       o.get_integerarrayparameters()
o.set_integerarray( comp, id, data.tostring(), elsiz, elsign, elem )
o.write_file(sys.argv[2], # filename
              pycbf.CBF,
              pycbf.MIME_HEADERS,
              pycbf.ENC_BASE64 )



~/cbflib_jun2010/CBFlib-0.9.0/pycbf % python demo.py image.cbf imagep10.cbf
Max 38182 Min -3 Mean 14124.4636993
Time 0.377423048019



Jon Wright wrote:
> Thanks Herbert - perhaps the problem is that I am using 0.7.9, and also 
> see the error on cif2cbf. Will try upgrading.
> Best
> 
> Jon
> 
> 
> Herbert J. Bernstein wrote:
>> Puzzling.  This image doe copy fine with cif2cbf.  I'll work on it
>> after the eye doctor. -- Herbert
>>
>> =====================================================
>>   Herbert J. Bernstein, Professor of Computer Science
>>     Dowling College, Kramer Science Center, KSC 121
>>          Idle Hour Blvd, Oakdale, NY, 11769
>>
>>                   +1-631-244-3035
>>                   yaya@dowling.edu
>> =====================================================
>>
>> On Tue, 22 Jun 2010, Jon Wright wrote:
>>
>>> Hi Herbert, Graeme,
>>>
>>> I had a quick look with the pycbf bindings and ran into a problem with the 
>>> example image (http://www.ccp4.ac.uk/xia/image.cbf). I get a slew of warnings 
>>> and finally an error coming from CBFlib:
>>>
>>> coral24:~/cbflib_jun2010/CBFlib_0.7.9/pycbf % python tst.py image.cbf 2>&1 | 
>>> more
>>> CBFlib: warning input line 26 (1) -- invalid character
>>> CBFlib: warning input line 26 (2) -- invalid character
>>> CBFlib: warning input line 26 (3) -- invalid character
>>> ..
>>> CBFlib: warning input line 26 (1978) -- invalid character
>>> CBFlib: warning input line 26 (1979) -- invalid character
>>> CBFlib: error input line 26 (1) -- value without tag
>>>
>>>
>>> The code in python is essentially:
>>>
>>>   import pycbf
>>>   o = pycbf.cbf_handle_struct()
>>>   o.read_file("image.cbf", pycbf.MSG_DIGEST)
>>>
>>> In c this translates into something like the following, and gives the same 
>>> output:
>>>
>>> #include "cbf_simple.h"
>>> #include <stdlib.h>
>>> #include <stdio.h>
>>> int main (int argc, char *argv [])
>>> {
>>>  FILE *in;
>>>  cbf_handle cbf;
>>>  if (!(in = fopen (argv[1], "rb"))) {
>>>     fprintf (stderr,"Couldn't open the input cbf file %s\n", argv[1]);
>>>     exit (1);
>>>   }
>>>  cbf_failnez (cbf_make_handle (&cbf));
>>>  cbf_failnez (cbf_read_widefile (cbf, in, MSG_DIGEST));
>>> }
>>>
>>>
>>> A workaround is to copy the binary section into a valid file and then read 
>>> that file, but I don't think it is what Graeme is after. Code to do that is 
>>> attached in case it is useful in the meantime.
>>>
>>> Sorry I can't be more useful,
>>>
>>> Jon
>>>
>>>
>>>
>>>
>>>
>>>
>>> -------- Original Message --------
>>> Subject: RE: [Imgcif-l] Reading CBF files from Python
>>> Date: Mon, 21 Jun 2010 14:20:38 +0100
>>> From: <Graeme.Winter@Diamond.ac.uk>
>>> To: <wright@esrf.fr>
>>> CC: <Graeme.Winter@Diamond.ac.uk>
>>> References: 
>>> <4854F2500EA8C4478A508D2D92973E5206D6B5F9@EXCHANGE25.fed.cclrc.ac.uk> 
>>> <4C1F2B5C.30400@esrf.fr><4854F2500EA8C4478A508D2D92973E5206D6B5FA@EXCHANGE25.fed.cclrc.ac.uk> 
>>> <4C1F3784.6060107@esrf.fr>
>>>
>>> Hi Jon,
>>>
>>> Thanks - I think that the problem is more than this image does not
>>> behave like a real cbf, though having some similarities. However I tried
>>> building pycbf as you suggest and the build failed - I think the
>>> configuration is not right in the way the cctbx folks use cbflib.
>>>
>>> I have attached the image (picture would be a better word) - when I try
>>> to load it in to cctbx using the MiniCBFImage loader it fails indicating
>>> I think that it is poorly structured.
>>>
>>> I am sure that there is code in cbflib which will do exactly what I
>>> need, the only problem is gaining access to it!
>>>
>>> Best wishes,
>>>
>>> Graeme
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: imgcif-l-bounces@iucr.org [mailto:imgcif-l-bounces@iucr.org] On
>>> Behalf Of Jon Wright
>>> Sent: 21 June 2010 10:57
>>> To: The Crystallographic Binary File and its imgCIF application to image
>>> data
>>> Subject: Re: [Imgcif-l] Reading CBF files from Python
>>>
>>> Hi Graeme,
>>>
>>> To use the binding the wrapper needs to be compiled, which should just
>>> mean typing "python setup.py build" in the pycbf subfolder. Then you
>>> just need to drop pycbf.py and _pycbf.so somewhere on your python path
>>> (import sys; print sys.path). The binding runs entirely via calls to
>>> CBFlib, so if there is something useful in there, it should be possible
>>> to include it. Herbert can probably help more for knowing where to look.
>>>
>>> If you can send an example image off-list I'll try to take a look
>>>
>>> Best
>>>
>>> Jon
>>>
>>> Graeme.Winter@Diamond.ac.uk wrote:
>>>> Hi Jon,
>>>>
>>>> Thanks - I see that this is bundled with cctbx and Labelit, which was 
>>>> awesome. However it seems that the modules from pycbf aren't available
>>>> as standard :o(  - I am slightly hamstrung by the fact that I use the cctbx 
>>>> python, so having access to other python extensions presents a problem. 
>>>> However I *really need* cctbx :o) - xia2 won't work without it now.
>>>>
>>>> My rate limiting step right now is the reading and writing of the byte
>>>> offset compression. I am going to guess that if I messed with the python 
>>>> paths and such I should be able to find your implementation. Do
>>>> you have a method in there which will allow me to just pack and unpack
>>>> the byte_offset compressed regions?
>>>>
>>>> This is the data in question:
>>>>
>>>> ###CBF: Version July 2008 generated by XDS
>>>>
>>>> data_FRAME.cbf
>>>>
>>>> _array_data.header_convention "XDS special"
>>>> _array_data.header_contents
>>>> ;
>>>> ;
>>>>
>>>> _array_data.data
>>>> ;
>>>> --CIF-BINARY-FORMAT-SECTION--
>>>> Content-Type: application/octet-stream;
>>>>      conversions="x-CBF_BYTE_OFFSET"
>>>> Content-Transfer-Encoding: BINARY
>>>> X-Binary-Size:   9450352
>>>> X-Binary-ID: 1
>>>> X-Binary-Element-Type: "signed 32-bit integer"
>>>> X-Binary-Element-Byte-Order: LITTLE_ENDIAN
>>>> X-Binary-Number-of-Elements:   9437184
>>>> X-Binary-Size-Fastest-Dimension:    3072
>>>> X-Binary-Size-Second-Dimension:    3072
>>>> (massive binary blob)
>>>>
>>>> The bit which is causing me problems is that the cbflib adaptor raises
>>>> an exception as most of the "proper" cbf stuff (i.e. the CIF) is not 
>>>> present. Erk.
>>>>
>>>> Any suggestions will be pounced apon!
>>>>
>>>> Many thanks,
>>>>
>>>> Graeme
>>>>
>>>> -----Original Message-----
>>>> From: imgcif-l-bounces@iucr.org [mailto:imgcif-l-bounces@iucr.org] On 
>>>> Behalf Of Jon Wright
>>>> Sent: 21 June 2010 10:06
>>>> To: The Crystallographic Binary File and its imgCIF application to image 
>>>> data
>>>> Subject: Re: [Imgcif-l] Reading CBF files from Python
>>>>
>>>> Hi Graeme,
>>>>
>>>> There is a swig based python binding included with CBFlib in the pycbf
>>>> subfolder. It should work for reading files (see pycbf_test1.py).
>>>> Further work was needed for writing. Let me know if you run into problems. 
>>>> Improvements are most welcome.
>>>>
>>>> Best
>>>>
>>>> Jon
>>>>
>>>> Graeme.Winter@Diamond.ac.uk wrote:
>>>>> Hi Folks,
>>>>>  Does anyone know if CBF images can be read from Python? I coded up 
>>>>> something in Python to do this but it's a spot slow to say the least.
>>>>> My interest is partly in reading the cbf "image" files calculated by XDS 
>>>>> to explain what it has been up to. I could code something up in 'c' or a 
>>>>> bridge to CBFlib but that would be hardish to maintain, so a
>>>>> solution using existing Python libraries would suit me better.
>>>>>  I looked quickly at PyCIFrw but this appears to be mostly about the CIF 
>>>>> side of things rather than the MIME encoded chunks. Please correct
>>>>> me if I am wrong. I also tried iotbx.detectors.cbf.CBFImage but hit an
>>>>> exception which probably relates to this being more of a picture than
>>>>> a diffraction image.
>>>>>  Thanks,
>>>>>  Graeme
>>>>>  Dr. Graeme Winter
>>>>> Software and MX Support Scientist
>>>>> Diamond Light Source
>>>>>  +44 1235 778091 (work)
>>>>> +44 7786 662784 (work mobile)
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> imgcif-l mailing list
>>>> imgcif-l@iucr.org
>>>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>>
>>> _______________________________________________
>>> imgcif-l mailing list
>>> imgcif-l@iucr.org
>>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>
>>>
>>>
>>> -- 
>>>
>>> This e-mail and any attachments may contain confidential, copyright and or 
>>> privileged material, and are for the use of the intended addressee only. If 
>>> you are not the intended addressee or an authorised recipient of the 
>>> addressee please notify us of receipt by returning the e-mail and do not use, 
>>> copy, retain, distribute or disclose the information in or attached to the 
>>> e-mail.
>>>
>>> Any opinions expressed within this e-mail are those of the individual and not 
>>> necessarily of Diamond Light Source Ltd.
>>>
>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
>>> attachments are free from viruses and we cannot accept liability for any 
>>> damage which you may sustain as a result of software viruses which may be 
>>> transmitted in or with the message.
>>>
>>> Diamond Light Source Limited (company no. 4375679). Registered in England and 
>>> Wales with its registered office at Diamond House, Harwell Science and 
>>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --------------090102040500010601060009
>>> Content-Type: text/plain;
>>> name="read_graeme.py"
>>> Content-Transfer-Encoding: 7bit
>>> Content-Disposition: inline;
>>> filename="read_graeme.py"
>>>
>>>
>>> import pycbf, sys, tempfile, time, numpy as np
>>>
>>> def fix_im( barecbf, template, tempname ):
>>>   temp = open(tempname, "wb")
>>>   tag = "--CIF-BINARY-FORMAT-SECTION--"
>>>   good = open(template,"rb").read()
>>>   s = good.find(tag) + len(tag)
>>>   temp.write( good[:s] )
>>>   eg = s + good[s:].find("--CIF")
>>>   bin = open(barecbf, "rb").read()
>>>   s = bin.find(tag)+len(tag)
>>>   e = s + bin[s:].find("--CIF")
>>>   temp.write( bin[s:e] )
>>>   temp.write( good[ eg: ] )
>>>   temp.flush()
>>>   temp.close()
>>>
>>>
>>>
>>>
>>> def read_binary(name ):
>>>   object = pycbf.cbf_handle_struct()
>>>   object.read_file(name,pycbf.MSG_DIGEST)
>>>   object.rewind_datablock()
>>>   object.select_datablock(0)
>>>   object.rewind_category()
>>>   categories = object.count_categories()
>>>   for i in range(categories):
>>>       object.select_category(i)
>>>       category_name = object.category_name()
>>>       rows=object.count_rows()
>>>       cols = object.count_columns()
>>>       loop=1
>>>       object.rewind_column()
>>>       while loop is not 0:
>>>           column_name = object.column_name()
>>>           try:
>>>               object.next_column()
>>>           except:
>>>               break
>>>       for j in range(rows):
>>>           object.select_row(j)
>>>           object.rewind_column()
>>>           for k in range(cols):
>>>               name=object.column_name()
>>>               object.select_column(k)
>>>               typeofvalue=object.get_typeofvalue()
>>>               if typeofvalue.find("bnry") > -1:
>>>                   s=object.get_integerarray_as_string()
>>>                   ar = np.fromstring(s,np.int32)
>>>               else:
>>>                   value=object.get_value()
>>>                   # print "Val:",value,i
>>>   del(object)
>>>   return ar
>>>
>>> start = time.time()
>>> fix_im(sys.argv[1], sys.argv[2], sys.argv[3] )
>>> print "fixing",time.time()-start
>>> start = time.time()
>>> image = read_binary(sys.argv[3])
>>> print "reading",time.time()-start
>>> print image.max(), image.min(), image.mean()
>>>
>>> --------------090102040500010601060009
>>> Content-Type: text/plain; charset="us-ascii"
>>> MIME-Version: 1.0
>>> Content-Transfer-Encoding: 7bit
>>> Content-Disposition: inline
>>>
>>> _______________________________________________
>>> imgcif-l mailing list
>>> imgcif-l@iucr.org
>>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>
>>> --------------090102040500010601060009--
>>>
>>>
>> _______________________________________________
>> imgcif-l mailing list
>> imgcif-l@iucr.org
>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
> 
> _______________________________________________
> imgcif-l mailing list
> imgcif-l@iucr.org
> http://scripts.iucr.org/mailman/listinfo/imgcif-l

_______________________________________________
imgcif-l mailing list
imgcif-l@iucr.org
http://scripts.iucr.org/mailman/listinfo/imgcif-l

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links