[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Reply to: [list | sender only]
Re: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
- To: The Crystallographic Binary File and its imgCIF application to image data <imgcif-l@iucr.org>
- Subject: Re: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
- From: "Herbert J. Bernstein" <yaya@bernstein-plus-sons.com>
- Date: Tue, 22 Jun 2010 08:04:16 -0400 (EDT)
- Cc: Graeme.Winter@Diamond.ac.uk
- In-Reply-To: <4C20A497.3080200@esrf.fr>
- References: <4C20A497.3080200@esrf.fr>
Puzzling. This image doe copy fine with cif2cbf. I'll work on it after the eye doctor. -- Herbert ===================================================== Herbert J. Bernstein, Professor of Computer Science Dowling College, Kramer Science Center, KSC 121 Idle Hour Blvd, Oakdale, NY, 11769 +1-631-244-3035 yaya@dowling.edu ===================================================== On Tue, 22 Jun 2010, Jon Wright wrote: > Return-path: <imgcif-l-bounces@iucr.org> > Received: from galena.iucr.org ([192.70.242.40]) > by hermes.dowling.edu with ESMTP; Tue, 22 Jun 2010 07:55:40 -0400 > Received: from ASSP.nospam (galena.iucr.org [127.0.0.1]) > by galena.iucr.org (Postfix) with ESMTP > id 08ECB532E3; Tue, 22 Jun 2010 12:55:18 +0100 (BST) > Received: from dynhost1.iucr.org ([192.70.242.92] helo=dynhost1.iucr.org) > with IPv4:25 by ASSP.nospam; 22 Jun 2010 12:55:18 +0100 > Received: from dynhost1.iucr.org (dynhost1.iucr.org [127.0.0.1]) > by dynhost1.iucr.org (Postfix) with ESMTP > id D26DF9B59A; Tue, 22 Jun 2010 12:55:17 +0100 (BST) > X-Original-To: imgcif-l@dynhost1.iucr.org > Delivered-To: imgcif-l@dynhost1.iucr.org > Received: from galena.iucr.org (galena.iucr.org [192.70.242.40]) > by dynhost1.iucr.org (Postfix) with ESMTP id 8ABF59B59A > for <imgcif-l@dynhost1.iucr.org>; Tue, 22 Jun 2010 12:55:15 +0100 > (BST) > Received: by galena.iucr.org (Postfix) > id 5D27A532F1; Tue, 22 Jun 2010 12:55:15 +0100 (BST) > Delivered-To: imgcif-l@iucr.org > Received: from ASSP.nospam (galena.iucr.org [127.0.0.1]) > by galena.iucr.org (Postfix) with ESMTP id 0C1C4532E3 > for <imgcif-l@iucr.org>; Tue, 22 Jun 2010 12:55:15 +0100 (BST) > Received: from gtwin2.esrf.fr ([193.49.43.102] helo=gtwin2.esrf.fr) with > IPv4:25 by ASSP.nospam; 22 Jun 2010 12:55:14 +0100 > Received: (from uucp@localhost) by gtwin2.esrf.fr (1.0.0) id o5MBtCNf335843; > Tue, 22 Jun 2010 13:55:12 +0200 (MEST) > Received: from mailsrv.esrf.fr(160.103.2.161) by gtwout2 via smap (V1.3) > id tmp335837; Tue Jun 22 13:55:09 2010 > Received: from novirus.esrf.fr (novirus.esrf.fr [160.103.2.175]) > by mailsrv.esrf.fr (8.13.8+Sun/8.13.8) with ESMTP id o5MBt80K004948; > Tue, 22 Jun 2010 13:55:08 +0200 (MEST) > Received: from [127.0.0.1] (pcmedstudent2.esrf.fr [160.103.106.183]) > by novirus.esrf.fr (8.13.1/8.13.1) with ESMTP id o5MBnP9r032172; > Tue, 22 Jun 2010 13:49:29 +0200 > Message-ID: <4C20A497.3080200@esrf.fr> > Date: Tue, 22 Jun 2010 13:55:03 +0200 > From: Jon Wright <wright@esrf.fr> > User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) > MIME-Version: 1.0 > To: "Herbert J. Bernstein" <yaya@bernstein-plus-sons.com>, > Graeme.Winter@Diamond.ac.uk > Content-Type: multipart/mixed; > boundary="------------090102040500010601060009" > X-Assp-Delay: not delayed (wright@esrf.fr on whitelistdb); 22 Jun 2010 > 12:55:15 +0100 > X-Assp-Whitelisted: Yes (wright@esrf.fr on whitelistdb) > X-Assp-Envelope-From: wright@esrf.fr > X-Assp-Passing: wright@esrf.fr on whitelistdb > X-Assp-ID: ASSP.nospam () > X-Assp-Version: 1.7.5.2(1.0.02) > Cc: The Crystallographic Binary File and its imgCIF application to image data > <imgcif-l@iucr.org> > Subject: [Imgcif-l] [Fwd: RE: Reading CBF files from Python] > X-BeenThere: imgcif-l@iucr.org > X-Mailman-Version: 2.1.9 > Precedence: list > Reply-To: The Crystallographic Binary File and its imgCIF application to > image data <imgcif-l@iucr.org> > List-Id: The Crystallographic Binary File and its imgCIF application to image > data <imgcif-l.iucr.org> > List-Unsubscribe: <http://scripts.iucr.org/mailman/listinfo/imgcif-l>, > <mailto:imgcif-l-request@iucr.org?subject=unsubscribe> > List-Archive: <http://scripts.iucr.org/pipermail/imgcif-l> > List-Post: <mailto:imgcif-l@iucr.org> > List-Help: <mailto:imgcif-l-request@iucr.org?subject=help> > List-Subscribe: <http://scripts.iucr.org/mailman/listinfo/imgcif-l>, > <mailto:imgcif-l-request@iucr.org?subject=subscribe> > Sender: imgcif-l-bounces@iucr.org > Errors-To: imgcif-l-bounces@iucr.org > X-Assp-Passing: acceptAllMail > X-Assp-ID: ASSP.nospam () > X-Assp-Version: 1.7.5.2(1.0.02) > > This is a multi-part message in MIME format. > --------------090102040500010601060009 > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > Content-Transfer-Encoding: 7bit > > Hi Herbert, Graeme, > > I had a quick look with the pycbf bindings and ran into a problem with the > example image (http://www.ccp4.ac.uk/xia/image.cbf). I get a slew of warnings > and finally an error coming from CBFlib: > > coral24:~/cbflib_jun2010/CBFlib_0.7.9/pycbf % python tst.py image.cbf 2>&1 | > more > CBFlib: warning input line 26 (1) -- invalid character > CBFlib: warning input line 26 (2) -- invalid character > CBFlib: warning input line 26 (3) -- invalid character > .. > CBFlib: warning input line 26 (1978) -- invalid character > CBFlib: warning input line 26 (1979) -- invalid character > CBFlib: error input line 26 (1) -- value without tag > > > The code in python is essentially: > > import pycbf > o = pycbf.cbf_handle_struct() > o.read_file("image.cbf", pycbf.MSG_DIGEST) > > In c this translates into something like the following, and gives the same > output: > > #include "cbf_simple.h" > #include <stdlib.h> > #include <stdio.h> > int main (int argc, char *argv []) > { > FILE *in; > cbf_handle cbf; > if (!(in = fopen (argv[1], "rb"))) { > fprintf (stderr,"Couldn't open the input cbf file %s\n", argv[1]); > exit (1); > } > cbf_failnez (cbf_make_handle (&cbf)); > cbf_failnez (cbf_read_widefile (cbf, in, MSG_DIGEST)); > } > > > A workaround is to copy the binary section into a valid file and then read > that file, but I don't think it is what Graeme is after. Code to do that is > attached in case it is useful in the meantime. > > Sorry I can't be more useful, > > Jon > > > > > > > -------- Original Message -------- > Subject: RE: [Imgcif-l] Reading CBF files from Python > Date: Mon, 21 Jun 2010 14:20:38 +0100 > From: <Graeme.Winter@Diamond.ac.uk> > To: <wright@esrf.fr> > CC: <Graeme.Winter@Diamond.ac.uk> > References: > <4854F2500EA8C4478A508D2D92973E5206D6B5F9@EXCHANGE25.fed.cclrc.ac.uk> > <4C1F2B5C.30400@esrf.fr><4854F2500EA8C4478A508D2D92973E5206D6B5FA@EXCHANGE25.fed.cclrc.ac.uk> > <4C1F3784.6060107@esrf.fr> > > Hi Jon, > > Thanks - I think that the problem is more than this image does not > behave like a real cbf, though having some similarities. However I tried > building pycbf as you suggest and the build failed - I think the > configuration is not right in the way the cctbx folks use cbflib. > > I have attached the image (picture would be a better word) - when I try > to load it in to cctbx using the MiniCBFImage loader it fails indicating > I think that it is poorly structured. > > I am sure that there is code in cbflib which will do exactly what I > need, the only problem is gaining access to it! > > Best wishes, > > Graeme > > > > -----Original Message----- > From: imgcif-l-bounces@iucr.org [mailto:imgcif-l-bounces@iucr.org] On > Behalf Of Jon Wright > Sent: 21 June 2010 10:57 > To: The Crystallographic Binary File and its imgCIF application to image > data > Subject: Re: [Imgcif-l] Reading CBF files from Python > > Hi Graeme, > > To use the binding the wrapper needs to be compiled, which should just > mean typing "python setup.py build" in the pycbf subfolder. Then you > just need to drop pycbf.py and _pycbf.so somewhere on your python path > (import sys; print sys.path). The binding runs entirely via calls to > CBFlib, so if there is something useful in there, it should be possible > to include it. Herbert can probably help more for knowing where to look. > > If you can send an example image off-list I'll try to take a look > > Best > > Jon > > Graeme.Winter@Diamond.ac.uk wrote: >> Hi Jon, >> >> Thanks - I see that this is bundled with cctbx and Labelit, which was >> awesome. However it seems that the modules from pycbf aren't available > >> as standard :o( - I am slightly hamstrung by the fact that I use the cctbx >> python, so having access to other python extensions presents a problem. >> However I *really need* cctbx :o) - xia2 won't work without it now. >> >> My rate limiting step right now is the reading and writing of the byte > >> offset compression. I am going to guess that if I messed with the python >> paths and such I should be able to find your implementation. Do > >> you have a method in there which will allow me to just pack and unpack > >> the byte_offset compressed regions? >> >> This is the data in question: >> >> ###CBF: Version July 2008 generated by XDS >> >> data_FRAME.cbf >> >> _array_data.header_convention "XDS special" >> _array_data.header_contents >> ; >> ; >> >> _array_data.data >> ; >> --CIF-BINARY-FORMAT-SECTION-- >> Content-Type: application/octet-stream; >> conversions="x-CBF_BYTE_OFFSET" >> Content-Transfer-Encoding: BINARY >> X-Binary-Size: 9450352 >> X-Binary-ID: 1 >> X-Binary-Element-Type: "signed 32-bit integer" >> X-Binary-Element-Byte-Order: LITTLE_ENDIAN >> X-Binary-Number-of-Elements: 9437184 >> X-Binary-Size-Fastest-Dimension: 3072 >> X-Binary-Size-Second-Dimension: 3072 >> (massive binary blob) >> >> The bit which is causing me problems is that the cbflib adaptor raises > >> an exception as most of the "proper" cbf stuff (i.e. the CIF) is not >> present. Erk. >> >> Any suggestions will be pounced apon! >> >> Many thanks, >> >> Graeme >> >> -----Original Message----- >> From: imgcif-l-bounces@iucr.org [mailto:imgcif-l-bounces@iucr.org] On >> Behalf Of Jon Wright >> Sent: 21 June 2010 10:06 >> To: The Crystallographic Binary File and its imgCIF application to image >> data >> Subject: Re: [Imgcif-l] Reading CBF files from Python >> >> Hi Graeme, >> >> There is a swig based python binding included with CBFlib in the pycbf > >> subfolder. It should work for reading files (see pycbf_test1.py). >> Further work was needed for writing. Let me know if you run into problems. >> Improvements are most welcome. >> >> Best >> >> Jon >> >> Graeme.Winter@Diamond.ac.uk wrote: >>> Hi Folks, >>> Does anyone know if CBF images can be read from Python? I coded up >>> something in Python to do this but it's a spot slow to say the least. >>> My interest is partly in reading the cbf "image" files calculated by XDS >>> to explain what it has been up to. I could code something up in 'c' or a >>> bridge to CBFlib but that would be hardish to maintain, so a > >>> solution using existing Python libraries would suit me better. >>> I looked quickly at PyCIFrw but this appears to be mostly about the CIF >>> side of things rather than the MIME encoded chunks. Please correct >> >>> me if I am wrong. I also tried iotbx.detectors.cbf.CBFImage but hit an >> >>> exception which probably relates to this being more of a picture than > >>> a diffraction image. >>> Thanks, >>> Graeme >>> Dr. Graeme Winter >>> Software and MX Support Scientist >>> Diamond Light Source >>> +44 1235 778091 (work) >>> +44 7786 662784 (work mobile) >>> >>> >>> >> >> _______________________________________________ >> imgcif-l mailing list >> imgcif-l@iucr.org >> http://scripts.iucr.org/mailman/listinfo/imgcif-l >> > > _______________________________________________ > imgcif-l mailing list > imgcif-l@iucr.org > http://scripts.iucr.org/mailman/listinfo/imgcif-l > > > > -- > > This e-mail and any attachments may contain confidential, copyright and or > privileged material, and are for the use of the intended addressee only. If > you are not the intended addressee or an authorised recipient of the > addressee please notify us of receipt by returning the e-mail and do not use, > copy, retain, distribute or disclose the information in or attached to the > e-mail. > > Any opinions expressed within this e-mail are those of the individual and not > necessarily of Diamond Light Source Ltd. > > Diamond Light Source Ltd. cannot guarantee that this e-mail or any > attachments are free from viruses and we cannot accept liability for any > damage which you may sustain as a result of software viruses which may be > transmitted in or with the message. > > Diamond Light Source Limited (company no. 4375679). Registered in England and > Wales with its registered office at Diamond House, Harwell Science and > Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > > > > > > > > > > --------------090102040500010601060009 > Content-Type: text/plain; > name="read_graeme.py" > Content-Transfer-Encoding: 7bit > Content-Disposition: inline; > filename="read_graeme.py" > > > import pycbf, sys, tempfile, time, numpy as np > > def fix_im( barecbf, template, tempname ): > temp = open(tempname, "wb") > tag = "--CIF-BINARY-FORMAT-SECTION--" > good = open(template,"rb").read() > s = good.find(tag) + len(tag) > temp.write( good[:s] ) > eg = s + good[s:].find("--CIF") > bin = open(barecbf, "rb").read() > s = bin.find(tag)+len(tag) > e = s + bin[s:].find("--CIF") > temp.write( bin[s:e] ) > temp.write( good[ eg: ] ) > temp.flush() > temp.close() > > > > > def read_binary(name ): > object = pycbf.cbf_handle_struct() > object.read_file(name,pycbf.MSG_DIGEST) > object.rewind_datablock() > object.select_datablock(0) > object.rewind_category() > categories = object.count_categories() > for i in range(categories): > object.select_category(i) > category_name = object.category_name() > rows=object.count_rows() > cols = object.count_columns() > loop=1 > object.rewind_column() > while loop is not 0: > column_name = object.column_name() > try: > object.next_column() > except: > break > for j in range(rows): > object.select_row(j) > object.rewind_column() > for k in range(cols): > name=object.column_name() > object.select_column(k) > typeofvalue=object.get_typeofvalue() > if typeofvalue.find("bnry") > -1: > s=object.get_integerarray_as_string() > ar = np.fromstring(s,np.int32) > else: > value=object.get_value() > # print "Val:",value,i > del(object) > return ar > > start = time.time() > fix_im(sys.argv[1], sys.argv[2], sys.argv[3] ) > print "fixing",time.time()-start > start = time.time() > image = read_binary(sys.argv[3]) > print "reading",time.time()-start > print image.max(), image.min(), image.mean() > > --------------090102040500010601060009 > Content-Type: text/plain; charset="us-ascii" > MIME-Version: 1.0 > Content-Transfer-Encoding: 7bit > Content-Disposition: inline > > _______________________________________________ > imgcif-l mailing list > imgcif-l@iucr.org > http://scripts.iucr.org/mailman/listinfo/imgcif-l > > --------------090102040500010601060009-- > > _______________________________________________ imgcif-l mailing list imgcif-l@iucr.org http://scripts.iucr.org/mailman/listinfo/imgcif-l
Reply to: [list | sender only]
- Follow-Ups:
- References:
- [Imgcif-l] [Fwd: RE: Reading CBF files from Python] (Jon Wright)
- Prev by Date: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
- Next by Date: Re: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
- Prev by thread: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
- Next by thread: Re: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
- Index(es):