473,804 Members | 3,750 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Manipulating large blobs in Python

I'm working on a module that will manipulate large blobs. I'm using a C
dll to allocate big blocks of memory, using PyMem_Malloc, which is
working quite well up until I try to manipulate a blob that exhausts
Python's heap. I'm guessing that to increase the heapsize, I'm going to
have to recompile python... (??) Am I on the right track? If so, can
you steer me toward some instructions on how to do that compile? I'm a
command-line make kind of dude, and I don't see any of that kind of
stuff in the source distribution. I see .vcproj xml-ish files... do I
need some version of visual studio ide to do the windows compile? It'd
be great if there was some way to increase the heapsize without all that
bother :)

Tim Stone
Jul 19 '05 #1
0 986

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
3663
by: Raaijmakers, Vincent (GE Infrastructure) | last post by:
It is the first time that I use blobs in mysql. Please help me out here..... Using MSSQLdb and python 2.3.4 I was surprised to see how my information was stored in the blob. My goal is to store JPG images in the blob. Well, it stores it but as a string of bytes: 'x0dx0fxffxa3......' So, when reading back that information into my python environment, it seems that I need to translate the value back to the original binary format? ...
1
3363
by: Kirby Urner | last post by:
I've been testing the Cookbook example 8.6 (2002 edition) re using cPickle to insert and retrieve BLOBs from mySQL, using Python's MySQLdb module. When I try to cPickle.loads(blob), I get an error telling me that loads wants a string, not type array.array. So I go cPickle.loads(blob.tostring()) instead and it works. My question is: has something changed in the Python API since this example was written?
1
6203
by: Good Man | last post by:
Hi there I'm developing a large web application. Part of this web application will be storing numerical chart data in a MySQL table - these numbers will be already calculated, and are just being stored for reference. In this particular table, the stored data will never be deleted or changed. The only actions performed will be SELECTs and INSERTs. There will never be any DELETEs or UPDATEs.
7
6955
by: Howard Lowndes | last post by:
My situation is that I am interacting PHP 4.1.2 to PostgreSQL 7.2.2 I have no difficulty inserting and managing BLOBs into the Large Object system table, and I have a user table called images which maintains the relationship between the BLOB loid and the identity that relates to it in my user tables. So far so good. When I RTFM obout psql it refers to the \lo_import, \lo_list, \lo_export and \lo_unlink functions.
4
1453
by: David Helgason | last post by:
What: I'm having trouble finding out how to find the current PGconn connection inside a C function. Looking through the documentation didn't give this up. Could anyone suggest where to look? I didn't even see anything similar to this in the SPI_* documentation. Perhaps I am totally mislead here? Why: I am writing an wrapper around librsync, allowing differential updating of large amounts of data (librsync wraps the clever algorithm of
5
5613
by: Louis LeBlanc | last post by:
Hey folks. I'm new to the list, and not quite what you'd call a DB Guru, so please be patient with me. I'm afraid the lead up here is a bit verbose . . . I am working on an application that uses very high volume DB transactions - in the order of tens of millions per day . . . Anyway, the current database which will remain nameless, but begins with O and rymes with debacle (sorta), has a problem with high volume work when it comes to...
1
2334
by: Jeremiah Jahn | last post by:
will an alter table that removes an oid type also remove all of the associated large objects. I've been using blobs but have converted to byte arrays, now I need to get rid of all of the blobs. Will this be enough? followed by a vaccum of course. -- Jeremiah Jahn <jeremiah@cs.earlham.edu>
2
1171
by: Aaron Watters | last post by:
Poking around I discovered somewhere someone saying that Python gc adds a 4-7% speed penalty. So since I was pretty sure I was not creating reference cycles in nucular I tried running the tests with garbage collection disabled. To my delight I found that index builds run 30-40% faster without gc. This is really nice because testing gc.collect() afterward shows that gc was not actually doing anything.
1
1495
by: FeelLikeANut | last post by:
My question is partly a language issue (memory) and partly a Web issue (where the program runs), so I hope everyone will be open to helping. Users may submit and store pictures with my program. I have the choice of storing these pictures either as files on the server or as a blob in the database. The pictures are just user data, and it seems logical to keep them in the database with all the other data. But I'm concerned about how to...
0
10583
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10082
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9160
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7622
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6854
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5525
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4301
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3822
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2995
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.