System resets when loading a large table

Beau E. Cox

Hi -

I have an application which is essentially 'read-only' -
once the database is loaded it is only read from. So,
I am trying to implement it with various indices and
a table containg some large content in blob columns.
I observe the max_allowed_pac ket setting and
'split' blobs too large among multiple rows. All is fine
during testing. However, when I try a production
load (which should load about 1.5G to the various
tables) the system resets at or near the 1G mark
leaving no logs/msgs (that I can find) telling me
what went wrong. I have tried the following to no
avail:
- 'disable' bin logging on the main 'content' table
- RAID that table into 8 chunks.
Sill, when the total in the .MYD chunks reaches
1GB - KABOOM!

My configuration:
SuSE Linux 8,1
Mysql 4.0.13
loading with perl 5.8 using DBI 1.37/
DBD::mysql 2.9002.

Can someone suggest a next step?

Aloha => Beau;
== please visit ==
<http://beaucox.com> => main site
<http://howtos.beaucox. com> => howtos
<http://PPM.beaucox.com > => perl PPMs
<http://CPAN.beaucox.co m> => CPAN
== thank you ==

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/my***********...ie.nctu.edu.tw

Jul 19 '05 #1

Subscribe Reply

1570

Aggro

Beau E. Cox wrote:

tables) the system resets at or near the 1G mark
leaving no logs/msgs (that I can find) telling me
what went wrong. I have tried the following to no
avail:

I'm no expert but could this be hardware failure? Perhaps something
wrong with cpu, memory or motherboard that causes the hardware to do
panic solution.

I had a problem with cpu. It couldn't take a lot of cpu load without
booting the machine.

Jul 19 '05 #2

Aggro

Beau E. Cox wrote:

tables) the system resets at or near the 1G mark
leaving no logs/msgs (that I can find) telling me
what went wrong. I have tried the following to no
avail:

Jul 19 '05 #3

Similar topics

2922

Slow loading of large in-memory tables

by: Philipp K. Janert, Ph.D. | last post by:

Dear All! I am trying to load a relatively large table (about 1 Million rows) into an sqlite table, which is kept in memory. The load process is very slow - on the order of 15 minutes or so. I am accessing sqlite from Python, using the pysqlite driver. I am loading all records first using cx.execute( "insert ..." ). Only once I have run cx.execute() for all records, I commit all

Python

3155

Loading dataset from XML

by: kids_pro | last post by:

I had come across a code block from Loading a DataSet from XML (.NET Framework Developer's Guide) ---------------------------------------- NOte If you call ReadXML to load a very large file, you may encounter slow performance. To ensure best performance for ReadXml, on a large file, call the DataTable.BeginLoadData method for each table in the DataSet, then call ReadXml. Finally, call DataTable.EndLoadData for each table in the DataSet as...

C# / C Sharp

2554

System.Web.UI.Page Inheritance

by: dan | last post by:

I have created a class called BasePage that inherits from System.Web.UI.Page so that I can implement a "template" for all pages in an app. My BasePage class is in a class library project and there is also a web app in the same solution that has a project reference back to the class library. When I create a new web form in VS.NET, I simply change the code behind class to inherit from BasePage instead of System.Web.UI.Page.

ASP.NET

1883

extreme memory use when loading in a lot of data

by: Vivek Khera | last post by:

I have some historic data that I want to analyze. To do this I set up postgres on a spare box I picked up for cheap, which just lucked into having tons of RAM (1.5G). I set up postgres to use 10000 buffers, and recompiled the kernel to allow 2Gb data size limit per process. Since this is historical data, I'm actually merging a couple of dumps that span the time range. I've dealt with eliminating any conflicting data (ie, clashing...

PostgreSQL Database

1589

Q about loading large table that is centered

by: Richard | last post by:

I have a table that is about 100 rows long and within that table I have tons of links. The table takes about 2 seconds to load. I think javascript might help somehow. The problem is that in FireFox and Netscape, as this document is written the page is being formated while this table is loading, and because the table is centered horizontally and I have some right-aligned divs above that, the table will appear to the right side of the...

Javascript

1921

Caching (?) problem with System.Web.UI.WebControls.Table ???

by: B. Chernick | last post by:

I am using a System.Web.UI.WebControls.Table control on a screen. (Dot Net 1.1) My problem is this: The table is programmatically reconstructed on every postback. Each table row contains two textboxes that are intially set from the database. None of the items in the table are 'databound'. As of now, if the previous instance of the screen contained data in the first row boxes, that data is carried over, even if the boxes text value...

ASP.NET

6265

Access XP / MySQL / MyODBC: Access' caching system?

by: onnodb | last post by:

Hi all, While working on an Access UI to a MySQL database (which should be a reasonable, low-cost, flexible interface to the DB, better than web-based, much less costly than a full-fledged .NET app or so.... is it?), I tried measuring the bandwith consumed by the Access/MyODBC/MySQL link, which came out to be, er, quite high. I fancied it would be interesting to look at the queries Access throws at MySQL through the ODBC link, so I...

Microsoft Access / VBA

9000

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

9577

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

9396

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...

Online Marketing

9256

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

8260

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

6804

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...

Microsoft Access / VBA

6081

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

3322

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

2225

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General