Dealing with very large binary files

sebastian.harko

Helllo,

What's the general accepted strategy for dealing with very large
binary files in C# ?
I have to do a program that reads some "multi frame bitmap " files
which can reach up to one hundred megs so I need to know how to
optimize reading a file..

Best regards,
Seb

Aug 6 '07 #1

Subscribe Reply

2358

Peter Duniho

se************* @gmail.com wrote:

What's the general accepted strategy for dealing with very large
binary files in C# ?

Very large binary files? Read a little bit at a time.

I have to do a program that reads some "multi frame bitmap " files
which can reach up to one hundred megs so I need to know how to
optimize reading a file..

Oh. I thought you were asking about "very large binary files". These
days, 100MB isn't really all that big. :)

That said, you are likely to find that the most important issue is to
make sure you read enough data at one time. You can use a
BufferedStream to ensure this, but my experience has been that even
FileStream has some built-in buffering (caching at the OS level) that
results in pretty good performance anyway.

Do you have a specific issue that is coming up in which performance is
not satisfactory? Generally, it's better to not waste time optimizing
until you know what performance problem you're trying to solve. Until
you have an actual performance problem, you can't answer that.

Pete

Aug 6 '07 #2

Jonathan Wood

It really depends on what you will do with those files. But you might check
out memory-mapped files.

--
Jonathan Wood
SoftCircuits Programming
http://www.softcircuits.com

<se************ *@gmail.comwrot e in message
news:11******** *************@5 7g2000hsv.googl egroups.com...

Helllo,

What's the general accepted strategy for dealing with very large
binary files in C# ?
I have to do a program that reads some "multi frame bitmap " files
which can reach up to one hundred megs so I need to know how to
optimize reading a file..

Best regards,
Seb

Aug 6 '07 #3

Ignacio Machin \( .NET/ C# MVP \)

HI,

<se************ *@gmail.comwrot e in message
news:11******** *************@5 7g2000hsv.googl egroups.com...

Helllo,

What's the general accepted strategy for dealing with very large
binary files in C# ?

Depend of your escenario and of the kind of processing you need to do.

I have to do a program that reads some "multi frame bitmap " files
which can reach up to one hundred megs so I need to know how to
optimize reading a file..

And do what with them? As I said depending of your intended action will be
the best way of handling it

Aug 6 '07 #4

Similar topics

3110

Uploading large files - error "stat failed"

by: Marc | last post by:

Hello, I have a page where i can upload binary file (using the HTML input type=file approach). This works fine for relatively small files (<6MB)) but when files get bigger (13MB) there is a problem. The code of the php file is based upon the article in http://www.onlamp.com/pub/a/php/2000/09/15/php_mysql.html: uploading large binary files using more than one table and more trhan 1 row per file.

PHP

2918

Remoting Question dealing with Microsoft.Samples...SSPI,Security

by: Sean Newton | last post by:

I am absolutely bewildered by now by the Microsoft.Samples SSPI and Security assemblies. I've been trying to set these up in a very straightforward harness in the way that I'd like to be able to use them. No IIS. Use TCP, binary. Standard server example with a console host and console client. .NET 1.1, windows XP. (I tried posting to the remoting newsgroup, no answers in the last couple days, trying here in hopes that more people watch this...

.NET Framework

5857

array iteration?

by: CSN | last post by:

Is it possible to iterate over an array in plpgsql? Something like: function insert_stuff (rel_ids int) .... foreach rel_ids as id insert into table (rel_id, val) values (id, 5);

PostgreSQL Database

2648

Querying Very Large XML

by: Greg | last post by:

I am working on a project that will have about 500,000 records in an XML document. This document will need to be queried with XPath, and records will need to be updated. I was thinking about splitting up the XML into several XML documents (perhaps 50,000 per document) to be more efficient but this will make things a lot more complex because the searching needs to go accross all 500,000 records. Can anyone point me to some best practices...

.NET Framework

2382

Dealing with large files (random access)

by: Patrick | last post by:

Hi, This post is the 'sequel' ;) of the "Data Oriented vs Object Oriented Design" post, but it can be read and treated apart from that one. I will just quote the beginning of my previous message to expose the problem: This post deals with type (2) files.

C# / C Sharp

4626

Transport large binary file from Window client application to Web Server & back

by: gauravkhanna | last post by:

Hi All I need some help for the below problem: Scenario We need to send large binary files (audio file of about 10 MB or so) from the client machine (.Net Windows based application, located outside the home network) to the Web Server and then retrieve the file back from the web server to the client.

.NET Framework

1961

Large XML files

by: jdev8080 | last post by:

We are looking at creating large XML files containing binary data (encoded as base64) and passing them to transformers that will parse and transform the data into different formats. Basically, we have images that have associated metadata and we are trying to develop a unified delivery mechanism. Our XML documents may be as large as 1GB and contain up to 100,000 images. My question is, has anyone done anything like this before?

.NET Framework

9908

safely reading large files

by: byte8bits | last post by:

How does C++ safely open and read very large files? For example, say I have 1GB of physical memory and I open a 4GB file and attempt to read it like so: #include <iostream> #include <fstream> #include <string> using namespace std; int main () {

C / C++

2035

dealing with huge data

by: pereges | last post by:

ok so i have written a program in C where I am dealing with huge data(millions and lots of iterations involved) and for some reason the screen tends to freeze and I get no output every time I execute it. However, I have tried to reduce the amount of data and the program runs fine. What could possibly be done to resolve this ?

C / C++

8348

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

8861

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

8549

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...

Windows Server

7375

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

6187

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...

Microsoft Access / VBA

5660

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

4351

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

2003

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP

1759

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General