473,406 Members | 2,451 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,406 software developers and data experts.

Custom File Format & Serialization

Hi there,

I'm developing a shape recognition application for the tablet PC for a)
fun b) university project. Currently I'm working on the learning stage
using neural networks, and have to store a load of learning data (a 25
by 25 matrix) each shape group has a number of user drawn shapes, then
the application will create variations of these shapes (by moving nodes
and drawing lines into the matrix between nodes, after normalization).
So as you can imagine there is allot of data floating around in the
program. I've used XML serialization to save the data to disk, and at
the moment it weighs in at a 7.3 meg file for 3 shape groups, with 4
user drawn shapes each, and each user drawn shape having 48 variations
generated.

I'm wondering if there is a better way to save this information, as I
plan to have many shape groups and as you can imagine the file size of
the learning data will go through the roof. I also want to package some
other data with the learning information (for example a graphic preview
of what the shape group actually is, and may stroke information),
without the file size being too huge. So any ideas here?

Personally I have thought about compression, but this means expanding
into memory which is never a good thing, and also the only compression
library of any note that I can find on the magic google is sharpzlip and
this is GPL and I'm not a fan of GPL.

Thanks in advance
--
Phil Price
CS BSc, University Of Hull
Microsoft Student Partner 2004
w: www.philprice.net
Jul 21 '05 #1
1 1936
Hello Phil,

I think the first thing you can do is start using BinaryFromatter class
from System.Runtime.Serialization namespace. It should produce more compact
files. If the file size is still too large, you can do the following things:
- write your own formatter that involves compression/decompression
algorithms
- Implement ISerializable interface in your classes and override the
standard way of serializing/deserializing objects. Probably you can come up
with more efficient encoding for your classes.

I hope this helps.

--
Victor Urnyshev [MSFT]
This post is "AS IS" with no warranties, and confers no rights.
--------------------
|NNTP-Posting-Date: Tue, 25 May 2004 06:44:27 -0500
|Date: Tue, 25 May 2004 12:44:33 +0100
|From: Phil Price <ph**@philprice.net>
|User-Agent: Mozilla Thunderbird 0.6 (Windows/20040502)
|X-Accept-Language: en-us, en
|MIME-Version: 1.0
|Newsgroups: microsoft.public.dotnet.general
|Subject: Custom File Format & Serialization
|Content-Type: text/plain; charset=us-ascii; format=flowed
|Content-Transfer-Encoding: 7bit
|Message-ID: <ud********************@karoo.co.uk>
|Lines: 32
|NNTP-Posting-Host: 213.249.237.180
|X-Trace:
sv3-wKxWJX0RZgjeT1B2+0MPc8a85CX6uzrF7+4YN5Py+5XFzm5CEg G8cTZ5TuDjU/lN3tJZyx2d
bMr7LIG!RLIyHvzbcTr52H0XsB6kc+Ie7iFHFWBP7UT1JME8q1 f8r9TfvXaRtTa2QDXngL6yoOHs
4EIwhhFP!+Q==
|X-Complaints-To: ab***@karoo.kcom.com
|X-DMCA-Complaints-To: ab***@karoo.kcom.com
|X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
|X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your
complaint properly
|X-Postfilter: 1.1
|Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!newsfeed00.s
ul.t-online.de!t-online.de!border2.nntp.dca.giganews.com!border1.nn tp.dca.gi
ganews.com!nntp.giganews.com!local1.nntp.dca.gigan ews.com!nntp.karoo.co.uk!n
ews.karoo.co.uk.POSTED!not-for-mail
|Xref: cpmsftngxa10.phx.gbl microsoft.public.dotnet.general:135105
|X-Tomcat-NG: microsoft.public.dotnet.general
|
|Hi there,
|
|I'm developing a shape recognition application for the tablet PC for a)
|fun b) university project. Currently I'm working on the learning stage
|using neural networks, and have to store a load of learning data (a 25
|by 25 matrix) each shape group has a number of user drawn shapes, then
|the application will create variations of these shapes (by moving nodes
|and drawing lines into the matrix between nodes, after normalization).
|So as you can imagine there is allot of data floating around in the
|program. I've used XML serialization to save the data to disk, and at
|the moment it weighs in at a 7.3 meg file for 3 shape groups, with 4
|user drawn shapes each, and each user drawn shape having 48 variations
|generated.
|
|I'm wondering if there is a better way to save this information, as I
|plan to have many shape groups and as you can imagine the file size of
|the learning data will go through the roof. I also want to package some
|other data with the learning information (for example a graphic preview
|of what the shape group actually is, and may stroke information),
|without the file size being too huge. So any ideas here?
|
|Personally I have thought about compression, but this means expanding
|into memory which is never a good thing, and also the only compression
|library of any note that I can find on the magic google is sharpzlip and
|this is GPL and I'm not a fan of GPL.
|
|Thanks in advance
|--
|Phil Price
|CS BSc, University Of Hull
|Microsoft Student Partner 2004
|w: www.philprice.net
|

Jul 21 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: psy000 | last post by:
Hi, I have a C# web service client that talks to a JAVA application sever. I use AXIS to generate the WSDL file, use wsdl.exe to generate proxy stub c# code. When I try to use c# client connect...
3
by: Nadav | last post by:
Hi, I have to read files generated by 3rd party application, these files are in a propriotary format ( not generated using the .NET framework ), I want to implement a custom DeSerializer to read...
4
by: Nick Gilbert | last post by:
Hi, I would like the ability to store the configuration settings for all versions of my site in a single web.config file by using different sections. Eg: <siteConfig> <machine name="XENON">...
1
by: Knecke | last post by:
Hi all. I have a problem with returning a custom Result object with webservice. The classes i use is described below (some fields and properties is removed) public class Result { int...
2
by: AMDRIT | last post by:
Hello everyone, I have created a custom component and one of its properties is a class object with it's own properties. During runtime, I can assign values to the class object properties just...
1
by: Phil Price | last post by:
Hi there, I'm developing a shape recognition application for the tablet PC for a) fun b) university project. Currently I'm working on the learning stage using neural networks, and have to store...
7
by: Flashster | last post by:
My program writes text to a file. Currently the files have a suffix of '.log' . I want to make sure that the user does not attempt to open a file that was not originally created by my program. ...
4
by: Val | last post by:
I have a complex object that I need to serialize. Rather than rely on a standard routine, which is called during the serialization/deserialization, I would like to be able to use my own functions...
11
by: =?Utf-8?B?bWljaGFlbCBzb3JlbnM=?= | last post by:
I have worked with application settings in VS2005 and C# for awhile, but usually with standard types. I have been trying to store a custom container/class/type in an application setting and I have...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.