473,562 Members | 2,600 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Working with Hastables

Hi all,

I am new to Hashtables, so at the moment, not fully familiar with them.

I have been experimenting with a search engine spider written in c#. It uses
hashtables to hold the catalog.

Now, if I have a large site, or I want to scan many websites, then the
hashtables would get very large. I am looking at writing them to disk and
reading them, though am not sure how this would work.

Now, I have found this on the net (vb code, I can convert, so no need to
worry about that)

Private Sub cmdSave_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles cmdSave.Click
'Save the hashtable
If File.Exists(App lication.Startu pPath & "\data.dat" ) = True Then
File.Delete(App lication.Startu pPath & "\data.dat" )
Dim fs As New FileStream(Appl ication.Startup Path & "\data.dat" ,
FileMode.Create New)
Dim bf As New
Runtime.Seriali zation.Formatte rs.Binary.Binar yFormatter()
bf.Serialize(fs , HashTest)
fs.Close()
End Sub

Private Sub cmdLoad_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles cmdLoad.Click
If File.Exists(App lication.Startu pPath & "\data.dat" ) = False Then
Exit Sub
Dim fs As New FileStream(Appl ication.Startup Path & "\data.dat" ,
FileMode.Open)
Dim bf As New
Runtime.Seriali zation.Formatte rs.Binary.Binar yFormatter()
HashTest = bf.Deserialize( fs)
fs.Close()
cmdIterate_Clic k(Nothing, Nothing)
End Sub

which looks like it might be suitable... but, my question is then,
1. Can I incrementally add to the hashtable file so that I don't run out of
memory when scanning for files to catalog
2. When reading it back off disk, do I have to read the whole lot into
memory in order to search through it?

If I can't do either of these, then what would you suggest?

The way I want to use it is somewhat like a sql database, where I can
quickly select the records I need.

--
Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available
Jan 29 '06 #1
3 2322
Dnia 29-01-2006 o 17:20:07 David
<da************ *****@revilloc. REMOVETHIS.com> napisa³:
Hi all,

I am new to Hashtables, so at the moment, not fully familiar with them.

I have been experimenting with a search engine spider written in c#. It
uses
hashtables to hold the catalog.

Now, if I have a large site, or I want to scan many websites, then the
hashtables would get very large. I am looking at writing them to disk and
reading them, though am not sure how this would work.

Now, I have found this on the net (vb code, I can convert, so no need to
worry about that)

Private Sub cmdSave_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles cmdSave.Click
'Save the hashtable
If File.Exists(App lication.Startu pPath & "\data.dat" ) = True Then
File.Delete(App lication.Startu pPath & "\data.dat" )
Dim fs As New FileStream(Appl ication.Startup Path & "\data.dat" ,
FileMode.Create New)
Dim bf As New
Runtime.Seriali zation.Formatte rs.Binary.Binar yFormatter()
bf.Serialize(fs , HashTest)
fs.Close()
End Sub

Private Sub cmdLoad_Click(B yVal sender As System.Object, ByVal e As
System.EventArg s) Handles cmdLoad.Click
If File.Exists(App lication.Startu pPath & "\data.dat" ) = False
Then
Exit Sub
Dim fs As New FileStream(Appl ication.Startup Path & "\data.dat" ,
FileMode.Open)
Dim bf As New
Runtime.Seriali zation.Formatte rs.Binary.Binar yFormatter()
HashTest = bf.Deserialize( fs)
fs.Close()
cmdIterate_Clic k(Nothing, Nothing)
End Sub

which looks like it might be suitable... but, my question is then,
1. Can I incrementally add to the hashtable file so that I don't run out
of
memory when scanning for files to catalog
2. When reading it back off disk, do I have to read the whole lot into
memory in order to search through it?
[PD] In the solution above I'm afraid the answer for both questions is no.
You would have to serialize and deserialize whole Hashtable.

If I can't do either of these, then what would you suggest?

The way I want to use it is somewhat like a sql database, where I can
quickly select the records I need.

[PD] If you want database functionality why not use database? Some SQL
servers (i.e. Firebird) can be embedded within your application so you
don't need to install them.
--
Piotr Dobrowolski
Piotr.Dobrowols ki@_usun_gmail. com
Jan 29 '06 #2
A hashtable is basically like a database table with a single, primary
key. (Actually, they're more analogous to in-memory indexed files, but
not everyone these days remembers what an indexed file is.)

You can store an object in the hashtable by specifying the key under
which to store it. You can retrieve the object again by giving its key.
I use them all the time: if you read some a table full of business
information from a database, often a Hashtable is a natural fit.

Take a table full of invoices, for example. Each invoice usually has a
unique invoice number. Put the invoices in a Hashtable, keyed by
invoice number. Later, when you get a foreign key in another table that
refers to an invoice number, just use that number to look up the
invoice object in your in-memory hash table.

In your case, why not store the information in a database? I believe
that Microsoft's new SQL Express is out (others will correct me if I'm
wrong about that). It's a free mini-database that will run on any
laptop / desktop / notepad machine. That way your app is scalable.

I wouldn't bother using a flat-file serialization like the one you
posted here. It may be fine for a quick test hack, but not for a real
application.

Jan 30 '06 #3
Thank you both Bruce and Piotr,

I thought what I was looking at would not be the best solution. I will look
at writing it to a database.

It is for a search engine and spider that uses Hashtables to store the
catalog. I don't like the idea of storing it this way, but need a quick
access.

I am looking at alternative search engine code as well, so I will start
another thread regarding that.

Best regards,
Dave Colliver.
http://www.AshfieldFOCUS.com
~~
http://www.FOCUSPortals.com - Local franchises available
"Bruce Wood" <br*******@cana da.com> wrote in message
news:11******** **************@ o13g2000cwo.goo glegroups.com.. .
A hashtable is basically like a database table with a single, primary
key. (Actually, they're more analogous to in-memory indexed files, but
not everyone these days remembers what an indexed file is.)

You can store an object in the hashtable by specifying the key under
which to store it. You can retrieve the object again by giving its key.
I use them all the time: if you read some a table full of business
information from a database, often a Hashtable is a natural fit.

Take a table full of invoices, for example. Each invoice usually has a
unique invoice number. Put the invoices in a Hashtable, keyed by
invoice number. Later, when you get a foreign key in another table that
refers to an invoice number, just use that number to look up the
invoice object in your in-memory hash table.

In your case, why not store the information in a database? I believe
that Microsoft's new SQL Express is out (others will correct me if I'm
wrong about that). It's a free mini-database that will run on any
laptop / desktop / notepad machine. That way your app is scalable.

I wouldn't bother using a flat-file serialization like the one you
posted here. It may be fine for a quick test hack, but not for a real
application.

Jan 30 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
6247
by: Gary | last post by:
I am trying to use the "System.Windows.Forms.SendKeys" class for triggering the Ctrl+P key. Syntax: System.Windows.Forms.SendKeys.Send("^(P)"); This is not working ..what could be the reason..? Situation: I have a menu in my form, which is given a short key "Ctrl+P", now when the
6
3053
by: Mullin Yu | last post by:
hi, i have a web service that has file operations on Windows OS, and there may be a file concurrency issue if only one working directory e.g. c:\working therefore, i want to have a unique sub directory for each request so as to get rid of any file concurrency issue, e.g c:\working\unique-identifier1 for Request #1...
3
3099
by: | last post by:
Hello, I am hoping someone else has thought about a date time calculation i need to perform. I would like to be able to calculate the number of "working minutes" between 2 dates, given my working week definition. Lets say I have a working week definition of Monday through Friday, 9 am to 5 pm. Date1 = January 1st, 2005 at 8 am
8
1822
by: Hardy Wang | last post by:
Hi: Is it possible for me to create/open web application from remote machine other than port 80? And create application directly under virtual web site instead of creating a virtual directory? Because there is a developing server in our company running IIS with multiple web sites, each one has its own port number. In real scenario, no...
4
1619
by: GP | last post by:
We are facing a strange problems everything works fine when we are using the visual studio environment .But when we deploy it in the webserver the save buttons/submit buttons which has the cause validation property to true is not working ..meaning the other aspx pages which doesn't have the page validator is working fine.Why does this problem...
5
2798
by: Martin Heuckeroth | last post by:
Hi We are working on a webservice application and are having some problems with the cookies and/or sessions. We have them working on our intranet but then its not working on the internet. We have been looking our asses off but can not find a working solution. the code we came up with is ...
5
3046
by: tshad | last post by:
I have been working with setting my drop boxes to allow double clicking to select an item. It worked fine until I made some changes. I then stripped the page down to the bare essentials to find out why it quit working. I found that if I didn't have a linkbutton, it quit working????????? If I changed the linkbutton to a regular button,...
8
2503
by: jojobar | last post by:
Okay, I am trying to do is to test the webresource in 2.0 1. I created a new project with assembly name (and default assembly name) "Office". 2. I added the following to the AssemblyInfo.cs file (present under the Properties folder of the project) 3. Now I created a file called test.js on the root folder of the project,
2
14814
by: Don | last post by:
I'm having problems with intellisense, autocomplete, etc. suddenly not working in certain classes of a project I'm working on. All the options are set, and it all works fine for most classes, but for some it just suddenly stopped working. After a build it'll work maybe once or twice, then stop. I've tried repairing my installation of Visual...
9
3195
by: MSDNAndi | last post by:
Hi, I have a set of simple webservices calls that worked fine using .NET Framework 1.0. I am calling a Java/Apache based webservices, the calling side is not able to supply a proper WSDL. What it does is to call a webservice with two parameters, one being a integer, the other one being a "String" which contains XML (not the best...
0
7874
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
8101
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
7940
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6228
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5198
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3626
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3609
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2075
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
907
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.