file access - design thoughts.

[J.Broeden]

Hi, I'm hoping to get some advice on this design query to see if I'm
heading in the right direction.

I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

Jay

[Ian Collins]

J.*******@gmail.com wrote:

Hi, I'm hoping to get some advice on this design query to see if I'm
heading in the right direction.

I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

If portability isn't an issue, you will probably be better of using your
operating system's means of memory mapping the file and simply using the
result as an array.

--
Ian Collins.

[Malcolm McLean]

<J.*******@gmail.comwrote in message
news:11*********************@w5g2000hsg.googlegrou ps.com...

Hi, I'm hoping to get some advice on this design query to see if I'm
heading in the right direction.

I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

Google for ggets and Chuck Falconer.
Chuck, a reg in this group, has kindly provided exactly what you want. I
believe entirely rights free, certainly for nothing.

[pete]

J.*******@gmail.com wrote:

>
Hi, I'm hoping to get some advice on this design query to see if I'm
heading in the right direction.

I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

get_line uses getc to read lines, and realloc to size the array.
Truncates lines at (size_t)-1) bytes.

http://www.mindspring.com/~pfilandr/...ine/get_line.c

--
pete

[Tor Rustad]

J.*******@gmail.com wrote:

Hi, I'm hoping to get some advice on this design query to see if I'm
heading in the right direction.

I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Your problem, is well known for compiler writers, they call it lexical
analysis. fgets() is not normally used in hand-written lexer's, but
getchar() and ungetc() is, because you analyze single character at a time.

Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

Yes, there is standard tool for this, called lex. Here is a lex
specification that print all the words in a file:

$ cat ex_lex.l
/*
Compile:
flex ex_lex.l
cc lex.yy.c -ll

Run:
../a.out < ex_lex.l
*/

%{
%}
%%

[^ \n\t]+ { printf("Word: '%s', len = %d\n", yytext, yyleng); }
..|\n ; /* ignore anything else */

%%

int main(void)
{
yylex();
}

--
Tor <torust [at] online [dot] no>

[CBFalconer]

Malcolm McLean wrote:

<J.*******@gmail.comwrote in message

>Hi, I'm hoping to get some advice on this design query to see if I'm
heading in the right direction.

I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

Google for ggets and Chuck Falconer.
Chuck, a reg in this group, has kindly provided exactly what you want.
I believe entirely rights free, certainly for nothing.

try: <http://cbfalconer.home.att.net/download/>
--
<http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.txt>
<http://www.securityfocus.com/columnists/423>
<http://www.aaxnet.com/editor/edit043.html>
<http://kadaitcha.cx/vista/dogsbreakfast/index.html>
cbfalconer at maineline dot net
--
Posted via a free Usenet account from http://www.teranews.com

[CBFalconer]

Tor Rustad wrote:

J.*******@gmail.com wrote:

.... SNIP ...

>>
I would like to read a file of many lines and then search for a
particular character to break up the sentence into separate words.
Once done I will sort them.

I thought of using fgets but the line could be very very long and I
won't know it's length of n characters that could be in a line.

Your problem, is well known for compiler writers, they call it lexical
analysis. fgets() is not normally used in hand-written lexer's, but
getchar() and ungetc() is, because you analyze single character at a time.

>Using fgetc seems the better bet and put each character into an array
so I can traverse the array.

However,if I put each character into an array, I still won't know how
large to make the array.

Is there a better strategy ?

Yes, there is standard tool for this, called lex. Here is a lex
specification that print all the words in a file:

$ cat ex_lex.l
/*
Compile:
flex ex_lex.l
cc lex.yy.c -ll

Run:
./a.out < ex_lex.l
*/

%{
%}
%%

[^ \n\t]+ { printf("Word: '%s', len = %d\n", yytext, yyleng); }
.|\n ; /* ignore anything else */

%%

int main(void)
{
yylex();
}

Why all that tortuous mess to copy text?.

#include <stdio.h>

int main(void) {
int ch;

while (EOF != (ch = getchar())) putchar(ch);
return 0;
}

--
<http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.txt>
<http://www.securityfocus.com/columnists/423>
<http://www.aaxnet.com/editor/edit043.html>
<http://kadaitcha.cx/vista/dogsbreakfast/index.html>
cbfalconer at maineline dot net

--
Posted via a free Usenet account from http://www.teranews.com

[Tor Rustad]

CBFalconer wrote:

[...]

Why all that tortuous mess to copy text?.

Hand-written lexer in C is easy, but writing a lex spec is easier.

Try again, this time print each *word* and it's length on a separate line on
the output. Please follow your own bogus advice, by using ggets().

--
Tor <torust [at] online [dot] no>

[Malcolm McLean]

"Tor Rustad" <to****@online.nowrote in message
news:ir*********************@telenor.com...

CBFalconer wrote:

[...]

>Why all that tortuous mess to copy text?.

Hand-written lexer in C is easy, but writing a lex spec is easier.

Try again, this time print each *word* and it's length on a separate line
on
the output. Please follow your own bogus advice, by using ggets().

Why is ggets() a problem for this task?
Granted les / flex is a dedicated lexical analyser and in its element with
token generation, but most people don't have the inclination to learn yet
another language for such a simple spec.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Tor Rustad]

Malcolm McLean wrote:

>
"Tor Rustad" <to****@online.nowrote in message
news:ir*********************@telenor.com...

>CBFalconer wrote:

[...]

>>Why all that tortuous mess to copy text?.

Hand-written lexer in C is easy, but writing a lex spec is easier.

Try again, this time print each *word* and it's length on a separate line
on the output. Please follow your own bogus advice, by using ggets().

Why is ggets() a problem for this task?

OP needed to split the file content into words, not lines! Hence, this is
what he wanted in C:

int lexer(char *word, size_t *wlen)
{
int t;

while (1)
{
t = getchar();

/* Eat white space */
if (t == ' ' || t == '\t' || t == '\n')
;
else if (t == EOF)
return EOF;

etc...
}
}

adding logic to handle lines, just complicate matters. Hence, using fgets(),
ggets(), getline() etc. was all bad advice.

Granted les / flex is a dedicated lexical analyser and in its element with
token generation, but most people don't have the inclination to learn yet
another language for such a simple spec.

OP asked if there was a better strategy, well there is, he can ignore the
line issue and look into using a dedicated tool for such problems.

Since I had already done the complete lex skeleton, all he needed to do, was
really to add that C sorting code. That hardly qualify as learning a new
language. <g>

--
Tor <torust [at] online [dot] no>

[Malcolm McLean]

"Tor Rustad" <to****@online.nowrote in message
news:Pp*********************@telenor.com...

Malcolm McLean wrote:

>>
"Tor Rustad" <to****@online.nowrote in message
news:ir*********************@telenor.com...

>>CBFalconer wrote:

[...]

Why all that tortuous mess to copy text?.
Hand-written lexer in C is easy, but writing a lex spec is easier.

Try again, this time print each *word* and it's length on a separate
line
on the output. Please follow your own bogus advice, by using ggets().

Why is ggets() a problem for this task?

OP needed to split the file content into words, not lines! Hence, this is
what he wanted in C:

int lexer(char *word, size_t *wlen)
{
int t;

while (1)
{
t = getchar();

/* Eat white space */
if (t == ' ' || t == '\t' || t == '\n')
;
else if (t == EOF)
return EOF;

etc...
}
}

adding logic to handle lines, just complicate matters. Hence, using
fgets(),
ggets(), getline() etc. was all bad advice.

>Granted les / flex is a dedicated lexical analyser and in its element
with
token generation, but most people don't have the inclination to learn yet
another language for such a simple spec.

OP asked if there was a better strategy, well there is, he can ignore the
line issue and look into using a dedicated tool for such problems.

Since I had already done the complete lex skeleton, all he needed to do,
was
really to add that C sorting code. That hardly qualify as learning a new
language. <g>

You are right in a sense. You can produce very compact programs that do
manipulations on streams.
However you are limited to fairly simple manipulations before the strategy
begins to break down, and you tie IO very closely to logic, which is
undesireable for all sorts of reasons. Functions (in the mathematical sense)
should be separated from the procedures - a procedure being something which
does things.
--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Tor Rustad]

Malcolm McLean wrote:

[...]

You are right in a sense. You can produce very compact programs that do
manipulations on streams.
However you are limited to fairly simple manipulations before the strategy
begins to break down, and you tie IO very closely to logic, which is
undesireable for all sorts of reasons.

Like I have already pointed out, regular expressions (via lex) is a *very*
useful abstraction when doing lexical analysis. For the problem at hand,
all OP needed, was to get next character input until EOF, he didn't even
need a lookahead.

So calling getchar() or fgetc(), just say "Hey, give me that next
character". Simple and guess what, it does not break down...

Functions (in the mathematical sense) should be separated from the
procedures - a procedure being something which does things.

???

--
Tor <torust [at] online [dot] no>

[Malcolm McLean]

"Tor Rustad" <to****@online.nowrote in message
news:G_*********************@telenor.com...

Malcolm McLean wrote:

[...]

>You are right in a sense. You can produce very compact programs that do
manipulations on streams.
However you are limited to fairly simple manipulations before the
strategy
begins to break down, and you tie IO very closely to logic, which is
undesireable for all sorts of reasons.

Like I have already pointed out, regular expressions (via lex) is a *very*
useful abstraction when doing lexical analysis. For the problem at hand,
all OP needed, was to get next character input until EOF, he didn't even
need a lookahead.

So calling getchar() or fgetc(), just say "Hey, give me that next
character". Simple and guess what, it does not break down...

But the stream can return an error state, because it does IO, and the
program cannot control things outside of itself. Which has important
implications.

>

>Functions (in the mathematical sense) should be separated from the
procedures - a procedure being something which does things.

???

Think "pure functions" and "model-view architecture".
--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Flash Gordon]

Malcolm McLean wrote, On 25/05/07 05:39:

>
"Tor Rustad" <to****@online.nowrote in message
news:G_*********************@telenor.com...

>Malcolm McLean wrote:

[...]

>>You are right in a sense. You can produce very compact programs that do
manipulations on streams.
However you are limited to fairly simple manipulations before the
strategy
begins to break down, and you tie IO very closely to logic, which is
undesireable for all sorts of reasons.

Like I have already pointed out, regular expressions (via lex) is a
*very*
useful abstraction when doing lexical analysis. For the problem at hand,
all OP needed, was to get next character input until EOF, he didn't even
need a lookahead.

So calling getchar() or fgetc(), just say "Hey, give me that next
character". Simple and guess what, it does not break down...

But the stream can return an error state, because it does IO, and the
program cannot control things outside of itself. Which has important
implications.

So?

>>Functions (in the mathematical sense) should be separated from the
procedures - a procedure being something which does things.

???

Think "pure functions" and "model-view architecture".

Think "there is more than one approach to SW design".
--
Flash Gordon

[Tor Rustad]

Malcolm McLean wrote:

"Tor Rustad" <to****@online.nowrote in message

>>
So calling getchar() or fgetc(), just say "Hey, give me that next
character". Simple and guess what, it does not break down...

But the stream can return an error state, because it does IO, and the
program cannot control things outside of itself. Which has important
implications.

In a lex scanner, you don't need to operate on stdin/stdout, it's just the
default streams yyin and yyout use, which can easily be changed. The caller
is also supposed to provide a yywrap() function, which is a callback on
EOF.

If there is an I/O error, this can of course be handled elsewhere. Why not
read a lex/flex tutorial first, before wasting more time here?

>>Functions (in the mathematical sense) should be separated from the
procedures - a procedure being something which does things.

???

Think "pure functions" and "model-view architecture".

FYI, there is no difference between functions and procedures in C, and
mathematical functions is something that can very much do things too:

y = f(x)

--
Tor <torust [at] online [dot] no>

[Malcolm McLean]

"Tor Rustad" <to****@online.nowrote in message
news:Uq*********************@telenor.com...

Malcolm McLean wrote:

>"Tor Rustad" <to****@online.nowrote in message

>>>
So calling getchar() or fgetc(), just say "Hey, give me that next
character". Simple and guess what, it does not break down...

But the stream can return an error state, because it does IO, and the
program cannot control things outside of itself. Which has important
implications.

In a lex scanner, you don't need to operate on stdin/stdout, it's just the
default streams yyin and yyout use, which can easily be changed. The
caller
is also supposed to provide a yywrap() function, which is a callback on
EOF.

If there is an I/O error, this can of course be handled elsewhere. Why not
read a lex/flex tutorial first, before wasting more time here?

>>>Functions (in the mathematical sense) should be separated from the
procedures - a procedure being something which does things.

???

Think "pure functions" and "model-view architecture".

FYI, there is no difference between functions and procedures in C, and
mathematical functions is something that can very much do things too:

y = f(x)

In C it is conventional to use the word "function" for "subroutine". exit()
is called a "function", but it isn't in the more generally accepted usage.
Language is like that, words shift in meaning and people aren't entirely
logical.

In C you can say that a procedure is a void function. This is useful to
compiler writers, because
"x = function()" is legal whilst
"x= procedure()" is not.
However for our purposes the distinction between

int foo();
and
void bar(int *ret)

doesn't really matter. It is just a case of how you lay out the code when
calling what is effectively the same routine.

A function in mathematics is a mapping of inputs to outputs. It follows that
a mathematical function can't do things.

As Flash pointed out, there is more than one approach to software
engineering. Separating IO from logic is one of the basic themes that crops
up time after time, though it is given different names.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Flash Gordon]

Malcolm McLean wrote:

>
"Tor Rustad" <to****@online.nowrote in message
news:Uq*********************@telenor.com...

>Malcolm McLean wrote:

>>"Tor Rustad" <to****@online.nowrote in message

So calling getchar() or fgetc(), just say "Hey, give me that next
character". Simple and guess what, it does not break down...

But the stream can return an error state, because it does IO, and the
program cannot control things outside of itself. Which has important
implications.

In a lex scanner, you don't need to operate on stdin/stdout, it's just
the
default streams yyin and yyout use, which can easily be changed. The
caller
is also supposed to provide a yywrap() function, which is a callback on
EOF.

If there is an I/O error, this can of course be handled elsewhere. Why
not
read a lex/flex tutorial first, before wasting more time here?

<snip>

As Flash pointed out, there is more than one approach to software
engineering. Separating IO from logic is one of the basic themes that
crops up time after time, though it is given different names.

Using Lex you *are* separating the IO from the processing. Lex is only
used to define processing.

In any case, another design theme that comes up time and again which is
keep it simple. A simple loop grabbing data a bit at a time from the
input and processing it as you grab it *is* keeping it simple. Doing
this I have before now simplified some code so much that it was then
provably fast enough to put straight on the interrupt service routing
thus allowing things to be simplified further because there was not need
to synchronise between the interrupt routine and a background task thus
speeding it up thus...

Moving the processing to the IO *greatly* simplified the entire design
and implementation. It also breached other "design rules" in order to
simplify it greatly.

There is more than one approach and separating the IO from the logic is
not always the correct approach.
--
Flash Gordon

[Malcolm McLean]

"Flash Gordon" <sp**@flash-gordon.me.ukwrote in message

In any case, another design theme that comes up time and again which is
keep it simple.
Moving the processing to the IO *greatly* simplified the entire design and
implementation. It also breached other "design rules" in order to simplify
it greatly.

There is more than one approach and separating the IO from the logic is
not always the correct approach.

I agree with you. Often there are two contradictory rules, neither of which
you want to break, but dogmatically insisting on one over the other is a
silly way of proceeding.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Flash Gordon]

Malcolm McLean wrote:

"Flash Gordon" <sp**@flash-gordon.me.ukwrote in message

>In any case, another design theme that comes up time and again which
is keep it simple.
Moving the processing to the IO *greatly* simplified the entire design
and implementation. It also breached other "design rules" in order to
simplify it greatly.

There is more than one approach and separating the IO from the logic
is not always the correct approach.

I agree with you. Often there are two contradictory rules, neither of
which you want to break, but dogmatically insisting on one over the
other is a silly way of proceeding.

So why were you criticising a nice simple solution for breaking a rule
by not separating the the IO? Tom's solution was nice and simple.
--
Flash Gordon

[Malcolm McLean]

"Flash Gordon" <sp**@flash-gordon.me.ukwrote in message
news:cf************@news.flash-gordon.me.uk...

Malcolm McLean wrote:

>"Flash Gordon" <sp**@flash-gordon.me.ukwrote in message

>>In any case, another design theme that comes up time and again which is
keep it simple.
Moving the processing to the IO *greatly* simplified the entire design
and implementation. It also breached other "design rules" in order to
simplify it greatly.

There is more than one approach and separating the IO from the logic is
not always the correct approach.

I agree with you. Often there are two contradictory rules, neither of
which you want to break, but dogmatically insisting on one over the other
is a silly way of proceeding.

So why were you criticising a nice simple solution for breaking a rule by
not separating the the IO? Tom's solution was nice and simple.

Tom:

>>adding logic to handle lines, just complicate matters. Hence, using
fgets(),
ggets(), getline() etc. was all bad advice.

I was pointing out that it isn't as straightforwards as that. Yes you
complify things, and break the rule KISS, by using ggets. On the other hand
you couple IO very tightly to logic, breaking the prodecures must be
separate from functions rule.

However you managed to persuade me that I wasn't entirely right after all
insisting on IO separate from logic. That's one of the characteristics of
intelligent people. They modify their opinions after listening to other
intelligent people.
--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[Flash Gordon]

Malcolm McLean wrote:

<snip>

However you managed to persuade me that I wasn't entirely right after
all insisting on IO separate from logic. That's one of the
characteristics of intelligent people. They modify their opinions after
listening to other intelligent people.

Indeed it is.
--
Flash Gordon

[Keith Thompson]

Flash Gordon <sp**@flash-gordon.me.ukwrites:

Malcolm McLean wrote:

<snip>

>However you managed to persuade me that I wasn't entirely right
after all insisting on IO separate from logic. That's one of the
characteristics of intelligent people. They modify their opinions
after listening to other intelligent people.

Indeed it is.

Well, I used to think so, but ...

8-)}

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Tor Rustad]

Keith Thompson wrote:

Flash Gordon <sp**@flash-gordon.me.ukwrites:

>Malcolm McLean wrote:

<snip>

>>However you managed to persuade me that I wasn't entirely right
after all insisting on IO separate from logic. That's one of the
characteristics of intelligent people. They modify their opinions
after listening to other intelligent people.

Indeed it is.

Well, I used to think so, but ...

Hmm.. since I didn't persuade Malcolm, it follows that one of us isn't
intelligent! :-/

Who is Tom? :-P

--
Tor <torust [at] online [dot] no>