473,320 Members | 1,857 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Adjusting the 1024 byte stdin buffer limit

Are there runtime settings that can be used to adjust the default 1024
byte stdin buffer limit or a buildtime setting in pyconfig.h? I have a
need to pump this up to permit input of a large data block via stdin.
Tim Jones

Jul 18 '05 #1
6 6704
"brucoder" <ti**@tolisgroup.com> wrote in message
news:11*********************@z14g2000cwz.googlegro ups.com...
Are there runtime settings that can be used to adjust the default 1024
byte stdin buffer limit or a buildtime setting in pyconfig.h? I have a
need to pump this up to permit input of a large data block via stdin.


What do you think you are being prevented from doing?
Jul 18 '05 #2
Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

Tim

Jul 18 '05 #3
Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

Tim

Jul 18 '05 #4
brucoder wrote:
Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

But surely the blocking will only last as long as the consuming process
leaves buffered input unread?

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
Jul 18 '05 #5
"brucoder" <ti**@tolisgroup.com> wrote:
Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.


you can pass in a buffer size when you open a file:
help(open)
class file(object)
| file(name[, mode[, buffering]]) -> file object
|
| Open a file. The mode can be 'r', 'w' or 'a' for reading (default),
| writing or appending. The file will be created if it doesn't exist
| when opened for writing or appending; it will be truncated when
| opened for writing. Add a 'b' to the mode for binary files.
| Add a '+' to the mode to allow simultaneous reading and writing.
| If the buffering argument is given, 0 means unbuffered, 1 means line
| buffered, and larger numbers specify the buffer size.
| Add a 'U' to mode to open the file for input with universal newline
| support. Any line ending in the input file will be seen as a '\n'
| in Python. Also, a file so opened gains the attribute 'newlines';
| the value for this attribute is one of None (no newline read yet),
| '\r', '\n', '\r\n' or a tuple containing all the newline types seen.
|
| 'U' cannot be combined with 'w' or '+' mode.
|
| Note: open() is an alias for file().

or use os.fdopen() to reopen an existing file handle:
help(os.fdopen)


fdopen(fd [, mode='r' [, bufsize]]) -> file_object

Return an open file object connected to a file descriptor.

assuming "sending via stdin" means using a pipe, this page explains why all
this probably won't matter:

http://www.opengroup.org/onlinepubs/...xsh/write.html

(see the "Write requests to a pipe or FIFO" section)

</F>

Jul 18 '05 #6
Hi All,
-
Thanks, but it seems I may have been a bit more confused that even I
thought. It turns out to be an issue with Apple's OS X pty/tty
interface. Even though the mediator tool is set up to read in up to
20K of data, the pty interface blocks at 1024 bytes. When testing this
under Linux, I'm fine. And, Solaris 8 weaves an even more diabolical
255 character limit :( .
-
Looks like I'll need to be a bit more creative than simply spawning a
shell.
-
Tim
Fredrik Lundh wrote:
"brucoder" <ti**@tolisgroup.com> wrote:
Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.
you can pass in a buffer size when you open a file:
>>> help(open)
class file(object)
| file(name[, mode[, buffering]]) -> file object
|
| Open a file. The mode can be 'r', 'w' or 'a' for reading

(default), | writing or appending. The file will be created if it doesn't exist | when opened for writing or appending; it will be truncated when | opened for writing. Add a 'b' to the mode for binary files.
| Add a '+' to the mode to allow simultaneous reading and writing. | If the buffering argument is given, 0 means unbuffered, 1 means line | buffered, and larger numbers specify the buffer size.
| Add a 'U' to mode to open the file for input with universal newline | support. Any line ending in the input file will be seen as a '\n' | in Python. Also, a file so opened gains the attribute 'newlines'; | the value for this attribute is one of None (no newline read yet), | '\r', '\n', '\r\n' or a tuple containing all the newline types seen. |
| 'U' cannot be combined with 'w' or '+' mode.
|
| Note: open() is an alias for file().

or use os.fdopen() to reopen an existing file handle:
>>> help(os.fdopen)

fdopen(fd [, mode='r' [, bufsize]]) -> file_object

Return an open file object connected to a file descriptor.

assuming "sending via stdin" means using a pipe, this page explains

why all this probably won't matter:

http://www.opengroup.org/onlinepubs/...xsh/write.html

(see the "Write requests to a pipe or FIFO" section)

</F>


Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Andreas Muck | last post by:
Hi! We have an application running on Linux (SuSE 7.2, kernel 2.4.16) that opens lots of connections to a Postgres database and occasionaly dies with segfault. Trying to reproduce the crash, I...
0
by: Andreas Muck | last post by:
Hi! We have an application running on Linux (SuSE 7.2, kernel 2.4.16) that opens lots of connections to a Postgres database and occasionaly dies with segfault. Trying to reproduce the crash, I...
23
by: herrcho | last post by:
What's the difference between STDIN and Keyboard buffer ? when i get char through scanf, i type in some characters and press enter, then, where do the characters go ? to STDIN or Keyboard...
5
by: siliconwafer | last post by:
Hi All, I want to take a string from stdin(say login Name).But I don't want a static array of fixed size.I don't want to restrict user to a perticular sizeof login name. So I want to allocate...
2
by: Ron | last post by:
Hello, I am trying to read a list of files from an FTP server (mainframe) to a byte array using sockets as follows, but not getting all the files in a given directory: private readonly static...
2
by: Brian Mitchell | last post by:
Hello, I am using a MemoryStream to serialize a class but when I use the GetBuffer method it returns exactly 1024 bytes even though the stream is only 675 bytes. If I look at the...
2
by: wizofaus | last post by:
Hi, If I write the following unmanaged C program: main() { char buffer; gets(buffer); }
1
by: Mark McDuff | last post by:
I'm trying to read one byte from stdin, without the newline. If I try something like: I can input a character, but then I have to press enter, which leaves a newline character in the stdin...
0
by: Fredrik Lundh | last post by:
Mark McDuff wrote: in addition to the cookbook article Diez posted, there's also a FAQ entry about this: http://effbot.org/pyfaq/how-do-i-get-a-single-keypress-at-a-time.htm </F>
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.