473,322 Members | 1,480 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,322 software developers and data experts.

Python advocacy in scientific computation

Someone asked me to write a brief essay regarding the value-add
proposition for Python in the Fortran community. Slightly modified to
remove a few climatology-related specifics, here it is.

I would welcome comments and corrections, and would be happy to
contribute some version of this to the Python website if it is of
interest.

===

The established use of Fortran in continuum models such as climate
models has some benefits, including very high performance and
flexibility in dealing with regular arrays, backward compatibility with
the existing code base, and the familiarity with the language among the
modeling community. Fortran 90 and later versions have taken many of
the lessons of object oriented programming and adapted them so that
logical separation of modules is supported, allowing for more effective
development of large systems. However, there are many purposes to which
Fortran is ill-suited which are increasingly part of the modeling
environment.

These include: source and version control and audit trails for runs,
build system management, test specification, deployment testing (across
multiple platforms), post-processing analysis, run-time and
asynchronous visualization, distributed control and ensemble
management. To achieve these goals, a combination of shell scripts,
specialized build tools, specialized applications written in several
object-oriented languages, and various web and network deployment
strategies have been deployed in an ad hoc manner. Not only has much
duplication of effort occurred, a great deal of struggling up the
learning curves of various technologies has been required as one need
or another has been addressed in various ad hoc ways.

A new need arises as the ambitions of physical modeling increase; this
is the rapid prototyping and testing of new model components. As the
number of possible configurations of a model increases, the expense and
difficulty of both unit testing and integration testing becomes more
demanding.

Fortunately, there is Python. Python is a very flexible language that
has captured the enthusiasm of commercial and scientific programmers
alike. The perception of Python programmers coming from almost any
other language is that they are suddenly dramatically several times
more productive than previously, in terms of functionality delivered
per unit of programmer time.

One slogan of the Python community is that the language "fits your
brain". Why this might be the case is an interesting question. There
are no startling computer science breakthroughs original to the
language, Rather, Python afficionados will claim that the language
combines the best features of such various languages as Lisp, Perl,
Java, and Matlab. Eschewing allegiance to a specific theory of how to
program, Python's design instead offers the best practices from many
other software cultures.

The synergies among these programming modes is in some ways harder to
explain than to experience. The Python novice may nevertheless observe
that a single language can take the place of shell scripts, makefiles,
desktop computation environments, compiled languages to build GUIs, and
scripting languages to build web interfaces. In addition, Python is
useful as a wrapper for Fortran modules, facilitating the
implementation of true test-driven design processes in Fortran models.

Another Python advocacy slogan is "batteries included". The point here
is that (in part because Python is dramatically easier to write than
other languages) there is a very broad range of very powerful standard
libraries that make many tasks which are difficult in other languages
astonishingly easy in Python. For instance, drawing upon the standard
libraries (no additional download required) a portable webserver
(runnable on both Microsoft and Unix-based platforms) can be
implemented in seven lines of code. (See
http://effbot.org/librarybook/simplehttpserver.htm ) Installation of
pure python packages is also very easy, and installation of mixed
language products with a Python component is generally not
significantly harder than a comparable product with no Python
component.

Among the Python components and Python bindings of special interest to
scientists are the elegant and powerful matplotlib plotting package,
which began by emulating and now surpasses the plotting features of
Matlab, SWIG, which allows for runtime interoperability with various
languages, f2py which specifically interoperates with Fortran, NetCDF
libraries (which cope with NetCDF files with dramatically less fuss
than the standard C or Fortran bindings), statistics packages including
bindings to the R language, linear algebra packages, various
platform-specific and portable GUI libraries, genetic algorithms,
optimization libraries, and bindings for high performance differential
equation solvers (notably, using the Argonne National Laboratory
package PetSC). An especially interesting Python trick for runtime
visualization in models that were not designed to support it, pioneered
by David Beazley's SWILL, embeds a web server in your model code.

See especially http://starship.python.net/~hinsen/ScientificPython/ and
http://scipy.org as good starting points to learn about scientific uses
of Python.

mt

Feb 14 '06
53 4259
Andy Salnikov wrote:
Actually os.system() is rather poor replacement for the shell's
capabilities, and it's _very_ low level, it's really a C-level code
wrapped in Python syntax.


Since os.system() spawns a shell to execute the command,
it's theoretically capable of anything that the shell
can do. It's somewhat inelegant having to concatenate
all the arguments into a string, though.

I gather there's a new subprocess management module
coming that's designed to clean up the mess surrounding
all the popen() variants. Hopefully it will make this
sort of thing a lot easier.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Mar 9 '06 #51
Peter Maas wrote:
This is hard to understand for an outsider. If you pass an int, a float,
a string or any other "atomic" object to a function you have "pass by
value" semantics. If you put a compound object like a list or a dictionary
or any other object that acts as an editable data container you can return
modified *contents* (list elements etc.) to the caller, exactly like in
Java and different from C/C++.


There's really no difference here -- when you pass an
int, you're passing a pointer to an int object, just
the same as when you pass a list, you're passing a
pointer to a list object. It's just that Python
doesn't provide any operations for changing the
contents of an int object, so it's hard to see
the difference.

The similarity is brought out by the following
example:
def a(x): .... x = 42
.... def b(x): .... x = [42]
.... y = 3
a(y)
print y 3 y = [3]
b(y)
print y

[3]

What this shows is that assignment to the parameter
*name* never affects anything outside the function,
regardless of whether the object passed in is mutable
or immutable.

It's best to avoid using terms like "by reference" when
talking about Python parameter passing, because it's
hard to tell whether the person you're talking to
understands the same thing by them. But if you
insist, the correct description in Algol terms is
that Python passes pointers to objects by value.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Mar 9 '06 #52
Robert Kern <ro*********@gmail.com> writes:
sturlamolden wrote: ....
5. Versioning control? For each program there is only one
developer and a single or a handful users.

.... This is one thing that a lot of people seem to get wrong: version
control is not a burden on software development. It is a great
enabler of software development. It helps you get your work done
faster and easier even if you are a development team of one. You can
delete code (intentionally!) because it's not longer used in your
code, but you won't lose it. You can always look at your history and
get it again. You can make sweeping changes to your code, and if
that experiment fails, you can go back to what was working
before. Now you can do this by making copies of your code, but
that's annoying, clumsy, and more effort than it's worth. Version
control makes the process easier and lets you do more interesting
things.

I would go so far as to say that version control enables the
application of the scientific method to software development. When
you are in lab, do you say to yourself, "Nah, I won't write anything
in my lab notebook. If the experiment works at the end of the day,
only that result matters"?

....
A slightly off topic note:

I find that version control (VC) has many advantages for
scientific research (I am a physicist).

1) For software as Robert mentions I find it indispensable.
2) Keeping track of changes to papers (as long as they are plain text
like LaTeX). This is especially useful for collaborations: using
the diff tools one can immediately see any changes a coauthor may
have made.

(I even use branching: maintaining one branch for the journal
submission which typically has space restrictions, and another for
preprint archives which may contain more information.)
3) Using VC allows you to easily bring another computer up to date
with your current work. If I go to a long workshop and use local
computing resources, I simply checkout my current projects and I
can work locally. When I am done, I check everything back in and
when I get home, I can sync my local files.
-------

Another aspect of python I really appreciate are the unit testing
facilities. The doctest, unittest, and test modules make it easy to
include thorough tests: crucial for making sure that you can trust the
results of your programs. Organizing these tests in MATLAB and with
other languages was such a pain that I would often be tempted to omit
the unit tests and just run a few simulations, finding errors on the
fly.

Now I almost always write unit tests along with---or sometimes
before---I write the code.

Michael.
Mar 9 '06 #53
Michael McNeil Forbes <mf*****@lnsDOTmit.edu> writes:

I find that version control (VC) has many advantages for
scientific research (I am a physicist).


Greg Wilson also makes that point in this note:

http://www.nature.com/naturejobs/200...7050-600b.html

Where he describes his excellent (Python Software Foundation sponsored)
course on software carpentry for scientists:

http://www.third-bit.com/swc2/index.html

Regards, Phil
Mar 9 '06 #54

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Jeff Rush | last post by:
I'd like to extend an invitation to those who would like to get involved in advocating the use of Python. In August, the PSF hired me, for a 6-mo contract, to coordinate the Python advocacy...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.