473,725 Members | 2,392 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Good programming style

I'm still learning python and would like to know what's a good way of
organizing code.

I am writing some scripts to scrape a number of different website that
hold similar information and then collating it all together. Obviously
each site needs to be handled differently, but once the information is
collected then more generic functions can be used.

Is it best to have it all in one script or split it into per site
scripts that can then be called by a manager script? If everything is
in one script would you have per site functions to extract the data or
generic function that contain vary slightly depending on the site, for
example

import firstSiteScript
import secondSiteScrip t

firstsitedata = firstSiteScript .getData('searc h_str)
secondsitedata = secondSiteScrip t.getData('sear ch_str)
etc etc

OR

def getFirstSiteDat a(search_str):
etc etc
def getSecondSiteDa ta(search_str):
etc etc

OR

def getdata(search_ str, website):
if website == 'firstsite':
....
elif website =='secondsite':

etc
Sep 12 '08 #1
14 1835
Astley Le Jasper a crit :
I'm still learning python and would like to know what's a good way of
organizing code.

I am writing some scripts to scrape a number of different website that
hold similar information and then collating it all together. Obviously
each site needs to be handled differently, but once the information is
collected then more generic functions can be used.

Is it best to have it all in one script or split it into per site
scripts that can then be called by a manager script?
If everything is
in one script would you have per site functions to extract the data or
generic function that contain vary slightly depending on the site,
As far as I'm concerned, I'd choose the first solution. Decoupling
what's varying (here, site-specific stuff) from "invariants " is so far
the best way I know to keep complexity manageable.
for
example

import firstSiteScript
import secondSiteScrip t

firstsitedata = firstSiteScript .getData('searc h_str)
secondsitedata = secondSiteScrip t.getData('sear ch_str)
etc etc
Even better :

- put generic functions in a 'generic' module
- put all site-specific stuff each in it's own module in a specific
'site_scripts' directory
- in your 'main' script, scan the site_scripts directory to loop over
site-specific modules, import them and run them (look for the __import__
function).

This is kind of a Q&D lightweight plugin system, that avoids having to
hard-code imports and calls in the main script, so you just have to
add/remove site-specific script to/from the site_scripts directory .

Also, imported modules are not recompiled on each import - only when
they change - while the 'main' script get recompiled on each invocation.

(snip)
OR

def getdata(search_ str, website):
if website == 'firstsite':
....
elif website =='secondsite':
This one is IMHO the very worst thing to do.

My 2 cents...
Sep 12 '08 #2
On 12 Sep, 12:44, Bruno Desthuilliers <bruno.
42.desthuilli.. .@websiteburo.i nvalidwrote:
Astley Le Jasper a crit :
I'm still learning python and would like to know what's a good way of
organizing code.
I am writing some scripts to scrape a number of different website that
hold similar information and then collating it all together. Obviously
each site needs to be handled differently, but once the information is
collected then more generic functions can be used.
Is it best to have it all in one script or split it into per site
scripts that can then be called by a manager script?
If everything is
in one script would you have per site functions to extract the data or
generic function that contain vary slightly depending on the site,

As far as I'm concerned, I'd choose the first solution. Decoupling
what's varying (here, site-specific stuff) from "invariants " is so far
the best way I know to keep complexity manageable.
for
example
import firstSiteScript
import secondSiteScrip t
firstsitedata = firstSiteScript .getData('searc h_str)
secondsitedata = secondSiteScrip t.getData('sear ch_str)
etc etc

Even better :

- put generic functions in a 'generic' module
- put all site-specific stuff each in it's own module in a specific
'site_scripts' directory
- in your 'main' script, scan the site_scripts directory to loop over
site-specific modules, import them and run them (look for the __import__
function).

This is kind of a Q&D lightweight plugin system, that avoids having to
hard-code imports and calls in the main script, so you just have to
add/remove site-specific script to/from the site_scripts directory .

Also, imported modules are not recompiled on each import - only when
they change - while the 'main' script get recompiled on each invocation.

(snip)
OR
def getdata(search_ str, website):
* if website == 'firstsite':
* * ....
* elif website =='secondsite':

This one is IMHO the very worst thing to do.

My 2 cents...
Excellent, thanks for that.
Sep 12 '08 #3
Astley Le Jasper <As************ *@gmail.comwrit es:
Is it best to have it all in one script or split it into per site
scripts that can then be called by a manager script? If everything
is in one script would you have per site functions to extract the
data or generic function that contain vary slightly depending on the
site, for example

import firstSiteScript
import secondSiteScrip t
First: each of these things you're importing is a "module" in Python.
A script is what I prefer, for clarity, to call a "program": it's
intended to be executed independently as the top level of execution.

Second: please do yourself a favour and drop the camelCaseNames.
Follow PEP 8 <URL:http://www.python.org/dev/peps/pep-0008for style
and naming in your Python code.
firstsitedata = firstSiteScript .getData('searc h_str)
secondsitedata = secondSiteScrip t.getData('sear ch_str)
etc etc
I'm presuming that there will be large areas of common functionality
between these different sites. On that basis, it's prbably best to
treat the differences as differences of *configuration* where
possible, instead of having separate modules for the entire site.

You might like to look at a web framework which gathers much of this
functionality together for you, and provides flexible ways to define
different sites in terms of those common elements
<URL:http://wiki.python.org/moin/WebFrameworks>.

--
\ “Following fashion and the status quo is easy. Thinking about |
`\ your users' lives and creating something practical is much |
_o__) harder.” —Ryan Singer, 2008-07-09 |
Ben Finney
Sep 14 '08 #4
On 2008-09-14, Ben Finney <bi************ ****@benfinney. id.auwrote:
Astley Le Jasper <As************ *@gmail.comwrit es:
>Is it best to have it all in one script or split it into per
site scripts that can then be called by a manager script? If
everything is in one script would you have per site functions
to extract the data or generic function that contain vary
slightly depending on the site, for example

import firstSiteScript
import secondSiteScrip t

First: each of these things you're importing is a "module" in
Python. A script is what I prefer, for clarity, to call a
"program": it's intended to be executed independently as the
top level of execution.

Second: please do yourself a favour and drop the camelCaseNames.
Follow PEP 8 <URL:http://www.python.org/dev/peps/pep-0008for style
and naming in your Python code.
If he finds camelcase more readable and easier to type (as do
I), how is switching to underscores "doing himself a favor"?

I'm generally in favor of using a consistent naming style
throughout a project, but I don't see why the naming style used
in my source code should be subject to somebody else's
arbitrary standard.

When it comes to writing code intended for the standard library
in the main Python distribution, I would certainly defer to the
existing standard as defined in PEP 8. However, I don't see
any reason that style should be imposed on all everybody else.

--
Grant

Sep 15 '08 #5
Grant Edwards <gr****@visi.co mwrites:
On 2008-09-14, Ben Finney <bi************ ****@benfinney. id.auwrote:
Second: please do yourself a favour and drop the camelCaseNames.
Follow PEP 8 <URL:http://www.python.org/dev/peps/pep-0008for style
and naming in your Python code.

If he finds camelcase more readable and easier to type (as do
I), how is switching to underscores "doing himself a favor"?

I'm generally in favor of using a consistent naming style
throughout a project, but I don't see why the naming style used
in my source code should be subject to somebody else's
arbitrary standard.
Because the code we write rarely stays isolated from other code. There
is an existing convention, and it's better to pick a (sufficiently
sane) style convention and stick to it than argue about what the
convention should be.
When it comes to writing code intended for the standard library
in the main Python distribution, I would certainly defer to the
existing standard as defined in PEP 8. However, I don't see
any reason that style should be imposed on all everybody else.
Who's imposing? I'm saying it's a good idea for everyone to do it, and
going so far as to say that one is doing oneself a favour by following
the convention. I have no more power than you to "impose" convention
on anyone.

--
\ “‘Did you sleep well?’ ‘No, I made a couple of mistakes.’” |
`\ —Steven Wright |
_o__) |
Ben Finney
Sep 15 '08 #6
On 2008-09-15, Ben Finney <bi************ ****@benfinney. id.auwrote:
Grant Edwards <gr****@visi.co mwrites:
>On 2008-09-14, Ben Finney <bi************ ****@benfinney. id.auwrote:
>>Second: please do yourself a favour and drop the
camelCaseName s. Follow PEP 8
<URL:http://www.python.org/dev/peps/pep-0008for style and
naming in your Python code.

If he finds camelcase more readable and easier to type (as do
I), how is switching to underscores "doing himself a favor"?

I'm generally in favor of using a consistent naming style
throughout a project, but I don't see why the naming style
used in my source code should be subject to somebody else's
arbitrary standard.

Because the code we write rarely stays isolated from other
code. There is an existing convention,
There are many existing conventions.
and it's better to pick a (sufficiently sane) style convention
and stick to it than argue about what the convention should
be.
I suppose if everybody agreed to pick one, and all the source
code in the world was changed to meet it, that would "a good
thing". It just seems like a goal so unrealistic as to make it
a bit of an overstatement to tell people they're better off
following convention X than following convention Y.

When packages as significant as wxPython use naming conventions
other than PEP 8, I find it hard to make a case that the PEP 8
naming convention is any better than any other.
>When it comes to writing code intended for the standard
library in the main Python distribution, I would certainly
defer to the existing standard as defined in PEP 8. However,
I don't see any reason that style should be imposed on all
everybody else.

Who's imposing? I'm saying it's a good idea for everyone to do
it, and going so far as to say that one is doing oneself a
favour by following the convention. I have no more power than
you to "impose" convention on anyone.
My apologies -- "impose" was too strong a word to use.

If we were starting from scratch and there was no extant source
code in the world, then it would make sense to encourage
everybody to pick one convention. [I still think it would be
rather quixotic.] But, there are so many projects out there
with naming conventions other than PEP 8, that I don't see how
there's an advantage to picking one over another (except for
the obvious also-rans like "all upper case, no vowels, and a
maximum length of 6 characters").

I'll agree that sticking with a single convention within a
project is definitely a good thing.

I'm personally aware of mixed/camel-case projects from 25+
years ago, so I'm afraid PEP 8 came along a bit too late...

--
Grant

Sep 15 '08 #7
Grant Edwards wrote:
When packages as significant as wxPython use naming conventions
other than PEP 8, I find it hard to make a case that the PEP 8
naming convention is any better than any other.
This relates to a question I was thinking about...

I'm looking at porting a library that was written for COM and .Net to
work as a Python module, and was wondering whether it would be better to
stick to the library's current naming convention so that the API is as
similar as possible on each platform, or to adopt a "when in Rome..."
policy and follow the "most mainstream" naming pattern for each
platform/language.

Adelle.

Sep 15 '08 #8
On Sep 14, 7:10*pm, Grant Edwards <gra...@visi.co mwrote:
On 2008-09-15, Ben Finney <bignose+hate s-s...@benfinney. id.auwrote:
Grant Edwards <gra...@visi.co mwrites:
On 2008-09-14, Ben Finney <bignose+hate s-s...@benfinney. id.auwrote:
>Second: please do yourself a favour and drop the
camelCaseNames . Follow PEP 8
<URL:http://www.python.org/dev/peps/pep-0008for style and
naming in your Python code.
If he finds camelcase more readable and easier to type (as do
I), how is switching to underscores "doing himself a favor"?
I'm generally in favor of using a consistent naming style
throughout a project, but I don't see why the naming style
used in my source code should be subject to somebody else's
arbitrary standard.
Because the code we write rarely stays isolated from other
code. There is an existing convention,

There are many existing conventions.
and it's better to pick a (sufficiently sane) style convention
and stick to it than argue about what the convention should
be.

I suppose if everybody agreed to pick one, and all the source
code in the world was changed to meet it, that would "a good
thing". *It just seems like a goal so unrealistic as to make it
a bit of an overstatement to tell people they're better off
following convention X than following convention Y.

When packages as significant as wxPython use naming conventions
other than PEP 8, I find it hard to make a case that the PEP 8
naming convention is any better than any other.
When it comes to writing code intended for the standard
library in the main Python distribution, I would certainly
defer to the existing standard as defined in PEP 8. *However,
I don't see any reason that style should be imposed on all
everybody else.
Who's imposing? I'm saying it's a good idea for everyone to do
it, and going so far as to say that one is doing oneself a
favour by following the convention. I have no more power than
you to "impose" convention on anyone.

My apologies -- "impose" was too strong a word to use.

If we were starting from scratch and there was no extant source
code in the world, then it would make sense to encourage
everybody to pick one convention. [I still think it would be
rather quixotic.] But, there are so many projects out there
with naming conventions other than PEP 8, that I don't see how
there's an advantage to picking one over another (except for
the obvious also-rans like "all upper case, no vowels, and a
maximum length of 6 characters").

I'll agree that sticking with a single convention within a
project is definitely a good thing.

I'm personally aware of mixed/camel-case projects from 25+
years ago, so I'm afraid PEP 8 came along a bit too late...

--
Grant
+1

CamelCase FTW!

~Sean
Sep 15 '08 #9
Adelle Hartley <ad****@akemi.c om.auwrites:
I'm looking at porting a library that was written for COM and .Net
to work as a Python module, and was wondering whether it would be
better to stick to the library's current naming convention so that
the API is as similar as possible on each platform, or to adopt a
"when in Rome..." policy and follow the "most mainstream" naming
pattern for each platform/language.
I think it's more important for Python library APIs to comply with the
Python coding guidelines (as specified in PEP 8) than to comply with
standards in other languages.

The Python library you're implementing isn't being used in those other
languages, so the conventions of other languages have little
relevance. It's being used in Python code, so it should mesh well with
PEP 8 compliant code — by having the API itself comply with PEP 8.

--
\ “When cryptography is outlawed, bayl bhgynjf jvyy unir |
`\ cevinpl.” —Anonymous |
_o__) |
Ben Finney
Sep 15 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

24
3605
by: matty | last post by:
Go away for a few days and you miss it all... A few opinions... Programming is a craft more than an art (software engineering, not black magic) and as such, is about writing code that works, first and foremost. If it works well, even better. The same goes for ease of maintenance, memory footprint, speed, etc, etc. Most of the time, people are writing code for a use in the *real world*, and not just as an academic exercise. Look at...
29
2011
by: Cheng Mo | last post by:
Recently I happens to read some sourcecode which has different style from my previous experience. In this style, all functions return a RETURN_TYPE_T which is defined as typedef unsigned int RETURN_TYPE_T; and the return value can only be enum value enum { SUCCESS = 0, FAILURE = 1,
20
5139
by: Clark | last post by:
Hi all. I'm looking for good C source code to study and be able to advance my C programming skills. Do you recomend any open source project in particular that in your opinion has good writen C code? Thanks.
60
3958
by: K. G. Suarez | last post by:
Hello everyone. I am new to programming and my uncle gave me a copy of "C For Dummies 2nd Edition". I am up to chapter 9 right now. He probably saw me struggling with "The C Programming Language" by Ritchie and Kernigahn and felt bad. Does anyone have experience with this book? I feel that it is helping me along pretty well. But how much will this book teach me? What would be the next book to read?
43
2656
by: Sensei | last post by:
Hi! I'm thinking about a good programming style, pros and cons of some topics. Of course, this has nothing to do with indentation... Students are now java-dependent (too bad) and I need some serious motivations for many issues... I hope you can help me :) I begin with the two major for now, others will come for sure! - function variables: they're used to java, so no pointers. Personally I'd use always pointers, but why could be better...
2
315
by: Harley | last post by:
Hello, I was VERY blessed with a Christmas gift of visual studio .net from a man I hardly know who had heard of my plans of software developement. So I am probably the only person in the world who actualy has this great IDE and don't even know vb.net (or c sharp etc.). I have some prior exposure to simple scripting language such as javascript and I understand data types etc. (basic programming concepts and procedures) but I don't have any...
0
1211
by: Andr | last post by:
I'm trying to change an app so that it uses gettext for translations rather than the idiosyncratic way I am using. I've tried the example on the wxPython wiki http://wiki.wxpython.org/index.cgi/RecipesI18n but found that the accented letters would not display properly. I have found a workaround that works from Python in a Nutshell; however it is said in that book that "...this is not good style". I would like to do things "in good...
206
8343
by: WaterWalk | last post by:
I've just read an article "Building Robust System" by Gerald Jay Sussman. The article is here: http://swiss.csail.mit.edu/classes/symbolic/spring07/readings/robust-systems.pdf In it there is a footprint which says: "Indeed, one often hears arguments against building exibility into an engineered sys- tem. For example, in the philosophy of the computer language Python it is claimed: \There should be one|and preferably only one|obvious...
2
1972
by: silvrfoxx | last post by:
I created this using notepad and was checking it in Firefox, but when i finished and checked it in IE it looks like crap. I even opened up Dreamweaver and it shows it looking great. I can't find what is causing it. I have double, triple checked to make sure that all my ", <, >, tr, td, etc. are opened and closed properly. Please can anyone help? Thanks in advance. Here is my code: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0...
0
8878
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main usage, and What is the difference between ONU and Router. Lets take a closer look ! Part I. Meaning of...
1
9165
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8079
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development projectplanning, coding, testing, and deploymentwithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
6006
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4507
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4776
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3216
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2626
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2153
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.