automatically grading small programming assignments

[Brian Blais]

Hello,

I have a couple of classes where I teach introductory programming using Python. What
I would love to have is for the students to go through a lot of very small programs,
to learn the basic programming structure. Things like, return the maximum in a list,
making lists with certain patterns, very simple string parsing, etc. Unfortunately,
it takes a lot of time to grade such things by hand, so I would like to automate it
as much as possible.

I envision a number of possible solutions. In one solution, I provide a function
template with a docstring, and they have to fill it in to past a doctest. Is there a
good (and safe) way to do that online? Something like having a student post code,
and the doctest returns. I'd love to allow them to submit until they get it, logging
each attempt.

Or perhaps there is a better way to do this sort of thing. How do others who teach
Python handle this?
thanks,
Brian Blais

--
-----------------

bb****@bryant.edu
http://web.bryant.edu/~bblais

[Beliavsky]

Brian Blais wrote:

Hello,

I have a couple of classes where I teach introductory programming using Python. What
I would love to have is for the students to go through a lot of very small programs,
to learn the basic programming structure. Things like, return the maximum in a list,
making lists with certain patterns, very simple string parsing, etc. Unfortunately,
it takes a lot of time to grade such things by hand, so I would like to automate it
as much as possible.

I envision a number of possible solutions. In one solution, I provide a function
template with a docstring, and they have to fill it in to past a doctest. Is there a
good (and safe) way to do that online? Something like having a student post code,
and the doctest returns. I'd love to allow them to submit until they get it, logging
each attempt.

On a different matter, coding style, you could run something such as
Pylint on submitted codes and penalize them based on the number of
warnings emitted.
Maybe some adjustments would be necessary -- my experience with picky
compilers is that most but not ALL warnings indicate problems.

[Paddy]

Brian Blais wrote:

Hello,

I have a couple of classes where I teach introductory programming using Python. What
I would love to have is for the students to go through a lot of very small programs,
to learn the basic programming structure. Things like, return the maximum in a list,
making lists with certain patterns, very simple string parsing, etc. Unfortunately,
it takes a lot of time to grade such things by hand, so I would like to automate it
as much as possible.

I envision a number of possible solutions. In one solution, I provide a function
template with a docstring, and they have to fill it in to past a doctest. Is there a
good (and safe) way to do that online? Something like having a student post code,
and the doctest returns. I'd love to allow them to submit until they get it, logging
each attempt.

Or perhaps there is a better way to do this sort of thing. How do others who teach
Python handle this?

It might turn out to be a poor substitute for the personal touch,
especially If they are just starting to program.

- Paddy.

[Steven Bethard]

Brian Blais wrote:

I have a couple of classes where I teach introductory programming using
Python. What I would love to have is for the students to go through a
lot of very small programs, to learn the basic programming structure.
Things like, return the maximum in a list, making lists with certain
patterns, very simple string parsing, etc. Unfortunately, it takes a
lot of time to grade such things by hand, so I would like to automate it
as much as possible.

I envision a number of possible solutions. In one solution, I provide a
function template with a docstring, and they have to fill it in to past
a doctest. Is there a good (and safe) way to do that online? Something
like having a student post code, and the doctest returns. I'd love to
allow them to submit until they get it, logging each attempt.

A few people were playing around with a secure Python online
interpreter/interactive prompt a while back:

http://groups-beta.google.com/group/...46d738a8859c2f

I don't think any of them was quite at the usability level you'd need
though.
Can you just have the students download one file for each program you
want them to run? I'd ship something like::

import unittest

# ========================================
# This is the function you need to fill in
# ========================================
def list_max(items):
'''Your code goes here'''
# ==================================
# Don't modify code below this point
# ==================================
class Test(unittest.TestCase):
def test_range(self):
self.failUnlessEqual(list_max(range(10)), 10)
def test_first_elem(self):
self.failUnlessEqual(list_max([1, 0, 0, 0]), 1)
...

if __name__ == '__main__':
unittest.main()

Then all your students would have to do is download the file, fill in
the definition of the appropriate function, and run the same file until
the unittest said that everything worked. I guess that doesn't solve
your "logging each attempt" problem though...

STeVe

[bearophileHUGS]

Brian Blais, just an idea. Create an online form to upload the tiny
program(s). Such programs can be one for file. Then on your PC you can
run a script that loads each of such programs, and runs a good series
of tests, to test their quality... Such tests can be about all things,
speed, coding quality, that the results are correct, etc. You can even
put some way to autenticate students... (BTW, good students don't
cheat. Cheating isn't a good way to learn to program, and they probably
know this).

Bye,
bearophile

[commander.coder]

be************@lycos.com wrote:
Then on your PC you can

run a script that loads each of such programs, and runs a good series
of tests, to test their quality...

What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?

[Dennis Benzinger]

Am Thu, 14 Dec 2006 12:27:07 -0500
schrieb Brian Blais <bb****@bryant.edu>:

Hello,

I have a couple of classes where I teach introductory programming
using Python. What I would love to have is for the students to go
through a lot of very small programs, to learn the basic programming
structure. Things like, return the maximum in a list, making lists
with certain patterns, very simple string parsing, etc.
Unfortunately, it takes a lot of time to grade such things by hand,
so I would like to automate it as much as possible.

I envision a number of possible solutions. In one solution, I
provide a function template with a docstring, and they have to fill
it in to past a doctest. Is there a good (and safe) way to do that
online? Something like having a student post code, and the doctest
returns. I'd love to allow them to submit until they get it, logging
each attempt.

Or perhaps there is a better way to do this sort of thing. How do
others who teach Python handle this?
thanks,
Brian Blais

Perhaps the Sphere Online Judge can help you: https://www.spoj.pl/info/
Dennis

[commander.coder]

be************@lycos.com wrote:
Then on your PC you can

run a script that loads each of such programs, and runs a good series
of tests, to test their quality...

What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?

[Carl J. Van Arsdall]

co*************@hotmail.com wrote:

be************@lycos.com wrote:
Then on your PC you can

>run a script that loads each of such programs, and runs a good series
of tests, to test their quality...

What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?

The system administrator should make sure that student user accounts (or
the auto testing account) doesn't have access to that. Probably should
make sure that user applications only get a limited amount of memory too.

--

Carl J. Van Arsdall
cv*********@mvista.com
Build and Release
MontaVista Software

[Brian Blais]

co*************@hotmail.com wrote:

be************@lycos.com wrote:
Then on your PC you can

>run a script that loads each of such programs, and runs a good series
of tests, to test their quality...

What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?

I was thinking of including a dummy os.py and sys.py, so import os, and import sys
would fail. Would this work? Is there anything else obvious. I can have student
authentication, that's not a problem.
bb
--
-----------------

bb****@bryant.edu
http://web.bryant.edu/~bblais

[Brian Blais]

Paddy wrote:

It might turn out to be a poor substitute for the personal touch,
especially If they are just starting to program.

Oh, I didn't mean it to completely replace me grading things, but I think it would be
useful if there were a lot of little assignments that could be done automatically,
and then some larger ones that I would grade by hand. The little ones could be set
up so that they can submit as many times as they want, until they get it right.

bb

--
-----------------

bb****@bryant.edu
http://web.bryant.edu/~bblais

[Paddy]

Brian Blais wrote:

co*************@hotmail.com wrote:

be************@lycos.com wrote:
Then on your PC you can

run a script that loads each of such programs, and runs a good series
of tests, to test their quality...

What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?

I was thinking of including a dummy os.py and sys.py, so import os, and import sys
would fail. Would this work? Is there anything else obvious. I can have student
authentication, that's not a problem.

Setting up a Crunchy server in a virtualized OS environment would give
you some security.

- Paddy.

[Paul Rubin]

Brian Blais <bb****@bryant.eduwrites:

Unfortunately, it takes a lot of time to grade such things by hand, so
I would like to automate it as much as possible.
...
Or perhaps there is a better way to do this sort of thing. How do
others who teach Python handle this?

I think you should not attempt this. It means grading the program on
pure functionality. Writing good code is about more than
functionality; it's also about communicating with humans. Supplying a
doctest is a reasonable idea but students should just run the test
themselves until their code passes, no need to log all their debugging
attempts. When they submit something that passes the test, you should
read the code and supply some feedback about about their coding style
and so forth. This matters as much as the pure issue of whether the
program works. Imagine an English teacher wanting to grade class
papers automatically by running them through a spelling and grammar
checker without reading them. If the workload of grading manually is
too high, see if you can bring on an assistant to help.

[Dan Bishop]

On Dec 14, 8:36 pm, Brian Blais <[e-mail]:

[e-mail] wrote:

[e-mail] wrote:
Then on your PC you can

run a script that loads each of such programs, and runs a good series
of tests, to test their quality...

What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?I was thinking of including a dummy os.py and sys.py, so import os, and import sys

would fail. Would this work?

How would they access their command-line arguments without sys.argv?

[Caleb Hattingh]

Hi Brian

You could make great use of XML-RPC here. XML-RPC is /really/ easy to
use.

Here is a simple example:

http://aspn.activestate.com/ASPN/Coo...n/Recipe/81549

You put procedures on the server that will check the args against a the
required result, and report back to the student whether it passes or
fails.

Here is another example using xml-rpc over https, for security:

http://aspn.activestate.com/ASPN/Coo.../Recipe/496786

So, the idea is that the student calls a procedure on the xml-rpc
server (which you set up), and passes his results as an argument, and
your server procedure can return True or False.

One benefit is that if you change the input to the tests, you need only
update the server. Actually, you could let the procedures on the
server accept test input and student results, and return True or False.
This would be cool :)

Caleb

On Dec 14, 6:27 pm, Brian Blais <[e-mail]:

Hello,

I have a couple of classes where I teach introductory programming using Python. What
I would love to have is for the students to go through a lot of very small programs,
to learn the basic programming structure. Things like, return the maximum in a list,
making lists with certain patterns, very simple string parsing, etc. Unfortunately,
it takes a lot of time to grade such things by hand, so I would like to automate it
as much as possible.

I envision a number of possible solutions. In one solution, I provide a function
template with a docstring, and they have to fill it in to past a doctest. Is there a
good (and safe) way to do that online? Something like having a student post code,
and the doctest returns. I'd love to allow them to submit until they get it, logging
each attempt.

Or perhaps there is a better way to do this sort of thing. How do others who teach
Python handle this?

thanks,

Brian Blais

--
-----------------

[e-mail]
http://web.bryant.edu/~bblais

[Brian Blais]

Dan Bishop wrote:

On Dec 14, 8:36 pm, Brian Blais <[e-mail]:

>[e-mail] wrote:

>>[e-mail] wrote:
Then on your PC you can
run a script that loads each of such programs, and runs a good series
of tests, to test their quality...
What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?I was thinking of including a dummy os.py and sys.py, so import os, and import sys

would fail. Would this work?

How would they access their command-line arguments without sys.argv?

the types of assignments that I am envisioning (finding the maximum in a list,
parsing strings, etc.) will not need anything offered in os or sys. Certainly, if
they were needed, another solution would need to be found.
bb

--
-----------------

bb****@bryant.edu
http://web.bryant.edu/~bblais

[=?iso-8859-1?B?QW5kcuk=?=]

Brian Blais wrote:

Dan Bishop wrote:

On Dec 14, 8:36 pm, Brian Blais <[e-mail]:

[e-mail] wrote:
[e-mail] wrote:
Then on your PC you can
run a script that loads each of such programs, and runs a good series
of tests, to test their quality...
What happens if someone-- perhaps not even someone in the class-- does
some version of os.system('rm -Rf /') ?I was thinking of including a dummy os.py and sys.py, so import os, and import sys
would fail. Would this work?

How would they access their command-line arguments without sys.argv?

the types of assignments that I am envisioning (finding the maximum in a list,
parsing strings, etc.) will not need anything offered in os or sys. Certainly, if
they were needed, another solution would need to be found.

If you do a search on the web, you will find that there are many other
security problems in Python that can not be prevented by simply
including dummy modules for os and sys.

Brett Cannon's PhD thesis is, afaik, based on looking at ways of
creating a secure Python environment. Other suggestions mentioned
before (like running in a virtual environment) might be the best way to
go for now. Having the user run the program on their own machine (like
would be done with the current version of Crunchy already mentioned in
this thread) would keep yours safe. Crunchy's doctest feature could be
"easily" modified so that it logs the number of attempts and mail the
results to a given address.

André

[Slawomir Nowaczyk]

On Thu, 14 Dec 2006 21:36:31 -0500
Brian Blais <bb****@bryant.eduwrote:

#Paddy wrote:

# It might turn out to be a poor substitute for the personal touch,
# especially If they are just starting to program.
#>
#Oh, I didn't mean it to completely replace me grading things, but I
#think it would be useful if there were a lot of little assignments
#that could be done automatically, and then some larger ones that I
#would grade by hand. The little ones could be set up so that they can
#submit as many times as they want, until they get it right.

Well, that sounds like a valid plan, but why would you want to grade the
little ones at all, then?

What I would most likely do is to publish those small assignments
together with a set of tests for each one, and say that they should
write programs and make sure their programs pass the tests.

If you wish, you could publish two sets of tests, the "easy" and
"tricky" ones, and have them use easy ones when writing program, and
only run it through the "tricky" tests when they believe the program is
bug-free. This can be a very valuable experience! (if you can devise the
right tests, of course ;)

If you either require the skills they develop doing "small" assignments
in the "big" assignments, or if you check 2-3 small assignments by hand,
you should be able to reduce cheating sufficiently... It's just a matter
of making sure they really *do* write programs and that those programs
*do* pass the tests.

Or just require students to hand in the small assignments, together with
the testing output, but do not check them at all (not too many will have
the guts to fake the outputs).

Then there is a whole range of ideas about peer review in the education
community, where you could get students to verify one another's
programs... But this can sometimes be tricky.

--
Best wishes,
Slawomir Nowaczyk
( Sl***************@cs.lth.se )

Someone Else's Code - a commonly used synonym for "Bad Code"