SNMP Toolkit

[Matthew Bell]

Hi,

I'm looking for a high-performance SNMP manager toolkit (SNMPv1 / v2,
GET / GETNEXT / GETBULK) I can use with Python on Windows2K/XP. I
wonder if anyone has got any suggestions? Commercial software is
fine, as it's for a specialised in-house application.

I've tried PySNMP which, while a fine piece of code (I've learnt a lot
looking through it!), it's fairly CPU intensive in my application.
I've also tried using UCDSNMP via popen() but that has a tendency to
hang in a multi-threaded environment.

So, does anyone know of a solid, multi-threading capable, fast SNMP
library with a reasonable Python interface that will run on Windows
without me needing to get my head round a C compiler to build it? I
know I should be all manly-man and hew my own C-based SNMP library
from scratch but, basically, while I'm only a mediocre Python
programmer, the last compiled language I used was COBOL...

Thanks!
Matthew.

[Les Smithson]

>>>>> "Matthew" == Matthew Bell <li****@cix.co.uk> writes:

Matthew> Hi, I'm looking for a high-performance SNMP manager
Matthew> toolkit (SNMPv1 / v2, GET / GETNEXT / GETBULK) I can use
Matthew> with Python on Windows2K/XP. I wonder if anyone has got
Matthew> any suggestions? Commercial software is fine, as it's
Matthew> for a specialised in-house application.

Matthew> I've tried PySNMP which, while a fine piece of code (I've
Matthew> learnt a lot looking through it!), it's fairly CPU
Matthew> intensive in my application. I've also tried using
Matthew> UCDSNMP via popen() but that has a tendency to hang in a
Matthew> multi-threaded environment.

Matthew> So, does anyone know of a solid, multi-threading capable,
Matthew> fast SNMP library with a reasonable Python interface that
Matthew> will run on Windows without me needing to get my head
Matthew> round a C compiler to build it? I know I should be all
Matthew> manly-man and hew my own C-based SNMP library from
Matthew> scratch but, basically, while I'm only a mediocre Python
Matthew> programmer, the last compiled language I used was
Matthew> COBOL...

Matthew> Thanks! Matthew.

Pysnmp is the only 100% pure Python SNMP implementation I'm aware
of. Perhaps you could address your program's inefficiencies instead?
What are you doing that's so CPU intensive? SNMP agents/managers that
I've worked on (admittedly not using Python) were always I/O bound,
rather than CPU bound.

[Matthew Bell]

Les Smithson <ls******@NOhare.SPAM.demon.co.uk> wrote in message news:<m3************@hare.demon.co.uk>...

>> "Matthew" == Matthew Bell <li****@cix.co.uk> writes:

Matthew> Hi, I'm looking for a high-performance SNMP manager
Matthew> toolkit (SNMPv1 / v2, GET / GETNEXT / GETBULK) I can use
Matthew> with Python on Windows2K/XP.

[...deletia...]
Pysnmp is the only 100% pure Python SNMP implementation I'm aware
of. Perhaps you could address your program's inefficiencies instead?
What are you doing that's so CPU intensive? SNMP agents/managers that
I've worked on (admittedly not using Python) were always I/O bound,
rather than CPU bound.

The application is monitoring 30+ big LAN switches with a large
variety of information being collected from each one. Using UCD-SNMP
then (if we ignore the way it tends to hang sometimes), indeed, I/O is
the limiting factor rather than CPU. The Python code I've written
that deals with all the data amazes me by how relatively little CPU it
consumes (yay for Python!); if I could find a way to get UCD-SNMP to
work reliably in a multi-threaded environment then I'd happily carry
on using that. But the hanging is getting really frustrating as not
even select() seems to be able to catch it. It just hangs.

Using PySNMP rather than UCD-SNMP certainly gets round the random
hanging, but it does make CPU go through the roof.

I'm not actually necessarily looking for a pure Python SNMP toolkit.
To be honest, given the large amounts of SNMP traffic that this
application has to generate, it would make an awful lot of sense to
have the low-level SNMP grunt work done in a C / C++ library and then
have the Python code just deal with the processed OIDs and values.
It's just that I can't find any such library that'll run on Win32 and
that has a convenient Python wrapper. I've got a recollection that I
stumbled across a commercial C++ based SNMP library that came with a
Python wrapper about a year ago, but no amount of googling has found
it again :-(

I have found any number of native COM/.NET/ASP/etc C++ SNMP toolkits
and tried using the Python Win32 extensions to talk to them but I just
don't understand enough about low-level Win32 calls, event handling
etc to get them to work, particularly as the code examples typically
expect you to either be using Visual C++ or Visual Basic.

Thanks anyway,
Matthew.

[Les Smithson]

>>>>> "Matthew" == Matthew Bell <li****@cix.co.uk> writes:

Matthew> I have found any number of native COM/.NET/ASP/etc C++
Matthew> SNMP toolkits and tried using the Python Win32 extensions
Matthew> to talk to them but I just don't understand enough about
Matthew> low-level Win32 calls, event handling etc to get them to
Matthew> work, particularly as the code examples typically expect
Matthew> you to either be using Visual C++ or Visual Basic.

Matthew> Thanks anyway, Matthew.

I have to ask this - did you look at snmpy
(http://snmpy.sourceforge.net)? This uses ucd-snmp/net-snmp for the
grunt. It doesn't claim to be ported to Windows, but net-snmp is, and
the C module in snmpy doesn't *look* that difficult to build on
Windows.

[Roy Smith]

In article <m3************@hare.demon.co.uk>,
Les Smithson <ls******@NOhare.SPAM.demon.co.uk> wrote:

>> "Matthew" == Matthew Bell <li****@cix.co.uk> writes:

Matthew> I have found any number of native COM/.NET/ASP/etc C++
Matthew> SNMP toolkits and tried using the Python Win32 extensions
Matthew> to talk to them but I just don't understand enough about
Matthew> low-level Win32 calls, event handling etc to get them to
Matthew> work, particularly as the code examples typically expect
Matthew> you to either be using Visual C++ or Visual Basic.

Matthew> Thanks anyway, Matthew.

I have to ask this - did you look at snmpy
(http://snmpy.sourceforge.net)? This uses ucd-snmp/net-snmp for the
grunt. It doesn't claim to be ported to Windows, but net-snmp is, and
the C module in snmpy doesn't *look* that difficult to build on
Windows.

What have been people's experiences with snmpy? The last time I looked
at it (over a year ago), I had troubles building it (on linux) because
it had dependencies on an oldish version of net-snmp.

I do a lot of SNMP work, but have never really found a good python SNMP
package. I looked at PySNMP (the pure python one) but never managed to
get that to work either. It also just seems wrong to be doing all the
low-level BER and packet building stuff in Python.

[Mike C. Fletcher]

Les Smithson wrote:

>>"Matthew" == Matthew Bell <li****@cix.co.uk> writes:
>>
>>

....
Matthew> I've tried PySNMP which, while a fine piece of code (I've
Matthew> learnt a lot looking through it!), it's fairly CPU
Matthew> intensive in my application. I've also tried using
Matthew> UCDSNMP via popen() but that has a tendency to hang in a
Matthew> multi-threaded environment.

....
SNMP agents/managers that
I've worked on (admittedly not using Python) were always I/O bound,
rather than CPU bound.

PySNMP is actually a *very* heavy library compared to C ASN.1 parsers.
I've certainly run into situations where it's just not fast enough to do
what we want to do. Thing is, it's flexible and can be readily
integrated with Twisted, so we still use it, even if it does mean it can
take 10 minutes to scan a plant. That's all processor-bound time, not
IO bound.

Some day I'm going to try to squeeze some money out of the budget to
spend a week on figuring out how to make PySNMP faster, but until then
I'm afraid I just don't have a good suggestion for large SNMP querying
tasks.

Have fun,
Mike

________________________________________________
Mike C. Fletcher
Designer, VR Plumber, Coder
http://members.rogers.com/mcfletch/
blog: http://zope.vex.net/~mcfletch/plumbing/

[Michael Ströder]

Mike C. Fletcher wrote:

PySNMP is actually a *very* heavy library compared to C ASN.1 parsers.
I've certainly run into situations where it's just not fast enough to do
what we want to do.
[..]
That's all processor-bound time, not
IO bound.

AFAIK the BER encoding/decoding is implemented in pure Python. If one has
the time to implement these routines in C it'd be much less CPU intensive I
guess.

I wish support for BER and DER to be in the standard lib...

Ciao, Michael.

[popov]

Have you tried yapsnmp: http://yapsnmp.sourceforge.net/ ?
It is a Python wrapper around net-snmp, so should be fast.
It also claims to be thread-safe.
The problem you could have is that it's advertised as a posix module
only. But the wrapper is built using SWIG, so it should be possible to
build it for Windows without too much hassle.
Note that I've never used it myself, so can't comment on the package
itself.
However, I will have to do some snmp coding soon, and it's the first
one I will try.

[Ilya Etingof]

Michael Str?der <mi*****@stroeder.com> wrote:

[ skipped ]

AFAIK the BER encoding/decoding is implemented in pure Python. If one has
the time to implement these routines in C it'd be much less CPU intensive I
guess.

I've got an impression, that building/parsing BER is not computationally
intensive. In fact, BER encoding has been designed to cope with rather limited
resources (in terms of about 20 yo hardware!).

As an alternative bottleneck I'd rather propose 1) object instantiation and
2) function calls. As of this writing, pysnmp.asn1 code is designed as a top-down
parser what implies intensive recursion and object creation. Although not a real
receipt but rater a workaround, I'd suggest caching and reusing top-level
ASN.1/SNMP objects (such as SNMP message) inside your app whenever possible.
This might safe lots of CPU on a long run.

-ilya

[Matthew Bell]

Les Smithson <ls******@NOhare.SPAM.demon.co.uk> wrote in message news:<m3************@hare.demon.co.uk>...

I have to ask this - did you look at snmpy
(http://snmpy.sourceforge.net)? This uses ucd-snmp/net-snmp for the
grunt. It doesn't claim to be ported to Windows, but net-snmp is, and
the C module in snmpy doesn't *look* that difficult to build on
Windows.

Yes, I did look at snmpy. And was instantly scared off by the obvious
need to hack about with the C module to get it to compile on Win32. I
haven't even been able to get net-snmp itself to compile on Win32 - I
tried to follow the net-snmp readme about compiling it with cygwin and
just ended up in a world of pain. The thought of doing the same for
snmpy brings me out in hives.

For better or for worse, I don't come from a UNIX / C background and
so while such things may be straightforward for some, they're not for
me.

Thanks anyway,
Matthew.

[Les Smithson]

>>>>> "Ilya" == Ilya Etingof <il**@cray.glas.net> writes:

Ilya> I've got an impression, that building/parsing BER is not
Ilya> computationally intensive. In fact, BER encoding has been
Ilya> designed to cope with rather limited resources (in terms of
Ilya> about 20 yo hardware!).

Ilya> As an alternative bottleneck I'd rather propose 1) object
Ilya> instantiation and 2) function calls. As of this writing,
Ilya> pysnmp.asn1 code is designed as a top-down parser what
Ilya> implies intensive recursion and object creation. Although
Ilya> not a real receipt but rater a workaround, I'd suggest
Ilya> caching and reusing top-level ASN.1/SNMP objects (such as
Ilya> SNMP message) inside your app whenever possible. This might
Ilya> safe lots of CPU on a long run.

Ilya> -ilya

Out of interest, I ran a simple pysnmp test using the Python
profiler. The test case was a 1000 loops of a GETREQUEST/GETRESPONSE
of sysDescr, using pysnmp 2.0.8 on a Linux box. The top 10, sorted by
time, were:
514003 function calls in 8.100 CPU seconds

Ordered by: internal time
List reduced from 63 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function)
18000 0.760 0.000 1.300 0.000 v1.py:240(update)
14000 0.560 0.000 2.140 0.000 asn1.py:284(decode)
3000 0.520 0.000 0.770 0.000 asn1.py:633(num2str)
13000 0.490 0.000 1.600 0.000 asn1.py:263(encode)
25000 0.370 0.000 0.510 0.000 asn1.py:206(__init__)
52000 0.350 0.000 0.350 0.000 asn1.py:245(update)
4000 0.330 0.000 1.340 0.000 asn1.py:750(decode)
2000 0.330 0.000 0.700 0.000 asn1.py:606(str2num)
30000 0.320 0.000 1.190 0.000 v1.py:210(__setitem__)
21000 0.270 0.000 0.270 0.000 asn1.py:142(decode_tag)
By fiddling with the args/ARGS dictionary merging code in
v1.py:240(update), I cut its internal time by around 50%. I haven't
found any significant optimisations elsewhere. I can't figure out
where the time goes in the asn1.py decode/encode/update functions -
they don't have any loops and there's not much to them.

Would using the array module for holding the PDU make a difference?