From: =?ISO-8859-15?Q?Rodrigo_Nuno_Bragan=E7a_da_Cunha?=
Subject: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <3d8f4d27@news.vizzavi.pt>
Hi!

Once upon a time, a few years ago, I learned Lisp. I remember the 
language for allowing me to produce useful programs relatively fast and 
with very little bugs...

Anyway: I now program mostly in perl, which seems better for most quick 
and dirty things, but recently I made a neat perl program witch happens 
to be too slow for it's own good. Apart from that it works ok, but 
considering I have >2000 router interfaces to poll each 300 seconds and 
I take about 1 second per interface the speed is not tolerable.

So... I started to re-implement it in the mighty C, only to discover how 
difficult it is to make half complex and reliable programs in C.

The program uses the SNMP perl library, and a similar one exists in 
Lisp. Text munching seems a no brainer, since there are nice regex 
libraries for Lisp. SNMP-Tables munching are also not a concern. The 
problem seems to be mysql: I couldn't find anything equivalent to the 
DBI/DBD perl suite. Anyone knows about this? I need some kind of fast 
interface to mysql.

Also another problem: I can't find the Xemacs package to run cmulisp 
inside xemacs. Hemlock is not quite polished, and emacs... is not quite 
my style :-) I remember this neat thing for xemacs... where can I find it?

--
Rodrigo
From: Joe Marshall
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <r8fksic7.fsf@ccs.neu.edu>
=?ISO-8859-15?Q?Rodrigo_Nuno_Bragan=E7a_da_Cunha?=   <·······@vodafone.pt> writes:

> 
> So... I started to re-implement it in the mighty C, only to discover
> how difficult it is to make half complex and reliable programs in C.
> 
> 

I disagree.  It is far too easy to make half-reliable programs in C.
From: Bulent Murtezaoglu
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <878z1sk3pg.fsf@acm.org>
>>>>> "RNBdC" == Rodrigo Nuno Bragan <Rodrigo> writes:
[...]
    RNBdC> good. Apart from that it works ok, but considering I have
    RNBdC> 2000 router interfaces to poll each 300 seconds and I take
    RNBdC> about 1 second per interface the speed is not tolerable.

Are you sure that it is perl that's causing the slowness?  There are 
quite a few things that can go wrong with a program like that (network
misconfiguration, latency in the router SNMP software etc.).   Are you 
doing the probing sequentially?  If so, maybe you can come up with an 
event-based scheme to do the querying in parallel.

[...]
    RNBdC> Also another problem: I can't find the Xemacs package to
    RNBdC> run cmulisp inside xemacs. Hemlock is not quite polished,
    RNBdC> and emacs... is not quite my style :-) I remember this neat
    RNBdC> thing for xemacs... where can I find it?

That thing is called ILISP.  http://sourceforge.net/projects/ilisp/

cheers,

BM
From: Will Deakin
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <amnk8j$ouc$1@paris.btinternet.com>
Bulent Murtezaoglu wrote:
> That thing is called ILISP.  http://sourceforge.net/projects/ilisp/
It is also available as one of the xemacs packages...

:)w
From: Rodrigo Cunha
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <amog17$7jo4o$1@ID-160389.news.dfncis.de>
First of all: thanks to all the responders: they solved most of my problems.

Now about the lousy performance I have in my perl program:

The objective is to get variables containing in/out counters for each 
class of IP traffic (TOS) in a given interface.

Seems easy and fast, right? So cisco made sure the variables floated 
like the wind. To make things short: I have to read 3 entire snmp tables 
(a few dozen entries), make 3 inverse searches in them, obtain a 
"sintetic" index (they call it that way) and only than get my variables. 
5 minutes later the tos-index has probably changed.

Inverse hash-table searching is BAD/SLOW... especially in perl. Also 
there is the initialization time for the perl (object oriented) library: 
slow again, each time a pseudo-snmp-connection is made. The hash 
insertion, as used by the library, is also slow... Making another snmp 
library? No thanks :-)

I was trying it in C, but if I can do it in a language in witch I will 
actually understand my code in a few months it's much better, even if 
it's not as fast. I only need it about 25x faster than perl or so :-)

Compiling the perl program would (perhaps) be nice, but even perl 5.8.0 
crashes bigtime with non-toy programs.

 > Bulent Murtezaoglu wrote:
> Are you sure that it is perl that's causing the slowness?  There are 
> quite a few things that can go wrong with a program like that (network
> misconfiguration, latency in the router SNMP software etc.).   Are you 
> doing the probing sequentially?  If so, maybe you can come up with an 
> event-based scheme to do the querying in parallel.
From: Raymond Wiker
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <867khb3ltz.fsf@raw.grenland.fast.no>
Rodrigo Cunha <····@netcabo.pt> writes:

> Inverse hash-table searching is BAD/SLOW... especially in perl. Also
> there is the initialization time for the perl (object oriented)
> library: slow again, each time a pseudo-snmp-connection is made. The
> hash insertion, as used by the library, is also slow... Making another
> snmp library? No thanks :-)

        If you have to do reverse hash-table searches, you should
probably create reverse hash-tables :-)

-- 
Raymond Wiker                        Mail:  ·············@fast.no
Senior Software Engineer             Web:   http://www.fast.no/
Fast Search & Transfer ASA           Phone: +47 23 01 11 60
P.O. Box 1677 Vika                   Fax:   +47 35 54 87 99
NO-0120 Oslo, NORWAY                 Mob:   +47 48 01 11 60

Try FAST Search: http://alltheweb.com/
From: Rodrigo Nuno Bragança da Cunha
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <3d905de3$1@news.vizzavi.pt>
Ainsi parlait Raymond Wiker:
>         If you have to do reverse hash-table searches, you should
> probably create reverse hash-tables :-)

Yes, but you see, that's not how the library works: you read a table, 
and the table gets indexed by oid, not by the variables value, witch can 
even be of assorted types.
From: Marc Spitzer
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <Xns9293B715C95A9mspitze1optonlinenet@167.206.3.3>
Rodrigo Nuno Bragan�a da Cunha <·······@vodafone.pt> wrote in
···············@news.vizzavi.pt: 

> Ainsi parlait Raymond Wiker:
>>         If you have to do reverse hash-table searches, you should
>> probably create reverse hash-tables :-)
> 
> Yes, but you see, that's not how the library works: you read a table, 
> and the table gets indexed by oid, not by the variables value, witch
> can even be of assorted types.
> 
> 

Well if we are talking about snmp, I will assume so or the rest makes no
sence, take another look at your mibs.  Many mibs have convienence
tables which allow you to map a peace of client info to a magic number.
For example in the docsis mibs(cable modem stuff) you have a table that
maps the modem mac address to an integer(aka index number, last value in
the oid string).  This allows you to do get the specific info you want
instead of walking and discarding the table to get your info.  The
lookup table is not dome for all mibs, but it does exist for a lot of
them. 

good luck

marc
From: Rodrigo Cunha
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <amono4$7jjbd$1@ID-160389.news.dfncis.de>
Bulent Murtezaoglu wrote:
> doing the probing sequentially?  If so, maybe you can come up with an 
> event-based scheme to do the querying in parallel.

Yes, fork() is my friend: it I can get the program to be fast in itself 
forking a few instances is a no brainer. In fact that's what I'm doing 
in other cases. The problem with this one right now is the lousy 
computation time. I could probably solve that, at least partially, by 
rewriting the snmp & sockets perl library to perform better in this 
rather peculiar (god bless cisco) problem, but I'm not gona do that :-)
From: Thomas F. Burdick
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <xcv65wvxqle.fsf@hurricane.OCF.Berkeley.EDU>
Rodrigo Cunha <····@netcabo.pt> writes:

> Bulent Murtezaoglu wrote:
> > doing the probing sequentially?  If so, maybe you can come up with an 
> > event-based scheme to do the querying in parallel.
> 
> Yes, fork() is my friend: it I can get the program to be fast in itself 
> forking a few instances is a no brainer. In fact that's what I'm doing 
> in other cases. The problem with this one right now is the lousy 
> computation time. I could probably solve that, at least partially, by 
> rewriting the snmp & sockets perl library to perform better in this 
> rather peculiar (god bless cisco) problem, but I'm not gona do that :-)

I had a similar problem once, but instead of just the Perl SNMP
library, I also had to deal with thousands and thousands of lines of
Perl "OO" code that amounted to vermicelli code (like spaghetti, but
more tangled ;).  The overly-complicated system written by someone
else was taking too long to run -- and, being run as a cron job, it
was running into the next instance of itself and losing information
left and right.  For this system, complete with large amounts of
unnecessary computation in poorly-written Perl, using a fork-bomb
approach was good enough.  For all practical purposes, this would have
scaled infinitely.

Although I'm not suggesting you stick with Perl rather than switch to
Lisp -- but you might try seeing how much a controlled fork-bomb
approach can buy you in Perl.

-- 
           /|_     .-----------------------.                        
         ,'  .\  / | No to Imperialist war |                        
     ,--'    _,'   | Wage class war!       |                        
    /       /      `-----------------------'                        
   (   -.  |                               
   |     ) |                               
  (`-.  '--.)                              
   `. )----'                               
From: Bulent Murtezaoglu
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <87vg4vi9ky.fsf@acm.org>
>>>>> "TFB" == Thomas F Burdick <···@hurricane.OCF.Berkeley.EDU> writes:
[...]
    TFB> Although I'm not suggesting you stick with Perl rather than
    TFB> switch to Lisp -- but you might try seeing how much a
    TFB> controlled fork-bomb approach can buy you in Perl.

I dislike forking for things like this, especially if there'll be some
shared data.  I don't know what functionality is available in perl for
event driven network IO.  I occasionally use tcl for net+GUI scripting 
and event-based net IO works fine there (but no udp AFAIR, so probably 
not a good idea for UDP-based SNMP).

Anyhow, back on to the topic, maybe the path of least resistance thing
to do to avoid a complete rewrite is to just punt the computationally
expensive things to lisp by calling a lisp script (clisp?) and feeding
it the right data.  Pipes to/from a spawned process should work OK in 
perl AFAIK.

cheers,

BM
From: Thomas F. Burdick
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <xcvhegfkuh8.fsf@apocalypse.OCF.Berkeley.EDU>
Bulent Murtezaoglu <··@acm.org> writes:

> >>>>> "TFB" == Thomas F Burdick <···@hurricane.OCF.Berkeley.EDU> writes:
> [...]
>     TFB> Although I'm not suggesting you stick with Perl rather than
>     TFB> switch to Lisp -- but you might try seeing how much a
>     TFB> controlled fork-bomb approach can buy you in Perl.
> 
> I dislike forking for things like this, especially if there'll be some
> shared data.  I don't know what functionality is available in perl for
> event driven network IO.  I occasionally use tcl for net+GUI scripting 
> and event-based net IO works fine there (but no udp AFAIR, so probably 
> not a good idea for UDP-based SNMP).

In general, I wouldn't recommend forking just to fake asynch-io, but
in this case, where the Perl SNMP library works well enough, but is
synchronous, it's probably a good engineering decision.  In practice,
these processes will mostly wait for io, and FreeBSD can take hundreds
of them doing this simultaneously (I think I used 512), with no
problem.  Even using freeze/thaw for serialization to communicate back
to the parent process, and even assuming a fair number of dead boxes,
on the system I did this on, this approach was going to scale to an
absurd number of boxes that it could query.

-- 
           /|_     .-----------------------.                        
         ,'  .\  / | No to Imperialist war |                        
     ,--'    _,'   | Wage class war!       |                        
    /       /      `-----------------------'                        
   (   -.  |                               
   |     ) |                               
  (`-.  '--.)                              
   `. )----'                               
From: Marc Spitzer
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <Xns9293B5348E9E6mspitze1optonlinenet@167.206.3.3>
Bulent Murtezaoglu <··@acm.org> wrote in ···················@acm.org:
 
> I dislike forking for things like this, especially if there'll be some
> shared data.  I don't know what functionality is available in perl for
> event driven network IO.  I occasionally use tcl for net+GUI scripting
> and event-based net IO works fine there (but no udp AFAIR, so probably
> not a good idea for UDP-based SNMP).
> 

tcl has a great SNMP library called scotty.  It also has a whole bunch 
of stuff in addition to snmp there: dns, icmp, nis(I think).  I just 
wish it had SNMPv3 also.  

marc

ps I used to be ····@oscar.eng.cv.net

marc

pps

anybody need some remote SA work done?
From: Rodrigo Cunha
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <amqsch$8d7pj$1@ID-160389.news.dfncis.de>
Thomas F. Burdick wrote:
> Although I'm not suggesting you stick with Perl rather than switch to
> Lisp -- but you might try seeing how much a controlled fork-bomb
> approach can buy you in Perl.

Been there, done that -> Most of the time the process is using CPU, as I 
said.
From: Rodrigo Cunha
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <amqsto$892ib$1@ID-160389.news.dfncis.de>
BTW, here goes the code...

#!/usr/local/bin/perl -w
#use diagnostics;
#use strict;
(my $seconds0, my $useconds0)=gettimeofday();


#get_qos_policy_counters000.pl cr2-alf 213.30.25.213 RT

use Net::SNMP qw( oid_lex_sort );
#use Env qw( ARGV );			#Para aceder ?s vari?veis de ambiente
use Time::HiRes qw( usleep ualarm gettimeofday tv_interval );
use POSIX "sys_wait_h";

if ($#ARGV < 0) {
	print "Parametros Incorrectos!\n";
	exit(1);
}

#print $ARGV[0],"\n";


my $session;
my $error;
($session, $error) = Net::SNMP->session(
	-hostname  => $ARGV[0],
	-version   => 'SNMPv2c',
	-community => 'yadayada',
	-timeout   => 2,
	-retries   => 5,
	-port      => 161,
	-debug     => 0,
	-translate => 0
	);

#my $tabela_Ifs = $session->get_table(-baseoid => '.1.3.6.1.2.1.2.2.1.2');
# RFC1213-MIB::ifDescr

#Neste caso o Argv[1] ? o IP da interface a monitorizar.
my $tabela_IfsIp = $session->get_request('.1.3.6.1.2.1.4.20.1.2.'.$ARGV[1]);
# RFC1213-MIB::ipAdEntIfIndex
my $ifIndex=$tabela_IfsIp->{'.1.3.6.1.2.1.4.20.1.2.'.$ARGV[1]};

my $tabela_QosIfs = $session->get_table(-baseoid => '.1.3.6.1.4.1.9.9.166.1.1.1.1.4');
# CISCO-CLASS-BASED-QOS-MIB::cbQosIfIndex
my $interface;
my @aux0;
my $virtual_ifIndex;
foreach $interface (keys(%{$tabela_QosIfs})) {
	#print $interface," ===== ",$tabela_QosIfs->{$interface};
	@aux0=split(/\./, $interface);
	#print " ===== ",$aux0[$#aux0],"\n";
	if ($tabela_QosIfs->{$interface} == $ifIndex) {
		$virtual_ifIndex=$aux0[$#aux0];
		#print "Our virtual interface index is ".$virtual_ifIndex."\n";
		last;
	}
}

my $virtual_QosIndex;
my $qosmap;
my $tabela_QosClassMap = $session->get_table(-baseoid => '.1.3.6.1.4.1.9.9.166.1.7.1.1.1');
# CISCO-CLASS-BASED-QOS-MIB::cbQosCMName
foreach $qosmap (keys(%{$tabela_QosClassMap})) {
	#print $qosmap," ===== ",$tabela_QosClassMap->{$qosmap};
	@aux0=split(/\./, $qosmap);
	#print " ===== ",$aux0[$#aux0],"\n";
	if ($tabela_QosClassMap->{$qosmap} eq $ARGV[2]) {
		$virtual_QosIndex=$aux0[$#aux0];
		#print "Our virtual qosmap index is ".$virtual_QosIndex."\n";
		last;
	}
}


my $virtual_QosIfIndex;
my $virtualqos;
my $tabela_QosConfig = $session->get_table(-baseoid => '.1.3.6.1.4.1.9.9.166.1.5.1.1.2.'.$virtual_ifIndex);
# CISCO-CLASS-BASED-QOS-MIB::cbQosConfigIndex.CISCO-CLASS-BASED-QOS-MIB::cbQosIfIndex
foreach $virtualqos (keys(%{$tabela_QosConfig})) {
	#print $virtualqos," ===== ",$tabela_QosConfig->{$virtualqos};
	@aux0=split(/\./, $virtualqos);
	#print " ===== ",$aux0[$#aux0],"\n";
	if ($tabela_QosConfig->{$virtualqos} == $virtual_QosIndex) {
		$virtual_QosIfIndex=$aux0[$#aux0];
		#print "Our virtual qosif index is ".$virtual_QosIfIndex."\n";
		last;
	}
}

my $tabela_contadores = $session->get_request(
			'.1.3.6.1.4.1.9.9.166.1.15.1.1.10.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex,
			#cbQosCMPostPolicyByte64 "The 64 bits count of outbound octets after executing QoS policies."
			'.1.3.6.1.4.1.9.9.166.1.15.1.1.17.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex
			#cbQosCMDropByte64 "The 64 bits counter of dropped bytes per class as the result of all features that can produce drops (e.g., police, random detect, etc.)."
			);
#print "Bytes Sent ===== ",
#	'.1.3.6.1.4.1.9.9.166.1.15.1.1.10.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex.' ===== ',
#	$tabela_contadores->{'.1.3.6.1.4.1.9.9.166.1.15.1.1.10.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex},
#	"\n";
#print "Bytes Drop ===== ",
#	'.1.3.6.1.4.1.9.9.166.1.15.1.1.17.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex.' ===== ',
#	$tabela_contadores->{'.1.3.6.1.4.1.9.9.166.1.15.1.1.17.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex},
#	"\n";

print $tabela_contadores->{'.1.3.6.1.4.1.9.9.166.1.15.1.1.10.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex},"\n";
print $tabela_contadores->{'.1.3.6.1.4.1.9.9.166.1.15.1.1.17.'.$virtual_ifIndex.'.'.$virtual_QosIfIndex},"\n";
print time(),"\n";
(my $seconds1, my $useconds1)=gettimeofday();
printf("comment %d.%.6d-%d.%.6d\n",
	$seconds1,
	$useconds1,
	$seconds0,
	$useconds0
	);

exit(0);
From: Rodrigo Cunha
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <amqte1$8eho0$1@ID-160389.news.dfncis.de>
Rodrigo Cunha wrote:
> BTW, here goes the code...

As it is you can use it as an external MRTG probe.

Another version exists witch inserts the values into a database, but 
that one sufficiently ilustrates the problems :-)
From: Bill Clementson
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <wku1kgy2ya.fsf@attbi.com>
Rodrigo Nuno Bragan�a da Cunha <·······@vodafone.pt> writes:

> The program uses the SNMP perl library, and a similar one exists in
> Lisp. Text munching seems a no brainer, since there are nice regex
> libraries for Lisp. SNMP-Tables munching are also not a concern. The
> problem seems to be mysql: I couldn't find anything equivalent to the
> DBI/DBD perl suite. Anyone knows about this? I need some kind of fast
> interface to mysql.

I don't know what the DBI/DBD perl suite does; however, there are a
number of different SQL packages for CL. See the CLiki page:
http://ww.telent.net/cliki/Database

> Also another problem: I can't find the Xemacs package to run cmulisp
> inside xemacs. Hemlock is not quite polished, and emacs... is not
> quite my style :-) I remember this neat thing for xemacs... where can
> I find it?

You're probably thinking of ILISP:
http://sourceforge.net/projects/ilisp/

--
Bill Clementson
From: synthespian
Subject: Re: mysql + cmulisp + xemacs + ...
Date: 
Message-ID: <pan.2002.09.24.02.45.30.237408.20476@debian-rs.org>
On Mon, 23 Sep 2002 14:19:36 -0300, Rodrigo Nuno Bragan�a da Cunha wrote:

> Hi!
> 
> 
> 
> Also another problem: I can't find the Xemacs package to run cmulisp
> inside xemacs. Hemlock is not quite polished, and emacs... is not quite
> my style :-) I remember this neat thing for xemacs... where can I find
> it?
> 
> --
> Rodrigo
Hi Rodrigo --

	Install it using xemacs package installation feature.
	If you use Debian, there's also a .deb package for Emacs

	Regs
	Henry
 


_________________________________________________________________
Micro$oft-Free Human         100% Debian GNU/Linux
     KMFMS              "Bring the genome to the people!
		···········@debian-rs.org
www.debian.org - www.debian-br.cipsga.org.br - www.debian-rs.org
comp.lang.lisp / 2002 / sep