Hello all,
I have inherited the administration of a web-enabled rental checkout
software package. It uses Apache, PostgreSQL, and Common LISP 3.0.8
(I believe this is correct.). This is running on a 2 GHz Pentium 4
with 1 GB of ram and a 35 GB scsi hd and runs on Debian with the
2.2.20 kernel. We have aprox. 10 users spread across 3 buildings about
3 or 5 blocks apart who are constantly using the system.
Anyway I was talking to the software company's tech support about why
this system bogs down. It can take 5 seconds or more from the moment
our barcode scanner scans a barcode before it returns the product name
and goes on to the next line of the checkout form. I was told by them
it was because their version of LISP was "serial". Not knowing
anything about LISP, I asked him if he meant that LISP was not
multi-threading and he said the version they are using is not
multi-threading. So my basic understanding is that when our 10 users
all scan a product (a typical order can have up to 200 items) 9 of the
users will have to wait for the lisp process to execute. Thus the
reason why our users are waiting 5 seconds for the system to process
their scan. Does this sound right?
What surprises me from reading this newsgroup is I thought all LISP
programs are compiled and that it is not an interpreted language, but
when I run the top command I see one lisp routine running for the last
32 hours, taking up 14.6% of memory and at times using 35% of the CPU.
Assuming they wrote 5 functions and compiled each separately
shouldn't we be seeing 5 programs running in top instead of one LISP
program?
Any insight on LISP would be greatly appreciated. This slow down has
been a problem since they bought the program.
Jason
From: Peter Seibel
Subject: Re: Newbie Question: LISP seems to be running very slow
Date:
Message-ID: <m3u1cr5zet.fsf@javamonkey.com>
····@jasonmartinez.com (Jason) writes:
> What surprises me from reading this newsgroup is I thought all LISP
> programs are compiled and that it is not an interpreted language,
> but when I run the top command I see one lisp routine running for
> the last 32 hours, taking up 14.6% of memory and at times using 35%
> of the CPU. Assuming they wrote 5 functions and compiled each
> separately shouldn't we be seeing 5 programs running in top instead
> of one LISP program?
It is the case that many Lisp implementations compile to native code.
But that is not the same as creating a standalone executable, which is
what--reading between the lines--I think you mean by "compiled Each
separately". In the Lisp world it's possible to compile on function at
a time (what a Lisper would mean if they said "compiled each
separately") but they're all part of the same Lisp process.
I'm sure all this Lisp stuff seems a bit strange if it just got dumped
in your lap--just be aware that some of the terminology and even
concepts are quite different between Lisp and other languages you may
be used to. So if you ask a question here and people tell you you
aren't making sense or *their* answers don't make any sense to you, be
aware that we may (on both sides) be tripping over either a
terminological difference or one of the areas where the concepts are
just different.
-Peter
--
Peter Seibel ·····@javamonkey.com
The intellectual level needed for system design is in general
grossly underestimated. I am convinced more than ever that this
type of work is very difficult and that every effort to do it with
other than the best people is doomed to either failure or moderate
success at enormous expense. --Edsger Dijkstra
From: Kenny Tilton
Subject: Re: Newbie Question: LISP seems to be running very slow
Date:
Message-ID: <3EA4709E.6020504@nyc.rr.com>
Jason wrote:
> What surprises me from reading this newsgroup is I thought all LISP
> programs are compiled and that it is not an interpreted language, but...
this does not sound like compiled-vs-interepreted, this sounds like a
single server process is handling all interesting work (such as
processing a scan) and that the developers did not think to hand off the
heavy-lifting to a subprocess so the server process could continue to
provide reasonable responsiveness to other users.
possibly this is a classic case of a design that seems OK in unit
testing but has a flaw which appears only when one attempts to scale it
to more users. i say that because you are asking here instead of talking
to tech support, who apparently have nothing to offer you.
have they offered to fix the problem? my hypothesis that there are no
bad lisp programmers is in danger.
> Any insight on LISP would be greatly appreciated.
This sounds more like a simple design mistake unrelated to Lisp. Any
application (in any language, whether it supports MP or not) can spawn a
subprocess or even communicate with a background process, and hand off
to them long-running tasks. this is a good idea even for single-user
apps--let me scan three things and then make a beer run while the system
works on the scans.
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
Kenny Tilton <·······@nyc.rr.com> writes:
> This sounds more like a simple design mistake unrelated to Lisp.
That's assuming that the operating system and hardware are up to the
task and are working properly. Without more data, it's not really
possible to say exactly what's happening.
For example, is it *really* a five minute wait for a process to run
through? Is there just one function of the application where this
happens? Are we talking about a lack of speed or too much latency
here? Is the Lisp the bottleneck, or is it the database? Is the
database running on an ATA disk?
We really need more data.
--
Matt Curtin, CISSP, IAM, INTP. Keywords: Lisp, Unix, Internet, INFOSEC.
Founder, Interhack Corporation +1 614 545 HACK http://web.interhack.com/
Author of /Developing Trust: Online Privacy and Security/ (Apress, 2001)
From: Kenny Tilton
Subject: Re: Newbie Question: LISP seems to be running very slow
Date:
Message-ID: <3EA5F164.3040907@nyc.rr.com>
Matt Curtin wrote:
> Kenny Tilton <·······@nyc.rr.com> writes:
>
>
>>This sounds more like a simple design mistake unrelated to Lisp.
>
>
> That's assuming ....
You did not give sufficient weight to "sounds more like".
> We really need more data.
Not really, we just need to crack the OP's confidence that the problem
is Lisp-related. If we can manage that, they will then investigate with
a more open mind.
But don't forget, the OP was:
> Anyway I was talking to the software company's tech support about why
> this system bogs down. It can take 5 seconds or more from the moment
> our barcode scanner scans a barcode before it returns the product name
> and goes on to the next line of the checkout form. I was told by them
> it was because their version of LISP was "serial".
Sounds like tech support was not aghast at the five second (not minute,
btw) pause. And tech support seems to have had no workaround /of any
kind/ to offer the OP. If so, that knocks down most of your suggestions
for further research; 5sec is as good as it gets.
ok, right, one assumption /was/ implicit in what I wrote: that the OS
offered multiple processes and IPC.
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
Jason wrote:
> Hello all,
>
> I have inherited the administration of a web-enabled rental checkout
> software package. It uses Apache, PostgreSQL, and Common LISP 3.0.8
> (I believe this is correct.).
There is no "Common LISP 3.0.8". You probably mean CMUCL 3.0.8 as packaged
with Debian.
> This is running on a 2 GHz Pentium 4
> with 1 GB of ram and a 35 GB scsi hd and runs on Debian with the
> 2.2.20 kernel. We have aprox. 10 users spread across 3 buildings about
> 3 or 5 blocks apart who are constantly using the system.
>
> Anyway I was talking to the software company's tech support about why
> this system bogs down. It can take 5 seconds or more from the moment
> our barcode scanner scans a barcode before it returns the product name
> and goes on to the next line of the checkout form. I was told by them
> it was because their version of LISP was "serial". Not knowing
> anything about LISP, I asked him if he meant that LISP was not
> multi-threading and he said the version they are using is not
> multi-threading. So my basic understanding is that when our 10 users
> all scan a product (a typical order can have up to 200 items) 9 of the
> users will have to wait for the lisp process to execute. Thus the
> reason why our users are waiting 5 seconds for the system to process
> their scan. Does this sound right?
It sounds plausible. Another problem is that even if the system supports
multi-threading the whole lisp-process might block when it calls out to a
C library via the foreign function interface.
A solution might be to have a lisp-process per user of this system. I don't
think there will be any intercommunication between the users in the system
so they would not necessarily have to be in the same process.
> What surprises me from reading this newsgroup is I thought all LISP
> programs are compiled and that it is not an interpreted language, but
> when I run the top command I see one lisp routine running for the last
> 32 hours, taking up 14.6% of memory and at times using 35% of the CPU.
> Assuming they wrote 5 functions and compiled each separately
> shouldn't we be seeing 5 programs running in top instead of one LISP
> program?
Hm... I don't know how you come to this conclusion.
Most Common Lisp implementations compile to native code - including CMUCL.
Why should you see 5 programs when there are 5 functions in the code?
> Any insight on LISP would be greatly appreciated. This slow down has
> been a problem since they bought the program.
Sidenote: Nowadays its called "Lisp" not "LISP".
ciao,
Jochen
Jochen Schmidt <···@dataheaven.de> writes:
> Jason wrote:
>
>> multi-threading. So my basic understanding is that when our 10 users
>> all scan a product (a typical order can have up to 200 items) 9 of the
>> users will have to wait for the lisp process to execute. Thus the
>> reason why our users are waiting 5 seconds for the system to process
>> their scan. Does this sound right?
Note though that if it takes n seconds of cpu to process a scan, if
you only have the one CPU it will still take 10n seconds to process
all of the scans, no matter whether you do them one at a time or you
multitask between them.
> A solution might be to have a lisp-process per user of this system. I don't
> think there will be any intercommunication between the users in the system
> so they would not necessarily have to be in the same process.
I'm guessing this system uses UncommonSQL (on the basis that it sounds
like the Onshore WebCheckout system). I would be kind of reluctant to
have multiple UncommonSQL instances doing transactions on the same
Postgres database, unless I were fairly well acquainted with the usql
code, because of the potential for having one process miss an update
to its cached data made by another process.
>> Any insight on LISP would be greatly appreciated. This slow down has
>> been a problem since they bought the program.
My experience with UncommonSQL has been that the speed of database
queries is more likely a bottleneck than the speed of Lisp. Try some
Postgres tuning first - e.g. changing the shared buffer cache size, if
you haven't looked at that yet.
-dan
--
http://www.cliki.net/ - Link farm for free CL-on-Unix resources
Daniel Barlow wrote:
> Jochen Schmidt <···@dataheaven.de> writes:
>
>> Jason wrote:
>>
>>> multi-threading. So my basic understanding is that when our 10 users
>>> all scan a product (a typical order can have up to 200 items) 9 of the
>>> users will have to wait for the lisp process to execute. Thus the
>>> reason why our users are waiting 5 seconds for the system to process
>>> their scan. Does this sound right?
>
> Note though that if it takes n seconds of cpu to process a scan, if
> you only have the one CPU it will still take 10n seconds to process
> all of the scans, no matter whether you do them one at a time or you
> multitask between them.
>
>> A solution might be to have a lisp-process per user of this system. I
>> don't think there will be any intercommunication between the users in the
>> system so they would not necessarily have to be in the same process.
>
> I'm guessing this system uses UncommonSQL (on the basis that it sounds
> like the Onshore WebCheckout system). I would be kind of reluctant to
> have multiple UncommonSQL instances doing transactions on the same
> Postgres database, unless I were fairly well acquainted with the usql
> code, because of the potential for having one process miss an update
> to its cached data made by another process.
Yes thats true. If I understood your point right then I tried to address
this with the handwaving statement "no intercommunication between users". I
think having multiple USQL instances should work well if they simply have
to write the scanned data into the database without reading any already
stored data. So missing an update would then not be a problem.
I sometimes thougth something like Postgres' NOTIFY and LISTEN might be an
interesting thing to include in USQL. AFAIR ORACLE supports events sent over
the database too.
ciao,
Jochen
From: Kenny Tilton
Subject: Re: Newbie Question: LISP seems to be running very slow
Date:
Message-ID: <3EA551E8.60206@nyc.rr.com>
Alain Picard wrote:
> I'm sure that multiple connections
> to the same Postgresql back end do the right thing vis-a-vis seeing
> consistent snapshots of the DB if all writes are done within a transaction.
Having come out in favor of processing the scans in the background, the
above reminds me that a multi-user DB can get bogged down with thrashing
when multiple users kick off transactions simultaneously, so possibly
the background scan processor should itself process tasks serially.
otoh, I agree (with someone) that possibly the scan processing involves
nothing but writing out new data, so... try it and see?
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
Daniel Barlow <···@telent.net> writes:
> I'm guessing this system uses UncommonSQL (on the basis that it sounds
> like the Onshore WebCheckout system). I would be kind of reluctant to
> have multiple UncommonSQL instances doing transactions on the same
> Postgres database, unless I were fairly well acquainted with the usql
> code, because of the potential for having one process miss an update
> to its cached data made by another process.
USQL provides in-memory transaction management, via the ODCL
transaction manager. USQL used to have it's own transaction manager
but it has been abstracted out and made general enough to support
multiple frameworks subjugating their transaction mgmt to it. By
"in-memory transaction management" I mean that CLOS instances in
memory which are mapped into some storage system, SQL or otherwise,
can be edited in a transaction, failed edits rolled back, written to
DB on commit. The reason for this is that the app driving USQL and
ODCL development uses several different storage subsystems presently,
filesystem, USQL OO->RDBMS, and a custom index mechanism which stores
its data in SQL. All of these hook into the ODCL transaction manager.
I think it is safe to assume that a useful feature for USQL in a
multi-process context is that modifications are shared across
processes, with proper transaction synchronization. One basic
building block of this is an event bus or some other mechanism of
syncing the transaction managers in the various processes. It would
need to provide the following:
1. Protocol for locking objects that are edited in one transaction so
they cannot be edited in the other. This is of course optional, it
could be a "last commit sticks" mecahnism, or perhaps fail upon
write to changed object mechanism.
2. Cache mgmt protocol for flushing cached instances in the other
processes which have been modified. This cache mgmt mechanism
needs to be general enough to support various storage subsystems.
3. Transaction manager lifecycle control, registering and
deregistering with the other managers.
ODCL recently acquired a event/messaging system which I hope will
provide the substrate upon which these protocols are built at some
point in the future. With those capabilities in place, ODCL could
provide storage subsystem agnostic transaction control across multiple
processes.
--
Sincerely, Craig Brozefsky <·····@red-bean.com>
No war! No racist scapegoating! No attacks on civil liberties!
Chicago Coalition Against War & Racism: www.chicagoantiwar.org
Free Scheme/Lisp Software: http://www.red-bean.com/~craig
From: Eduardo Muñoz
Subject: Re: Newbie Question: LISP seems to be running very slow
Date:
Message-ID: <uy923sewf.fsf@terra.es>
····@jasonmartinez.com (Jason) writes:
> What surprises me from reading this newsgroup is I thought all LISP
> programs are compiled and that it is not an interpreted language, but
> when I run the top command I see one lisp routine running for the last ...
This is extrange. Are you sure that the code is compiled? I
have a setup just like yours (debian+apache+cmucl+postgresql)
running on a 120 MHz Pentium and serves simple pages in less
than one second. If the machine is new maybe someone forgot
to compile the code before loading.
--
Eduardo Mu�oz | (prog () 10 (print "Hello world!")
http://213.97.131.125/ | 20 (go 10))
"Jason" <····@jasonmartinez.com> wrote in message
·································@posting.google.com...
> Hello all,
>
> I have inherited the administration of a web-enabled rental checkout
> software package. It uses Apache, PostgreSQL, and Common LISP 3.0.8
> (I believe this is correct.). This is running on a 2 GHz Pentium 4
> with 1 GB of ram and a 35 GB scsi hd and runs on Debian with the
> 2.2.20 kernel. We have aprox. 10 users spread across 3 buildings about
> 3 or 5 blocks apart who are constantly using the system.
Was has Apache to do with your application? Do the terminals use
HTTP to communicate with some kind of bar-code/price server? If so,
how does Apache communicate with the CL app? Is that Apache to
CL app protocol the serial bottleneck you are talking about?
Wade