Hello group,
I have a question regarding lisp memory footprint - in particular,
that of SBCL.
When I simply start SBCL, room reports the following:
* (room nil)
Dynamic space usage is: 24,563,128 bytes.
Read-only space usage is: 1,912 bytes.
Static space usage is: 1,936 bytes.
Control stack usage is: 1,244 bytes.
Binding stack usage is: 328 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.
while ps -u victor -o pid,rss,command | grep sbcl reports the
following:
21820 4296 sbcl
21822 528 grep sbcl
- I don't know by ps thinks that sbcl is using just 4M while sbcl
itself thinks it uses 24M, but so far, so good
When I load some _necessary_ package needed for web development e.g.
Hunchentoot, CL-WHO, CL-PPCRE and CLSQL - YMMV of course - the usage
jumps to 50M:
sbcl --load test.lisp --eval '(gc :full t)' --eval '(room nil)'
where test.lisp is just
(require 'asdf)
(require 'asdf-install)
(require 'sb-md5)
(asdf:oos 'asdf:load-op 'clsql)
(asdf:oos 'asdf:load-op 'clsql-postgresql)
(asdf:oos 'asdf:load-op 'cl-ppcre)
(asdf:oos 'asdf:load-op 'hunchentoot)
(asdf:oos 'asdf:load-op 'cl-who)
(use-package 'hunchentoot)
(use-package 'cl-who)
(use-package 'clsql)
prints
Dynamic space usage is: 44,402,560 bytes.
Read-only space usage is: 1,912 bytes.
Static space usage is: 2,512 bytes.
Control stack usage is: 732 bytes.
Binding stack usage is: 104 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.
which is consistence with ps output. Again, extra 20M of overhead for
having necessary libraries is OK.
Now the interesting thing happen - when I defined the following
function:
(defun check-if-robot-p (row)
"Check if it's robot's visit"
(let ((url (fourth row))
(browser (third row)))
(or (cl-ppcre:scan "\\.php($|\\?)" url)
(cl-ppcre:scan "livejournal.13gb.com" url)
(cl-ppcre:scan "http://canjo.net" url)
(cl-ppcre:scan "/ljimages" url)
(cl-ppcre:scan "webcollage" browser)
)))
memory usage jumps by 30M !!
Dynamic space usage is: 75,696,072 bytes.
Read-only space usage is: 1,912 bytes.
Static space usage is: 2,512 bytes.
Control stack usage is: 1,244 bytes.
Binding stack usage is: 328 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.
(of course, I called (gc :full t) right after function definition).
Now, this is something that I'm not happy with - my small project
consists just of two dozen functions, and the memory footprint is ~
100M right after application start, even before it's starts processing
any data - not a good thing if you're hosted on a shared hosting with
memory limitations.
Now, I know the 'business' answer to this: memory is cheap, get your
own server, get a better hosting option etc. (or even go ahead and
purchase a commercial lisp) - and still I'm wondering what's going on
here and why defining a function leads to such memory usage.
Any points would be really appreciated,
Regards,
Victor.
PS. My system is linux, SBCL 1.0.3.38.
On Mar 27, 7:10 pm, Edi Weitz <········@agharta.de> wrote:
> On 27 Mar 2007 16:23:42 -0700, "Victor Kryukov" <··············@gmail.com> wrote:
>
> > Now the interesting thing happen - when I defined the following
> > function:
>
> > (defun check-if-robot-p (row)
> > "Check if it's robot's visit"
> > (let ((url (fourth row))
> > (browser (third row)))
> > (or (cl-ppcre:scan "\\.php($|\\?)" url)
> > (cl-ppcre:scan "livejournal.13gb.com" url)
> > (cl-ppcre:scan "http://canjo.net" url)
> > (cl-ppcre:scan "/ljimages" url)
> > (cl-ppcre:scan "webcollage" browser)
> > )))
>
> > memory usage jumps by 30M !!
>
> Try (SETQ CL-PPCRE:*USE-BMH-MATCHERS* NIL) before you define the
> function.
>
> http://weitz.de/cl-ppcre/#use-bmh-matchers
>
> HTH,
> Edi.
>
> --
>
> Lisp is not dead, it just smells funny.
>
> Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Thank you Edi - that was exactly the root of the problem. Once I did
it, the memory usage is back to 44M for the whole application. I could
never imagined that reckless regexp usage can lead to that kind of
memory usage.
BTW, thank you very much for your libraries - I must admit that
Hunchentoot is the most elegant and easy-to-use web framework among
those that I've tried - which includes TurboGears, perl CGI and
Portable AllegroServe.
Best wishes,
Victor.
"Victor Kryukov" <··············@gmail.com> writes:
> BTW, thank you very much for your libraries - I must admit that
> Hunchentoot is the most elegant and easy-to-use web framework among
> those that I've tried - which includes TurboGears, perl CGI and
> Portable AllegroServe.
I have to second that. I'm basing my work on Hunchentoot and cl-who
and both are excellent. Thank you!
On 27 Mar 2007 17:37:59 -0700, "Victor Kryukov" <··············@gmail.com> wrote:
> Thank you Edi - that was exactly the root of the problem. Once I did
> it, the memory usage is back to 44M for the whole application. I
> could never imagined that reckless regexp usage can lead to that
> kind of memory usage.
It's not really reckless regexp usage on your side, but rather
reckless optimization on my side... :)
The reason this gets so big on SBCL is that CHAR-CODE-LIMIT on
Unicode-enabled SBCL is 1114112 which doesn't work very well with
Boyer-Moore-Horspool matching. So, you should either switch it off
completely or (if you think you need the optimization) play with this
variable:
http://weitz.de/cl-ppcre/#regex-char-code-limit
> BTW, thank you very much for your libraries - I must admit that
> Hunchentoot is the most elegant and easy-to-use web framework among
> those that I've tried - which includes TurboGears, perl CGI and
> Portable AllegroServe.
Thanks... :)
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")