From: rjf
Subject: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <f85c5ae9-b4c1-45d4-a4a0-4dcada4ad609@r35g2000prm.googlegroups.com>
in http://www.cs.berkeley.edu/~fateman/lisp/bughash.lisp

is a file that executes correctly for small tests on Allegro CL
version 8.0, SBCL 1.0.13 and GCL 2.6.8
on Windows (XP)/ Intel Pentium D.

For a large test it runs fine on Allegro.
On GCL it crashes somehow, silently.  (Not sure how to debug this. I
loaded it into Maxima, and it just stops.)
On SBCL it says fatal error, some gc problem.

To run the test, compile and load the file, and look at the comments
at the end.

The program, in case you care, multiplies sparse polynomials using a
hash table to collect terms that should be added together.  Yes, it is
far more elaborate than necessary, but it runs faster than simpler
versions, on some inputs.

If this is of interest to you, I hope you will look at it.

RJF

From: D Herring
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <MIqdnb3EO81y3zfVnZ2dnUVZ_gydnZ2d@comcast.com>
rjf wrote:
> in http://www.cs.berkeley.edu/~fateman/lisp/bughash.lisp

I took only a cursory look at your file, but
- comment out the defvar x and y (what!?! no *earmuffs*?!?)
- redefine the test
(defun test(n m &optional (d 5)) ;try these parameters for now.
   (let ((x (make-racg n d))
         (y (make-racg m d)))
     (length (car  (mulhash1 x y))) ;; some number returned
     ))

gets rid of the crash without changing the results of the first three 
tests.

No idea whether it changes other functionality significantly.

- Daniel
From: Richard J. Fateman
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <g8f2vq$mhi$1@agate.berkeley.edu>
Thanks!  Juho's comment was exactly right; Daniel's fix is probably 
accidental. (I was reading beyond the end of an array, but never looked 
at the bogus values... maybe the GC did though!)

The reason I had no *earmuffs* is that I was typing x and y in 
repeatedly for tests, and didn't want to type *x* and *y* so much.
But I agree stylistically it's preferable.

No need to look further.

Anecdote -- when I was first learning Lisp in 1966 at MIT, I found that 
programs could be debugged by emptying my bladder.

  How? ...It meant leaving a program listing by a computer terminal (one 
of maybe 6 or so in a big room on the 9th floor of Project MAC/AI lab, a 
room which housed a PDP-6 too) for a few minutes.  When I returned, some 
anonymous hacker would have gone over to "my" terminal, picked up the 
listing, and scribbled a bug fix on the paper.  Considering that many 
listings had no comments...

Thanks again.
RJF


D Herring wrote:
> rjf wrote:
>> in http://www.cs.berkeley.edu/~fateman/lisp/bughash.lisp
> 
> I took only a cursory look at your file, but
> - comment out the defvar x and y (what!?! no *earmuffs*?!?)
> - redefine the test
> (defun test(n m &optional (d 5)) ;try these parameters for now.
>   (let ((x (make-racg n d))
>         (y (make-racg m d)))
>     (length (car  (mulhash1 x y))) ;; some number returned
>     ))
> 
> gets rid of the crash without changing the results of the first three 
> tests.
> 
> No idea whether it changes other functionality significantly.
> 
> - Daniel
From: Rob Warnock
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <TZSdnV2I5obS_DDVnZ2dnUVZ_sTinZ2d@speakeasy.net>
Richard J. Fateman <·······@eecs.berkeley.edu> wrote:
+---------------
| The reason I had no *earmuffs* is that I was typing x and y in 
| repeatedly for tests, and didn't want to type *x* and *y* so much.
+---------------

Use DEFLEX to get (well, simulate) top-level lexical variables:

    http://rpw3.org/hacks/lisp/deflex.lisp

[There are lots of other variations on the theme; that's just
the one I wrote & use.]

Then you can define all the short REPL variables you want without
accidentally rebinding global specials, e.g.:

    > (deflex foo 23)

    FOO
    > (defun foo () foo)

    FOO
    > (defvar bar 91)     ; UNSAFE!! Should be named *BAR*

    BAR
    > (defun bar () bar)

    BAR
    > (let ((foo 37)
            (bar 103))
        (list foo (foo) bar (bar)))

    (37 23 103 103)
    > 

Does that help?


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: ·············@gmail.com
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <46fd771f-dafc-412e-9a40-ae5a4605b0f9@j22g2000hsf.googlegroups.com>
On Aug 21, 8:36 am, ····@rpw3.org (Rob Warnock) wrote:
> Richard J. Fateman <·······@eecs.berkeley.edu> wrote:
> +---------------
> | The reason I had no *earmuffs* is that I was typing x and y in
> | repeatedly for tests, and didn't want to type *x* and *y* so much.
> +---------------
>
> Use DEFLEX to get (well, simulate) top-level lexical variables:
>
>    http://rpw3.org/hacks/lisp/deflex.lisp
>
> [There are lots of other variations on the theme; that's just
> the one I wrote & use.]
>
> Then you can define all the short REPL variables you want without
> accidentally rebinding global specials, e.g.:
>
>     > (deflex foo 23)
>
>     FOO
>     > (defun foo () foo)
>
>     FOO
>     > (defvar bar 91)     ; UNSAFE!! Should be named *BAR*
>
>     BAR
>     > (defun bar () bar)
>
>     BAR
>     > (let ((foo 37)
>             (bar 103))
>         (list foo (foo) bar (bar)))
>
>     (37 23 103 103)
>     >
>
> Does that help?
>
> -Rob
>
> -----
> Rob Warnock                     <····@rpw3.org>
> 627 26th Avenue                 <URL:http://rpw3.org/>
> San Mateo, CA 94403             (650)572-2607

Rob, this is really nitpicking, (and possibly wrong), but I am
wondering if in your deflex gem one could do

CL-USER> (defmacro deflex2 (var val &optional (doc nil docp))
  (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
	 (s1 (symbol-name var))
	 (s2 (symbol-name '#:*))
	 (s3 (symbol-package var))	; BUGFIX [see above]
	 (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
    `(prog2
	 (defparameter ,backing-var ,val ,(when docp `,doc))
	 (define-symbol-macro ,var ,backing-var)
        ,(when docp
		 `(setf (documentation ',var 'variable) ,doc)))))

My only problem (i.e. lack of macro expertise) with this is that the
when forms instead of returning nothing place a NIL in the macro
expansion.  But that does not seem to hurt things.

Mirko
From: Rob Warnock
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <K4idnUgy9ofwGDDVnZ2dnUVZ_v3inZ2d@speakeasy.net>
<·············@gmail.com> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) wrote:
| > Richard J. Fateman <·······@eecs.berkeley.edu> wrote:
| > +---------------
| > | The reason I had no *earmuffs* is that I was typing x and y in
| > | repeatedly for tests, and didn't want to type *x* and *y* so much.
| > +---------------
| >
| > Use DEFLEX to get (well, simulate) top-level lexical variables:
| >    http://rpw3.org/hacks/lisp/deflex.lisp
...
| Rob, this is really nitpicking, (and possibly wrong), but I am
| wondering if in your deflex gem one could do
| 
| CL-USER> (defmacro deflex2 (var val &optional (doc nil docp))
|   (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
| 	 (s1 (symbol-name var))
| 	 (s2 (symbol-name '#:*))
| 	 (s3 (symbol-package var))	; BUGFIX [see above]
| 	 (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
|     `(prog2
| 	 (defparameter ,backing-var ,val ,(when docp `,doc))
| 	 (define-symbol-macro ,var ,backing-var)
|         ,(when docp
| 		 `(setf (documentation ',var 'variable) ,doc)))))
| 
| My only problem (i.e. lack of macro expertise) with this is that the
| when forms instead of returning nothing place a NIL in the macro
| expansion.  But that does not seem to hurt things.
+---------------

The idea of pushing the IF into the body of the PROGN is certainly
reasonable, and I've done it both ways at various times, but I left
the IF outside in the <http://rpw3.org/hacks/lisp/deflex.lisp> version
because I thought it would be a bit clearer to the newcomer.

However, since you brought it up...  ;-}  ;-}

First, my plain ,DOC gives identical results as your ,(WHEN DOCP `,DOC)
in the DEFPARAMETER's third arg. And neither solves the problem of the
third arg being always provided regardless of the value of DOC.[1]
Fixing that requires using the same splicing hack used below for
the (SETF (DOCUMENTATION ...)) clause.

Anyway, to the main point: if I were to push the IF down in the PROGN,
I would do it the following way instead, which uses a bit more magic
macro syntax but doesn't require the out-of-order thinking that your PROG2
introduced (and also doesn't leave orphan NILs anywhere in the code):

    (defmacro deflex (var val &optional (doc nil docp))    
      (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
	     (s1 (symbol-name var))
	     (s2 (symbol-name '#:*))
	     (s3 (symbol-package var))
	     (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
	`(progn
	   (defparameter ,backing-var ,val ,@(when doc (list doc)))
	   ,@(when docp
	       (list `(setf (documentation ',var 'variable) ,doc)))
	   ;; Must be last, so the value of the form is the symbol VAR.
	   (define-symbol-macro ,var ,backing-var))))

Yes, this version is *slightly* more compact than my original, but
also perhaps a bit harder to figure out.

Note that this splicing trick is quite useful for both optional and
keyword parameters. But you *must* use the LIST and not just QUOTE
the optional value, that is, write ,@(WHEN DOC (LIST DOC)) and not
just ,@(WHEN DOC '(DOC)), since the latter can result in constant
structure being modified, which is a big no-no. Whereas LIST will
allocate a fresh cons that can be cheerfully spliced.

Back to you...


-Rob

[1] For maximum portability one should probably use the splicing hack
    on the third arg of DEFPARAMETER, too, since the CLHS defines that arg
    to be "a string", *not* "a string or NIL". Fortunately, this doesn't
    seem to matter in all the CLs I have access to -- (DEFPARAMETER FOO val)
    gives the same result as (DEFPARAMETER FOO val NIL). But I included that fix in the above
    version, too.

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: ·············@gmail.com
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <31c58427-0222-4c01-9d9a-86b287147da6@m3g2000hsc.googlegroups.com>
On Aug 21, 11:10 am, ····@rpw3.org (Rob Warnock) wrote:
> <·············@gmail.com> wrote:
>
> +---------------
> | ····@rpw3.org (Rob Warnock) wrote:
> | > Richard J. Fateman <·······@eecs.berkeley.edu> wrote:
> | > +---------------
> | > | The reason I had no *earmuffs* is that I was typing x and y in
> | > | repeatedly for tests, and didn't want to type *x* and *y* so much.
> | > +---------------
> | >
> | > Use DEFLEX to get (well, simulate) top-level lexical variables:
> | >    http://rpw3.org/hacks/lisp/deflex.lisp
> ...
> | Rob, this is really nitpicking, (and possibly wrong), but I am
> | wondering if in your deflex gem one could do
> |
> | CL-USER> (defmacro deflex2 (var val &optional (doc nil docp))
> |   (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
> |        (s1 (symbol-name var))
> |        (s2 (symbol-name '#:*))
> |        (s3 (symbol-package var))      ; BUGFIX [see above]
> |        (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
> |     `(prog2
> |        (defparameter ,backing-var ,val ,(when docp `,doc))
> |        (define-symbol-macro ,var ,backing-var)
> |         ,(when docp
> |                `(setf (documentation ',var 'variable) ,doc)))))
> |
> | My only problem (i.e. lack of macro expertise) with this is that the
> | when forms instead of returning nothing place a NIL in the macro
> | expansion.  But that does not seem to hurt things.
> +---------------
>
> The idea of pushing the IF into the body of the PROGN is certainly
> reasonable, and I've done it both ways at various times, but I left
> the IF outside in the <http://rpw3.org/hacks/lisp/deflex.lisp> version
> because I thought it would be a bit clearer to the newcomer.
>
> However, since you brought it up...  ;-}  ;-}
>
> First, my plain ,DOC gives identical results as your ,(WHEN DOCP `,DOC)
> in the DEFPARAMETER's third arg. And neither solves the problem of the
> third arg being always provided regardless of the value of DOC.[1]
> Fixing that requires using the same splicing hack used below for
> the (SETF (DOCUMENTATION ...)) clause.
>
> Anyway, to the main point: if I were to push the IF down in the PROGN,
> I would do it the following way instead, which uses a bit more magic
> macro syntax but doesn't require the out-of-order thinking that your PROG2
> introduced (and also doesn't leave orphan NILs anywhere in the code):
>
>     (defmacro deflex (var val &optional (doc nil docp))
>       (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
>              (s1 (symbol-name var))
>              (s2 (symbol-name '#:*))
>              (s3 (symbol-package var))
>              (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
>         `(progn
>            (defparameter ,backing-var ,val ,@(when doc (list doc)))
>            ,@(when docp
>                (list `(setf (documentation ',var 'variable) ,doc)))
>            ;; Must be last, so the value of the form is the symbol VAR.
>            (define-symbol-macro ,var ,backing-var))))
>
> Yes, this version is *slightly* more compact than my original, but
> also perhaps a bit harder to figure out.
>
> Note that this splicing trick is quite useful for both optional and
> keyword parameters. But you *must* use the LIST and not just QUOTE
> the optional value, that is, write ,@(WHEN DOC (LIST DOC)) and not
> just ,@(WHEN DOC '(DOC)), since the latter can result in constant
> structure being modified, which is a big no-no. Whereas LIST will
> allocate a fresh cons that can be cheerfully spliced.
>
> Back to you...
>
> -Rob
>
> [1] For maximum portability one should probably use the splicing hack
>     on the third arg of DEFPARAMETER, too, since the CLHS defines that arg
>     to be "a string", *not* "a string or NIL". Fortunately, this doesn't
>     seem to matter in all the CLs I have access to -- (DEFPARAMETER FOO val)
>     gives the same result as (DEFPARAMETER FOO val NIL). But I included that fix in the above
>     version, too.
>
> -----
> Rob Warnock                     <····@rpw3.org>
> 627 26th Avenue                 <URL:http://rpw3.org/>
> San Mateo, CA 94403            (650)572-2607

Well, your approach not to complicate things too much for newbies is
much appreciated :-)

And thanks for the splicing pointer.

Mirko (still newbie)
From: Juho Snellman
Subject: Re: large hashtable crashes SBCL and GCL
Date: 
Message-ID: <87abf93d90.fsf@vasara.proghammer.com>
rjf <·······@gmail.com> writes:
> in http://www.cs.berkeley.edu/~fateman/lisp/bughash.lisp
> 
> is a file that executes correctly for small tests on Allegro CL
> version 8.0, SBCL 1.0.13 and GCL 2.6.8
> on Windows (XP)/ Intel Pentium D.
> 
> For a large test it runs fine on Allegro.
> On GCL it crashes somehow, silently.  (Not sure how to debug this. I
> loaded it into Maxima, and it just stops.)
> On SBCL it says fatal error, some gc problem.

You know, the proper first step when your code compiled in unsafe mode
crashes is not reporting compiler bug :-) It's compiling the code in
safe mode. At least SBCL with safety and debug 3 would immediately
pinpoint you to the place where you're doing an out of bounds array
access (the code being (aref zcoefs m)).

-- 
Juho Snellman