I'm a big fan of regression testing. I develop as much as possible in
Common Lisp, and I wonder if anyone has come to the same problem I
have:
When developing with regression testing, there is a slight problem
(easily overcome with discipline) that arises due to developing code
in a running lisp image. It is best illustrated in an example:
Let's say I have the following code:
(defun foo ()
....)
(defun bar ()
...
(foo)
....)
Both of these functions have good, complete tests, and they pass them
all.
Now, I want to change the name of the function foo. It now looks like
this:
(defun hoo ()
...)
Now I run my unit tests, and they still pass, even though the tests
still refer to function foo, as does function bar.
What's more frightening is that I can now modify and break function
hoo, see all of my tests pass, and think that everything is fine.
The function foo still exists in the running lisp image. The unit
tests will continue to pass until I restart the lisp image, reload all
of the code, and then run the unit tests. If I don't do this, I won't
know what will happen when I reload the image (say, if the machine
goes down and needs a reboot).
Has anyone come up with a good method for making sure this will
happen? Does anyone worry that the code files and the function
definitions in memory get out of sync without you being aware?
I really appreciate the fact that lisp does provide an interactive
environment, but it has bitten me before. The main problem as I see
it: there is duplication of information -- textual code in files/
editor window, and compiled code in the lisp environment.
Eric
http://www.lispcast.com
············@gmail.com writes:
> The function foo still exists in the running lisp image. The unit
> tests will continue to pass until I restart the lisp image, reload all
> of the code, and then run the unit tests. If I don't do this, I won't
> know what will happen when I reload the image (say, if the machine
> goes down and needs a reboot).
Well, to be really sure that everything builds and runs correctly, your
really do need to start up a new lisp image. Fortunately, that
shouldn't take too long to do. As long as it doesn't take forever to
actually load your code, the best solution is to restart before doing
any serious regression testing.
--
Thomas A. Russ, USC/Information Sciences Institute
From: Richard M Kreuter
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <87sl408cyp.fsf@progn.net>
············@gmail.com writes:
> Let's say I have the following code:
>
> (defun foo ()
> ....)
>
> (defun bar ()
> ...
> (foo)
> ....)
>
> Both of these functions have good, complete tests, and they pass them
> all.
>
> Now, I want to change the name of the function foo. It now looks like
> this:
>
> (defun hoo ()
> ...)
>
> Now I run my unit tests, and they still pass, even though the tests
> still refer to function foo, as does function bar.
When you change the name of a function, you can FMAKUNBOUND the old
name. But note that if FOO was inlined in BAR, then code that tests
BAR probably won't detect that FOO isn't fbound anymore.
> I really appreciate the fact that lisp does provide an interactive
> environment, but it has bitten me before. The main problem as I see
> it: there is duplication of information -- textual code in files/
> editor window, and compiled code in the lisp environment.
Lisp supports an interactive style, but it doesn't preclude batchy
operation; you can periodically build your projects and run your tests
in a fresh Lisp instance.
--
RmK
············@gmail.com wrote:
> I'm a big fan of regression testing. I develop as much as possible in
> Common Lisp, and I wonder if anyone has come to the same problem I
> have:
>
> When developing with regression testing, there is a slight problem
> (easily overcome with discipline) that arises due to developing code
> in a running lisp image. It is best illustrated in an example:
>
> Let's say I have the following code:
>
> (defun foo ()
> ....)
>
> (defun bar ()
> ...
> (foo)
> ....)
>
> Both of these functions have good, complete tests, and they pass them
> all.
>
> Now, I want to change the name of the function foo. It now looks like
> this:
>
> (defun hoo ()
> ...)
>
> Now I run my unit tests, and they still pass, even though the tests
> still refer to function foo, as does function bar.
That is not a Lisp problem, you were supposed to search your source for
foo and change it to hoo at the moment you decided to make the name change.
>
> What's more frightening is that I can now modify and break function
> hoo, see all of my tests pass, and think that everything is fine.
What's more frightening is that you changed /none/ of your code to use
hoo but are already revising it and rerunning tests to check the output
and are not worried at all that the output is not changing.
Are you making this all up?
If hoo does not change internally such that you cannot tell foo is
running, then running foo by mistake is fine. If you change hoo and are
not bothered that the program behavior does not change, again, do not
blame Lisp.
If you ar enot making this up, then you just need to spend more time in
a dynamic environment until you change hoo, the program behavior does
not change, you are astonsihed that it did not change (this is the piece
you are missing), and then (thanks to experience) you realize, oh, that
old code is still running. Then bounce your Lisp instead of trying to
guess all the places you forgot to change or all the methods you need to
delete or all the symbols you need to unbind.
kt
--
http://www.theoryyalgebra.com/
"Career highlights? I had two. I got an intentional walk
from Sandy Koufax and I got out of a rundown against the Mets."."
- Bob Uecker
On Oct 24, 4:03 pm, ············@gmail.com wrote:
> Has anyone come up with a good method for making sure this will
> happen?
Discipline and practice. Like most others say, I will start up a new
Lisp image or delete the appropriate package(s), if necessary.
It's simpler to get into the habit of using FMAKUNBOUND and possibly
UNINTERN when you change a function name.
> Does anyone worry that the code files and the function
> definitions in memory get out of sync without you being aware?
>
No, because that was something I acknowledge as an inherent property
of working in an environment that evolves. And as Kent mentioned,
files are just a handy way to store Lisp objects when not using Lisp.
I don't think in terms of them when writing Lisp code.
Thank you all for your responses. I appreciate the time and thought
you've put into them.
Though I feel like, with your help, I am closer to the answer I'm
looking for, as is often the case with USENET posts, I think I need to
clarify my original question.
I know that I am *supposed to* make sure that I modify all of the
references to foo when I change it to hoo. But unit tests are
supposed to guard me against all of those things I'm supposed to do
but forget to do. The example is perhaps simple, but it makes sense
if you see that it's possible that bar is in another file (which I
apologize for not making clear in my original post), possibly in a
different package, and that I just plain forgot about it in the
hypothetical situation. I make mistakes all the time.
I am not making this example up. I realize that it's a silly mistake
to make, and I apologize for what you may consider a silly question,
but I thought people with more experience than me would have more
insight than I have had into how to resolve the issue, and that I
might learn something from them. And I did: from these posts, I
realize that most people deal with it by keeping track of it all in
their heads, deleting the package in general or unbinding the symbol-
function, or else they reload the lisp image.
I want to make it clear that I do not blame Lisp for how it works.
Many people much smarter than me developed it the way it is for
reasons I perhaps will never fathom. I just want to fix a problem
with my code that I inadvertently created, and prevent myself from
making further silly mistakes.
It may come as a surprise to you, but I already do keep track of what
functions have been loaded, and how it differs from the text files I'm
editing. I realize that posts to newsgroups can sometimes lead to
arguments, misunderstandings, etc. However, I keep reviewing my post
to see what indications I gave that would make anyone speak in defense
of Common Lisp (which I truly love -- I go to bed thinking about how
Common Lisp is doing in the world). I don't see any insult or blame,
or any implications of such, in my original post, though I take all
responsibility for any perceived transgressions.
I was asking about the issue of one of the fundamental assumptions of
Regression Testing: that they do a relatively thorough job of allowing
changes to the code which do not alter the correctness of the code,
while alerting you to changes that do alter the correctness of the
code. Lisp is different from most other languages, in that it
maintains a dynamic environment, which exists apart from the files of
code inputted to it. There is therefore a breach of the Regression
Testing assumption. I want to rectify that, in some way. Being
human, I want as automated and as foolproof a way to do that as I can
get, while maintaining the joyous power that Common Lisp affords. I
was wondering if there was some mechanism *besides* the obvious and
error-prone you-just-have-to-deal-with-it approach. I am very much
for incremental improvement -- which means that I look for
opportunities and possibilities for improvement *despite* there
existing a de facto solution, and despite opposition from those who
have invested in their own ways.
Like I said before, I love Common Lisp. Without going into details, I
bliss out thinking about the potential for such a great language. I
just want to develop what Common Lisp already does really well, so
that it does more even better.
I hope I've given a proper introduction, which will allow me to
continue:
Some constraints:
* Unit tests are meant to test the *code*, and so must test a clean
lisp environment with only the files loaded (or some equivalent
approach)
* Unit tests are meant to be automated -- to prevent error and to
encourage running them often, so they must be easy to run (one
command, button, etc). Automated unit testing has improved many a
developer's life
I have, with all of your help, come up with a few solutions to the
above constraints:
* Create a shell script which loads all of the files of the
application into a new lisp image, then runs the tests, prints out the
results, then exits
Pros:
+ simple to implement
+ automated
Cons:
- loading some code/tests may take a long time
* In the file containing the tests, add code to delete and reload the
package containing the code to be tested as well code to run the
tests. A simple recompile of that file will flush the symbols from
that package, reload the symbols of that package from its code file,
and run the tests.
Example:
Say you have a file unit-tests.lisp which tests code in
package :package-to-test
unit-tests.lisp would contain
;; flush package
(eval-when (:compile-toplevel)
(delete-package :package-to-test)
(load "package-to-test")) ;; or some other
mechanism
;; unit tests here
.......
;; run tests
(run-tests)
Pros:
+ totally automated, pushbutton testing
Cons:
- Any? I don't know Lisp well enough to say
* Be sure to fmakunbound and/or uninter symbols when you no longer
need them
Pros:
+ will catch problems when you remember it
Cons:
- requires a lot of discipline
- if you forget it, you've broken the constraints of the
problem
Do you have any more solutions, or comments on the ones I've provided?
Thanks for you time.
Eric Normand
http://www.lispcast.com
From: Kent M Pitman
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <utzofseh9.fsf@nhplace.com>
············@gmail.com writes:
> I know that I am *supposed to* make sure that I modify all of the
> references to foo when I change it to hoo. But unit tests are
> supposed to guard me against all of those things I'm supposed to do
> but forget to do.
This is the first source of confusion.
Unit tests are programs. Nothing more, nothing less. They can, therefore
have bugs.
If they see observed behavior and it's what they expect, that doesn't
make the program right; the program is right only if (a) the program
is in fact right and (b) the unit test is ALSO right (i.e., is
observing the rightness of the program's behavior and correctly
opining on that rightness). It is still possible for the program to
be wrong if both the program is wrong and the unit test is wrong.
If they see observed behavior and it's what they don't expect, that
doesn't make the program wrong; the program is wrong only if (a) the
program is in fact wrong and (b) the unit test is right (i.e., is
observing the wrongness of the program's behavior and correctly
opining on that wrongness). It is still possible for the program to
be right if both the program is right and the unit test is wrong.
Unit tests are nothing more than an extra chance to notice a problem.
They are not magic. They do not operate by magic. You are still
obliged to write programs that are correct.
For example, consider a c# program that contains
public class MyApp {
public int m_foo = 17;
public int Foo { get { return m_foo; } set { m_foo = value; } }
public void DoStuff() { m_foo++ }
}
and a "unit test" in another file that does, effectively
static bool testfoo() { return m_SomeApp.Foo == 17 }
where m_SomeApp is bound to a MyApp of somewhat vague origin that
you'd like to test.
This program might seem in some ways like a valid unit test because
you expect the unit test to be run at startup time in lieu of running
the application. And, very often, unit tests are done by first
instantiating a new instance of the class being tested. So startup
time would really mean at the time the app was initialized by 'new'.
So if you've managed to call m_SomeApp.DoStuff(), then testfoo()
will report failure. Meanwhile if you have
static bool testfoo2() { return m_SomeApp.Foo == 18 }
in your quiver, testfoo2() will have been reporting failure all along,
but will suddenly start reporting success... for a while. until you
do m_SomeApp.DoStuff() again.
So this comes back to the "somewhat vague origin". If the program has
been running for a while, the damage may be done. If the unit test,
for example, is given to an app that has been legally running for a
while, as for a test that is activated by pressing a "Run Unit Tests"
command in the application, it will do weird stuff if the unit tests
are not written extra carefully.
It is for this reason that most unit tests are done on freshly-built
objects. That sounds clean and pretty and makes a lot of unit tests
not fail. But it doesn't mean much for the methodology of unit
testing--it says that if you test all your weapons back home at the
factory when you're milling them, they'll work well under jungle
conditions.
This didn't work well for US soldiers using the M-16 rifle in Viet Nam
if I recall... or maybe it was he M-16. I did a few Google searches
and couldn't find out. Presumably I'm on some watch list now, though,
just for trying to be conversationally colorful. Sigh. Anyway, my
point was it worked well when deployed conditions are just like
factory conditions... but the deployed situation wasn't the factory
situation. And that meant, in practice, it jammed. So who is right?
The factory or the world?
> The example is perhaps simple, but it makes sense
> if you see that it's possible that bar is in another file (which I
> apologize for not making clear in my original post), possibly in a
> different package, and that I just plain forgot about it in the
> hypothetical situation. I make mistakes all the time.
I don't think anyone reading along didn't assume this could happen.
The real misunderstanding you have is that there's anything wrong in
this circumstance, with requiring you to exit Lisp and reload your
application if you need this strong a check.
Going back to my example above, if what you're trying to test is
initial setup, the you have no substitute but to do it initially.
Unless you write your unit test program to account for things that may
have been going on in the app, you've not written your unit test program
correctly.
> I am not making this example up. I realize that it's a silly mistake
> to make, and I apologize for what you may consider a silly question,
> but I thought people with more experience than me would have more
> insight than I have had into how to resolve the issue, and that I
> might learn something from them. And I did: from these posts, I
> realize that
The verb you want here is "infer that". To "realize", properly used,
means "x is true and I have inferred x". To "infer" does not imply
the truth of the world.
> most people deal with it by keeping track of it all in their heads,
I didn't read this fact into what people said. What I head was that
most people don't create the circumstance of this change and then expect
unit testing to recognize them.
Some people responding probably don't do unit tests. That doesn't
require any keeping track of anything.
Some people responding may do unit tests at load-up time only. That
doesn't requrie any keeping track of anything.
Some people responding do unit tests at other times, but keep a single
bit in their head ("have I done anything that is suspicious at all in
this image that would make running unit tests not work here"). That's
not exactly keeping track of "it all" since (a) a single bit is not
usually called "all" and (b) there are more things to know than this
one bit.
A few people may keep track of specific state such as you suggest.
But the one unifying thing I heard people say was that if you're
simply going to do changes, then run unit tests, then assume you have
no affirmative obligation to restart lisp and try it in a fresh
environment, you're going to lose. That also requires no "keeping
track of anything in your head". It's a static, unvarying truth.
> deleting the package in general or unbinding the symbol-
> function,
Yes, we said this. However, to be clear, I don't know that everyone was
recommending it. I wasn't. I was merely answering the "how" question.
I don't generally ever recommend calling these from user code (including
typing them by hand) because the effects are so devastating. It's a bit
like thinking it's ok to write on your hand with a permanent marker to take
notes in class and knowing you can always use a scalpel to erase and that
your skin will grow back. It's harsh and works well if you're keeping your
immune system up to date, but if you make a mistake, the damage you can do
is way worse than just doing things right in the first place.
Some environments [e.g., LispWorks] offer something like Meta-X Undefine
so you can undefine a definition before changing it. Incidentally, this
isn't a matter of keeping track of state. It's a matter of good hygiene
when making a name change. (Hygiene is all about NOT having to remember,
NOT about keeping track of a lot of stuff, as you characterized.)
> or else they reload the lisp image.
If they want to know what the program would do initially. (See my
rifle example above.) And that answers some questions. But the real
thing you want to do is to understand that your unit tests have limitations
and one of them is that if you're going to interactively change definitions,
you can't assume that looking for certain definitions
> I want to make it clear that I do not blame Lisp for how it works.
> Many people much smarter than me developed it the way it is for
> reasons I perhaps will never fathom.
Only if you don't try. The reasons are powerful, but not intellectually
deep. You will easily come to understand the reasons if you simply use
the language for what it's intended for and don't try to make it be what
it is not.
If you think Unit Test implies "newly instantiated, factory-pure instance",
then you want to be reloading Lisp.
If you think "rename" means "I get to load a new definition and tell no one
and nothing that secretly it is a redefinition instead of just a new
definition", that counts as not newly instantiated, just in case there is
some question.
> I just want to fix a problem with my code that I inadvertently
> created,
Did someone say something that keeps you from fixing your code?
I think they only said that fixing one bug doesn't imply (a) that you
have fixed others and (b) that you haven't introduced others.
If the use of foo rather than hoo is a bug, then you have the bug in two
places. If you fix one of them, you haven't fixed the problem.
So your question reduces to "can I interactively be sure that when I fix
one thing, the system will magically know that I teleported back in time
and will run programs as if I'd never run the programs I have". The answer
is no. Reloading lisp is the closest you get to that--and is not a bad
solution if that's really what you want.
> and prevent myself from making further silly mistakes.
I suspect the term "prevent" is what has gotten people up in arms here.
No technology will prevent making mistakes. If you start talking in terms
of reduced probabilities, people can help you. If you think of this as
a science, they probably cannot.
> It may come as a surprise to you, but I already do keep track of what
> functions have been loaded, and how it differs from the text files I'm
> editing. I realize that posts to newsgroups can sometimes lead to
> arguments, misunderstandings, etc. However, I keep reviewing my post
> to see what indications I gave that would make anyone speak in defense
> of Common Lisp (which I truly love -- I go to bed thinking about how
> Common Lisp is doing in the world).
You're using absolutist statements that imply things you may not mean to
imply. When you say, as above,
| I just want to fix a problem with my code that I inadvertently created,
it seems to imply someone is keeping you from doing it. They are not.
When you say, as above,
| and prevent myself from making further silly mistakes.
it seems to imply that this is a valid goal. It is not.
So when you appear to be saying you have valid goals and you appear to be
saying the goals are not being achieved, the reason people are confused into
thinking you are saying something about the system is that you are failing
to acknowledge that you might have a confusion. You have fixed your goal
in a way that doesn't seem to allow for the idea that unit tests have to
be realistic in scope.
> I don't see any insult or blame,
> or any implications of such, in my original post, though I take all
> responsibility for any perceived transgressions.
And no intent here on my part to say you should feel bad. My remarks are
just trying to help you see why people get confused.
> I was asking about the issue of one of the fundamental assumptions of
> Regression Testing: that they do a relatively thorough job of allowing
> changes to the code which do not alter the correctness of the code,
> while alerting you to changes that do alter the correctness of the
> code.
Incidentally, I use regression tests a lot in my Lisp code and I have
a lot of things to say on this subject. I just don't plan to say it
here. I may say something about this at another time on another day.
Sorry for being vague, but that's all I can be at this hour.
> Lisp is different from most other languages, in that it
> maintains a dynamic environment, which exists apart from the files of
> code inputted to it. There is therefore a breach of the Regression
> Testing assumption.
I don't think this is so. Rather, I think that there is a failure to
see all the assumptions that go into some tests, even in non-dynamic
languages. I think you're overstating what they do.
I would not use "relatively thorough" because it gives a misleading sense
that you can characterize their coverage completely, for example. How about
Regression Testing is generally better than no testing when making
changes a lot because it increases the chance that if a problem occurs
it will be reported.
That's a much more neutral statement and is as true of Lisp as of other
languages. Now the question of what's testable is different between
languages, but there's nothing magic about Regression Testing.
> I want to rectify that, in some way. Being
> human, I want as automated and as foolproof a way to do that as I can
> get, while maintaining the joyous power that Common Lisp affords.
Freedom and foolproofness are in tension with one another.
Cue not only Ben Franklin with his recently-much-quoted "They who
would give up an essential liberty for temporary security, deserve
neither liberty or security." but also see:
www.wired.com/wired/archive/1.04/gibson.html
So there is a balance.
> I was wondering if there was some mechanism *besides* the obvious and
> error-prone you-just-have-to-deal-with-it approach.
There is no substitute for writing correct programs is how we in the
peanut gallery would say it. Unit tests are not something you just whip out.
If you are writing a program that appears to assess the correct execution
of something, you had better understand what that is testing and know what
the limitations of those tests are. It's easy to write things like my
testfoo() example above, but that doesn't mean that it can or should be
written by someone with just that much programming skill. The skill of
knowing that's the right test is quite a bit higher.
A manager asking "have you written/run your unit tests" may have a
very different model in his head about what he's asking than in fact
what is implemented, and it's your ethical burden to understand and
report on not only the results but the significance of the results. I
doubt nearly as many programmers do that latter part as merely write
the tests.
> I am very much
> for incremental improvement -- which means that I look for
> opportunities and possibilities for improvement *despite* there
> existing a de facto solution, and despite opposition from those who
> have invested in their own ways.
>
> Like I said before, I love Common Lisp. Without going into details, I
> bliss out thinking about the potential for such a great language. I
> just want to develop what Common Lisp already does really well, so
> that it does more even better.
Good to see the optimism here. Hope I have not trounced it by being
detailed in my response.
> I hope I've given a proper introduction, which will allow me to
> continue:
>
> Some constraints:
>
> * Unit tests are meant to test the *code*, and so must test a clean
> lisp environment with only the files loaded (or some equivalent
> approach)
The unqualified use of "unit tests" continues to disturb me.
Do you mean
* I'd like to write unit tests to test the *code*, and so to accomplish
this I plan to write in a lisp environment with only the files loaded
(or some equivalent approach)
OR
* Any program worthy of the venerable name Unit Test must test the code,
and so by nature must, if the are to continue to hold this title,
run in a clean lisp environment with only the files laoded.
There are probably shades of gray between. But I'm guessing you mean
the former, even though the text you wrote can be read as the latter
and will get extreme pushback by people if they think you are owning
the term (in just the way "free" gets in trouble in "free software",
when many advocates don't say "this is my meaning" but instead assert
"all people must mean this if they dare to use this term").
But assuming you meant the first choice (or something like it), then
I think the problem is that you're limiting what your tests can do,
and what you infer from them. You're basically resigning yourself to
never running them at other times because you're assuming the data is bad
and you can't trust the results.
A better conclusion would be:
If I run the tests in ANY image, clean or dirty, and they pass, that
isn't proof that things are ok, but at least it means I didn't find
anything new that was a problem; I could have still found problems
worth looking at, and I might still find problems at another time.
> * Unit tests are meant to be automated -- to prevent error and to
> encourage running them often, so they must be easy to run (one
> command, button, etc). Automated unit testing has improved many a
> developer's life
Again, do you mean "my unit tests" or "anything worthy of the name".
Sure, it's useful to automate them, but it's not a requirement of the name.
The reason the word "automated unit tests" has three words is linguistically
to allow other modifiers than "automated" to still refer to "unit tests".
If you make the three word and two word versions be synonyms, then you get
to where to talk about non-automated or semi-automated, you have to say
"things that would be unit tests except they aren't automated". That seems
uselessly clumsy.
Moreover, this isn't a constraint, it's either a goal statement or a
political manifesto.
> I have, with all of your help, come up with a few solutions to the
> above constraints:
>
> * Create a shell script which loads all of the files of the
> application into a new lisp image, then runs the tests, prints out the
> results, then exits
> Pros:
> + simple to implement
> + automated
> Cons:
> - loading some code/tests may take a long time
Isn't every detail of this, implementation and pro and con, precisely what
is routinely done in other languages?
> * In the file containing the tests, add code to delete and reload the
> package containing the code to be tested as well code to run the
> tests.
Make this be separable.
> A simple recompile of that file will flush the symbols from
> that package, reload the symbols of that package from its code file,
> and run the tests.
> Example:
> Say you have a file unit-tests.lisp which tests code in
> package :package-to-test
> unit-tests.lisp would contain
> ;; flush package
> (eval-when (:compile-toplevel)
> (delete-package :package-to-test)
> (load "package-to-test")) ;; or some other
> mechanism
> ;; unit tests here
> .......
>
> ;; run tests
> (run-tests)
>
> Pros:
> + totally automated, pushbutton testing
> Cons:
> - Any? I don't know Lisp well enough to say
This destroys your environment. You're assuming that's healthy.
It's easy to write applications that are devastated by this.
What if I've done
(defvar *saved-state* (find-symbol "SOMETHING" "PACKAGE-TO-TEST"))
or even just
(defvar *saved-state-too* 'package-to-test:something)
upon loading my package in with yours? Doing DELETE-PACKAGE will forcefully
recreate the package but cannot go find other applications that have done
reflection and found and saved away pointers to your symbols. You will test
a new package, but that doesn't mean other packages will use the new package.
Same problem as before.
> * Be sure to fmakunbound and/or uninter symbols when you no longer
> need them
> Pros:
> + will catch problems when you remember it
> Cons:
> - requires a lot of discipline
> - if you forget it, you've broken the constraints of the
> problem
>
> Do you have any more solutions, or comments on the ones I've provided?
Your problem isn't techniques in the first place. Your problem is what
you are inferring from them. Follow instead this simple strategy:
Don't assume unit tests are foolproof. They have only unidirectional
implication, not a bidirectional one: When they tell you something is
the matter, investigate it. When they tell you all is well, assume
you know nothing.
Run them in as many different modes as you can stand so that they have
as many chances as you can of finding problems to look into.
From: Holger Schauer
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <yxzfxzy1ahi.fsf@gmx.de>
On 5167 September 1993, Kent M. Pitman wrote:
> Some environments [e.g., LispWorks] offer something like Meta-X Undefine
> so you can undefine a definition before changing it. Incidentally, this
> isn't a matter of keeping track of state. It's a matter of good hygiene
> when making a name change. (Hygiene is all about NOT having to remember,
> NOT about keeping track of a lot of stuff, as you characterized.)
Actually, I think that the main topic of the OP is not regression
testing but refactoring (here: "rename"). The crucial bit here is that
we're inherently constrained (as of today) to do refactoring not in
code (as in running code) but in text files. What the OP asks for is
"I want to refactor every use of 'foo' to 'hoo'", but all he can do is
load 'hoo' into the running image, which doesn't do anything to 'foo'
(actually, CL does offer a lot more here than most other languages, as
-- inlining set apart -- CL will notice that a function gets redefined
without *requiring* one to reload the entire code base).
I wonder if some (maybe commercial) vendor has some extension to allow
such refactorings; I think CL would be well suited to something like
it.
Holger
--
--- http://hillview.bugwriter.net/ ---
"Debian-Nazi!"
-- Aus "Wie starte ich einen Endlosthread ?", Teil 57
From: Kamen TOMOV
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <uk5padv89.fsf@cybuild.com>
On Fri, Oct 26 2007, Holger Schauer wrote:
> On 5167 September 1993, Kent M. Pitman wrote:
(decode-universal-time (+ (encode-universal-time 0 0 0 1 9 1993)
(hours-to-seconds (* 5167 24))))
Wow 5167 days after Sept 1st, 1993 are exactly Oct 25th, 2007 when KMP
send the letter (in terms of boring Gregorian calendar). Something
historical must had happened with you in September 1993 that made you
create your own calendar.
>> Some environments [e.g., LispWorks] offer something like Meta-X
>> Undefine so you can undefine a definition before changing it.
>> Incidentally, this isn't a matter of keeping track of state. It's
>> a matter of good hygiene when making a name change. (Hygiene is
>> all about NOT having to remember, NOT about keeping track of a lot
>> of stuff, as you characterized.)
>
> Actually, I think that the main topic of the OP is not regression
> testing but refactoring (here: "rename").
An excellent point!
> The crucial bit here is that we're inherently constrained (as of
> today) to do refactoring not in code (as in running code) but in
> text files.
Why do you think so? One can do refactoring while interacting with a
running image. Even I do so :)
> What the OP asks for is "I want to refactor every use of 'foo' to
> 'hoo'", but all he can do is load 'hoo' into the running image,
> which doesn't do anything to 'foo' (actually, CL does offer a lot
> more here than most other languages, as -- inlining set apart -- CL
> will notice that a function gets redefined without *requiring* one
> to reload the entire code base).
Therefore what he can do is:
1. Load 'hoo';
2. Modify and load the form that use 'foo' to use 'hoo';
3. Unload 'foo' if he wants;
Of course if there are more than one forms that use 'foo' one might
consider an extra step.
1.5
(defun foo (params)
(hoo params))
> I wonder if some (maybe commercial) vendor has some extension to
> allow such refactorings; I think CL would be well suited to
> something like it.
See above.
--
�����
From: Holger Schauer
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <yxzd4v212f5.fsf@gmx.de>
On 5168 September 1993, Kamen TOMOV wrote:
> On Fri, Oct 26 2007, Holger Schauer wrote:
> Wow 5167 days after Sept 1st, 1993 are exactly Oct 25th, 2007 when KMP
> send the letter (in terms of boring Gregorian calendar). Something
> historical must had happened with you in September 1993 that made you
> create your own calendar.
Search for "september that never ended".
>> The crucial bit here is that we're inherently constrained (as of
>> today) to do refactoring not in code (as in running code) but in
>> text files.
> Why do you think so? One can do refactoring while interacting with a
> running image. Even I do so :)
Not in the same sense as refactoring happens in some of todays more
nifty IDEs. In those you say "refactor function 'foo' to 'hoo'" and
/the IDE/ replaces (in the text files) all references to function
'foo' to 'hoo'.
> Therefore what he can do is:
> 1. Load 'hoo';
> 2. Modify and load the form that use 'foo' to use 'hoo';
> 3. Unload 'foo' if he wants;
> Of course if there are more than one forms that use 'foo' one might
> consider an extra step.
The point in having the system do this is that it's not you, the
programmer, which needs to know which forms to modify.
> (defun foo (params)
> (hoo params))
This comes closest to what one would want, yes. But of course, this
comes at the cost at actually *not* having refactored the calls to
'foo' (only one point out of several).
Holger
--
--- http://hillview.bugwriter.net/ ---
"Compulsive whiners are not welcome in any professional setting and I
fail to see how they are welcome in any personal company, either."
-- Erik Naggum in comp.lang.lisp
Holger Schauer <··············@gmx.de> writes:
> The point in having the system do this is that it's not you, the
> programmer, which needs to know which forms to modify.
I think that in some cases, it would be very hard for the system to
figure out how to do the renaming/refactoring, and it will depend on
the programmer to trigger it. A contrived example (badly written and
untested):
(defvar *number-names* '("ZEROE" "ONE" "TWO" "THREE"))
(defun make-number-functions (list)
(loop for name in list
for i from 1
do (let ((return-value i))
(setf (symbol-function (intern name))
#'(lambda () return-value)))))
(make-number-functions *number-names*)
Now I use these functions as defaults for optional arguments to other
functions. Eventually someone tells me that "zero" is spelled without
an "e". I change the string and re-run the make- function. It would
have to be a very smart system to figure out what to do with calls to
#'ZEROE.
Or maybe I'm off-base here.
From: Tobias C. Rittweiler
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <87myu5wyla.fsf@freebits.de>
·······@ncsu.edu (Rob St. Amant) writes:
> I think that in some cases, it would be very hard for the system to
> figure out how to do the renaming/refactoring, and it will depend on
> the programmer to trigger it. A contrived example (badly written and
> untested):
>
> (defvar *number-names* '("ZEROE" "ONE" "TWO" "THREE"))
>
> (defun make-number-functions (list)
> (loop for name in list
> for i from 1
> do (let ((return-value i))
> (setf (symbol-function (intern name))
> #'(lambda () return-value)))))
>
> (make-number-functions *number-names*)
>
> Now I use these functions as defaults for optional arguments to other
> functions. [...]
You do know about CONSTANTLY, don't you?
(If you do already, and the sole purpose of this fragment was arguing
for your point, please consider this to be directed to the silent
readership, Rob.)
-T.
"Tobias C. Rittweiler" <···@freebits.de.invalid> writes:
> ·······@ncsu.edu (Rob St. Amant) writes:
>
>> I think that in some cases, it would be very hard for the system to
>> figure out how to do the renaming/refactoring, and it will depend on
>> the programmer to trigger it. A contrived example (badly written and
>> untested):
>>
>> (defvar *number-names* '("ZEROE" "ONE" "TWO" "THREE"))
>>
>> (defun make-number-functions (list)
>> (loop for name in list
>> for i from 1
>> do (let ((return-value i))
>> (setf (symbol-function (intern name))
>> #'(lambda () return-value)))))
>>
>> (make-number-functions *number-names*)
>>
>> Now I use these functions as defaults for optional arguments to other
>> functions. [...]
>
> You do know about CONSTANTLY, don't you?
Actually, I didn't, and so while it wasn't my point (as you recognize
below), posting to cll is always a good learning experience. Thanks.
> (If you do already, and the sole purpose of this fragment was arguing
> for your point, please consider this to be directed to the silent
> readership, Rob.)
From: Kamen TOMOV
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <uabq6dnd2.fsf@cybuild.com>
On Fri, Oct 26 2007, Holger Schauer wrote:
> On 5168 September 1993, Kamen TOMOV wrote:
>> On Fri, Oct 26 2007, Holger Schauer wrote:
>> Wow 5167 days after Sept 1st, 1993 are exactly Oct 25th, 2007 when
>> KMP send the letter (in terms of boring Gregorian
>> calendar). Something historical must had happened with you in
>> September 1993 that made you create your own calendar.
>
> Search for "september that never ended".
Oh, OK.
>>> The crucial bit here is that we're inherently constrained (as of
>>> today) to do refactoring not in code (as in running code) but in
>>> text files.
>
>> Why do you think so? One can do refactoring while interacting with a
>> running image. Even I do so :)
>
> Not in the same sense as refactoring happens in some of todays more
> nifty IDEs. In those you say "refactor function 'foo' to 'hoo'" and
> /the IDE/ replaces (in the text files) all references to function
> 'foo' to 'hoo'.
I suppose these nifty IDEs work with text files and not in code (as
you defined the terms). Anyway, the Emacs/Slime IDE can also be used
for that purpose to a particular extend.
Hint: 'C-C, <' would find all the forms that use the symbol above the
cursor.
One can automate the modification and give me his code ;) Obviously we
need a running program if we want to use it. There are some risks
though, and they are with code that is not loaded yet but contains the
symbol in question. As far as this problem it's a point where a nifty
IDE can step in.
--
�����
Kamen TOMOV <·····@cybuild.com> writes:
> On Fri, Oct 26 2007, Holger Schauer wrote:
>
>> On 5167 September 1993, Kent M. Pitman wrote:
>
> (decode-universal-time (+ (encode-universal-time 0 0 0 1 9 1993)
> (hours-to-seconds (* 5167 24))))
>
> Wow 5167 days after Sept 1st, 1993 are exactly Oct 25th, 2007 when KMP
> send the letter (in terms of boring Gregorian calendar). Something
> historical must had happened with you in September 1993 that made you
> create your own calendar.
That's the September that never ended, back when AOL added Usenet. Some
claim that now that AOL has removed Usenet, September 1993 is over and
now we are in October 1993, which is expected to last until Google
Groups is no more.
--
Robert Uhl <http://public.xdi.org/=ruhl>
I remember when everybody posted to Usenet with their real, deliverable
e-mail address. Of all the sins committed by the spammers, destroying
the viability of the open Internet was the worst.
--Shmuel (Seymour J.) Metz
From: Kent M Pitman
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <uwst9frnl.fsf@nhplace.com>
Holger Schauer <··············@gmx.de> writes:
> Actually, I think that the main topic of the OP is not regression
> testing but refactoring (here: "rename").
Well, he said regression testing and I think he means it. Regression
testing is important to verify refactoring has been done correctly,
since even commercial refactoring tools do make mistakes (in part
because they often ask for advice from humans :).
> The crucial bit here is that we're inherently constrained (as of
> today) to do refactoring not in code (as in running code) but in
> text files.
Interlisp did not do it that way, btw. It did all its changes in the
running image and saved from file to disk for the purpose of later
reloads, but kept the master source "in core". A few Maclisp users did
the same, using what was sometimes called the Kludgey Binford Editor
to do in-memory editing of expressions. But ultimately, Teco and
later Emacs (initially Teco-based and then later Lisp-based) won out
as the preferred technology.
Strictly, though, it's not CL but its various commonly available
environments that make you deal with source files. Another paradigm
could be created, but one would have to spec that out. The language
doesn't preclude it, and certainly offers more support for it than
most languages do.
But yes, CL makes you change the file IF the file is what was how you
did it originally--that is, nothing substitutes for changing the way
you loaded it before if you want to have doing that action again do
something different.
> What the OP asks for is "I want to refactor every use of 'foo' to
> 'hoo'", but all he can do is load 'hoo' into the running image,
I think indeed he wants this, but I didn't perceive this as the locus
of his complaint. I tried to obliquely address this issue, too, but
my message was already long, and I didn't want to give my post two
foci, lest the tangent lead to a hyperbola.
> I wonder if some (maybe commercial) vendor has some extension to allow
> such refactorings; I think CL would be well suited to something like
> it.
I assume they're reading along. Who knows?
On Oct 26, 5:16 pm, Kent M Pitman <······@nhplace.com> wrote:
> Holger Schauer <··············@gmx.de> writes:
> > Actually, I think that the main topic of the OP is not regression
> > testing but refactoring (here: "rename").
>
> Well, he said regression testing and I think he means it. Regression
> testing is important to verify refactoring has been done correctly,
> since even commercial refactoring tools do make mistakes (in part
> because they often ask for advice from humans :).
That's exactly what I meant -- I was only using renaming a function as
an example (though see below).
>
> Strictly, though, it's not CL but its various commonly available
> environments that make you deal with source files. Another paradigm
> could be created, but one would have to spec that out. The language
> doesn't preclude it, and certainly offers more support for it than
> most languages do.
Interesting. I've been wondering if CL had a way to read the
originally inputed source code for a function, then let you could edit
it, then resubmit it. It seems like a more logical way to edit the
code in a running environment. The Lisp system could figure out how
to persist it. That would come in handy for some applications.
One possible alternative is the Squeak system, where the objects are
their own editors.
> But yes, CL makes you change the file IF the file is what was how you
> did it originally--that is, nothing substitutes for changing the way
> you loaded it before if you want to have doing that action again do
> something different.
I think I'm going to settle on a batch testing approach: load
everything up in a new Lisp instance and then run the tests.
But this thread has got me thinking: is there a name for testing
running software (eg, testing the in-memory data structures for
invariants)? I'd like to know more about that. I think it would be
useful to run that periodically just as a sanity check.
> > What the OP asks for is "I want to refactor every use of 'foo' to
> > 'hoo'", but all he can do is load 'hoo' into the running image,
>
> I think indeed he wants this, but I didn't perceive this as the locus
> of his complaint. I tried to obliquely address this issue, too, but
> my message was already long, and I didn't want to give my post two
> foci, lest the tangent lead to a hyperbola.
You are perceptive! I do want that. I think one of the secrets of
the sensationalism behind some of the newer programming languages is
their extensive, powerful tool set. I would love to see/know of some
refactoring tools in Emacs.
Eric
http://www.lispcast.com
Kent M Pitman wrote:
> Holger Schauer <··············@gmx.de> writes:
>> I wonder if some (maybe commercial) vendor has some extension to allow
>> such refactorings; I think CL would be well suited to something like
>> it.
>
> I assume they're reading along. Who knows?
One of c.l.lisp's own, Pascal Bourguignon, has published Image Based
Common Lisp <http://www.cl-user.net/asp/libs/IBCL>. It is still
categorized as experimental, though. Pascal, any comments to add? I
thought I remember there being a similar project in the works to add
support for a more smalltalk-ish development environment to some CL
implementations, but my Google-fu isn't quite up to the task. Any help?
From: Holger Schauer
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <yxzzlxz78ip.fsf@gmx.de>
Sorry for coming back to this post this late, but I knew I would have
to re-read your post carefully again.
On 5167 September 1993, Kent M. Pitman wrote:
> ············@gmail.com writes:
>> I know that I am *supposed to* make sure that I modify all of the
>> references to foo when I change it to hoo. But unit tests are
>> supposed to guard me against all of those things I'm supposed to do
>> but forget to do.
[...]
> Unit tests are programs. Nothing more, nothing less. They can, therefore
> have bugs.
[...]
> If they see observed behavior and it's what they don't expect, that
> doesn't make the program wrong; the program is wrong only if (a) the
> program is in fact wrong and (b) the unit test is right (i.e., is
> observing the wrongness of the program's behavior and correctly
> opining on that wrongness). It is still possible for the program to
> be right if both the program is right and the unit test is wrong.
I think that this is indeed the heart of the matter, in combination
with this refactoring thingy I mentioned in my first response. Of
course, I know nothing about the OPs approach to programming, but the
usual scenario for unit tests is test driven development, in which you
develop the tests first. Now, I am really neither an expert in TDD nor
refactoring, but I would assume that refactoring in a TDD scenario
would imply refactoring the tests *first*. If one would do so in the
given example, all unit tests should fail first, then you would
refactor the code, after which all unit tests hopefully succeed
again. This has nothing to do with Lisp, btw.
> It is for this reason that most unit tests are done on freshly-built
> objects. That sounds clean and pretty and makes a lot of unit tests
> not fail. But it doesn't mean much for the methodology of unit
> testing--it says that if you test all your weapons back home at the
> factory when you're milling them, they'll work well under jungle
> conditions.
Hmm. That reminds me of pre-/post-conditions and invariants in code
verification, or better said, that those tests with freshly built
objects assume that one could generalize from the success of such
tests to the general validity (correctness) of the tested code. As you
rightly point out that's a questionable assumption.
Very interesting post, Kent, thanks.
Holger
--
--- http://hillview.bugwriter.net/ ---
Fachbegriffe der Informatik - Einfach erkl�rt
51: Version x.5
Mit neuen Icons! (Kristian K�hntopp)
············@gmail.com writes:
> * Create a shell script which loads all of the files of the
> application into a new lisp image, then runs the tests, prints out the
> results, then exits
> Pros:
> + simple to implement
> + automated
> Cons:
> - loading some code/tests may take a long time
Do it at night using crontab. I build and run the regression tests for my
main application at 3am on three different computers. The result is mailed
to me.
Nicolas
············@gmail.com wrote:
> Do you have any more solutions, or comments on the ones I've provided?
Use a proper IDE with a multi-file search capability, preferably
integrated with its project manager so that it will reliably search all
your source no matter how it is organized directory-wise.
kt
--
http://www.theoryyalgebra.com/
"Career highlights? I had two. I got an intentional walk
from Sandy Koufax and I got out of a rundown against the Mets."."
- Bob Uecker
From: Richard M Kreuter
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <87lk9r86bq.fsf@progn.net>
············@gmail.com writes:
> I was asking about the issue of one of the fundamental assumptions
> of Regression Testing: that they do a relatively thorough job of
> allowing changes to the code which do not alter the correctness of
> the code, while alerting you to changes that do alter the
> correctness of the code. Lisp is different from most other
> languages, in that it maintains a dynamic environment, which exists
> apart from the files of code inputted to it. There is therefore a
> breach of the Regression Testing assumption.
If you're interested in this topic, you might find it useful to look
at how people do regression testing on relational databases, which
also have a dynamic environment independent of files. (I suspect,
however, that database refactoring methodologies might also start with
"assume a freshly initialized database...", which would be about the
same as "assume a freshly initialized Lisp instance...".)
--
RmK
············@gmail.com wrote:
> I know that I am *supposed to* make sure that I modify all of the
> references to foo when I change it to hoo. But unit tests are
> supposed to guard me against all of those things I'm supposed to do
> but forget to do.
I find it easy to remember to "fmakunbound" a function before renaming
it. That will cause the tests that use the old name to fail. And, in
case you wonder about inlined functions, I don't use them until I really
need them, which is about never during development.
Hth.
From: Kent M Pitman
Subject: Re: Common Lisp and RegressionTesting
Date:
Message-ID: <u3aw03tff.fsf@nhplace.com>
············@gmail.com writes:
> Now, I want to change the name of the function foo. It now looks like
> this:
>
> (defun hoo ()
> ...)
>
> Now I run my unit tests, and they still pass, even though the tests
> still refer to function foo, as does function bar.
How does Lisp know you've changed foo?
Maybe you've just hidden the definition somewhere.
How would it tell the difference?
This is entirely a programming environment issue, not a language issue.
The language supports the possibility that you could type in exactly
what you've loaded from your file. So if you write
(defun foo () 'foo)
(defun hoo () 'hoo)
How is Lisp supposed to know that HOO used to be FOO and that you've both
redefined and renamed it? As Richard Kreuter points out, your obligation
is to call something like FMAKUNBOUND or UNINTERN if you want to say "oh,
forget what I wrote before". Otherwise, you are augmenting a dynamic
environment.
Lisp is not a file-based language. Indeed, the evaluation semantics are
not even defined at all on text. The input to the lisp compiler are
objects that have long ago lost association with files. READ's
responsibility is to do that, and so the agent that calls READ and
presents the result to EVAL or COMPILE for evaluation or compilation
is what's responsible for managing identity and adjusting definition.
The only reason that static languages manage to win on this is that
they have a very limited model of how definition works at all, which
allows them to infer a lot more because your options to start with are
fewer.
Just like the government would have a simpler job fighting crime if
they could assume you weren't allowed to walk the streets without
being hassled. That leads to a lot more clever optimization of street
crime, it just doesn't allow ordinary walking of the street.
Once you get what Lisp is up to, as emphasized by both Richard and Ken
Tilton, and probably other posters to follow, you'll understand that the
effect you're looking for comes at way too high a price.
Mariano Montone <··············@gmail.com> writes:
> http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/73609b97c0e0f2f4?tvc=2&q=image
>
Wow, good reference. What a post! Peter Seibel facing the same
"problem" - Kenny giving advice without any hitting of the OP ;-) and
- tatah! Erik Naggum explaining in a very educating and entertaining
way what the - as he says - "more general problem" underlying it is.
That's three Common Lisp Gods in one thread.
Bookmarked.
;-)
Frank
--
Frank Goenninger
frgo(at)mac(dot)com
"Don't ask me! I haven't been reading comp.lang.lisp long enough to
really know ..."
Frank Goenninger DG1SBG wrote:
> Mariano Montone <··············@gmail.com> writes:
>
>
>>http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/73609b97c0e0f2f4?tvc=2&q=image
>>
>
>
> Wow, good reference. What a post! Peter Seibel facing the same
> "problem" - Kenny giving advice without any hitting of the OP ;-) and
> - tatah! Erik Naggum explaining in a very educating and entertaining
> way what the - as he says - "more general problem" underlying it is.
>
> That's three Common Lisp Gods in one thread.
> Bookmarked.
> ;-)
I am just a simple application programmer. JP Massar was a developer of
Star Lisp and is in the Lisp Hall of Fame, Burdick aka Tfb the Younger
is a future Hall of Famer, and Tfb the elder has by now I am sure
dispatched the helicopters for leaving him out.
:)
kzo
--
http://www.theoryyalgebra.com/
"Career highlights? I had two. I got an intentional walk
from Sandy Koufax and I got out of a rundown against the Mets."."
- Bob Uecker
On Oct 25, 8:27 pm, Mariano Montone <··············@gmail.com> wrote:
> http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/73...
Whoah! Deja Vu!
Eric