Hello all,
I am trying to write a benchmark program for functions that looks like
this:
(defun benchmark fun arglist &optional (experiments 5) (runtimes
10000)
...
)
I want the function fun to be evaluated with the given arglist, a
number of times (runtimes). I want to record the time that is needed
and repeat the experiment a number of times (experiments) to get a
fair result. At the end I want to return the average time needed by
the function to evaluate "runtimes" times.
I have been pointed out to use (get-internal-run-time) to record the
beginning and end time. Also I tried to implement benchmark, only I
end up using (progn (...)) or (block...) and setf. Probably I am doing
something wrong.
Can you guys help me with this?
Grtz,
Michiel
Lisp has a 'time' macro.
(time
(dotimes (i 50000)
(my-function arg0 arg1 ...)))
The result will be printed to the console.
By the way, I combined it into experiments = 1, runtimes = 50000
because I feel that's just as fair.
··············@gmail.com (Michiel Borkent) wrote in message news:<····························@posting.google.com>...
> Hello all,
>
> I am trying to write a benchmark program for functions that looks like
> this:
>
> (defun benchmark fun arglist &optional (experiments 5) (runtimes
> 10000)
> ...
> )
>
> I want the function fun to be evaluated with the given arglist, a
> number of times (runtimes). I want to record the time that is needed
> and repeat the experiment a number of times (experiments) to get a
> fair result. At the end I want to return the average time needed by
> the function to evaluate "runtimes" times.
Thank you all for your helpful replies.
I didn't know how to use the let form properly and that it was similar
to progn in a way.
After discovering it, I came up with the following with a lot of help
from a site I found with Google and some-one who told me the specific
use of loop. I still need to check out all this, I am only a noob.
I distinguished single-experiments that run a number of times and
experiments that are being runned by benchmark a number of times, to
see if it makes any difference doing it this way. Maybe another
process interrupts, for instance the garbage collector, and influences
on the real used time, I don't know.
(defun benchmark (fun arglist &optional (times 10000) (experiments 5))
(loop repeat experiments
as (run real) = (single-experiment fun arglist times)
sum run into runsum
sum real into realsum
finally (return (list (float (/ runsum experiments))(float (/
realsum experiments))))))
(defun single-experiment (fun arglist &optional (times 100000))
(let ((real1 (get-internal-real-time))
(run1 (get-internal-run-time)))
(dotimes (n times)
(apply fun arglist))
(let ((run2 (get-internal-run-time))
(real2 (get-internal-real-time)))
(let ((totalrun (- run2 run1))
(totalreal (- real2 real1)))
;(format t "~% Experiment took ~f seconds of runtime... "
totalrun)
;(format t "~% Experiment took ~f seconds of real time... ~%"
totalreal)
(list totalrun totalreal)))))
···············@yahoo.com (Mark McConnell) wrote in message news:<····························@posting.google.com>...
> Lisp has a 'time' macro.
>
> (time
> (dotimes (i 50000)
> (my-function arg0 arg1 ...)))
>
> The result will be printed to the console.
>
> By the way, I combined it into experiments = 1, runtimes = 50000
> because I feel that's just as fair.
>
> ··············@gmail.com (Michiel Borkent) wrote in message news:<····························@posting.google.com>...
> > Hello all,
> >
> > I am trying to write a benchmark program for functions that looks like
> > this:
> >
> > (defun benchmark fun arglist &optional (experiments 5) (runtimes
> > 10000)
> > ...
> > )
> >
> > I want the function fun to be evaluated with the given arglist, a
> > number of times (runtimes). I want to record the time that is needed
> > and repeat the experiment a number of times (experiments) to get a
> > fair result. At the end I want to return the average time needed by
> > the function to evaluate "runtimes" times.
··············@gmail.com (Michiel Borkent) writes:
> [...]
> (return (list (float (/ runsum experiments))(float (/
> realsum experiments))))
>
> [...]
> (list totalrun totalreal)
By the way, you can also return multiple values with VALUES (instead
of wrapping them in a list). One benefit is that it shows more
clearly what is going on.
---Vassil.
--
Vassil Nikolov <········@poboxes.com>
Hollerith's Law of Docstrings: Everything can be summarized in 72 bytes.
Michiel Borkent wrote:
> Hello all,
>
> I am trying to write a benchmark program for functions that looks like
> this:
>
> (defun benchmark fun arglist &optional (experiments 5) (runtimes
> 10000)
> ...
> )
>
> I want the function fun to be evaluated with the given arglist, a
> number of times (runtimes). I want to record the time that is needed
> and repeat the experiment a number of times (experiments) to get a
> fair result. At the end I want to return the average time needed by
> the function to evaluate "runtimes" times.
>
> I have been pointed out to use (get-internal-run-time) to record the
> beginning and end time. Also I tried to implement benchmark, only I
> end up using (progn (...)) or (block...) and setf. Probably I am doing
> something wrong.
>
> Can you guys help me with this?
I find it hard to understand what your real problem is. The section
about progn, block and setf doesn't make any sense to me.
Do you know about APPLY for calling functions on argument lists, and
TIME for doing something very close to what you want? Both are standard
functions in Common Lisp.
Pascal
--
Pascal Costanza University of Bonn
···············@web.de Institute of Computer Science III
http://www.pascalcostanza.de R�merstr. 164, D-53117 Bonn (Germany)
··············@gmail.com (Michiel Borkent) wrote:
> I am trying to write a benchmark program for functions that looks like
> this:
>
> (defun benchmark fun arglist &optional (experiments 5) (runtimes
> 10000)
> ...
> )
>
> I want the function fun to be evaluated with the given arglist, a
> number of times (runtimes). I want to record the time that is needed
> and repeat the experiment a number of times (experiments) to get a
> fair result. At the end I want to return the average time needed by
> the function to evaluate "runtimes" times.
what is the difference between 50,000 runtimes and 10,000 runtimes * 5
"experiments" ? For something similiar to TIME you can use a macro:
(defmacro time-n (count &body body)
`(let ((sum 0))
(loop repeat ,count do
(let ((start (get-internal-run-time)))
,@body
(setq sum (+ sum (- (get-internal-run-time) start)))))
(let ((sum-seconds (float (/ sum internal-time-units-per-second))))
(format t "iterations: ~D, runtime: ~D, single run: ~D~%"
,count
sum-seconds
(/ sum-seconds ,count)))))
CL-USER> (time-n 1000000 (sqrt 1234))
iterations: 1000000, runtime: 4.0658464, single run: 4.0658465E-6
But I don't know how exact this is, because there might be too much
overhead by the surrounding loop.
--
Frank Bu�, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
Frank Buss <··@frank-buss.de> writes:
> But I don't know how exact this is, because there might be too much
> overhead by the surrounding loop.
The surrounding loop shouldn't affect anything, since you are only
accumulating the time around the body. Of course, for timing really
short things, there might be issues with the resolution of
GET-INTERNAL-RUN-TIME that could cause trouble.
--
Thomas A. Russ, USC/Information Sciences Institute
Frank Buss <··@frank-buss.de> writes:
> what is the difference between 50,000 runtimes and 10,000 runtimes * 5
> "experiments" ?
You can get some estimate of the variability or repeatability of the
quantity being measured in the latter case. Books on experimental
science will explain this in more detail.
Christophe
Christophe Rhodes <·····@cam.ac.uk> wrote:
> You can get some estimate of the variability or repeatability of the
> quantity being measured in the latter case. Books on experimental
> science will explain this in more detail.
yes, but I don't think this will result in much difference for measuring
Lisp functions, if you don't provide another arguments.
BTW: My macro was bad, because of course I should time the whole loop
instead of summing the inner loop. And DOTIMES looks better for this,
because LOOP has to much overhead:
CL-USER> (time (dotimes (i 1000000) (sqrt 1234)))
Real time: 2.9642625 sec.
Run time: 2.9242048 sec.
Space: 24000740 Bytes
GC: 46, GC time: 0.0801152 sec.
NIL
CL-USER> (time (loop repeat 1000000 do (sqrt 1234)))
Real time: 4.6967535 sec.
Run time: 4.576581 sec.
Space: 24004028 Bytes
GC: 46, GC time: 0.0901296 sec.
NIL
CL-USER>
And it should be a good idea to subtract the overhead, for example by
running the function n times and 2*n times:
(defmacro run-time (&body body)
`(let ((start (get-internal-run-time)))
,@body
(- (get-internal-run-time) start)))
(defmacro time-n (count &body body)
`(float (/ (run-time (dotimes (i ,count) ,@body))
,count
internal-time-units-per-second)))
(defmacro time-2n (count &body body)
`(let ((count-n (time-n ,count ,@body))
(count-2n (time-n (* 2 ,count) ,@body)))
(let ((sum-seconds (- count-2n count-n)))
(format t "iterations: ~D, single runtime: ~D ~D ~D"
,count sum-seconds count-n count-2n))))
But it doesn't help much, looks like there are some optimization in
CLisp, so sometimes the result is known before the calculation starts :-)
CL-USER> (defun fac (n) (if (> n 1) (* n (fac (1- n))) 1))
FAC
CL-USER> (time-n 100000 (fac 42))
1.0985797E-4
CL-USER> (time-n 100000 (fac 20))
5.0773007E-5
CL-USER> (time-2n 100000 (fac 20))
iterations: 100000, single runtime: 8.0115205E-7 5.0672865E-5 5.1474017E-
5
NIL
CL-USER> (time-2n 100000 (fac 42))
iterations: 100000, single runtime: -2.9542425E-6 1.1456473E-4
1.1161049E-4
NIL
CL-USER>
--
Frank Bu�, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de