I'm writing yet another flat file parser, and considering how often I
do this I would very much like to write:
(defun parse-test-case (stream)
(map 'vector #'parse-line (read-lines stream)))
I can do this easily myself if I write write read-lines to simply
accumulate the lines in a sequence. I would rather do it lazily,
though, to avoid reading the entire file into memory. As far as I can
tell, a list is a list is a list. It's a concrete thing, so I can't
create my own list type. That means I need to use a sequence type.
Is there an existing sequence type that does this? If not, I need to
implement a sequence type. Section 17.1 of the Hyperspec listst
sequence functions; are these generic functions that I need to
specialize for my sequence type? Is there a smaller subset that is
sufficient?
Also, I often want to throw away the map results, so I want X where
map : X :: mapcar : mapc. I will need to write this myself, correct?
Thanks,
David
grackle wrote:
> I'm writing yet another flat file parser, and considering how often I
> do this I would very much like to write:
>
> (defun parse-test-case (stream)
> (map 'vector #'parse-line (read-lines stream)))
>
> I can do this easily myself if I write write read-lines to simply
> accumulate the lines in a sequence. I would rather do it lazily,
> though, to avoid reading the entire file into memory. As far as I can
> tell, a list is a list is a list. It's a concrete thing, so I can't
> create my own list type. That means I need to use a sequence type.
>
> Is there an existing sequence type that does this? If not, I need to
> implement a sequence type. Section 17.1 of the Hyperspec listst
> sequence functions; are these generic functions that I need to
> specialize for my sequence type? Is there a smaller subset that is
> sufficient?
>
> Also, I often want to throw away the map results, so I want X where
> map : X :: mapcar : mapc. I will need to write this myself, correct?
I'm confused as to what you're trying to do... you want to be able to
call PARSE-LINE on the result of READ-LINE for each line of the file
and collect the results in a vector? Do you know the length of the
vector beforehand?
(defun parse-test-case (stream)
(let ((done (gensym))
xx)
(do ((line (read-line stream nil done)
(read-line stream nil done))
((eq line done) (coerce (nreverse xx) 'vector))
(push (parse-line line) xx))))
or is your problem not being able to use a generic MAPC/MACPAR in this
situation?
(defun do-lines ((var stream &optional return) &body body)
(let ((done (gensym))
(s (gensym)))
`(let ((,s ,stream))
(do ((var (read-line ,s nil ,done)
(read-line ,s nil ,done)))
((eq ,var ,done) ,return)
,@body))))
(defun mapc-lines (fn stream)
(do-lines (l stream)
(funcall fn l)))
(defun mapcar-lines (fn stream)
(let (xx)
(do-lines (l stream (nreverse xx))
(push (funcall fn l) xx))))
(defun parse-test-case (stream)
(coerce (mapcar-lines #'parse-line stream) 'vector))
hth
Nick
>
> Thanks,
> David
········@gmail.com wrote:
> I'm confused as to what you're trying to do... you want to be able to
> call PARSE-LINE on the result of READ-LINE for each line of the file
> and collect the results in a vector? Do you know the length of the
> vector beforehand?
Sometimes I want to collect, sometimes not. I usually don't know the
length of the input beforehand.
> or is your problem not being able to use a generic MAPC/MACPAR in this
> situation?
Yes. There are many macros and functions that accept sequence types,
and I don't want to reimplement every one I use for every sequence-like
type I create! I would like to create a sequence object that allows me
to use any macro or function that expects a sequence.
Andrew Philpot wrote:
> You could implement a lazy list, and give it an API that mirrors that
> of built-on CONS-constructed lists, starting with MAP, etc.
The list and sequence APIs are there and they work. I think all I need
to know is how to create a subtype of list or sequence. Then only a
single function make-lazy-readline-list or make-lazy-readline-sequence
is needed to take advantage of map, mapc, mapcar, dolist, etc.
> > Also, I often want to throw away the map results, so I want X where
> > map : X :: mapcar : mapc. I will need to write this myself, correct?
>
> For this, I think you want MAP with first argument output-type-spec
> NIL.
Thanks!
-David
In article <·······················@m73g2000cwd.googlegroups.com>,
"grackle" <···········@gmail.com> wrote:
> Yes. There are many macros and functions that accept sequence types,
> and I don't want to reimplement every one I use for every sequence-like
> type I create! I would like to create a sequence object that allows me
> to use any macro or function that expects a sequence.
Sequences are not object-oriented, so there's no standard way to extend
the sequence functions.
You also can't define new arithmetic types -- for instance, you can't
make the + function operate on matrices.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
On Thu, 16 Nov 2006 21:01:54 -0500, Barry Margolin wrote:
> In article <·······················@m73g2000cwd.googlegroups.com>,
> "grackle" <···········@gmail.com> wrote:
>
>> Yes. There are many macros and functions that accept sequence types,
>> and I don't want to reimplement every one I use for every sequence-like
>> type I create! I would like to create a sequence object that allows me
>> to use any macro or function that expects a sequence.
>
> Sequences are not object-oriented, so there's no standard way to extend
> the sequence functions.
>
> You also can't define new arithmetic types -- for instance, you can't
> make the + function operate on matrices.
cl:+ isn't a method, but you could create a my-cl:+ that is:
(defpackage :my-cl
(:use :cl)
(:shadow :+))
(in-package :my-cl)
(defclass not-really-a-matrix ()
((a :accessor a-of :initarg :a :initform 0)
(b :accessor b-of :initarg :b :initform 0)))
(defmethod + ((x number) &rest args)
(apply #'cl:+ x args))
(defmethod + ((x not-really-a-matrix) &rest args)
(apply #'+ (a-of x) (b-of x) args))
..or something..then `:use my-cl' in your application-package.
--
Lars Rune Nøstdal
http://lars.nostdal.org/
In article <······························@gmail.com>,
Lars Rune Nøstdal <···········@gmail.com> wrote:
> On Thu, 16 Nov 2006 21:01:54 -0500, Barry Margolin wrote:
>
> > In article <·······················@m73g2000cwd.googlegroups.com>,
> > "grackle" <···········@gmail.com> wrote:
> >
> >> Yes. There are many macros and functions that accept sequence types,
> >> and I don't want to reimplement every one I use for every sequence-like
> >> type I create! I would like to create a sequence object that allows me
> >> to use any macro or function that expects a sequence.
> >
> > Sequences are not object-oriented, so there's no standard way to extend
> > the sequence functions.
> >
> > You also can't define new arithmetic types -- for instance, you can't
> > make the + function operate on matrices.
>
> cl:+ isn't a method, but you could create a my-cl:+ that is:
...
> ..or something..then `:use my-cl' in your application-package.
That won't make all the functions not written by him use it.
Relating this back to the OP's problem, he wants built-in functions like
MAPCAR to be able to iterate over his lazy sequences. To make use of
your suggestion he would have to create MY-CL:MAPCAR, MY-CL:LOOP,
MY-CL:DO, etc.
Compare this will languages like CLU and C++, which have a standard
generic protocol for iterators, so that programmers can define their own
sequences and make use of the built-in for-loop.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
Barry Margolin wrote:
> Compare this will languages like CLU and C++, which have a standard
> generic protocol for iterators, so that programmers can define their own
> sequences and make use of the built-in for-loop.
This is what I'm accustomed to and was expecting. It's very un-Lispy
to make a fundamental part of the language off-limits to users.
-David
grackle wrote:
> Barry Margolin wrote:
>> Compare this will languages like CLU and C++, which have a standard
>> generic protocol for iterators, so that programmers can define their own
>> sequences and make use of the built-in for-loop.
>
> This is what I'm accustomed to and was expecting. It's very un-Lispy
> to make a fundamental part of the language off-limits to users.
>
It's really not off-limits. It is there so that the core of the Common
Lisp language can be compiled efficiently. You can
create your own package that does just what you want. You can even
create new syntax to support your needs. But, you are strongly
discouraged from changing the core system. There is a reason
and there are practical arguments for it.
You seem concerned that there needs to be a standard way.
Why?
Wade
Wade Humeniuk wrote:
> But, you are strongly
> discouraged from changing the core system. There is a reason
> and there are practical arguments for it.
As Common Lisp is defined, adding a sequence type would involve
changing the core system. I don't understand why it has to be that
way. There's already a menagerie of sequence types, including lists,
multidimensional arrays, and bit vectors. If an implementation can
detect and handle all those cases, would it kill performance to add a
generic case?
>
> You seem concerned that there needs to be a standard way.
> Why?
It doesn't make sense for a Common Lisp implementation to do all the
work of implementing sequence operations and leave the user with no way
of reusing that work for his own types, and no alternative but to
reproduce the work on his own.
A few weeks ago Juho Snellman wrote this about SBCL:
> It doesn't generally [*]. But the inlined and non-inlined versions of
> MAP-INTO will actually use radically different implementation
> strategies. The non-inlined version is pretty hideous, and has
> probably only survived since people who use MAP-INTO care about
> performance enough to declare their sequence types.
This demonstrates that writing a looping macro is a nontrivial task.
After all, some SBCL implementor did a hideous job on his first try,
and fixing it is a big enough job that no one is volunteering to do it
without demonstrated demand. For me, that is sufficient evidence that
while it might be instructive for a newb to reimplement map, mapcar,
mapc, map-into, etc., it is not conducive to creating good software.
A programmer can use an *alternative* to a standard type (I'm looking
at Series right now), but it will be inferior to using an *extension*
of a standard type, because it can't be used with code written for the
standard type. I can't (AFAICT) pass a Series object to any of my
existing code or to any third-party code that operates on sequences.
Not only that, portable library code will have inferior performance (if
I understand correctly) compared to functionality that is specialized
for an implementation.
-David
grackle <···········@gmail.com> wrote:
>> It doesn't generally [*]. But the inlined and non-inlined versions of
>> MAP-INTO will actually use radically different implementation
>> strategies. The non-inlined version is pretty hideous, and has
>> probably only survived since people who use MAP-INTO care about
>> performance enough to declare their sequence types.
>
> This demonstrates that writing a looping macro is a nontrivial task.
> After all, some SBCL implementor did a hideous job on his first try,
> and fixing it is a big enough job that no one is volunteering to do it
> without demonstrated demand.
Well, not quite. First of all, that code is about 15 years old, and
probably just written as the fall-back case with the expectation of
all real use going through the open coded versions. Second, the code
itself is pretty simple, it's hideous mainly if you consider it from a
performance viewpoint (which that thread was about). And third, I
don't expect that fixing it would be a big job in absolute terms, just
not very cost-effective.
> For me, that is sufficient evidence that
> while it might be instructive for a newb to reimplement map, mapcar,
> mapc, map-into, etc., it is not conducive to creating good software.
That, of course, is true. I've never bought the "if you need
extensibility for standard CL operators, just write your own versions
of the functions and shadow the symbols" argument.
For what it's worth, it's likely that SBCL will include support for
user-extensible sequences as a documented extension in the near
future.
--
Juho Snellman
grackle <···········@gmail.com> wrote:
+---------------
| Wade Humeniuk wrote:
| > But, you are strongly discouraged from changing the core system.
| > There is a reason and there are practical arguments for it.
|
| As Common Lisp is defined, adding a sequence type would involve
| changing the core system. I don't understand why it has to be that
| way. There's already a menagerie of sequence types, including lists,
| multidimensional arrays, and bit vectors.
+---------------
Sorry, SEQUENCE is the union of only *two* types: LIST & VECTOR.
Multidimensional arrays are *not* included. [And bit vectors are,
well, just vectors...] So there's only a two-way dispatch being done.
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> grackle <···········@gmail.com> wrote:
> | As Common Lisp is defined, adding a sequence type would involve
> | changing the core system. I don't understand why it has to be that
> | way. There's already a menagerie of sequence types, including lists,
> | multidimensional arrays, and bit vectors.
>
> Sorry, SEQUENCE is the union of only *two* types: LIST & VECTOR.
This is not correct. Please look up the definition of the SEQUENCE
type in the standard.
Christophe
Christophe Rhodes <·····@cantab.net> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| > Sorry, SEQUENCE is the union of only *two* types: LIST & VECTOR.
|
| This is not correct. Please look up the definition of the SEQUENCE
| type in the standard.
+---------------
Uh... I did, *really* I did!! ;-} ;-}
http://www.lispworks.com/documentation/HyperSpec/Body/17_a.htm
17.1 Sequence Concepts
A sequence is an ordered collection of elements, implemented as
either a vector or a list.
...
That sure sounded definitive to me!! My mistake was not remembering
that the standard is not always completely self-consistent. I *knew*
that, but forgot to browse around a bit to see if this was one of
those times. (Oops.) Looking further now, I see:
http://www.lispworks.com/documentation/HyperSpec/Body/t_seq.htm
System Class SEQUENCE
...
The types VECTOR and the type LIST are disjoint subtypes of type
SEQUENCE, but are not necessarily an exhaustive partition of sequence.
...
But to address the OP's original complaint about "a menagerie of
sequence types" [which my response was an attempt to rebut], do you
know of any implementation that actually exploits that permission?
E.g., do you know of any implementation that supports multi-dimensional
arrays as sequences? [I suppose one could always traverse such an
array in row-major order...]
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Christophe Rhodes <·····@cantab.net> wrote:
> +---------------
> | ····@rpw3.org (Rob Warnock) writes:
> | > Sorry, SEQUENCE is the union of only *two* types: LIST & VECTOR.
> |
> | This is not correct. Please look up the definition of the SEQUENCE
> | type in the standard.
> +---------------
>
> Uh... I did, *really* I did!! ;-} ;-}
>
> http://www.lispworks.com/documentation/HyperSpec/Body/17_a.htm
> 17.1 Sequence Concepts
>
> A sequence is an ordered collection of elements, implemented as
> either a vector or a list.
> ...
>
> That sure sounded definitive to me!! My mistake was not remembering
> that the standard is not always completely self-consistent.
Heh, and so was mine... I wasn't aware of that sentence you've just
quoted. :-/ Sorry for being so brusque.
> http://www.lispworks.com/documentation/HyperSpec/Body/t_seq.htm
> System Class SEQUENCE
> ...
> The types VECTOR and the type LIST are disjoint subtypes of type
> SEQUENCE, but are not necessarily an exhaustive partition of sequence.
> ...
This is what I was thinking of, yes.
> But to address the OP's original complaint about "a menagerie of
> sequence types" [which my response was an attempt to rebut], do you
> know of any implementation that actually exploits that permission?
> E.g., do you know of any implementation that supports multi-dimensional
> arrays as sequences? [I suppose one could always traverse such an
> array in row-major order...]
I know of one implementation which is considering allowing users to
define their own subclasses of sequence, usable in all the places that
CL allows sequences to be used. (I don't know of any which would
allow multi-dimensional arrays to be treated as sequences, for what
it's worth; I think probably such an implementation would be formally
non-compliant because of the restrictions on superclasses: I don't
think an implementation is allowed to put SEQUENCE in the superclass
list of ARRAY, from CLHS 1.4.4.5).
Christophe
Rob Warnock wrote:
> Sorry, SEQUENCE is the union of only *two* types: LIST & VECTOR.
> Multidimensional arrays are *not* included. [And bit vectors are,
> well, just vectors...] So there's only a two-way dispatch being done.
I stand corrected about multidimensional arrays, but I think there must
be more than two-way dispatch. The code to access elements of a bit
vector will differ from the code to access elements of other kinds of
vectors, because bits can't be directly addressed on most machines. It
will also differ for byte-sized objects on machines that can't address
byte offsets. I think there are at least three cases for retrieving
the next object: next list item, vector item at addressable offset,
and vector item at possibly non-addressable offset.
Juho Snellman wrote:
> For what it's worth, it's likely that SBCL will include support for
> user-extensible sequences as a documented extension in the near
> future.
Sounds great! I can't wait to start writing sequence adaptors instead
of specialized loops.
(map nil (lambda (n) (format t "Node: ~a~%" n)) (depth-first my-tree))
(map nil (lambda (n) (format t "Leaf: ~a~%" n)) (leaves-only my-tree))
It's nice to be able to operate on constructs in a functional style
without instantiating them concretely.
-David
"grackle" <···········@gmail.com> writes:
> I stand corrected about multidimensional arrays, but I think there must
> be more than two-way dispatch. The code to access elements of a bit
> vector will differ from the code to access elements of other kinds of
> vectors, because bits can't be directly addressed on most machines. It
> will also differ for byte-sized objects on machines that can't address
> byte offsets. I think there are at least three cases for retrieving
> the next object: next list item, vector item at addressable offset,
> and vector item at possibly non-addressable offset.
Not at the gross language level. For any sequence, you can use ELT or
for any vector you can use AREF to gain access to a particular element.
Exactly how that gets translated into machine code, is, indeed an
important consideration for the compiler writer, but as far as the
sequence functions themselves are concerned, they don't need to be
concerned with it. They can all be implemented in terms of the element
access functions and thus leverage the underlying implementation work.
--
Thomas A. Russ, USC/Information Sciences Institute
In article <·······················@f16g2000cwb.googlegroups.com>,
"grackle" <···········@gmail.com> wrote:
> Wade Humeniuk wrote:
> > But, you are strongly
> > discouraged from changing the core system. There is a reason
> > and there are practical arguments for it.
>
> As Common Lisp is defined, adding a sequence type would involve
> changing the core system. I don't understand why it has to be that
> way. There's already a menagerie of sequence types, including lists,
> multidimensional arrays, and bit vectors. If an implementation can
> detect and handle all those cases, would it kill performance to add a
> generic case?
Because the basic CL language was not designed with OO in mind. CLOS
was a later add-on. What you describe would have required all
implementations to rewrite much of the basic functionality, and that
would have been an unacceptable impediment to adoption of the language.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: Lazy sequence for reading file
Date:
Message-ID: <87odr3oetx.fsf@qrnik.zagroda>
Wade Humeniuk <··················@telus.net> writes:
>> This is what I'm accustomed to and was expecting. It's very un-Lispy
>> to make a fundamental part of the language off-limits to users.
>
> It's really not off-limits. It is there so that the core of the
> Common Lisp language can be compiled efficiently.
I thought that Lisp prefers flexibility over ease of optimizations by
language implementations.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
On Nov 16, 12:39 pm, "grackle" <···········@gmail.com> wrote:
> situation?Yes. There are many macros and functions that accept sequence types,
> and I don't want to reimplement every one I use for every sequence-like
> type I create! I would like to create a sequence object that allows me
> to use any macro or function that expects a sequence.
First: There are no facilities to write your own sequence types and
have them work with the Common Lisp sequence functions. Those aren't
generic functions. As others have mentioned, you could easily write
your own package that re-exports the bulk of the CL symbols, and
shadows the CL sequence functions with generic replacements.
Presumably, the default primary method of the generic replacements
would call the CL version.
Second: All of the built-in sequence functions are written with the
assumption that the entire contents of the sequence are available
immediately. For example, if you fed MAP a lazy sequence, you would
probably like to be able to get a lazy sequence back from it... but the
MAP function was written to apply itself to everything immediately.
Reusing the Common Lisp sequence functions on lazy sequences is futile,
since they're written with different assumptions in mind.
Dick Waters did some work regarding lazy value generation. Some
implementations might support his proposals, or there might be
libraries out there.
joelh
Joel Ray Holveck wrote:
> First: There are no facilities to write your own sequence types and
> have them work with the Common Lisp sequence functions.
Thanks; this is the answer I needed. I would be curious to hear why.
> Second: All of the built-in sequence functions are written with the
> assumption that the entire contents of the sequence are available
> immediately. For example, if you fed MAP a lazy sequence, you would
> probably like to be able to get a lazy sequence back from it... but the
> MAP function was written to apply itself to everything immediately.
> Reusing the Common Lisp sequence functions on lazy sequences is futile,
> since they're written with different assumptions in mind.
It would work in the case where you want to immediately process as much
of the sequence as is available, which would include reading from a
file or a socket. Potentially infinite sequences could be used as long
as one of the sequences being mapped could be depended on to block or
terminate when appropriate:
(mapc (lambda (line number) (format t "Line ~a: ~a~%" number line))
(make-readlines-sequence my-input-stream)
(make-integer-sequence :start 1 :step 1))
I'm just looking for the most carpal-friendly way to parse flat files,
so Nick's suggestion is the way to go, I think.
-David
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: Lazy sequence for reading file
Date:
Message-ID: <87k61roegl.fsf@qrnik.zagroda>
"grackle" <···········@gmail.com> writes:
>> First: There are no facilities to write your own sequence types and
>> have them work with the Common Lisp sequence functions.
>
> Thanks; this is the answer I needed. I would be curious to hear why.
Probably because CLOS was a late addition to Common Lisp, so there was too
little time to update older designs to use CLOS before ANSI standarization.
And there was little evolution of Common Lisp after the standarization.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
"Joel Ray Holveck" <·····@piquan.org> writes:
> First: There are no facilities to write your own sequence types and
> have them work with the Common Lisp sequence functions. Those aren't
> generic functions.
This is true at the moment, though nothing (or almost nothing: see CDR
3) in the standard prevents an implementation from allowing this.
> Second: All of the built-in sequence functions are written with the
> assumption that the entire contents of the sequence are available
> immediately. For example, if you fed MAP a lazy sequence, you would
> probably like to be able to get a lazy sequence back from it... but the
> MAP function was written to apply itself to everything immediately.
> Reusing the Common Lisp sequence functions on lazy sequences is futile,
> since they're written with different assumptions in mind.
The MAP example is a good one, because it's not amenable to
specialization in the ordinary CLOS manner. However, for many of the
CL sequence functions it is in fact possible to get the lazy -> lazy
behaviour that one might want from sequence functions, so one could
imagine (say) REMOVE implemented lazily on a lazy sequence.
Cheers,
Christophe
Christophe Rhodes wrote:
> The MAP example is a good one, because it's not amenable to
> specialization in the ordinary CLOS manner. However, for many of the
> CL sequence functions it is in fact possible to get the lazy -> lazy
> behaviour that one might want from sequence functions, so one could
> imagine (say) REMOVE implemented lazily on a lazy sequence.
Oh, I think you misunderstand me. I'm just saying that the only thing
that he could reuse would be the names... the implementations wouldn't
be reusable. Sure you could implement REMOVE against a lazy sequence,
but it would have to be an all-new implementation.
joelh
"Joel Ray Holveck" <·····@piquan.org> writes:
> Christophe Rhodes wrote:
>
>> The MAP example is a good one, because it's not amenable to
>> specialization in the ordinary CLOS manner. However, for many of the
>> CL sequence functions it is in fact possible to get the lazy -> lazy
>> behaviour that one might want from sequence functions, so one could
>> imagine (say) REMOVE implemented lazily on a lazy sequence.
>
> Oh, I think you misunderstand me. I'm just saying that the only thing
> that he could reuse would be the names... the implementations wouldn't
> be reusable. Sure you could implement REMOVE against a lazy sequence,
> but it would have to be an all-new implementation.
OK. I think we can agree, then, but I might put a slightly different
spin on it: being able to use the same name for the same concept is a
non-trivial benefit; additionally, there will be cases where
implementations of sequence functions can usefully be shared across
different sequence-like things, so facilitating that would be a good
thing.
Christophe
In article <·······················@f16g2000cwb.googlegroups.com>,
grackle wrote:
> I'm writing yet another flat file parser, and considering how often I
> do this I would very much like to write:
>
> (defun parse-test-case (stream)
> (map 'vector #'parse-line (read-lines stream)))
>
> I can do this easily myself if I write write read-lines to simply
> accumulate the lines in a sequence. I would rather do it lazily,
> though, to avoid reading the entire file into memory. As far as I can
> tell, a list is a list is a list. It's a concrete thing, so I can't
> create my own list type. That means I need to use a sequence type.
You could implement a lazy list, and give it an API that mirrors that
of built-on CONS-constructed lists, starting with MAP, etc. Since you
don't use the lazy list as a data structure (assign, mutate, etc.),
one might argue this would be of little practical benefit.
In Lisp, "lazy" sometimes ends up as "implemented with a closure."
Another way to do this would be do forgo the lazy list directly and
define a function that embodies the closure:
(map-over-lines-of-file #'parse-line stream)
or a macro wrapping of something like the above
(with-lines-from-file (stream file)
(parse-line stream))
Of course you could implement the lazy list, then the mapper on top of
it, then the macro on top of it!
When I recapitulate this kind of idiom, my solutions look like the
WITH-LINES-FROM-FILE version. Not surprisingly, these tend to take
quite a few keyword arguments like WITH-OPEN-FILE (often but not
always defaulted). Including all of these might tend to make your
READ-LINES lazy list specification a lot less perspicuous. With
either but particularly well with the WITH-LINES-FROM-FILE syntax, you
can use either functional filters or control flow to limit what gets
collected...
>
> Is there an existing sequence type that does this? If not, I need to
> implement a sequence type. Section 17.1 of the Hyperspec listst
> sequence functions; are these generic functions that I need to
> specialize for my sequence type? Is there a smaller subset that is
> sufficient?
>
> Also, I often want to throw away the map results, so I want X where
> map : X :: mapcar : mapc. I will need to write this myself, correct?
For this, I think you want MAP with first argument output-type-spec
NIL.
>
> Thanks,
> David
>
#A
grackle wrote:
> I'm writing yet another flat file parser, and considering how often I
> do this I would very much like to write:
>
> (defun parse-test-case (stream)
> (map 'vector #'parse-line (read-lines stream)))
>
> I can do this easily myself if I write write read-lines to simply
> accumulate the lines in a sequence. I would rather do it lazily,
> though, to avoid reading the entire file into memory. As far as I can
> tell, a list is a list is a list. It's a concrete thing, so I can't
> create my own list type. That means I need to use a sequence type.
>
> Is there an existing sequence type that does this? If not, I need to
> implement a sequence type. Section 17.1 of the Hyperspec listst
> sequence functions; are these generic functions that I need to
> specialize for my sequence type? Is there a smaller subset that is
> sufficient?
>
> Also, I often want to throw away the map results, so I want X where
> map : X :: mapcar : mapc. I will need to write this myself, correct?
>
It's been done before. Look at the SERIES package.
http://series.sourceforge.net/
Wade