I have the following scenario:
- I have a bunch of files (userspecified filenames)
- I want to create one stream from all these files, in the form of
something like the result of (make-concatenated-stream <files-steams>)
- (read-line <mystream>) should result in one line. This line should be a line
from one of the logs, sorted by some rule.[1]
- the next (readl-line <mystream>) should return the next in the sort order
etc.
Using Scheme I could do things like this with "force" and "delay" to create
a highly personalized stream. Can I to this in CL too, or do I have to create
functions for this (e.g. implement my own read-files-stream function)?
[1]: make-concatenated-stream provides this (except for the sorting) for
example.
--
Oyvin
On 21 Apr 2005 13:22:15 +0200, <·····@remove.spam.oyvins.net> wrote:
>
> - I have a bunch of files (userspecified filenames)
> - I want to create one stream from all these files, in the form of
> something like the result of (make-concatenated-stream <files-steams>)
> - (read-line <mystream>) should result in one line. This line should be a line
> from one of the logs, sorted by some rule.[1]
Since you mention that these are logfiles, you might want to start by
looking at syslog-ng. My various logfiles are combined with named
pipes to the master logfile, under a FIFO sorting rule. Note that
syslog-ng will work to combine any set of logging inputs and doesn't
need to be the traditional syslog function, as it is very flexible.
--
Everyman has three hearts;
one to show the world, one to show friends, and one only he knows.
GP lisper <········@CloudDancer.com> writes:
> On 21 Apr 2005 13:22:15 +0200, <·····@remove.spam.oyvins.net> wrote:
> >
> > - I have a bunch of files (userspecified filenames)
> > - I want to create one stream from all these files, in the form of
> > something like the result of (make-concatenated-stream <files-steams>)
> > - (read-line <mystream>) should result in one line. This line should be a line
> > from one of the logs, sorted by some rule.[1]
>
> Since you mention that these are logfiles, you might want to start by
> looking at syslog-ng. My various logfiles are combined with named
> pipes to the master logfile, under a FIFO sorting rule. Note that
> syslog-ng will work to combine any set of logging inputs and doesn't
> need to be the traditional syslog function, as it is very flexible.
OK. Thanks for the tip. However, the app I'm writing is supposed to handle
all kind of different log-systems (it's written for FreeBSD syslog for now,
a syslog implementation with quite advanced features btw.). Therefore I need to
write the concat-sort-ish stuff myself.
--
Oyvin
Oyvin Halfdan Thuv <·····@remove.spam.oyvins.net> writes:
> - (read-line <mystream>) should result in one line. This line should be a
> line from one of the logs, sorted by some rule.[1]
^^^^
That is "files". I'm working on log-files.
--
Oyvin
> Using Scheme I could do things like this with "force" and "delay" to create
> a highly personalized stream. Can I to this in CL too, or do I have to create
> functions for this (e.g. implement my own read-files-stream function)?
I think there's some confusion here. What CL calls ``streams'' is
called ``I/O ports'' in Scheme; what Scheme calls ``streams'' is
called ``lazy lists'' by the rest of the universe.
There are a few ways to solve your problem. The simple way would be
to build a vector or an ordinary (eager) list of all the lines in your
logs and sort them; while not as elegant as using an abstraction, it
is likely to be more efficient than the other solutions (since you're
sorting the data, using lazy data structures doesn't buy you anything
in that case since you're sorting the data).
If you insist on using an abstraction, you could map your lines to a
CL stream by using the so-called ``Gray Streams'' library; not
standard Common Lisp, but present in every implementation of CL that I
know of.
If you want a higher-level abstraction, there's nothing preventing you
from implementing lazy lists in CL. Here's an example of an implemen-
tation of Scheme-style promises:
(defstruct promise
value
� thunk)
(defun force (promise)
(when (promise-thunk promise)
(setf (promise-value promise) (funcall (promise-thunk promise))
(promise-thunk promise) nil))
(promise-value promise))
(defmacro delay (form)
`(make-promise :thunk #'(lambda () ,form)))
There's nothing preventing you from using closures to store the thunk,
as is traditionally done in Scheme, but the above feels more idiomatic
to me.
An alternative implementation would be to define a lazy CONS rather
than promises.
Finally, you could also use generators rather than lazy lists, but at
least to me that feels less intuitive.
Juliusz Chroboczek
Juliusz Chroboczek <···@pps.jussieu.fr> writes:
> > Using Scheme I could do things like this with "force" and "delay" to create
> > a highly personalized stream. Can I to this in CL too, or do I have to create
> > functions for this (e.g. implement my own read-files-stream function)?
>
> I think there's some confusion here. What CL calls ``streams'' is
> called ``I/O ports'' in Scheme; what Scheme calls ``streams'' is
> called ``lazy lists'' by the rest of the universe.
Aha.
> There are a few ways to solve your problem. The simple way would be
> to build a vector or an ordinary (eager) list of all the lines in your
> logs and sort them; while not as elegant as using an abstraction, it
> is likely to be more efficient than the other solutions (since you're
> sorting the data, using lazy data structures doesn't buy you anything
> in that case since you're sorting the data).
The data is already sorted (by timestamp) in each file. Therefore I do not
need to read more than one line at a time into memory. The files are rather
large as well, so it's not feasible to read them into memory anyway. Efficiency
isn't really an issue right here.
I've already written a piece of code that pops lines in a correct manner from
each file. I just wanted the "stream" of data from all the files to be
represented by the same abstraction as any other file-stream. For now I have
a (get-next-line) function that works, but it's inelegant. Elegance is
important :)
> If you insist on using an abstraction, you could map your lines to a
> CL stream by using the so-called ``Gray Streams'' library; not
> standard Common Lisp, but present in every implementation of CL that I
> know of.
>
> If you want a higher-level abstraction, there's nothing preventing you
> from implementing lazy lists in CL. Here's an example of an implemen-
> tation of Scheme-style promises:
>
> (defstruct promise
> value
> � thunk)
>
> (defun force (promise)
> (when (promise-thunk promise)
> (setf (promise-value promise) (funcall (promise-thunk promise))
> (promise-thunk promise) nil))
> (promise-value promise))
>
> (defmacro delay (form)
> `(make-promise :thunk #'(lambda () ,form)))
Yes, this is elegant. I like it.
> There's nothing preventing you from using closures to store the thunk,
> as is traditionally done in Scheme, but the above feels more idiomatic
> to me.
>
> An alternative implementation would be to define a lazy CONS rather
> than promises.
>
> Finally, you could also use generators rather than lazy lists, but at
> least to me that feels less intuitive.
>
> Juliusz Chroboczek
This was interesting reading, thanks.
--
Oyvin