Hello all,
I am new at lisp, and would like to try making an actual program to
help me practise and learn. I spend a lot of my time doing market
analysis, and would like to make a program (Or set of programs) that
would help me do this with lisp.
The raw data that I would have to work with is a set of open, high,
low, close, time, and volume numbers. I figure that packages each of
these from a CSV file into a class or structure object would be the best
way to begin, and then running programs on top of those objects would be
the analysis phase.
I would like to know if there is some code floating about out there
that already does this, so that I can have a guideline to use when
making my own. Again, I want to do this to begin programming (I learn
best by practise) and learning the Lisp way.
Thank you
On 19 June, 14:12, Bata <·········@yahoo.ca> wrote:
> Hello all,
> I am new at lisp, and would like to try making an actual program to
> help me practise and learn. I spend a lot of my time doing market
> analysis, and would like to make a program (Or set of programs) that
> would help me do this with lisp.
> The raw data that I would have to work with is a set of open, high,
> low, close, time, and volume numbers. I figure that packages each of
> these from a CSV file into a class or structure object would be the best
> way to begin, and then running programs on top of those objects would be
> the analysis phase.
> I would like to know if there is some code floating about out there
> that already does this, so that I can have a guideline to use when
> making my own. Again, I want to do this to begin programming (I learn
> best by practise) and learning the Lisp way.
>
> Thank you
I've got code for parsing CSV into various formats somewhere, but it's
a straightforward task so probably a good one to learn a bit more
about the language, file interaction, and if you like the excellent cl-
ppcre library (regular expressions) or split-sequence mini-library.
There are two schools of thought about structures vs. classes vs.
lists vs. arrays for this kind of task. I'd start by working with
objects, and when you know what you really want to do and if you're
facing performance issues, move to arrays or structs if you have
performance issues (although structure access may not always be faster
than objects - YMMV).
Usually I start a quick interactive development of file-processing by
importing files into lists to make sure my parser works since it's so
easy to introspect on the resulting data. So, before performance
tuning, you would simply read a CSV line into a string, use cl-ppcre
or split-sequence to turn that into a list of symbols (using intern)
and numbers (using parse-integer), and then use that list to create
objects.
Have fun!
Ian
ianeslick <·········@gmail.com> wrote:
+---------------
| Bata <·········@yahoo.ca> wrote:
| > The raw data that I would have to work with is a set of open, high,
| > low, close, time, and volume numbers. I figure that packages each of
| > these from a CSV file into a class or structure object would be the best
| > way to begin, ...
...
| I've got code for parsing CSV into various formats somewhere, but it's
| a straightforward task so probably a good one to learn a bit more
| about the language, file interaction...
+---------------
I agree: writing a PARSE-LINE-AS-CSV function is an excellent
learning tool! One can start with the simple case, then add
the more complex special cases incrementally [things like double
quotes, commas within double quotes, escape characters other
than double quotes (e.g., some applications permit "\"), escaped
commas & double quotes, fields with a mixture of all of these, etc.].
<http://en.wikipedia.org/wiki/Comma-separated_values> and
<http://tools.ietf.org/rfc/rfc4180.txt> mention some of the
subtleties [except the additional escape characters].
+---------------
| ...and if you like the excellent cl-ppcre library (regular expressions)
| or split-sequence mini-library.
+---------------
Unfortunately, those "more complex special cases" I mentioned
above aren't easy to handle with either CL-PPCRE or SPLIT-SEQUENCE.
I would suggest using a simple TAGBODY-based state machine
[or LOOP (ECASE state), if you don't like TAGBODY].
+---------------
| Usually I start a quick interactive development of file-processing by
| importing files into lists to make sure my parser works since it's so
| easy to introspect on the resulting data. So, before performance
| tuning, you would simply read a CSV line into a string, use cl-ppcre
| or split-sequence to turn that into a list of symbols (using intern)
| and numbers (using parse-integer), and then use that list to create
| objects.
+---------------
I would differ only slightly, suggesting first a READ-FILE-LINES function
that slurps a whole file into a list of lines, and then writing/debugging
your PARSE-LINE-AS-CSV function with individual test-ase lines, then
doing (MAPCAR #'PARSE-LINE-AS-CSV (READ-FILE-LINES "file")). That will
yield a list if lists of strings, which you can further parse as numbers
or names or whatever.
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> I agree: writing a PARSE-LINE-AS-CSV function is an excellent
> learning tool!
Indeed, when you start making errors at the name of the function,
you're bound to learn a lot...
CSV records may span several lines!
--
__Pascal Bourguignon__
Pascal J. Bourguignon <···@informatimago.com> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| > I agree: writing a PARSE-LINE-AS-CSV function is an excellent
| > learning tool!
|
| Indeed, when you start making errors at the name of the function,
| you're bound to learn a lot...
|
| CSV records may span several lines!
+---------------
Touch�! That's one of those "special cases"
that my own CSV parser doesn't handle yet! ;-} ;-}
PARSE-STREAM-AS-CSV, then...? ;-}
Plus, since CL strings may contain #\newline, for testing
at the REPL one could write a PARSE-STRING-AS-CSV:
(defun parse-string-as-csv (string)
(with-input-from-string (s string)
(parse-stream-as-csv s)))
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
On 19 June, 19:30, ····@rpw3.org (Rob Warnock) wrote:
> Pascal J. Bourguignon <····@informatimago.com> wrote:
> +---------------
> | ····@rpw3.org (Rob Warnock) writes:
> | > I agree: writing a PARSE-LINE-AS-CSV function is an excellent
> | > learning tool!
> |
> | Indeed, when you start making errors at the name of the function,
> | you're bound to learn a lot...
> |
> | CSV records may span several lines!
> +---------------
>
> Touché! That's one of those "special cases"
> that my own CSV parser doesn't handle yet! ;-} ;-}
>
> PARSE-STREAM-AS-CSV, then...? ;-}
>
> Plus, since CL strings may contain #\newline, for testing
> at the REPL one could write a PARSE-STRING-AS-CSV:
>
> (defun parse-string-as-csv (string)
> (with-input-from-string (s string)
> (parse-stream-as-csv s)))
>
> -Rob
>
> -----
> Rob Warnock <····@rpw3.org>
> 627 26th Avenue <URL:http://rpw3.org/>
> San Mateo, CA 94403 (650)572-2607
I love how we always like to make things more complicated than they
need to be. For learning purposes, unless your files are truly
generic CSV files, you don't need a general parser, just one that gets
the job done. Premature generalization is almost as bad a sin as
premature optimization. Of course I speak with authority, as these
are two of my worst habits!
Perhaps the proper name is parse-line-as-my-csv or qadp-line-as-csv
(qadp = quick and dirty parse) :)
Ian
On Jun 20, 12:25 am, ianeslick <·········@gmail.com> wrote:
> On 19 June, 19:30, ····@rpw3.org (Rob Warnock) wrote:
>
>
>
> > Pascal J. Bourguignon <····@informatimago.com> wrote:
> > +---------------
> > | ····@rpw3.org (Rob Warnock) writes:
> > | > I agree: writing a PARSE-LINE-AS-CSV function is an excellent
> > | > learning tool!
> > |
> > | Indeed, when you start making errors at the name of the function,
> > | you're bound to learn a lot...
> > |
> > | CSV records may span several lines!
> > +---------------
>
> > Touché! That's one of those "special cases"
> > that my own CSV parser doesn't handle yet! ;-} ;-}
>
> > PARSE-STREAM-AS-CSV, then...? ;-}
>
> > Plus, since CL strings may contain #\newline, for testing
> > at the REPL one could write a PARSE-STRING-AS-CSV:
>
> > (defun parse-string-as-csv (string)
> > (with-input-from-string (s string)
> > (parse-stream-as-csv s)))
>
> > -Rob
>
> > -----
> > Rob Warnock <····@rpw3.org>
> > 627 26th Avenue <URL:http://rpw3.org/>
> > San Mateo, CA 94403 (650)572-2607
>
> I love how we always like to make things more complicated than they
> need to be. For learning purposes, unless your files are truly
> generic CSV files, you don't need a general parser, just one that gets
> the job done. Premature generalization is almost as bad a sin as
> premature optimization. Of course I speak with authority, as these
> are two of my worst habits!
>
> Perhaps the proper name is parse-line-as-my-csv or qadp-line-as-csv
> (qadp = quick and dirty parse) :)
>
> Ian
I think I just wrote a generic 'split string' and used read-line last
time I wanted to do simple CSV stuff.
(Also look up parse-integer and the substitute/find functions to
convert all of the imported data into valid/idiomatic lisp objects).
But yeah, laziness is definitely something I consider a programming
virtue.
Another option for the data stuff would be to write it all out as lisp
data structures (for example with a custom 'defrecord' style macro)
and then just read it all in using the reader.
Maybe you (Bata) can do that as an exercise for stuff that you want to
save between sessions.
On Jun 19, 11:12 pm, Bata <·········@yahoo.ca> wrote:
> Hello all,
> I am new at lisp, and would like to try making an actual program to
> help me practise and learn. I spend a lot of my time doing market
> analysis, and would like to make a program (Or set of programs) that
> would help me do this with lisp.
> The raw data that I would have to work with is a set of open, high,
> low, close, time, and volume numbers. I figure that packages each of
> these from a CSV file into a class or structure object would be the best
> way to begin, and then running programs on top of those objects would be
> the analysis phase.
> I would like to know if there is some code floating about out there
> that already does this, so that I can have a guideline to use when
> making my own. Again, I want to do this to begin programming (I learn
> best by practise) and learning the Lisp way.
>
> Thank you
the RSM-STRING package (sourceforge, but also somewhere on Debian) has
a 2 nice parsers which handle numbers and strings and mixed values.
It returns list-of-list structures which one can array-ize.
Also see some of the top-level code in ls-demo.lisp and maybe
TODO.lisp (I think it's the former) in the Common Lisp Stat repository
at http://github.com/blindglobe