[mostly newbie, sbcl, linux]
I recently tried to do a seemingly simple thing and could not see how.
Given a string, I wanted to substitute a CR-LF for each LF, or if you
prefer, insert a CR before each LF. I'm sure it can be done with some
crufty characater by character loop, but it seems like there should be
an elegant solution.
Either the case of a small string, allowing exploding it into large
data structures, or a constant-memory streaming algorithm would be
interesting.
Any suggestions?
[I ended up doing it in python which was very simple, but my intent is
to learn practical lisp]
-- George Young
············@gmail.com wrote:
> I recently tried to do a seemingly simple thing and could not see how.
> Given a string, I wanted to substitute a CR-LF for each LF, or if you
> prefer, insert a CR before each LF. I'm sure it can be done with some
> crufty characater by character loop, but it seems like there should be
> an elegant solution.
Strings are sequences. Let's make one with a LF in it:
CL-USER> (defparameter lf-string (concatenate 'string "foo" (string
#\Return)))
LF-STRING
CL-USER> lf-string
"foo^M"
So, you want to replace one element in a sequence with two elements.
You have map, which can apply a function to every element in a
sequence, and return a new sequence with the result, so replacing
that LF with something else is easy:
CL-USER> (map 'string (lambda (e)
(if (eql e #\Return)
#\x
e))
lf-string)
"foox"
We need something like that, but something that can handle more
than one return value. Let's build it:
(defun map-values (result-type function first-sequence)
(let ((new-seq ()))
(loop for e across first-sequence do
(dolist (v (multiple-value-list (funcall function e)))
(push v new-seq)))
(map result-type #'identity (nreverse new-seq))))
Now we can say this:
CL-USER> (map-values 'string (lambda (e)
(if (eql e #\Return)
(values #\Newline #\Return)
e))
lf-string)
"foo
^M"
Which I believe is what you wanted? I'm fairly new to CL myself,
so there might be an easier way. Still, this is trivial and a
general solution (I think, haven't tested it much).
Joel Wilsson wrote:
> (defun map-values (result-type function first-sequence)
map can handle more than one sequence. I forgot to, err... I mean,
extending map-values to more than one sequence is left as an exercise
for the reader! *cough*
Apologies for the spam, but here's the "final" version. I hope it's
more beautiful than whatever you cooked up in Python:
(defun map-values (result-type function sequence)
(map result-type #'identity
(loop for e across sequence
append (multiple-value-list (funcall function e)))))
Joel Wilsson wrote:
> Apologies for the spam, but here's the "final" version. I hope it's
> more beautiful than whatever you cooked up in Python:
>
> (defun map-values (result-type function sequence)
> (map result-type #'identity
> (loop for e across sequence
> append (multiple-value-list (funcall function e)))))
That's pretty cool, thanks. In python the whole task is done by:
def lf2crlf(s):
return s.replace('\n', '\r\n')
Maybe not beautiful, but it sure is concise ;-)
-- George
············@gmail.com wrote:
> In python the whole task is done by:
>
> def lf2crlf(s):
> return s.replace('\n', '\r\n')
>
> Maybe not beautiful, but it sure is concise ;-)
With CL-PPCRE and CL-INTERPOL:
(defun lf-to-crlf (string)
(regex-replace-all #?"\n" string #?"\r\n"))
Maybe not beautiful, but about the same level of conciseness, I'd
say... :)
Cheers,
Edi.
http://weitz.de/cl-ppcre/
http://weitz.de/cl-interpol/
············@gmail.com wrote:
> That's pretty cool, thanks. In python the whole task is done by:
>
> def lf2crlf(s):
> return s.replace('\n', '\r\n')
>
> Maybe not beautiful, but it sure is concise ;-)
>
> -- George
The CL equivalent would make use of CL-PPCRE, as Andr� showed you.
map-values isn't just for strings, it works on other stuff too:
CL-USER> (map-values 'simple-vector
(lambda (e)
(if (oddp e)
(values 'odd e)
(values 'even e)))
#1A(1 2 3 4))
#(ODD 1 EVEN 2 ODD 3 EVEN 4)
············@gmail.com wrote:
> Joel Wilsson wrote:
> > Apologies for the spam, but here's the "final" version. I hope it's
> > more beautiful than whatever you cooked up in Python:
> >
> > (defun map-values (result-type function sequence)
> > (map result-type #'identity
> > (loop for e across sequence
> > append (multiple-value-list (funcall function e)))))
>
> That's pretty cool, thanks. In python the whole task is done by:
>
> def lf2crlf(s):
> return s.replace('\n', '\r\n')
>
> Maybe not beautiful, but it sure is concise ;-)
Doing it that way is not very useful, since it requires loading the
whole file before transforming it. It's generally better to treat a
file incrementally. The above program would fail on a file that is
larger than available virtual memory. It is also not very efficient.
Similar criticisms apply to the code using cl-pcre.
Code that is small but has significant flaws is not that useful, in any
language. Their are great examples of how limited code can be made
incredibly concise in the International obfusticated C competition.
Such code need not be obtuse, but it often isn't really that useful.
That said sometimes it is, I've done my share of stuff in dodgy ways.
On Nov 30, 7:45 am, ············@gmail.com wrote:
> [mostly newbie, sbcl, linux]
> I recently tried to do a seemingly simple thing and could not see how.
> Given a string, I wanted to substitute a CR-LF for each LF, or if you
> prefer, insert a CR before each LF. I'm sure it can be done with some
> crufty characater by character loop, but it seems like there should be
> an elegant solution.
>
> Either the case of a small string, allowing exploding it into large
> data structures, or a constant-memory streaming algorithm would be
> interesting.
> Any suggestions?
> [I ended up doing it in python which was very simple, but my intent is
> to learn practical lisp]
>
> -- George Young
To do character substitution, try:
(substitute #\k #\c "cevin") -> "kevin"
To do entire string substitution, this would work:
(defun str-replace (to-replace replace-with str)
(let* ((start (search to-replace str))
(end (+ start (length to-replace))))
(concatenate 'string (subseq str 0 start)
replace-with (subseq str end))))
(str-replace "peter" "kevin" "peter ryan")
(Note: the string version [ie, the function] doesn't work on characters
- but luckily, CL proovides substitute which I mentioned above).
I think that should help you.
A character by character loop writing to a string-output-stream would
to me seem the simplest although some stream implementations can do
this as part of the output encoding.
>>>>> "georgeryoung" == georgeryoung <············@gmail.com> writes:
georgeryoung> [mostly newbie, sbcl, linux] I recently tried to do
georgeryoung> a seemingly simple thing and could not see how.
georgeryoung> Given a string, I wanted to substitute a CR-LF for
georgeryoung> each LF, or if you prefer, insert a CR before each
georgeryoung> LF. I'm sure it can be done with some crufty
georgeryoung> characater by character loop, but it seems like
georgeryoung> there should be an elegant solution.
georgeryoung> Either the case of a small string, allowing
georgeryoung> exploding it into large data structures, or a
georgeryoung> constant-memory streaming algorithm would be
georgeryoung> interesting. Any suggestions? [I ended up doing it
georgeryoung> in python which was very simple, but my intent is to
georgeryoung> learn practical lisp]
georgeryoung> -- George Young
--
John A.R. Williams
············@gmail.com schrieb:
> [mostly newbie, sbcl, linux]
> I recently tried to do a seemingly simple thing and could not see how.
> Given a string, I wanted to substitute a CR-LF for each LF, or if you
> prefer, insert a CR before each LF. I'm sure it can be done with some
> crufty characater by character loop, but it seems like there should be
> an elegant solution.
>
> Either the case of a small string, allowing exploding it into large
> data structures, or a constant-memory streaming algorithm would be
> interesting.
> Any suggestions?
> [I ended up doing it in python which was very simple, but my intent is
> to learn practical lisp]
You could do it with cl-ppcre:
(cl-ppcre:regex-replace-all
(string #\Newline)
target-string
(concatenate 'string (string #\Newline) (string #\Return)))
cl-ppcre is a regex engine for Lisp.
You can download it here:
http://weitz.de/cl-ppcre/
Andr�
--
On Thu, 30 Nov 2006 14:41:57 +0100, Andr� Thieme
<······························@justmail.de> wrote:
>
> You could do it with cl-ppcre:
>
> (cl-ppcre:regex-replace-all
> (string #\Newline)
> target-string
> (concatenate 'string (string #\Newline) (string #\Return)))
>
I would replace concatenate.. with (coerce '(#\Newline #\return) 'string)
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Andr� Thieme <······························@justmail.de> writes:
> ············@gmail.com schrieb:
>> [mostly newbie, sbcl, linux]
>> I recently tried to do a seemingly simple thing and could not see how.
>> Given a string, I wanted to substitute a CR-LF for each LF, or if you
>> prefer, insert a CR before each LF. I'm sure it can be done with some
>> crufty characater by character loop, but it seems like there should be
>> an elegant solution.
>>
>> Either the case of a small string, allowing exploding it into large
>> data structures, or a constant-memory streaming algorithm would be
>> interesting.
>> Any suggestions?
>> [I ended up doing it in python which was very simple, but my intent is
>> to learn practical lisp]
>
> You could do it with cl-ppcre:
>
> (cl-ppcre:regex-replace-all
> (string #\Newline)
> target-string
> (concatenate 'string (string #\Newline) (string #\Return)))
This is incorrect. #\Newline has no relation with CR or LF.
(cl-ppcre:regex-replace-all
(string #\Linefeed)
target-string
(concatenate 'string (string #\Return) (string #\Linefeed)))
--
__Pascal Bourguignon__ http://www.informatimago.com/
"Do not adjust your mind, there is a fault in reality"
-- on a wall many years ago in Oxford.
Pascal Bourguignon schrieb:
> Andr� Thieme <······························@justmail.de> writes:
>
>> ············@gmail.com schrieb:
>>> [mostly newbie, sbcl, linux]
>>> I recently tried to do a seemingly simple thing and could not see how.
>>> Given a string, I wanted to substitute a CR-LF for each LF, or if you
>>> prefer, insert a CR before each LF. I'm sure it can be done with some
>>> crufty characater by character loop, but it seems like there should be
>>> an elegant solution.
>>>
>>> Either the case of a small string, allowing exploding it into large
>>> data structures, or a constant-memory streaming algorithm would be
>>> interesting.
>>> Any suggestions?
>>> [I ended up doing it in python which was very simple, but my intent is
>>> to learn practical lisp]
>> You could do it with cl-ppcre:
>>
>> (cl-ppcre:regex-replace-all
>> (string #\Newline)
>> target-string
>> (concatenate 'string (string #\Newline) (string #\Return)))
>
> This is incorrect. #\Newline has no relation with CR or LF.
>
> (cl-ppcre:regex-replace-all
> (string #\Linefeed)
> target-string
> (concatenate 'string (string #\Return) (string #\Linefeed)))
>
>
At least on sbcl there seems to be a relation:
CL-USER> #\Linefeed
#\Newline
I was following this table:
http://www.robelle.com/smugbook/ascii.htmlb
And after checking that (code-char 10) -> #\Newline
I put it in the code.
Andr�
--
Andr� Thieme <······························@justmail.de> writes:
> Pascal Bourguignon schrieb:
>> This is incorrect. #\Newline has no relation with CR or LF.
>>
>> (cl-ppcre:regex-replace-all
>> (string #\Linefeed)
>> target-string
>> (concatenate 'string (string #\Return) (string #\Linefeed)))
>>
>>
>
> At least on sbcl there seems to be a relation:
Implementation specific.
--
__Pascal Bourguignon__ http://www.informatimago.com/
"Specifications are for the weak and timid!"
Pascal Bourguignon schrieb:
> Andr� Thieme <······························@justmail.de> writes:
>
>> Pascal Bourguignon schrieb:
>>> This is incorrect. #\Newline has no relation with CR or LF.
>>>
>>> (cl-ppcre:regex-replace-all
>>> (string #\Linefeed)
>>> target-string
>>> (concatenate 'string (string #\Return) (string #\Linefeed)))
>>>
>>>
>> At least on sbcl there seems to be a relation:
>
> Implementation specific.
Okay, possible.
But I can say that at least sbcl works this way.
And clisp. Not to forget Allegro. Oh, and cmucl.
But maybe all the other lisps act different in that regard, so probably
you are right, and there mostly is no relation between #\Linefeed and a
#\Newline. ;)
Andr�
--
Andr� Thieme schrieb:
> Pascal Bourguignon schrieb:
>> Andr� Thieme <······························@justmail.de> writes:
>>
>>> Pascal Bourguignon schrieb:
>>>> This is incorrect. #\Newline has no relation with CR or LF.
>>>>
>>>> (cl-ppcre:regex-replace-all
>>>> (string #\Linefeed)
>>>> target-string
>>>> (concatenate 'string (string #\Return) (string #\Linefeed)))
>>>>
>>>>
>>> At least on sbcl there seems to be a relation:
>>
>> Implementation specific.
>
> Okay, possible.
> But I can say that at least sbcl works this way.
> And clisp. Not to forget Allegro. Oh, and cmucl.
> But maybe all the other lisps act different in that regard, so probably
> you are right, and there mostly is no relation between #\Linefeed and a
> #\Newline. ;)
Forgot to mention Lispworks.
Andr�
--
On Thu, 30 Nov 2006 20:34:48 +0100, Andr� Thieme
<······························@justmail.de> wrote:
> Pascal Bourguignon schrieb:
>> Andr� Thieme <······························@justmail.de> writes:
>>
>>> Pascal Bourguignon schrieb:
>>>> This is incorrect. #\Newline has no relation with CR or LF.
>>>>
>>>> (cl-ppcre:regex-replace-all
>>>> (string #\Linefeed)
>>>> target-string
>>>> (concatenate 'string (string #\Return) (string #\Linefeed)))
>>>>
>>>>
>>> At least on sbcl there seems to be a relation:
>> Implementation specific.
>
> Okay, possible.
> But I can say that at least sbcl works this way.
> And clisp. Not to forget Allegro. Oh, and cmucl.
> But maybe all the other lisps act different in that regard, so probably
> you are right, and there mostly is no relation between #\Linefeed and a
> #\Newline. ;)
>
>
> Andr�
All running Unix right?
That is because #\Linefeed is a gives a newline under Unix.
However under Windows it just goes to the next line.
A #\Return is needed to send it to the beginning of line.
(Like on a typewriter)
I come across this a lot having to convert #\Newline to
#\Return #\Newline. By the way a Mac used (does it still use?)
just #\Return. This is probably the reason for the routine.
I should have caught the problem myself. For Unix compatabillity
Emacs typically handles both cases for Windows and Unix.
So LispWorks for Windows handles #\Linefeed as #\Newline.
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
John Thingstad schrieb:
> All running Unix right?
Yes, nearly. I just tested Lispworks under Windows.
> Emacs typically handles both cases for Windows and Unix.
> So LispWorks for Windows handles #\Linefeed as #\Newline.
Yes.
Andr�
--
"John Thingstad" <··············@chello.no> writes:
> I come across this a lot having to convert #\Newline to
> #\Return #\Newline.
But this is plain wrong.
(format t "~{~C~}" '(#\Return #\Newline)) produces:
On Unix: CR LF
On Mac: CR CR
On MS-Windows and MS-DOS: CR CR LF
The correct way is:
(format t "~{~C~}" '(#\Return #\Linefeed)) which produces:
On Unix: CR LF
On Mac: CR LF
On MS-Windows and MS-DOS: CR LF
as wanted.
--
__Pascal Bourguignon__ http://www.informatimago.com/
"This statement is false." In Lisp: (defun Q () (eq nil (Q)))
On Thu, 30 Nov 2006 23:27:39 +0100, Pascal Bourguignon
<···@informatimago.com> wrote:
> "John Thingstad" <··············@chello.no> writes:
>> I come across this a lot having to convert #\Newline to
>> #\Return #\Newline.
>
>
> The correct way is:
>
> (format t "~{~C~}" '(#\Return #\Linefeed)) which produces:
> On Unix: CR LF
> On Mac: CR LF
> On MS-Windows and MS-DOS: CR LF
> as wanted.
>
Right. That is what I thought I said. Sorry for the confusion.
I think of it as \n or \r\l I guess.
Anyhow I have a feeling substituting #\Linefeed for #\Newline
is what is needed. That should output correctly on all systems.
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
> I come across this a lot having to convert #\Newline to
> #\Return #\Newline.
For the record:
#\Linefeed to #\Return #\Linefeed
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Andr� Thieme <······························@justmail.de> writes:
> Pascal Bourguignon schrieb:
> > Andr� Thieme <······························@justmail.de> writes:
> > Implementation specific.
Or perhaps better said, platform-specific, but see a previous Mac OS
post indicating implementation specific behavior on a single platform.
> Okay, possible.
> But I can say that at least sbcl works this way.
> And clisp. Not to forget Allegro. Oh, and cmucl.
> But maybe all the other lisps act different in that regard, so probably
> you are right, and there mostly is no relation between #\Linefeed and a
> #\Newline. ;)
Well, on a Unix system, I would expect the implementations to work the
same way. But if you tried it under Windows or MacOS, or on a VAX,
etc. you would find different behavior.
It's harder to write portable code if you only test in one environment.
--
Thomas A. Russ, USC/Information Sciences Institute
Pascal Bourguignon <···@informatimago.com> writes:
> Andr� Thieme <······························@justmail.de> writes:
>
> > Pascal Bourguignon schrieb:
> >> This is incorrect. #\Newline has no relation with CR or LF.
> >
> > At least on sbcl there seems to be a relation:
>
> Implementation specific.
And thus non-portable.
On my Mac,
MCL maps #\Newline to #\Return
OpenMCL maps #\Newline to #\Linefeed
somewhat reflecting the dualist nature of the platform.
IIRC the Spec says that #\Newline should be the natural end of line
character for the platform in question. (I'm not sure what the Windows
convention is, given that there isn't a single character for it. I
suspect that linefeed is the more common choice.)
--
Thomas A. Russ, USC/Information Sciences Institute
···@sevak.isi.edu (Thomas A. Russ) writes:
> IIRC the Spec says that #\Newline should be the natural end of line
> character for the platform in question. (I'm not sure what the
> Windows convention is, given that there isn't a single character for
> it. I suspect that linefeed is the more common choice.)
No single byte denotes end-of-line on Windows, but Common Lisp
characters aren't bytes.
#\Newline represents whatever is the line division convention in a
particular file. For example, writing a #\Newline means emitting
#\Linefeed for Unix files, a #\Return for (older?) MacOS files, and
#\Return#\Linefeed for Windows files. But on a record-structured file
system, writing a #\Newline might mean emitting a length-encoded
sequence of bytes, no particular subsequence of which would signify
end-of-line.
--
RmK
Helu
* georgeryoung <························@l12g2000cwl.XXXXXXX.com> :
| I recently tried to do a seemingly simple thing and could not see how.
| Given a string, I wanted to substitute a CR-LF for each LF, or if you
| prefer, insert a CR before each LF. I'm sure it can be done with some
| crufty characater by character loop, but it seems like there should be
| an elegant solution.
IMO, at best, you can chunk (i.e. buffer up) the input and
output. This but this involves more cruft, not less.
| Either the case of a small string, allowing exploding it into large
| data structures, or a constant-memory streaming algorithm would be
| interesting.
| Any suggestions?
If you are looking at modifying the string in place, I think you have
to have at least as much memory as there are CRs in the input string.
| [I ended up doing it in python which was very simple, but my intent is
| to learn practical lisp]
OK, Given that goal, here is some [lightly tested] code to look at.
The idea is to read and write the same string. The string grows to the
right --- We assume it is adjustable and has a fill-pointer. We will
also need a tail-consing queue data structure with 2 methods:
(defstruct queue list tail-cons (length 0))
(defun enqueue (X queue)
"Insert X at the end of QUEUE's list."
(let ((tail-cons (queue-tail-cons queue))
(new-cons (cons X NIL)))
(if tail-cons
(setf (cdr tail-cons) new-cons)
(setf (queue-list queue) new-cons))
(setf (queue-tail-cons queue) new-cons))
(values X (incf (queue-length queue))))
(defun dequeue-n (queue &optional (n 1) (return-type 'list))
"Dequeue and return the first N elements of QUEUE's list as a
sequence coerced to RETURN-TYPE."
(let* ((old-queue (queue-list queue))
(prev-cons (nthcdr (1- n) old-queue))
(new-queue (cdr prev-cons))) ; (nthcdr n old-queue)
(cond ((consp prev-cons)
(decf (queue-length queue) n)
(setf (cdr prev-cons) nil (queue-list queue) new-queue)
(coerce old-queue return-type))
(t (cerror "Return NIL"
"Cannot dequeue ~D elements from Queue of length ~D"
N (queue-length queue))))))
And another helper function:
(defmacro ensure-array-capacity (array size)
"Ensure adjustable ARRAY with fill-pointer has length SIZE."
`(let ((orig-size (array-total-size ,array)))
(if (<= ,size orig-size)
(prog1 ,array (setf (fill-pointer ,array) ,size))
(setq ,array (adjust-array ,array ,size :fill-pointer t)))))
(defvar $string2
(make-array 0 :element-type 'character :adjustable t :fill-pointer t))
Assume the input is available in $string2 with fill-pointer. If your
data is in $STRING1 you can copy it to $STRING2 in two steps:
(ensure-array-capacity $string2 (length $string1))
(replace $string2 $string1)
Now the actual code:
(defvar +Linefeed+)
(defvar +Return+)
We wany the code below to work with binary arrays as well as character
strings. Assuming you only have character strings, you should:
(setq +Linefeed+ #\Linefeed
+Return+ #\Return)
(defun convert-crlf (string &key (queue (make-queue)) (flush-threshold 100))
"Replace CR with CRLF inplace in STRING. STRING is an
adjustable array with a fill-pointer. Use QUEUE as scratch
buffer, STRING is written to in counts of FLUSH- THRESHOLD"
(let ((total-read-length (length string))
(read-index 0)
(write-index 0)
(nchars-written 0))
(flet ((dochar (c)
(enqueue c queue)
(let ((nchars-writeable (- read-index write-index)))
(when (> nchars-writeable flush-threshold)
(assert (< nchars-writeable (queue-length queue)))
(replace string (dequeue-n queue nchars-writeable)
:start1 write-index)
(incf write-index nchars-writeable)
(incf nchars-writeable nchars-writeable)))))
(loop (let ((c (aref string read-index)))
(incf read-index)
(cond ((eql c +Return+)
(dochar +Return+)
(dochar +LineFeed+))
(t (dochar c)))
(when (= read-index total-read-length)
(ensure-array-capacity string (+ write-index
(queue-length queue)))
(replace string (queue-list queue) :start1 write-index)
(incf nchars-written (queue-length queue))
(return nchars-written)))))))
This tries to cons as little as possible while reading STRING exactly
once. The algorithm should also work if you want to modify a file
inplace, since it only grows. You can open the file with :DIRECTION
:IO and :IF-EXISTS :OVERWRITE (assuming your lisp does not illegaly
truncate the file). You have to change REPLACE appropriately to
WRITE-SEQUENCE.
However, I dont know how to open a file in :OVERWRITE mode and then
truncate it portably.
-- Madhu