I apologize if similar requests have been made frequently to this news group in the past.
I need a function wich separates a string into substrings. Each substring is separated by
one or more characters from a set of "token dividers".
Example:
(separate-tokens "This$is a list$ of $tokens" '(#\space #\$))
("This" "is" "a" "list" "of" "tokens")
I am sure this problem has been solved many times before, so I would like to avoid to do
this over again. Does anybody care to show me the implentation of a similar function?
--
Regards, K}re Telnes
Division of Engineering Cybernetics : Phone +47 7 594376
The Norwegian Institute of Technology : Fax +47 7 594399
N-7034 Trondheim : Email ······@itk.unit.no
NORWAY
In article <·····················@ugle.unit.no> ······@itk.unit.no (Kaare Telnes) writes:
>(separate-tokens "This$is a list$ of $tokens" '(#\space #\$))
>("This" "is" "a" "list" "of" "tokens")
>I am sure this problem has been solved many times before, so I would like to avoid to do
>this over again. Does anybody care to show me the implentation of a similar function?
I've probably done it before, but I don't know where the code is, so I did
it again. It only took ten minutes.
(defun separate-tokens (string delimiters)
(loop with length = (length string)
for start = 0 then (1+ end) until (> start length)
for end = (or (position-if #'(lambda (char) (member char delimiters))
string :start start)
length)
unless (= start end) ;; ignore consecutive delimiters
collect (subseq string start end)))
--
Barry Margolin
System Manager, Thinking Machines Corp.
······@think.com {uunet,harvard}!think!barmar
From: Nick Chapman
Subject: Re: Identifying token
Date:
Message-ID: <1538@trotsky>
······@itk.unit.no (Kaare Telnes) writes:
> I apologize if similar requests have been made frequently to this news group in the past.
> I need a function wich separates a string into substrings. Each substring is separated by
> one or more characters from a set of "token dividers".
>
> Example:
>
> (separate-tokens "This$is a list$ of $tokens" '(#\space #\$))
> ("This" "is" "a" "list" "of" "tokens")
>
(defun separate-tokens (String SeparatorSet)
(remove-if
#'(lambda (x) (string= x ""))
(reduce
#'(lambda (a b)
(if (member a SeparatorSet)
(cons "" b)
(cons
(concatenate
'string
(make-string 1 :initial-element a)
(car b))
(cdr b))))
String
:initial-value '("")
:from-end t)))
Ugly, but it does the trick.
Nick.
--
-------------------------------------------------------------------------------
Nick Chapman, GEC-Marconi Research Centre | INTERNET: ···@uk.co.gec-mrc
Great Baddow, Chelmsford, Essex. CM2 8HN | Tel: +44 245 73331 x3245
-------------------------------------------------------------------------------