I have a project that could possibly benefit from using Lisp as an
intermediate language. Hopefully I can describe this simply enough and
someone can give me some simple code as an example? (note: this will be
done in Corman Lisp).
I would like to create a simple assembler in Lisp. I think the Lisp syntax
handles itself quite well to assembly, because it could allow me to use
Lisp syntax to handle forward branches and loops without labels. So, I'm
currently putting together a bunch of functions that return the correct
opcodes for the ARM7:
(defun compile (instruction) ... )
(defmacro bx (rm) (compile ...))
Now, I'm hoping that I can then use this as an intermediate language. The
original compiler will produce Lisp code like so:
(declare my-adder
(pop r1)
(add r0 r1 r0)
(bx lr))
When the IL compiler executes, it will interpret and write the binary for
the final image. I've already coded it in Forth, the theory is sound and
works great. The difficulty I'm having is that Forth is rather obscure and
difficult for my target audience. I'm hoping Lisp will be easier for them.
So, now onto my question: I'd like to limit the scope of the IL compiler so
that *only* the functions I create for it (like "declare" above) are
accessible, and the end-user can't use the entire Lisp system. I'm assuming
this can be done with packages, but I have never used them before in any
past Lisp code that I've written.
Thanks in advance.
--
Best regards,
Jeff jma[at]mfire.com
http://www.jm-basic.com
Jeff Massung <···@NOSPAM.mfire.com> writes:
> I have a project that could possibly benefit from using Lisp as an
> intermediate language. Hopefully I can describe this simply enough and
> someone can give me some simple code as an example? (note: this will be
> done in Corman Lisp).
>
> I would like to create a simple assembler in Lisp. I think the Lisp syntax
> handles itself quite well to assembly, because it could allow me to use
> Lisp syntax to handle forward branches and loops without labels. So, I'm
> currently putting together a bunch of functions that return the correct
> opcodes for the ARM7:
>
> (defun compile (instruction) ... )
> (defmacro bx (rm) (compile ...))
>
> Now, I'm hoping that I can then use this as an intermediate language. The
> original compiler will produce Lisp code like so:
>
> (declare my-adder
> (pop r1)
> (add r0 r1 r0)
> (bx lr))
>
> When the IL compiler executes, it will interpret and write the binary for
> the final image. I've already coded it in Forth, the theory is sound and
> works great. The difficulty I'm having is that Forth is rather obscure and
> difficult for my target audience. I'm hoping Lisp will be easier for them.
>
> So, now onto my question: I'd like to limit the scope of the IL compiler so
> that *only* the functions I create for it (like "declare" above) are
> accessible, and the end-user can't use the entire Lisp system. I'm assuming
> this can be done with packages, but I have never used them before in any
> past Lisp code that I've written.
>
> Thanks in advance.
A package is a little world of symbols, that can intersect other
worlds. What I've done in a similar situation is to make three
packages: one contains the symbols defined for your system, another is
the package in which you implement your system, and a third is for
your user code to be loaded into:
(defpackage :il
(:use)
(:export #:declare ...))
(defpackage :il-impl
(:use :common-lisp))
(defpackage :il-user
(:use #:il))
The Common Lisp code that implements your IL go in il-impl. To refer
to symbols in your IL's package, you use an explicit prefix:
(in-package :il-impl)
(defmacro IL:DEFINE (name &body instructions)
`(progn
(eval-when (compile load eval)
(setf (get (quote ,name) 'il-definition)
(quote ,instructions)))
(quote ,name)))
...
Now you can load files written in your IL into the il-user package:
(let ((*package* (find-package :il-user)))
(load "some-file.il"))
The only way user code can refer to symbols outside of your IL's world
is by using explicit prefixes. If you're not worried about malicious
code, this solution works well.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> The only way user code can refer to symbols outside of your IL's
> world is by using explicit prefixes. If you're not worried about
> malicious code, this solution works well.
What if one is worried about malicious code? I'm trying to come up
with some example of what "malicious code" could do (like deleting
kernel32.dll), but maybe that kind of protection is better left to the
operating system's permission model.
The range of malice expands when one considers running code from a
remote untrusted source. Java applets encourage this scenario, and the
Java virtual machine has a permission model to control which
facilities are restricted.� Is it possible (or at least practical) to
run Common Lisp code with a similarly restricted subset of facilities?
Footnotes:
� Not that I've ever used it, so correct me if the example is
specious.
--
Steven E. Harris :: ········@raytheon.com
Raytheon :: http://www.raytheon.com
"Steven E. Harris" <········@raytheon.com> writes:
> ···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>
> > The only way user code can refer to symbols outside of your IL's
> > world is by using explicit prefixes. If you're not worried about
> > malicious code, this solution works well.
>
> What if one is worried about malicious code?
Then you can't cheat by simply extending CL, you need to implement a
complete interpreter for your language, avoiding CL:EVAL. You can
probably get away with using CL:READ, though. Really, this is an
entirely different question.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
In article <···············@L75001820.us.ray.com>,
"Steven E. Harris" <········@raytheon.com> wrote:
> What if one is worried about malicious code? I'm trying to come up
> with some example of what "malicious code" could do (like deleting
> kernel32.dll), but maybe that kind of protection is better left to the
> operating system's permission model.
>
> The range of malice expands when one considers running code from a
> remote untrusted source. Java applets encourage this scenario, and the
> Java virtual machine has a permission model to control which
> facilities are restricted.� Is it possible (or at least practical) to
> run Common Lisp code with a similarly restricted subset of facilities?
This type of protection generally needs to be designed into the runtime
system of the language, it's not something you can add on with
user-written code. As you note, the permission model is part of the JVM.
It certainly would be possible to make a CL implementation with a
similar security model, but it's not part of the standard language. And
the package system should not be viewed as any form of this. It's
intended as a way to avoid name collisions when loading unrelated pieces
of software into the same Lisp image. Exporting and importing are just
conveniences that allow for abbreviated references to symbols that are
useful outside their home packages.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Jeff Massung <···@NOSPAM.mfire.com> writes:
> I have a project that could possibly benefit from using Lisp as an
> intermediate language. Hopefully I can describe this simply enough and
> someone can give me some simple code as an example? (note: this will be
> done in Corman Lisp).
>
> I would like to create a simple assembler in Lisp.
I suppose you are aware that some Lisps include an assembler, and its
source code, where you could get some inspiration on how to approach
things (AFAIK Corman is among them).
> (defun compile (instruction) ... )
> (defmacro bx (rm) (compile ...))
>
> Now, I'm hoping that I can then use this as an intermediate language. The
> original compiler will produce Lisp code like so:
>
> (declare my-adder
> (pop r1)
> (add r0 r1 r0)
> (bx lr))
>
> When the IL compiler executes, it will interpret and write the binary for
> the final image. I've already coded it in Forth, the theory is sound and
> works great. The difficulty I'm having is that Forth is rather obscure and
> difficult for my target audience. I'm hoping Lisp will be easier for them.
Heh. Lisp being easier/less obscure than anything else is not
something you hear too often. :-)
> So, now onto my question: I'd like to limit the scope of the IL compiler so
> that *only* the functions I create for it (like "declare" above) are
> accessible, and the end-user can't use the entire Lisp system. I'm assuming
> this can be done with packages, but I have never used them before in any
> past Lisp code that I've written.
Basically yes, packages could help you define your own language that
doesn't have much in common with Common Lisp, and has a completely
disjoint set of operators.
If you define a package like ARM-ASSEMBLER which does not use the
COMMON-LISP package, and arrange for code in the immediate language to
be read in that package, there won't be any name clashes. This would
however make it neccessary that you use explicit package qualifiers
for either the symbols in the COMMON-LISP package or those in
ARM-ASSEMBLER. That is, given a package declaration like
(defpackage :arm-assembler (:use))
you'd have to write either
(in-package :arm-assembler)
(cl:defun compile (instruction) ...)
(cl:defmacro bx (rm) (compile ...))
or
(defun arm-assembler:compile (...) ...)
(defmacro arm-assembler:bx (...) ...)
If your ARM-ASSEMBLER package would use the CL package, you'd get in
trouble, because 1) the symbols COMPILE and DECLARE in that package
would be the same as COMMON-LISP:COMPILE/COMMON-LISP:DECLARE and you
must not redefine those and 2) all other operators defined by ANSI CL
would be accessible in this package.
Another thing is how you arrange for the IL code to be read in the
correct package, i.e. the ARM-ASSEMBLER package. If you just use the
Common Lisp LOAD function, you'll have to start each IL file with a
(in-package :arm-assembler) form, because load always binds the
current package to COMMON-LISP-USER. If this is not possible because
you cannot be sure that all IL files satisfy this requirement, you
could bind the *PACKAGE* special variable (which controls in which
package symbols are read by default, absent a package qualifier) and
use READ and EVAL instead of LOAD, like in
(defun read-il-file (file)
(with-open-file (stream file)
(let ((*package* (find-package :arm-assembler)))
(do ((form (read stream) (read stream nil :eof)))
((eq form :eof))
(eval form)))))
(Your implementation would probably be radically different, depending
on how you actually define the operators of your IL.)
Note however this all this does not implement any kind of security -
it is always possible to refer to standard Common Lisp operators
simply by using the operators full name, like in
(declare my-adder
(cl:delete-file "C:\\winnt\system32\kernel32.dll"))
If this is not acceptable, you'll have to implement a lot of
functionality yourself. Basically, the only thing you can use from the
standard CL toolset is READ to parse the textual representation of
your IL to a tree (but be sure to bind *READ-EVAL* to NIL if you don't
trust your input, otherwise one could write #.(delete
"C:\\winnt\system32\kernel32.dll") and the harm is done at read-time!),
everything after that would be up to you.
Henrik Motakef <············@henrik-motakef.de> wrote in
···················@crocket.internal.henrik-motakef.de:
> Jeff Massung <···@NOSPAM.mfire.com> writes:
>
>> When the IL compiler executes, it will interpret and write the binary
>> for the final image. I've already coded it in Forth, the theory is
>> sound and works great. The difficulty I'm having is that Forth is
>> rather obscure and difficult for my target audience. I'm hoping Lisp
>> will be easier for them.
>
> Heh. Lisp being easier/less obscure than anything else is not
> something you hear too often. :-)
Well, I don't plan on implementing ' #' ` , etc. Or anything else that
really makes CL as wonderful as it is. It really will be a lot like just
a regular assembler:
(db 13 "Hello, world!" 0)
(...
(begin '<=
(add r1 r1 4)
(cmp r0 0)))
There are other things planned, though that will make it a lot more
powerful than just a simple assembler.
> If you define a package like ARM-ASSEMBLER which does not use the
> COMMON-LISP package, and arrange for code in the immediate language to
> be read in that package, there won't be any name clashes. This would
> however make it neccessary that you use explicit package qualifiers
> for either the symbols in the COMMON-LISP package or those in
> ARM-ASSEMBLER. That is, given a package declaration like
When you say "does not use the COMMON-LISP package" do you mean that
inside of ARM-ASSEMBLER I couldn't make use of CL at all? or just that I
would need to prefix all CL functions with cl: ? BTW, is the package
"CL" the common name or does it change across implementations?
> Another thing is how you arrange for the IL code to be read in the
> correct package, i.e. the ARM-ASSEMBLER package. If you just use the
> Common Lisp LOAD function, [...]
Thanks for pointing that out. That is something I wouldn't have thought
of. I'll probably end up creating a very small package that only contains
functionality for loading and interpreting IL code.
> Note however this all this does not implement any kind of security -
> it is always possible to refer to standard Common Lisp operators
> simply by using the operators full name, like in
>
> (declare my-adder
> (cl:delete-file "C:\\winnt\system32\kernel32.dll"))
Is there any way to get around this? For example, can I override the
prefixing ability of CL?
Thanks for the help!
--
Best regards,
Jeff jma[at]mfire.com
http://www.jm-basic.com
Jeff Massung <···@NOSPAM.mfire.com> writes:
> Henrik Motakef <············@henrik-motakef.de> wrote in
> ···················@crocket.internal.henrik-motakef.de:
>
> > Note however this all this does not implement any kind of security -
> > it is always possible to refer to standard Common Lisp operators
> > simply by using the operators full name, like in
> >
> > (declare my-adder
> > (cl:delete-file "C:\\winnt\system32\kernel32.dll"))
>
> Is there any way to get around this? For example, can I override the
> prefixing ability of CL?
>
> Thanks for the help!
In order to get a good answer to your question, you need to spell out
your level of trust in the input. Is this trusted code, written by
competant users? Is this mostly trusted code, but you want to prevent
against accidental mistakes (ie, nothing sneaky, just normal
semi-competent user mistakes)? Or do you need to protect against
potentially clever malicious intent from your users?
All of these are solvable problems, and without too much difficulty,
but the approach you take will be different in those situations (eg,
in the first one, I'd consider it a possible benefit to be able to
access full CL if you need to).
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) wrote in
····················@famine.OCF.Berkeley.EDU:
> Jeff Massung <···@NOSPAM.mfire.com> writes:
>
>> Henrik Motakef <············@henrik-motakef.de> wrote in
>> ···················@crocket.internal.henrik-motakef.de:
>>
>> > Note however this all this does not implement any kind of security -
>> > it is always possible to refer to standard Common Lisp operators
>> > simply by using the operators full name, like in
>> >
>> > (declare my-adder
>> > (cl:delete-file "C:\\winnt\system32\kernel32.dll"))
>>
>> Is there any way to get around this? For example, can I override the
>> prefixing ability of CL?
>>
>> Thanks for the help!
>
> In order to get a good answer to your question, you need to spell out
> your level of trust in the input. Is this trusted code, written by
> competant users? Is this mostly trusted code, but you want to prevent
> against accidental mistakes (ie, nothing sneaky, just normal
> semi-competent user mistakes)? Or do you need to protect against
> potentially clever malicious intent from your users?
>
> All of these are solvable problems, and without too much difficulty,
> but the approach you take will be different in those situations (eg,
> in the first one, I'd consider it a possible benefit to be able to
> access full CL if you need to).
>
Good point. The end-goal is to have an IL (as stated before), but where
libraries in assembly could be created and imported for use. For example,
if a graphics library were created, it could be included and assembled
along with all the other code, giving access to new API functions.
Most of the users would probably never touch the source code. Right now
they have access to the Forth versions of the current libraries, and no
attempts have been made (except minor) to add functionality.
In some ways, it can be a toss-up. I would like to give access to the CL
package so that some very cool libraries could be created in ways that I
never thought of. At the same time, there is the possible malicious coder
that could prey on unsuspecting users (my user-base is generally under 20
years old).
Lastly, as stated I'm using Corman Lisp. I *don't* want to give access to
the Win32 API and compiler functionality of Corman Lisp to the end-user.
This (to me) would be very wrong -- basically giving away a Lisp compiler
that someone else made along with my product. The same problem exists
with the current Forth version, but I can hide the Forth vocabulary
(package) from the user so they can't use it at all.
Let me try and put some parameters around the problem, and perhaps the
solution will be more easy:
o The end user will *not* have access to read-print-eval
o At startup, a filename will be specified that is assembled
For example (given the name as.exe for the assembler):
c:>as.exe il_code.lisp
The only output to this program will be text output specifying success or
an error of some kind. So, if a simple function could be made that uses a
LOAD function of my own creation (that limits packages), this will
probably be good enough for what I want. If there is a function that
checks to see if a symbol is defined within a given package may work,
too.
One last possibility that I'm willing to look at is if this would be
easier done compiling to a DLL that I could then load into the C++
compiler... completely by-passing the second step. I don't know if this
would help at all, but thought I'd let you know that it is an option.
Thanks again, all, for the assistance.
--
Best regards,
Jeff Massung jma[at]mfire.com
http://www.jm-basic.com/
Jeff Massung <jma[at]nospam.mfire.com> writes:
> Let me try and put some parameters around the problem, and perhaps the
> solution will be more easy:
>
> o The end user will *not* have access to read-print-eval
> o At startup, a filename will be specified that is assembled
>
> For example (given the name as.exe for the assembler):
>
> c:>as.exe il_code.lisp
>
> The only output to this program will be text output specifying success or
> an error of some kind. So, if a simple function could be made that uses a
> LOAD function of my own creation (that limits packages), this will
> probably be good enough for what I want. If there is a function that
> checks to see if a symbol is defined within a given package may work,
> too.
I think what you want to do in this case is make your own interpreter
for this IL. A simple interpreter is very easy to write in Lisp: read
your source using CL:READ, with *READ-EVAL* bound to NIL; instead of
using the function and value slots on symbols, use a il-function and
il-variable properties; keep an alist for local function/macro
bindings, and one for local variable bindings; make sure you don't
accidentally make it dynamic scope (rebind your binding stacks on
function call). With this approach, the user can refer to
CL:DELETE-FILE all they want, but unless it has a function defined for
it in your IL, it won't do them any good. All in all, a simple
interpreter is <=100 LOC. Changing an interpreter to a naive compiler
that targets CL is pretty easy, too, if you're worried about the
overhead.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) wrote in
····················@famine.OCF.Berkeley.EDU:
> Jeff Massung <jma[at]nospam.mfire.com> writes:
>
>> Let me try and put some parameters around the problem, and perhaps
>> the solution will be more easy:
>>
>> o The end user will *not* have access to read-print-eval
>> o At startup, a filename will be specified that is assembled
>>
>> For example (given the name as.exe for the assembler):
>>
>> c:>as.exe il_code.lisp
>>
>> The only output to this program will be text output specifying
>> success or an error of some kind. So, if a simple function could be
>> made that uses a LOAD function of my own creation (that limits
>> packages), this will probably be good enough for what I want. If
>> there is a function that checks to see if a symbol is defined within
>> a given package may work, too.
>
> I think what you want to do in this case is make your own interpreter
> for this IL. A simple interpreter is very easy to write in Lisp: read
> your source using CL:READ, with *READ-EVAL* bound to NIL; instead of
> using the function and value slots on symbols, use a il-function and
> il-variable properties; keep an alist for local function/macro
> bindings, and one for local variable bindings; make sure you don't
> accidentally make it dynamic scope (rebind your binding stacks on
> function call).
Is there any sample code out there describing this process? While I've
used CL before, I've never re-bound system variables or done anything
like what you propose. Just a little website or push with even pseudo
code would be helpful.
Thanks.
--
Best regards,
Jeff jma[at]mfire.com
http://www.jm-basic.com
Jeff Massung <···@NOSPAM.mfire.com> writes:
> ···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) wrote in
> ····················@famine.OCF.Berkeley.EDU:
>
> > Jeff Massung <jma[at]nospam.mfire.com> writes:
> >
> >> Let me try and put some parameters around the problem, and perhaps
> >> the solution will be more easy:
> >>
> >> o The end user will *not* have access to read-print-eval
> >> o At startup, a filename will be specified that is assembled
> >>
> >> For example (given the name as.exe for the assembler):
> >>
> >> c:>as.exe il_code.lisp
> >>
> >> The only output to this program will be text output specifying
> >> success or an error of some kind. So, if a simple function could be
> >> made that uses a LOAD function of my own creation (that limits
> >> packages), this will probably be good enough for what I want. If
> >> there is a function that checks to see if a symbol is defined within
> >> a given package may work, too.
> >
> > I think what you want to do in this case is make your own interpreter
> > for this IL. A simple interpreter is very easy to write in Lisp: read
> > your source using CL:READ, with *READ-EVAL* bound to NIL; instead of
> > using the function and value slots on symbols, use a il-function and
> > il-variable properties; keep an alist for local function/macro
> > bindings, and one for local variable bindings; make sure you don't
> > accidentally make it dynamic scope (rebind your binding stacks on
> > function call).
>
> Is there any sample code out there describing this process? While I've
> used CL before, I've never re-bound system variables or done anything
> like what you propose. Just a little website or push with even pseudo
> code would be helpful.
Oh, you don't need to touch anything in the guts of the host system.
Here's a very simple evaluator for a lexically scoped Lisp, which
should give you the idea. You can build a reasonable Lisp up from
this using macros. Well, assuming you also fix it's lousy handling of
lambda-lists :-)
I know that _Paradigms of Artificial Intelligence Programming_ covers
simple interpreters and compilers, and Lisp In Small Pieces as well.
Note that because functions are represented with the host CL system's
functions, you can call actual CL functions to do the work of IL
functions and macros, if you make them available to the IL world.
;;; Typed directly into the post, untested
(defconstant +unbound+ '+unbound+)
(defvar *value-bindings* ())
(defvar *function-bindings* ())
(defun get-binding (var type)
(let* ((stack (ecase type
(il-value *value-bindings*)
(il-function *function-bindings)))
(cons (assoc var stack))
(result (if cons (cdr cons)
(get var type +unbound+))))
(if (eq result +unbound+)
(error "Unbound ~S ~S" type var)
result)))
(defun (setf get-binding) (newval var type)
(let* ((stack (ecase type
(il-value *value-bindings*)
(il-function *function-bindings)))
(cons (assoc var stack)))
(if cons (setf (cdr cons) newval)
(setf (get var type) newval))))
(defun il-eval (form)
(cond
((symbolp form) (get-binding form 'il-value))
((not (consp form)) form)
(t (case (car form)
(quote (second form))
(if (if (il-eval (second form))
(il-eval (third form))
(il-eval (fourth form))))
(lambda (let ((lambda-list (second form))
(body (cddr form))
(lexenv (list *value-bindings* *function-bindings*)))
(lambda (&rest args)
;; restore saved binding stacks, so we're lexically scoped!
(progv '(*value-bindings* *function-bindings*) lexenv
;; fixme: fixed position params only in lambda-list
(assert (= (length lambda-list) (length args)))
(loop for var in lambda-list and arg in args
do (push (cons var arg) *value-bindings*))
(loop for form in body
for result = (il-eval form)
finally (return result))))))
(setq (setf (get-binding (second form) 'il-value)
(il-eval (third form))))
(t ;; function call or macro-expand
(destructuring-bind (type fn) (get-ginding (car form) 'il-function)
(ecase type
(:function (let ((args (mapcar #'il-eval (cdr form))))
(apply fn args)))
;; These macros recieve the whole form.
(:macro (let ((new-form (funcall fn form)))
(il-eval new-form))))))))))
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Oops, lunch break resumed momentarily.
···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
[ In il-eval ]
> (t ;; function call or macro-expand
> (destructuring-bind (type fn) (get-ginding (car form) 'il-function)
> (ecase type
> (:function (let ((args (mapcar #'il-eval (cdr form))))
> (apply fn args)))
> ;; These macros recieve the whole form.
> (:macro (let ((new-form (funcall fn form)))
> (il-eval new-form))))))))))
You can also add special forms implemented in CL by adding this case
to the ECASE above:
(:special-form (funcall fn form))
Eg, to add FLET:
(defun eval-flet (form)
(destructuring-bind ((&rest flets) &rest body) form
(loop for (name lambda-list body) in flets
when (let ((binding (get-binding name 'il-function)))
(and (consp binding) (eq (first binding) :special-form)))
do (error "Trying to shadow the special form ~S" name)
collect (list name :function (make-il-closure lambda-list body))
into bindings
finally (setf *function-bindings*
(nconc bindings *function-bindings*)))
(loop for form in body
for result = (il-eval form)
finally (return result))))
(setf (get-binding 'flet 'il-function)
(list :special-form 'eval-flet))
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Jeff Massung <jma[at]nospam.mfire.com> writes:
> Good point. The end-goal is to have an IL (as stated before), but where
> libraries in assembly could be created and imported for use. For example,
> if a graphics library were created, it could be included and assembled
> along with all the other code, giving access to new API functions.
>
> Most of the users would probably never touch the source code. Right now
> they have access to the Forth versions of the current libraries, and no
> attempts have been made (except minor) to add functionality.
Hm. I think if your users need to use libraries that they cannot
trust, you are in deep trouble, which is very hard to solve by
technical means. The evildoers could always include some malicious
code in their library that works as a trojan horse in the generated
output, without messing with CL at all.
> Let me try and put some parameters around the problem, and perhaps the
> solution will be more easy:
>
> o The end user will *not* have access to read-print-eval
> o At startup, a filename will be specified that is assembled
>
> For example (given the name as.exe for the assembler):
>
> c:>as.exe il_code.lisp
>
> The only output to this program will be text output specifying success or
> an error of some kind. So, if a simple function could be made that uses a
> LOAD function of my own creation (that limits packages), this will
> probably be good enough for what I want. If there is a function that
> checks to see if a symbol is defined within a given package may work,
> too.
There is such a function. The "home package" of a symbol is accessible
through symbol-package, and the other packages where it can be used without
prefix through (package-used-by-list *home-package*).
As for your home-grown LOAD function, it could read the file, then
check for validity, and only if all symbols used are OK go on and
process them in whatever way you want them to be processed.
For example (untested):
(defconstant +allowed-symbols+ '(+ - * / cons car cdr ...))
(defun symbol-allowed-p (symbol)
(or (eql (symbol-package symbol) (find-package :asm-user))
(eql :external (nth-value 1 (find-symbol (symbol-name symbol) :asm)))
(member symbol +allowed-symbols+)))
(defun form-allowed-p (form)
(typecase form
(symbol (symbol-allowed-p form))
(list (every #'form-allowed-p form))
((or number string pathname) t)
(otherwise nil)))
(defun read-il-file (pathname)
(with-open-file (stream pathname)
(with-standard-io-syntax
(let ((*read-eval* nil)
(*package* (find-package :asm-user)))
(loop for form = (read stream nil :eof) then (read stream nil :eof)
until (eql form :eof)
collecting form)))))
(defun load-il-file (pathname)
(let ((forms (read-il-file pathname)))
(if (not (every #'form-allowed-p forms))
(error "Boo!")
(mapcar #'eval forms))))
This would allow people to use symbols that are either in the
:asm-user package, which is where unprefixed symbols are read in, are
exported from the :asm package (where you'd implement your stuff), or
that are explicitly allowed by inclusion in +allowed-symbols+. The
hard part would be figuring out which symbols to allow - you probably
wouldn't want to allow the whole CL package, especially not things
like LOAD, INTERN, EVAL, READ-FROM-STRING etc., as well as stuff that
manipulates packages (INTERN, USE-PACKAGE,...). A problem is how to
allow SETF, but prevent +allowed-symbols+ from being messed with - I
don't know how picky Corman Lisp is with regard to changing constants,
many implementations provide a "go on and change the constant anyway"
restart.
You could of course also implement a more sophisticated way of
checking things. Googling for "code walker" will probably turn up some
inspirations.
Henrik Motakef wrote:
> Hm. I think if your users need to use libraries that they cannot
> trust, you are in deep trouble, which is very hard to solve by
> technical means. The evildoers could always include some malicious
> code in their library that works as a trojan horse in the generated
> output, without messing with CL at all.
I think that proof-carrying code could help.
Pascal
--
Pascal Costanza University of Bonn
···············@web.de Institute of Computer Science III
http://www.pascalcostanza.de R�merstr. 164, D-53117 Bonn (Germany)
Henrik Motakef wrote:
>
> Jeff Massung <jma[at]nospam.mfire.com> writes:
>
> > Good point. The end-goal is to have an IL (as stated before), but where
> > libraries in assembly could be created and imported for use. For example,
> > if a graphics library were created, it could be included and assembled
> > along with all the other code, giving access to new API functions.
> >
> > Most of the users would probably never touch the source code. Right now
> > they have access to the Forth versions of the current libraries, and no
> > attempts have been made (except minor) to add functionality.
>
> Hm. I think if your users need to use libraries that they cannot
> trust, you are in deep trouble, which is very hard to solve by
> technical means. The evildoers could always include some malicious
> code in their library that works as a trojan horse in the generated
> output, without messing with CL at all.
What is really needed for untrusted code to be run safely is
a decent security model for the operating system. Specifically,
you need a capability-based security model, where a program has
to request permission to do something in order to get an interface
to do it, and the kernel can revoke that permission without the
program's cooperation by simply disconnecting the interface that
the program has access to.
This would also mean that the kernel could maintain a complete
and correct database of which programs have what capabilities
at all times, so you can go look and see what program had requested
the authority to write /opt/packages/foo/ if you found that
something funny had been written there.
The problem is that, unless you design your capabilities system
carefully and meaningfully, this can become a huge hassle.
Bear
In article <···············@corp.supernews.com>,
Jeff Massung <···@NOSPAM.mfire.com> wrote:
> > (declare my-adder
> > (cl:delete-file "C:\\winnt\system32\kernel32.dll"))
>
> Is there any way to get around this? For example, can I override the
> prefixing ability of CL?
Not without completely replacing the reader.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
In article <····························@netnews.comcast.net>, Barry Margolin wrote:
> In article <···············@corp.supernews.com>,
> Jeff Massung <···@NOSPAM.mfire.com> wrote:
>
> > > (declare my-adder
> > > (cl:delete-file "C:\\winnt\system32\kernel32.dll"))
> >
> > Is there any way to get around this? For example, can I override the
> > prefixing ability of CL?
>
> Not without completely replacing the reader.
>
Couldn't he use the CL reader, then scan the resulting list for
symbols not in the current package, either raising an error
or replacing these symbols with something else?
Of course that won't solve all problems with executing untrusted code.
He would still have to deal with things like infinite loops and
memory hogging. A solution would be not to allow looping constructs :-)
Eric Daniel
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----
Eric Daniel <···········@barberic.vancouver.wa.us> writes:
> Couldn't he use the CL reader, then scan the resulting list for
> symbols not in the current package, either raising an error
> or replacing these symbols with something else?
#.(cl::delete-file ...)
--
Rahul Jain
·····@nyct.net
Professional Software Developer, Amateur Quantum Mechanicist
In article <··············@nyct.net>, Rahul Jain wrote:
> Eric Daniel <···········@barberic.vancouver.wa.us> writes:
>
> > Couldn't he use the CL reader, then scan the resulting list for
> > symbols not in the current package, either raising an error
> > or replacing these symbols with something else?
>
> #.(cl::delete-file ...)
>
Well (let ((*read-eval* nil)) ...) was implied :-)
But I can think of other drawbacks to using the Lisp reader
for reading untrusted code:
- symbol pollution. The code being read can cause a bunch
of symbols to be interned in other packages, making it difficult
to clean them up afterwards.
- depending on the input stream (a socket for example) it may be
possible to cause (read) to exhaust all memory or never return.
I don't know if these problems are relevant to the original poster.
But it does seem that in the general case one can't use the Lisp
reader as part of an application server.
Not that writing your own simple reader for a Lisp-like syntax
is difficult mind you.
Eric Daniel
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----
Jeff Massung <···@NOSPAM.mfire.com> writes:
> > If you define a package like ARM-ASSEMBLER which does not use the
> > COMMON-LISP package, and arrange for code in the immediate language to
> > be read in that package, there won't be any name clashes. This would
> > however make it neccessary that you use explicit package qualifiers
> > for either the symbols in the COMMON-LISP package or those in
> > ARM-ASSEMBLER. That is, given a package declaration like
>
> When you say "does not use the COMMON-LISP package" do you mean that
> inside of ARM-ASSEMBLER I couldn't make use of CL at all? or just that I
> would need to prefix all CL functions with cl: ?
The latter. I meant "use" as in USE-PACKAGE, PACKAGE-USE-LIST or the
:USE-keyword to DEFPACKAGE. That is, if your package uses the
COMMON-LISP package, all symbols in this package are also accessible
in your package without prefixing them. Using prefixed symbols works
pretty much always - remember that packages are a way to implement
namespaces, not modules. They are not about strong "information
hiding".
> BTW, is the package "CL" the common name or does it change across
> implementations?
The package is called COMMON-LISP and nicknamed CL. That means that
every conforming implementation will recognize both COMMON-LISP:DEFUN
and CL:DEFUN as the same symbol, the one that happens to be used to
define functions. It is used by COMMON-LISP-USER a.k.a CL-USER ("used"
in the sense outlined above) which is the package you are in by
default in the REPL.
In case you are not aware, and for the sake of the google groups
archive otherwise, this is discussed in the Hyperspec section 11.1
"Package Concepts"
(http://www.lispworks.com/reference/HyperSpec/Body/11_a.htm).
> > Note however this all this does not implement any kind of security -
> > it is always possible to refer to standard Common Lisp operators
> > simply by using the operators full name, like in
> >
> > (declare my-adder
> > (cl:delete-file "C:\\winnt\system32\kernel32.dll"))
>
> Is there any way to get around this? For example, can I override the
> prefixing ability of CL?
Hm. There may be a way to change the semantics of : (and ::) in the
readtable you use (the readtable, an opaque object stored in the
*READTABLE* special variable, can be used to change the CL syntax in
rather funky ways, see most of section 2 in the hyperspec for more
info), but I seem to remember that this isn't as easy as it would
seem. If there is, SET-SYNTAX-FROM-CHAR will probably be a good
starting point.
Unless someone who knows more about that than I do speaks up, I'd try
to find another solution to that problem - CL really isn't about
strong, uncircumventable enforcement of package boundaries. The
standard mechanisms mostly adopt a spirit of letting you know when you
do stuff you are not supposed to do, but let people who know what they
are doing do the dirty stuff anyway. If you need a stricter policy,
prepare to implement some things on your own.
Henrik Motakef <············@henrik-motakef.de> wrote in
···················@crocket.internal.henrik-motakef.de:
> Jeff Massung <···@NOSPAM.mfire.com> writes:
>
>> Is there any way to get around this? For example, can I override the
>> prefixing ability of CL?
>
> Hm. There may be a way to change the semantics of : (and ::) in the
> readtable you use (the readtable, an opaque object stored in the
> *READTABLE* special variable, can be used to change the CL syntax in
> rather funky ways, see most of section 2 in the hyperspec for more
> info), but I seem to remember that this isn't as easy as it would
> seem. If there is, SET-SYNTAX-FROM-CHAR will probably be a good
> starting point.
What about something easier? Not that I mind going the hard route, just
trying to hit all the bases before making a decision to move forward. Is
is possible to redefine a package (override the name?)
I know that I can do:
(defconstant x 1)
(defconstant x 2)
It works, there is just a warning generated by the compiler. What about
just doing:
(defpackage arm-assembly (:use))
(cl:defun read (path)
(cl:read path)) ; Making use of only arm-assembly
;; At end of code...
(defpackage cl)
And then not having anything inside of it, essentially overwriting the
function table for the first. Would this produce the desired results?
--
Best regards,
Jeff jma[at]mfire.com
http://www.jm-basic.com
In article <···············@corp.supernews.com>,
Jeff Massung <···@NOSPAM.mfire.com> wrote:
> (defpackage arm-assembly (:use))
>
> (cl:defun read (path)
> (cl:read path)) ; Making use of only arm-assembly
>
> ;; At end of code...
> (defpackage cl)
>
> And then not having anything inside of it, essentially overwriting the
> function table for the first. Would this produce the desired results?
I'm not sure if the language spec ever specifically says it, but I find
it hard to believe that we considered the consequences to be
well-defined if you destroy any of the standard packages. Just as
you're not allowed to redefine standard functions, changing the meaning
of standard package names shouldn't be expected to be safe.
Anyway, even if you obliterate the CL and COMMON-LISP packages, there
may be implementation-specific packages that the untrusted user could
access. In fact, it's a common implementation technique for all the
standard functions to be defined in some internal package, then imported
and re-exported from the COMMON-LISP package. E.g. something like:
(in-package :system-internals)
(defun read ...)
(defpackage common-lisp
(:import-from :system-internals ... read ...)
(:export ... read ...)
...)
So even though you might have destroyed cl:read, they can still access
system-internals::read.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Henrik Motakef <············@henrik-motakef.de> wrote:
>> Is there any way to get around this? For example, can I
>> override the prefixing ability of CL?
> Hm. There may be a way to change the semantics of : (and
> ::) in the readtable you use (the readtable, an opaque
> object stored in the *READTABLE* special variable, can be
> used to change the CL syntax in rather funky ways, see
> most of section 2 in the hyperspec for more info), but I
> seem to remember that this isn't as easy as it would seem.
A few months ago I found out what was confusing me about
*READTABLE* and *RANDOM-STATE*. I had mostly been writing
code in a functional style, and had missed a key point about
setf and defstruct. Assignment in C copies the top level of
a structure for you. setf is not the same as C's = . setf
shares structures. If you need a copy you have to ask to it.
After (defstruct pair l r)
(let ((a (make-pair :l 1 :r 2)) b)
(setf b a)
(setf (pair-l b) 3)
(list a b))
=> (#S(PAIR :L 3 :R 2) #S(PAIR :L 3 :R 2))
^
|
`- L in A has changed!
(let ((a (make-pair :l 1 :r 2)) b)
(setf b (copy-pair a))
(setf (pair-l b) 3)
(list a b))
(#S(PAIR :L 1 :R 2) #S(PAIR :L 3 :R 2))
^
|
`- L in A left alone
After I recovered from the culture shock I realised that I
knew the problem that (copy-readtable) was designed to
solve. With this vital context, the documentation for
(copy-readtable) made sense, and (set-macro-character)
started to work for me.
(defpackage test (:export a))
(setf test:a 'secret)
test:a => SECRET
(set-macro-character #\: #'(lambda(stream character) #\:))
test:a => *** - EVAL: variable TEST has no value
1. Break [6]>
but
(symbol-value (intern "A" "TEST")) => SECRET
(symbol-value
(with-standard-io-syntax
(read-from-string "test:a")))
=> SECRET
It seems perfectly practical to disable : so as to prevent
accidents and discourage mucking about.
Can one provide real security by putting the user in a
package that only imports selected symbols from COMMON-LISP
and disables #\: ? This seems like quite an important
question now that the World Wide Web encourages software
architectures that download and run snippets of code from
strangers. One would have to leave out
(with-standard-io-syntax), and the readtable functions, and
put in a limited version of intern without the optional
package argument. I've no feel for whether this is practical
or whether there would be an endless succession of security
holes.
Alan Crowe
Edinburgh
Scotland
Alan Crowe <····@cawtech.freeserve.co.uk> wrote in message news:<··············@cawtech.freeserve.co.uk>...
>
> Can one provide real security by putting the user in a
> package that only imports selected symbols from COMMON-LISP
> and disables #\: ? This seems like quite an important
> question now that the World Wide Web encourages software
> architectures that download and run snippets of code from
> strangers. One would have to leave out
> (with-standard-io-syntax), and the readtable functions, and
> put in a limited version of intern without the optional
> package argument. I've no feel for whether this is practical
> or whether there would be an endless succession of security
> holes.
I'm not sure you can disable :, since I think it's completely magic -
it's not a reader macro, its part of the symbol parser, which isn't
exposed to you. I might be wrong about this. I'm sure this has been
argued endlessly on cll: if I was less lazy I'd go back and check,
and/or read the spec (which I don't actually have to hand, or at least
that's my excuse).
What you can do is to read forms in a `good' package and with `safe'
reader settings (no read-time eval!), and then check what you get for
things you don't want. To do this you have to walk over the form,
looking for bad things, which include symbols you don't want, and so
on. You need an occurs check, and you need to know how to descend any
object that you allow (so you likely need to reject arbitrary
structures...).
Once you've done that then what you have left should be safe. `safe'
here means `relatively safe': just because it uses `good' things
doesn't have to mean it terminates, and I think that there are no hard
promises that errors are actually signalled in all cases. Finally
you're assuming that the reader can actually take anything you throw
at it without blowing up in some bad way, which it should, but may
not. I wouldn't want to put such a thing up on the web without a lot
of testing.
My conduit package thing was originally part of a little toy system
which allowed you to define a `good' (and small!) subset of CL for
this purpose. You'd define a conduit package with just safe things
from CL in it, and then the form walker would just check for symbols
that were interned this package.
--tim
PS you also may need to worry about flags being left by code in the
form of interned symbols.
Tim Bradshaw wrote:
> I'm not sure you can disable :, since I think it's
> completely magic - it's not a reader macro, its part of
> the symbol parser, which isn't exposed to you.
I've convinced myself that the spec does permit disabling
of: even though it is magic and not a reader macro.
I think it hinges on the difference between Constituent
Character Attributes and Character Syntax Types.
Constituent Character Attributes cannot be changed, but they
do get turned of whenever a character is escaped
foo\:bar == |FOO:BAR|
* (symbol-package 'a:b\:c)
#<The A package, 0/7 internal, 2/2 external>
and more importantly they get shadowed when the Character
Syntax is other than constituent. So setting the syntax of :
to be macro-character makes the constituent attribute go
away because : isn't a constituent any more.
I expect the code I posted early to suppress : to work
everywhere and not just on CMUCL.
I wish Erik Naggum were here. He would know for sure.
Alan Crowe
Edinburgh
Scotland
* Alan Crowe wrote:
> I've convinced myself that the spec does permit disabling
> of: even though it is magic and not a reader macro.
I think this is right - you can make #\: not work as a package
marker. What you can't do (I think) is make something else be a
package marker, or make #\: be a real constituent - so
(read-from-string "foo:bar") -> |FOO:BAR| for instance.
Of course the problem with doing this is that this will break
keywords, and thus half of CL. I suppose you could make it so that it
read a KW symbol or something, but it would be pretty ick.
--tim