A first PR for a standard lib

[dhsorens]

This is a first iteration on a standard library. As I am new to programming in Lisp, please make comments about style/form/correctness/anything else liberally.

This is a first pass at a standard set of useful functions. They're not necessarily optimized for efficiency. They might not be fully correct either. Some of them might not even be useful. But they're here as a first brain-dump on what a std-lib could look like. Very happy to collaborate on this to add/eliminate some of the functions.

Some TODOs:

• Still need to write several of the tests. Right now, only the tests on bool are fully done.
• Add std functions such as cadr, cddr, reverse, nth, last, etc.
• Add take, drop, filter

[weissjeffm]

Very impressive, you have a lot of good functions there!

One that I would like to see but I suspect it needs to be implemented in rust, is apply. (which takes a function f, and a list p, and calls f with the arguments taken from the list p.)

[weissjeffm]

I think this can just be called length. I think that matches common lisp. The same probably goes for most of these other functions - you usually don't need to have list in the name.

Also I think we have automatic currying here, so you can try

(letrec ((length_iter (lambda (cntr lst)
			(if (car lst)
			    (length_iter (+ 1 cntr) (cdr lst))
			    cntr)))
	 (list_length (length_iter 0)))
  (current-env))

[dhsorens]

Would you say the same goes for list-eq?

[porcuquine]

list-eq is just eq. There's no need for such a function at all.

[porcuquine]

Very impressive, you have a lot of good functions there!

One that I would like to see but I suspect it needs to be implemented in rust, is apply. (which takes a function f, and a list p, and calls f with the arguments taken from the list p.)

APPLY would be fun to write, and may actually be pretty simple because of Lurk's currying. I had been assuming we would support it in the core language, which might still make sense. But it would be well worth implementing here.

[porcuquine]

LIST probably requires a macro or language support to be done right (variable number of arguments).

Explicit use of EVAL is probably not desirable, but if it were, you probably want the current env to be involved.

I'm chalking this up to probably not being a useful function as written.

[weissjeffm]

I came up with that one, and i had a newer version that uses current-env, so that you can do nested lists.

         (list (lambda (q)
                 (if (car q)
                     (cons (eval (car q) (current-env))
                           (list (cdr q)))
                     nil)))

I don't see any other way to produce a literal list (that's not quoted, but rather requires eval'ing args). I mean, aside from a bunch of consing. Obviously using eval is a terrible hack and yes list is supposed to be varargs. Edit: but i think it's useful until there's a native list.

[porcuquine]

I think this is a clue that the concept involved is off. If we really want a function that takes a list and outputs a list, we can just use the identity function.

Normal Lisp LIST, apart from being variadic, doesn't perform explicit evaluation. It just so happens that the args are evaluated before LIST ever sees them.

For example, here is how SBCL defines LIST:

(defun list (&rest args)
  "Construct and return a list containing the objects ARGS."
  args)

[weissjeffm]

Yes, I understand this.

What I really want is lisp's LIST, but I don't think that's possible to define in lurk itself right now. I'm sure you understand why I want it, it's for the same purpose everyone uses LIST in lisp - to make a literal list with expressions in it that I want evaluated first. Without that the only way I can think to make such a list is with repeated consing.

[porcuquine]

Yep, I understand. I think one urge is for Lisp's LIST, which can indeed be implemented with macros or when we have support for &REST parameters.

I think you also want something that is definitionally not LIST, but that would accomplish a practical purpose. That something is basically MAP-EVAL of some flavor — which is fine.

I just would not call it LIST, since that's confusing and also will clash with some future implementation of LIST which works as it should.

[weissjeffm]

Ok yes I see your point, I started out trying to implement LIST but realized it's not possible and then kept the name on something that's not the same. What I ended up with is basically (map eval ...) like you said. I am fine with whatever it is not being in the stdlib but I will need something like this as a utility function at least until there's something better. I am ok with just copying it around. Putting it in the stdlib is probably bad because later it will have no purpose and should go away.

[porcuquine]

I'm not even saying not to have such a function. I'm just saying don't call it LIST. You could call it MAP-EVAL or something.

[weissjeffm]

Nah, if we have map (which we will), writing (map eval lst) is too trivial to be worth creating a new function.

[porcuquine]

I think it might be a little more complicated than you want to try to inline by hand. Remember:

• EVAL isn't a function
• you need the correct environment

As a test, you want to make sure something like this works:

(let ((a 1))
  (do-the-thing '(a 2 (+ a 2))))

I think for that to work, you probably need to also pass (current-env) to do-the-thing.

[weissjeffm]

Same here - suggest the op argument goes first so you can do this:

  (letrec 
      ((list_fold (lambda (op accum lst)
          (if (car lst)
              (list_fold op (op (car lst) accum) (cdr lst))
              accum)))
       (sum (list_fold (lambda (x y) (+ x y)) 0)))
    (sum '(1 2 3 4 5)))
=> 15

(by the way I notice + isn't a first class function, so I had to introduce the lambda there, not sure if this is something I should open as a separate lurk issue or if it's intentional.)

[porcuquine]

No, it's how it works. All the built-in operators are hardcoded in the circuit and are not actually 'functions'. We could define functions bound to the symbols in a library in the future, as a convenience — or not. Depending on how details of future versions evolve, along with the circuit, it's possible we revisit this, but for now it's how things are by design.

[dhsorens]

Ah good to know, I had the same question about +

[dhsorens]

Re: implementing apply, as I understand it (apply f '(x y z ...)) evaluates to something equal to (f x y z ...) (I may have this wrong). In particular, it fails if you give it the wrong number of arguments.

You can also write (apply f x y '(z ...)) which is the same as (apply (f x y) '(z ...)).

If this ^^ is accurate, then I just pushed a version of apply implemented that accepts a function and a list and executes the semantics of the first paragraph. I'm not sure how to write a function which takes variable numbers of args. Do we need macros for that sort of thing?

With what is implemented, writing the usual notion of (apply f 1 2 3 '(4 5 6)) would just need to be written (apply (f 1 2 3) '(4 5 6))

[weissjeffm]

Haven't tried it yet but this looks right. We can't have varargs currently afaik, so just can handle the first case you outlined. You also just have to assume the user passed in a list that is no longer than the function's arglist (otherwise it's undefined behavior). I think that's the best we can do for apply written in lurk. We can't tell how many args a function should take, so we can't check error conditions, and I don't think lurk has a way to signal error yet anyway. Maybe for convenience we could write apply1 apply2 etc that takes n individual items before the list, but it's not necessary imo. We do need to think about what to call this though, since the real apply is varargs. Maybe we should name this something slightly different?

[weissjeffm]

Shouldn't this whole file be one big letrec? As it stands, aren't we returning a bunch of different envs, each with one function in it? Maybe I'm misunderstanding what (current-env) does.

[weissjeffm]

Also what is the difference between list-eq and eq?

> (eq '(a (b)) '(a (b)))
[3 iterations] => T

[dhsorens]

haha, that's funny, somehow I thought that eq didn't work on bool and on lists, but it works on both! These are removed

[dhsorens]

re it being one big letrec or several, that's a good question. I don't know what convention is; as I understand it, as written each let and letrec updates the current environment to include that variable binding. I've only bundled related/helper functions into the same let or letrec. @porcuquine thoughts on this stylistically?

[weissjeffm]

Maybe (current-env) doesn't do what I thought.

I guess it is not only returning the env at that point, it's also setting the top level's state to include all the variables from that point. In that case the name is a little misleading, the name sounds like it doesn't mutate any state. I'd call it something like capture-env, or use-env or set-top-level-env etc.

[porcuquine]

(current-env) returns the current lexical environment.

Separately, when a .lurk file is processed by !(:load …), the toplevel lexical environment used by the REPL is replaced with the result of evaluating each form encountered.

This has the net effect that sequential forms, each extending the environment, can be used to build up an environment containing all the definitions. This trick allows us to define libraries in different files than use them, and for libraries to depend on other libraries. It also allows more modular definitions within a single file.

This is mostly a hack that won't be so important once the RAM subset is available, but it certainly makes things easier in the current world, which we do want and need to support.

[weissjeffm]

So @porcuquine what's your thought on whether to define the lib in one big letrec or a bunch of different ones? I lean toward just using one (assuming that would work just as well).

[porcuquine]

Probably multiple, but logically grouped. I'm not 100% sure, but that's my instinct. That gives us an extra layer of structure and will probably help as intermediate stage toward breaking sections out into their own file/module/whatever. It will also facilitate re-ordering for performance.

[porcuquine]

We do need to think about what to call this though, since the real apply is varargs. Maybe we should name this something slightly different?

Maybe apply1? We could use a similar naming strategy for other cases following this pattern.

[porcuquine]

Why are you checking to see if the car of the list is NIL? NIL shouldn't be treated specially when treated as an argument to f.

[weissjeffm]

I think that is intended as a check to see if the list is empty. I don't know if (if list ...) is falsey when the list is empty, but that would be nice.

What you're getting at though is that nil is a valid list item, right?

[porcuquine]

Yes, that's right.

To your other point, NIL is the empty list, and in Lurk as in Common Lisp, NIL is the false value.

[porcuquine]

In general, it only makes sense to be checking to see if a list is NIL (when traversing lists). If you're checking the CAR, that's probably wrong. Even if traversing a binary tree, you probably put the NIL check at the top of the function.

This is just a heuristic I'm noting since @durlicc is new to Lisp.

[weissjeffm]

I think we should probably either leave these out, or name them something else. The "real" and and or should be macros that have short-circuit logic (in or, if the first arg evals to true, it immediately returns true without evaluating the 2nd arg).

[dhsorens]

That sounds reasonable to me 👍

[porcuquine]

For stylistic, consistency, let's use ACC for accumulator variable. I'd personally use L for lists also, but I feel less strongly about that. LST doesn't feel very Lispy. LIST would be better, but more verbose (hence my preference for L).

[weissjeffm]

I don't like list because it collides with the function list. Although that brings up the question of whether lurk is a lisp-1 or lisp-2. let's see:

> (let ((a (lambda () 2))
	(a 1))
    (a))
[6 iterations] => ERROR!

lisp-1. So naming collision is a problem for LIST. I agree with L then.

[porcuquine]

This is treating NIL specially and will terminate the computation if a NIL is encountered in LST.

[porcuquine]

As I think I mentioned earlier, NIL checks on the CAR of a list when recursing are usually a sign of confused logic (not an absolute rule, but it holds in this case).

[dhsorens]

Yeah, good point, I have to revisit a lot of the functions I wrote to correct this logic error 🤔

[porcuquine]

As an aside, I'm providing an example which implements this slightly differently, in a way that is more efficient for Lurk. We don't necessarily need to use this, but it may be instructive. I'll provide output showing the difference in iteration count so you see the point.

> (let ((fold (lambda (op acc l)
              (letrec ((f (lambda (acc l)
                            (if l
                                (f (op acc (car l)) (cdr l))
                                acc))))
                (f acc l)))))
  (fold (lambda (a b) (+ a b))
        0
        '(1 2 3 4 5 6 7 8 9 10)))
[415 iterations] => 55

Compare with the cleaned-up version of the above code:

[225 iterations] => 15
> (letrec ((fold (lambda (op acc l)
                 (if l
                     (fold op (op acc (car l)) (cdr l))
                     acc))))
  (fold (lambda (a b) (+ a b))
        0
        '(1 2 3 4 5 6 7 8 9 10)))
[460 iterations] => 55

[dhsorens]

I just pushed some functions for trees. I'm assuming these are not implemented well, as they're my first pass at these kinds of things. I will be iterating on these to make them nicer over time (also, comments warmly welcome).

A tree is defined by expressions of the form (cons br1 br2) where br1 and br2 are of the form (cons br1' br2') if they are non-leaf subtrees, or (cons nil val) if they are leafs of value val. Leafs are implemented this way to be able to distinguish from a non-leaf branch and a leaf-branch, since for a leaf br, (= nil (car br)).

Sets are implemented as balanced trees, though nested loads don't work?