Proposal for Lisp as the Extension Mechanism

[habruening]

On the contribution page you invite users to make proposals. I have a proposal how the extension mechanism could be. Sorry, it is nearly an essay. But the proposal contains very unusual ideas. It needs a few more words to explain why I make this proposal.

Problem Statement

The challenge is to provide an extension mechanism with the following properties.

• The programming language for extensions shall be a versatile real programming language, that is powerful enough for all sorts of extensions.
• The programming language shall be easy to learn. Easy things shall be easy without big overhead, and complicated things shall be possible.
• Installing an extension shall not require a complicated installation, compilation or setup.
• Extensions shall be delivered as source code and not as something like native binaries.
• Extensions shall be as fast as native code.
• Extensions shall be fully interoperable with the Rust world.
• Extension need a safe runtime environment, so that the main editor parts and the Rust world is protected from them.
• Zed shall be "hackable". The extension mechanism shall be very dynamic. Adding new functions, new menus, new languages, new widgets etc. shall be possible at any time.

Solution Proposal

These requirements are hard to fulfil with standard programming languages.

I propose the programming language Common Lisp for the extension mechanism.

Rationale

• Lisp is easy to learn. If you compare Lisp to other languages, Lisp has fewer language concepts. The only structuring concept are parenthesis. If you look at typical JavaScript code, for example an anonymous function with curley braces as a function argument in parenthesis and commata and simocolon, you wish you had a syntax that makes this easier and not more complicated than it is. With Lisp You get this easier.

• Lisp is an interactive programming environment. A text editor is interactive text processing. Both fit well together. Example: With multi cursor editing you edit a Java file. You place a cursor at each variable assignment. Now you want to reduce this selection to those assignments, where the target variable is a global variable. Normally this is impossible with text editors. You would have to ask the LSP in order to find this out, and then you have to reduce the selections dependong on what comes out of this query. You need a small programming interface for such such editing problems. Lisp gives you this.

• Lisp is very high level. You only have parenthesis. You can write an expression like (open-file :file-name) and it can be anything. It can be an option in a configuration file. It can be en entry in a menu. It can be a function that you call. Or maybe it is a list. It can be all together, because it is just a symbol followed by a keyword in parentheses. You can define a function with the name open-file and then execute the expression. These ideas allow us to write program code that completely abstracts away the implementation details. And that is what we need for an extension mechanism. If someone writes (open-file :file-name) in the configuration file that configures for example a menu, everybody knows what this means, but nobody cares what it technically is. But if you look at a typical C thing like open_file("filename") everybody in the world recognises this as a function call, and nobody would like to have such things written in a human readable configuration file. So if you want to make this configurable you are forced to invent an ad-hoc language, while in Lisp you all the time work like this.

• The reason why I propose Lisp is that Lisp feels like interpreted, but technically it is a compiled language. This is what an extension mechanism meeds. You get the advantage of an interpreted language, where extensions are deliverd as source code only. But you also get the advantage of a compiled language with natively compiled fast code. This works with a concept that no other programming language offers: When you create an executable or a library with Lisp, the whole compiler is normally baked into it. The whole program execution is a dynamic thing. That allows you to dynamically at runtime compile more things, evaluate things or change things. This would mean in practice, when you load an extension for the first time, it gets automatically compiled. When you load it for the second time, it is already compiled. Only when you change the source code of the an extension it would be recompiled in that moment, so that not even a restart is required. This is what Lisp implementations typically provide, so you get this for free.

• Many Lisp implementations provide a native interface to foreign languages (https://cffi.common-lisp.dev/), into Rust or C that is used by many people.

Drawbacks

The Lisp runtime environment is not a small library. If we talk only about a small extension mechanism with the only purpose to hook into your Rust world, you may not want such a huge mechanism. But if you take into account how much of the functionality of nowadays text editors are provided by extensions, it may be worth it.

Motivation

One reason why Visual Studio Code gained a lot of popularity is the programming language in which it is written. JavaScript performs very well and JavaScript attracts developers, who contribute all sorts of extensions. The succcess of JavaScript is not a random coincidence. Many people regard JavaScript as a dialect of Lisp (https://raganwald.com/2013/07/19/javascript-is-a-lisp.html, https://www.crockford.com/javascript/javascript.html). If we look at two important editors: VS Studio Code uses JavaScript and Emacs uses Lisp. The Emacs community has proven, that Lisp is appropriate for the best editors, and the VS Studio Code community has proven that ideas of dynamic programming matter for masses. But both editors have certain usability problems.

TreeSitter by the way already uses S-Expressions, which is the Lisp syntax. But TreeSitter is probably a special topic anyway, because it comes as C code. This proposal has no answer to that problem. But if Zed becomes successful and more popular it may be worth to think about a TreeSitter implementation in a way that is not based so much on binaries. That could be Lisp or whatever mechanism Zed in the future has. Lisp is an option, because it is a real programming language.

Perspective: Dynamic Behaviour

With dynamic behaviour I mean something like the following ideas.

• A document contains its own rules for syntax highlighting or autocompletion or navigation.
• A document contains a piece of code, that adds a tool bar or a manu entry in Zed. If you open the document, that code is executed and the tool bar appears. If you close it, the toolbar disappears.
• A documents contains a piece of code that implements a document specific mechanism for the outline view.
• A document contains even something like a "major mode" in Emacs.

These ideas are quite fantastic, because there is no editor that supports such things. It is hard to imagine something which we have never seen. It requires a very dynamic extension mechanism. You don't get this with a extension mechanism that requires a dedicated build or process. You want that you extension language is pure text, because this fits everywhere. You achieve this with Lisp or JavaScript or Lua. There are very heavy and impressive tools like Jetbrains MPS that try to achieve such a user experience. MPS is a great piece of software. But it is based on Java, which makes it very complex. With Lisp, as extension mechanism, where code is data you can achieve this much easier.

[1925381584]

When it comes to ease of learning and use, I think ruby is also a good option. Personally, I feel that ruby's chained calls are more elegant than bracketed expressions. When I look at elisp code, I always feel like I'm struggling to understand it, whereas ruby code is very intuitive and beautiful. But I'm not sure ruby is suitable as a language embedded in an editor!

[habruening]

Ruby should be possible. Sublime Text made a similar decision. They use Python. But my proposal goes in a different direction. Lisp has some unique selling points that I don't see with other languages. In my opinion these matter for the purpose of being an embedded language.

[matj1]

Elisp is the ugliest of all widespread Lisp dialects IMO. Racket and Clojure are much more readable and intuitive.

[zhengying]

Hi, Lua is a more lightweight scripting language. It has significant advantages as a configuration and scripting language, and it's also easy for developers to maintain. Neovim is a great example of this. Lua has a well-established ecosystem and is simple and easy to use."

[mocenigo]

The problem with Lua is that it does not seem to have a very clear direction, and it has been forked several times as a result. Python and Ruby are more suitable. Ideally, the scripting language should be portable, support multi-threading well, maybe even compile to WASM. Guile is in my opinion better for the purpose than a full CL, and has a WASM compiler (Hoot).

@habruening has mentioned Clojure — it would be a great choice, but I fear that the WASM backend is not mature enough? If it has reached sufficient maturity, then it would be my first choice.

[habruening]

Lua is one of the few options. I don't know Lua so much. No doubt that Lua for NeoVim is a great success. I would be a bit concerned if Lua is really versatile enough. It is nice to have a real programming language that addresses all sorts of problems. But the web page of Lua clearly sais, that it is a scripting language. I am sceptical if a scripting language is appropriate for real GUI heavy stuff. I suppose we want this. This will be a bit more complicated for Zed than for Emacs or NeoVim, because Zed is not a console program.

So you need a language that addresses the challenges, including multi threading. Lua looks to me like it has a bit of everything, but not enough of everything. For example it does not have map and reduce from functional programming or no multimethods or no native OOP. Some people really need these things to be productive. I prefer a real general purpose language, so that the editor can get so powerful that I can live the whole day in the editor like Emacs people do it. Assume you want to implement an E-Mail client in Zed. What language do you want to use? Obviously things like Scala are not an option. In my opinion Lisp is the hottest candidate. And the performance of Lisp is quite often still much faster than Lua. Lua is normally compared to other scripting languages und has the reputation to be very fast in this comparison. But Lisp is often compared to C.

But I suppose, the Zed team goes more into the direction of Web Assembly. They already use Web Assembly for Tree Sitter. With Web Assembly they also get a good solution for safety. With Web Assembly they are language independent. The Zed team may consider this as a benefit. I personally see it as a disadvantage, because it causes much fragmentation in the community. Everybody would use his favourite language. With Lisp you don't get such a fragmentation. You find tons of projects where programming language A is implemented in programming language B. Lua in Python. Lisp in Lua. Python in Java. Everything in everything. Because people are never happy with what they have. But what you don't find, people who try to implement Lua in Lisp, Python in Lisp, JavaScript in Lisp. Lisp developers are normally very happy with their language (my observation). If you take Lisp, you get one language that can rule everything.

What you get with Lisp is not only a language for extensions. You also get a dynamic environment. See here what I mean. That Clojure language (which is a Lisp dialect) enables impressive dynamic features in VS Code by exposing the JavaScript API to the user space. The whole infrastructure that enables a comfortable working experience with this foreign API already existed. That stuff implements the idea that in the middle of an arbitrary text file you can have pieces of code, which you can execute in your editor. You want to write something like 30 + 12 and quickly evaluate it without the need to install a math evaluation plugin. Or you want to copy something into the clipboard, filter it, sort it and paste it back. This is quickly done in any programming language. But you need a programming language that is available for you in that moment and that has access to your editor. You want to be able to have a text file that contains its own GUI, which helps editing itself. If you plan to enable this with Lua, fine.

[zhengying]

Thank you for your clarification! It's fun to discuss these topics, and your points make sense. I just want to emphasize that Lua is one of the choices available.

[habruening]

Yes, you convinced me, that Lua is a good and fits.

[tarsioonofrio]

Just my experience, I gave up on emacs for two motives: it is too slow and elisp.

Elisp, and functional languages in general, it is interesting, amazing, but too hard for the majority of devs, and for me. If zed wants to be a major force in editors wars, he needs to have a very engaged community that develops a multitude of plugins. Unfortunately lisp does not promote this because to be productive in that language takes too much time to deeply understand. As far I understand that is one major of the problems of emacs: a heavily engaged community, that builds an impressive software that makes everything, but that is still a small community because not everyone understands lisp.

Lua, is fast, and a script language, so it is easy to understand. We can see this in the neovim: a big and engaged community.

I believe that it is not merely a software decision but a choice that will impact the development of the community. And if zed chooses lisp I will give up too unfortunately.

Well, I prefer python, but I understand that is a really bad software decision for obvious reasons.

@habruening , I believe maybe you love lisp, and I understand that, it is a amazing language, but functional languages, that are beautiful, are hard and not for everyone unfortunately.

[tarsioonofrio]

Maybe a good option to plugin language will Rhai a Rust script language.

[JoeriDijkstra]

I personally love Emacs and use it all the time but honestly I also think Lua may be a better option, Emacs uses elisp because it's old, not because its the best option.

[habruening]

I looked a bit what options exist in the world. Unfortunatelly not so many. Let me summarise what I found out.

My personal conclusion at the moment is that Lua is indeed the best candidate and the fastest you can get. But in my opinion Lua is not perfect.

Just one example. Emacs has an XML outline. NeoVIM not. Why not? We need a language, where such things are easy to implement. What is the ideal way for XML editing? You need a view with the XML file, a view with your user configurable WYSIWYG editing (e.g. something like https://www.altova.com/de/authentic) and one view for an outline. All these three parts must be editable and responsive. If you change something in one part (e.g. drag and drop something in the outline, the other parts must update simultaneously at the speed of thought. That is hard to achieve. That works only if the thing that you edit is not ASCII text. ASCII text is the enemy of effective programming. That is hard to say in a discussion about text editors. But we must face the reality. You cannot parse and update ASCII text everywhere, because that is hard to implement. You want data models instead. Otherwise you will never be better than VS Code. So you need a real powerful programming language, that enables this. I don't think, that Lua is that powerful. The web page clearly sais, it is a scripting language.

Also I am a bit sceptical if Lua has answers for the next 30 years. It looks to me more like a trendy language for now. But that is maybe unly until someone comes with a new alternative. Lua is from 1993. The authors of Lua in my opinion are a bit too pragmatic and do not seem to have a long term strategy. On the one side they say "We did not consider LISP or Scheme because of their unfriendly syntax." But then they say "Semantically, Lua has many similarities with Scheme, even though these similarities are not immediately clear because the two languages are syntactically very different. The influence of Scheme on Lua has gradually increased during Lua’s evolution." I am not sure, how serious I can take Lua. That reminds me at JavaScript, where many people say, it is a dialect of Lisp. But many people are actually not happy wiht JavaScript and try to replace it by something else. Or why does nobody consider JavaScript for Zed? And now look how people use JavaScript: React is Functional Programming. What I want to say: There is still a lot happening in the world of programming languages. Betting on a language is a very hard decision. But, what I can clearly see, is that Emacs with Elisp made the right decision 50 years ago. Although not very popular, there is nobody who seriously wants to replace Elisp with something else. So what is the right decision for the next 50 years?

A few alternatives (but none of them viable):

• Embeddable Common Lisp. A good way to embedd Lisp into native C programs. But in order to get the full execution speed, it must be baked together with a C compiler. I don't think, that is a good option, especially, because Zed is not a C program.
• Guile. That is a good option. But it is not available on Windows.
• Janet. That in my opinion would be the best language. I think, it is similarly easy as Lua, but has a "richer core language", "more batteries" included and has a REPL. But performance is not a goal, and they don't plan to make it a goal. So Janet is no candidate.
• Fennel. That is also not a real alternative. It is a language that compiles to Lua. I only mention it, because that demonstrates, I am not the only one who wants Lisp instead of Lua.
• SBCL. So what remains are the super big Lisp environments. But these are probably not the ideal solution for an embedded langauge.

Unfortunatelly I have no good proposal. But the Zed team will have to make a decision. If they decide for WASM, I could understand that. But in my opinion it would only be an escape. I don't think, that Emacs or NeoVIM became so successful if there would not have been the one programming language that unifies everything. I don't think, that WASM would reproduce that "your text editor is a whole operating system" experience. That Emacs idea of being a whole operating system has a long tradition in the Lisp world (https://www.youtube.com/watch?v=7RNbIEJvjUA).

But anyway, Zed would be a reason for me to learn Lua, even if I prefer Lisp.

[mocenigo]

I gave this some thought, and I also looked at what other editors do. Even though, as a lisp person, I have my preference for languages like Guile, I think Lua is better, for three reasons:

1. It is very mature. I have complained that sometimes its development seems to lack a direction at the moment. But this may also be because it is a complete language for its purposes, so it is more into a mantainance mode while adding smaller features. It is also a good thing.

2. It is widely used and therefore it has several implementations. This means that the likelihood that it could be suddenly not supported is quite low. It is more likely that some crate that is needed in Zed may be orphaned. There is even a compiler to WASM and in any case converting the Lua bytecode to WASM should not be a task beyond @mrnugget because of his experience.

3. It is used by a very popular other editor, namely Neovim. If a future Zed extension API is similar to the Neovim one — which would be a smart more — porting useful functions would be simplified.

Therefore, my vote for Lua.
The second choice, because of similar, but not exactly the same, arguments, would be to use JS (like Nova and VSCode).

[mocenigo]

User red-co replied with

wasm=lisp+js+rust,
It solves all problems,

... then apparently deleted the message. I agree that the wasm VM in Zed allows us to use any language that can be compiled to wasm, BUT I interpreted this discussion as having a first class scripting language. Having more choices is nice, but if somebody wants to modifiy an extension, than another, chances are that they have to learn more than one language. One language should have a special status and have a rich set of bindings for all sorts of manipulations and written by the Zed team. These bindings then would be ported to other languages, possibly by others.

[mocenigo]

A few observations.

1. Since zed includes node, we can ask it to execute TypeScript programs.
2. There is wasmoon, written in TypeScript, that can compile Lua into webasm.
3. So we just need the right bindings, and maybe some extra support, and we could make Lua into a first class language for zed.

What would be missing? Is for instance a node.js runtime necessary in the wasm runtime for wasmoon compiled programs?

[habruening]

Jank (https://jank-lang.org/) is a new interesting language. Not yet ready, but interesting. It is a general purpose language with maybe better capabilities for abstraction than Lua. Jank can be compiled ahead of time or just in time to native code (or probably WASM later).

That would allow a few very nice ideas.

• Plugins can be delivered as source code. They are compiled when started.
• Plugins can be delivered precompiled as native code.
• Code can be changed at runtime and recompiled at runtime.
• Unsafe or non-time demanding plugins can be executed in WASM
• Safe plugins can be executed as native code. Users could even decide to run individual plugins in WASM and othe plugins native.
• We could implement TreeSitter in Jank. As I see it problematic anyway that TreeSitter grammers are compiled into C code. That is very unflexible. People cannot quickly define or change a TreeSitter grammer, because they need the C compiler. If TreeSitter grammars are translated to Jank code, they can be compiled at runtime.

Jank is not more than a proof of concept at the moment. I think, it demonstrates how a good plugin mechanism could work. But I am not sure if Jank runs without a C++ compiler.

There is an interesting Video: https://youtu.be/ncYlHfK25i0 In the video there are also a few statements about Rust.

I think, when following this, we understand a bit better that really good technologies for a perfect plugin mechanism do not yet exist. Everything will be a compromise.

[mocenigo]

Hi @habruening , this is definitely interesting, but once you can compile to wasm, any language can be used, provided you have a good FFI to the Rust API. I am very partial to javascript/typescript at the moment because of the recent efforts to integrate a webview based on the tauri code. Add deno, and you can easily port many vscode extensions...

[habruening]

I understand the ambition of the Zed team to be super fast. So fast that you see no lag for things like auto completion, highlighting, outlines or menu and GUI things. If this is the ambition, native code is the goal. WASM is still much slower than native code. The great thing of WASM is, that you use the same code native or in WASM. WASM gives you both options. You only have to learn one language (C++, Rust or whatever) and you can apply it for high speed native plugins and for low speed unsafe WASM plugins. If you don't want both options, WASM makes not much sense. WASM enables only a quite low level programming style. Plugin developers do not really want to work in C++ or care for memory management. You don't even have garbage collection in WASM. If you only want a quite fast platform solution that fits for plugin development , there are better options than WASM, namely the JVM. There are plenty of languages that run on the JVM. The JVM enables a much higher level of abstration than WASM could ever do. But I think the JVM is not really an option. The JVM is not very popular, it has a long startup time, it has a big footprint. But it is comparable fast as WASM.
So we are at the beginning. Super fast and safe enough for plugin development is a contradiction. JS is super safe and super slow. Native code is super fast and unsafe. Platform languages (JVM, Common Lisp) are in the middle. With WASM we have principally both, but not at the same time.

[mocenigo]

I understand the ambition of the Zed team to be super fast. So fast that you see no lag for things like auto completion, highlighting, outlines or menu and GUI things. If this is the ambition, native code is the goal. WASM is still much slower than native code.

WASM can be transformed to native machine code. In fact, this is exactly what happens in Zed. Wasmtime is the wasm runtime environment, which leverages cranelift to execute it natively. WASM has the advantage that one compiles it once (like also Java promises to do) and can be distributed independently from the underlying ISA. The compilation from bytecode to machine code is much faster than the compilation from source code to either bytecode or native nachine code.

The great thing of WASM is, that you use the same code native or in WASM. WASM gives you both options. You only have to learn one language (C++, Rust or whatever) and you can apply it for high speed native plugins and for low speed unsafe WASM plugins.

I am not sure whether you are misunderstanbding quite a few things or you are confused. If you use a safe language, the code will be safe regardless of whether you compile it to native machine code ot WASM. Rust plugins are safe (unless unsafe code is present) also as WASM, and once transpiled upon loading, they should ahve roughly the same performance as if they were compiled directly to machine code.

If you don't want both options, WASM makes not much sense. WASM enables only a quite low level programming style.

What what what? But one does not write directly in WASM...

Plugin developers do not really want to work in C++ or care for memory management. You don't even have garbage collection in WASM. If you only want a quite fast platform solution that fits for plugin development , there are better options than WASM, namely the JVM. There are plenty of languages that run on the JVM. The JVM enables a much higher level of abstration than WASM could ever do. But I think the JVM is not really an option. The JVM is not very popular, it has a long startup time, it has a big footprint. But it is comparable fast as WASM. So we are at the beginning.

Super fast and safe enough for plugin development is a contradiction.

MMMMM. No.

JS is super safe and super slow.

There are large applications that are very responsive written in Javascript or Typescript.

Native code is super fast and unsafe. Platform languages (JVM, Common Lisp) are in the middle. With WASM we have principally both, but not at the same time.

You need to correct some of your assumptions.

Also, it seems we are conflating two things: plugins and scripting.

[habruening]

Indeed, I have to correct my words.

With the same code native and in WASM I mean, that the same C code can be either compiled to WASM or to a native binary.

With the words "WASM enables only a quite low level programming style" I mean, that in my opinion there is a difference between languages like Python, JavaScript, Clojure, that are all high level languages and languages like C, C++, Rust, Ada, which are all low level languages. But that is only my opinion. For example none of them has a garbage collector. I agree that my words were very unclear.

With unsafe code I actually don't mean the programming language, but the platform on which it is executed. A buggy WASM program can only crash the WASM environment. A buggy native program can crash the whole application.

Sorry for the confusion. My words were really incorrect. I agree to most of your words.