wat: lisp-y text format for wasm

[sbinet]

to ease with sharing the spec tests with WebAssembly, we should be able to parse the .wast files.

There are already a few Go-based lisp-y parsers/lexers:

• https://github.com/glycerine/zygomys
• https://github.com/janne/go-lisp
• https://zhehaomao.com/project/2014/03/28/glisp.html, https://github.com/zhemao/glisp
• https://pkelchte.wordpress.com/2013/12/31/scm-go/
• https://github.com/DangerOnTheRanger/schego

[Spriithy]

Hello there,

I was wondering if you would like some help for this particular issue ?
I don't have much experience for lisp-y languages but some with lexers / parsers.

I would be glad to contribute to this project in my free time.

[sbinet]

hi @Spriithy

yes. definitely yes!

[Spriithy]

Thanks @sbinet for your consideration,

After looking at the file format specification and based on the knowledge that .wast files are lisp syntax based, should we implement an in-house lexer/parser to generate some pre-defined IR ?

From the above references, I think Schego and Zygomys looks quite robust.

Anyways, what are the specifications for the wast format interaction ? Would it be meant for:

• Execution
• Sanitization
• Verification
• Optimization

We should first better define the needs.

[sbinet]

I am not super familiar with the design space of lexers/parsers, so I can't answer your question about the generation of a pre-defined IR.

The first and immediate value for a wat (or wast) parser is to be able to reduce the scaffolding in the tests of the interpreter.

the official wasm tests contain wast files that describe the content of wasm modules.
these modules are then fed to the interpreter and the result of which is then checked against the expectations that are also described in the wast files.

I'd like to closely reproduce this modus operandi.

I think this involves being able to parse a wast file, detect the parts that describe a wasm module, create a proper wasm.Module out of that and then let the exec package loose.

I am not yet sure whether directly creating a wasm.Module from a wast file is better over first creating a wasm file from the wast one and then parsing the wasm file to create a wasm.Module.
(I guess the second solution is more modular but it involves more moving pieces...)

what do you think?

[Spriithy]

As stated in the official documentation, the .wast format is a superset of .wat.

Note: The .wast format understood by some of the listed tools is a superset of the .wat format that is intended for writing test scripts.

Anyways, after comparing a handful of wat and wast files it seems, to me, that creating a wasm file from a wast one and then parsing the resulting file to create a wasm.Module is just almost (I cannot emphasize this word enough 😄) doing the same work twice.

To give you an idea, wast files are to be seen as some sort of test files, where each top-level element is a test unit/case asserting one particular behavior (see the i64 test suite for instance). As you can see, the file contains a module, exporting several functions that are then tested.

It's like a combination of the source file and the test file, if you want.

It seems to me that parsing a wast file should generate []wasm.Module containing all the declared modules along with a test module wrapping all the test cases. The latter would include the former as needed and sequentially run the statements (after indexing possible declarations).

[Spriithy]

Obviously, all the test suites from the official WebAssembly repositories are given as standalone wast files. I would like to add that, parsing and generating the code from such files and then executing it on-the-fly seems to be the right approach to me.

Nevertheless, it would be interesting to have some external feedback / suggestions.

[Spriithy]

Hey there,

I allow myself to bump this issue to notify those interested that I am currently working on it.
I have decided to go for a in-house lexer/parser for the wast file format has very special specifications.

I plan to create a reviewable pull request that is not meant for immediate merge but rather see the advancement of the implementation and propose an early insight in my work.

[sbinet]

great!

[andreimatei]

Just wondering, without knowing what I'm talking about: instead of building an assembler, have you considered at all using wabt through CGo? If you want to keep wagon buildable through go build, I think wabt's build system can't be used and instead one has to list all the .c files in CGo directives. It's ugly but probably doable - CockroachDB used to build large C++ dependencies that way.

[andreimatei]

Btw, another option for using wabt as a dependency is to ask user of wagon to install and build wabt themselves and then wagon would link against it through a directive like

// #cgo LDFLAGS: -lwabt

This directive could be put in a file gated behind a build flag, if the integration is designed in such a way that it is optional.

[sbinet]

well, the whole point of wagon is to be pure-Go.
I wouldn't be against having another go-interpreter/repo with either a CGo package that links against wabt or one that shells out to one of its commands.

but I'd like to keep wagon free of any cgo business :)