C++20 migration woes

 I have an extensive list of subjects I can work on in the Egel language.  Among them: performance (it needs to become an order faster),  the introduction of quotation and local combinator definitions, fixing small unfinished corners,  mobile code, and -last but not least- various language experiments around features not easily expressible in typed languages.

But I also want to migrate to C++20 modules and introduce an egel namespace. And I also want to do that before all the above since that should simplify and sanitize the interpreter; modules should allow for a more principled approach to the implementation and I'll be visiting, subsequently cleaning, a lot of code not touched in years. I prefer single-file (complete) modules for a bit of a silly reason: I mostly program in vim and it's just faster/easier to modify declarations not spread over different files.

Egel is intended to be an interpreted language implemented in C++, closely following the C++ standard and not much more (except for Unicode/libicu) and to be easily extendable and embeddable in C++. This already gives problems since C++20 compilers (gcc, clang, msvc) are at various stages of supporting the C++20 module system. This adds to the problems I already have supporting gcc/clang, the interpreter usually only builds on up-to-date systems due to various moving targets, i.e., cmake and libraries.

I am at the start of the process, and I decided to document a number of problems I encountered.

• Naming. A silly observation but do I go for `.ixx`, `.cppm`, or `.cpp` files? I decided on `.ixx`. Rationale: I don't like 4 letter extensions,  and `.cpp` is too overloaded for my taste. This is weird because I don't support msvc at the moment, I only work with clang since my shift to a Mac M1, and I want to primarily support the venerable gcc.
• Macros.  I make extensive use of macros. A number of them I have been phasing out (like casts) but there are two types of macros I would prefer to keep.  Assertions (like `PANIC(m)`, abort with a message) and multiple assignment helpers (like `SPLIT_LAMBDA(l, vv, t)`, split a lambda AST object into variables and a term).  Then there are boiler-plate macros (like DYADIC_PREAMBLE which sets up all the boilerplate of a dyadic combinator.) Where do they go? Do I need to include a `macro.hpp` header file in every module?
• Constants.  A number of constants are defined as macros in the interpreter.  Like `#define LPAREN '('`. I can easily switch over to constexpr but that implies I need to decide on a type, in my case char or UChar32.  I would prefer not to.  It gets worse for string constants: char*, string, or icu::UnicodeString? Sometimes it makes sense to treat something as text instead of a typed expression.  This is a minor detail but switching over to constexpr actually makes the code more tedious to support long term,  instead of the opposite.
• Inclusion of other libraries.  Where are `<iostream>` and the libicu c++20 module equivalents? Do I include, do I import, if I import, then what to do I import? Where do I find the information, i.e., what website documents where the objects are defined?
• CMake.  I have no clue how to write cmake modules that support c++20 modules, not even considering in a portable manner.

These are the problems I have now. What I need are migration guides.

Rewriting the emitter

 The bytecode currently produced by the egel interpreter is pretty brain-dead at the moment.  For a simple expression like "1+1" it will produce code like "a=reduce(1);b=reduce(1);result=reduce(+ a b)". Needs to be fixed.

Compacting GC?

A neat trick is that directed acyclic graphs, the run-time structures egel programs manipulate, do allow for an efficient in-place compacting garbage collector. Which is what I am tempted to write at the moment.

Looks like sectorlisp uses that idea.

For future reference

cmake -DCMAKE_BUILD_TYPE=PROFILE ..




egel -e "import \"prelude.eg\";; using System;; using List;; foldl (+) 0 (from_to 0 1000000)"

/opt/homebrew/Cellar/llvm@12/12.0.1_1/bin/llvm-profdata merge -sparse default.profraw -o default.profdata

/opt/homebrew/Cellar/llvm@12/12.0.1_1/bin/llvm-profdata show -all-functions default.profraw

Testing 1.. 2.. 3..

// memory test

#include <memory>

#include <new>

#include <type_traits>

#include <utility>

// model values = atom int | array value*

static constexpr size_t roundup(size_t x, size_t m)

{

    return x % m == 0 ? x : (x + m) / m * m;

};

class Value;

typedef std::shared_ptr<Value> ValuePtr;

class Value {

public:

    Value() = default;

    virtual ~Value() {

    }

};

class Atom: public Value {

public:

    Atom(const int v): _value(v) {

    }

    virtual ~Atom() {

    }

    static ValuePtr create(const int v) {

        return std::shared_ptr<Atom>(new Atom(v));

    }

private:

    int _value;

};

struct ValueField {

    ValuePtr _value;

};

class Array: public Value {

public:

    Array() = default;

    virtual ~Array() {

        while (_size-- > 0)

            std::addressof(operator[](_size))->~ValueField();

    }

    ValueField &operator[](size_t i) {

        char *r0 = reinterpret_cast<char *>(this) + roundup(sizeof(*this), alignof(ValueField));

        ValueField *t0 = reinterpret_cast<ValueField *>(r0);

        return t0[i];

    }

    ValueField &at(size_t i) {

        if (i >= _size)

            throw std::out_of_range("duuude!");

        return operator[](i);

    }

    static ValuePtr create(size_t _sizes) {

        return std::shared_ptr<Array>(new(_sizes) Array(_sizes));

    }

    void *operator new(size_t count, size_t _sizes) {

        auto z = roundup(count, alignof(ValueField)) + sizeof(ValueField) * _sizes;

        return ::operator new(z);

    }

    void operator delete(void *x) {

        ::operator delete(x);

    }

    // not implemented for brevity reasons.

    // Also, it would have to handle (if only to throw an exception)

    // if someone attempted to move an Array(30) into an Array(20)

    /*

    Array(Array &&) = delete;

    void operator=(Array &&) = delete;

    */

private:

    Array(size_t n) {

        for (_size = 0; _size < n; ++_size)

            new(std::addressof(operator[](_size))) ValueField;

    }

    size_t _size = 0;

};

int main()

{

    Array fixed;

    auto flex = Array::create(20);

    // (*flex)[1]._value = 42;

}

Serialization

 I am more pondering than writing code at the moment. In order to get to mobile code, I need to be able to serialize, in order to serialize I need some representation independent of the runtime. Because when the serialized graph isn't independent it might corrupt the runtime when it is being loaded among other things.

It looks like I have little choice but to copy almost the complete runtime hierarchy into a serialization format.  Which sounds worse than it is, only four primitive types, arrays, then combinators. And then the combinators are the problem. Data is easy, bytecode can be (dis-)assembled, internal objects are always loaded, then the problem is externally defined C++ combinators. Or I do everything on a per-module basis, that's unclear at the moment.

Then there's the question of what goes into the graph. Say I have a bytecode combinator, do I only write the symbol or also the bytecode? No idea yet but I am tempted to have that separate from the graph.

The good part, serialization makes me think hard about how the interpreter is organized. The bad part, now I want to refactor again and get rid of some warts (macros), introduce namespaces, and employ c++20 modules. It's annoying to code against old code.

Also, pretty sure I am going to write the mobile code on top of grpc, which brings other problems since that's a client/server architecture and I want bidirectionality and more general a cloud of nodes which can talk to each other. But ah well, not everything is perfect. Maybe I'll just use it anyway.

Slides

 I felt like writing so I made some technical slides.