The C3 Blog¶

Regarding programming forums and such

Originally from: https://c3.handmade.network/blog/p/8863-regarding_programming_forums_and_such

An observation: I notice that by virtue of people being mostly anonymous, a curious effect occurs on programming discords (and by extension elsewhere):

People who are "chat savvy" (or whatever we should call being good at writing in a way that is similar to being good at social interactions in real life) are able to dominate discussions by virtue of this.

In addition, people tend to "cluster" when it comes to opinions, so that followers of such persons may sway others ("everyone else agrees on this").

However, such savvy has nothing to do with actual programming skill or knowledge. Many of these "leaders" are in fact fairly inexperienced, if not outright beginners. Age is similarly obfuscated, so that teenagers might be seen as old and middle-aged persons may be taken for teenagers.

The most obvious example when someone new comes to a Discord and enthusiastically starts presenting ideas. If these ideas are not "approved" by the leaders, the person might immediately be mocked and treated as a beginner / idiot.

I've seen this played out several times, at one occasion a 60+ year old gentleman presenting a language and compiler, where he solved several long standing practical problems he'd encountered over the course of his career. He was laughed at and ridiculed as knowing nothing about programming or real world problems by kids 1/3rd his age because he wasn't presenting it in the way that was "expected" in that community. It was painful to see.

Outside of direct bullying, criticizing well-known people in the business is a favorite past-time. You often find people confidently deriding others as "not having any experience", "not knowing what they're talking about", "is just making things up etc". These critics are very sure of themselves, with aforementioned followers to echo those feelings. So you end up with a bunch of 16-year olds deriding 50+ year old programmers with multiple hit games under their belt as "not knowing anything about programming" and collecting pats on the back by the crowd for saying something so profound!

So what is my point? Nothing really except for these observations and to conclude that there is no wisdom of the crowds online, you need to find truth on your own.

Comments

Comment by Christoffer Lernö

An observation: I notice that by virtue of people being mostly anonymous, a curious effect occurs on programming discords (and by extension elsewhere):

People who are "chat savvy" (or whatever we should call being good at writing in a way that is similar to being good at social interactions in real life) are able to dominate discussions by virtue of this.

In addition, people tend to "cluster" when it comes to opinions, so that followers of such persons may sway others ("everyone else agrees on this").

However, such savvy has nothing to do with actual programming skill or knowledge. Many of these "leaders" are in fact fairly inexperienced, if not outright beginners. Age is similarly obfuscated, so that teenagers might be seen as old and middle-aged persons may be taken for teenagers.

The most obvious example when someone new comes to a Discord and enthusiastically starts presenting ideas. If these ideas are not "approved" by the leaders, the person might immediately be mocked and treated as a beginner / idiot.

I've seen this played out several times, at one occasion a 60+ year old gentleman presenting a language and compiler, where he solved several long standing practical problems he'd encountered over the course of his career. He was laughed at and ridiculed as knowing nothing about programming or real world problems by kids 1/3rd his age because he wasn't presenting it in the way that was "expected" in that community. It was painful to see.

Outside of direct bullying, criticizing well-known people in the business is a favorite past-time. You often find people confidently deriding others as "not having any experience", "not knowing what they're talking about", "is just making things up etc". These critics are very sure of themselves, with aforementioned followers to echo those feelings. So you end up with a bunch of 16-year olds deriding 50+ year old programmers with multiple hit games under their belt as "not knowing anything about programming" and collecting pats on the back by the crowd for saying something so profound!

So what is my point? Nothing really except for these observations and to conclude that there is no wisdom of the crowds online, you need to find truth on your own.

How bad is LLVM really?

Originally from: https://c3.handmade.network/blog/p/8852-how_bad_is_llvm_really

LLVM used to be hailed as a great thing, but with language projects such as Rust, Zig and others complaining it's bad and slow and they're moving away from it – how bad is LLVM really?

What is LLVM?

LLVM of today is not just a compiler backend, it's a whole toolchain, and the project also provides a linker (lld), a C compiler available as a library (Clang) and much more.

Except for the anomaly of Zig (Zig also uses the entire Clang as a library), most language projects simply use the LLVM backend, and possibly also the lld linker.

LLVM, Clang, lld and most other parts of the project are written in C++, with a C API available for LLVM, but not for most of the other libraries.

The speed problem

When Clang was released, a selling point was that it compiled faster than GCC. Since then this has slipped a bit and GCC and Clang is about equally slow.

The problem is not in optimized builds - most people accept that optimized builds will compile slowly. No, the problem is that unoptimized builds compile slowly. How slow? LLVM codegen and linking takes over 98% of the total compilation time for the C3 compiler when codegen is single threaded with no optimizations.

If codegen is 2 magnitudes slower than parsing, lexing and semantic checking combined, then you can see why compiler writers might not be totally happy with LLVM's performance.

Why is LLVM slow?

First a disclaimer: I have only read the LLVM source code a bit, I haven't contributed anything beyond a few small fixes so I'm not an expert.

However, it seems to me that LLVM has a fairly traditional C++ OO design. One thing this results in is an abundance of heap allocations. An early experiment switching the C3 compiler to mimalloc improved LLVM running times with a whopping 10%, which could only be true if memory allocations were a large contributor to the runtime cost. I would have expected LLVM to use arena allocators, but that doesn't seem to be the case for most code.

Heap allocations aside, using C++ or similar languages often invite certain inefficient patterns. It's easy to just rely on high level constructs to solve problems:

Need to check if a list has duplicates? No problem, just grab a hash set and check!

Except if the list is typically only 2-3 entries, then just doing a linear search might be much faster and require no setup. It doesn't matter how clever and fast the hash set is. And they're usually fast – LLVM has lots of optimized containers, but if no container was needed, then it doesn't matter how fast it was.

It's not necessarily bad code, but it's not code this is likely to be highly performant.

Why is LLVM "bad"?

There are other warts LLVM has. First up, the documentation isn't particularly great. It's not worse than many other libraries I've used, so this is more of a "we all wish it could be better because understanding the backend is hard enough as it is".

More fundamentally though, LLVM is very much a backend for C/C++. While LLVM has test suites, Clang is ultimately the product in the LLVM umbrella that really tests the backend. This results in codegen not used by Clang being notoriously unreliable, as well as often poorly optimized (passing structs around by value for instance).

Another consequence is that LLVM often has mandatory UB where C/C++ does. For example, integer division by zero is currently an unescapable undefined behaviour in LLVM – which is bad if your language wanted to define x / 0 to be 0 for example. Another example is when i << x overflows due to x being the same or larger than the bit width of i. This yields a poison value in LLVM, so if you wanted, say, it to be 0, you would have add a select on every such shift as there is no way to request well defined behaviour. At least in this case the result is a poison value and not UB. C/C++ of course considers i << x undefined behaviour for these overflow cases.

So: bugs, not-so-great documentation and assumption of C/C++ semantics are probably the main complaints I've seen.

The problem with alternatives

Alternatives to LLVM that pop up are Cranelift, QBE etc. However, at the moment none of those offers the same kind of complete solution that LLVM provides - and some of them are slower than using LLVM! If you already started using LLVM's advanced features, you will struggle with feature parity, not to mention the limited platform support.

Integrating with GCC is an alternative, but it doesn't solve the compilation speed problem, nor the other "bad" things about LLVM.

At this point, a lot of projects will start thinking about writing their own backend, and honestly this is probably a better alternative than using anything incomplete off the shelf at the moment, as this ensures there isn't some missing functionality that is impossible to handle later.

So while there are some promising upcoming backends (Tilde Backend comes to mind), there isn't really a drop in replacement for LLVM today.

LLVM the good parts

While there are these downsides to LLVM, we shouldn't lose track of what it actually brings to the table. It's a full fledged backend that is way more field tested than anything one could hope to write by oneself. It's reliable in the sense that it's not going away tomorrow or in five years. Buried in LLVM + Clang is a treasure trove of domain knowledge that a single developer can't be expected to accumulate on their own.

Being able to use LLVM is a huge service to language developers. What it lacks in speed it wins back in completeness.

Final words

We all love to complain about LLVM. It's far from perfect, not the least in regards to speed. But at the same time, it is allowing language designers to build compilers that produces production quality machine code on a wide variety of platforms. So really, starting out with LLVM is a good idea. Once there is a backend that works there is plenty of time to explore other backends without any pressure.

So is LLVM bad? Well it has its bad parts, but it's also probably the best backend you can pick for your compiler when you start out (not counting transpiling to C).

You can worry about the bad parts later.

Comments

Comment by Christoffer Lernö

LLVM used to be hailed as a great thing, but with language projects such as Rust, Zig and others complaining it's bad and slow and they're moving away from it – how bad is LLVM really?

What is LLVM?

LLVM of today is not just a compiler backend, it's a whole toolchain, and the project also provides a linker (lld), a C compiler available as a library (Clang) and much more.

Except for the anomaly of Zig (Zig also uses the entire Clang as a library), most language projects simply use the LLVM backend, and possibly also the lld linker.

LLVM, Clang, lld and most other parts of the project are written in C++, with a C API available for LLVM, but not for most of the other libraries.

The speed problem

When Clang was released, a selling point was that it compiled faster than GCC. Since then this has slipped a bit and GCC and Clang is about equally slow.

The problem is not in optimized builds - most people accept that optimized builds will compile slowly. No, the problem is that unoptimized builds compile slowly. How slow? LLVM codegen and linking takes over 98% of the total compilation time for the C3 compiler when codegen is single threaded with no optimizations.

If codegen is 2 magnitudes slower than parsing, lexing and semantic checking combined, then you can see why compiler writers might not be totally happy with LLVM's performance.

Why is LLVM slow?

First a disclaimer: I have only read the LLVM source code a bit, I haven't contributed anything beyond a few small fixes so I'm not an expert.

However, it seems to me that LLVM has a fairly traditional C++ OO design. One thing this results in is an abundance of heap allocations. An early experiment switching the C3 compiler to mimalloc improved LLVM running times with a whopping 10%, which could only be true if memory allocations were a large contributor to the runtime cost. I would have expected LLVM to use arena allocators, but that doesn't seem to be the case for most code.

Heap allocations aside, using C++ or similar languages often invite certain inefficient patterns. It's easy to just rely on high level constructs to solve problems:

Need to check if a list has duplicates? No problem, just grab a hash set and check!

Except if the list is typically only 2-3 entries, then just doing a linear search might be much faster and require no setup. It doesn't matter how clever and fast the hash set is. And they're usually fast – LLVM has lots of optimized containers, but if no container was needed, then it doesn't matter how fast it was.

It's not necessarily bad code, but it's not code this is likely to be highly performant.

Why is LLVM "bad"?

There are other warts LLVM has. First up, the documentation isn't particularly great. It's not worse than many other libraries I've used, so this is more of a "we all wish it could be better because understanding the backend is hard enough as it is".

More fundamentally though, LLVM is very much a backend for C/C++. While LLVM has test suites, Clang is ultimately the product in the LLVM umbrella that really tests the backend. This results in codegen not used by Clang being notoriously unreliable, as well as often poorly optimized (passing structs around by value for instance).

Another consequence is that LLVM often has mandatory UB where C/C++ does. For example, integer division by zero is currently an unescapable undefined behaviour in LLVM – which is bad if your language wanted to define x / 0 to be 0 for example. Another example is when i << x overflows due to x being the same or larger than the bit width of i. This yields a poison value in LLVM, so if you wanted, say, it to be 0, you would have add a select on every such shift as there is no way to request well defined behaviour. At least in this case the result is a poison value and not UB. C/C++ of course considers i << x undefined behaviour for these overflow cases.

So: bugs, not-so-great documentation and assumption of C/C++ semantics are probably the main complaints I've seen.

The problem with alternatives

Alternatives to LLVM that pop up are Cranelift, QBE etc. However, at the moment none of those offers the same kind of complete solution that LLVM provides - and some of them are slower than using LLVM! If you already started using LLVM's advanced features, you will struggle with feature parity, not to mention the limited platform support.

Integrating with GCC is an alternative, but it doesn't solve the compilation speed problem, nor the other "bad" things about LLVM.

At this point, a lot of projects will start thinking about writing their own backend, and honestly this is probably a better alternative than using anything incomplete off the shelf at the moment, as this ensures there isn't some missing functionality that is impossible to handle later.

So while there are some promising upcoming backends (Tilde Backend comes to mind), there isn't really a drop in replacement for LLVM today.

LLVM the good parts

While there are these downsides to LLVM, we shouldn't lose track of what it actually brings to the table. It's a full fledged backend that is way more field tested than anything one could hope to write by oneself. It's reliable in the sense that it's not going away tomorrow or in five years. Buried in LLVM + Clang is a treasure trove of domain knowledge that a single developer can't be expected to accumulate on their own.

Being able to use LLVM is a huge service to language developers. What it lacks in speed it wins back in completeness.

Final words

We all love to complain about LLVM. It's far from perfect, not the least in regards to speed. But at the same time, it is allowing language designers to build compilers that produces production quality machine code on a wide variety of platforms. So really, starting out with LLVM is a good idea. Once there is a backend that works there is plenty of time to explore other backends without any pressure.

So is LLVM bad? Well it has its bad parts, but it's also probably the best backend you can pick for your compiler when you start out (not counting transpiling to C).

You can worry about the bad parts later.

Comment by Christoffer Lernö

Lexing, parsing and analysis is about 1-2% of the entire compile time when compiling C3 code with no optimizations. The rest is LLVM + linking, where linking is a small part of the time.

You can compare some compiler benchmarks here: https://github.com/nordlow/compiler-benchmark

Not that such benchmark really gives the real time compilation will take on general code, but it gives a rough level of magnitude between different compilers (and consequently compiler backends, as this is where the most of the time is spent for something like C)

Syntax - when in doubt, don't innovate

Originally from: https://c3.handmade.network/blog/p/8851-syntax_-_when_in_doubt%252C_don%2527t_innovate

One of the most attractive things about language design is to be able to tweak the syntax of a language on its fundamental level, so not surprisingly you'll see language designers coming up with all sorts of alternatives to conventional syntax.

The problem is that it takes a while – I'd say a year at least – to figure out if some particular new syntax is good. It often takes less time figure out if it's bad, but in some cases it might not be obvious until very late. So just because it's not immediately bad, it doesn't mean you won't find out something later.

Even worse, it's hard to weed out false negatives: sometimes syntax might appear to be "bad" simply because it is unfamiliar.

For that reason I think a good rule of thumb when working on syntax might be "when in doubt, do not innovate".

Just like other language features should "carry their weight" (that is, their value should outweigh their cost), so should syntax. "It's setting the language apart" or "I like how it looks" is fairly low on the value scale if the language is intended for use by others. If you're not sure whether some new syntax is necessary then it's better to wait until you know if it is. Meanwhile, there are established syntax conventions out there you can lean on.

New syntax shines where it enables (possibly new and innovative) language features to be expressed cleanly and clearly. It is probably better to prioritize such syntax innovations than, say, innovate new symbol combinations for arithmetics.

Comments

Comment by Christoffer Lernö

One of the most attractive things about language design is to be able to tweak the syntax of a language on its fundamental level, so not surprisingly you'll see language designers coming up with all sorts of alternatives to conventional syntax.

The problem is that it takes a while – I'd say a year at least – to figure out if some particular new syntax is good. It often takes less time figure out if it's bad, but in some cases it might not be obvious until very late. So just because it's not immediately bad, it doesn't mean you won't find out something later.

Even worse, it's hard to weed out false negatives: sometimes syntax might appear to be "bad" simply because it is unfamiliar.

For that reason I think a good rule of thumb when working on syntax might be "when in doubt, do not innovate".

Just like other language features should "carry their weight" (that is, their value should outweigh their cost), so should syntax. "It's setting the language apart" or "I like how it looks" is fairly low on the value scale if the language is intended for use by others. If you're not sure whether some new syntax is necessary then it's better to wait until you know if it is. Meanwhile, there are established syntax conventions out there you can lean on.

New syntax shines where it enables (possibly new and innovative) language features to be expressed cleanly and clearly. It is probably better to prioritize such syntax innovations than, say, innovate new symbol combinations for arithmetics.

C3 0.5.3 Released

Originally from: https://c3.handmade.network/blog/p/8848-c3_0.5.3_released

It's almost 2 months since 0.5.0 was released and we're now at 0.5.3. This is the change list from 0.5.2:

Changes / improvements

• Migrate from using actual type with GEP, use i8 or i8 array instead.
• Optimize foreach for single element arrays.
• Move all calls to panic due to checks to the end of the function.

Fixes

• Single module command line option was not respected.
• Fixed issue with compile time defined types (String in this case), which would crash the compiler in certain cases.
• Projects now correctly respect optimization directives.
• Generic modules now correctly follow the implicit import rules of regular modules.
• Passing an untyped list to a macro and then using it as a vaarg would crash the compiler.
• Extern const globals now work correctly.

Stdlib changes

• init_new/init_temp deprecated, replaced by new_init and temp_init.

What about 0.5.1 and 0.5.2?

Unfortunately I never blogged about those. So here is a short recap on what happened in 0.5.1 and 0.5.2:

Changes / improvements

• Allow trailing comma in calls and parameters #1092.
• Improved error messages for const errors.
• Do not link with debug libraries unless using static libraries.
• Add 'print-linking' build option.
• System linker may be used even if the target arch is different from current.
• Slice -> array/vector works for constant slice lengths.

Fixes

• Fixes issue where single character filenames like 'a.c3' would be rejected.
• Better errors when index type doesn't match len() when doing user defined foreach.
• Fixes to to_int for hexadecimal strings.
• Fixed issue when using a generic type from a generic type.
• Bug with vector parameters when the size > 2 and modified.
• Missing error on assigning to in-parameters through subscripting.
• Inference of a vector on the lhs of a binary expression would cause a crash.
• Fixes to PriorityQueue
• On Aarch64 use the correct frame pointer type.
• On Aarch64 macOS, ensure the minimum version is 11.0 (Big Sur)
• Fixes to the yacc grammar.
• Dsym generation on macOS will correctly emit -arch.
• Stacktrace on signals on Linux when backtrace is available.

Stdlib changes

• Allow to_int family functions take a base, parsing base 2-10 and 16.
• delete and delete_range added to DString.
• Splitter iterator added.
• splitter and iterator String methods.
• load_new, load_buffer and load_temp std::io::file functions.

0.5 has feature stability guarantees, so any code written for 0.5.0 will with on all of 0.5.x.

If you want to read more about C3, check out the documentation: https://c3-lang.org or download it and try it out: https://github.com/c3lang/c3c

Comments

Comment by Christoffer Lernö

It's almost 2 months since 0.5.0 was released and we're now at 0.5.3. This is the change list from 0.5.2:

Changes / improvements

• Migrate from using actual type with GEP, use i8 or i8 array instead.
• Optimize foreach for single element arrays.
• Move all calls to panic due to checks to the end of the function.

Fixes

• Single module command line option was not respected.
• Fixed issue with compile time defined types (String in this case), which would crash the compiler in certain cases.
• Projects now correctly respect optimization directives.
• Generic modules now correctly follow the implicit import rules of regular modules.
• Passing an untyped list to a macro and then using it as a vaarg would crash the compiler.
• Extern const globals now work correctly.

Stdlib changes

• init_new/init_temp deprecated, replaced by new_init and temp_init.

What about 0.5.1 and 0.5.2?

Unfortunately I never blogged about those. So here is a short recap on what happened in 0.5.1 and 0.5.2:

Changes / improvements

• Allow trailing comma in calls and parameters #1092.
• Improved error messages for const errors.
• Do not link with debug libraries unless using static libraries.
• Add 'print-linking' build option.
• System linker may be used even if the target arch is different from current.
• Slice -> array/vector works for constant slice lengths.

Fixes

• Fixes issue where single character filenames like 'a.c3' would be rejected.
• Better errors when index type doesn't match len() when doing user defined foreach.
• Fixes to to_int for hexadecimal strings.
• Fixed issue when using a generic type from a generic type.
• Bug with vector parameters when the size > 2 and modified.
• Missing error on assigning to in-parameters through subscripting.
• Inference of a vector on the lhs of a binary expression would cause a crash.
• Fixes to PriorityQueue
• On Aarch64 use the correct frame pointer type.
• On Aarch64 macOS, ensure the minimum version is 11.0 (Big Sur)
• Fixes to the yacc grammar.
• Dsym generation on macOS will correctly emit -arch.
• Stacktrace on signals on Linux when backtrace is available.

Stdlib changes

• Allow to_int family functions take a base, parsing base 2-10 and 16.
• delete and delete_range added to DString.
• Splitter iterator added.
• splitter and iterator String methods.
• load_new, load_buffer and load_temp std::io::file functions.

0.5 has feature stability guarantees, so any code written for 0.5.0 will with on all of 0.5.x.

If you want to read more about C3, check out the documentation: https://c3-lang.org or download it and try it out: https://github.com/c3lang/c3c

Say hello to C3 0.5

Originally from: https://c3.handmade.network/blog/p/8824-say_hello_to_c3_0.5

C3 is a programming language that builds on the syntax and semantics of the C language, with the goal of evolving it while still retaining familiarity for C programmers. It's an evolution, not a revolution: the C-like for programmers who like C.

It is finally time to release C3 0.5. This version is the first version of the C3 compiler (and by extension, the C3 language) which is feature-stable.

Before 0.5, the language changed in the same minor version, so the 0.4.1 version of the compiler might not compile code written for 0.4.20 and vice versa.

From 0.5 and forward this changes: each future version will have its own branch where bug fixes will happen, but otherwise the features are frozen. New features will be reserved for the dev and master branches. Consequently, as we announce 0.5, work will actually move on to 0.6 which is where the active development will happen.

This allows people to pick a version to confidently work with, knowing that there will be no changes to language semantics or the standard library.

Feature complete

With 0.5, C3 language itself can also be considered feature complete, and for 0.6, 0.7, 0.8, 0.9 the focus will be on the standard library. A good standard library should address real life use-cases, to solve commonly encountered issues of the users.

In order to properly know what those use-cases are, a diverse set of projects must be written in C3. And for people to build non-trivial projects in C3 without problems there must be some stability guarantees to the compiler itself. This is what 0.5 provides, and why we now switch forward to refining the standard library.

Explore C3

Interested in trying out C3 0.5? Learn more on the language's official site: https://c3-lang.org. Obtain the compiler from GitHub at https://github.com/c3lang/c3c/issues and join the community shaping the future of the C3 programming language.

Comments

Comment by Christoffer Lernö

C3 is a programming language that builds on the syntax and semantics of the C language, with the goal of evolving it while still retaining familiarity for C programmers. It's an evolution, not a revolution: the C-like for programmers who like C.

It is finally time to release C3 0.5. This version is the first version of the C3 compiler (and by extension, the C3 language) which is feature-stable.

Before 0.5, the language changed in the same minor version, so the 0.4.1 version of the compiler might not compile code written for 0.4.20 and vice versa.

From 0.5 and forward this changes: each future version will have its own branch where bug fixes will happen, but otherwise the features are frozen. New features will be reserved for the dev and master branches. Consequently, as we announce 0.5, work will actually move on to 0.6 which is where the active development will happen.

This allows people to pick a version to confidently work with, knowing that there will be no changes to language semantics or the standard library.

Feature complete

With 0.5, C3 language itself can also be considered feature complete, and for 0.6, 0.7, 0.8, 0.9 the focus will be on the standard library. A good standard library should address real life use-cases, to solve commonly encountered issues of the users.

In order to properly know what those use-cases are, a diverse set of projects must be written in C3. And for people to build non-trivial projects in C3 without problems there must be some stability guarantees to the compiler itself. This is what 0.5 provides, and why we now switch forward to refining the standard library.

Explore C3

Interested in trying out C3 0.5? Learn more on the language's official site: https://c3-lang.org. Obtain the compiler from GitHub at https://github.com/c3lang/c3c/issues and join the community shaping the future of the C3 programming language.

Comment by Christoffer Lernö

The change list for 0.5:

Changes / improvements

• Trackable allocator with leak allocation backtraces.
• $defined can take a list of expressions.
• $and compile time "and" which does not check expressions after the first is an error.
• $is_const returns true if an expression is compile time const.
• $assignable returns true is an expression may be implicitly cast to a type.
• $checks and @checked removed, replaced by an improved $defined
• Asm string blocks use AT&T syntax for better reliability.
• Distinct methods changed to separate syntax.
• 'exec' directive to run scripts at compile time.
• Project key descriptions in --list command.
• Added init-lib to simplify library creation.
• Local const work like namespaced global const.
• Added $$atomic_fetch_* builtins.
• vectors may now contain pointers.
• void! does not convert to anyfault.
• $$masked_load / $$masked_store / $$gather / $$scatter for vector masked load/store.
• $$select builtin for vector masked select.
• Added builtin benchmarks by benchmark, compile-benchmark commands and @benchmark attribute.
• Subtype matching in type switches.
• Added parentof typeid property.
• Slice assignment is expanded.
• Enforced optional handling.
• Better dead code analysis, and added dead code errors.
• Exhaustive switches with enums has better analysis.
• Globals may now be initialized with optional values.
• New generic syntax.
• Slice initialization.
• $feature for feature flags.
• Native stacktrace for Linux, MacOS and Windows.
• Macro ref parameters are now of pointer type and ref parameters are not assignable.
• Added nextcase default.
• Added $embed to embed binary data.
• Ad hoc generics are now allowed.
• Allow inferred type on method first argument.
• Fix to void expression blocks
• Temporary objects may now invoke methods using ref parameters.
• Delete object files after successful linking.
• Compile time subscript of constant strings and bytes.
• @if introduced, other top level conditional compilation removed.
• Dynamically dispatched interfaces with optional methods.
• $if now uses $if <expr>: syntax.
• $assert now uses $assert <expr> : <optional message>
• $error is syntax sugar for $assert false : "Some message"
• $include, $echo no longer has mandatory () around the arguments.
• $exec for including the output of files.
• assert no longer allows "try unwrap"
• Updated cpu arguments for x86
• Removed support for ranged case statements that were floats or enums, or non-constant.
• nextcase with a constant expression that does not match any case is an error.
• Dropped support for LLVM 13-14.
• Updated grammar and lexer definition.
• Removal of $elif.
• any / anyfault may now be aliased.
• @stdcall etc removed in favor of @callconv
• Empty fault definitions is now an error.
• Better errors on incorrect bitstruct syntax.
• Internal use wildcard type rather than optional wildcard.
• Experimental scaled vector type removed.
• Disallow parameterize attributes without parameters eg define @Foo() = { @inline }.
• Handle @optreturn contract, renamed @return!.
• Restrict interface style functions.
• Optional propagation and assignment '!' and '?' are flipped.
• Add l suffix (alias for i64).
• Allow getting the underlying type of anyfault.
• De-duplicate string constants.
• Change @extname => @extern.
• define and typedef removed.
• define is replaced by def.
• LLVM "wrapper" library compilation is exception free.
• private is replaced by attribute @private.
• Addition of @local for file local visibility.
• Addition of @public for overriding default visibility.
• Default visibility can be overridden per module compile unit. Eg module foo @private.
• Optimized macro codegen for -O0.
• Addition of unary +.
• Remove possibility to elide length when using ':' for slices.
• Remove the : and ; used in $if, $switch etc.
• Faults have an ordinal.
• Generic module contracts.
• Type inference on enum comparisons, e.g foo_enum == ABC.
• Allow {} to initialize basic types.
• String literals default to String.
• More const modification detection.
• C3L zip support.
• Support printing object files.
• Downloading of libraries using vendor "fetch".
• Structural casts removed.
• Added "native" option for vector capability.
• $$shufflevector replaced with $$swizzle and $$swizzle2.
• Builtin swizzle accessors.
• Lambdas, e.g a = int(x, y) => x + y.
• $$FILEPATH builtin constant.
• variant renamed any.
• anyerr renamed anyfault.
• Added $$wasm_memory_size and $$wasm_memory_grow builtins.
• Add "link-args" for project.
• Possible to suppress entry points using --no-entry.
• Added memory-env option.
• Use the .wasm extension on WASM binaries.
• Update precedence clarification rules for ^|&.
• Support for casting any expression to void.
• Win 32-bit processor target removed.
• Insert null-check for contracts declaring & params.
• Support user defined attributes in generic modules.
• --strip-unused directive for small binaries.
• $$atomic_store and $$atomic_load added.
• usz/isz replaces usize and isize.
• @export attribute to determine what is visible in precompiled libraries.
• Disallow obviously wrong code returning a pointer to a stack variable.
• Add &^| operations for bitstructs.
• @noinit replaces = void to opt-out of implicit zeroing.
• Multiple declarations are now allowed in most places, eg int a, b;.
• Allow simplified (boolean) bitstruct definitions.
• Allow @test to be placed on module declarations.
• Updated name mangling for non-exports.
• defer catch and defer try statements added.
• Better errors from $assert.
• @deprecated attribute added.
• Allow complex array length inference, eg int[*][2][*] a = ....
• Cleanup of cast code.
• Removal of generic keyword.
• Remove implicit cast enum <-> int.
• Allow enums to use a distinct type as the backing type.
• Update addition and subtraction on enums.
• @ensure checks only non-optional results.
• assert may now take varargs for formatting.

Stdlib changes

• Tracking allocator with location.
• init_new/init_temp for allocating init methods.
• DString.printf is now DString.appendf.
• Tuple and Maybe types.
• .as_str() replaced by .str_view()
• Added math::log(x , base) and math::ln(x).
• Hashmap keys implicitly copied if copy/free are defined.
• Socket handling.
• csv package.
• Many random functions.
• Updated posix/win32 stdlib namespacing
• process stdlib
• Stdlib updates to string.
• Many additions to List: remove, array_view, add_all, compact etc
• Added dstringwriter.
• Improved printf formatting.
• is_finite/is_nam/is_inf added.
• OnStack allocator to easily allocate a stack buffer.
• File enhancements: mkdir, rmdir, chdir.
• Path type for file path handling.
• Distinct String type.
• VarString replaced by DString.
• Removal of std::core::str.
• JSON parser and general Object type.
• Addition of EnumMap.
• RC4 crypto.
• Matrix identity macros.
• compare_exchange added.
• printfln and println renamed printfn and printn.
• Support of roundeven.
• Added easings.
• Updated complex/matrix, added quaternion maths.
• Improved support for freestanding.
• Improved windows main support, with @winmain annotations.
• SimpleHeapAllocator added.
• Added win32 standard types.
• Added saturated math.
• Added @expect, @unlikely and @likely macros.
• Temp allocator uses memory-env to determine starting size.
• Temp allocator is now accessed using mem::temp(), heap allocator using mem::heap().
• Float parsing added.
• Additions to std::net, ipv4/ipv6 parsing.
• Stream api.
• Random api.
• Sha1 hash function.
• Extended enumset functionality.
• Updated malloc/calloc/realloc/free removing old helper functions.
• Added TrackingAllocator.
• Add checks to prevent incorrect alignment on malloc.
• Updated clamp.
• Added Clock and DateTime.
• Added posix socket functions.

Fixes

• Structs returned from macros and then indexed into directly could previously be miscompiled.
• Naked functions now correctly handles asm.
• Indexing into arrays would not always widen the index safely.
• Macros with implicit return didn't correctly deduct the return type.
• Reevaluating a bitstruct (due to checked) would break.
• Fix missing comparison between any.
• Fix issue of designated initializers containing bitstructs.
• Fix issue of designated initializers that had optional arguments.
• Fixed ++ and -- for bitstructs.
• Fix to bug where library source files were sometimes ignored.
• Types of arrays and vectors are consistently checked to be valid.
• Anonymous bitstructs check of duplicate member names fixed.
• Assignment to anonymous bitstruct members in structs.
• Fix casts on empty initializers.
• Fix to DString reserve.
• Fix where aliases did not do arithmetic promotion.
• @local declarations in generic modules available by accident.
• Fixes missing checks to body arguments.
• Do not create debug declaration for value-only parameter.
• Bug in alignment for atomics.
• Fix to bug when comparing nested arrays.
• Fix to bug when a macro is using rethrow.
• Fixes bug initializing a const struct with a const struct value.
• Fixes bug when void is passed to an "any"-vararg.
• Fixed defer/return value ordering in certain cases.
• Fixes to the x64 ABI.
• Updates to how variadics are implemented.
• Fixes to shift checks.
• Fixes to string parsing.
• Bug when rethrowing an optional from a macro which didn't return an optional.
• Fixed issues with ranged cases.
• Disallow trailing ',' in function parameter list.
• Fixed errors on flexible array slices.
• Fix of readdir issues on macOS.
• Fix to slice assignment of distinct types.
• Fix of issue casting subarrays to distinct types.
• Fixes to split, rindex_of.
• List no longer uses the temp allocator by default.
• Remove test global when not in test mode.
• Fix sum/product on floats.
• Fix error on void! return of macros.
• Removed too permissive casts on subarrays.
• Using C files correctly places objects in the build folder.
• Fix of overaligned deref.
• Fix negating a float vector.
• Fix where $typeof(x) { ... } would not be a valid compound literal.
• Fix so that using var in if (var x = ...) works correctly.
• Fix int[] -> void* casts.
• Fix in utf8to16 conversions.
• Updated builtin checking.
• Reduce formatter register memory usage.
• Fixes to the "any" type.
• Fix bug in associated values.
• More RISC-V tests and fixes to the ABI.
• Fix issue with hex floats assumed being double despite f suffix.
• Fix of the tan function.
• Fixes to the aarch64 ABI when passing invalid vectors.
• Fix creating typed compile time variables.
• Fix bug in !floatval codegen.
• Fix of visibility issues for generic methods.
• Fixes to $include.
• Fix of LLVM codegen for optionals in certain cases.
• Fix of $vasplat when invoked repeatedly.
• Fix to $$DATE.
• Fix of attributes on nested bitstructs.
• Fix comparing const values > 64 bits.
• Defer now correctly invoked in expressions like return a > 0 ? Foo.ABC! : 1.
• Fix conversion in if (int x = foo()).
• Delay C ABI lowering until requested to prevent circular dependencies.
• Fix issue with decls accidentally invalidated during $checked eval.
• Fold optional when casting slice to pointer.
• Fixed issue when using named arguments after varargs.
• Fix bug initializing nested struct/unions.
• Fix of bool -> vector cast.
• Correctly widen C style varargs for distinct types and optionals.
• Fix of too aggressive codegen in ternary codegen with array indexing.

Comment by Christoffer Lernö

It allows the language to be easily parsable. The classic problem in a C-like grammar is that it is ambiguous with respect to types vs variables. In C this is typically solved using the "lexer hack", where the parser feeds types back into the lexer. Other methods include outlawing certain types of expressions and using infinite lookahead, this is the method D uses for example.

In C3, the distinct naming rules for types disambiguates the grammar, making it LL(1). Also see here: https://c3-lang.org/faq/#syntax-language-design

So to be clear, it's not about trying to enforce some arbitrary name standards, but rather to simplify the grammar. Picking PascalCase for the types was pretty much the only possible choice. I might write a blog post about this some time.

Too much power, too poor accuracy - the story of $checks in C3

Originally from: https://c3.handmade.network/blog/p/8810-too_much_power%252C_too_poor_accuracy_-_the_story_of_checks_in_c3

Recently C3 lost its $checks() function. It would take any sequence of declarations and expressions, and if it failed to semantically check anywhere, return false.

It was an extremely powerful and flexible way of testing pretty much anything at compile time. Some examples:

// Test if a value may be indexed:
$checks(a[0]);
// Test if something supports addition:
$checks(a + a);
// Test if you can assign something to the type of another variable
$checks(b = a);
// Test if you can call a function with the values of two variables
$checks(foo(a, b));
// Check if a type has a particular field
$checks(Foo x, x.my_field);
// Check if a type is ordered
$checks(Foo x, x < x);

In essence, $checks was a Swiss Army knife for compile-time validation, making it redundant to employ multiple compile-time functions like $defined(x). So, why did we part ways with $checks (and its contract counterpart @checked)?

Well, it turns out that with power comes also lack of clarity. Take, for example, the $checks(foo(a, b)) call – it could potentially fail for a multitude of reasons:

1. foo might not be visible in the scope.
2. foo needs to be called with the module name, e.g. my_module::foo
3. foo might not be a callable variable pointer or function.
4. a might not be visible in the scope.
5. b might not be visible in the scope.
6. foo might take fewer than 2 or more than 2 arguments.
7. There could be a type mismatch between a and the first parameter of foo.
8. There could be a type mismatch between b and the second parameter of foo.

So while we might have wanted to test for some of these, it might fail for any of the listed cases and there is no way we can determine which one, unless we move it out of the $checks and test it so that it errors just the same way.

While this is a problem when writing the $checks, it also poses a problem when refactoring, as it is hard to tell when you accidentally change something that breaks inside of $checks, causing it to reject legitimate parameters.

So $checks unfortunately combines power with inexactness. In fact, its power comes from being inexact and just bundling all the implicit checks together.

The alternative solution

C3 already had $defined(...) which would do a lightweight check if a variable or a field was defined. Its functionality had almost completely been eclipsed by $checks(...) but now got a new life: $defined would semantically check all but the outermost part of a nested expression. The final expression would then be conditionally checked.

The new behaviour was reminiscent of $checks, but would only have a single "tested" semantic check. For example, $defined(foo(a, b)) would return true if it checked correctly, and false only if "foo" wasn't callable or didn't accept 2 arguments.

The downside is that $defined must be carefully crafted to correctly do each "test" it supports.

But all in all, this is a substantial upgrade to correct compile time checking, which is very important in C3.

Addition: without $checks the various examples instead become:

// Test if a value may be indexed:
$defined(a[0]);
// Test if something supports addition:
types::is_numerical($typeof(a))
// Test if you can assign something to the type of another variable
$assignable(a, $typeof(b));
// Test if you can call a function with the values of two variables
$defined(foo(a, b));
// Check if a type has a particular field
$defined(Foo{}.my_field);
// Check if a type is ordered
Foo.is_ordered

Comments

Comment by Christoffer Lernö

Recently C3 lost its $checks() function. It would take any sequence of declarations and expressions, and if it failed to semantically check anywhere, return false.

It was an extremely powerful and flexible way of testing pretty much anything at compile time. Some examples:

// Test if a value may be indexed:
$checks(a[0]);
// Test if something supports addition:
$checks(a + a);
// Test if you can assign something to the type of another variable
$checks(b = a);
// Test if you can call a function with the values of two variables
$checks(foo(a, b));
// Check if a type has a particular field
$checks(Foo x, x.my_field);
// Check if a type is ordered
$checks(Foo x, x < x);

In essence, $checks was a Swiss Army knife for compile-time validation, making it redundant to employ multiple compile-time functions like $defined(x). So, why did we part ways with $checks (and its contract counterpart @checked)?

Well, it turns out that with power comes also lack of clarity. Take, for example, the $checks(foo(a, b)) call – it could potentially fail for a multitude of reasons:

1. foo might not be visible in the scope.
2. foo needs to be called with the module name, e.g. my_module::foo
3. foo might not be a callable variable pointer or function.
4. a might not be visible in the scope.
5. b might not be visible in the scope.
6. foo might take fewer than 2 or more than 2 arguments.
7. There could be a type mismatch between a and the first parameter of foo.
8. There could be a type mismatch between b and the second parameter of foo.

So while we might have wanted to test for some of these, it might fail for any of the listed cases and there is no way we can determine which one, unless we move it out of the $checks and test it so that it errors just the same way.

While this is a problem when writing the $checks, it also poses a problem when refactoring, as it is hard to tell when you accidentally change something that breaks inside of $checks, causing it to reject legitimate parameters.

So $checks unfortunately combines power with inexactness. In fact, its power comes from being inexact and just bundling all the implicit checks together.

The alternative solution

C3 already had $defined(...) which would do a lightweight check if a variable or a field was defined. Its functionality had almost completely been eclipsed by $checks(...) but now got a new life: $defined would semantically check all but the outermost part of a nested expression. The final expression would then be conditionally checked.

The new behaviour was reminiscent of $checks, but would only have a single "tested" semantic check. For example, $defined(foo(a, b)) would return true if it checked correctly, and false only if "foo" wasn't callable or didn't accept 2 arguments.

The downside is that $defined must be carefully crafted to correctly do each "test" it supports.

But all in all, this is a substantial upgrade to correct compile time checking, which is very important in C3.

Addition: without $checks the various examples instead become:

// Test if a value may be indexed:
$defined(a[0]);
// Test if something supports addition:
types::is_numerical($typeof(a))
// Test if you can assign something to the type of another variable
$assignable(a, $typeof(b));
// Test if you can call a function with the values of two variables
$defined(foo(a, b));
// Check if a type has a particular field
$defined(Foo{}.my_field);
// Check if a type is ordered
Foo.is_ordered

Some guidelines to new syntax design

Originally from: https://c3.handmade.network/blog/p/8778-some_guidelines_to_new_syntax_design

Syntax discussions tend to be highly contextual. The syntax of a language is not a standalone, separate entity, but rather interacts with what type of algorithmic solutions you envision users to employ. On top of that, one must be aware of that syntax shapes the solutions users will prefer in sometimes unpredictable ways.

This makes completely new syntax very hard to analyze. And also hard to write any guidelines for.

That said, I think there are some things we can say about syntax design, to form some very simple (and obvious) guidelines:

1. In general, an easy-to-parse syntax tend to be easier for a user to read quickly than a complex-to-parse syntax.
2. Newly invented syntax will initially be harder for people to grok than established syntax. So it is bad if you try to make experienced programmers understand it "at a glance".
3. Newly invented syntax does makes the language feel more "different" (unique, inventive etc) than established syntax. So it is good if you want to make the language stand out as being different at a glance.
4. It's harder to know the downsides of newly invented syntax. So much more research is needed, and it's important to be ready to change it down the line if it doesn't work out.
5. One's personal opinions of what "nice looking syntax" is very unlikely to be the objectively most accurate opinion, so be aware how that "beautiful" syntax might be hideous to someone else.

Happy hacking!

Comments

Comment by Christoffer Lernö

Syntax discussions tend to be highly contextual. The syntax of a language is not a standalone, separate entity, but rather interacts with what type of algorithmic solutions you envision users to employ. On top of that, one must be aware of that syntax shapes the solutions users will prefer in sometimes unpredictable ways.

This makes completely new syntax very hard to analyze. And also hard to write any guidelines for.

That said, I think there are some things we can say about syntax design, to form some very simple (and obvious) guidelines:

1. In general, an easy-to-parse syntax tend to be easier for a user to read quickly than a complex-to-parse syntax.
2. Newly invented syntax will initially be harder for people to grok than established syntax. So it is bad if you try to make experienced programmers understand it "at a glance".
3. Newly invented syntax does makes the language feel more "different" (unique, inventive etc) than established syntax. So it is good if you want to make the language stand out as being different at a glance.
4. It's harder to know the downsides of newly invented syntax. So much more research is needed, and it's important to be ready to change it down the line if it doesn't work out.
5. One's personal opinions of what "nice looking syntax" is very unlikely to be the objectively most accurate opinion, so be aware how that "beautiful" syntax might be hideous to someone else.

Happy hacking!

Compile-time and short-circuit evaluation

Originally from: https://c3.handmade.network/blog/p/8773-compile-time_and_short-circuit_evaluation

Recently a user had a problem with the following code in C3:

$if $foo != "" && $foo[0] != '_':
    ...
$endif

As a reminder, compile time evaluation is distinguished using a $ sigil, so in this case the idea was to check whether the compile time variable $foo was an empty string, and if it wasn't, compare the first character with '_'.

If $foo is indeed an empty string, this code will fail at compile time.

This is because constant folding in C3 follows semantic evaluation, and a binary expression will first type check the sub expressions && was evaluated. That is, at compile time there is no short-circuit evaluation.

The curious effect of short-circuit evaluation

We could say that for && we only evaluate the left hand side, and if that one is false, then we don't evaluate the rest. This is perfectly legitimate behaviour BUT it would mean this would pass semantic checking as well:

if (false && okeoefkepofke[3.141592])
{
    ...
}

Why? Because constant folding would need to work the same way: we evaluate the first part to false, so now we never check the expression okeoefkepofke[3.141592].

So now we got this big piece of code that is wrong and never checked...

But obviously no one would write that, right? Except for something like this is quite reasonable code:

macro foo($foo)
{
  if ($foo && abc()) { ... }
}

This problem is not unique, people using any sort of dynamically typed scripting languages will be familiar with this exact problem. And the solution – if you care about the code actually working – is to write more tests.

Trying to eat the cake and keep it

One possibility one can consider, is to have short-circuit behaviour only in compile time constant environments, so:

// Const global? Don't evaluate the right hand side.
const bool FOO = false && foewkfoewkf[fefeji]; 

fn void test()
{
    // Compile time conditional? Don't evaluate the right hand side
    $if false && foofoekfe[kfiejfie]:
        ...
    $endif
    // And same with switch:
    $switch
        $case false && fokeokfe[ofkeofk]:
            ...
    $endswitch
    // But this would be an error:
    bool b = false && fokefoek[ofofke]; // Error!
}

But if "never short-circuiting" is annoying and unexpected, and "always short-circuiting" requires much more testing, this "a little of both", creates a corner in the language which can be just as problematic as the former two. Having expression evaluation behave differently depending on where it's evaluated, is something likely to confuse even experienced users.

As usual, language design is a trade-off

For C3, semantic checking is prioritized over compile time convenience. I think everyone who's been working with macros in C3 knows the lazy evaluation of macros can easily hide bugs already, and having short-circuiting constant evaluation would just magnify this problem.

There are languages that consistently uses short-circuiting constant evaluation at compile time instead. This allows leveraging this the feature for all its conditional compilation. Where C3 uses $if or $switch and very clear "this is evaluated at compile time" blocks to facilitate finding compile-time bugs, other languages may prefer to streamline the look of the code allowing compile-time and runtime evaluation blur but also being consistent in following the same rules. While this comes at the aforementioned added cost of testing, it might be a trade-off its users prefer.

Comments

Comment by Christoffer Lernö

Recently a user had a problem with the following code in C3:

$if $foo != "" && $foo[0] != '_':
    ...
$endif

As a reminder, compile time evaluation is distinguished using a $ sigil, so in this case the idea was to check whether the compile time variable $foo was an empty string, and if it wasn't, compare the first character with '_'.

If $foo is indeed an empty string, this code will fail at compile time.

This is because constant folding in C3 follows semantic evaluation, and a binary expression will first type check the sub expressions && was evaluated. That is, at compile time there is no short-circuit evaluation.

The curious effect of short-circuit evaluation

We could say that for && we only evaluate the left hand side, and if that one is false, then we don't evaluate the rest. This is perfectly legitimate behaviour BUT it would mean this would pass semantic checking as well:

if (false && okeoefkepofke[3.141592])
{
    ...
}

Why? Because constant folding would need to work the same way: we evaluate the first part to false, so now we never check the expression okeoefkepofke[3.141592].

So now we got this big piece of code that is wrong and never checked...

But obviously no one would write that, right? Except for something like this is quite reasonable code:

macro foo($foo)
{
  if ($foo && abc()) { ... }
}

This problem is not unique, people using any sort of dynamically typed scripting languages will be familiar with this exact problem. And the solution – if you care about the code actually working – is to write more tests.

Trying to eat the cake and keep it

One possibility one can consider, is to have short-circuit behaviour only in compile time constant environments, so:

// Const global? Don't evaluate the right hand side.
const bool FOO = false && foewkfoewkf[fefeji]; 

fn void test()
{
    // Compile time conditional? Don't evaluate the right hand side
    $if false && foofoekfe[kfiejfie]:
        ...
    $endif
    // And same with switch:
    $switch
        $case false && fokeokfe[ofkeofk]:
            ...
    $endswitch
    // But this would be an error:
    bool b = false && fokefoek[ofofke]; // Error!
}

But if "never short-circuiting" is annoying and unexpected, and "always short-circuiting" requires much more testing, this "a little of both", creates a corner in the language which can be just as problematic as the former two. Having expression evaluation behave differently depending on where it's evaluated, is something likely to confuse even experienced users.

As usual, language design is a trade-off

For C3, semantic checking is prioritized over compile time convenience. I think everyone who's been working with macros in C3 knows the lazy evaluation of macros can easily hide bugs already, and having short-circuiting constant evaluation would just magnify this problem.

There are languages that consistently uses short-circuiting constant evaluation at compile time instead. This allows leveraging this the feature for all its conditional compilation. Where C3 uses $if or $switch and very clear "this is evaluated at compile time" blocks to facilitate finding compile-time bugs, other languages may prefer to streamline the look of the code allowing compile-time and runtime evaluation blur but also being consistent in following the same rules. While this comes at the aforementioned added cost of testing, it might be a trade-off its users prefer.

Inspirations for C3's features

Originally from: https://c3.handmade.network/blog/p/8723-inspirations_for_c3%2527s_features

When designing a new programming language, research is incredibly important. While research can be investigating new syntax and new semantics, most of it is actually looking at other language's features and seeing if anything worked extra well and wether it could be useful for your own language.

C3 is derived from C2, which in turn is an evolution of C, so the basis of the language itself is clear. But what about the features on top of C -where do they come from? I thought it might be amusing to list the features and where they originated.

Features and where they come from

Modules – Java was probably the primary inspiration for a lot of it, since it has a very simple and well understood system with packages. However, Java's imports are actually only about visibility, not about really importing anything, so there are very clear differences. I've written more in detail here.

Generic modules - This was inspired by macro based container libraries in C, as well as ASTEC's "@module" macro.

Faults/optionals - Originally this was similar to Zig's design, but took on inspiration from Herbceptions, Haskell/Rust results, C and Go error handling into something original.

Macros - This was based largely on ASTEC but added things like iteration.

Struct subtyping - This is a Plan9 feature that also ended up in Go. I got it from reading about the Plan9 C compiler.

Slices - This exists in many languages, it's hard to say what languages I based it on.

Slicing syntax - The ^1 syntax comes from C#, otherwise it's mostly D with some looks at Swift and Odin.

Contracts - While a lot of languages try to add a bit of contract support, Eiffel is the language I looked at. Placing the contracts in the docs was a change for C3.

Def - I started by looking at D. The inclusion of "distinct" types comes from Odin. The restriction that function types only to be accessed through def is from C2. The idea that generic modules are instantiated using def occurs in earlier languages, I remember looking at Ada in particular.

Reflection - I'd say Jai got the ball rolling here, with some additional inspiration from Odin. It was clear from the start that reflection like Java or Objective-C was out of question, and that Jai's runtime information was more than I wanted. I read about reflection in other languages as well, with D having quite a bit of influence on the syntax.

Operator overloading - I certainly looked at overloading in C++, D and other languages, but in the end the result was a bit in between everything.

Dynamic calls - This is from Objective-C.

Undefined behaviour - The C3 attitude to UB is strongly influenced by Odin, but doesn't go quite that far.

Implicit conversions - Originally this borrowed from Zig, but after a lot of research, it ended as a unique blend of C and Java ideas, without the need for untyped literals.

Precedence rules - Just trying to avoid retaining the poor precedence rules of C.

Project files - Derived from C2, but modified.

Any and typeid types - mainly inspired by Odin.

Enum associated values - derived from Java enums.

Bitstructs - inspired by PacketC.

Extended switch - pattern matching in many languages.

Flowtyping to unwrap - Java / Kotlin in JetBrains' IDEs.

Foreach - ObjC and Java originally. The idea (and syntax!) to allow getting values by ref comes from PHP.

Base64 and hex literals - Inspired by language "wish lists" on the web :D.

Zero init by default - Ultimately Odin convinced me this was a good idea.

Array/slice arithmetics - A subset of Odin and D functionality.

Type methods - An extension of C2 struct functions.

Attributes - Based on C2 attributes

Defer - Based on Swift and Jai defer. Extensions defer catch and defer try were added on top. While Zig has a errdefer which works like defer catch the C3 feature was developed without knowledge of that Zig addition(!)

Special syntax for compile time - Mostly driven by a need to make compile time clearer than compile time code in Zig.

Visibility rules - I did lots of research on this, so it's hard to say where it comes from. Certainly some I made up for C3. "Public by default" comes from Odin. Some ideas for export and visibility came from D.

Raw strings - I experimented with a lot of different styles, ultimately I picked Go style from comparing with Odin. Escaping a single backtick by having two in a row is also from some language, but unfortunately I don't recall which one.

Ranges in initializers - This is a GCC extension.

Expression block - This is a variant of the GCC statement expression that I changed be a self contained block where return only jumped out of the block. So it's an evolution of the GCC feature.

Ranges in case statements - Yes, this is a GCC extension as well.

Named arguments - Probably borrowed from Swift originally.

Trailing macro body - This is a unique functionality, but it is somewhat similar to trailing body lambdas in Ruby and later Swift.

Lambdas - These are syntactically very similar to Java's lambdas. But of course C3 does not capture closures.

Static initializers and finalizers - Syntactically somewhat derived from Java static blocks.

Function syntax - This is from C2, but in shortened form (C2 uses func)

Allocators - Influences from Jai, Odin and Zig, but ultimately C3 picks its own trade off.

Temp allocators - Mostly based off Odin originally.

Inline asm - Mostly based on MSVC inline asm.

Final words

On top of the above, C3 is of course indebted to all the people I've engaged in language discussions with over the years. I should mention Jon Goodwin (Cone) and Andrey Penechko (Vox) in particular, but I want to thank everyone who helped with thoughts and feedback (and complaints!) over the years.

Thank you!

If you are curious about C3 you can try it at https://learn-c3.org or download the compiler from https://github.com/c3lang/c3c

P.S. A bonus tidbit: the use of printn and printfn instead of println and printfln comes from F#

Comments

Comment by Christoffer Lernö

When designing a new programming language, research is incredibly important. While research can be investigating new syntax and new semantics, most of it is actually looking at other language's features and seeing if anything worked extra well and wether it could be useful for your own language.

C3 is derived from C2, which in turn is an evolution of C, so the basis of the language itself is clear. But what about the features on top of C -where do they come from? I thought it might be amusing to list the features and where they originated.

Features and where they come from

Modules – Java was probably the primary inspiration for a lot of it, since it has a very simple and well understood system with packages. However, Java's imports are actually only about visibility, not about really importing anything, so there are very clear differences. I've written more in detail here.

Generic modules - This was inspired by macro based container libraries in C, as well as ASTEC's "@module" macro.

Faults/optionals - Originally this was similar to Zig's design, but took on inspiration from Herbceptions, Haskell/Rust results, C and Go error handling into something original.

Macros - This was based largely on ASTEC but added things like iteration.

Struct subtyping - This is a Plan9 feature that also ended up in Go. I got it from reading about the Plan9 C compiler.

Slices - This exists in many languages, it's hard to say what languages I based it on.

Slicing syntax - The ^1 syntax comes from C#, otherwise it's mostly D with some looks at Swift and Odin.

Contracts - While a lot of languages try to add a bit of contract support, Eiffel is the language I looked at. Placing the contracts in the docs was a change for C3.

Def - I started by looking at D. The inclusion of "distinct" types comes from Odin. The restriction that function types only to be accessed through def is from C2. The idea that generic modules are instantiated using def occurs in earlier languages, I remember looking at Ada in particular.

Reflection - I'd say Jai got the ball rolling here, with some additional inspiration from Odin. It was clear from the start that reflection like Java or Objective-C was out of question, and that Jai's runtime information was more than I wanted. I read about reflection in other languages as well, with D having quite a bit of influence on the syntax.

Operator overloading - I certainly looked at overloading in C++, D and other languages, but in the end the result was a bit in between everything.

Dynamic calls - This is from Objective-C.

Undefined behaviour - The C3 attitude to UB is strongly influenced by Odin, but doesn't go quite that far.

Implicit conversions - Originally this borrowed from Zig, but after a lot of research, it ended as a unique blend of C and Java ideas, without the need for untyped literals.

Precedence rules - Just trying to avoid retaining the poor precedence rules of C.

Project files - Derived from C2, but modified.

Any and typeid types - mainly inspired by Odin.

Enum associated values - derived from Java enums.

Bitstructs - inspired by PacketC.

Extended switch - pattern matching in many languages.

Flowtyping to unwrap - Java / Kotlin in JetBrains' IDEs.

Foreach - ObjC and Java originally. The idea (and syntax!) to allow getting values by ref comes from PHP.

Base64 and hex literals - Inspired by language "wish lists" on the web :D.

Zero init by default - Ultimately Odin convinced me this was a good idea.

Array/slice arithmetics - A subset of Odin and D functionality.

Type methods - An extension of C2 struct functions.

Attributes - Based on C2 attributes

Defer - Based on Swift and Jai defer. Extensions defer catch and defer try were added on top. While Zig has a errdefer which works like defer catch the C3 feature was developed without knowledge of that Zig addition(!)

Special syntax for compile time - Mostly driven by a need to make compile time clearer than compile time code in Zig.

Visibility rules - I did lots of research on this, so it's hard to say where it comes from. Certainly some I made up for C3. "Public by default" comes from Odin. Some ideas for export and visibility came from D.

Raw strings - I experimented with a lot of different styles, ultimately I picked Go style from comparing with Odin. Escaping a single backtick by having two in a row is also from some language, but unfortunately I don't recall which one.

Ranges in initializers - This is a GCC extension.

Expression block - This is a variant of the GCC statement expression that I changed be a self contained block where return only jumped out of the block. So it's an evolution of the GCC feature.

Ranges in case statements - Yes, this is a GCC extension as well.

Named arguments - Probably borrowed from Swift originally.

Trailing macro body - This is a unique functionality, but it is somewhat similar to trailing body lambdas in Ruby and later Swift.

Lambdas - These are syntactically very similar to Java's lambdas. But of course C3 does not capture closures.

Static initializers and finalizers - Syntactically somewhat derived from Java static blocks.

Function syntax - This is from C2, but in shortened form (C2 uses func)

Allocators - Influences from Jai, Odin and Zig, but ultimately C3 picks its own trade off.

Temp allocators - Mostly based off Odin originally.

Inline asm - Mostly based on MSVC inline asm.

Final words

On top of the above, C3 is of course indebted to all the people I've engaged in language discussions with over the years. I should mention Jon Goodwin (Cone) and Andrey Penechko (Vox) in particular, but I want to thank everyone who helped with thoughts and feedback (and complaints!) over the years.

Thank you!

If you are curious about C3 you can try it at https://learn-c3.org or download the compiler from https://github.com/c3lang/c3c

P.S. A bonus tidbit: the use of printn and printfn instead of println and printfln comes from F#

Language design bullshitters

Originally from: https://c3.handmade.network/blog/p/8721-language_design_bullshitters

Inevitably people will ask "what language should I choose for my compiler?".

The answer is really: "you can use any language, so all things being equal, pick one you're good at."

Of course there are caveats:

1. You want it to go really fast? Then C is better than Python.
2. Are you making a DSL? Then you probably want to do it in the host language.
3. Do you want to experiment with some parsing techniques? Then some languages might be a better fit than others

... and so on.

So when someone says something like "C is a bad choice for writing a compiler" as a general statement, you know they are just making it up as they go along.

The C3 compiler is written in C, and there is frankly no other language I could have picked that would have been a substantially better choice. – Sure, writing it in C2 or Odin would certainly have avoided some of C's warts, but the difference would not have been significant. And doing an OO-style C++, or worse, Java, would just have pushed the compiler to slower and more bloated, with no additional benefits other than there are more Java programmers than C programmers.

"Say you are bad at programming without saying you're bad at programming"

So what do you think are the arguments against C?

"C memory management is hard".

My god, if you think your compiler has to have a lot of free and that is the hard part about writing a compiler then you have ABSOLUTELY ZERO business handing out advice on compilers - or programming.

(Memory allocation can be handled in different ways in a compiler, with the simplest way being using arena allocators)

"C doesn't have feature X, so it will be a nightmare writing a compiler for it"

This is the prime argument for people arguing for writing the compiler in Ocaml or some other functional language. "C doesn't have extensive support for pattern matching, how can I use [my preferred technique] without that??? IT'S IMPOSSIBLE AND NO ONE SHOULD TRY IT"

If you point out that there are plenty of compilers written in C, the argument becomes "yes, but they are old and it's not modern to use C".

A carnival of made up arguments

There is no lack of people who want to give advice on language design. Even language designers that actually know what they talk about struggle to give good advice that are applicable to your particular design if you ask about it. It's just hard.

With that in mind, guess what the quality of advice is from people who just have some theoretical knowledge of compiler and language design? Yes, it is as bad as you might guess.

Also, somewhat unfortunately, the group that has little experience is the ones who tend to have the most time to argue for things. Of course their arguments are made up of what they just happen to think is true and what they read on blogs they liked.

To sum it up

So you want to write a compiler? Get some advice for language design? Well do ask, but just keep in mind that most of what you read is just trash advice made up by people who actually don't know what they talk about. Especially on forums where there aren't many people who actually write compilers. You'll get bad advice in places dedicated to programming language design as well, but your odds of picking up some good advice is better.

And the proof is in the pudding: if you actually look at what compilers are written and what languages they're written in, you at least know what's been proven to be production ready. And do look at compiler performance too, because that will matter if you're serious about the project.

And to me, if I find out that someone is making things up, then clearly other things they say isn't trustworthy either. Language design seems to be one of those things people like to have opinions on because they know the risk of being called out for lying is low.

Comments

Comment by Christoffer Lernö

Inevitably people will ask "what language should I choose for my compiler?".

The answer is really: "you can use any language, so all things being equal, pick one you're good at."

Of course there are caveats:

1. You want it to go really fast? Then C is better than Python.
2. Are you making a DSL? Then you probably want to do it in the host language.
3. Do you want to experiment with some parsing techniques? Then some languages might be a better fit than others

... and so on.

So when someone says something like "C is a bad choice for writing a compiler" as a general statement, you know they are just making it up as they go along.

The C3 compiler is written in C, and there is frankly no other language I could have picked that would have been a substantially better choice. – Sure, writing it in C2 or Odin would certainly have avoided some of C's warts, but the difference would not have been significant. And doing an OO-style C++, or worse, Java, would just have pushed the compiler to slower and more bloated, with no additional benefits other than there are more Java programmers than C programmers.

"Say you are bad at programming without saying you're bad at programming"

So what do you think are the arguments against C?

"C memory management is hard".

My god, if you think your compiler has to have a lot of free and that is the hard part about writing a compiler then you have ABSOLUTELY ZERO business handing out advice on compilers - or programming.

(Memory allocation can be handled in different ways in a compiler, with the simplest way being using arena allocators)

"C doesn't have feature X, so it will be a nightmare writing a compiler for it"

This is the prime argument for people arguing for writing the compiler in Ocaml or some other functional language. "C doesn't have extensive support for pattern matching, how can I use [my preferred technique] without that??? IT'S IMPOSSIBLE AND NO ONE SHOULD TRY IT"

If you point out that there are plenty of compilers written in C, the argument becomes "yes, but they are old and it's not modern to use C".

A carnival of made up arguments

There is no lack of people who want to give advice on language design. Even language designers that actually know what they talk about struggle to give good advice that are applicable to your particular design if you ask about it. It's just hard.

With that in mind, guess what the quality of advice is from people who just have some theoretical knowledge of compiler and language design? Yes, it is as bad as you might guess.

Also, somewhat unfortunately, the group that has little experience is the ones who tend to have the most time to argue for things. Of course their arguments are made up of what they just happen to think is true and what they read on blogs they liked.

To sum it up

So you want to write a compiler? Get some advice for language design? Well do ask, but just keep in mind that most of what you read is just trash advice made up by people who actually don't know what they talk about. Especially on forums where there aren't many people who actually write compilers. You'll get bad advice in places dedicated to programming language design as well, but your odds of picking up some good advice is better.

And the proof is in the pudding: if you actually look at what compilers are written and what languages they're written in, you at least know what's been proven to be production ready. And do look at compiler performance too, because that will matter if you're serious about the project.

And to me, if I find out that someone is making things up, then clearly other things they say isn't trustworthy either. Language design seems to be one of those things people like to have opinions on because they know the risk of being called out for lying is low.

Comment by Christoffer Lernö

Absolutely, I don't know if needed to add that. General purpose languages should all be easy to make languages in.

And while asm isn't exactly the nicest abstraction to write a compiler in, that used to be what a lot of (AST-less) compilers were written in back in the days.