Scriptensity

Emscripten stuff on other blog

2011-12-05T10:22:00.001-08:00

As mentioned previously, I've moved to working fulltime on Emscripten now. So Emscripten-related blogposts will now be on my other blog. (You can also follow me on twitter.)

I'll use this blog for Syntensity-specific stuff, that is, about porting Sauerbraten and/or Syntensity which is based on it to the web. I have not had any progress to report on that recently, since I am completely blocked on OpenGL issues: I can compile the C++ to JS, but I can't convert the OpenGL code to WebGL. I've asked around for help but so far no luck, hopefully it will happen though.

Emscripten Updates

2011-11-10T17:35:00.001-08:00

Lots of stuff has happened with Emscripten, I haven't blogged because I've been too busy. Here are some updates:

Emscripten was used to compile the Android H264 codec to JavaScript, in a project called Broadway. (live demo)
I gave a talk about Emscripten at SPLASH 2011 in Portland. (slides)
I gave a talk about Emscripten at JSConf.EU in Berlin. (slides)
Performance of the generated code has been improving due to progress in the relevant projects (JS engines, Emscripten and Closure Compiler). Some numbers appear in the slides linked to above (the upper link is more recent).
Bundled headers. This makes it easier to use Emscripten on non-Linux platforms (Linux being the platform that most development is done on) and in a portable way that does not depend on your local system headers.
Some library bugfixes that resolved almost all the open issues on speak.js, the Emscripten port of eSpeak to JavaScript which lets you do text-to-speech on the web.
j2k.js, a port of OpenJPEG to JavaScript with a nice API, letting you decode JPEG2000 images on the web. This might be helpful with pdf.js.
Support for LLVM svn (soon to be 3.0). Note that revision 141881 is known to work, others should but not necessarily.
Many other improvements and bugfixes to Emscripten. I should probably formally release a 2.0 version, but I can't seem to decide when.
Finally, I am now working fulltime on Emscripten and Emscripten-related things (at Mozilla, where I already worked but on other stuff before). So progress on Emscripten will be faster :)

llvm-svn branch has been merged

2011-10-09T07:53:00.000-07:00

The llvm-svn branch of Emscripten has been merged to master, in preparation for Emscripten 2.0, after all tests have been fixed and all speed regressions resolved. If you are currently using master, the consequences of that are:

You should use LLVM's svn (soon to be 3.0). LLVM 2.9 might still work, but it isn't guaranteed. Also, LLVM 3.0 is deprecating llvm-gcc, so Emscripten no longer uses that in its tests (as with 2.9, it might still work, but it might not). clang in 3.0 is much improved and is able to compile much more code than 2.9, so llvm-gcc is less necessary; there is also dragonegg which combines LLVM and GCC in a different manner, but its website says it is not mature yet.
Emscripten now uses its own header files, not your system headers. That means that Emscripten should now work on all platforms exactly the same. However, if you were using Emscripten to compile something that relied on your system headers, you might need to change how your project is built (that is, tell it to use those headers and not just Emscripten's bundled ones in system/include). Note that if you do not use the bundled headers, you will probably need to use the -H flag with emscripten.py, which tells it what headers to parse for constants (library.js needs to be aware of constants in your library headers, so that it is synchronized with them).

Please report bugs if you find them. If there are no show-stoppers, Emscripten 2.0 will be released soon.

Road to Emscripten 2.0

2011-09-24T14:05:00.000-07:00

Lots and lots of work has been taking place on emscripten (the LLVM-to-JS compiler). I haven't been breaking things out into smaller releases, instead, there will be a 2.0 release in the near future (last release was 1.5). The remaining issues before that are:

Fix any remaining regressions in the llvm-svn branch compared to the master branch, and merge llvm-svn to mastert. The llvm-svn branch uses LLVM's svn, which will soon become 3.0. Some of the code changes in LLVM have hurt our generated code, however most of the issues are now fixed. The latest update is a fix for exception handling which has led to a 5% smaller ammo.js build (compared to 2.9), with no speed decrease :)
Bundle headers with emscripten. As more people have begun to use emscripten, we have been seeing more issues with platform-specific problems, almost all due to using different system headers (for example, issues #82 and #85 on github). Bundling working headers will fix that, in a similar way as to how SDKs and NDKs typically bundle a complete build environment. Currently the plan is to use the newlib headers for libc and start from there.

As mentioned in a previous blog post, emscripten 2.0 will require LLVM 3.0, and will no longer officially support the deprecated llvm-gcc compiler (it might still work though). This is a significant change which may affect projects using emscripten, please let me know if you are aware of any issues there.

Note also that I will be merging llvm-svn to master before LLVM 3.0 goes stable. That means that you will need to build LLVM from source at that time since there won't be LLVM binaries. Of course, you will still be able to use a previous revision of emscripten, that works with LLVM 2.9, with no issues.

LLVM 3.0, llvm-svn Branch

2011-09-09T19:42:00.000-07:00

LLVM 3.0 will probably be released in about a month. In preparation for that, I've gotten emscripten to work properly with LLVM svn in the llvm-svn branch.

As in the past, I intend to only support one version of LLVM at a time, since it takes too much effort to do any more - our automatic tests already require several hours, doubling that for another LLVM version is a huge burden.

With LLVM 3.0 it looks like llvm-gcc is pretty much obsoleted. It isn't being developed much, and remains on gcc 4.2 (I am guessing due to Apple's aversion to the GPL3?). There is Dragonegg, which is a plugin for recent GCC versions which uses LLVM as the backend, however as of 2.9 Dragonegg is not considered mature. I am not sure if 3.0 will be sufficiently stable or not.

So, the question is what compilers to use with LLVM 3.0. Clang goes without saying. The good news is that Clang can finally build all of the source code in the emscripten automatic tests which is very nice (although I did need to file a bug last week for libc++ - which is kind of ironic considering it's an LLVM project ;) but kudos to the LLVM people for the quick fix). As for other compilers, llvm-gcc seems of little importance if Clang can compile the same code, since llvm-gcc is deprecated. Dragonegg is interesting but not sure it makes sense to use it before it is fully ready - we might end up wasting a lot of time on bugs.

So my current plan is to move to a single compiler, Clang, in the emscripten test suite. That is what is currently done in the llvm-svn branch. The main risk here is of code that gcc can compile but Clang cannot. Is anyone aware of any significant cases of that? In particular I am curious about Python, I have not tried to compile it with Clang yet (the emscripten test suite has a prebuilt .ll file - we should fix that).

If no one raises any concerns about this plan, I'll merge the llvm-svn branch into master in the near future.

The VTable Customization Hack

2011-09-08T22:03:00.000-07:00

Recently my main focus in emscripten (the LLVM-to-JavaScript compiler) has been on the bindings generator: A tool to make it easy to use C++ code from within JavaScript. Why is this needed? Well, assume you have some C++ class,

class MyClass {
public:
MyClass();
virtual void doSomething();
};

The bindings generator will autogenerate bindings code so that you can do the following from JavaScript:

var inst = new MyClass;
inst.doSomething();

In other words, use that class from JavaScript almost as if it was a native JavaScript class.

Turns out that really doing this is not easy to do ;) One issue is callbacks from C++ into JavaScript: Imagine that you compiled some C++ library into JavaScript, and at some point the C++ code will expect to receive an object on which is a virtual function, which it will call. The virtual function is a common design pattern where you can basically get a callback to your own code. Typically you would create a new subclass, implement that virtual function, create an instance, and pass it to the library. That function will then be called when needed from the library.

Why is this difficult when mixing C++ and JavaScript? The main issue is that in C++ you would be creating those new classes and functions at compile time. But in JavaScript you are doing it at runtime. Creating a new class at runtime is not simple, but it was one option I considered. However compilation speed was too much of a concern. Instead, I went for a vtable customization approach.

The vtable of a class is a list of addresses to its virtual functions. Virtual functions at runtime work as follows: The code goes to the vtable, and to the proper index into it, loads the address, and calls that function. So by replacing the vtable you can change what gets called. However this still turned out to be fairly difficult. The reason is that the bindings code gets you into this situation:

// 1: Original C++ codevoid MyClass::doSomething();

// 2: Autogenerated C++ bindings code
void emscripten_bind_MyClass_doSomething(MyClass *self)
{ self->doSomething(); }

// C++/JS barrier

// 3: Autogenerated JS bindings code
MyClass.prototype.doSomething = function() {
_emscripten_bind_MyClass_doSomething(this.ptr);
};

// 4: Handwritten JS code
myClassInstance.doSomething();

The top layer is the original C++ code in the library you are compiling. Next is the generated C++ bindings code. This does almost nothing except for it being defined as "extern C", so that there is no C++ name mangling. Below that is the JS bindings code, which also seems fairly trivial here, but generally speaking it handles type conversions, object caching and a few other crucial things. Finally, at the bottom is the handwritten JS code you create yourself.

So, the idea of the vtable customization hack is to receive a concrete object, then copy and modify its vtable, replacing functions as desired. The replacements can be native, normal JS functions, and presto: Your C++ library is calling back into your handwritten JS code. However, how do you modify the vtable, exactly? When your handwritten code wants to modify it, what it specifies is code on the third level, something like this:

customizeVTable(myClassInstance, [{
original: MyClass.prototype.doSomething,
replacement: function() { print('hello world!') }
}]);

Here we want to replace doSomething with a custom JS function. But what appears in the vtable is not the third-layer function specified here. It isn't even the second-layer function! It's the first-layer one. How can you get to there, from here..?

A natural idea is to add something to the second layer,

// 2: Autogenerated C++ bindings code
void emscripten_bind_MyClass_doSomething(MyClass *self)
{ self->doSomething(); }
void *emscripten_bind_MyClass_doSomething_addr = &MyClass:doSomething;

- basically, have the address of the function in the bindings code. You can then read it at runtime and use that. But there are a few problems here. The first is that this code won't compile! The right-hand-side is a two-part pointer, consisting of a class and an representation of the function in the class. You can't convert that to void* (well, GCC will let you, but it won't work). Even if you do get around the compilation issue, though, you will be left with that representation of the function. I had hoped it was a simple offset into the vtable - but it isn't, at least not in Clang. After some mucking around with trying to figure out what in the world it was, I realized there was a better solution anyhow, because of the other reason that this approach is a bad idea: This approach forces you to add a lot of bindings code, a little for every single function. That's a lot of overhead, considering you will likely use that information for very few functions!

So instead, I arrived at the following hack:

Add a terminating 0 to all vtables at compile time. (This adds some overhead, but there is one vtable per class, and it's just one 32-bit value for each).
Copy the object's vtable.
Replace all the vtable elements with 'canary functions', that report back to you with their index in the vtable.
Call the function you want to replace, through the third-layer function you have available in JavaScript.
Since you replaced the entire vtable, you end up calling one of those. The canary function then reports back by setting a value. That value is the index of the function you want to replace in the vtable.
Copy the vtable again, this time the only modification is to replace the function at the index that you just found with the replacement function you want run instead.
(There are some additional complications, for example due to how emscripten handles C++ function pointers in JavaScript - pointers to functions are just integers, like all pointers, so there is a lookup table to map them to actual JS functions. Another issue is that the third-layer JS bindings code will try to convert types, and if you pass it the wrong things it will fail, so calling the canaries must be done very carefully. But the description above is the main idea.)

This ends up working properly. You can see the code in tools/bindings_generator.js (search for customizeVTable), and you can see it used in the latest version of ammo.js (the README there has been updated with documentation for it).

Rewritten Physics Engine Demo

2011-08-11T20:53:00.000-07:00

Initial testing of ammo.js (a port of Bullet Physics to JavaScript using Emscripten) found some issues, but they have been quickly resolved. ammo.js should be ready for use now.

Completing that allowed me to rewrite the original Emscripten Bullet demo using ammo.js. That is, the original demo code - creating the scene and so forth - was written in C++, and was compiled alongside Bullet into JavaScript for the original demo. What I did now was to write the scene generating code in JavaScript, where it uses Bullet through ammo.js's autogenerated bindings. Check out the demo here, and read the JavaScript embedded in the HTML file to see a complete example of using ammo.js.

I'm very happy with the result: The JavaScript code in the demo is very nice to work with now, and in addition it outperforms the original demo due to build system improvements that were completed since the original demo was finished.

ammo.js - Ready for Testing!

2011-08-08T18:59:00.000-07:00

After months of work, ammo.js (the Bullet physics engine compiled to JavaScript using Emscripten) is now ready for testing. To do that, grab builds/ammo.js, and look at examples/hello_world.js for a code example.

examples/hello_world.js is almost a 1 to 1 manual translation of HelloWorld.cpp (which you can find in bullet/Demos/HelloWorld/HelloWorld.cpp). That is possible since the ammo.js bindings let you write very natural JavaScript, for example this C++ code

btTransform groundTransform;
groundTransform.setIdentity();

becomes this JavaScript code

var groundTransform = new btTransform();
groundTransform.setIdentity();

There are some limitations to the automatically generated bindings code (see the ammo.js README for more details), but overall it basically works.

This is the result of a lot of hard work by bretthart on CppHeaderParser and by me on the bindings generator in Emscripten that uses it. Turns out it's pretty hard to automatically generate bindings from C++ to JavaScript, who would have thought ;) In any case we appear to have things in good shape now. I will probably write a separate blogpost about the bindings methodology later on.

The speed of ammo.js should be fairly decent, most optimizations are applied to the underlying Bullet code except for the LLVM ones and for typed arrays. However the binding code itself is not optimized at all yet. Overall things should be fast enough for testing, with some additional speedups still possible later on.

Please test ammo.js and file issues on github :)

Emscripten 1.5!

2011-07-31T11:15:00.000-07:00

Version 1.5 of Emscripten, the LLVM to JavaScript compiler, is out. Lots of new stuff:

A Text-to-Speech demo using eSpeak. Not much had to be done to get this to work, a few library functions were missing but that is pretty much it. I did need to bundle getopt and strtok C sources in the project though. Also, I had to use typed arrays type 2, since the eSpeak source code is not as platform independent as we would like (so this ended up being a good test of typed arrays 2 actually). For more details, source code etc., see the demo page.
max99x has written a nice Filesystem API. See that link for documentation. It makes the emulated filesystem much more flexible and useful. The text-to-speech demo uses it, as do all the automatic tests. Aside from the API itself, this update comes with a ton of library additions for IO related things.
max99x also wrote parsing code to detect field names in LLVM metadata. This lets you use the original C/C++ field names in your JavaScript, so integrating compiled code and JavaScript becomes much easier. I am thinking about extending this for use in the bindings generator.
Speaking of the bindings generator, it has seen a lot of work and things are finally starting to run with Bullet, at least a 'hello world' of creating a btVector3. There is still some work ahead before it is finished, not sure how much.

Emscripten 1.4!

2011-07-10T10:55:00.000-07:00

Version 1.4 of Emscripten, the open source LLVM to JavaScript compiler, has been released.

Some significant improvements this time, including

Support for compiling and loading dynamic libraries, thanks to max99x for writing this very useful (and not easy to write!) feature. You can now compile a module as a shared library, and load it from your main compiled script just like you would load a normal shared library in native code, using dlopen() and so forth. This can potentially be very useful, both in not needing to rewrite code that is already split up into modules, and also in that it lets you load the main module quickly since other stuff is split out into other files, which can be loaded later on demand. I hope to see a demo of this up soon.
Automatic bindings generation. Until now, you could compile a C or C++ library and run it on the web, but using it from normal JavaScript was clunky. Thankfully bretthart pointed me to CppHeaderParser, a pure Python C++ header parser, which Emscripten can now use to generate bindings (for more details on the header parser, see here). The result is a set of JavaScript objects that wrap the compiled C++ code, so you can write quite natural JavaScript code to access them, for example, var inst = new CppClass() to create an instance, inst.doSomething() to call a function, etc. A lot of basic stuff already works (see part 2 of test_scriptaclass), I am currently investigating the use of this with Bullet in ammo.js, hopefully I will succeed there and have a more detailed blogpost afterwards.
Library stuff, lots of fixes and additions there, thanks to max99x and timdawborn.

Emscripten 1.3!

2011-06-22T19:47:00.001-07:00

Version 1.3 of Emscripten, the open source LLVM to JavaScript compiler, has been released.

No new demo this time, sorry. However the Python demo has been updated to improve performance and enable raw_input to work (it prompts for input using window.prompt). Press 'execute' in the demo to see it work.

Main updates:

Support for a new usage of typed arrays, TA2. In TA2, a single shared buffer is used, and int8, int32 etc. are all accessed through views into that buffer. The main benefit here is memory usage - this mode takes much less memory than TA1 (the original typed array usage), and in most cases probably less than the non-typed array case.

In theory, this can also be faster. However that doesn't appear to be the case in my benchmarks, due to the need to divide pointers by 2 or 4 constantly (pointers are raw addresses, while indexes into typed arrays take into account the size of the element. int32vec[1] is at address 4!), and since JS engines still do not heavily optimize typed arrays.

One thing you can do, though, is use dangerous nonportable LLVM optimizations with TA2. TA2 lets you write an int and read the first character, and get the 'right' result. Of course 'right' will depend on the endianness, so this is very dangerous and not recommended. However you can compile two versions, one for each endianness. This can potentially be faster.
Some relooper optimizations were done, which gave us a nice speed improvement. I'll probably do a full blogpost on performance issues, but to briefly summarize, we seem to be getting close to the speed of handwritten JS code, which is to say, as fast as we can probably get. In absolute terms, compared to gcc -O3 (the fastest native code), we are around 5X slower (on the latest development versions of SpiderMonkey and V8). But there is a big spread: In raw numeric processing we are often just 2-3X slower, which is about the same as Scala, Haskell, and Mono, but certain other operations are costlier and in some benchmarks we are up to 10X slower.

Other news:

Still hoping for people to help out with OpenGL/WebGL stuff. Please step up! I don't know what to do there myself.
Next main project for me personally will probably be better tools to integrate compiled code with normal JS code. One option is to use SWIG to generate bindings. We could then compile a C++ library and use it in a natural way on the web, which would be very cool. If you know SWIG, or don't but want to see this happen (like me ;) then please get in touch.
I had some discussions with people interested in compiling certain large projects to the web, for example Second Life and Mono. Both have significant technical difficulties (rendering and networking for Second Life, the non-existence of an interpreter and the limitations of mono-llvm in Mono), but if the people interested in each are serious enough to do the work to overcome the respective difficulties, I have promised to do the raw C++ to JS conversion for each of those projects. Hopefully cool things will happen here.

Followup to Doom on the Web

2011-05-31T18:42:00.000-07:00

The Doom on the Web demo has been viewed over 35,000 times so far. Based on the responses, I'd like to clarify some things that I should have mentioned before (sorry for not doing so).

First off, I should have linked to the details page more clearly. It explains a bit about the demo, where it is currently known to work, etc.

Now, to the main issue: This demo is not a good benchmark of anything. The goal here was not to run a version of Doom with good performance, but to run DOOM itself, the original, with as few changes as possible, on the web. I made no effort to optimize the code, which was written and heavily optimized for a completely different architecture (it uses fixed-point arithmetic! :) I wasn't sure if it would be playable at all.

That is, the point of the demo is, "Doom can be run, with hardly any modifications, on the web." Showing that sounded like a cool thing to do, so I spent several evenings and a weekend or two on it.

As for frame rates, Doom caps them at 35 - you won't see it get any better than that, simply because of how the main loop works (and as mentioned before, I didn't try to improve it). It will also max out your CPU, even if it doesn't need to, for the same reasons. It might be possible to optimize this with some modifications to the Doom source code, but I didn't look into that. So, if you are seeing 35fps and 100% CPU, that doesn't mean your machine is actually working hard to generate it.

For example, I get close to 35fps on a slow 1.2GHz laptop. A modern machine would probably be able to get over 100fps with a proper main loop. And again, even this is not a fair measure of how fast Doom could be, if it were actually optimized for JavaScript. So please don't run this demo, be disappointed by the speed, and say "JavaScript is too slow, we need Flash/NaCl/Java/native apps" etc. The demo can't be used to conclude anything like that. Valid benchmarks (which this demo is not) show that JavaScript is quite fast, and getting faster more quickly than any other language - something that shouldn't be surprising, given that it probably has the most developer effort put into it, these days.

I hope the above explains what frame rates in the demo actually mean (that is, almost nothing). Now, aside from that, some people said the demo was very slow for them. I suspect that depends on the browser, as both Firefox and Safari play it very speedily even on older machines (as I mentioned earlier, my old laptop at 1.2GHz runs it well on FF7). On the other hand, Opera runs it slowly, while Chrome is unplayable (I reported the issue to them, and am doing my best to help figure it out). So, performance depends on the browser. That's disappointing, clearly, but that is another point of this demo - to push the limits, and hopefully to motivate JavaScript engine devs to fix whatever bugs are in the way, and be even faster and more awesome.

P.S. I apologize for the quality of sound in the demo. I never used any audio generating API before, and I still don't know what the numbers I am passing from Doom to the Audio Data API actually mean ;) I basically just hacked together something quickly, got it to the point it is in the demo, and stopped there. Someone that knows this stuff could probably make it sound right.

ammo.js: Bullet on the Web

2011-05-30T11:10:00.000-07:00

We already had a demo of the Bullet physics engine in JavaScript a while ago. People asked for an easier way to use it, so I set up ammo.js, a separate project to port Bullet to the web.

The starting point is Bullet compiled to JavaScript using Emscripten, and the main challenge is to make a friendly API for JavaScript applications to use. See Issue #1 in that github repo, we are currently looking for ideas and help in doing this. Discussion also takes place on Emscripten's IRC channel (#emscripten on irc.mozilla.org).

Emscripten 1.2, Doom on the Web

2011-05-30T10:19:00.000-07:00

Emscripten, the LLVM to JavaScript compiler, is now at version 1.2. The main updates in this release were to enable this demo of Doom on the Web - a playable version of the classic game Doom, compiled from C to JavaScript and rendering using Canvas.

The demo is known to work on Firefox and Safari. It works, but slowly, on Opera. I can't get it to run properly in Chrome due to a problem with V8. I have no idea if it runs on IE9, since I don't have a Windows machine, but since IE9 has a fast JS engine and supports canvas, it should (please let me know if you try it there). Edit: Here's a screencast of the demo running on Firefox Nightly if you can't run it yourself.

Highlights of Emscripten 1.2:

Many improvements to Emscripten's implementation of the SDL API in JavaScript, including support for color palettes (Doom uses a 256-color palette), input events (we translate normal web keyboard events into their SDL forms), and audio (for now, just using the Mozilla Audio Data API - it's the most straightforward API at this point. Patches are welcome for other ones).
Many improvements to the CHECK_* and CORRECT_* options, which are very important for generating optimized code using Emscripten. In particular, there is a new AUTO_OPTIMIZE option which will output a summary of which checks ran how any times, and how many of those checks failed, giving you a picture of which lines are important to be optimized, and which can be.
Some additional experimental work is ongoing about supporting OpenGL in WebGL. I don't know either OpenGL or WebGL very well, I'm learning as I go, and I'm not sure how feasible this project is. If you can help here, please do!
Various bug fixes. Thanks to all the people that submitted bug reports. In addition compiling Doom uncovered a few small bugs, for example we were not doing bit shifts on 64-bit integers properly.

Emscripten 1.1!

2011-05-01T18:27:00.000-07:00

Emscripten is an LLVM to JavaScript compiler, allowing you to run code written in C or C++ on the web. I released version 1.1 today, with the following updates:

A much improved Bullet demo - check it out! This version is much faster. The main differences are use of memory compression (see below), LLVM optimizations, and CubicVR.js for rendering.
QUANTUM_SIZE == 1, a.k.a memory compression. This is an advanced, and somewhat risky, optimization technique. I see speedups of around 25%, but take note, this must be used carefully. See the docs.
Dead function elimination tool: A Python script that scrubs an .ll file to remove unneeded functions. This is useful to reduce the size of the generated code and speed up compilation. Note though that if you want to compile a library, then this tool will remove functions that you probably want left in - it removes everything that cannot be reached by main(). The test runner now uses this by default.
Various performance improvements and bug fixes.

Emscripten Moves to GitHub

2011-04-10T15:28:00.000-07:00

After starting on Google Code, and later adding a git mirror, Emscripten has now moved entirely to GitHub. emscripten.org and so forth should now forward to there.

The main reason for the change is the inconvenience in maintaining two clones. hg-git helped greatly, but it remained a constant hassle. Meanwhile GitHub has been getting more popular and more useful. The last straw was GitHub's recent addition of a much nicer issue tracker. So I decided to make the move today, which coincides nicely with the release of Emscripten 1.0: A fresh start on the road ahead to 2.0.

Code will not be updated in the Google Code page anymore, and I pushed a final commit there to warn people they are running old code if they get there by mistake. I moved the important wiki pages to GitHub, which leaves only one thing left behind, the open issues. If you have an open issue on Google Code that you care about, please either open a new issue on GitHub for it, or tell me and I'll do it for you.

Emscripten 1.0!

2011-04-09T13:34:00.000-07:00

It's been almost a year since I started Emscripten (which, if you haven't heard of it, is a tool to compile LLVM to JavaScript), during which it took up much of my spare time. So I am very pleased to announce that today Emscripten has reached the 1.0 milestone. This release comes with a demo of rendering PDFs on the web (warning: that page downloads >12MB, since it includes Poppler and FreeType. It's like downloading an entire desktop app, almost).

Other highlights in this release:

Very significant optimization of memory use in the compiler. This was necessary for the PDF demo to build, since it is far larger than previous demos.
Full support for the recently released LLVM 2.9.
The Emscripten documentation paper is finished. It explains how Emscripten works, so you might be interested in it if you care what Emscripten does under the hood (but if you just want to use Emscripten you don't need to read it).

Overall Emscripten is now in very good shape. It can probably compile most any C/C++ project out there, subject to some limitations (like JS not allowing C-style multithreading). At times some manual intervention is needed, like changing the project's settings so it doesn't generate inline assembly, and of course bugs probably still exist, but recently the code I have compiled has tended to just work (hence the rate of commits has greatly decreased recently).

The speed of the generated code can be quite good. By default Emscripten compiles with very conservative settings, so the code will be slow, but optimizing the code is not that hard to do. Optimized code tends to run around 10x slower than gcc -O3, which is obviously not great, but on the other hand fairly decent and more than good enough for many purposes. And of course, that ratio will improve along with advancements in JavaScript engines, LLVM, and the Closure Compiler.

So, Emscripten 1.0 is in my opinion pretty solid. There are no major outstanding bugs, and no major missing features. (But I do have plans for some major improvements, which are difficult, but should end up with code that runs at least twice as fast.) Now that Emscripten is at 1.0, I am hoping to see it used in more places. I'm starting to propose at Mozilla that we use it in various ways, and also I'd love to see things like GTK or Qt ported to the web - if anyone wants to collaborate on that, let me know.

Emscripten moving to LLVM 2.9

2011-03-19T10:02:00.000-07:00

LLVM 2.9 will be released very soon, and Emscripten has just been updated to support it.

Emscripten has a lot of automatic tests - they take over 2 hours to run on my laptop - so I won't be running tests for LLVM 2.8 anymore (that would double the time the tests take). Until LLVM 2.9 is formally released with binary builds, you can build LLVM from svn source (the instructions on the Emscripten wiki are useful), or use LLVM 2.8 with Emscripten 0.9 (the last release of Emscripten that supports 2.8).

If you do build LLVM 2.9 and put it in a different location than 2.8 was, don't forget to update your ~/.emscripten file so it uses the version you want. Also, if you update LLVM to 2.9 and want to use llvm-gcc, you need to update that to their current svn as well.

There were not a lot of changes for Emscripten to support 2.9, so it is possible 2.8 will still work. But as mentioned above, I am not testing it, so I can't say for sure.

Puzzles on the Web

2011-03-06T11:20:00.000-08:00

Check out this very cool port of Simon Tatham's Portable Puzzle Collection to the web, by Jacques Le Roux, using Emscripten.

Nice quote from there:

This was basically just an experiment to see how hard it would be to port C code to a web application running entirely on the client (turns out not that hard).

Emscripten 0.9!

2011-03-05T19:36:00.000-08:00

The demo this time is OpenJPEG: JPEG 2000 decoding in JavaScript.

Aside from OpenJPEG, lots of stuff in this release, including

Line number debugging: Emscripten can optionally add the original source file and line to the generated JavaScript (if you compiled the source using '-g'). Useful for debugging when things go wrong, especially with the new autodebugger tool, which rewrites an LLVM bitcode file to add printouts of every store to memory. Figuring out why generated code doesn't work is then as simple as running that same code in lli (the LLVM interpreter) and in JavaScript, and diff'ing the output, then seeing which original source code line is responsible.
Line-specific CORRECT'ing: The main speed issue with Emscripten is that JavaScript and C have different semantics. For example, -5/2 in C is -2, while +5/2 is +2, whereas in JavaScript, naive division gives floating point numbers, but worse, there is no single operator that will create the same behavior as C (Math.floor on -5/2 gives -3, and Math.ceil on +5/2 gives +3). So in this example (unless we have a trick we can use, like |0 if the value is 32-bit and signed), we must check the sign of the value, and round accordingly - and that is slow. Similar things happen not just in rounding, but also with signedness and numerical overflows, and therefore Emscripten has the CORRECT_SIGNS, CORRECT_OVERFLOWS and CORRECT_ROUNDINGS options.

With line-specific correcting in the 0.9 release, you can find out which lines actually run into such problems, and tell Emscripten to generate the 100% correct code only in them. Most of the time, the slow and correct code isn't needed, so this option is very useful. I will write a wiki page soon to give more examples of how to use it to optimize the generated code (meanwhile, check out the linespecific test).
20% faster compilation, mainly from optimizing the analyzer pass.
Strict mode JavaScript. The compiler will now generate strict mode JavaScript, which is simpler, less bug-prone, and in the future will allow JS engines to run it more quickly.

Synchronizing a git mirror with hg-git

2011-02-12T17:02:00.001-08:00

Posting this since it might save someone else the time it took me to figure out.

There are several tutorials for creating a git repo from an hg one using hg-git. But none of them go much into how to keep the git repo up to date when the hg one changes (assuming work is done in the hg repo). And, just doing a push with hg-git to github fails... silently :( (But it succeeded the first time, to create the repo.) Even adding -v --debug in hopes of getting some useful information is no help.

The solution is to push to a local git repo first, then push from there to github. But even pushing to a local git repo won't just work - you will get

error: refusing to update checked out branch: refs/heads/master
error: By default, updating the current branch in a non-bare repository
error: is denied, because it will make the index and work tree inconsistent
error: with what you pushed,

To get around that, create another branch in the local git repo (git branch side), switch to it (git checkout side), then push to it using hg-git. Then switch back to master (git checkout master) and finally push that to github.

I think it might also work to make the local git repo into a bare repo, but the above worked for me so I stopped there.

Git Mirror for Emscripten

2011-02-11T17:50:00.000-08:00

We now have a git mirror on GitHub! Getting the Emscripten code is now as easy as

git clone git://github.com/kripken/emscripten.git

Code will be mirrored there, so you can either use git with that GitHub repo, or hg with the Google Code repo, and the result will be the same. (However for project stuff - issues, wiki, etc. - we will continue to use the Google Code project page.)

FreeType Demo

2011-02-08T19:17:00.000-08:00

A simple demo of FreeType in JavaScript can be seen here.

Emscripten 0.8!

2011-02-06T20:05:00.000-08:00

The main highlights of this release are:

Tests for FreeType and zlib, two important real-world codebases. Aside from all the fixes and improvements necessary to get them to work, the test infrastructure now runs the entire build procedure (using emmaken) for those two tests, giving even more complete test coverage.
File emulation. Just enough to let compiled C/C++ code think it is accessing a filesystem. For example, the FreeType test loads a TrueType font from a file (but really it's a virtual filesystem, set up in JavaScript).
Additional compilation options for overflows and signedness (CHECK_OVERFLOWS, CORRECT_OVERFLOWS, CHECK_SIGNS). These allow even more C/C++ code to be compiled and run properly, but are switchable, so code that doesn't need them can run fast.

I'll post a web demo soon.

Emscripten Overview Writeup

2011-01-20T20:30:00.001-08:00

If you're interested in how Emscripten works, then this writeup I am working on may interest you. It is currently the best explanation of the underlying techniques Emscripten uses to compile LLVM to JavaScript, including the memory model, the Relooper algorithm, etc.

It isn't meant to be a manual or a practical guide. For that, as always see the wiki.

LIL browser demo

2011-01-18T17:48:00.000-08:00

I just saw this demo of LIL (a Little Interpreted Language) running in the browser, compiled from C to JavaScript using Emscripten. Cool stuff!

A Completely Speculative History & Future of H.264 and WebM

2011-01-15T11:17:00.000-08:00

Several years ago, Google decides something needs to be done about web video, because (1) H.264 requires royalties, which means that parts of the web are proprietary (even if it is a standard), and Google believes an open web is in its best interest, and (2) for similar reasons, H.264 is incompatible with the W3C, Mozilla's Firefox, and Opera, so it will never become universal anyhow. Google is willing to go to great lengths to solve this issue, including large sums of money and developer time.
Google approaches MPEG-LA (or major entities that are members), and quietly floats the idea of 'freeing' H.264, by way of a large one-time payment from Google, after which H.264 will be royalty-free, and can then be blessed by the W3C.
Negotiations fail. Google offers large amounts of cash, but it isn't enough for MPEG-LA, which believes it is close to having a complete lock on the market, which it can leverage for even more cash later on.
Google threatens to support a competing format with all its resources, thereby threatening the future profitability of H.264.
MPEG-LA decides to call Google's bluff.
Google makes good on its threat, buying On2 and freeing its VP8 video codec as part of WebM, a royalty-free format for web video. (Note: I'll use 'WebM' to refer to 'VP8', a lot of the time.)
Mozilla, Opera, etc. naturally support this move, as it is good for the open web. Apple and Microsoft, whose motivations are otherwise, do not support this move - they are both already heavily invested in H.264, and for them life would be simplest if WebM never existed.
As a reaction to WebM, MPEG-LA makes H.264's licensing less expensive, and for a longer period of time.
Google makes good on another part of its threat to MPEG-LA, removing H.264 support from Chrome. MPEG-LA is surprised Google is willing to hobble its own browser in order to get a leg up in this fight.

(This brings us to the present time.)
Nothing much changes, at first. Most web video is seen through Flash anyhow. However, the block of WebM supporters, which is now Firefox, Chrome and Opera - whose share in the market is large, and growing - gets video providers on the web to pay close attention to WebM.
Flash introduces WebM support. Most video encoded for desktop viewing can now be encoded in WebM, and viewable through Flash or an HTML5 video element in Firefox, Chrome and Opera. Even video shown with DRM can be encoded in WebM, but must be shown in Flash. On the other hand, in the mobile space, a complete stack of hardware&software support is still really just present for H.264, and Apple doesn't support anything else, so video encoded for mobile viewing is primarily done in H.264.
WebM's video quality improves, in part benefiting from the fact that while open source and royalty-free, WebM is not a formal specification or standard, so rapid development and changes are possible. WebM becomes equivalent or superior to H.264.
Google switches YouTube to primarily use WebM for encoding video meant for desktop use. There is hardly any impact on users, due to most video being shown in Flash anyhow (which now supports WebM). However, video for mobile viewing remains encoded in H.264.
Hardware support for WebM begins to ship in a great deal of new mobile devices, and eventually in a majority of new mobile devices.
As a reaction to WebM's rise, MPEG-LA once more lessens the royalties for H.264, in an attempt to make it more competitive.
A new version of Google's Android ships, on a new flagship phone from Google, that has complete hardware and software support for WebM. The device primarily views YouTube video in WebM format.
Google, stating WebM's superior quality, begins to 'favor' WebM over H.264 on YouTube, for mobile content. More specifically, while both WebM and H.264 are supported, WebM content is encoded at higher quality levels (this is accomplished not by decreasing H.264 quality, but by adding a higher level of quality exclusively for WebM). The result is that mobile devices viewing YouTube give a better user experience if the device does so using WebM.
Apple makes the rational decision and supports WebM on new iOS devices - hardware support is already there, and Apple cannot compromise on user experience. Whatever monetary benefit Apple gains from MPEG-LA from H.264 is completely eclipsed by Apple's iOS business, so this is a no-brainer.
With the majority of new mobile devices shipping with WebM support (Android and iOS), smaller players (Blackberry, WebOS, Windows Phone) are forced to support it as well.
The online video market has its anti-DRM moment, just like online audio already had. Video is shown without DRM, which simplifies delivery and cuts costs, and piracy remains at the same levels as before (just as with audio).
Once it is clear H.264 has lost in the mobile space, and that DRM is no longer needed, there is no reason for Microsoft and Apple not to support WebM in the HTML5 video element, on Windows and OS X respectively, in order to ensure their users the best experience.
With DRM no longer an issue and widespread support for WebM in the HTML5 video element, WebM becomes the universal standard for video on the web, on both desktop and mobile. Content producers have little reason to even support a fallback to Flash - some do, but many do not, at little detriment to them or their users.
Google wins the fight, and the open web greatly benefits.

Some things that can change this future history:

MPEG-LA deciding to make H.264 100% royalty-free. This will kill MPEG-LA's profits, but may still be worthwhile for MPEG-LA members, since if done properly - and promptly - it can ensure H.264 becomes the standard for web video. Whether there remains enough profit from H.264 (from hardware, services, etc.) for this move to make sense, is not clear. But if this does happen, WebM loses, but really Google wins, since it got what it set out to get.
A new video format can appear, or a newer version of an existing format, which requires new hardware support but has benefits to justify the switch. Given the battle between H.264 and WebM, I would expect the new format's backers to learn the lessons of the past and make it free on the web (or, if they are not willing to do that - then to not even bother to create a new format). If such a new format appears, and becomes the universal standard for web video, the result is that Google wins in this case as well.
The fight gets taken to the courts. I doubt a simple injunction will be granted to either side, as both are powerful, influential, and have many patents to back up their claims - so there is no quick victory. Instead there is a lengthy court battle. To justify the cost, there must be a significant chance of large future profits, and if H.264 is already in decline, that might not be the case. However, it might still make sense for MPEG-LA to take Google to court, just to get it to settle for some amount of money, in which case Google wins overall, but MPEG-LA gets a little more money than otherwise. However, if the goal isn't a settlement, but an actual attempt to kill WebM, then things can get interesting. I don't think anyone can say for sure how that fight would turn out - does WebM infringe on H.264 patents? Does H.264 infringe on WebM patents (VP8 patents, granted to On2, and now owned by Google, which would countersue)? Perhaps both? Such a 'fight to the death' in the courts seems unlikely, in part due to that unpredictability, so all we can say for sure in this case is that several law firms will greatly benefit.

DISCLAIMER: I have no inside knowledge about any of this.

Emscripten Usage Change

2011-01-14T22:49:00.000-08:00

I refactored the python scripts in Emscripten to make them more sane. There is one difference in how Emscripten is used: If ~/.emscripten does not exist, it will copy tests/settings.py to ~/.emscripten, at which point you would edit the paths etc. in ~/.emscripten - not in tests/settings.py, which is how things were before (and that was bad).

If you already have a file at ~/.emscripten, and you probably do if you already ran Emscripten in the past, then that file will not contain all the necessary information. The simplest thing is for you to copy your edited tests/settings.py into ~/.emscripten. Or, you can delete ~/.emscripten and run the tests (python tests/runner.py), which will copy tests/settings.py for you into ~/.emscripten (but remember to change the paths, if you need to).

Sorry for the inconvenience, but the previous way things were was just a hack, which had to be fixed.

Wiki pages on the project site have been updated.

JavaScript, Native Client, and Emscripten

2011-01-01T18:59:00.000-08:00

This is a response to this blog post, which is titled "Mozilla’s Rejection of NativeClient Hurts the Open Web", which appeared prominently on Hacker News.

I disagree with the thesis in that blog post. My reasons are entirely technical.

First off, NaCl is not portable yet. PNaCl is working towards that, but it will take time. Until it is portable, comparing it to JavaScript is comparing apples and oranges. They simply do very different things - one is fast, the other is portable.

Second, while PNaCl is being worked on, at the same time a lot of effort is being put into making JavaScript engines faster. Now, for purposes of comparison to PNaCl, we don't need all JavaScript code to run as fast as native code. For our purposes here, we can care only about 'implicitly statically typed' JavaScript code - the code that can, in theory, be compiled so it runs as fast as native code. Implicitly statically typed code is code that uses a single type for each variable, and even though it's written in a dynamic language, could correspond almost 1-to-1 to code in a fast statically typed language like C.

Such code can be created automatically from C or C++ using something like Emscripten. Or, you can write such code on purpose, for example, PyPy is written in RPython, which is basically implicitly statically typed Python. More generally, you might write the performance-sensitive parts of a JavaScript application in an implicitly statically typed manner, while the rest can have fun with dynamic typing and the benefits that gives.

The important issue is that JavaScript engines can optimize implicitly statically typed code very well, both theoretically and practically. Techniques like tracing, type analysis, etc. are already being used in JavaScript engines like SpiderMonkey and V8, and progress is very fast. Also, PyPy can give us an idea of the long-term potential in such an approach.

So, while NaCl is working towards portability, JavaScript is working on speed. To clarify, again, I'm not talking about running all JavaScript code at native speed. But there is a big, relevant subset which can be run very quickly. Once JavaScript engines achieve the goal of running that code at native speed, then the performance advantage of NaCl will have vanished. At that point both NaCl and JavaScript will be fast (and, if PNaCl is completed, they will both also be portable).

Once we get there, I believe JavaScript will be preferable for the simple fact that it has a natural fallback built in - implicitly statically typed JavaScript is perfectly valid JavaScript, so even if your JS engine doesn't achieve the full speed of native code, at least it can run it. Whereas NaCl will simply not run at all unless the NaCl plugin is installed (and that may never happen on iOS devices, and may never happen by default on any desktop browser but Chrome).

Note that one might devise a fallback for NaCl by writing an emulator, or even a compiler, for PNaCl in JavaScript - perhaps using Emscripten (which does exactly what is needed - convert LLVM into JavaScript). If the speed-intensive parts of the code in that approach are implicitly statically typed, then we have come full circle, and the two approaches of JavaScript+Emscripten or PNaCl+Emscripten essentially converge, with a minor disadvantage to NaCl for being more complex (a special NaCl compiler, and a special NaCl runtime).

Is the Web Ready for Compiled Code?

2010-12-23T23:31:00.001-08:00

CPython compiled to JavaScript is now reasonably fast on Firefox 4 and Chrome. Startup takes a few seconds, but it could be done in a web worker, so as not to stall a loading page. Executing code afterwards is far slower than native CPython, but already fast enough to be useful for some things, and getting faster all the time. (There are other limitations, like importing of modules, but I'm mainly focusing on speed for now.) So, from the perspective of those two browsers, we might be able to use Python on the web.

However, for it to make sense in general, it needs to work well on all or almost all browsers people use. So, what's the situation with other browsers?

The Python demo tends to run very slowly in older browsers, that don't have a modern JavaScript engine. But, those people will upgrade eventually, so this will be solved (except perhaps for a minority using IE6...).
Currently Python will not run on Opera, due to a bug in their JS engine. They have been notified of the problem. As Opera has a quite fast JS engine, I am hopeful that compiled code will work there at some point.
IE9 has a fast JS engine as well, but I don't have a Windows machine to test on. Has anyone tried to run the Python demo there? Update: Issue 22.
Safari has had a fast JS engine for a while (these days perhaps not as fast as the other open source ones, given recent speedups in Firefox and Chrome, but still quite good). As in the previous point, I lack a machine to test on - has anyone tried there? Update: Works in Safari and even iPads.

Emscripten 0.7!

2010-12-18T17:38:00.000-08:00

Main changes in this release:

Lots of minor fixes and additions, in order to get CPython working. As a result there is now a web demo of Python, which seems to work quite well aside for being very slow in Chrome.
Figuring out what to do with LLVM optimizations. It looks like all of them generate suitable code except for -instcombine, which apparently combines instructions in a CPU-specific way (so, it isn't portable, and confuses Emscripten). All the tests now pass with LLVM optimizations enabled (all but the problematic one just mentioned).

So, not much in the way of new features: As mentioned before, we are already pretty much feature complete at this point.

Python Demo

2010-12-12T22:55:00.000-08:00

With help from rasjidw, there is now a web demo of Python. Check it out, and let me know if you find a bug!

The demo is of CPython, the original/standard Python implementation, compiled from C to JavaScript. What's cool about compiling CPython itself is that we get all the language features 'for free', both the common stuff you'd expect and think of first, and also things that if you were writing a new implementation, you might leave for last.

Known limitations:

No import of modules - just the core language works. It should be possible to get imports working, of both C modules and Python ones, but it would require writing code for dynamic linking etc.
I tested in Firefox 4 beta and Chrome 8, and for some reason the demo runs very slowly in Chrome 8 (perhaps related to V8 Issue 947?). Firefox 4 runs much faster but still obviously slower than native CPython. In any case that should improve once we get LLVM optimizations working.

A Ridiculously Recursive Bug

2010-12-11T13:17:00.000-08:00

Work has been going on to get Emscripten to compile CPython (the standard implementation of Python) into JavaScript. It's very close to working, so I started to write an automatic test. But the test failed even though running the test manually worked.

That's weird, I thought.

Turns out that something quite amusing was happening. The automatic test system and other scripts are written in Python, and the test runner checks for script crashes and failures by scanning for the text that appears in a Python stack trace, which is to say something like 'Traceback (most recent call last)' etc.

Now, when Emscripten compiles Python itself, it generates JavaScript code that includes, unsurprisingly, all the constant strings in Python - including that very string! So the right output was erroneously detected as a failure.

I guess that's what can happen if your test compiles the very same runtime in which the test infrastructure is built... ;)

Lua Demo += Skywriter

2010-11-28T18:47:00.000-08:00

The Lua demo now has a nicer interface using Skywriter, and some additional bugfixes under the hood.

Check it out, and let me know if you find any problems!

Lua Improvements

2010-11-26T17:20:00.000-08:00

It seems there are people interested in running Lua on the web :) The Lua demo was on Hacker News, and some people noticed that certain things don't work - thanks for the feedback! - so I spent a few hours figuring out what was wrong. After some small fixes, the demo will now handle these statements:

for k,v in pairs({'foo','bar'}) do print(k,v) end
for x = 1,10 do print(x) end
for k,v in pairs(_G) do print("->", k,v) end

which people mentioned did not work before.

The output seems ok, but I'm not totally sure that it's 100% correct, since my Lua skills are pretty rusty. That's also why I didn't notice the issues in the beginning - I saw that 'hello world' worked, and thought all was well ;)

If people really want to see this work properly, let me know about any other problems. I'm happy to work on it, I just can't do it without feedback since I don't know where to start testing. (Is there a standard test suite for Lua?)

The issues that had to be fixed in Emscripten were

Lua uses varargs in ways that the existing code didn't support. So I improved that. Varargs are now emulated in a way that is basically identical to C.
A few missing stdlib functions.

So, small stuff. In fact I'd be surprised if there were any big issues left, at this point, given the amount of code known to compile properly, but there are probably a lot of small things left to fix.

Emscripten 0.6!

2010-11-25T17:09:00.000-08:00

Demo: Lua. Thanks to rasjidw and puffnfresh!

Other major changes since the last release:

SAFE_HEAP checks for invalid reads/writes and nonportable LLVM bitcode. This is important as LLVM optimizations can lead to nonportable bitcode. We need to figure out which optimizations are at fault, so we can use the other optimizations - they should make us much faster.
Much more optimized compilation of very large projects - both in terms of memory and speed. I am now able to compile Sauerbraten without the JS engine giving up ;) - still a very long way from getting it to run, though.

See the Changelog file for additional updates.

Worth mentioning...

2010-11-16T20:24:00.000-08:00

First committed patch by someone other than me! Thanks puffnfresh :)

Now What?

2010-11-12T19:54:00.001-08:00

For several months I've known that when I get home at night, I have tasks X and Y to do in Emscripten in order to move it forward. Last night, however, I took out the guitar instead. Suddenly, there is not much to do - basically the goals of the project have been achieved, Emscripten can compile things like the Bullet physics engine and run it on the web. As far as the core code-generating capabilities are concerned, Emscripten is pretty much complete.

So, what now?

There are some additional optimizations and enhancements that can be done, like nativizing structures or emulating multithreaded code.
There are various tooling improvements that can be done, like making it easier to glue together web code and compiled code.
Emscripten could be used in other ways, for example, it could be combined with something like Rubinius that generates LLVM code from Ruby, allowing running Ruby code on the web.
Various code cleanups and refactorings could be done.

I'm not in a rush to do any of these - none is urgent or essential. I guess I'll get around to them eventually, or perhaps someone else will.

Right now, I'm considering doing one or both of the following:

Return to my original goal, that of bringing Syntensity to the web. In other words, to compile a version of Syntensity using Emscripten. The time of 3D-environments-on-the-web is almost upon us, and when it is, we need to make sure that the main tools for it are open source and platform agnostic. Sadly, currently the main contenders are not such.
Some other side project, got at least two ideas in my head of things I'd like to hack up. They are very experimental and speculative though, so they may end up a waste of time. But if they succeed...

Bullet/WebGL Demo

2010-11-09T20:46:00.000-08:00

Click on the screenshot for a live demo.

Emscripten 0.5!

2010-11-01T19:44:00.000-07:00

Another Emscripten release has arrived, this time the focus was on building a large real-world library: Bullet Physics. For those unfamiliar with it, Bullet is an awesome physics library written in C++, which has been used in many places, including commercial games and movies.

Here is the demo for this release:

----------------------------------------------------------------------
Ran 168 tests in 1060.329s

OK

:P Just kidding, a real demo will follow. I'm working on something with Bullet and WebGL, but didn't want to delay the release for it.

About the work in this release: Bullet is much larger than other projects Emscripten was tested on before, and a large part of the work was to speed up the compiler. The result is that Emscripten now takes a reasonable amount of time - if it takes 10 minutes to build the Bullet C++ sources into LLVM bitcode, then it takes a similar amount of time to convert that into JavaScript using Emscripten (and it takes an even longer amount of time to run that through the Closure Compiler for additional optimization). So at this point there isn't much more reason to optimize for speed of compilation - it's in very good shape, fitting into the build process without slowing things down noticeably.

As expected, various small bugs were found while building Bullet, and a few additional tests were added to check for specific issues. There is also an automatic test for Bullet as a whole.

Overall, at this point Emscripten is capable of compiling large projects effectively; quite a lot of the original goals of this project have been achieved. Still lots of cool stuff left, though! More about plans for the near future in another blog post very soon.

Emscripten 0.4!

2010-10-17T10:50:00.000-07:00

The focus of this release was on making the generated code faster. As the chart shows, we went from being 100X slower than hand-optimized JavaScript code, to around 5X slower. You can see the difference in the raytrace demo, which has been updated to use all the current optimizations.

5X slower is still slower. It will be hard to do much better, though, without either better JS engine support, or much more clever code analysis. Both will hopefully happen over time. Meanwhile, 5X slower is not too terrible, and there are some advantages over hand-written code - we hardly use garbage collection, so no GC pauses. Also, the speed really depends on the code - the comparison in the chart above uses benchmarks for which we have comparable code in both C++ and JavaScript. But the most interesting uses of Emscripten are to convert code for which we don't have a JavaScript equivalent. Also worth mentioning is that it is perfectly possible to hand-optimize the crucial parts of the code that Emscripten generates.

Some technical details about the optimizations implemented in this release:

Use typed arrays, if available in the JS engine (thanks to pcwalton and njn for the idea)
Optimize after the relooper runs, removing unneeded code flow overhead
Nativize many more variables than before (i.e., move them off the emulated stack, and into native JS variables)
Optimized stack emulation
Inlining of various runtime code fragments
Integration with the Closure Compiler: We generate output that it is very good at optimizing (thanks to Anders Riggelsen for the idea)

Also added in this release is support for the brand-new LLVM 2.8. That is now the version being tested against.

Emscripten 0.3!

2010-10-05T20:07:00.000-07:00

Demo for this release: Raytracing. It isn't very fast, since the focus hasn't been on code speed yet, but it does show that a ray tracer written in C++, using SDL, can be emscriptened and run on the web.

So, I ended up doing more in this release than I had intended, causing it to take longer than planned. But it was for the best. Major changes include:

Clang support: All tests now work in both llvm-gcc and Clang. The two produce somewhat different llvm bitcode, to the degree that different methods are needed with Clang, causing it to run 1/2 as fast as llvm-gcc code. That is mainly because llvm-gcc is more explicit with what it does, while Clang uses memcpy and such, with hardcoded C size values (4 bytes for an int, etc.).

Emscripten therefore now supports optional 'C memory layout' (QUANTUM_SIZE in settings.js). For example, an array of ints of values 1,2,3 with that enabled is [1,0,0,0,2,0,0,0,3,0,0,0] (since each int is 4 bytes), and when it is disabled, [1,2,3]. The latter works fine with llvm-gcc-generated llvm bitcode. Note that things get even more complicated with structures here, which need to be aligned and so forth. Anyhow, after that effort Emscripten should now be able to support anything that C/C++ can throw at it.
Faster compilation speed: The original goal of the release. Compilation speed is 2-3 times faster now. Still lots of room for improvement, but it isn't a major nuisance like it was.
Proper memory management: A call stack is implemented, and static memory allocation (for global variables, etc.) is also possible. sbrk() is emulated as well, allowing dlmalloc, a popular malloc() implementation, to be emscriptened properly. In particular that lets you use a real malloc() in your emscriptened code.
Much better native flow regeneration (the 'relooper'): A major challenge of translating LLVM to JavaScript is to implement native flow structures - if, while, for, etc. LLVM bitcode only provides chunks of code (I call them 'labels', but that's not the right name) and branchings between then. So Emscripten needs to figure out from that low-level data the high-level code flow patterns. Native flow structures are extremely important for good performance of the generated code.

The first relooper worked on most tests, but was slow and buggy. I wrote a new version almost from scratch, and it now properly processes all the test code. It isn't very fast, though, I didn't focus on that. For that reason it is off by default, which means that Emscripten will not generate native flow structures (instead it will emulate code flow using a switch in a loop, which is very slow - but trivial to generate).
The above-mentioned raytracing demo: For this, initial work was done on supporting SDL - just showing video data so far. The SDL Surface is implemented in JavaScript using a Canvas. I found this very amusing, to write C++ code with SDL, compile and run it natively using gcc, and be able to run that same unmodified code on the web through Emscripten ;)
Lots more tests: There are now 37 separate tests, from small LLVM features to high-level tests like the CubeScript engine, dlmalloc, and raytracing; each test is run through both Clang and llvm-gcc, and with relooping&optimization both on and off, for a total of 148 tests. This takes 7 minutes on my slow laptop, which is starting to be significant, but it's extremely important in a project like this.

Next goals include performance of the generated code - lots, lots of low-hanging fruit there - and compiling yet more real-world code.

Emscripten, Now With More Clang

2010-09-21T21:10:00.001-07:00

Emscripten can now work with Clang. It turns out that the llvm bitcode that Clang generates is slightly different from that of llvm-gcc, which uncovered various minor bugs and missing elements in Emscripten (for example, the 'phi' command).

All the tests now pass both llvm-gcc and Clang, and with both optimization and relooping on (however, relooping has been weakened, due to some bugs that were discovered).

The benefits of allowing Clang to be used is that having two sources of LLVM bitcode is better than one - more chances to catch bugs (but even more important would be to add non-C/C++ sources of LLVM bitcode as well!). Another benefit is that Clang is simpler to build so it will allow more people to play with Emscripten.

The wiki has been updated with full instructions, so you can get Clang and try it out with Emscripten right now.

CubeScript on the Web

2010-09-11T21:16:00.001-07:00

Emscripten 0.2 is out, and here's a silly demo: the CubeScript engine from Sauerbraten, compiled from C++ to JavaScript, and running in a web page. Finally, you can use a script language on the web ;P

A few more details about the demo appear on that page. Other details about the 0.2 release are in the changelog.

After successfully compiling the CubeScript engine (which was mainly to see if Emscripten could do it - fixed a lot of bugs on the way), I think most C/C++ stuff should work (but there are probably a lot of minor corner cases left). The next steps are something like this:

Version 0.3: Optimize the compiler for speed. Right now compiling CubeScript, about 2,500 lines of code, takes a minute on my (slow) laptop. The goal is to compile large projects, so this needs to be much faster.
Version 0.4: Optimize the generated code for speed. Some radical solutions for making it faster are possible, but might take a lot of time, so maybe post-1.0. But some straightforward optimizations should be done in the near future.
Version 0.5: Tools and integration. Make it easy to build multi-file projects, and to connect the generated code to web JavaScript (calling functions both ways).
???
1.0!

Emscripten

2010-08-29T10:22:00.000-07:00

As mentioned in the last post, I am checking out ways to bring Syntensity to the web. As part of that, I just put up Emscripten on Google Code. Emscripten is an LLVM-to-JavaScript compiler. Combined with llvm-gcc, it lets you compile C/C++ code into JavaScript, and run that on the web.

Emscripten is still in an early stage, but can already compile some benchmarks. At this point I think I will start to try to port code I am interested in, and fix bugs along the way as I find them. I'll probably begin with a tiny subset of sauerbraten (probably one out of command.cpp, rendermodel.cpp, or physics.cpp) and see how that works, then continue from there.

The long-term goal is to run Syntensity, or a smaller version of it, on the web. Aside from Emscripten this will require some other tools, and probably a lot of hard work, so I can't say when I expect this to be done. Also I am working on it in my spare time, so that will vary depending on other stuff I'm doing. But now that Emscripten is in good enough shape to start using, the fun part can begin.

Side note: I'll post updates on this blog regarding using Emscripten for porting Syntensity to the web. I'll probably post stuff about Emscripten itself, that isn't related just to Syntensity, on my more general blog, here.

Update on Where Things are Going

2010-08-14T20:09:00.000-07:00

It looks like the Syntensity codebase (and community) is going in two directions:

A few community members (quaker, BiosElement) have started CubeCreate, a fork of Syntensity that is not afraid of breaking compatibility with Sauerbraten, in that they will rewrite Sauer code and add major new features. It definitely looks very interesting and they are putting a lot of effort into it. For more details, visit their IRC channel (#cubecreate on FreeNode), or forums.
As for me, I have been thinking about something sort of in the opposite direction. I think gaming should move to the web - like everything else pretty much already has or is in the process of doing. To do that, things need to be - at least in the near future - lighter and simpler, to make moving to the web easier. So I've been working in my spare time on tools for that. One tool is an LLVM-to-JS compiler, which lets you run (some) C++ code inside of a web browser, by compiling it to LLVM and then to JavaScript. My goal is to move existing C++ code - Sauer, Syntensity, Bullet, etc. (or maybe simplified versions of those) - to the web that way. The tool is far from finished but already can run various benchmarks, I'll probably put the project up on some open source hosting site soon.

So those are the main directions in which things are going. They sort of go in opposite ways, but of course there's nothing wrong with that. The more the merrier ;)

Plan for Next Release: Metallblech

2010-06-20T10:23:00.001-07:00

gk has a great concept for the next game, which you can see here. That game will be part of a new release.

Help from community members would be great, and let us get to the release faster. Stuff we will need help with includes

Coding the new gk game linked to above. Mainly we will use existing scripts and customize them, but there will also be a few new things as well
Testing that game
Need to decide if to merge from the development branch for this release - if so, will require more testing time. Opinions?
Making binary builds for Linux, Windows, OS X

There is no timeline, it really depends on how much help we get. But it will be awesome when it arrives! :)

Development Update

2010-06-13T10:39:00.000-07:00

After the recent changes, we have been working out how to move forward. This is what makes sense for now.

Regarding code branches:

The main github repo will remain as the stable branch. It will always be in sync with the running servers at syntensity.com, so if you want to connect to there, you should run that code.
quaker66 has been doing excellent work on his branch, which is now the official development ('unstable') branch. quaker66 has worked very hard to port all the recent features from sauer svn (lots of great stuff that eihrul has been doing there) as well as awesome new features like flashlights, better cutscene management, etc. You can see videos of that stuff on his YouTube channel. In addition, T7g and BiosElement have been doing very interesting work as well, some of which has already been merged into quaker66's branch. New stuff should likewise be added to that branch.

Regarding versioning: The current version is still 1.1.6. For a new release, we would merge from the development branch, bringing all the changes from there. To do so, we would need:

For the code to be stable and well-tested - after a reasonable code freeze. We might also want to wait for a short while after a new sauer release.
At least one new cool game, to showcase the latest features
To update the syntensity.com servers
To have binaries for people to download

Short Absence - Almost Over

2010-05-07T11:15:00.000-07:00

I just wanted to clarify why I haven't been around much the last 2 weeks. I relocated to Mountain View, California, and I started my new job at Mozilla. All very exciting stuff, but also kept me very busy.

However, I still would have been on IRC etc. on the weekend and possibly evenings, but I have had some trouble in getting my apartment connected to the internet (in silicon valley - somewhat ironic, heh). But that should be resolved in less than a week and I will finally be internet-enabled.

EDIT: AT&T fail - I am still without internet. They hope to get it working next week though.

Windows Installer

2010-04-24T08:07:00.001-07:00

Thanks to community members gk and rapiscan, we now have a proper installer for Windows. It includes the latest code (1.1.6). You can check it out on our download page, which links to Mod DB.

If you do try it, please let us know how it works.

2 Interesting Videos

2010-04-14T13:23:00.001-07:00

Two interesting Syntensity-related videos are embedded on the Bios Interactive blog, worth taking a look at.

New License: MIT

2010-04-12T00:35:00.000-07:00

Today I changed the license on all my code in Syntensity/the Intensity Engine to the MIT (X11) license, for the reasons previously discussed in this blog.

The MIT license is a permissive software license, compatible with the GPL, and similar to the BSD. It it simpler than the BSD license in that the BSD has several variants, some of which have clauses that are not immediately clear - which was recently pointed out to me. (What does "Neither the name of the nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission" mean, exactly?)

Mapmodel Mesh Collisions

2010-04-11T03:53:00.000-07:00

The video shows a mapmodel with mesh collisions - so the player can collide with the mapmodel properly, and not with a bounding box. Note how the player can move through the holes in the model (but not through where the model is solid).

The mesh data is read from the sauer model, and converted and passed to Bullet, which handles the collisions. Scripts can also, optionally, create collision meshes on the fly.

EDIT: Btw, this is still a little buggy. For example, you need to rotate the mapmodel 180 degrees, and it doesn't notice other rotations for now. Use the bullet debugging option to see what is going on (/bulletdebug 1). EDIT 2: Fixed.

Personal and Project Changes

2010-04-06T04:46:00.000-07:00

Some of you may have noticed that I have been less active on IRC (and in general) recently. The reason is that I was in the US for some job interviews. And the reason for that is that, as already mentioned on this blog, it looks like the business side of Syntensity won't work out, at least for me. So I can't continue to work on it fulltime as I have been for almost 2 years.

I still believe there is a lot of business potential here. But it would take someone with more industry contacts, business development skills, and money in the bank, than I do. Syntensity's technology works very well - I am very happy with it, and I think others are too - but I guess I was too optimistic about finding business partners and people to invest money in the venture and so forth.

Syntensity is open source, though, and that means anyone else can try to use it for a profitable business. And in fact some people are, and more will probably follow. (For more on that, and licensing changes, see below.)

I just signed a contract with one of the companies I interviewed at, and I'm going to start that job very soon, which I am very excited about (this blog post isn't about my new job, though, so I won't go into more detail now). But I am by no means leaving this open source project - I will continue to work on it in my spare time. I feel that this is a reasonable time for me to decrease my involvement, as

The engine is feature-complete, at least for v1.0. More stuff would be nice, and will be added (in particular I'll work on finishing experimental stuff like Bullet, but also entirely new stuff that can be useful), but actually there is a lot of potential not even used by the current games. I have said it on IRC several times, what Syntensity really needs now is not more features, but more games.
The engine is stable. Bugs are found, but most are in experimental features. The core stuff has been used daily for several months now, with few issues (and what issues did arise, have been resolved).
More people are getting involved in development, in various ways. Hopefully the dev team will expand.
There is a growing group of users that are helping others, which is a very good thing.

Basically, the engine is in a fairly mature state, and the community has come a long way from where it began. I would have been worried had I decreased my involvement before or around the 1.0 launch (and in fact that is one reason why I only started looking for a job recently), but now I have a very good feeling about the future of the project. And it is a healthy thing for the project as well, not to rely too much on a single person, at least once it gets to the proper stage - which I think it has. Also, after almost 2 years, it is time for a change for me personally, and I believe the new job I will soon start will be a great opportunity for me to do both interesting and important work, alongside some great people.

Some important stuff about Syntensity:

Now that my personal business goals are irrelevant, the Syntensity/Intensity Engine name split is not really necessary (it was inspired by things like StatusNet/identi.ca, which clearly separate the open source project from a commercial use of it). So, henceforth the Intensity Engine may also be called the Syntensity Engine or just Syntensity. I won't bother to go through all the code and docs to change every reference to "Intensity Engine" to "Syntensity", but Syntensity is now the preferred name (it's shorter, and easier to google for). Note that syntensity.com will remain up of course, as a 'hub' for Syntensity game development, hosted by me, without change.
I will be changing the licensing of the Intensity Engine/Syntensity, from GPL3 to the (new) BSD. Some of the reasons have already been mentioned in previous blog posts. Another is for the following reason: Allowing commercial use is a major goal of Syntensity, but for games that means non-GPL licenses are a must (you can't run GPL code on consoles, etc.). Dual-licensing with the GPL is possible, and was the general idea so far. But since I will have significantly less time for this project, and the business opportunities of dual-licensing no longer relevant for me, it doesn't make sense to deal with the complexities of dual-licensing (copyright assignment to some central entity, etc.). It is just simpler to move to the BSD license for all involved.
Once I start working, I will be much less on IRC. However, I will of course continue to be active on the Forums, so feel free to use that in order to contact me, for stuff both important and nonimportant.

New Script Tutorial/Docs

2010-03-18T04:12:00.000-07:00

Continuing the recent trend of writing docs, I wrote a new scripting tutorial, which is now present in the git repo. See

docs/scripting.markdown

That document refers to the example scripts in

docs/scripting_examples/

This should be a clear and easy way to get started with scripting in the Intensity Engine. I'll add more examples later on.

'Making a Game' Tutorial

2010-03-16T07:47:00.001-07:00

There is an increasing amount of interest in using the Intensity Engine separately from syntensity.com, in some cases without a master server at all (like, say, for a singleplayer game). The engine supports this very well, but documentation has been lacking. So, I started to write a tutorial for that. Feedback is welcome.

Rain Effect

2010-03-14T10:10:00.000-07:00

Somebody on IRC mentioned today that a rain effect would look nice, so I made a quick one:

Note how the rain is realistic in that it falls until it hits something, then shows a splash. So there is no rain inside the castle or underneath the scenery. This takes a little CPU, so very heavy rain (thousands of drops at once) might slow you down. In the video there are about 1,000 and speed seems ok.

The effect is about 40 lines of code, in library/1_3/CustomEffect.js. You can see it in action if you run the latest git, and play the storming_test map.

Editing Tools in JavaScript

2010-03-14T04:39:00.000-07:00

The video shows a 'slope tool' that makes it easy to create sloped surfaces in cube geometry. It isn't finished, but it shows what can be achieved by this approach, I think.

Basically, this is done using new JavaScript API bindings to editing code that works on the cube octree. Actually the editing code is fairly old - it was used already in a very early version of the Intensity Engine - but it was not enabled in the current JavaScript API until this week.

Using the expressive power of JavaScript, it is easy to create editing tools like this one, for all sorts of purposes. For example it can be used to import maze files to create maps with them, etc.

You need to run the latest compiled engine for this. The code is in library/1_3/Editing.js, and you can see it in action in kripken/editdemo, which you can run by entering that name into the server_runner plugin GUI, after logging in to the master (see README-standalone.txt). To see it work as in the video, do /actionkey0 when hovering on one corner, then the same command when hovering on the opposite corner (this is just for demo purposes, not how the final controls should be).

Move to a Permissive License? (Part 2)

2010-03-12T00:37:00.001-08:00

This is a followup to this post.

Following the previous post, I got some feedback in comments and on IRC. Here are some of the points raised, and comments on them:

It was pointed out to me that zlib is the most permissive license of those mentioned, allowing binary copies to be made without attribution, whereas BSD still requires that you keep the copyright notice in that case (so you don't need to distribute the source, but you do need to document that you are using code written by whomever wrote it). It seems MIT does the same thing, but the wording is slightly less clear - BSD explicitly mentions source and binary distributions, while MIT just talks about 'copies of the software' (which many people seem to interpret as intending both source and binaries, but I would have expected to see something like 'copies or transformations of the software', where 'transformations' means compilation to binary form, etc.). So, out of zlib/BSD/MIT, I lean towards ('new') BSD.
I should clarify my goals with the license change. There are basically two: First, I want the code to be useful for people, however it can be, which includes making money off of it in commercial games. My second goal is to improve the situation of open source in gaming - which is currently one of the last fields of software almost entirely dominated by proprietary code (compare the situation to operating systems, web browsers, web servers, databases, programming languages, etc. - open source is doing very well in all of those). I guess the two goals are not necessarily in perfect alignment, hence the need to think carefully about the license.
Also I should clarify my own personal goals&status: I am looking for a job right now, so I will likely have less time for this project than I have had, but I certainly intend to continue contributing. Anyhow since we are past a stable 1.0 release, and the engine is in very good shape, really what we need now is more people creating games with it rather than more engine features that I add.

About my role in the project, I would be very happy if the development team were to expand, and have more sharing of responsibilities. I've been talking to related projects and various people about that, and hopefully there will be progress on that front soon. Permissive licensing might make that easier, since with GPL projects it matters who owns copyright, whether you have copyright assignment or not, and all that, whereas in permissively licensed projects it is far less of an issue.
I've been looking at open source code that is successful in gaming, for inspiration. Basically three projects come to mind, detailed below. I'm not sure what lesson to learn from them yet, but they are worth thinking about.
- Quake Engine(s): Released as open source some time after the commercial releases, GPL with dual licensing. The GPL option has been used extensively in the open source community. However, the GPL uses did not become a significant force in gaming. Basically the GPL code releases play catchup, always being behind the closed code. Also, it is not certain that GPL code releases will continue after id Software was acquired by ZeniMax.
- Ogre 3D: Used to be LGPL + optional commercial licenses, has moved to MIT. It is not clear yet if moving to the MIT license will change anything, as Ogre 3D was already doing very well before, including being used in commercial games. I am guessing that Ogre 3D has support for consoles, but that code isn't in the public repo (likely it can't be, because of the console SDK terms of use), which leads to a quasi-dual licensing model: Free for use (MIT) on regular computers, requires a license for consoles (or, implement all the console integration yourself, in theory).
- Bullet: zlib licensed. The Bullet physics library is very popular, and has been used in many commercial applications, including ones that significantly modified Bullet, presumably without contributing changes back (but not sure). Unlike Ogre 3D, Bullet can be used on consoles freely, as there is really no need for special console integration code.

Move to a Permissive License?

2010-03-11T07:14:00.000-08:00

I am considering relicensing all the code I wrote in this project to something more permissive, like zlib, BSD, Apache, etc. - basically, to let people use the code entirely freely. Please let me know whether you think that is a good idea or not, and if it is, which license we should switch to. This is a big decision - involving some 65,000 lines of code (about the same as Cube 2, btw) - so it should be done carefully.

As background, I chose the AGPL when I started, because I wanted to keep open the possibility of dual-licensing (to generate income for the project), and to make fragmentation of the community less likely. The situation with both of these has changed: I don't intend to make money myself from this project anymore (I still think it can be done - but it would take time, and before that time arrives, I will have run out of money myself...), and so far there seems little worry of fragmentation and incompatible forks.

So, I am considering moving to a license like zlib/BSD/etc., which are simpler and make life easier for people using the code - clear benefits, and could lead to more people using the code. Of course the flip side is that the code could then be used and modified in closed source projects without returning anything to the community. These are the usual tradeoffs between the BSD and GPL approaches.

Some thoughts on possible licenses:

zlib: Same license as Cube 2, so kind of simple in that respect - a single license for almost the entire project (almost, because things like V8, ENet, Django, etc., have other licenses).
BSD: Probably the most common permissive license out there, and the most familiar to people. Also worth mentioning, zlib is not an option on Google Code, while BSD is.
Apache: Permissive, while adding some patent protections, which is useful. However, it does make the license text quite long and complex compared to zlib/BSD/MIT, which is a downside. Another downside is Apache is not compatible with GPL2, but is with GPL3. There are many codebases out there that are GPL2-only (including ones relevant to this project) so that is an issue.
LGPL, MPL, etc.: More permissive than GPL&AGPL, but still some requirement of giving back, at least for changes to existing code. If the license is to be changed, I kind of tend not to go this route, because the current license is already quite close to it, since we don't consider games to be derivative works anyhow, which makes our AGPL sort of like the LGPL in that respect (a good comparison is how the Linux kernel is GPL, but you can run apps in userspace that are not GPLed). So really the actual change in going to one of these licenses is the removal of the 'A' from AGPL, that is, removing the condition of providing the source even if running over a network.

So, again, I am looking for feedback on this topic. We can also hold a vote if there are strong opinions and that makes sense. One thing though: Please let's not make this into a GPL vs. BSD flamewar with all the usual arguments we have all heard before - I ask that the discussion be about this project, which license makes sense for it (and not which license in general is 'better').

EDIT: Followups in this post.

Running Locally & Singleplayer

2010-03-03T00:46:00.000-08:00

The server runner plugin can now work entirely offline - without connecting to anything remote, not even a master. In such a case it will run using the local files. Basically, running a map is now as easy as entering the name of the map in the GUI and clicking 'start'.

In such a situation you don't have any of the benefits of the master server, like getting the latest version of the game, being able to upload changes, play multiplayer or co-op games, etc. But, it is useful for when you don't have a network connection, for demos, or if you just want to run entirely standalone - like for a singleplayer game.

Speaking of singleplayer games, in a few lines of script the client can now be set up to automatically run a certain map using the plugin, which would basically turn the client into a single-purpose game: Start the client, and that game runs. If you also change the branding (logo, menus, etc.), it is now easy to use the Intensity Engine for your own singleplayer games.

(The underlying functionality for these new features is that you can set [Activity] force_location on the server, and it will run the map with that location, without querying the master. It will also communicate that fact to the client, so it does the same. The server runner plugin basically runs the server in a transparent way, with that feature enabled.)

Local Server Runner Plugin

2010-03-01T06:15:00.000-08:00

People have complained that local editing and playing were harder than it should be. A plugin that I finished writing today, server_runner, should help with that. When the plugin is active, you can start a local server with just a few clicks in the GUI. The plugin will run the server in a separate process (so if your scripts crash the server, the client won't crash - kind of like how modern web browsers run plugins in separate processes).

Usage:

Activate the plugin: Add intensity.components.server_runner to [Components] list in your settings.cfg
After logging in to the master, click on 'plugins...' in the main menu. You will then see the status of the local server.
Tell the plugin which map to run (simply by writing the location of the map, e.g. "racetrack" for the racing map). Then click start, and wait a bit while the server is started up for you. As soon as it is ready you will automatically connect to it.
The server will be shut down automatically when you close the client, or connect to another server, so no need to worry about that. You can also shut it down in the GUI if you want (which is necessary to start it up with a different map).

Limitations:

Requires you to be logged in to the master, but allowing master-less running would be possible if people want it.
If the server crashes, output is saved to ~/.intensityengine_client/out_server.txt (note: not in the install directory, but the engine home directory). A nice GUI for viewing the output should be added later.
You can't write the location of a map that you never played on before, as it only looks locally for translating the map location to the asset info.
Tested on Linux. Should work on other OSes (it is 99% pure Python), but not tested yet.
The server still uses a network port, so your firewall may ask about that. It would be possible to move to a pure IPC solution later, without that problem, but it would be additional work.

You must be running the latest git source code to use the plugin. If you do, I'd be happy to hear feedback on how things work.

Background:

Sauerbraten, the engine that the Intensity Engine is based on, lets you easily run a local server - it runs the server inside the client, in fact. This makes sense, since the Sauerbraten server is very simple, it is basically a network message router.

However, the Intensity Engine lets you run scripts and physics on the server, which raises the possibility of the server crashing (due to bugs in scripts, etc.). The server also can take a lot more CPU and RAM because of that. Consequently, the server is strictly separated from the client - there is no option to run it inside the client. Another reason is since the server now does a lot of the things the client does (scripts, physics), then maintaining a single codebase with the server defined as separate - and not two options, separate or internal - is much easier.

Bullet Physics

2010-02-15T08:42:00.000-08:00

The embedded video shows the Bullet physics library being used in a Syntensity world. 100 boxes are simulated, as well as the static world geometry. As you can see, the boxes behave in a physically realistic way.

This is just an early demo - it would take more work to get this ready for actual games. But the hard integration part is now finished.

If you're curious to run this yourself, then instructions are at the end of the COMPILE.txt file.

Windows Building Improvements

2010-02-10T06:43:00.001-08:00

Building on Windows should now be much easier:

SDL (all parts) is bundled in the git repo in binary form, so you don't need to get that or set it up. These are the latest versions of SDL, which include some useful bugfixes.
Likewise the latest zlib is bundled in binary form.
Boost.Python is bundled in source code form, and compiles correctly, which prevents the need to get or set up Boost at all. This is very nice as the precompiled Boost we previously used assumed you were using Python 2.5. Consequently, the engine will now build using Python 2.6.

The process should now be very easy, basically just

Get git, scons, CMake and Python.
Get the Intensity Engine source code.
Build V8.
Run CMake to generate a VC++ project file.
Build the project in VC++.

For specific details, as usual see the COMPILE.txt file.

I develop on Linux, so sometimes the Windows stuff gets delayed, sorry about that. Speaking of which, volunteers willing to help with Windows and OS X stuff would be very welcome.

Browser Plugin: Proof of Concept

2010-02-02T06:22:00.000-08:00

In the following video you can see Syntensity being run inside a browser, as a plugin:

This is a followup to my post from last week, 3D and the Open Web. After the speculation in that post, I got to coding, and the result is a working proof of concept.

Status

The plugin works, as you can see in the video, and you can enter Intensity Engine servers (including Syntensity's) normally, just as you would with the regular client. This is still in early stages, however, so you should expect some limitations:

The plugin will currently only build on Linux (where I develop). However the code on which I worked is cross-platform (see below), so this should not be a problem to remedy later. Meanwhile, Linux users can try it out for themselves using the instructions at the end of the COMPILE.txt file.
Works in Firefox and Chrome, not tested elsewhere (that's Chrome you see in the video).
Mouse input works reasonably well. Keyboard input should work as well, but doesn't, probably for some minor GDK reason I will look into later.

Technical Details and Notes

Built using the O3D plugin code (which in turn is closely related to the Chrome codebase). The O3D code is not meant to be a general-purpose basis for browser plugins in general, so it took some hacking to get things working, especially to integrate with SDL (which we use for rendering and input). Not surprising I guess. Anyhow, the O3D code was very useful here, thank you to the authors!
The implementation I wrote runs client instances in separate processes, which is good for security and stability. In other words, the browser plugin is a small component that runs actual clients (that render, run physics etc. etc.) in separate processes.
Interprocess communication is done using boost::interprocess (Chrome's more powerful IPC turned out to be more complicated to set up, and anyhow it's probably overkill for what we need).
Very few modifications were necessary in the main Intensity Engine codebase. Rendering is still done using SDL, etc. A couple of minor tweaks were needed here and there, though.
A combination of Firefox and Chrome turned out to be useful during development - Firefox is easier to use in debug mode, to figure out crashes, while Chrome is more resilient to crashes bringing down the entire browser (although I hear Firefox will be adding a similar feature soon).
Overall, my conclusion is that the browser plugin approach is not easy (there were several technical hurdles that I ran into, like this) - but it is possible.

Other Approaches

As an experiment, I think it is now clear that the browser plugin way is feasible. It's still an open question of whether it is the best approach, or whether one of the alternatives (mentioned in the previous post) would make more sense. I tend to favor this approach though (making our own browser plugin), for the following reasons.

I have gotten no response on the O3D mailing list, where I asked about possible collaboration (like contributing our code to there, to add features games need like networking, physics, etc.). I guess it isn't relevant for them. In that case, it probably doesn't make sense for us to move to O3D - porting the rendering engine would be a lot of work, and it turns out that making our own browser plugin (in part using their non-rendering code) is feasible.

WebGL seems to be favored by some people in the virtual worlds community. It's definitely an interesting technology, but I am doubtful it would work for games, or more specifically action games like we have in Syntensity, for several reasons:

WebGL is not complete, and WebGL's performance limitations are not yet clear (and not sure when they will be).
WebGL + JavaScript - a pure web implementation - would be too slow for very complex clientside physics. NaCl might help with that, eventually.
Network protocol-wise, TCP (even with nifty things like Comet, etc.) can't give the performance that UDP can, in terms of realtime updates.
Also, it is unclear if WebGL will ever make its way into Internet Explorer. After all, WebGL is based on OpenGL, which Microsoft has been trying to kill. A related issue is that not only is a major browser hostile to WebGL, but also a major desktop operating system - users often need to manually install OpenGL drivers from the manufacturer's website, which is no better than installing a browser plugin (in some ways worse).

That said, the Intensity Engine has always been a web-oriented project, what with JavaScript used for scripting, HTTP for asset downloads, asset and server management done using a web interface (on Django), etc., so if later on WebGL matures into something relevant for us, we can certainly consider moving to it.

Bottom Line

The open source community can have something like a FOSS Quake Live, or more generally a proper open solution for 3D content on the web, from casual stuff to multiplayer action games. The potential here is vast, and it is within reach, basically right now.

If we are to go forward with this, we will need help from the open source community, as browser plugins need a lot of testing on all the different operating systems and browsers and so forth. So, help is welcome. Also, ideas on how to do things would be appreciated, in particular about how to get the licensing issue right - as mentioned in the previous post, if changing our license will get more people on board, we are open to discussing that.

3D and the Open Web

2010-01-24T05:35:00.000-08:00

This post is a call for feedback, about Syntensity's engine (the Intensity Engine) being used on the web. The basic idea is that we - people on the web - should avoid what happened with Flash, a single vendor controlling a closed technology, which became the de facto standard for video and interactive content for many years. Instead, we need an open source solution, not controlled by any one corporation. I think the Intensity Engine is close to being suitable for that, and am looking for feedback on this idea, and help in achieving it.

A big part of the success of the web is its openness: We can run websites on any of several webservers, some of them open source (Apache, lighttpd, nginx, etc.), and visit those websites using likewise open source web browsers (Firefox, Chromium, etc.). That's an amazing achievement. And let's not forget that just a few years ago Internet Explorer was dangerously close to a choke hold on the web browser side.

Video, these days, is often in the tech news: Specifically, using open, standardized technologies to play video on the web, using HTML5. There are some problems along the way, but overall we might be close to getting past video on the web being entirely reliant on Flash - a closed-source product controlled by a single company, and one that doesn't necessarily perform as well or the same on all platforms.

Here I'd like to talk about the "3D Web". The term was overhyped in the past, but really all I mean here is 3D content, mainly games and virtual worlds, that are accessible on the web. This is currently a smaller area than the web in general, or even video on the web, but it is growing in importance. My concern is that the open web should avoid 'Flashification' of 3D content, where a single closed-source product becomes the de-facto standard in the area, like Flash had (and mostly still has) in video, 2D gaming and interactive content. If we want to avoid that, the time is now.

There are some open technologies that show promise, mainly WebGL and Google's O3D. These may well end up succeeding. However, neither is a complete game engine, like for example the Unity 3D web plugin. There is a lot more that is necessary over what is present in WebGL and O3D - physics, content creation tools, a proper API and useful libraries, network protocols (for multiplayer), etc. etc. Some of that might be added to WebGL and O3D using JavaScript. However, many games are too computationally intensive, even with the best JavaScript engines out there.

Perhaps at some point Google Native Client (NaCl) will allow running game engines on the web. But instead of entirely relying on that, I think the open web needs an open source 3D gaming engine. The time to do it is now, before something else non-open comes to dominate the field. I'd like to suggest the Intensity Engine for that purpose: It is a complete, stable, cross-platform game engine. It works right now (outside of browsers) and has been in production for several months, successfully, on syntensity.com. It is 100% open source, and the current license, the AGPL, can be modified immediately to something else, like the BSD license, if that makes sense for this purpose. Also, the Intensity Engine was built with something like the web in mind - we use JavaScript to create games (Google V8 right now, and we also did some tests with SpiderMonkey), for example. In our mind, the ability to download and run games was always in parallel to how web browsers download and run web pages.

One concrete idea among others is to port the Intensity Engine's rendering system to O3D, and build a browser plugin of the result. The benefit being O3D is already set up as a browser plugin, while the Intensity Engine provides all the other game engine stuff. Alternatively, we can just port the Intensity Engine as-is to be a web browser plugin, assuming that would work with SDL (if not, would need to work on replacing that).

So, I'm looking for feedback about this topic, and ideas and help for how to move it forward. I really feel it isn't just us over here (in 3D gaming) that care about this stuff - lots of people want the web to remain open, and that should include 3D content and games.

Thanks for your responses!

Edit: I posted on relevant mailing lists about this,

Local Storage Plugin

2010-01-24T03:26:00.000-08:00

A new plugin has just been committed, local storage, which lets you store key-value pairs locally. It's currently being used to persist the high scores in the Racing game.

Usage:

Add intensity.components.local_storage to the proper place ([Components] list) in your settings.cfg, so the plugin will be loaded.
In your scripts, do CAPI.signalComponent('LocalStorage', 'read|KEY'); to read the value for KEY. You will get in response a string (all output from signalComponent is in string form), so do eval() on that. The result will be a list of responses from components. Normally you will have only one storage component, so just get the value at index [0].
To write values, do CAPI.signalComponent('LocalStorage', 'write|KEY|VALUE'); to write that KEY-VALUE pair.

That's basically it. See also the source file for some technical docs and notes.

New Forums on FreeGameDev

2010-01-18T09:33:00.001-08:00

We have recently been given a space on the FreeGameDev Forums, which is a much better place for community discussion than our current forum page. You can check out our new forums there at this link.

Aside from a more standard and useful forum approach - subforums, topics, etc. - it also makes a lot of sense for us to do it on FreeGameDev, because we share their approach about combining free and open source software with gaming. Gaming is, sadly, one of the last areas to which open source has not yet made a big impression. Most game engines are entirely closed source, and in fact many games also run entirely on locked-down hardware (consoles).

But that isn't how things need to be, and I predict they will change. The are reasons that led to our having open source operating systems, web browsers, web servers, and so forth, that are either better than or equivalent to competing closed source products, and those reasons are essentially the same in gaming.

So, having forums as part of the FreeGameDev Forums makes a lot of sense, and I hope it will lead to more interaction between Syntensity people and other open source game creators.

Thanks to the FreeGameDev people for agreeing to host our forums there!

Progress on Mac OS X support

2010-01-17T23:31:00.000-08:00

Thanks to abs1nth, the client now compiles on Snow Leopard (and possibly other versions). To do so, get the latest git source and follow the OS X section in the COMPILE.txt file.

The move to CMake, and removing the external dependency on Boost, made this more feasible than before (and they also make building on Windows easier).

HOWTO: Make your own Razanak maps/levels

2010-01-15T08:45:00.000-08:00

In Syntensity everything is moddable. That includes Razanak, the game we launched recently. Below is a step-by-step guide to starting your own Razanak-type map.

"Razanak-type map" means: Everything will be exactly the same as in Razanak - gameplay, entity types, HUD, etc. etc. - except for the map itself and entities in it. That means you can create the level however you want: You can add corridors, caverns, water, spawnpoints for the spider robots, rocket turrets, doors, keycards, pickups, and so forth, however you want. You can make levels similar to the existing one, or entirely different ones - it's up to you.

Here are the steps to follow, to prepare a new map based on Razanak:

Click on Razanak in the list of running servers and activities. This gets you into the Razanak activity.
Click on the map asset. This gets you into the actual asset containing the map. (You could have gone here directly if you had a link, instead of through the activity, so step 1 isn't really needed. But it's how you might usually do things.)
Click 'clone'. You now have a copy of the map that you can edit, as its owner. (Note that to do this step you need to be logged in on the master website.)
Change the 'location' of the asset. The location is where it will be downloaded to in the asset storage areas on the client and server. If you don't change it, it will stay the same as Razanak itself, which means that you will re-download Razanak and this map if you play one and then the other (since each such download will overwrite the one before it). Instead, change the location to something like base/USERNAME/mapname.tar.gz where USERNAME is your username (you don't need to do it that way, but it is nice and orderly).
Click "Create an activity based on this asset" to create an activity. You now have an activity, which you can run on servers (assets by themselves cannot be run).
Rename the activity, to something you like.
Click "Requisition a server instance to run this activity". One of the Syntensity servers will then run your copy of the map. You should see it in the list of your requisitioned servers.
Connect to the running server instance. The simplest way is to run the client, login, enter the lobby, and look in the side room on the right. It will have portals to all the currently running servers. Walk into the one running your new map. You should then see the familiar Razanak map load.
Press 'e' to enter edit mode. You can't do that on Razanak, but you are an owner of this copy, so you can do it now.
Do /clearallents. That will erase all the entities in the map. Razanak has about 1,000 entities, and some of them require CPU to manage, so after doing this you will probably see the frame rate go nicely up.
Do /newmap. That will erase all the world geometry. Note that you might not see any ground at all, but don't worry about that right now. Also, the map textures have been erased, so you shouldn't try to actually do any editing right now. Just go on to the next step.
Edit the map.js for your new map, as follows. You can edit it with an editor of your choice, the file is in ~/.intensityengine_client/packages/base/LOCATION-OF-YOUR-MAP (that you decided on in step 4). (Note that you should edit the map.js in the directory with the name of the location - and without .tar.gz - and not the archive ending in tar.gz.) (In Windows, replace ~ with C:\Users\USER or C:\Documents and Settings\USER, etc.). Or, you can use the in-game editor, press Escape and then editing commands...->cfg (then do 'load'). The actual editing you need to do is to add these two lines at the *top* of the very short map.js already there:

Global.entitiesFile = './entities.json';
Global.noCutscenes = true;

The first line tells the map to use your entities data (not Razanak's), and the second tells the map not to use Razanak's cutscenes. Again, be sure to place these two lines *before* the existing line.
Do 'upload map...' (press Escape first for the menu). When the upload finishes, the new version of the map will run on the server and you will be placed in it automatically.

You can now start mapping! More details about mapping and creating in general are on the wiki, and as usual feel free to ask for help on IRC (#syntensity on FreeNode) and the forum.

Tip: To get started, you might want to add a WorldMarker (press F8 in edit mode, and select WorldMarker in the second tab), for which you edit the 'tags' field (rightclick on the entity) to [start_red] (note the [, ] symbols). That will be the position where the players start ('red' is the name of the team, which should really be named something else, but the default team names are 'red' and 'blue').

Razanak a.k.a Swarm

2010-01-08T23:18:00.000-08:00

After much work, 'Swarm' has been launched, and is in the lobby. It is now named Razanak (thanks BiosElement!), which is 'Swarm' in Hungarian (give or take a few letters).

To see all the visuals correctly, you need the 1.1.5 or 1.1.6 builds. You can currently log in with older versions, but that will be disabled very soon (possibly by the time you read this). The latest builds can be downloaded from the download page (optionally, you can also compile from source if you want). We hope to have builds with nicer installers, and for more platforms, soon.

Currently there is one (big) level, with a boss at the end (that can be quite hard to kill). We plan to add more levels soon, they should be much faster to create now that the code is all ready.

Have fun, and let us know what you think!

Help us name and test 'Swarm'

2010-01-04T09:18:00.001-08:00

We're almost finished with 'Swarm', and we'd really appreciate some help with naming it, and playtesting. So, if you want to help, check it out here:

http://www.syntensity.com:8888/tracker/activity/view/46183c34_3b4f_44dd_9a63_4ca498603bf1/

and here is a video:

'Swarm' isn't in the lobby world yet, since it isn't finished. To test it, you need to requisition a server. If you need help in doing that, there are usually people on IRC (#syntensity on FreeNode) that can assist you.

So, as already mentioned, at this point we are looking for a final name for this map ('Swarm' is just the codename). Feel free to suggest any and all names that you think make sense!

Also, the map is ready for playtesting - everything should basically work, except for the boss at the end, which works but wasn't tested much yet. So if you find a bug or have a suggestion for how to improve something, that would be great (we probably won't change anything major at this point, though - just minor stuff. Big changes will wait for after we release the stable version of the game).

Note that a recent build of the engine is necessary, preferably from source. Otherwise the visuals might not look as good as they should. We should have new builds available soon.

CMake and Git

2009-12-31T01:32:00.001-08:00

This week the Intensity Engine has undergone a minor 'reboot': I moved it from SCons to CMake, and from bazaar+Launchpad to git+GitHub. The reasons are as follows.

SCons is a really cool tool, especially for Python lovers like me, but CMake's ability to produce 'native' project files and makefiles is quite important. Basically, on Windows it is easier to make proper builds and installers if you use a Visual C++ project file, as opposed to SCons. I wasn't fully aware of this when I started the Intensity Engine a year and a half ago (being a Linux guy). Also, this will help on OS X, as CMake can produce Xcode project files. So, it made sense to move to CMake.

Moving from bazaar to git was for related reasons. While bazaar works well in my experience, on both Linux and Windows, more than one person has told me it is very problematic for them on OS X. So, since working on OS X is very important to us, and also since in general people seem to prefer git, the switch was done. Basically, I prefer bazaar, but my personal preference isn't enough of a reason to stick with it. Anyhow, after finishing the move, I admit it is nice how fast git is, and GitHub has some nice features as well.

So, the overall goal with these changes is to allow more people to more easily use and contribute to the Intensity Engine. The project is growing, in both lines of code and users, and it makes sense to use the best tools for the most people.

For people currently compiling from source, the COMPILE.txt file explains how to get the source using git and compile it using CMake. If you have any trouble, as usual feel free to ask for help on IRC (#syntensity on FreeNode) or on the forum page.

Slower Languages for Faster Code

2009-12-24T06:37:00.000-08:00

This post is both an update on performance improvements in the engine, and a general comment about how slower languages can lead to faster code, in a somewhat ironic way.

So, the next game (codename 'swarm') has really pushed the Intensity Engine to new limits: the map is huge, both in terms of raw size (4096 cube units on each side, so imagine something like 500x500x500 meters), geometry (the compressed binary octree is 1.1MB), entities (about 1,000 of them), and includes several new gameplay elements (new bot types, new plot triggers and items, etc.).

So, it initially ran very slowly, which was not really surprising. At first glance, this could lead someone to speculate that the engine, with its approach of doing as much as possible in a script language - JavaScript on Google V8 - simply isn't a good idea. After all, Cube 2, the engine we are a mod of, is written entirely in C++ (well, with cubescript in a very, very minor role), and therefore our engine should be much slower than where we started from. So didn't we just make things slower, needlessly?

First of all, that isn't an apples-to-apples comparison, simply because Cube 2 can't run the new gameplay elements that we have. But more importantly to the current topic, if you did mod Cube 2 to run the new gameplay elements, it would almost certainly run this map more slowly. That seems strange at first, but it really isn't. The reason is that, while code-for-code JavaScript is slower than C++ (even with the best JavaScript engines like V8), using JavaScript lets you implement optimizations that don't make sense in C++.

For example, if you have a fixed engine written in C++, you can of course tune it quite a lot, but there are additional things you can do if each map/game/level can customize things through scripts. For example, the swarm map has a lot of mapmodels, and the profiling tool reported that they were in fact a significant factor in slowing down the frame rate. Since the code that renders them is a script, I was able to easily customize it to run faster, by applying some heuristics that work well in this map (like certain ways of deciding when not to render a mapmodel). Those tricks might not work well in other maps, but that is exactly the beauty of a secure scripting language - each map can have its own scripts, downloaded on demand with the map. Also, I customized how physics for collision checks and so forth is done, and various other things.

Basically, you don't want to download a whole new engine for each map. But that is what modding the C++ code would require. With a script language, on the other hand, you can code a lot of optimizations and distribute them with the map, in an easy way. So you end up being able to optimize a lot more.

(There is another factor here, of lesser importance, but also worth mentioning: It is also simply easier to write those optimizations in a script language than C++, because script languages are faster to code with.)

Of course, C++ has its place - the core engine has to be written in C++. Even Java or C# wouldn't be possible, because while being slower by a factor of 2 is fine for 95% of applications, that is not the case with game engines. Half the frame rate is not acceptable! But on top of a 'core' C++ engine, I believe it really makes sense to integrate a script language, as we do, and not just because it lets you run more types of games, but also for performance.

Anyhow, to get back to the swarm game, at this point performance is good: On my not-powerful-at-all machine I get 60fps in the slower areas, dropping perhaps to 50fps with a lot of action. But in most of the map the frame rate is actually much higher. So, the optimization phase is almost over. What is left is to write the scripts for the final boss battle, and then to do a lot of playtesting and tweaking of the gameplay.

Adding Plot Elements

2009-12-18T22:51:00.001-08:00

I made a small video to demonstrate how plot elements are added to Syntensity games:

Plot elements are things like doors that need to be opened somehow, or platforms that appear so you can cross otherwise impassable areas. They are typically used in single-player games like Doom. Note that in Syntensity they function in multiplayer mode as well (there is no 'singleplayer mode' or 'multiplayer mode' in Syntensity - all games are always multiplayer-ready. But you can also play them locally with just one player, if you want).

As the video shows, when in edit mode you can see some visualization hints, like effects connecting doors and platforms to the triggers that activate them. The visualization hints show both what is connected to what and in what direction.

There are some other features of plot elements not shown in the video:

You can have multiple triggers for a door or a platform, so the player(s) need to trigger them all before it opens.
A trigger can also be defined to hold an item. In that case, to open the thing it is connected to, the player must reach the trigger, at which point they get the item, and then they must bring it to the right place.

Community Update: Ubuntu Lobby

2009-12-16T09:39:00.001-08:00

One of our community members, TheAncientGoat, has been working on a nice Ubuntu-themed map:

Besides looking cool, the map includes some web pages, of the Ubuntu homepage etc. (for which you need to enable the experimental web browser plugin). With the web pages updating live, it can be a nice way to keep up to date on Ubuntu news for people inside Syntensity.

Definitely an interesting project to watch.

Media Plugins

2009-12-04T09:46:00.000-08:00

I've been working on media plugins recently, here are short videos of the results:

That is the VLC media player, playing on a surface inside a Syntensity map. Note that sound works, but I didn't record it for this video.

And this video shows embedded web browser windows that you can interact with. The plugin for that might be useful for other projects, so I spun it off as a separate project, Qtonium. The name is inspired by Awesomium and Berkelium, two projects doing similar things.

As to why a new project and not to use Awesomium or Berkelium: Awesomium is apparently going closed-source, or at least that is the only way I can interpret their taking down the code and mentioning 'new licensing terms coming soon'. Berkelium is open source, but it is unclear if it is functional or useful outside of Sirikata, the project in which it is being developed and used.

And in any case, both are based on the approach of taking Chromium and hacking it to work in an embedded manner. In Qtonium I took a very different approach: No C++ at all, and no hacking of a massive and rapidly-changing codebase. Instead, PyQt is used in a straightforward manner, that is, Qtonium is written in Python. It's dependencies are Qt and PyQt.

The only downsides I can see to the Qtonium approach are:

Qt WebKit isn't a complete browser, just a 'widget'. For example, when embedding Chromium as a whole, you get plugins (Flash, etc.), caching, various default behaviors (handling page load errors, etc. etc.). That stuff is obviously useful, so this is a clear downside. But the cost (hacking Chromium) is also very high in my opinion. In any case, over time those things can be implemented in Qtonium if people want.
Qtonium doesn't share Chromium's separate process security model. This is true at the moment, but it would actually be very simple to adopt, due to Qtonium being written in Python, and Python 2.6's improved support for handing separate processes. In fact if you look at the code, how to do this becomes quite obvious.

Note that this is part of the approach of having all plugins be Python modules. Those modules can of course load C++ code if they need to (like Qtonium loads Qt through PyQt), so this isn't a limitation, but actually the hope is that a lot of things can be done with pure Python (including ctypes). That way, plugins don't need to be compiled, and it's generally very easy to make them work on multiple platforms. C++ plugin systems tend to be much more complex to work with.

Per-Client Variables

2009-11-25T07:09:00.001-08:00

By default state variables are synchronized between all clients: changes to data are reflected everywhere. This is simple and appropriate for most FPS games. However, some information should only be sent to some clients, like say the inventory in an RPG game. Library 1.3 now allows such state variables, which are called clientPrivate.

A clientPrivate state variable is only sent to the client whose avatar that is. In other words, clientPrivate variables on a player entity are only updated to that player - other clients will not see a value for that variable on that entity. (Note that clientPrivate has no effect on non-player entities.) Of course, the server sees all of the values (and when it updates one, the update is only sent to that player).

Here is an example of how to add such a variable to a logic entity class:

secretItem: new StateString({ clientPrivate: true }),

Library 1.3

2009-11-25T01:45:00.000-08:00

library/1_2 has been forked into library/1_3. The goal is to keep 1_2 as it is, so all the older games still work, while development moves on to 1_3 (which may include API changes, etc.). So, for new games, 1_3 should be used, especially if you will need new library features, as those will be added to library/1_3.

To migrate a map to 1_3, currently the following is necessary:

Remove the map asset dependency on library/1_2 and add one for library/1_3.
The map script should include the following line before any other Library.include commands:

Library.include('library/1_3/');
Imports of 1_2 libraries should be changed as follows:

Library.include('library/1_2/Projectiles');

should become

Library.include('library/' + Global.LIBRARY_VERSION + '/Projectiles');

(this way of doing things will help make future library migrations much easier).

After doing those, the map should work exactly as it did with library/1_2.

New Game: Racing

2009-11-19T09:18:00.000-08:00

A new game is ready, you can get to it from the lobby and play it right now. Here are some screenshots and videos:

As you can see, this is a racing game. We wanted to do something different than the current games (first/third person shooters, etc.), to show the flexibility of the engine, so this is that. Another difference is that this game can be played in both single player or multiplayer, so you can have fun even if people aren't online at the same time (but it's even more fun with them).

IRC Plugin + Integration

2009-11-17T03:38:00.000-08:00

A server plugin for IRC has been written, and is now used in the main servers - the lobby and the three games it has portals to.

This basically integrates text chat in the game servers with #syntensity on FreeNode, so that:

All text messages sent in the game servers are shown on #syntensity
Messages sent to the server bots (syntensity0 to syntensity3) in #syntensity will relay the message to people on that game server

This should be useful in several ways. Perhaps mainly in that it lets people on a game server, that have questions, be heard on IRC. So we can give technical support very quickly from there.

The implementation was fairly simple: A plugin in Python that integrates python-irclib. We can probably write similar plugins for other instant messaging systems, if it makes sense.

Thanks to TheAncientGoat for the idea.

1.1 Release

2009-11-15T00:54:00.000-08:00

Revision 1190 is tagged as version 1.1. Binaries for Windows and Ubuntu 9.10 are on the Syntensity Mod DB downloads page. I'll link to them from the main website's download page after a little more testing (if you test them yourself, please say how it went, here, on IRC, or in the forum - thanks!).

The new version has many changes, but most won't be noticeable until maps use the new features. The improvements include:

Enable maps to customize move, strafe and jump
Support for cutscenes
WoW-like mouse movement in Mouselook mode - cursor at borders will rotate the view
Allow client and server to (naively) share their home dirs, and do this by default when the server does not specify a home dir explicitly. This makes it much easier to set up a local server.
ZeroG module, allowing maps to have zero gravity, custom gravity effects, etc.
Vehicles module, to let entities move like simple vehicles (pitch based on floor, thrust for movement, etc.)
World.getSurfaceNormal, which allows doing bouncing physics entirely in scripts, + Grenade example
WorldSequences module - lets you track players progression through areas
WorldSignals module - lets you emit and listen for signals that are relevant to areas in space
New Steering module for bot movement control
Add absolute x,y to clientClick, allowing maps to have menus
Allow scripts on the server to intercept and handle text messages (e.g. can be used as a simple way to let people issue commands to the server, etc.)

Community Update

2009-11-12T01:43:00.000-08:00

Since we launched Syntensity a month and a half ago, we posted on some online forums to let people know about this project, but we have been very low-key about it. The reason is that we are trying to do things gradually: Both with building the technology, and with spreading the word. The idea is to get feedback from people about Syntensity, and then to improve it, after which it makes sense to show it to more people, get even more feedback, and so forth.

This approach is appropriate for us, with our open source development model. It is very different from what people do with closed source development, which is usually to build something behind closed doors for a long time, and then release it in a big public launch, at which point they hope to get a large amount of users as fast as possible.

With that background, I'm happy to report that Syntensity has now reached over 500 registered accounts. So, even though we've been low-key about spreading the word, things are heating up. Another sign of increasing activity can be seen on our forum. This is therefore an appropriate time to make a 'Community Update' post on this blog, to take a look at a few of the neat things people in the Syntensity community have been doing. From time to time I'll make more of these posts.

Psycho v1.0 by Psychosilocybin is an interesting work-in-progress map with an attacker-defender gameplay: Only one team has an object to protect. That team gets points for keeping it safe, the other team gets points for stealing it. Accordingly, the map has an asymmetrical structure, unlike typical CTF maps (that are usually very symmetrical). Here are some screenshots:

Realm Arena is a mod of Map2 by adrix89, which adds some movement capabilities, like double tapping to move in order to 'dash' in that direction ('dash' means to move quickly for a brief time). I think it makes for very interesting gameplay, with much more control of your character.

(Note that currently this requires that you edit your autoexec.cfg, as the map says to do. This will not be necessary with the upcoming release, which makes customizing movement easier.)

(The two things mentioned in this blog post are by no means a complete list - I hope I am not offending anybody by leaving them out. These are just two things that happened to recently impress me.)

4-Weapon CTF Update

2009-11-08T00:39:00.000-08:00

The 4-Weapon CTF game, also known as 'Map 2', has been updated to Map 2.1. When you enter that game from the lobby you will be playing the new version.

The changes are mainly bugfixes and minor improvements, in response to feedback we've been getting. They include:

Headshot notifications are now shown to everyone, not just the shooter (so you will know if a headshot was done to you, or to anyone else).
The sound for scoring a point now always plays when points are scored (before it was done unreliably).
Fixed an annoying bug where fragged characters could, in some cases, change their animation and not look fragged (even though they were).
Rockets now do twice as much damage as before (making it possible to frag people with a single direct hit).

More updates and new games are coming soon.

Cutscenes

2009-10-30T08:04:00.000-07:00

Cutscenes are now possible:

Cutscenes can be written in scripts, so you can have different ones in each map.

See packages/library/1_2/Cutscenes.js for the code and some examples. This is in /trunk, and will be in the upcoming 1.0.3 release.

Syntensity Celebrates Ubuntu 9.10

2009-10-29T01:51:00.000-07:00

If you read this blog then you probably already know that I am a big fan of open source, and of Linux in particular. I'm also an Ubuntu user, so today the Syntensity lobby world is lighting up with fireworks to celebrate the release of Ubuntu 9.10, Karmic Koala:

Log in to the lobby world to see it in person. (The fireworks effect, btw, like everything else is reusable for your own Syntensity worlds. See packages/base/welcome/KarmicKoala.js in your home directory for example code. Note that that file will only be there after you connect to the lobby world, at which point it will be automatically downloaded.)

I've been using Ubuntu 9.10 for a while now, and it was definitely worth the upgrade for me: It feels significantly faster, and also better supports my hardware. Thank you to the Ubuntu people, and all the open source projects that are included in Ubuntu (the Linux kernel, etc. etc.)!

Btw, speaking of Ubuntu 9.10, the nice people at Playdeb have packaged Syntensity for Ubuntu 9.04 and 9.10, 32- and 64-bit.

Easier to Run Your Own Server

2009-10-27T02:40:00.000-07:00

Some improvements to server setup have just been committed, which make it much easier to run your own server. The improvements are already available if you are running the latest source code, otherwise, they will be in the 1.0.3 release.

The improvements are:

Interactive setup wizard, which you can run by adding --wizard to the commandline when you run the server,

Linux: ./intensity_client.sh --wizard
Windows: intensity_client.bat --wizard

The wizard will ask some questions and create a settings.cfg for you.
You can now tell the server what activity to run in a much easier way. Simply paste the URL to the activity (which you can find on the activities page) into the settings.cfg file, specifically into [Activities] force_activity_id. The server will then autodiscover the activity and the map asset IDs for you. (You can still use an actual activity ID, for backwards compatibility.)

For more info, see the wiki page on running a local server.

New Lobby

2009-10-24T01:47:00.001-07:00

(click for a bigger version)

The default world is now the new lobby, shown above. It has portals to other games, and also an interactive tutorial.

You will enter the lobby if you do 'connect to selected...' (and you haven't selected another world), or if you do 'connect to lobby...' in the latest build (1.0.2, available on the download page).

Live changes and improvements to the lobby will be happening fairly frequently.

1.0.2 Release

2009-10-22T09:52:00.000-07:00

Revision 1082 is tagged as version 1.0.2. A new build for Windows and a .deb installer for Ubuntu 9.10 are on the download page.

This is a minor, recommended but non-mandatory update (which means you can continue to use the previous version - the network protocol has not changed, nor anything else that would be a problem for you). It includes the following:

'connect to lobby...' option in the main menu, for directly connecting to a lobby world (which has portals to other ones)
Interactive tutorial example code (that is, a map which interactively teaches you the controls)
Better debug traces from JavaScript
Various improvements and additions to library 1.2 (Roles/Classes, WorldNotices, etc.)
Support for CherryPy 3.1+
Support for gcc 4.4+
A few minor bugfixes

Portals and the Welcome World

2009-10-18T07:01:00.000-07:00

Support for jumping between worlds was added just before version 1.0. To make it useful, I wrote a 'Portal' class which is basically a screenshot of a world, that when you move into (that is, when your avatar walks into the portal), you are sent to that world. In the screenshot above you can see two such portals, which lead to map2 and sketchworld.

This will be used in the 'welcome world', the first world people encounter when they log into Syntensity. The welcome world will be a map with portals to the most important/popular maps, and will also have an interactive tutorial for learning the controls. The tutorial isn't written yet, but you can check out the portals by doing a manual connection to www.syntensity.com:10000 in the Syntensity client.

1.0.1

2009-10-14T08:06:00.000-07:00

Two weeks after 1.0, we are now releasing 1.0.1, which adds several new features:

New example content: A new, bigger map, and new gameplay elements including homing missiles, rockets, a shotgun, headshots with the sniper gun, nicer jumppads, etc. (It was kind of our goal to see how much new stuff we could add in a short time, as a test of the API.)
Script API support for displaying HUD text and images
Various minor bugfixes and polish

You can see some of this in our new video:

or better yet, try it out yourself. Note that this is an engine update, so you need to download the latest build,

http://www.syntensity.com/toplevel/download/

As always, if you haven't already signed up for Syntensity, you need to do so, which takes just a few seconds. See the links on the main page.

Notes:

If you are copying the new version into an existing directory with the old version, you will NOT get the new keybindings (see next note). It is simplest to just unpack the new download into a new directory. (Or, you can delete syntensity/client/config.cfg, which will then be created from scratch the next time the client is run, and you will get the new keybindings and so forth.)
Change weapons with 1-4, or cycle by pressing the middle mouse button. H shows a little help.
The homing missiles can only be shot if you are locked on a target (another player). The lock remains for a short while after your crosshair is no longer over the target (a notification appears at the top of the HUD).
This is an early release, we will probably adjust the weapon strengths, ranges, etc., depending on feedback (we can push those updates between releases). So, let us know what you think needs changing.

P.S. I was very impressed by OpenShot, the open source video editor I used to make the video. Kudos to the OpenShot team for making a very useful program.

New Weapons

2009-10-11T02:35:00.000-07:00

The second game is almost finished, and it will include several new weapons, a new map, as well as various other features, many of them in response to the feedback we received so far.

You can see the current state of the new game if you run the latest from the source code repo (do a manual connect to www.syntensity.com:10002). Otherwise, binary builds will be available soon. Meanwhile here is a screenshot of some rocket fire:

Updates and Updating

2009-10-05T05:30:00.000-07:00

We have a new Twitter account that we are starting to use for real-time updates, things like

New features notifications as they go live
Server status reports

Speaking of new features, I might as well take this opportunity to explain a little about how that works in Syntensity (and of course the Intensity Engine): We are able to add new features between releases, without people needing to download a new version of the client, since

The master server is a website, so adding things like the new map creation wizard is possible just like updating any website.
Game content is downloaded automatically using our asset system, which will get whatever content you need. And as game logic is written in JavaScript, this includes entire games, not just textures, models, etc. (but also those, of course).

Aside from the map creation wizard already mentioned, we have been working on adding new weapons, weapon recoil, and fixing some bugs with animation speeds and other minor things. Note that the default map you enter is still the original version of the gk1 map, so you won't see these things unless you specifically connect to a server running gk1_a.

Server Requisitioning Issues / Logs

2009-10-03T01:12:00.000-07:00

People are already starting to requisition server instances, and sometimes there are problems, like if an asset or a script isn't set up correctly. That will cause the server instance to shut down, and an error log is uploaded to the master server - but we don't yet have a convenient interface for non-admin users to view it. So we are seeing the error logs arrive, but sadly they don't get to you yet.

We are working on improving this. In the meantime, if you requisition a server and don't see it start up, this might be the reason. You can do the following:

Ask an admin for your server logs, on IRC (#syntensity on FreeNode).
Run a local server instead of requisitioning a remote one. You will be able to see exactly what output it gives, which should help fix any problems with assets or scripts.

We apologize for this inconvenience. That's the nature of the early stages of an open beta - problems are found, and fixed. Bear with us :)

Edit: This issue has now been mostly resolved. You can view the last server log on your account page (you must be logged in), by clicking on "View last error log from requisitioned server." So, if you requisition a server and it doesn't appear in the list of servers, check the server log there for what caused the problem. A comment about this has also been added beside the 'requisition' button. We are evaluating ways to make this even easier.

Launch: Syntensity Open Beta, Intensity Engine 1.0

2009-10-02T10:32:00.000-07:00

Today we are officially launching Syntensity's open beta, as well as releasing version 1.0 of the Intensity Engine. This brings to a close an intensive year of development, but is also just the beginning for this project.

Instructions for getting started with Syntensity can be found on the main page.

Syntensity's open beta launches with one complete game (the gk1 map, which is an insta ctf-type game that also has some automatic gun turrets). But Syntensity is really more of a platform for game creation, not one or two games that we created ourselves. This initial game is just to show what the engine can do, and also to be a basis people can work from (by modding gk1). We'll be making some more example games during the beta (to show more of the engine's capabilities - gk1 covers only a small part of that), but our main focus will be to help people use the engine to create the games they want.

Regarding the Intensity Engine (the open source project that comprises Syntensity's code), it is now at 1.0. It's basically feature-complete for our initial goals, and decently stable - enough for people to start using it, both for Syntensity and other projects. There are some bugs we are aware of, and testing will probably uncover more, but overall things are in fairly good shape. In the near future we will focus on fixing bugs and polish, later on, there are a lot of engine features on our roadmap.

Automatic Testing: Client and Server

2009-10-01T10:33:00.000-07:00

(This will be a mostly technical post.)

There is a lot of work planned for the engine after the launch of the open beta (which is, btw, just about here), so automatic testing is crucial to prevent regressions. Basically, every time you commit a change, you want to be able to run a completely automatic battery of tests that checks you didn't break anything. So far in the Intensity Engine only the master server had automatic tests for it, using Django's test client. Those tests are actually very comprehensive, and have already come in handy. There are also some unit tests for the JavaScript API. But for the client and server as a whole, writing automatic tests isn't as easy: They don't have nice test setups like Django, and what's more, the client is a GUI program, using OpenGL (if it were using a GUI like GTK or Qt, there would actually be some tools to help out).

The tests/ directory contains what I did so far towards this goal, over the last few days. The test setup uses pexpect, a very useful Python module that lets you communicate with processes, sending them input and checking their output. This, combined with the server's Python console, lets the server be tested in a nice fashion: Start the server in a separate process, issue it commands - for example, check such and such state variable of such and such logic entity - and validate the output. Using Python's unittest module, in each test the server environment (home directory, with downloaded assets and so forth) is created from scratch. So all of this together gives an appropriate way to test the server.

The client was a little harder. First off, I added an option to run a Python console, just like on the server (which took some refactoring of the console code). Then, I had to write bindings for 'injecting' user events. That is, by issuing commands through the Python console, the test runner can manipulate the client as if it were an actual person: Move the mouse, click, press a key, and so forth. This took some time to get working correctly, due to how SDL events behave and how the engine processes them. But it appears to now be working as it should.

Currently there are 4 tests. Each of them starts up a master, a server, and a client, runs a map ('storming') and then does one of the following:

Modify a state variable on the client and see that it propagates to the server
Modify a state variable on the server and see that it propagates to the client
Modify a state variable, restart the map, and see that it returned to the original value
Modify a state variable, upload the map, and see that the new value is used

In other words, these are high-level functional tests, and they already cover most of the basics: Starting up the client and server, loading a map in each, network synchronization, and map uploading. In particular, they cover basically what someone starting out with the Intensity Engine would do (if they follow README-standalone.txt). So these tests are a good start, while of course many more should be added over time.

The tests take about a minute to run, what with creating an entirely new environment for the master, server and client for each test, and starting them all. (Also, for some reason the injected SDL events are slower than I would expect.) But it's actually kind of funny to watch the tests running, with the mouse jumping around, clicking and keypress sounds, and so forth. (Although I imagine I will get tired of it soon enough, and just go do something else while the tests run...)

Release Candidate

2009-09-27T03:18:00.000-07:00

I'm happy to announce that there is a release candidate :)

The goal is to check that the download binaries work (that is, to check if I didn't forget some DLL, asset file, etc.), and also to check for any major bugs that I missed.

To try the release candidate, do the following:

Download it:

Windows: http://download.syntensity.com/windows_1.0.zip
Ubuntu 9.04: http://download.syntensity.com/ubuntu9.04_1.0.tar.gz

For Ubuntu 9.04 you also need to get some deps, with a simple

sudo apt-get install libsdl1.2debian libsdl-image1.2 libsdl-mixer1.2 python libboost-python1.35.0 zlib1g

Note that the Ubuntu 9.04 version probably will NOT work on other Ubuntu versions (except *K*ubuntu 9.04, etc.) or other Linuxes - you would need to compile from source so it links correctly against your distro's libraries. Compiling from source is actually very easy on Linux, see instructions on http://www.syntensity.com/toplevel/intensityengine/ and of course feel free to ask for help (here or on IRC, #intensityengine or #syntensity on FreeNode).
Unpack the download and run intensity_client.bat (Windows) or intensity_client.sh (Linux).
You also need to sign up for a user account, for the release candidate you should do that here:

http://www.syntensity.com:8888/accounts/releasecandidate/register/
Aside from that, instructions for how to do stuff (log in, join a game, etc.) are on the wiki, in particular you should read

http://wiki.syntensity.com/introduction

Note that I am announcing the release candidate here, and not on the main website, because I don't want too many people to try it (since there might be a problem with the binaries) - I'm just looking for a small amount of feedback on the release candidate for now. Then if all is well with the release candidate, the release itself can be in a few days. So, until the release, please don't tell anybody about the release candidate, I'd rather their first impression be of the actual release (which will have an announcement on the main website, new video and screenshots, etc. etc.).

Thanks in advance for testing the release candidate!

Licensing

2009-09-22T09:51:00.000-07:00

In this post I'll clarify some things regarding licensing.

First of all, there are two separate topics here: Licensing for the Intensity Engine and for Syntensity. The Intensity Engine is an open source project, while Syntensity is one particular project built using the Intensity Engine (that happens to be run by the same team).

Regarding the Intensity Engine, it uses the AGPL license. I recently elaborated a little more on the website about this, mainly that game content is not considered a derivative work. That is, if you write a game using the API, the game does not need to be AGPL licensed. It's yours and you can license it however you want. Only if you modify the engine itself, as opposed to creating a game using it, does the AGPL license become an issue. It's pretty much how Blender licensing works - make something with Blender and license it however you want; modify Blender itself, and you need to be GPL licensed.

Regarding Syntensity, my original idea was to require that all content be under a creative commons license or something similar, since I see Syntensity as closely related to the participatory culture movement (but in the context of gaming). By requiring such licensing, all the content on Syntensity would be free for use, in Syntensity and elsewhere. But it turned out that wasn't workable - not everyone is willing to use the same license (some artists insist on -NC licenses, for example). So by picking any one license, it would leave a lot of people out. And by picking several incompatible ones, there would be a lot of confusion. So in the end Syntensity will use the most minimalistic and generic licensing for user-created content: Whatever you create in Syntensity is yours, to do with as you want, but you grant everyone a license to use it freely within Syntensity. So how you let your content be used outside of Syntensity is entirely up to you (and you can completely disallow such use), while inside Syntensity we uphold the idea of everyone working together to create more and better games.

(If granting a license to people to use your content within Syntensity is a problem, then don't upload that content into Syntensity - use the Intensity Engine by yourself to run your content however you want, and with whatever license you want. It's open source, and that's what it's for.)

Intensity Engine 0.9.8

2009-09-22T05:57:00.000-07:00

I just tagged version 0.9.8 of the engine, which might end up as our release candidate. Changes include:

An apparent fix for the big, worrying bug
Smoother camera movement (even in mouselook mode)
Flag model (4 colors)
Some optional, for-future-use API features (action keys, server teleport)
Various tool improvements (texture converter & packager, mass texture replacer, etc.)
Various bug fixes

Intensity Engine API Documentation

2009-09-17T02:43:00.000-07:00

A first version of documentation for the scripting API - the API you write games/activities in, also known as "the API" ;) - has been uploaded to docs/. You can see it here. It has a 'hello world' example, a tutorial, and a few big examples of actual working code (a plugin, an activity, etc.) with detailed explanations. This should be enough for people to get started with scripting.

We don't expect to have complete documentation in time for the launch, but with this document and the work-in-progress wiki, we should have enough for now. (The source code itself also has comments, particularly in the important places, and the example games and activities are helpful as well.)

EDIT: We moved to github meanwhile, the latest docs are here: http://github.com/kripken/intensityengine/blob/master/docs/scripting_docs.html