Emscripten 1.5!

Version 1.5 of Emscripten, the LLVM to JavaScript compiler, is out. Lots of new stuff:

• A Text-to-Speech demo using eSpeak. Not much had to be done to get this to work, a few library functions were missing but that is pretty much it. I did need to bundle getopt and strtok C sources in the project though. Also, I had to use typed arrays type 2, since the eSpeak source code is not as platform independent as we would like (so this ended up being a good test of typed arrays 2 actually). For more details, source code etc., see the demo page.
  
• max99x has written a nice Filesystem API. See that link for documentation. It makes the emulated filesystem much more flexible and useful. The text-to-speech demo uses it, as do all the automatic tests. Aside from the API itself, this update comes with a ton of library additions for IO related things.
  
• max99x also wrote parsing code to detect field names in LLVM metadata. This lets you use the original C/C++ field names in your JavaScript, so integrating compiled code and JavaScript becomes much easier. I am thinking about extending this for use in the bindings generator.
• Speaking of the bindings generator, it has seen a lot of work and things are finally starting to run with Bullet, at least a 'hello world' of creating a btVector3. There is still some work ahead before it is finished, not sure how much.

Emscripten 1.4!

Version 1.4 of Emscripten, the open source LLVM to JavaScript compiler, has been released.

Some significant improvements this time, including

• Support for compiling and loading dynamic libraries, thanks to max99x for writing this very useful (and not easy to write!) feature. You can now compile a module as a shared library, and load it from your main compiled script just like you would load a normal shared library in native code, using dlopen() and so forth. This can potentially be very useful, both in not needing to rewrite code that is already split up into modules, and also in that it lets you load the main module quickly since other stuff is split out into other files, which can be loaded later on demand. I hope to see a demo of this up soon.
  
  
• Automatic bindings generation. Until now, you could compile a C or C++ library and run it on the web, but using it from normal JavaScript was clunky. Thankfully bretthart pointed me to CppHeaderParser, a pure Python C++ header parser, which Emscripten can now use to generate bindings (for more details on the header parser, see here). The result is a set of JavaScript objects that wrap the compiled C++ code, so you can write quite natural JavaScript code to access them, for example, var inst = new CppClass() to create an instance, inst.doSomething() to call a function, etc. A lot of basic stuff already works (see part 2 of test_scriptaclass), I am currently investigating the use of this with Bullet in ammo.js, hopefully I will succeed there and have a more detailed blogpost afterwards.
  
  
• Library stuff, lots of fixes and additions there, thanks to max99x and timdawborn.
  

Emscripten 1.3!

Version 1.3 of Emscripten, the open source LLVM to JavaScript compiler, has been released.

No new demo this time, sorry. However the Python demo has been updated to improve performance and enable raw_input to work (it prompts for input using window.prompt). Press 'execute' in the demo to see it work.

Main updates:

• Support for a new usage of typed arrays, TA2. In TA2, a single shared buffer is used, and int8, int32 etc. are all accessed through views into that buffer. The main benefit here is memory usage - this mode takes much less memory than TA1 (the original typed array usage), and in most cases probably less than the non-typed array case.
  
  In theory, this can also be faster. However that doesn't appear to be the case in my benchmarks, due to the need to divide pointers by 2 or 4 constantly (pointers are raw addresses, while indexes into typed arrays take into account the size of the element. int32vec[1] is at address 4!), and since JS engines still do not heavily optimize typed arrays.
  
  One thing you can do, though, is use dangerous nonportable LLVM optimizations with TA2. TA2 lets you write an int and read the first character, and get the 'right' result. Of course 'right' will depend on the endianness, so this is very dangerous and not recommended. However you can compile two versions, one for each endianness. This can potentially be faster.
  
  
• Some relooper optimizations were done, which gave us a nice speed improvement. I'll probably do a full blogpost on performance issues, but to briefly summarize, we seem to be getting close to the speed of handwritten JS code, which is to say, as fast as we can probably get. In absolute terms, compared to gcc -O3 (the fastest native code), we are around 5X slower (on the latest development versions of SpiderMonkey and V8). But there is a big spread: In raw numeric processing we are often just 2-3X slower, which is about the same as Scala, Haskell, and Mono, but certain other operations are costlier and in some benchmarks we are up to 10X slower.

Other news:

• Still hoping for people to help out with OpenGL/WebGL stuff. Please step up! I don't know what to do there myself.
  
• Next main project for me personally will probably be better tools to integrate compiled code with normal JS code. One option is to use SWIG to generate bindings. We could then compile a C++ library and use it in a natural way on the web, which would be very cool. If you know SWIG, or don't but want to see this happen (like me ;) then please get in touch.
  
• I had some discussions with people interested in compiling certain large projects to the web, for example Second Life and Mono. Both have significant technical difficulties (rendering and networking for Second Life, the non-existence of an interpreter and the limitations of mono-llvm in Mono), but if the people interested in each are serious enough to do the work to overcome the respective difficulties, I have promised to do the raw C++ to JS conversion for each of those projects. Hopefully cool things will happen here.
  

Emscripten 1.2, Doom on the Web

Emscripten, the LLVM to JavaScript compiler, is now at version 1.2. The main updates in this release were to enable this demo of Doom on the Web - a playable version of the classic game Doom, compiled from C to JavaScript and rendering using Canvas.

The demo is known to work on Firefox and Safari. It works, but slowly, on Opera. I can't get it to run properly in Chrome due to a problem with V8. I have no idea if it runs on IE9, since I don't have a Windows machine, but since IE9 has a fast JS engine and supports canvas, it should (please let me know if you try it there). Edit: Here's a screencast of the demo running on Firefox Nightly if you can't run it yourself.

Highlights of Emscripten 1.2:

• Many improvements to Emscripten's implementation of the SDL API in JavaScript, including support for color palettes (Doom uses a 256-color palette), input events (we translate normal web keyboard events into their SDL forms), and audio (for now, just using the Mozilla Audio Data API - it's the most straightforward API at this point. Patches are welcome for other ones).
• Many improvements to the CHECK_* and CORRECT_* options, which are very important for generating optimized code using Emscripten. In particular, there is a new AUTO_OPTIMIZE option which will output a summary of which checks ran how any times, and how many of those checks failed, giving you a picture of which lines are important to be optimized, and which can be.
• Some additional experimental work is ongoing about supporting OpenGL in WebGL. I don't know either OpenGL or WebGL very well, I'm learning as I go, and I'm not sure how feasible this project is. If you can help here, please do!
• Various bug fixes. Thanks to all the people that submitted bug reports. In addition compiling Doom uncovered a few small bugs, for example we were not doing bit shifts on 64-bit integers properly.

Emscripten 1.1!

Emscripten is an LLVM to JavaScript compiler, allowing you to run code written in C or C++ on the web. I released version 1.1 today, with the following updates:

• A much improved Bullet demo - check it out! This version is much faster. The main differences are use of memory compression (see below), LLVM optimizations, and CubicVR.js for rendering.
  
• QUANTUM_SIZE == 1, a.k.a memory compression. This is an advanced, and somewhat risky, optimization technique. I see speedups of around 25%, but take note, this must be used carefully. See the docs.
• Dead function elimination tool: A Python script that scrubs an .ll file to remove unneeded functions. This is useful to reduce the size of the generated code and speed up compilation. Note though that if you want to compile a library, then this tool will remove functions that you probably want left in - it removes everything that cannot be reached by main(). The test runner now uses this by default.
• Various performance improvements and bug fixes.

Emscripten 1.0!

It's been almost a year since I started Emscripten (which, if you haven't heard of it, is a tool to compile LLVM to JavaScript), during which it took up much of my spare time. So I am very pleased to announce that today Emscripten has reached the 1.0 milestone. This release comes with a demo of rendering PDFs on the web (warning: that page downloads >12MB, since it includes Poppler and FreeType. It's like downloading an entire desktop app, almost).

Other highlights in this release:

• Very significant optimization of memory use in the compiler. This was necessary for the PDF demo to build, since it is far larger than previous demos.
• Full support for the recently released LLVM 2.9.
• The Emscripten documentation paper is finished. It explains how Emscripten works, so you might be interested in it if you care what Emscripten does under the hood (but if you just want to use Emscripten you don't need to read it).

Overall Emscripten is now in very good shape. It can probably compile most any C/C++ project out there, subject to some limitations (like JS not allowing C-style multithreading). At times some manual intervention is needed, like changing the project's settings so it doesn't generate inline assembly, and of course bugs probably still exist, but recently the code I have compiled has tended to just work (hence the rate of commits has greatly decreased recently).

The speed of the generated code can be quite good. By default Emscripten compiles with very conservative settings, so the code will be slow, but optimizing the code is not that hard to do. Optimized code tends to run around 10x slower than gcc -O3, which is obviously not great, but on the other hand fairly decent and more than good enough for many purposes. And of course, that ratio will improve along with advancements in JavaScript engines, LLVM, and the Closure Compiler.

So, Emscripten 1.0 is in my opinion pretty solid. There are no major outstanding bugs, and no major missing features. (But I do have plans for some major improvements, which are difficult, but should end up with code that runs at least twice as fast.) Now that Emscripten is at 1.0, I am hoping to see it used in more places. I'm starting to propose at Mozilla that we use it in various ways, and also I'd love to see things like GTK or Qt ported to the web - if anyone wants to collaborate on that, let me know.

Emscripten 0.8!

The main highlights of this release are:

• Tests for FreeType and zlib, two important real-world codebases. Aside from all the fixes and improvements necessary to get them to work, the test infrastructure now runs the entire build procedure (using emmaken) for those two tests, giving even more complete test coverage.
• File emulation. Just enough to let compiled C/C++ code think it is accessing a filesystem. For example, the FreeType test loads a TrueType font from a file (but really it's a virtual filesystem, set up in JavaScript).
  
• Additional compilation options for overflows and signedness (CHECK_OVERFLOWS, CORRECT_OVERFLOWS, CHECK_SIGNS). These allow even more C/C++ code to be compiled and run properly, but are switchable, so code that doesn't need them can run fast.
  

I'll post a web demo soon.

Emscripten 0.7!

Main changes in this release:

• Lots of minor fixes and additions, in order to get CPython working. As a result there is now a web demo of Python, which seems to work quite well aside for being very slow in Chrome.
• Figuring out what to do with LLVM optimizations. It looks like all of them generate suitable code except for -instcombine, which apparently combines instructions in a CPU-specific way (so, it isn't portable, and confuses Emscripten). All the tests now pass with LLVM optimizations enabled (all but the problematic one just mentioned).

So, not much in the way of new features: As mentioned before, we are already pretty much feature complete at this point.

Emscripten 0.4!

The focus of this release was on making the generated code faster. As the chart shows, we went from being 100X slower than hand-optimized JavaScript code, to around 5X slower. You can see the difference in the raytrace demo, which has been updated to use all the current optimizations.

5X slower is still slower. It will be hard to do much better, though, without either better JS engine support, or much more clever code analysis. Both will hopefully happen over time. Meanwhile, 5X slower is not too terrible, and there are some advantages over hand-written code - we hardly use garbage collection, so no GC pauses. Also, the speed really depends on the code - the comparison in the chart above uses benchmarks for which we have comparable code in both C++ and JavaScript. But the most interesting uses of Emscripten are to convert code for which we don't have a JavaScript equivalent. Also worth mentioning is that it is perfectly possible to hand-optimize the crucial parts of the code that Emscripten generates.

Some technical details about the optimizations implemented in this release:

• Use typed arrays, if available in the JS engine (thanks to pcwalton and njn for the idea)
  
• Optimize after the relooper runs, removing unneeded code flow overhead
• Nativize many more variables than before (i.e., move them off the emulated stack, and into native JS variables)
• Optimized stack emulation
• Inlining of various runtime code fragments
• Integration with the Closure Compiler: We generate output that it is very good at optimizing (thanks to Anders Riggelsen for the idea)
  

Also added in this release is support for the brand-new LLVM 2.8. That is now the version being tested against.

1.1 Release

Revision 1190 is tagged as version 1.1. Binaries for Windows and Ubuntu 9.10 are on the Syntensity Mod DB downloads page. I'll link to them from the main website's download page after a little more testing (if you test them yourself, please say how it went, here, on IRC, or in the forum - thanks!).

The new version has many changes, but most won't be noticeable until maps use the new features. The improvements include:

• Enable maps to customize move, strafe and jump
• Support for cutscenes
• WoW-like mouse movement in Mouselook mode - cursor at borders will rotate the view
• Allow client and server to (naively) share their home dirs, and do this by default when the server does not specify a home dir explicitly. This makes it much easier to set up a local server.
• ZeroG module, allowing maps to have zero gravity, custom gravity effects, etc.
• Vehicles module, to let entities move like simple vehicles (pitch based on floor, thrust for movement, etc.)
• World.getSurfaceNormal, which allows doing bouncing physics entirely in scripts, + Grenade example
• WorldSequences module - lets you track players progression through areas
• WorldSignals module - lets you emit and listen for signals that are relevant to areas in space
• New Steering module for bot movement control
• Add absolute x,y to clientClick, allowing maps to have menus
• Allow scripts on the server to intercept and handle text messages (e.g. can be used as a simple way to let people issue commands to the server, etc.)

1.0.2 Release

Revision 1082 is tagged as version 1.0.2. A new build for Windows and a .deb installer for Ubuntu 9.10 are on the download page.

This is a minor, recommended but non-mandatory update (which means you can continue to use the previous version - the network protocol has not changed, nor anything else that would be a problem for you). It includes the following:

• 'connect to lobby...' option in the main menu, for directly connecting to a lobby world (which has portals to other ones)
• Interactive tutorial example code (that is, a map which interactively teaches you the controls)
• Better debug traces from JavaScript
• Various improvements and additions to library 1.2 (Roles/Classes, WorldNotices, etc.)
• Support for CherryPy 3.1+
• Support for gcc 4.4+
• A few minor bugfixes

1.0.1

Two weeks after 1.0, we are now releasing 1.0.1, which adds several new features:

• New example content: A new, bigger map, and new gameplay elements including homing missiles, rockets, a shotgun, headshots with the sniper gun, nicer jumppads, etc. (It was kind of our goal to see how much new stuff we could add in a short time, as a test of the API.)
  
• Script API support for displaying HUD text and images
• Various minor bugfixes and polish
  

You can see some of this in our new video:



or better yet, try it out yourself. Note that this is an engine update, so you need to download the latest build,

http://www.syntensity.com/toplevel/download/

As always, if you haven't already signed up for Syntensity, you need to do so, which takes just a few seconds. See the links on the main page.

Notes:

• If you are copying the new version into an existing directory with the old version, you will NOT get the new keybindings (see next note). It is simplest to just unpack the new download into a new directory. (Or, you can delete syntensity/client/config.cfg, which will then be created from scratch the next time the client is run, and you will get the new keybindings and so forth.)
• Change weapons with 1-4, or cycle by pressing the middle mouse button. H shows a little help.
• The homing missiles can only be shot if you are locked on a target (another player). The lock remains for a short while after your crosshair is no longer over the target (a notification appears at the top of the HUD).
• This is an early release, we will probably adjust the weapon strengths, ranges, etc., depending on feedback (we can push those updates between releases). So, let us know what you think needs changing.
  

P.S. I was very impressed by OpenShot, the open source video editor I used to make the video. Kudos to the OpenShot team for making a very useful program.

Launch: Syntensity Open Beta, Intensity Engine 1.0

Today we are officially launching Syntensity's open beta, as well as releasing version 1.0 of the Intensity Engine. This brings to a close an intensive year of development, but is also just the beginning for this project.

Instructions for getting started with Syntensity can be found on the main page.

Syntensity's open beta launches with one complete game (the gk1 map, which is an insta ctf-type game that also has some automatic gun turrets). But Syntensity is really more of a platform for game creation, not one or two games that we created ourselves. This initial game is just to show what the engine can do, and also to be a basis people can work from (by modding gk1). We'll be making some more example games during the beta (to show more of the engine's capabilities - gk1 covers only a small part of that), but our main focus will be to help people use the engine to create the games they want.

Regarding the Intensity Engine (the open source project that comprises Syntensity's code), it is now at 1.0. It's basically feature-complete for our initial goals, and decently stable - enough for people to start using it, both for Syntensity and other projects. There are some bugs we are aware of, and testing will probably uncover more, but overall things are in fairly good shape. In the near future we will focus on fixing bugs and polish, later on, there are a lot of engine features on our roadmap.

Release Candidate

I'm happy to announce that there is a release candidate :)

The goal is to check that the download binaries work (that is, to check if I didn't forget some DLL, asset file, etc.), and also to check for any major bugs that I missed.

To try the release candidate, do the following:

1. Download it:
   
   Windows: http://download.syntensity.com/windows_1.0.zip
   Ubuntu 9.04: http://download.syntensity.com/ubuntu9.04_1.0.tar.gz
   
   For Ubuntu 9.04 you also need to get some deps, with a simple
   
   sudo apt-get install libsdl1.2debian libsdl-image1.2 libsdl-mixer1.2 python libboost-python1.35.0 zlib1g
   
   Note that the Ubuntu 9.04 version probably will NOT work on other Ubuntu versions (except *K*ubuntu 9.04, etc.) or other Linuxes - you would need to compile from source so it links correctly against your distro's libraries. Compiling from source is actually very easy on Linux, see instructions on http://www.syntensity.com/toplevel/intensityengine/ and of course feel free to ask for help (here or on IRC, #intensityengine or #syntensity on FreeNode).
   
   
2. Unpack the download and run intensity_client.bat (Windows) or intensity_client.sh (Linux).
   
   
3. You also need to sign up for a user account, for the release candidate you should do that here:
   
   http://www.syntensity.com:8888/accounts/releasecandidate/register/
   
   
4. Aside from that, instructions for how to do stuff (log in, join a game, etc.) are on the wiki, in particular you should read
   
   http://wiki.syntensity.com/introduction
   

Note that I am announcing the release candidate here, and not on the main website, because I don't want too many people to try it (since there might be a problem with the binaries) - I'm just looking for a small amount of feedback on the release candidate for now. Then if all is well with the release candidate, the release itself can be in a few days. So, until the release, please don't tell anybody about the release candidate, I'd rather their first impression be of the actual release (which will have an announcement on the main website, new video and screenshots, etc. etc.).

Thanks in advance for testing the release candidate!

Intensity Engine 0.9.5 Release

More screenshots can be seen here.

Since 0.9, a month and a half ago, a lot of work has gone into the 0.9.5 release which is out today (rev. 324). This release is still not considered stable, but is a significant step towards that (in fact it may be the last release before 1.0). Two main areas of focus in this release are:

• Allowing people to run their own infrastructure - master server, asset servers, and server instances - without any connection to Syntensity. This allows both federation (separate independent servers, with loose connections - like the WWW) and makes it easier to customize the Intensity Engine for non-standard uses.
  
• Improving content creation and collaboration. This is now in a good enough state to allow focusing on content creation, which is one of the main tasks remaining before the Intensity Engine 1.0 release and public launch of Syntensity.

In more detail, here are some of the changes:

• 'clientSet' state variables, which are applied first on the client, allowing better responsiveness in an easy way
  
• Initial login to server instances made faster
• AreaTriggers made significantly faster, and now work correctly on server
• Allow maps to extend the position protocol messages, for faster updates of map-relevant information like custom animation settings
  
• Allow rendering models (including players) from map scripts, for more flexibility in visuals
• Added almost-finished Stromar character model
• Update of server-side NPC/bot system to current API
• Various API extensions (e.g., allowing gravity to be changed)
  
• Various minor GUI and usability improvements
• Full support for building on Ubuntu 8.10, Ubuntu 9.04, Windows XP and Windows Vista
• Plenty of bug fixes
  

Binary builds aren't planned for this release, as things are still moving quite fast.


Towards 1.0

The Intensity Engine is now in feature freeze. Only bugs in existing features will be fixed for the 1.0 release, which is dependent upon

• a decent level of stability and polish, and
• a reasonable amount of working content that can be distributed with the engine or at least used to demo it