Friday, September 14, 2012

Monkey stuff!

Ok, this was initially going to be a forum post, but it got a bit longwinded so it's probably worthy of a post in my poor, neglected blog.

On the Monkey front, I've pretty much got async loading of images/sounds going now - usage looks something like this:

Class MyApp Extends App Implements IOnLoadImageComplete

Field _image:Image

Method OnCreate()
LoadImageAsync "cheerio.jpg",,,Self
SetUpdateRate 60

Method OnUpdate()
UpdateAsyncEvents 'must do this!

Method OnRender()
If _image DrawImage _image,MouseX,MouseY

Method OnLoadImageComplete:Void( image:Image,path:String,source:IAsyncEventSource )
If image
'image loaded OK!
'problem loading image!
Error ""

This works pretty much identically on all targets, except for PSM which can't do async image loading, even with a thread - it complains when you try to use the graphics context from a background thread. In this case, it's done synchronously so still at least works.

On c++/java targets, the load executes on a separate thread. On html5/flash, 'onloaded' callbacks are used internally to 'fake' threading. Note that on targets with embedded data (xna/flash) this is really just decompressing data on a separate 'thread', so may not be all that useful. Also, on OpenGL targets textures must be created on the main thread so there may still be a 'pause' after the image has loaded in the background when it gets sent to GL. This is theoretically fixable, although my experience with multi-thread GL contexts in bmx was less than encouraging...

In addition, image/sound loaders can now load stuff outside the data directory where possible. File paths are now more URL-ish, so (on some targets) you can do things like:

Local image:=LoadImage( "" )

This is down to the capabilities of the underlying native loaders, so for example http: currently only works on html5/android/ios. On Flash, it works if the image comes from the same domain, otherwise you need to configure the resource server - something Monkey can't help with. In the case of c++ targets, this can be used to load stuff from the filesystem but not http: (yet).

This has meant some tweaks to the file path system, as the path "cheerio.jpg" can now mean several things: It could be a file in the app's 'data/' dir, or it could be a file in the (c++/java) current dir or the (html/flash) document base.

To deal with this, I've added the 'pseudo' URI monkey: which works a bit like http: and can be used to locate monkey specific resources. Currently, there's only monkey://data/ to get at resources in the app's data dir.

To maintain compatiblity, Mojo will for now convert all relative paths into monkey://data/ paths. It will not convert paths that start with "./", "/" or "blah:", so to load something from the 'real' current dir you will need to use the "./" prefix.

This means these are equivalent:

Local image:=LoadImage( "cheerio.jpg" )
Local image:=LoadImage( "monkey://data/cheerio.jpg" )

But these are not:

Local image:=LoadImage( "cheerio.jpg" ) 'gets converted to "monkey://data/cheerio.jpg"
Local image:=LoadImage( "./cheerio.jpg" ) 'does not get converted

Whew! Sounds complex, but I think it's a reasonably sensible approach.

The new Mojo async loaders are built on a new interal async event system in the brl module, brl.asyncevent. The core of this system is the IAsyncEventSource interface that provides a single method...

Interface IAsyncEventSource
Method UpdateAsyncEvents:Void()

...and a bunch of global functions...

Function UpdateAsyncEvents:Void()
Function AddAsyncEventSource:Void( source:IAsyncEventSource )
Function RemoveAsyncEventSource:Void( source:IAsyncEventSource )

UpdateAsyncEvents() needs to be called regularly - once per OnUpdate() should be enough - and is what actually updates/polls the state of any async operations currently in progress. This is when various OnBlahComplete methods will be called.

You can add your own async event sources by writing a class that implements IAsyncEventSource and using AddAsyncEventSource/RemoveAsyncEventSource to add/remove objects of this class to the global list of async event sources.

Note that there is no 'event' class as such - each async event source must provide it's own mechanism for notifying the main app of progress/completion, eg: by providing an IOnBlahComplete interface.

There's also a new AsyncStream class that uses the async event system to let you read/write from some types of stream in the background. Currently, only async TCP streams are supported, but future versions could include include AsyncFileStream (once we HAVE a file stream!) for streaming from 'slow' filesystems such as Dvd, and even AsyncAudioStream for realtime mixing.

I've also made several reasonably major changes to the config var system.

First up, you can now use '+=' to append to config vars (you still can't overwrite them with '=' though). This will append the RHS to the var, inserting a ';' if necessary to keep 'em separated. This is very much a WIP feature right now and mainly for the sake of REFLECTION_FILTER, although MODPATH and data file filters might also work. Also, both ';' and '|' are now valid separators for REFLECTION_FILTER.

The reflection module no longer does a #REFLECTION_FILTER="*" by default, so it's up to you to set the correct filter(s) or it will be empty. Being able to append to config vars now means modules can add themselves to the filter with eg: #REFLECTION_FILTER+="mymodule*|mojo*" etc.

CONFIG.TXT files are now plain Monkey files, although they're only ever preprocessed. I've renamed them CONGIG.MONKEY to try and limit any confusion here. Note that this means each line of CONFIG.MONKEY must now start with a '#', just like real Monkey code.

The handling of bool true/false config vars has been changed/unified - you should now use the Monkey tokens True or False (no quotes, case insensitive - ie: Monkey code) instead of "true" or "false" for all bool vars. There's a minor issue with bool->string conversion here, since all config vars are actually stored as strings and Monkey doesn't support bool->string conversion. To deal with this, I've hacked the preprocessor so true gets converted to "1" and false gets converted to "0". This will only be an issue for those writing code that uses config vars internally, but it's something to be aware of. It's meant that I had to change any existing code that compared config vars with "true" to compare with "1" instead.

This all represents some potentially 'breaking' changes for anyone who's mainting their own version of trans - apologies for that, and perhaps I should have saved these for a future release as there are more breaking changes coming...

As for the future of Monkey, here's the current plan...

A Windows 8 C++/DirectX target is underway and will hopefully be done 'soon'. I have gone with C++/DirectX because it seems to be the lowest level/highest performance/most flexible way to go. C# appears to involve using some GUI markup language called XAML, and doesn't appear to be able to access directx, erm, directly. Please correct me if I'm wrong here, but the impression I get is that MS is going 'native first' to compete with ios, android.

Trans-wise, I want to add the ability to import source/lib files from modules into a project. Just how to do this cleanly became clear once += was added, eg: SRCS+="{CD}/native/lang.${LANG}". This way, the 'lists' of SRCS, LIBS etc are completely target dependant and 'quoted imports' can eventually be killed off, something I've always wanted to do.

Also, I'd like to make it easier/possible to 'roll your own' targets. This is the biggy, and I think there are 3 main changes that need to be made to make this possible:

First, targets should have their own modules/ dir, allowing them to provide target specific monkey APIs without having to provide a separate modules/ package.

Second, all the 'native' app-specific classes/code such as view, viewcontroller, activity etc should be moved from Mojo to the target. This at first sounds like a step backwards, but it means each target becomes entirely self contained so can easily be kludged as necessary to support ad systems, social networking etc. It also means you can use interface builder and other high-level target SDK systems to create the initial target environment etc. So basically, target projects will become standalone, runnable projects that can be built and run without ANY Monkey code (except perhaps a nop Main) and the target modules provide the Monkey side interface to the target.

Finally, the 'Target' class in trans should become 'Builder', and targets should specify which builder to use in a TARGET.MONKEY file. Examples of builders would include GccBuilder, XcodeBuilder, MsvcBuilder, MonoBuilder, FlexBuilder etc. So, for example, both the Mac ios and glfw targets could use the XcodeBuilder.

The goal here is for people to be able to simply copy and paste a target dir, tweak the source code and/or  TARGET.MONKEY settings, and end up with a standalone target that can be dropped into any Monkey release.

The last 'big job' regarding trans is to split up the monolithic output 'main.cpp' style files into per module sources, and to be able to compile them separately. Once all of the above is going, this should be quite doable.

Last but not least, there WILL be an official Linux version, although it will be Makefile based in keeping with the Linux spirit! Given what MS and Apple are doing with forcing developers to sign apps etc, the day when Linux becomes the last, free OS is, I fear, coming much sooner than I once thought.

As for Steam-on-Linux, I'm in two minds about this. On the one hand, I agree with Linus's POV that anyone should be able to do anything with Linux, even DRM etc. On the other hand, there is zero evidence to suggest that Valve/Steam, as cool as they are now, wont eventually end up abusing their potential power. I already don't like the 'exclusivity' aspect of having to get your game approved for Steam, whether or not that's by popular vote or whatever. I think overall I tend to agree with Linus here: freedom has to be *absolute* or it's not freedom - even the freedom to be an asshole. But I also think we also need guys like RMS to keep reminding us where we stand in terms of the big picture, or we'll start forgetting that THERE IS ALWAYS AN ALTERNATIVE!

Peace out!


  1. Thats awesome news to me! Specially the reflection thing and the changes on the way targets can be easilly added to Trans in the future. Also looking forward for the windows 8 target.

  2. Brilliant work! The async stuff looks sexy and Monkey is getting bigger day by day. Have a great Sunday!

  3. An official target for Linux coming sometime? Great news! The biggest reason I haven't moved from Windows to Linux, is the lack of support for games I like on Linux. Wine can only do so much without without active support on an app module, unfortunately.

    And, your comments about Steam on Linux warmed my heart a bit.

  4. Brilliant update, great read.
    Glad to get a little of the "behind the code" thoughts from you especially like in the last paragraph.

    Monkey is excellent way of App/HTML5 coding, and it's marvelous that it just keeps getting richer and more versatile.

  5. I don't agree with your statement about freedom and whether it's a good thing to have games approved by Valve to be on Steam. Personally, seeing how hard it is to navigate through other, less discerning, online catalogues I would say that having someone select the games before they get on the store is good for me. If I have the time to search for more niche things I will do on my own time, but so far I've never been disappointed by anything I bought there, I can't say the same for, say, Desura, which is much less discerning.


Note: Only a member of this blog may post a comment.