Showing posts with label Graphics. Show all posts
Showing posts with label Graphics. Show all posts

HTML 5 Web Workers and image processing

One of the more exiting things we are getting via the new HTML specification is Web Workers. If you have no idea what Web Workers (lets call them Worker from now on) are you can basically think of them as "sandboxed" threads. By threads I do refer to threads as in «multithreading», an architectural feature of modern CPUs and operating systems. Workers allow us to run tasks separately from the main thread where all the drawing, animation and DOM manipulation is going on. This allows us to do calculations without leaving the impression on the user that the browser is "hanging". If you have worked with other "thread wrapping" technologies like Grand Central Dispatch (GCD) you will get Workers without fuzz, however Workers do have some limitations that other similar technologies don't have.

Workers limitations and usage
First of all, Workers are separate code blocks of JavaScript not found in the spawning script, kinda.. In most cases a Worker is a separate JavaScript file which you DO NOT refer to in you HTML script tags in the HTML document. However, Workers can also be defined inline through the BlobBuilder interface. In this post I will use only external Workers. Also, there are two kinds of Workers, «Shared» and «Dedicated». I will only use «Dedicated Workers» in this post.

Workers cannot access the following:
  • The DOM or the DOM APIs
  • The window object
  • The document and the parent object. 
A worker can access:
  • The «navigator» object
  • «XMLHttpRequest»
  • A read-only version of the «location» object
  • The Application cache
  • setTimeout(), clearTimeout() and setInterval(), clearInterval()
A Worker can also spawn other Workers and import external scripts via «importScripts()».

The demo (aka. the fun part)
For the demo this time I have created a simple web-app which loads and displays 3 pictures n times. You can randomize the position of the images by clicking the randomize button. You can pick the number of images displayed as well as their size. The two image manipulation buttons will grayscale and invert the images, if you check the checkbox the images will perform the randomize animation concurrently with the image processing. It's this latter part which calls for the usage of Workers.

The demo will show you how to spawn a separate Worker for each of the images, thereby allowing for concurrent processing and animation without much degradation in performance. I won't explicitly go through all the layout and animation code here, rather focusing on the usage of the Workers.

var worker = new Worker('DWGrayscale.js');
worker.addEventListener('message', function(e){
   ctxAr[e.data.index].context.putImageData(e.data.imagedata,0,0);
});
worker.postMessage(...imagedata...);

On line 126 - 133 within the grayscale function we create a Worker for each of the images, then we add a listener for the «onmessage» event allowing the Worker to communicate with the main thread. We finish off by posting a message to the worker using «postMessage» and passing the image data. These messages are the only way to communicate with a Worker. This means that we also need to implement this interface in the Worker itself.
addEventListener('message', function(e){
 var imageData = e.data.imagedata;
 //....process image data....
 postMessage(...imagedata...);
});
In the Worker «DWGrayscale.js» we grab the passed image data from the «data» property of the event. We then go on processing the image data, when done we post a message back to the main thread passing the image data which then in turn can be used to update the canvas element on screen.

Why not pass the Canvas element or at least the Canvas context?
Workers do not have access to the DOM, because it would not be thread safe. Workers are limited to passing object which can be serialized into JSON, which does not support cyclic objects. The Canvas element is a DOM element and the context is a cyclic object, hence we need to get the actual pixel array which we can pass back and forth to the Worker.

Take a look at the demo and the source code to learn in more detail how the demo was created.
Note also that you should probably run the demo in Safari as it has no limitation on the number of Workers concurrently running like Chrome has, also the animations uses the WebKit prefix and will not work in non WebKit browsers. So, please don't use this code in an production environment!

References:

Downscaling images in Flash applications

The other day I ended up in an interesting discussion regarding how to downscale a large image in a Flash based application. What we figured out was that even though the FlashPlayer can do bilinear smoothing on scaled images automatically, this will fail when the scaling factor is more than 2 times of the source image. Hence a custom method is needed.

The simple approach (scaling down less than 2 times the size)
If the loaded image is to be scaled less than 2 times (400 -> 200) simply use the build in "smoothing" property of the Bitmap Class. This can be set directly on the Bitmap itself or via different DisplayElements such as the mx:Image in Flex. If you set this property to true the Bitmap will use bilinear scaling, achieving quite good results.

Grayscale bilinear interpolation
Now, if you need to do actual resampling of an image, in other words create a new smaller image, then you need to do this manually using the "draw()" method of the BitmapData Class. Use the bitmapData.draw() method and set the "smoothing" property to true. This will give you the same result as setting the smoothing property on an Bitmap container, however it uses less memory as only the pixels for the resulting image is stored (if you remember to dispose of the original data!).

Scaling down more than 2 times
Let's say that you have an image that is quite large and you want to create a thumbnail from it. A thumbnail is often way smaller than 2 times smaller than the original image. It would be quite nice to use the above method to just scale the image, or better resample the image with the "smoothing" property set to true. However, if you try to do this the FlashPlayer will simply fail and default back to using the "nearest neighbor" algorithm. If you try it you'll soon discover that this will give you a horrible result.

What you'll need to do in this case is to do an iterative or recursive resampling of the image at hand. In layman terms this means resampling the image several times, each time no more than 2 times downscaling factor. You could call this a multi pass resampler (actually it would be a multi-multi pass resampling as the draw() method is also multi pass). You would still use the build in "BitmapData.draw()" method to accomplish this, you only divide the work into multiple jobs. This will give you a much better result. Note that in my experience, function calls in ActionsScript 3 are quite expensive, hence a iterative approach would maybe be preferable to a recursive one.

Using PixelBender
If you have decided to go for the manual approach of resampling the picture, you can use a PixelBender shader to do the job. In this case the actual resampling will be a lot faster (as the PB language is based on C), however applying the PB shader and passing the image back and forth takes time. See the speed section..

Combining the two
So, what if you have an application and you don't know in advance the downscale factor. Well, then you'll need to use both of the approaches like so:

  1. Calculate the scale factor
  2. If the scale factor is less than 2, set "smoothing" to true on the container or do a singe pass resampling.
  3. If the scale factor is more than two, use the multi pass scaling approach. And set the "smoothing" property of the container to false (else it will be blurry).
What about speed?
Let's say that you have an image that is 1200 x 1200 pixels and you're resampling it to 200 x 200. This is a 6 times resample and will look like ass using only the "smoothing" property, but it is faster.

Let look at some hard results. In this test we see how much time is used extra compared to no "nearest neighbor" scaling (default in Bitmaps if smoothing is off).

MethodTime in ms
Smoothing property on
0 ms
Multi pass resample
9 ms
Multi pass PixelBender
32 ms
Using the smoothing property takes no extra time, because it fails and defaults back to nearest neighbor. The other two are both pretty fast, however even though the PB algorithm is way faster than the others, we see that the applying of the shader eats a lot of time, thereby making the multi pass resampler method the way to go on a large rescaling operation.