The way this filter works is explained in documentation very clear, but people still say they do not understand it. My guess is that they never bothered to rtfm, actually.
Just to remind you, the displacement map is 2D displacement vector field dr = (dx,dy). For every location r = (x,y) the filter uses color at r + dr instead of r. The map is stored as two 8-bit integers per pixel (you can pack 2 maps together in single BitmapData, therefore). Mathematically, this means you use dot product dr · (28i,28j) as a color to store the map.
Therefore, all you need to do in order to generate displacement map for some arbitrary transformation, is to calculate source pixel coordinates for every pixel in the map, subtract current coordinates, and pack the result into selected color channels. Here is code stub to do it:
import flash.display.BitmapData;
// BitmapData to hold the map
var map:BitmapData = new BitmapData (...),
w:Number = map.width, h:Number = map.height;
// color factors
var af = 16777216, rf = 65536, gf = 256, bf = 1;
// this may take a while
for (var x:Number = 0; x < w; x++)
for (var y:Number = 0; y < h; y++) {
// set source pixel coordinates for (x,y)
var xS:Number = ...;
var yS:Number = ...;
// calculate & store displacement in B & G channels
map.setPixel (x, y, bf * (128 + xS - x) + gf * (128 + yS - y));
}
// show the result: make the screenshot, crop it, and import
_root.attachBitmap (map, _root.getNextHighestDepth ());
Here is code stub to create a filter in your FLA:
import flash.filters.BitmapFilter;
import flash.filters.DisplacementMapFilter;
import flash.display.BitmapData;
import flash.geom.Point;
// channels
var ac = 8, bc = 4, gc = 2, rc = 1;
// making a filter
var b:BitmapData = BitmapData.loadBitmap (id);
var f:BitmapFilter = new DisplacementMapFilter (b,
new Point(), bc, gc, 256 * scaleX, 256 * scaleY);
Hope that helps :) Now go check some examples.