Take a snapshot
from a Kinect and transform it into a MineCraft save file. The Depth and Color
information will be used to approximate the position and types of blocks to
best represent the 3D point cloud in MineCraft.
script has been modified to take very quick successive snapshots. To take
them quickly as possible, they are all stored in memory and dumped at the
end. If you try to take too many, your PC may get upset…
Verbose: Set to true to show logging on the console
testEdit: When true, nothing is actually saved to the
minecraft save. Instead, an image “mapImage.png” is created to show a
cutout of what will be saved projected in 2D
colorMatch: When true, the various colors of the 3D image
are mapped. Currently uses a manhatten distance RGB algorithm.
If false, will
create statues out of randomized smooth stone, cobblestone, and mossy
storeOldChunks: True if the pre-edited chunks should be
pushed to a zip file so they can be restored afterwards
ImageNames: the depth and color image from getSnapshot.py
that will be processed
pathToSaves: The absolute path to the minecraft save file
minmaxDepth: This can be toggled to exclude regions outside
these bounds. The background walls, or close static can be removed by
toggling these values.
blocksPerMeter: When set to 100, each block will represent
1cm^3. Can toggle this to shrink or grow models.
floorBlock: The “ground” of the structures. If you lower
the world, be sure to lower this from 63
test attribute that will put a glass tower wherever the projection will
test attribute that places colums at the boundaries of the chunks
centerChunksX,Z: The projection is centered around the X:0, Z:0
corner of this chunk
Note: All the
modified chunks must exist prior to the save
PossibleBlocks: A list of all the blocks that can be color
matched. The name, ID, and associated RGB values that represent the
block. Current values were chosen by sampling the various textures from
helper script that takes a zip file with pre-edited chunks in it created
by processImages.py and loads them back into the world. This effectively ‘clears’
the last edit if used properly.
savedZipName: The name of the zip file that has the pre-edited files inside it
a windows machine to automatically process the many frames of data
captured from getSnapshot.py. Since it can capture up to 30 frames a
second, a way to automate:
processing a depth/color image into a statue.. navigate to the
MineCraft process, drive clicks to open the target world, position the
player, take a screenshot, close out of the world, and then restore the
world back to its previous state.
Make sure Minecraft is open on the main menu before running the
script, or the handle will not be able to be found!
15 new colors for wool were added. Red wood and
Birch may also make useful blocks for color
Assuming these are the 0-15 values of ‘Data’
for a wool block, but not sure. Create a script to add wool, and change
the data element to differing values
The new terrain.png is:
note the wool->
15 colors which are set by changing the 4 bit
Get a representitive RGB for each of the
blocks, and add them to the RGB list
Take out pink as it’s annoying and doesn’t
Create a heuristic to check and see if a color
should be grayscale.
If the average of the RGB is close to all 3
components of the RGB, then force it to choose from the grayscale. This
should get rid of the light/dark colored blocks from mixing in with the
grayscale ones. The false positives should be minor since they were
close to a gray anyway. (wound up removing this)
What already exists?
Velentin created an Image->Schematic
program in C, his code has been released and is at:
The colors look marginally better, will keep
this setup for now
Results Color Test: (please excuse the
shirt!) – 100 blocks per meter
Color mapped image using RGB representation
I love how my garnet/silver bracelet was
turned to a ring of redstone! So cool!
Getting glass out of the equation as having
empty see-through holes isn’t so cool.
Update RGB manhatten distance equations with
world for the action
How to set the stage
Have to anchor the statues to the ground, and
get a point of view as close too the kinects as possible. This will yield
the best result in an animation.
Created a ‘floorBlock’ variable that will
anchor the statue to a Y level between 0-127
Instead of ‘clearing’ the stage as done in processImages.py, lower a world so more
room for massive statues is present. The look and feel of the world will
be preserved in the background
Lowering by 30 gives a great effect, while
does not slow down the world too much.
Make it so that the stage always centers, no
matter what BlocksPerMeter is set too. This will allow the growing and
shrinking of models while retaining a correct POV
Determine how clouds work as they are an
inevitable part of a massive sculpture animation
My assumption is that they are based off the
time element in the Level.dat file.
Two successive logins to the same world have
the clouds in the same position
Try modifying the time and see if that screws
with the cloud position
If not, try changing chunks and see if that
screws with it.. If not, we’re golden!
If clouds will play nicely, then we do not
need to worry about cutting them out, as the quickness of the automated
snapshots will make them look like they’re progressing normally
If clouds don’t play nicely, the Point Of View
of the user will have to be positioned in such a way as to
Build worlds until a good canvas for an
animation is found. Maybe something flatish with a mountain in the
Lowering the world
In order to have truly monsterously huge
statues, the world has to be lowered. Sea level is around 63 on the
available 0-127 spread. Lower the entire world…. But how?
Create a branched version of processImages.py
that takes a saved world, iterates through every chunk and lowers the
Go through the block data, take every block
above the cutoff and transfer it’s data down Y by 60. Do the same for
the skylight, heightmap, and blocklight. That way, other than the world
being much taller, noone should ever know that it was lowered!
BlockID – Get the index and just update the
target. Clear the original if necessary
Data – has a low and high component to each 4
bit word. Numpy is atomic to bytes, so need functions to split the low
and high sections
Skylight/Blocklight – same as Data
HeightMap – lower all heights by 60. If lower
than 0, set to 0
First pass looks okay, minus some re-rendering
of chunks. Why does this happen?
Also, maybe bedrock or something should be put
at lvl 0 so you wont’ fall through the world while swimming and stuff.
This will change some light calculations, but hopefully not bad enough
to cause major issues
Can we clear the entities for each chunk? I
have a suspicision that they are causing the re-rendering of select
chunks.Mob spawners and chests will not be aligning with the entity
data, could be causing problems.
As we can see from the mc-map render, the chunks were properly
lowered, but the logging in respawn death caused the respawn to move and
thus the world was expanded
Created a module to lower the player during
the save so they will not fall to their death on login. Hopefully this
gets rid of the stuttering and crashing problem seen so far.
While it makes me not die at login, it crashes
not long after loading, even on peaceful.
North is X-, South is X+
East is Z-, West is Z+
Set Entities and TileEntities of every chunk
to NULL or whatever
Both Entities and TileEntities are in the
chunks [“Level”][“Entities”] and [“Level”][“TileEntities”] areas. When
they have data, a TAG_Compound is found, and when they don’t a TAG_Byte
By just setting them all to TAG_Byte with a
value of 0, and saving, should be enough. This is based on the
assumption that most things will be brought under the world and trimmed
Couldn’t get that to work Scanned the chunks
for TAG_Lists where empty, and basically copied them into every chunk.
This should be sufficient for ‘clearing’ every entity in the world.
If this doesn’t work, set the floor to bedrock
so entities don’t fall through. That is my gut feeling as to what is
Didn’t wind up doing this, as clearing the
entities stopped the crashes! My guess is that entities fell through
the world and kept falling.. so the bedrock fix may have worked too,
but they the calculations of blocklight would be wrong.. so..
Turns out if the skylight for air is not set
to 15, then new blocks that are added to the air with a skylight of 0 is
shaded incorrectly. For now force the top slide of new air from the
lowering of the world to have a skylight of 15.
Anchor the statues
Instead of anchoring the statues to the top of
the world like currently, use the XYZ positioning found from the kinect
to have a 1-1 mapping of the world and the 3d region sensed by the
kinect. This will make it so that when people or objects move in/out of
the Kinects range, the statues will not jitter around as they anchor to
the top of the world.
The ‘PlaySpace’ of the MineCraft world is
currently dependant on the width of a single frame. That won’t work, as
the various frames will cause a jitter back and forth as their extents
change.. choose a real region for the Kinect data to be rendered.
Use current images to find a good spread (in
meters) to use as a working minecraft area
Hacked up processImages.py to not save
anything, but instead find the depth of the ‘players’
Pic#63 will be good for horizontal spread:
X [-1.4m to 1.26m] Y[-.88m to 1.0m]
Z[1.76m,2.6m] was detected for: 136 pixels across resulting in:
Reasonably, -1.25m to 1.25m will work
well, this results in 2.5m of working area.
Pic#74 will be good for depth spread:
X[-1.46,-1.26] Y[-.57m to 1.03m] Z[1.7m to
2.5m] was detected, creating 138px:
A little closer would probably be better,
adjusting to [1.15m to 2.55m] will give a 1.4m working area.
This should work well.
How to anchor Y is the tricky part. We really
want the models anchored at the bottom, but that is difficult… as the
player moves forward and his legs are cut off from the image, we will
lose the information about where his feet hit the ground, giving him the
illusion of floating. We will always have 128 blocks vertically to deal
with, and it looks like going to about +1.1m or so in the positive will
As the blocks representing the 3D space get
bigger, we want the models to orient onto the gorund. Having them float
high above isn’t very useful.
The ‘floor’should be a new variable allowing
control over where the Y variable anchors. This will give some
flexibility to the user to allow them to try to match up the models
with the ground before creating the full stop-action animation.
The floor will initially be set to -.65m,
which is kinda an average of the two images above
Note: Y is reversed when read in from the
kinect. Negative is high, and positive is low. Why, I have no clue.
Looks good! Chose a floor of .85 as it seems
to be decently centered at the bottom so the statues will look like
they’re actually standing. As the models are sized, they should
continue to be on the ground.
When the models are scaled via the
‘BlocksPerMeter’ variable, they now anchor correctly on the ground.
When they are resized, they will get smaller,
but stay on the ground!
Notice it is 128 pixels high, and that is
exactly how many blocks of resolution we have to play with for the
Make sure the area is centered for different
scales. The ‘center’ chunk can be toggled, and (X:0,Z:0) of this chunk
should be the center of the diplay.
To test, need some way to test. Make it so
that the 0,0 coordinate of the initial start chunk is marked with a
tower of glass or something.
Here is a world I created for the movie, that
was lowered 40. A glass tower was created in the 0,0 of this chunk, it
wil be the center
All corners have a tower of bedrock. Center
corner has a tower of glass.
Make sure block spread is being correctly
Center the projection to always be by the
Setting the POV
Save the world to be used as a canvas, and
create a copy. Take that copy and import in a statue’d image of two
people. The location should be as close as possible to the Kinects
location relative to the actors in the real world, for best effect.
Created world until one with water in the
front, flat area behind it, with a mountain in back was found.. took 3
The wooden linein the water is the line that
represents the Line of Sight of the Kinect
Find a good center for the statues, remember
the chunk orientation (bottom left of the image is North)
Create a ledge with layers so the vantage can
be raised/lowered when the scale changes
The player in these cases is pointing directly
south. Extract the players rotation information and see what it is set
too when facing south.
The specification states Rotation is a list of
2 floats that represent the yaw and pitch respectively. West is 0.
Using twoolies NBTFile parser, can access the
Yaw, and set to due south by:
This is the meat of the project. We have many
images read to be transformed into many worlds which in turn will turn
into many snapshots which can be stop-actioned. While this *can* be done
manually, it would take a very long time, and creating a smooth stop
motion animation would be tedious and time consuming. What ever can we do
about this? How much of it can be automated using Python?
Overview - For each frame:
Save the pre-edited chunks so we can revert
after the frame is rendered in the world
Save the players position, pitch, and yaw so
that even if it moves somehow for one frame, it will not effect
Take the particular RGB and Disparity images
from the Kinect and process them into the world using processImages.py
Take the screenshot
Revert the world back by taking replacing all
edited chunks with their originals
Breaking up the scripts:
Both the main scripts have to be tweaked so
they can be imported and called from a controller script. Both have to
utilize main, and be able to pass in the variables that can change.
worldPath of the save
savedZipName of the zip file to save the
pre-edited chunks too
name of the image that is being processed
worldPath of the save
savesZipName of the zip file with the
Now that the scripts can be called seperately,
create a new script, driverAnimation.py, to handle the outer layer of the
With minecraft open:
Go to a designated directory with all the
images, and create a list of all the .png,.tiff image pairs
Extract the player position, pitch, and yaw so
that it will be consistent for every frame
Extract the world time (or just set it to
near-dawn) so that it is bright and lots of time can expire until dark
Render the world for the specific .png, .tiff
Move focus to minecraft
Some info on focus here: On Windows, you can use the pywin32
modules to assist you. There's a sub-module in them called win32gui that
has a SetForegroundWindow method. I usually use win32gui.EnumWindows()
to loop through all the open processes and then use both
win32gui.ShowWindow() and win32gui.SetForegroundWindow() to make the
window I want appear.
Documentation of what it looks like I need to
do is at: win32gui.html
Set window size so images are 720x480
Anchor to top left of screen so mouse clicking
Using window positioning, move to
Windows Movie Maker has a stop-motion option up
to 1/8th of a second. Need a higher FPS.
Have a MAGIX Movie Edit Pro 11 license
When importing many still frames as 1-frame
components, many blanks are inserted. This required manual fixing which
is not fun for lots of frames.
VirtualDub allows for quick importing of many
Minecraft default size is 854x480. How close is
this to the various video options?
Hatching a plan
Import the screenshots into VirtualDub 1.9.11
to create a .avi.
File->Open Video File…
Select the first image in the series. Tested
this out using the .PNG captures from the Kinect and got an
00000OutputColor.avi that mashed all the frames together
Import this .avi into MEP
All snapshots are imported as one frame, with
Result: (click to see wmv created from Kinect
Creating the movie
Import all the stills made from the script
Import the VirtualDub .avi into MEP
Create the a 3.3 second 100 frame test movie
of my crappy juggling skills
Now that the stop motion is technically
complete, it could be uploaded as is. However, it would look more polished
if there was sound, effects, or something else to really make it stand
out. Some ideas are:
Use something un-licensed so that soulless
lawyers don’t try to take my BlackBerry shirt from me
Something that goes along with the animation
would be good
Craig Viniconis may be a good resource, as he
spends a good chunk of his time writing and playing music
Multiple stop-actions can be stitched
together. They should be of different scenes, and have different actors,
to really show the versitility of the application.
This will only be realistic if a majority of
the steps to create the stop action can be automated, otherwise it
would be tedious and noone (minus people I paid) would do it!
Maybe start with presentations of the familiar
images of the first project, then have one move! That should cause some
Try to come up with more!
Motion capture setting/plot
Can try hacky sacking or juggling. Both of
these should create cool effects.
Get girls. Everyone likes girls.
Colorful and fun
Idea: Anything you intersect with effects
future frames by removing the block from the world.
Say I’m swinging a sword and it cuts through a
house. The first world rendered where there is an intersection of old
block->new block, the position and chunk are remembered. Then, after
the old world saves are loaded, the intersected ones are cleared and
turned to air.
This could give some cool effects, like
walking through a mountain, punching houses, etc.
Could also have an ‘add’ mode, where blocks of
a certain color leave a trail and replace the original air with some
other type of block.
A five digit index 00000->00001->etc will be prepended to each image
The number of images to be taken, one after another
The number of seconds between each image taken
·Inserting a snapshot into a
oThis script takes a color and depth image from
getSnapshot.py and pushes them into a minecraft save. The statue is anchored to
the top of the world by default. The depth and spread of the area is determined
by the area of the 3D image capture from the Kinect
§When testing, set testEdit to True. This will prevent
the save from happening, and the mapImage.png file can be checked for results.
to true to show logging on the console
When true, nothing is actually saved to the minecraft save. Instead, an image
“mapImage.png” is created to show a cutout of what will be saved projected in
When true, the various colors of the 3D image are mapped. Currently uses a
manhatten distance RGB algorithm.
·If false, will create statues out
of randomized smooth stone, cobblestone, and mossy cobblestone
the depth and color image from getSnapshot.py that will be processed
The absolute path to the minecraft save file
This can be toggled to exclude regions outside these bounds. The background
walls, or close static can be removed by toggling these values.
When set to 100, each block will represent 1cm^3. Can toggle this to shrink or
where the image should be inserted into the world.
·Note: All the modified chunks must
exist prior to the save
False if blocks should be ‘cleared’ prior to the model insertion.
How deep to clear the chunks too if Additive is set to False
The type of block to put at the bottom of the cleared chunk
A list of all the blocks that can be color matched. The name, ID, and
associated RGB values that represent the block. Current values were chosen by
sampling the various textures from terrain.png
oThe first step is to undistort rgb and depth images using the
estimated distortion coefficients. Then, using the depth camera intrinsics,
each pixel (x_d,y_d) of the depth camera can be projected to metric 3D space
using the following formula:
oP3D.x = (x_d - cx_d) * depth(x_d,y_d) / fx_d
oP3D.y = (y_d - cy_d) * depth(x_d,y_d) / fy_d
oP3D.z = depth(x,y)
oWe can then reproject each 3D point on the color image and get its color:
In order to “open” minecraft save files,
an understanding of the save-format is needed. There are a few different
formats that minecraft has used throughout it’s life, but the latest is the
alpha format described here: http://www.minecraftwiki.net/wiki/Alpha_Level_Format
Some tidbits of knowledge are as follows:
·The terrain is split into 16x16x128 chunks and saved into their
·The world folder can have 64 subdirectories, which can each
contain 64 subdirectories
·Each chunk is identified by it’s xPos and zPos.
oThe chunks name
is created by base36ing xPos and zPos.
oThe names of
the folders are found by taking xPos and zPos, modulo 64, and converting to
oEx: to find the
cunk at position (-13, 44)
§The first folder name
is base36(-13 % 64). This is base36(51) which is "1f".
§The second folder
name is base36(44 % 64). This is base36(44) which is "18".
§The chunk file's name
is "c." + base36(-13) + "." + base36(44) +
".dat". This evaluates to "c.-d.18.dat"
§Thus, the chunk at
(-13, 44) is stored in "1f/18/c.-d.18.dat"
·Level.dat stores environmental data like time of day, health,
·There are 128x16x16 (32768) blocks per chunk
·There are many bits of light optimization (skylight, blocklight,
oBlocks are laid
out in sets of vertical columns, rows go E->W, columns go N->S
ounsigned char BlockID = Blocks[ y + ( z * ChunkSizeY(=128) + ( x *
ChunkSizeY(=128) * ChunkSizeZ(=16) ) ) ];
§Simple: char BlockID = Blocks[ y + ( z * 128 + ( x * 128 * 16) ) ]
oX increases south, decreases north
oY increased upwards, decreases downwards
oZ increases West, decreases East
Now that we have some knowledge of the format, lets fill in some
overviewer does a good job with having a small simple python script for
converting to base36. Use theirs!
minecraft world is only 128 high, a 480 high depth map may not work… So lets
scale by 1/8th. This will turn a 640x480 to a 80x60.
§Sampling for now, but
turn to an average if required
Now that we have the data, we have to
process is so that the image will properly align in the minecraft universe so
that renders of the world, with N to the Bottom Left, will see the image
created from the Kinect data.
·Determine the area of drawing. How deep should these images be?
to 0-64 grayscale via sampling
·What chunks will be effected?
oWe want it
starting at 0,0 for ease of math reasons
o5 chunks will
spread in the E->W direction, so the following chunks will be the ‘z-axis’
of the image
§0,0 0,1 0,2 0,3
oIt will go 4
chunks deep in the N->S direction
§0,0 -1,0 -2,0 -3,0
oTurn the normal
NSEW 90deg counterclockwise
§EW is Z axis
§NS is X axis.
(N)(X+) ------------(X-) (S)
§EW is Z
·Calculate which chunks actually have to be opened
python function that translates the required chunks into the paths it should
·After all the data is read in, ungzipped, the TAG_ structures
have to be parsed so we can edit the appropriate part of the file
§Find a way to find
the position, name, and size
oWith the NBT package installed,
·Update heightmap, skymap
The overall flow of the save will be:
·Open and Scale Kinect images
·Check Minecraft Save and Open
·For each X in the width of the
oEvery 16 pixels:
§Save old strip of chunks if loaded
§Load the strip of chunks coorelating to next 16 horz
oFor each Y in the height of the image, 0->59
§Calculate chunk XZ
§Calculate Block XYZ
§Update chunk data
Attempt a NBTFile creation file from:
Preliminary “in the world” save:
Note: This is saving basic
information. Every XY point in the image had it’s disparity found and a Z was
calculated. This is just a projection into 3D rather than a true 3D image. It
was based off a scaled depth map from the kinect(image 1), and originally
turned to a glass sculpture(image2) and then as a stone/wood statue(image3)
blocks can be: (4 bits per block – same index as blockID)
§If these are saved incorrectly, or become incorrect by
updating a blockID that had a value that is now invalid, there can be crashes.
This data should be set imediately after setting a new block in a pos.
oPumpkins: 0123 for ESWN
§North (3) or East(0) can be used to give an additional
block of color
oTorches: 1234 for SNWE, 5 is standing
·Lets start with 4 basic colors for
the blocks and get the algorithm working
§BlockID 1, HSL(0,1,115)
§BlockID 5, HSL(25,119,103)
§BlockID 57, HSL(120, 240, 152)
§BlockID 12, HSL(33,136,192)
·Create color different algorithim
oInitial: .5dist(H) + .3dist(L) + .2dist(S)
·Then add access to the Data for
oThen recalc Data while adding blocks. Parallel to
·The dividing of the depth data by
4 is losing too much resolution. Turning it into a 64 spread
First attempt at color matching using
straight HLS distance:
Changed HLS to be 50,30,20 weight:
·Adding more colors:
oUse multiple HLS’s for each block. Start with 1 each,
then add other tints you want that block to represent
oPrint a pallett out so you can see the possible colors
Refine the colors!
oTurn this into a minecraft save:
§Instead of going through every X and Y pixel of the
depth image, and looking for its cooresponding depth, the projection points
from the above formula are used.
·For every X,Y of the image,
determine its depth from the disparity
·Determine it’s 3D X,Y,Z position
in space in meters
·Re-project onto the RGB to get the
color of the particular pixel
·Map to blocks in the minecraft
·Find the type for the RGB and
change the block
·Save the file
oFirst working attempt, Color mapping data (in real 3D
X,Y coords – newer overwrite Z) and MC:
It is annoying to load a world that is
night. Modify the world after the save to set it to noon.
Level.dat file format contains the time
of day, it’s format can be found here
Stores the current "time of day" in ticks. There are 20 ticks per
real-life second, and 24000 ticks per Minecraft day, making the day length 20
minutes. 0 appears to be sunrise, 12000 sunset and 24000 sunrise again.
it to 12000 and the sun was going down
to 18000 and it was night!
·Creating a heightmap
chunk that is cleared and saved should have it’s heightmap recalculated. This
is required in order for the blocklight and skylight to be calculated correctly