New world file format

In the last post I told you that we need a more efficient way to save the world data. Survivalcraft 2.2 doubled world height, and the data files have become too huge. For example, The Walking Dead world (one of the largest out there), occupies 1.4GB. And that is in 2.1 format, with chunks height of 128. Once you load it in 2.2, and it gets upgraded to the height of 256, the size doubles to 2.8GB. Ooops. This is too much:

I have experimented heavily with various new file formats. In fact, I have created three of them.

Here’s how the one I want to put in Survivalcraft 2.3 fares:

From 2.8GB to 26MB. Over 100x reduction! That’s better than I expected. The new format uses compression: a combination of custom RLE followed by a deflate. The RLE is clever, it uses the 4 bits of light data that does not need to be saved. The major issue is that the compressed chunks now have changing sizes, so simple array-style chunks table is out. More about the new format later.

Not all is rosy, however. The compression slows down loading, and most importantly, saving of chunks. The slowdown is considerable, about 5-10x. This is only to be expected, previously the game was just dumping memory direct to disk. Now, it needs to heavily process it first. The question is: does it affect the game? Fortunately, as far as I can see, not very much. Loading of chunks is not a problem, because it happens in a background thread anyway. Saving is more problematic, because it’s done on the main thread (to avoid having to keep chunk copies in save queue, memory is tight on the phones). Fortunately, saving happens infrequently. Plus there are various ways we can mitigate it, if needed.

The new system definitely needs more testing. I will be indirectly doing it when working on the rest of my todo list. At the moment it passes the brutal monster hot grind fatality crunch decapitation test (tens of millions of random chunk saves and loads).

I will check it in today and work on further items from my todo list for 2.3.