In the last post I told you that we need a more efficient way to save the world data. Survivalcraft 2.2 doubled world height, and the data files have become too huge. For example, The Walking Dead world (one of the largest out there), occupies 1.4GB. And that is in 2.1 format, with chunks height of 128. Once you load it in 2.2, and it gets upgraded to the height of 256, the size doubles to 2.8GB. Ooops. This is too much:
I have experimented heavily with various new file formats. In fact, I have created three of them.
Here’s how the one I want to put in Survivalcraft 2.3 fares:
From 2.8GB to 26MB. Over 100x reduction! That’s better than I expected. The new format uses compression: a combination of custom RLE followed by a deflate. The RLE is clever, it uses the 4 bits of light data that does not need to be saved. The major issue is that the compressed chunks now have changing sizes, so simple array-style chunks table is out. More about the new format later.
Not all is rosy, however. The compression slows down loading, and most importantly, saving of chunks. The slowdown is considerable, about 5-10x. This is only to be expected, previously the game was just dumping memory direct to disk. Now, it needs to heavily process it first. The question is: does it affect the game? Fortunately, as far as I can see, not very much. Loading of chunks is not a problem, because it happens in a background thread anyway. Saving is more problematic, because it’s done on the main thread (to avoid having to keep chunk copies in save queue, memory is tight on the phones). Fortunately, saving happens infrequently. Plus there are various ways we can mitigate it, if needed.
The new system definitely needs more testing. I will be indirectly doing it when working on the rest of my todo list. At the moment it passes the brutal monster hot grind fatality crunch decapitation test (tens of millions of random chunk saves and loads).
I will check it in today and work on further items from my todo list for 2.3.
30 Comments
FIRST!
WOW!!!!!
Perfect. I hope this means we’re going to seeing new features being shared.
Multiplayer, Kaalus! Focus on that. That is paramount.
i agree
i think now is the perfect time also.
I dreamed about survivalcraft2 adding grills and dressers and fancy sound effects like wind gravel and the ocean had fancy water and waves! It was crazy but I can’t prove it to be real.
It’s either speed or memory. Most of the time you cannot have both so you have to sacrifice something.
Hope next posts are going be more frequent
Looks great!
Wow! The difference between old and file sizes is huge, great job!
Nice to such success! Great job! But what about publishing in Steam? And maybe Ubuntu support? 😄
I need some clarification about chunks issue, it takes longer to load chunks or may compromise/corrupt chunks data (as for now)? If if just a matter of loading then is more than fine if we safe MB (or even GB) of memory usage!
That truly is a surprise compression ratio! Let’s just hope it doesn’t cause more lagging.
This is epic!!!
I do feel like mp should be done very soon. Minecraft is churning out updates and now there is a huge difference between the to. It will close the gap if we add mp.
Yep. That’s what I was thinking as well.
And once you get it on steam you should do “snapshot” of the new update we can test.
I agree but that might take a lot of work to set up.
100x reduction, most impressive!
About SurvivalCraft Official Wiki, To make it really official, Kaalus, did you ever tried talking with FANDOM?
Congratulations, Kaalus. You always achieved what you wanted!
Nice
Why is Amazon better than the playmarket?
Yes! I have issues uploading worlds, because of the file size, and thi s should help drastically! (2.8 GB to 26 MB is at least 100 times as small!
Woah that’s amazing!! Minecraft has small world files and now if you can get your files 100x compressed then survival craft will definitely be the ultimate! LAN multiplayer would be amazing too!
LAN or online?
Put rabbits and make the survivalcraft online pleazzzzz
follow my reasoning if the worlds become lighter they would be easier to “go online” so with the worlds lighter could it be easier to have an online multiplayer? that would be cool
That is a great idea! I have a world I want to access but I can’t because it says it needs at least 228 MB to convert and I don’t have enough space on the device it’s on.
great work