New world file format

In the last post I told you that we need a more efficient way to save the world data. Survivalcraft 2.2 doubled world height, and the data files have become too huge. For example, The Walking Dead world (one of the largest out there), occupies 1.4GB. And that is in 2.1 format, with chunks height of 128. Once you load it in 2.2, and it gets upgraded to the height of 256, the size doubles to 2.8GB. Ooops. This is too much:

I have experimented heavily with various new file formats. In fact, I have created three of them.

Here’s how the one I want to put in Survivalcraft 2.3 fares:

From 2.8GB to 26MB. Over 100x reduction! That’s better than I expected. The new format uses compression: a combination of custom RLE followed by a deflate. The RLE is clever, it uses the 4 bits of light data that does not need to be saved. The major issue is that the compressed chunks now have changing sizes, so simple array-style chunks table is out. More about the new format later.

Not all is rosy, however. The compression slows down loading, and most importantly, saving of chunks. The slowdown is considerable, about 5-10x. This is only to be expected, previously the game was just dumping memory direct to disk. Now, it needs to heavily process it first. The question is: does it affect the game? Fortunately, as far as I can see, not very much. Loading of chunks is not a problem, because it happens in a background thread anyway. Saving is more problematic, because it’s done on the main thread (to avoid having to keep chunk copies in save queue, memory is tight on the phones). Fortunately, saving happens infrequently. Plus there are various ways we can mitigate it, if needed.

The new system definitely needs more testing. I will be indirectly doing it when working on the rest of my todo list. At the moment it passes the brutal monster hot grind fatality crunch decapitation test (tens of millions of random chunk saves and loads).

I will check it in today and work on further items from my todo list for 2.3.

30 Comments

  1. Posted February 8, 2020 at 13:41 | Permalink | Reply

    FIRST!

  2. A Nice guy
    Posted February 8, 2020 at 13:48 | Permalink | Reply

    WOW!!!!!

  3. Posted February 8, 2020 at 13:50 | Permalink | Reply

    Perfect. I hope this means we’re going to seeing new features being shared.

  4. Cristóbal Demetrius
    Posted February 8, 2020 at 13:52 | Permalink | Reply

    Multiplayer, Kaalus! Focus on that. That is paramount.

    • Leinad Yllek
      Posted February 8, 2020 at 16:22 | Permalink | Reply

      i agree

    • Wyatt Moe
      Posted February 9, 2020 at 15:42 | Permalink | Reply

      i think now is the perfect time also.

  5. Wyatt Moe
    Posted February 8, 2020 at 13:59 | Permalink | Reply

    I dreamed about survivalcraft2 adding grills and dressers and fancy sound effects like wind gravel and the ocean had fancy water and waves! It was crazy but I can’t prove it to be real.

  6. Posted February 8, 2020 at 14:01 | Permalink | Reply

    It’s either speed or memory. Most of the time you cannot have both so you have to sacrifice something.
    Hope next posts are going be more frequent

  7. Johan Steyn
    Posted February 8, 2020 at 14:04 | Permalink | Reply

    Looks great!

  8. Posted February 8, 2020 at 14:10 | Permalink | Reply

    Wow! The difference between old and file sizes is huge, great job!

  9. Posted February 8, 2020 at 14:54 | Permalink | Reply

    Nice to such success! Great job! But what about publishing in Steam? And maybe Ubuntu support? 😄

  10. Chris Drake
    Posted February 8, 2020 at 14:58 | Permalink | Reply

    I need some clarification about chunks issue, it takes longer to load chunks or may compromise/corrupt chunks data (as for now)? If if just a matter of loading then is more than fine if we safe MB (or even GB) of memory usage!

  11. Stanimus
    Posted February 8, 2020 at 15:24 | Permalink | Reply

    That truly is a surprise compression ratio! Let’s just hope it doesn’t cause more lagging.

  12. Nathan Hohnberger
    Posted February 8, 2020 at 16:20 | Permalink | Reply

    This is epic!!!

  13. Nathan Hohnberger
    Posted February 8, 2020 at 16:23 | Permalink | Reply

    I do feel like mp should be done very soon. Minecraft is churning out updates and now there is a huge difference between the to. It will close the gap if we add mp.

    • Cristóbal Demetrius
      Posted February 9, 2020 at 06:18 | Permalink | Reply

      Yep. That’s what I was thinking as well.

  14. Nathan Hohnberger
    Posted February 8, 2020 at 16:27 | Permalink | Reply

    And once you get it on steam you should do “snapshot” of the new update we can test.

    • Nater Boggy
      Posted March 20, 2020 at 15:36 | Permalink | Reply

      I agree but that might take a lot of work to set up.

  15. Posted February 8, 2020 at 21:01 | Permalink | Reply

    100x reduction, most impressive!

  16. ivanalbrieu
    Posted February 8, 2020 at 23:50 | Permalink | Reply

    About SurvivalCraft Official Wiki, To make it really official, Kaalus, did you ever tried talking with FANDOM?

  17. Dima Vaskivskij
    Posted February 9, 2020 at 13:17 | Permalink | Reply

    Congratulations, Kaalus. You always achieved what you wanted!

  18. Posted February 10, 2020 at 09:09 | Permalink | Reply

    Nice

  19. Dima Vaskivskij
    Posted February 10, 2020 at 18:38 | Permalink | Reply

    Why is Amazon better than the playmarket?

  20. Posted February 14, 2020 at 05:32 | Permalink | Reply

    Yes! I have issues uploading worlds, because of the file size, and thi s should help drastically! (2.8 GB to 26 MB is at least 100 times as small!

  21. Cameron Barratt
    Posted February 15, 2020 at 11:21 | Permalink | Reply

    Woah that’s amazing!! Minecraft has small world files and now if you can get your files 100x compressed then survival craft will definitely be the ultimate! LAN multiplayer would be amazing too!

    • Stefix1
      Posted June 26, 2020 at 23:34 | Permalink | Reply

      LAN or online?

  22. john thal
    Posted April 29, 2020 at 11:01 | Permalink | Reply

    Put rabbits and make the survivalcraft online pleazzzzz

  23. JAY JAY
    Posted May 14, 2020 at 01:24 | Permalink | Reply

    follow my reasoning if the worlds become lighter they would be easier to “go online” so with the worlds lighter could it be easier to have an online multiplayer? that would be cool

  24. beastsuper2
    Posted June 18, 2020 at 19:04 | Permalink | Reply

    That is a great idea! I have a world I want to access but I can’t because it says it needs at least 228 MB to convert and I don’t have enough space on the device it’s on.

  25. Johan Steyn
    Posted June 28, 2020 at 13:46 | Permalink | Reply

    great work

Leave a Reply. No swearing please.

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s