2.3 progress

As you all know, 2.2 doubled the height of the world. This was a long overdue change that allows you to build properly high structures. It also doubled the size of the world files. It’s this doubling effect that I want to focus on this post.

Back in 2016 when releasing 1.29, I doubled the size of each block to use 32 bits. This doubled the size of the world files too. In 2.2, the increase in height doubled it again. The files have become huge – some larger worlds are reaching multiple gigabytes. This is something I haven’t paid enough attention to.

Another aggravating change I eventually relented on for 2.2 – after getting much feedback from you – was to increase of the maximum worlds limit on device from 20 to 30. In hindsight, this may have been a mistake.

The problem is that many, many people are running out of space on their devices. They download many large worlds from Community Content and everything grinds to a painful halt.

Another issue caused by large worlds is exceeding of Dropbox bandwidth limit for people who host their worlds for Community Content. This has always happened, but now, with worlds twice as large, it happens twice as often. The result is worlds becoming unavailable and dropping down through the ranks, until Dropbox unblocks the account after a few days. At that point, the world is so far down it might never climb out again, no matter how good it is.

As a result, one of my priority tasks for 2.3 is to change the world storage format to reduce world sizes.

2.2 and earlier use a naive format, where chunk data is stored verbatim – every block taking 32 bits. This is very inefficient, but very fast, simple and error-resilient. It also means that every chunk has exactly the same size on disk, which is extremely important. Here’s the description of the current file format: https://kaalus.wordpress.com/chunks-dat-file-format/

There are multiple compression schemes we can use to reduce the size of the chunks. Even something as simple as RLE (run length encoding) is likely to cut the size by a factor of 5 or even 10. But compression is the simple part – it’s not the issue here. The problem is, once we compress chunks, their size will no longer be the same. This opens up a Pandora’s box of pain. Fragmentation. Splitting chunks into little bits to fit in between other chunks, which have became smaller in the meantime. Etc.

Basically, what we need is a proper filesystem, with chunks being files with random size, that can both grow and shrink. And it has to be RELIABLE. Application on a phone leads a perilious life. It can, and frequently does die with no warning, at any second, due to user pressing the home button, or system running low on memory, or the battery going flat. What if it dies while chunks are being written to disk? At the moment, at most one chunk might get corrupted, and even this is unlikely, as chunk write is nearly always an atomic operation. Either it all goes in, or nothing does.

With the new complex format, we will suddenly be opening ourselves to complete mangling of the chunks file in case something goes wrong. As a result, the world will become lost forever. This cannot be.

This is what I am wrangling with at the moment. It’s proving to be more complex that I though it will be. We actually need a journaling filesystem, as ext4 or NTFS are.

One might question, why don’t we just use the Android’s or iOS’s filesystem, and save chunks in separate files? The answer is performance. File open operations are very slow, especially so on mobile systems. Even desktop Minecraft doesn’t do it this way. It combines multiple chunks into one “region” file. Plus, I’d rather not make myself a hostage to particular implementations of filesystems on various phones, which may become faster or slower over time, with no control from our side. I’ve been there before, and no, thank you, never again.