2.3 Bugfixes contd.

I will very soon release a bugfix update for the 2.3 release. Here’s the list of changes:

• Fixed broken uploading/downloading of 2.3 worlds
• Crouching deaths in low spaces fixed, general crouching improvements
• Reduced Sparrows spawn rate
• Different, more varied new world names
• Fixed glitches and improved wooden ladder
• Fixed depth clipping when crouching near a wall
• Hammer makes appropriate material sound when used to make furniture
• Reduced chance of spawning of many creatures of the same type
• Reduced number of pickables created by explosions
• Fixed cairn collision box

The version numbers of the update will be 2.3.10.1 (Google Play), 2.3.61.0 (Apple), 2.3.50.1 (Amazon) and 2.3.41.0 (Windows Store).

We will need to wait for the stores to approve the update – hopefully only about a day. It used to be over a week back in 2013…

Some of you have been complaining that motion sensors detecting movement of projectiles, pickables and moving blocks (a new feature in 2.3) broke their builds, because they were using pistons to block off visibility of sensors. As pistons are moving blocks, they trigger motion sensors in 2.3 – so the blocking no longer works.

Motion sensor

I think this is unfortunate, but the fact the sensor detects more movement is good. What I would do, is to move away from mechanical blocking of sensors, and use the electronic way instead. Like that:

2.3 bugfixes

I am working on the usual bugfix release for the new 2.3 series. Thank you for reporting all the problems you encountered. The most important one is the new crouch mechanics having various problems, such as dying when attacked or respawning in the low-ceiling area. This is what I am working on now – should be releasing that and other fixes soon.

Boring Optimizations

I think the most important change in 2.3 is that it uses a new, compressed storage format for the world files. I told you about it in one of the previous posts. It is a more complex system compared to the old one, which stored raw block values divided into constant-size chunks.

Here’s some details. The new format divides the world into regions of 16×16 chunks (where chunk is 16x16x256 blocks). Each region gets a separate file. The chunks within the region file are compressed using a two-step process:

Step 1. RLE compression to exploit the repetitive nature of the world blocks.

RLE compression stores the value of the block followed by the repeat count, instead of storing all blocks separately. The count value is stored in the “light” section of the block data, so it doesn’t take any more space than the raw, uncompressed block (light value does not have to be saved – it is generated anyway). Only if the count is too large to fit in the light section (4 bits), a separate byte is allocated for the count.

Additionally, the order of blocks storage is changed from how they are kept in memory. This is to maximize gains from the RLE scheme. Block values are more repeatable in horizontal directions than in the vertical. Deep down you get hundreds and hundreds of the same granite/basalt blocks, up in the air it’s all air, obviously.

Step 2. Standard deflate compression (the same as used by zip format) on the resulting RLE-compressed data

This does not need much explanation. Deflate has been the workhorse of compression for many decades now. There are more modern and better compression algorithms – like Zstandard, which is both faster and provides better compression. Maybe one day I will switch to using it. But Deflate is so ubiquitous and so easy to use – it is good enough for us at the moment.

The main complication with the new format is that chunks no longer have the same size – some compress more, some less. A simple fixed-size system, as used in 2.2, cannot work. Therefore, within each region file there is a mini-filesystem, which handles variable-size compressed chunks. It uses a simple overprovisioning scheme, which leaves extra 1024 bytes after each chunk to accommodate small changes. Once a chunk grows out of this extra space, the entire region file is rewritten, with the file enlarged and chunks rearranged to get their 1024 spare bytes again. The rewriting does not happen too frequently – a large change to the chunk data, such as an explosion or digging/placement of many blocks, is required to use up the 1024 bytes and trigger the rewriting.

The result is quite satisfactory. I’ve already shown you this:

World in 2.3 format

World in 2.2 format

2.78GB to 26MB – that’s over 100x reduction in size! But we badly needed that. The large number of crash reports I have from the game are almost always caused by the device running out of storage space. Followed by running out of memory, when someone with a low-memory phone ups the visibility range to the max…

Such changes are always risky – we don’t want world data losses. To reduce the chance of this happening, I used both automated and manual testing. For the automated testing, the world was randomly modified and camera randomly shifted in quick succession. This was left to run overnight – for many, many hours – stress testing the new format. For manual testing, me and friends played 2.3 in many different settings, with up to 4 players, for many hours in total.

As far as I can tell, it works. There are no bugs. But we’ll see once YOU start using it :-)

The new file format is not the only optimization in 2.3. Memory allocations in terrain engine have been massively reduced by use of caching. This should reduce garbage collection-related stuttering. In 2.2 and earlier, as the player was moving quickly, the chunks which were going out of visibility range behind the player were immediately freed, and memory for the new chunks was allocated afresh. In 2.3, the blocks of memory from old chunks are cached for a while to see whether the new chunks will fit in them, instead of allocating fresh memory.

Nothing too clever, but it hugely reduces pressure on the garbage collector. We had a system like that back in the old days of Alpha 1.xx around 2013, but I had to take it out, because back then memory was more precious than gold dust. The game had to run on some Apple devices with just 256MB of RAM. Fortunately, Apple has changed its ways, and the worst we can expect from them is 1GB on iPhone 6, or 512MB on iPad 2 – would you believe that scores of people still play on that ancient museum piece? That’s the sole reason I keep supporting iOS 9.x.

There’s more optimizations to do with light propagation as well. As usual, the full list of changes will be published soon on the Updates History page.

By the way, if you read all the way to here without getting bored: I released 2.3 earlier today on all platforms. Should already be in the stores, apart from Amazon, which is still in review. Amazon will probably be out tomorrow.

Have fun!

Mountaineering

Here’s a new video for you, shot high in the coastal mountains!

More Birds

You’ve noticed it right away. Flat blocks in 2.3 are rendered in extruded 3D style instead of flat. This only happens while the block is held in hand – pickables and projectiles in the world still display as billboarded particles. It looks better that way, I experimented with it. The code to extrude textures into 3D models was written by my son Mikołaj.

These are not 3D models. They are automatically generated from the blocks texture.
If you load a custom texture, the models will match.

Some more new stuff: 2.3 will include two new bird species. A sparrow and a pigeon. I have long thought that it is too easy to feed yourself hunting birds or eating their eggs. To make it more difficult, these two new bird species are not edible, and they are quite common, competing with the edible species. They still lay eggs, but their eggs are small (pigeon) and tiny (sparrow), and do not provide as much nutritional value as duck, raven or seagull eggs.

Sparrow. A tiny, agile, cute bird which lays equally tiny but edible eggs. Makes little chirping sounds that must be familiar to everyone on the planet (unless you live in the Antarctic?). Not edible. Lives everywhere apart from the very coldest places.

Pigeon. A small bird with a very familiar song. Lays larger eggs than sparrow, but is also inedible.

As a result, survival is harder. But you get the new default Survival mode instead. I think I told you about it already in one of the previous posts?

Sparrow is so tiny it can easily be killed with a rock, you don’t even need a spear. But why would you do that, it’s so cute!

All bird models have been updated to make them slightly more detailed and smoother to better match other animals:

I am almost ready to release, just a few more things.

Can you spot it?

Can you figure out what’s new in this screenshot from 2.3?

2.3 is coming out soon

The release is imminent. I have been busy working on other projects in the past months, but there’s a lot of changes in 2.3 ready to be released.

I have long wanted to make mining with explosives more viable than it used to be in 2.2 and before. The problem with explosives was that they were too expensive to make. It took a lot of gunpowder, which in turn depended on rare sulphur, and even more rare saltpeter. More importantly, explosions had a large chance of destroying the minerals instead of letting you pick them up after the blast. This is changing in 2.3.

And electric detonator has a new looks too:

The restriction of using certain tools to mine certain minerals has been removed. Now, explosion is just as good as a diamond pickaxe. Minerals pickables are not destroyed by explosions any more – they might fly all around the place, but they will not disappear. Plus, natural rocks have been made less resilient to explosives. And powder kegs are more powerful. All in all, I think blasting will be the best way to mine quickly.

Here’s a little video for you showing the process:

https://youtu.be/1Uz_Pz_suJA

If you watch carefully you might notice some other new things coming in 2.3…

SC Editor

Hi Everyone. One of the members of the community has been busy working on a world editor for Survivalcraft. It’s called SCEditor and there’s a beta version available for download here: https://drsmcraft.github.io/SCEditor/

I haven’t had much time to test it myself, but it seems to be functional and looks like it has full editing capabilities, down to the smallest details like block data values or obscure world options. Kudos to the creator. It even allows you to load worlds directly from Survivalcraft data folder. It’s still a beta though, so things may not always work as expected. Give it a try!

I have been temporarily sidetracked from working on the 2.3 update by a new project that came along this spring. This project is another multiplayer strategy game, using similar network protocol as Ruthless Conquest does, but more complex and advanced in every way. I want to get it finished before I get back to Survivalcraft. That hopefully shouldn’t take too long. So far I spent about a month doing base tech and another month on the game itself. It seems that the Coronavirus lockdown is very conductive to hard work :-)

 

New game mode

I have read your comments about the game difficulty. To fix it, there’s a new game mode coming in 2.3. It’s simply called “Survival” and will become the default game mode. “Harmless” is the easier variant of it. “Challenging” (the current default) is the more difficult.

Here’s what will be easier in “Survival” mode, compared to “Challenging”:

• Player will be significantly less susceptible to cold
• Player will be more resilient to injury (25%)
• Player will be able to place blocks when in the air
• Dangerous animals will spawn less frequently
• Player will get one cooked fish on game start

Please let me know in the comments if you think something else needs to be made easier for this new default game mode. Keep in mind that there’s still “Harmless” below it, and it gets all these benefits too, on top of what it already has.

Challenging mode difficulty stays unchanged. There’s also a new UI for game mode selection. The old way with a button that changes mode on every click is now too confusing, with so many game modes.

 

 

New world file format

In the last post I told you that we need a more efficient way to save the world data. Survivalcraft 2.2 doubled world height, and the data files have become too huge. For example, The Walking Dead world (one of the largest out there), occupies 1.4GB. And that is in 2.1 format, with chunks height of 128. Once you load it in 2.2, and it gets upgraded to the height of 256, the size doubles to 2.8GB. Ooops. This is too much:

I have experimented heavily with various new file formats. In fact, I have created three of them.

Here’s how the one I want to put in Survivalcraft 2.3 fares:

From 2.8GB to 26MB. Over 100x reduction! That’s better than I expected. The new format uses compression: a combination of custom RLE followed by a deflate. The RLE is clever, it uses the 4 bits of light data that does not need to be saved. The major issue is that the compressed chunks now have changing sizes, so simple array-style chunks table is out. More about the new format later.

Not all is rosy, however. The compression slows down loading, and most importantly, saving of chunks. The slowdown is considerable, about 5-10x. This is only to be expected, previously the game was just dumping memory direct to disk. Now, it needs to heavily process it first. The question is: does it affect the game? Fortunately, as far as I can see, not very much. Loading of chunks is not a problem, because it happens in a background thread anyway. Saving is more problematic, because it’s done on the main thread (to avoid having to keep chunk copies in save queue, memory is tight on the phones). Fortunately, saving happens infrequently. Plus there are various ways we can mitigate it, if needed.

The new system definitely needs more testing. I will be indirectly doing it when working on the rest of my todo list. At the moment it passes the brutal monster hot grind fatality crunch decapitation test (tens of millions of random chunk saves and loads).

I will check it in today and work on further items from my todo list for 2.3.