01-08-2019, 01:04 AM
Mojang is already giving out 1.14 snapshots, so I'd say it's about time we started working on a 1.13 support 
This will most likely be a huge challenge for us. For those who haven't seen it yet, the 1.13 update changes the way Minecraft works with block in such a way that it breaks more or less everything (possibly this is the reason why 1.13 took even Mojang so long to get right - over a year in development!) The most breaking change is that the blocks are no longer represented by their type and meta (12 + 4 bits), but rather by their name and state (string + map of key->value). In order to make this change, most Cuberite sources will be affected hugely. Almost everything block-related will need to be rewritten, sometimes from scratch. Most APIs that we used will need to change, so all plugins will break as well. Since the change is so massive, it begs the question whether to try for backward compatibility, as we always have, or just cut away and support only the 1.13+ from this moment onwards. Personally, I think cutting away the old version support is the way to go, since we are already stretched way too thin, we don't want any extra development burden coming from the compatibility code.
I have thought hard about the new way of storing the world block data. In the end, I think I've found a way to make it work with as little work as possible (but the work is still horribly large). When looking at blocks, we must not only consider the chunks themselves, but also the BlockArea structure used for manipulating multiple block at a time (such as from plugins) and the terrain generator. We don't want to implement three (or more) different storages for these (and other) systems, so the natural solution comes to mind: Use BlockArea instances for everything. The BlockArea class should be the one that handles the block storage, translation, merging etc.; the others should rely on it to do its job, rather than implement their own solutions. Specifically, this means getting rid of the AllocationPool that we use for the Chunk data, and handling out-of-memory issues in another way.
So the first step is to implement a BlockArea class to support the new way of storing blocks. Then add a 1.13-compatible world loading and saving (but keep the older versions so that worlds can be migrated), and change the terrain generators to use the new BlockArea, rather than the old one. Finally, change the Chunks to store blocks using the new BlockArea, change the simulators to support that, implement a 1.13 protocol that uses low-level access to the BlockArea's internals for sending the data to the clients and drop all the old protocols.
Since most block-related API functions now handle blocks as a pair of BlockType and BlockMeta values, they will need replacing. This will break all plugins, so as a side note, I have a proposition. The naming convention of Cuberite classes has been bothering me since forever, and this is the perfect moment to make things right - we're already breaking compatibility, so we can break it thoroughly. How about dropping the "c" prefix to all classes? And while we're at it, how about switching to a style that has long been used and is proven good? Personally, I like the Java-like style with membership prefixes (ClassName, doSomethingFunction(), mMemberVariable, aParameter). Once changed, we could simply identify plugins that use the old API by having wrong function names (old: Initialize(), new: initialize()) and it should be rather easy to add a check to the Lua engine for any access to cClassName to emit a warning and disable the plugin. In the end, it would help us with migrating the plugins. Yes, there will be pain, there will be gripe, but I can't think of a better way, things need to change).
Concerning block types: Right now we have a fixed set of known blocks (E_BLOCK_...) and there's no way for plugins to add new blocks. With the 1.13 rewrite, we could have a "Block type registry" where block types would get registered together with their properties (need to find out what properties to track), so there's only a small additional step to allow plugins to register their custom block types in the registry as well. Once we have the registry, we'll have all the pieces missing - we already have ways for plugins to register callbacks, and we have callbacks capable of being unloaded. A similar approach will be needed for item types.
Simulators are most likely need major re-thinking. Right now most of them work by the logic of "this BlockType doesn't concern me", "this BlockType produces this BlockType on this change" etc. They will need to take into account the block type registry. It would be great to open the simulators to plugins as well, but this could be done later on.
In more detail: As the BlockArea class will be the center piece of the new "design", we will need to take care implementing it and making it as performant as possible. It needs to store an array of blocks of any (reasonable) size, and it needs to manipulate that array. The easiest way of doing this is to allocate an array of numbers, one number per block, and the numbers pointing to a look-up table (palette) that stores the block name and state (kinda like the Cubeset file format we already use for loading / saving). This could even be gradual - first implement only 32-bit numbers (2^32 different block types in a single BlockArea instance should be more than enough), make it work, and only then optimize by implementing shrinking to smaller numbers. Even then, it might not make sense to use weird-bit numbers (such as 3-bit or 5-bit) because then the manipulation of such representation could become too CPU-intensive (subject to tests . Of all the BlockArea operations, the merging is the one I fear the most. Imagine: two block areas, each with a different palette and different number size, need to be merged. How to do that efficiently?
So the basic operations on a BlockArea are:
- Get Block
- Set Block
- Merge another BlockArea using a bounding box
Of course, specialized operations will still be needed - such as when a terrain generator creates a new chunk (one 16x256x16 BlockArea) it needs to get split into the 16 sections (16x16x16 BlockAreas) for Chunk representation, and this will occur rather frequently, so it makes sense to have a "subArea" operation. However, they can at first be implemented through merging.
To sum up, these are the changes:
- Block type (and Item type) registry
- BlockArea based on a palette
- Load world into BlockArea-based structure
- Generate world into BlockArea (and provide empty old-style chunks)
- Switch everything from old cBlockArea to new BlockArea
- optional: rename API symbols
- Optimize BlockArea, both CPU-wise and RAM-wise
I guess the proper way to do this is to start a new git branch with all the changes, so that multiple people can contribute at the same time, and finally replacing the master branch with the new one.
I'm quite looking forward to this, and I hope I'm not the only one
This will most likely be a huge challenge for us. For those who haven't seen it yet, the 1.13 update changes the way Minecraft works with block in such a way that it breaks more or less everything (possibly this is the reason why 1.13 took even Mojang so long to get right - over a year in development!) The most breaking change is that the blocks are no longer represented by their type and meta (12 + 4 bits), but rather by their name and state (string + map of key->value). In order to make this change, most Cuberite sources will be affected hugely. Almost everything block-related will need to be rewritten, sometimes from scratch. Most APIs that we used will need to change, so all plugins will break as well. Since the change is so massive, it begs the question whether to try for backward compatibility, as we always have, or just cut away and support only the 1.13+ from this moment onwards. Personally, I think cutting away the old version support is the way to go, since we are already stretched way too thin, we don't want any extra development burden coming from the compatibility code.
I have thought hard about the new way of storing the world block data. In the end, I think I've found a way to make it work with as little work as possible (but the work is still horribly large). When looking at blocks, we must not only consider the chunks themselves, but also the BlockArea structure used for manipulating multiple block at a time (such as from plugins) and the terrain generator. We don't want to implement three (or more) different storages for these (and other) systems, so the natural solution comes to mind: Use BlockArea instances for everything. The BlockArea class should be the one that handles the block storage, translation, merging etc.; the others should rely on it to do its job, rather than implement their own solutions. Specifically, this means getting rid of the AllocationPool that we use for the Chunk data, and handling out-of-memory issues in another way.
So the first step is to implement a BlockArea class to support the new way of storing blocks. Then add a 1.13-compatible world loading and saving (but keep the older versions so that worlds can be migrated), and change the terrain generators to use the new BlockArea, rather than the old one. Finally, change the Chunks to store blocks using the new BlockArea, change the simulators to support that, implement a 1.13 protocol that uses low-level access to the BlockArea's internals for sending the data to the clients and drop all the old protocols.
Since most block-related API functions now handle blocks as a pair of BlockType and BlockMeta values, they will need replacing. This will break all plugins, so as a side note, I have a proposition. The naming convention of Cuberite classes has been bothering me since forever, and this is the perfect moment to make things right - we're already breaking compatibility, so we can break it thoroughly. How about dropping the "c" prefix to all classes? And while we're at it, how about switching to a style that has long been used and is proven good? Personally, I like the Java-like style with membership prefixes (ClassName, doSomethingFunction(), mMemberVariable, aParameter). Once changed, we could simply identify plugins that use the old API by having wrong function names (old: Initialize(), new: initialize()) and it should be rather easy to add a check to the Lua engine for any access to cClassName to emit a warning and disable the plugin. In the end, it would help us with migrating the plugins. Yes, there will be pain, there will be gripe, but I can't think of a better way, things need to change).
Concerning block types: Right now we have a fixed set of known blocks (E_BLOCK_...) and there's no way for plugins to add new blocks. With the 1.13 rewrite, we could have a "Block type registry" where block types would get registered together with their properties (need to find out what properties to track), so there's only a small additional step to allow plugins to register their custom block types in the registry as well. Once we have the registry, we'll have all the pieces missing - we already have ways for plugins to register callbacks, and we have callbacks capable of being unloaded. A similar approach will be needed for item types.
Simulators are most likely need major re-thinking. Right now most of them work by the logic of "this BlockType doesn't concern me", "this BlockType produces this BlockType on this change" etc. They will need to take into account the block type registry. It would be great to open the simulators to plugins as well, but this could be done later on.
In more detail: As the BlockArea class will be the center piece of the new "design", we will need to take care implementing it and making it as performant as possible. It needs to store an array of blocks of any (reasonable) size, and it needs to manipulate that array. The easiest way of doing this is to allocate an array of numbers, one number per block, and the numbers pointing to a look-up table (palette) that stores the block name and state (kinda like the Cubeset file format we already use for loading / saving). This could even be gradual - first implement only 32-bit numbers (2^32 different block types in a single BlockArea instance should be more than enough), make it work, and only then optimize by implementing shrinking to smaller numbers. Even then, it might not make sense to use weird-bit numbers (such as 3-bit or 5-bit) because then the manipulation of such representation could become too CPU-intensive (subject to tests
. Of all the BlockArea operations, the merging is the one I fear the most. Imagine: two block areas, each with a different palette and different number size, need to be merged. How to do that efficiently?So the basic operations on a BlockArea are:
- Get Block
- Set Block
- Merge another BlockArea using a bounding box
Of course, specialized operations will still be needed - such as when a terrain generator creates a new chunk (one 16x256x16 BlockArea) it needs to get split into the 16 sections (16x16x16 BlockAreas) for Chunk representation, and this will occur rather frequently, so it makes sense to have a "subArea" operation. However, they can at first be implemented through merging.
To sum up, these are the changes:
- Block type (and Item type) registry
- BlockArea based on a palette
- Load world into BlockArea-based structure
- Generate world into BlockArea (and provide empty old-style chunks)
- Switch everything from old cBlockArea to new BlockArea
- optional: rename API symbols
- Optimize BlockArea, both CPU-wise and RAM-wise
I guess the proper way to do this is to start a new git branch with all the changes, so that multiple people can contribute at the same time, and finally replacing the master branch with the new one.
I'm quite looking forward to this, and I hope I'm not the only one