Towards 1.13

[xoft]

Mojang is already giving out 1.14 snapshots, so I'd say it's about time we started working on a 1.13 support

This will most likely be a huge challenge for us. For those who haven't seen it yet, the 1.13 update changes the way Minecraft works with block in such a way that it breaks more or less everything (possibly this is the reason why 1.13 took even Mojang so long to get right - over a year in development!) The most breaking change is that the blocks are no longer represented by their type and meta (12 + 4 bits), but rather by their name and state (string + map of key->value). In order to make this change, most Cuberite sources will be affected hugely. Almost everything block-related will need to be rewritten, sometimes from scratch. Most APIs that we used will need to change, so all plugins will break as well. Since the change is so massive, it begs the question whether to try for backward compatibility, as we always have, or just cut away and support only the 1.13+ from this moment onwards. Personally, I think cutting away the old version support is the way to go, since we are already stretched way too thin, we don't want any extra development burden coming from the compatibility code.

I have thought hard about the new way of storing the world block data. In the end, I think I've found a way to make it work with as little work as possible (but the work is still horribly large). When looking at blocks, we must not only consider the chunks themselves, but also the BlockArea structure used for manipulating multiple block at a time (such as from plugins) and the terrain generator. We don't want to implement three (or more) different storages for these (and other) systems, so the natural solution comes to mind: Use BlockArea instances for everything. The BlockArea class should be the one that handles the block storage, translation, merging etc.; the others should rely on it to do its job, rather than implement their own solutions. Specifically, this means getting rid of the AllocationPool that we use for the Chunk data, and handling out-of-memory issues in another way.

So the first step is to implement a BlockArea class to support the new way of storing blocks. Then add a 1.13-compatible world loading and saving (but keep the older versions so that worlds can be migrated), and change the terrain generators to use the new BlockArea, rather than the old one. Finally, change the Chunks to store blocks using the new BlockArea, change the simulators to support that, implement a 1.13 protocol that uses low-level access to the BlockArea's internals for sending the data to the clients and drop all the old protocols.

Since most block-related API functions now handle blocks as a pair of BlockType and BlockMeta values, they will need replacing. This will break all plugins, so as a side note, I have a proposition. The naming convention of Cuberite classes has been bothering me since forever, and this is the perfect moment to make things right - we're already breaking compatibility, so we can break it thoroughly. How about dropping the "c" prefix to all classes? And while we're at it, how about switching to a style that has long been used and is proven good? Personally, I like the Java-like style with membership prefixes (ClassName, doSomethingFunction(), mMemberVariable, aParameter). Once changed, we could simply identify plugins that use the old API by having wrong function names (old: Initialize(), new: initialize()) and it should be rather easy to add a check to the Lua engine for any access to cClassName to emit a warning and disable the plugin. In the end, it would help us with migrating the plugins. Yes, there will be pain, there will be gripe, but I can't think of a better way, things need to change).

Concerning block types: Right now we have a fixed set of known blocks (E_BLOCK_...) and there's no way for plugins to add new blocks. With the 1.13 rewrite, we could have a "Block type registry" where block types would get registered together with their properties (need to find out what properties to track), so there's only a small additional step to allow plugins to register their custom block types in the registry as well. Once we have the registry, we'll have all the pieces missing - we already have ways for plugins to register callbacks, and we have callbacks capable of being unloaded. A similar approach will be needed for item types.

Simulators are most likely need major re-thinking. Right now most of them work by the logic of "this BlockType doesn't concern me", "this BlockType produces this BlockType on this change" etc. They will need to take into account the block type registry. It would be great to open the simulators to plugins as well, but this could be done later on.

In more detail: As the BlockArea class will be the center piece of the new "design", we will need to take care implementing it and making it as performant as possible. It needs to store an array of blocks of any (reasonable) size, and it needs to manipulate that array. The easiest way of doing this is to allocate an array of numbers, one number per block, and the numbers pointing to a look-up table (palette) that stores the block name and state (kinda like the Cubeset file format we already use for loading / saving). This could even be gradual - first implement only 32-bit numbers (2^32 different block types in a single BlockArea instance should be more than enough), make it work, and only then optimize by implementing shrinking to smaller numbers. Even then, it might not make sense to use weird-bit numbers (such as 3-bit or 5-bit) because then the manipulation of such representation could become too CPU-intensive (subject to tests . Of all the BlockArea operations, the merging is the one I fear the most. Imagine: two block areas, each with a different palette and different number size, need to be merged. How to do that efficiently?

So the basic operations on a BlockArea are:
- Get Block
- Set Block
- Merge another BlockArea using a bounding box

Of course, specialized operations will still be needed - such as when a terrain generator creates a new chunk (one 16x256x16 BlockArea) it needs to get split into the 16 sections (16x16x16 BlockAreas) for Chunk representation, and this will occur rather frequently, so it makes sense to have a "subArea" operation. However, they can at first be implemented through merging.

To sum up, these are the changes:
- Block type (and Item type) registry
- BlockArea based on a palette
- Load world into BlockArea-based structure
- Generate world into BlockArea (and provide empty old-style chunks)
- Switch everything from old cBlockArea to new BlockArea
- optional: rename API symbols
- Optimize BlockArea, both CPU-wise and RAM-wise

I guess the proper way to do this is to start a new git branch with all the changes, so that multiple people can contribute at the same time, and finally replacing the master branch with the new one.

I'm quite looking forward to this, and I hope I'm not the only one

[xoft]

Seriously? No answer?

[NiLSPACE]

I wasn't able to check the forums yesterday, sorry.

Personally, I think cutting away the old version support is the way to go ..

I agree. Maintaining the older versions would become too much of a hassle.

... Use BlockArea instances for everything. ...

If I'm completely honest I was surprised cBlockArea wasn't used for it already. What were the reasons for not unifying them before?

... Personally, I like the Java-like style with membership prefixes ...

I've been programming C# for years now which has the following convention: ClassName, MethodName, _fieldName, argumentName, localVariable
I personally like that as well.

Concerning block types: Right now we have a fixed set of known blocks (E_BLOCK_...) and there's no way for plugins to add new blocks. With the 1.13 rewrite, we could have a "Block type registry" where block types would get registered together with their properties (need to find out what properties to track), so there's only a small additional step to allow plugins to register their custom block types in the registry as well. Once we have the registry, we'll have all the pieces missing - we already have ways for plugins to register callbacks, and we have callbacks capable of being unloaded. A similar approach will be needed for item types.

This is something I wanted years ago because I wanted to recreate a bukkit plugin If it would be possible I'm all for it!

... This could even be gradual - first implement only 32-bit numbers ...

Would it be possible to dynamically change the size? For example if only 100 blocks are registered we could change all the numbers to 8 bit numbers, but if they use more we'd automagically use more bits.


I'm having trouble visualizing how block manipulation would happen. Would we return strings to specify a block type? What happens when two plugins register a type of the same name?

[xoft]

If I'm completely honest I was surprised cBlockArea wasn't used for it already. What were the reasons for not unifying them before?

Two reasons:
1. cBlockArea was created much later than cChunk, for different initial purposes (loading and saving prefabs)
2. Until now the chunk sections are exactly the same memory size, and we are using this to provide a special allocator just for the sections (cAllocationPool) that was meant to provide an out-of-memory and almost-out-of-memory handling.

I've been programming C# for years now which has the following convention: ClassName, MethodName, _fieldName, argumentName, localVariable

So we agree on ClassName and localVariable; partially on argumentName (by forcing an "a" prefix). Whether methods use first capital or not is not a big issue with me, but what I resent is the underscore. It is the most unfortunate character - its very low visibility makes it too easy to mix up with a space. I'll be adamant about having as little underscores as possible.

... This could even be gradual - first implement only 32-bit numbers ...

Would it be possible to dynamically change the size?

That's the general idea - for each storage use only as few bits as possible. For performance reasons, we may want to limit to 4-, 8-, 16- and 32-bit numbers. There are special cases when we'll need to use more bits than actually needed - when merging two BAs.

I'm having trouble visualizing how block manipulation would happen. Would we return strings to specify a block type? What happens when two plugins register a type of the same name?

I was thinking something like this (in a *single* BlockArea):

/** Sets a single block using its full blockspec */
void setBlock(const Vector3i & aPos, const AString & aBlockName, const BlockState & aBlockState);

/** Sets a single block using an index to the palette (retrieved earlier by paletteIndex()). */
void setBlock(const Vector3i & aPos, UInt32 aPalleteIndex);

/** Returns the index into the palette that is used by the specified full blockspec.
Adds the blockspec to palette if not already there. */
UInt32 paletteIndex(const AString & aBlockName, const BlockState & aBlockState);

/** Returns the index into the palette that is used by the specified full blockspec.
Returns <undefined, false> if blockspec not in palette. */
std::pair<UInt32, bool> maybePaletteIndex(const AString & aBlockName, const BlockState & aBlockState);

/** Returns the index into the palette for the block at the specified pos. */
UInt32 blockPaletteIndex(const Vector3i & aPos);

/** Returns the full (copy of) blockspec of the block at the specified position. */
std::pair<AString, BlockState> block(const Vector3i & aPos);

/** Returns (an editable (?) blockspec represented by the specified palette index. */
std::pair<AString &, BlockState &> paletteEntry(UInt32 aPaletteIndex);

[xoft]

Two plugins will not be allowed to register the same block name. The second plugin will fail the registration. The registry will remember the name of the plugin that registered the block type name (mainly to allow un-loading and re-loading the plugin).

[NiLSPACE]

What if a plugin wants to override Cuberite's default behavior? Would we allow that?

.. but what I resent is the underscore. It is the most unfortunate character - its very low visibility makes it too easy to mix up with a space. I'll be adamant about having as little underscores as possible.

Fair enough. I find it useful in C# as I can type a single underscore and IntelliSense instantly provides all the member variables, but I'm not sure if it works equally well in C++ or another IDE. It's been a while since I've touched any C++ code

[ShadowCone]

Meh.... Hint hint, normally an ide gives you all the members when typing the dot (or dash greater than or whatever else access operator)

Edit: I guess I should read the whole thread and reply. Might take some time.

[xoft]

What if a plugin wants to override Cuberite's default behavior? Would we allow that?

Not initially. But I like the approach taken by the Minetest team, that all blocks, items and interactions are handled by plugins, so the final state of things could be that the vanilla behavior is completely implemented by a plugin (and we could have a different plugin for Minecraft and another one for Minetest, ...) However, this is not the main point of the 1.13 update, only a sweet positive side-effect possibility.

Please let's not derails this thread with "who's got a bigger better IDE" chitchat.

[NiLSPACE]

But I like the approach taken by the Minetest team, that all blocks, items and interactions are handled by plugins, so the final state of things could be that the vanilla behavior is completely implemented by a plugin

Ah, yes, that could work!

[ShadowCone]

I agree with almost everything you wrote here.

About optimization: the idea of coding into numbers doesn't sound too bad to me, however we should think _a bit_ now about layout of the numbers + the storage, to either let the compiler do optimizations right away or do them later (I can think of, for example, if the total actual block state (maybe even including some surrounding blocks) is encoder in a single number, I'd happily trade the few bits for a cache of known state transformations.

One possible option: a plugin registers a block, which has a method similar to javas hash code that combines all relevant state into a single number, (which also defines whether two block states are equal and so on). This should definitely be a fast hash and computed rather often. Additionally, we have a tick function, which takes the current state plus the surrounding blockarea and transforms it into some other state. If a plugin does not depend on neighboring blocks, it can mark itself as isolated, in which case no surrounding blockarea is passed in.

If the surrounding area changed, the block is marked as dirty (well if a block changed, it's surrounding blocks are marked dirty), in which case tick needs to be re-evaluated. Block state changes which aren't dirty could be looked up in a table that maps previous state to next state.

It's just an idea, but with a little bit more work into it, it could make ticking large areas really efficient because i would guess a lot of water simulation eg is pretty similar and could thus be optimized.

Also, a possible option of optimizing would be looking into ways to optimize data layout and computation make the compiler produce SIMD instructions, as that could potentially result in a fairly high performance gain.