WorldEdit

[xoft]

He's right in the youtube comment - even though it's 3.8 sec, it's still a drop in the tick rate, which is what he wanted to avoid. And we can't. Right now there's no way to split the paste job up into smaller units that could be laid out over multiple ticks.

[NiLSPACE]

If we'd create an API in WorldEdit that allows you to execute a string using loadstring, a plugin could override cBlockArea:Write to set an X amount of blocks instead of the whole lot.

[SBPrime]

Hi everyone.

NiLSPACE contacted me on youtube regarding WorldEdit for Cuberite. I took the liberty of looking at WorldEdit's source code. It looks like it would be possible to make it lag free. All you need to do is (I assume that commands in Cuberite are executed in the main thread same as in Bukkit):
1. Fork the command to a new thread
2. Store all block changes
3. Place the changed block in packages from a task thats running on the main thread.

In addition you also need to provide a safe way to read the world from non main thread.

[xoft]

Hello, welcome to the forum. Nice of you to come and look around

Actually, we have an object that represents an entire .schematic file (cBlockArea class) and it knows how to write itself into the world (cBlockArea:Write). It does so using extremely fast copy operation, which is several orders of magnitude faster than setting individual blocks, but on the other hand it cannot be sliced into smaller work packages effectively. And there's no point in offloading it to another thread, the world's chunkmap needs to be locked during the write operation, so the world tick thread wouldn't be able to run anyway.

What we'd need is something that can write the intersection of cBlockArea and a single specified chunk, then split larger cBlockArea writing into per-chunk packages using this new API function. It would even make sense to use it with the cWorld:ChunkStay() API, because that reports individual chunks being made available.

@SBPrime, in case you're interested, here is the documentation on the API accessible to Lua plugins: http://apidocs.cuberite.org/

[NiLSPACE]

@xoft, can't we use cBlockArea:Crop? With that you should be able to split the cBlockArea into multiple block areas, and thus can spread the write action over multiple ticks.

[SBPrime]

Ok I understand that its fast and it works directly on chunks (btw. NiLSPACE thats what I'm currently adding to AsyncWorldEdit). What I said was for global WorldEdit operations not only the paste operation. You have a lock while writing to the world and that causes lag.

<< IF COMMANDS ARE EXECUTED IN A NEW THREAD IGNORE THIS >>
You can also experience lag while you load the schematic into the object (or are the commands done in a different thread?). C++ is fast and there is no doubt about it but still you need to consider the IO operations while reading the file, therefore you might experience tps drop when you load the schematic into memory.

Regarding the block placing, thats exactly what I had in mind. You need to place the blocks in chunks, not all at once. Its best if you do it one by one adding a delay between each place to allow other threads world access.

[worktycho]

For us, the easiest way to ensure delays would be to use the scheduleTask function. You can schedule all the tasks ahead of time, and execute them in n tasks spread over m ticks. As for spliting up into chunks, it would be more efficient to create a write overload that allows writing of part of a blockarea. It would be relatively easy to add to the server.

[xoft]

@SBPrime We use threads extensively, but this particular thing won't work too well, because the Lua plugins are not thread-safe (by design), so in order to execute any plugin code, the plugin holds a (plugin-specific) mutex. Unfortunately that means that if the same plugin also registers a hook that comes from the world tick thread, the tick thread must block until the first call releases the mutex. There's no simple way around this, the plugins are forced into this pseudo single-threading. Changing that would require not only a substantial rewrite of the entire API, but also of the plugin writers' mindset - suddenly they'd need to be aware of multithreading issues that were hidden from them before, the learning curve would become too steep.

@NiLSPACE: I'm afraid the Crop operation would become the bottleneck in this - for each chunk you'd need to create a copy of the original area (duplicate it all), then crop it (reallocate and copy) and finally write it. As @worktycho said, it would be easier to create a WritePartial function.

[SBPrime]

Ok now I c. how it works and what you can and can't do using the current API. In my opinion you should add additional classes to the server API (and do not touch underlying logic!) that are used for threading. The plugins don't need to be multi-thread in the basic mode but if someone wants to use multi-threading allow it! If something fails its his/hers problem.

What classes I would add:
cLock --> simple mutex. It allows read write synchronization point
cReaderWritterLock -> a more complex mutex that allows single writer access or multiple readers
cSemaphore -> a multi thread barrier that allows X clients to enter into the critical section
cThreadPool -> a class that allows you to run a method in a new thread,

The bare minimum: cLock and a method to allow to tun a method on a new thread.

If you do not have any threading you will be able to implement the async mode to WorldEdit similar to first release of AsyncWorldEdit. The world editing is done in chunks (stage 2) and the blocks preparation (stage 1) is done on the main thread. This is ok for 75% of all situations, but its not enough for extremely large edits. Don't get me wrong because you are using C++ as the base it will work for much larger edits then in Bukkit/Spigot.

[NiLSPACE]

There is an issue on Github about Async plugins: https://github.com/cuberite/cuberite/issues/2071

If we can make cBlockArea paste only an X amount of blocks, for example from 0, 0, 0 to 10, 10, 10 then we could simply spread the whole edit over multiple ticks.