APIFuzzing - a lua plugin for fuzzing and checking the API

[Seadragon91]

APIFuzzing - a lua plugin for fuzzing and checking the API

Github
https://github.com/cuberite/APIFuzzing

Commands
fuzzing
checkapi

Fuzzing
The runme file has to be copied to the root folder of Cuberite.

Running

Under windows run the file runme.bat and it will startup Cuberite.
Under linux run the file runme.sh and it will startup Cuberite.

The server will be started and runs the console command fuzzing. If an crash occurs:

If an crash occurs:

• Under linux the script will automatically restart cuberite and run the command again
  
• Under windows, you need to close the debugger message box that will appear, then cuberite will start and run the command again
  

The message Fuzzing completed! will be printed in the console, if the plugin is finished. If an crash has occurred, in the home directory of the plugin will be a file named crashed_table.txt. It contains the class name, function name and the function call of all crashes.


CheckAPI

Start the server and run the console command checkapi. The plugin will be finished if the message CheckAPI completed! appears. The results, if any, are in the console output and in cuberite log files.

Features
It can catch:

• Syntax errors, indicates a problem in code generation of plugin
  
• Runtime errors, function doesn't exists, is not exported or flag IsStatic is missing in APIDoc
  

It checks the return types of the function call with the APIDoc


TODOs

• The function CreateSharedIgnoreTable in file tables.lua contains a list of classes and functions that are currently ignored
  
• Find a way to run the command fuzzing on startup again, for travis use Works now, but currently not much use as there are open issues for crashes already
  
• The table g_FalsePositives contains functions that are false positives, the generated code needs to be improved for them
  

[Seadragon91]

This a single output of the return type checker. What do you think?  :P

    [21:31:39] ########################## AMOUNT OF RETURN TYPES DOESN'T MATCH ########################################
    [21:31:39] 
    [21:31:39]                                              ## Code ##
    [21:31:39] 
cRoot:Get():GetDefaultWorld():SetBlock(10, 100, 10, 144, 0) cRoot:Get():GetDefaultWorld():DoWithMobHeadAt(10, 100, 10, function(a_BlockEntity) GatherReturnValues(a_BlockEntity:SetType(SKULL_TYPE_CREEPER)) end)
    [21:31:39] 
    [21:31:39] Class = cMobHeadEntity
    [21:31:39] Function = SetType
    [21:31:39] Got = number
    [21:31:39] APIDoc = nil
    [21:31:39] #########################################################################################################

[xoft]

You mean in the forum? Use shcode instead of code tag

[Seadragon91]

If I use shcode and then preview to see how it looks, it shows code tags, confusing 

[Seadragon91]

Want to work on the reports from APIFuzzing.


This are functions that all returns the passed strings. Manual export all of them?

- cEnchantments:
	- AddFromString
	- StringToEnchantmentID

- cWorld:
	- BroadcastChatLeave
	- BroadcastChatSuccess
	- BroadcastChatWarning
	- GetWorld
	- SetCommandBlockCommand

- cRoot:
	- BroadcastChatDeath
	- BroadcastChatFailure
	- BroadcastChatFatal
	- BroadcastChatInfo
	- BroadcastChatJoin
	
- cBlockInfo:GetPlaceSound

- BlockStringToType
- EscapeString
- NoCaseCompare
- ReplaceString
- SetCustomName
- StringToBiome
- StringToDamageType
- StringToItem
- StringToMobType
- StripColorCodes
- TrimString

Functions that are documented in APIDoc, but didn't found them in code. Okay for remove?

• cWorld:QueueSetBlock
  
• cEntity:GetRot
  
• cRoot:SetPrimaryServerVersion, GetPrimaryServerVersion
  
• cBlockInfo:RequiresSpecialTool
  

[xoft]

Might be easier to fix ToLua++ from generating those return values in the first place.

Yeah, remove that unused documentation. Those are obsolete functions anyway.

[Seadragon91]

Created a commit in a new branch in toluacpp 06f46c7f84e51cea405dc9ef5ec11abeeb141538

Code:

if self.is_parameter and string.match(self.type, "AString") then
return 0
end

I had to add a check for AString, because there are a few functions that returns a passed number as reference. For example AddFaceDirection, accepts 3 numbers changes and returns them.

The function ReplaceString breaks with this changes, it has returned all the passed strings back. This one could be fixed by manual exporting it, not sure if there are more functions. Only found this one with APIFuzzing, but currently it doesn't cover the whole api.

[xoft]

No, that's not the right fix. The proper fix should disable returning all `const &` parameters - those are always only-in, so there's no point in returning them.

[Seadragon91]

Changed it

Code:

-- Bail out, if is_parameter is true and it's a const &
if self.is_parameter and string.match(self.type, "const") and self.ret == "&" then
    return 0
end

Works correctly now.

[xoft]

Looks good.

Now if you search through src/Bindings/ManualBindings.cpp for the string "extra return value", you'll see the functions that should no longer need manual bindings.