I've been using Lua for quite some time but I've only just begun learning how to embed Lua, so my reply may not be completely correct. I did come from MTA:SA and over there we use plugins too (except we call them "resources" instead of "plugins"). Right now we do use one lua VM per plugin/resource (I think that's what you meant when you said "one lua state per plugin") but later on we found that it was very frustrating when it came to libraries. As a result of this we've had a major fork in the project with loads of improvements, therefore I would recommend that you actually split it into one very big VM but sandboxing each plugin to an extent.
[strike]Here is a fancy image I've taken from the project documentation to show you what I mean[/strike] Sorry, I couldn't find the right picture, but this shows it a bit. But yeah, the point is, all the plugins run in one VM whilst being sandboxed. You get the security and the ability to share custom made classes between plugins. Resources for us are very high security (they are sandboxed, certain functions like os.execute are removed) but I'd be able to provide my insight if I knew whether you're intending plugins to have a limited access to the host server. Or an ACL system, allowing the ability to grant/limit access to functions in the VM.
Anyway like I said, I'm not experienced with the implementation very much but I do know people and I've asked the main developers of multiple mods I'm involved in that has a similar mechanism to what you're doing
Specifically, this and this.
[strike]Here is a fancy image I've taken from the project documentation to show you what I mean[/strike] Sorry, I couldn't find the right picture, but this shows it a bit. But yeah, the point is, all the plugins run in one VM whilst being sandboxed. You get the security and the ability to share custom made classes between plugins. Resources for us are very high security (they are sandboxed, certain functions like os.execute are removed) but I'd be able to provide my insight if I knew whether you're intending plugins to have a limited access to the host server. Or an ACL system, allowing the ability to grant/limit access to functions in the VM.
Anyway like I said, I'm not experienced with the implementation very much but I do know people and I've asked the main developers of multiple mods I'm involved in that has a similar mechanism to what you're doing
Specifically, this and this.