We haven't run into this error before and the documentation on InstallUtil is quite scarce, so I'll suggest a few things to try:
* Check that you're running the installer with administrative privileges. When you open the VS Command Prompt, do so as an administrator if your account is not an administrator.
* Unblock the ServerService.exe and other assemblies if by chance they were copied from elsewhere and aren't trusted.
* Make sure that you're using the .NET 4 version of InstallUtil.
If none of the above helps, can you tell us which version of DevForce you're running, and which OS?