Preparing gRPC for Unreal Engine 4

It is possible to build gRPC for a wide range of languages. However, this post will focus on C++ since it is the programming language used for Unreal Engine projects.

This guideline can still be used for cases other than Unreal, with the appropriate changes.

Table of Contents

1. Files required by Unreal
2. Generating the files
3. Integrating gRPC with Unreal
4. Generating C++ code from a proto file
5. Integrating the proto API with Unreal
6. FAQ
   • How to build gRPC using Visual Studio 2019?
   • Cannot link, symbol defined multiple times
   • Other unknown link errors

Files required by Unreal

By default, building gRPC will produce plenty of libraries and binaries, but only a handful of them will be required.

Also, the gRPC repository references and links to several other repositories, including OpenSSL and zlib. However, Unreal already provides them as ThirdParty modules in the engine, so we can ignore them.

In order to integrate them into an Unreal project, we need to get ahold of, at least:

• Includes
  • /google/protobuf
  • /grpc
  • /grpcpp
• Libraries
  • address_sorting.lib
  • cares.lib
  • gpr.lib
  • grpc.lib
  • grpc++.lib
  • libprotobuf.lib
  • upb.lib
• Binaries
  • protoc.exe
  • grpc_cpp_plugin.exe

The binaries will be necessary to produce C++ code from .proto files, that will use gRPC.

Generating the files

The official repository for gRPC can be found here:

grpc/grpc

The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#) - grpc/grpc

grpcGitHub

The problem is: the process to build it is tedious and prone to errors. Fortunately, it can be automatised and generated on the background.

At first, I tried to build the thing from scratch, following one of the many instructions provided in the repo. After some time being unsuccessful, I came across Vizor Infraworld, "a solution that enables Unreal Engine 4 to work with Google gRPC services from either C++ or Blueprints."

Their repository contains a batch file that will automatically download the gRPC repository, build the Visual Studio solution and extract the appropriate files at the end of the process.

A customised version of their Windows script can be found here:

Batch file that clones gRPC, builds it for Windows and extracts the necessary includes, libs and bin files out of it.

Batch file that clones gRPC, builds it for Windows and extracts the necessary includes, libs and bin files out of it. - grpc_setup.bat

262588213843476Gist

It also contains a shell script for GNU/Linux, although I have been running with issues trying to get it work properly. I'll update this post once I have a clean version of it.

Integrating gRPC with Unreal

Next, we need to create the appropriate module boilerplate code for gRPC. This consists, amongst other things, on a .Build.cs file.

Spoiler alert: the module won't build right out of the box. I found this article, GRPC with UE4 on Windows, which suggests a series of changes that will sort the problems.

These definitions are necessary to avoid some of the compile errors that will pop-up.

PublicDefinitions.Add("GOOGLE_PROTOBUF_NO_RTTI=1");
PublicDefinitions.Add("GOOGLE_PROTOBUF_USE_UNALIGNED=0");
PublicDefinitions.Add("GPR_FORBID_UNREACHABLE_CODE=0");

To link with Unreal's ThirdParty libraries, add:

AddEngineThirdPartyPrivateStaticDependencies(Target, "OpenSSL");
AddEngineThirdPartyPrivateStaticDependencies(Target, "zlib");

Generating C++ code from a proto file

With protoc.exe and grpc_cpp_plugin.exe generated, it is now possible to generate the appropriate C++ code from any given .proto file.

In order to produce such files, it is necessary to run the following from the command line:

%GRPC_BIN_ROOT%\protoc --proto_path=%PROTO_PATH% --grpc_out=%OUTPUT_DIRECTORY% --plugin=protoc-gen-grpc=%GRPC_BIN_ROOT%/grpc_cpp_plugin.exe %PROTO_FILENAME%.proto
%GRPC_BIN_ROOT%\protoc --proto_path=%PROTO_PATH% --cpp_out=%OUTPUT_DIRECTORY% %PROTO_FILENAME%.proto

Where:

• %GRPC_BIN_ROOT%: Path where protoc.exe and grpc_cpp_plugin.exe are located.
• %PROTO_PATH%: Path of the proto file.
• %OUTPUT_DIRECTORY%: Path where the resulting .h and .cc files will be stored.
• %PROTO_FILENAME%: name of the .proto file to be processed.

A batch file that will automatically do this can be found here:

Batch file that creates the appropriate C++ code from a .proto file

Batch file that creates the appropriate C++ code from a .proto file - generate_cpp_from_proto.bat

262588213843476Gist

Integrating the proto API with Unreal

Similarly to the integration of gRPC into Unreal, this one will also fail to compile at first. The aforementioned article, GRPC with UE4 on Windows, contains other fixes that still need to be implemented.

Basically, the following needs to be called before including any gRPC code.

#pragma warning (push)

// forcing value to bool true or false
#pragma warning (disable : 4800)
// decimal digit terminates octal escape sequence
#pragma warning (disable : 4125)
// behavior change __is_pod has different value in previous version
#pragma warning (disable : 4647)
// 'symbol' is not defined as a preprocessor macro,
// replacing with '0' for 'directives'
#pragma warning (disable : 4668)
// constructor is not implicitly called
#pragma warning (disable : 4582)
// destructor is not implicitly called
#pragma warning (disable : 4583)
// reinterpret_cast
#pragma warning (disable : 4946)

// Reader beware: this should only be done if the
// compile platform AND the build target are Windows.
static void MemoryBarrier() {}
#include "AllowWindowsPlatformTypes.h"
#pragma intrinsic(_InterlockedCompareExchange64)
#define InterlockedCompareExchangeAcquire64 _InterlockedCompareExchange64
#define InterlockedCompareExchangeRelease64 _InterlockedCompareExchange64
#define InterlockedCompareExchangeNoFence64 _InterlockedCompareExchange64
#define InterlockedCompareExchange64 _InterlockedCompareExchange64

And, after the relevant subsequent code, append the following line:

#pragma warning( pop )
// Reader beware: this should only be done if the
// compile platform AND the build target are Windows.
#include "HideWindowsPlatformTypes.h"

FAQ

How to build gRPC using Visual Studio 2019?

In the call cmake .. -G "Visual Studio 15 2017 Win64", replace the parameter with "Visual Studio 16 2019".

By default, the builds in 2019 will be targeted for x64. It is possible to build for 32-bits by using this instead: cmake -G "Visual Studio 16 2019" -A Win32.

For more information, read here.

Cannot link, symbol defined multiple times

By default, the gRPC repository pulls and uses the OpenSSL repository in order to build libprotobuf.lib. However, Unreal already includes such library, which leads to the redefinition error.

Make sure that CMake is called pointing to the path where Unreal's OpenSSL is located.

The following arguments are required:

• -DgRPC_SSL_PROVIDER=package: This tells CMake that we will use an external OpenSSL library.
• -DOPENSSL_INCLUDE_DIR="%PATH%"
• For libcrypto.lib:
  • -DLIB_EAY_LIBRARY_RELEASE="%PATH%"
  • -DLIB_EAY_RELEASE="%PATH%"
• For libssl.lib
  • -DSSL_EAY_LIBRARY_RELEASE="%PATH%"
  • -DSSL_EAY_RELEASE="%PATH%"

Where %PATH% needs to be replaced with the appropriate path of each argument.

This shouldn't be an issue with the provided scripts since they are already doing this.

Other unknown link errors

This happened the first times I tried to build gRPC using VS2019. The solution I found at the time was to build it using VS2017 instead.

However, it eventually worked alright for 2019. IIRC, this was after I stopped trying to build gRPC by myself and instead used the batch file mentioned in this post, so it might not be an issue for everybody.