weblog/_posts/2023-09-10-build-a-game-engine.markdown

layout, title, author, categories, thumbnail, highlight

layout	title	author	categories	thumbnail	highlight
post	Build a game engine from scratch	JackCarterSmith	Programming C++ GameDev	build_an_engine	true

• Contents {:toc}

Creating a game inevitably involves choosing an engine to support our assets and mechanics. In 2023, the numbers of moderns game engines are quite large: Unity, Unreal, Godot, Cry, Source, idTech, OGRE, Monogame etc. Each one came with it's own features, and also it's own weakness. When I talk about benefits or consequences of an engine implementation, can be anything from what you can do with it to design rules.

When I started to look for an engine for my game rewrite idea, there are two questions that came up to me:

• should the engine adapt to the game, or the game to the engine?
• and why not design an engine?

To these questions, I've found many opinions and explainations, not a really precise answer. So after some comparisons and a pinch of personal opinion, I can now answer an "engine selection" checklist:

1. You have an idea of your game concept, artistic touch, mechanics and features.
2. Search for an existing engine that suits your game's needs. Check not only the advantages, but also the constraints.
3. If there is no matching engine, can you adjust your game's specifications to select a candidate?
4. If the previous answer is always no, then you should consider making your own game engine.

Just as your game exploits its engine, the engine exploits your skills to produce a game that more or less run So don't underestimate the importance of working your coding-skills. The advantage of having so many game engines today is that you can find engines to suit all levels! If you have already practiced programming, Unity or Godot can do the job for you. And if you didn't, GameEditor could be an option, but keep in mind that it's your programming skills that open up game design possibilities.

You can also choose to design your own engine just for the state-of-the-art aspect or as a personal technical challenge. It's not a bad choice, and it's the job of this series of articles to explain why I chose this option.

Be patient, really! Don't give up on setbacks.

Why use your own game engine?

Freedom! Since you define the design rules using your own engine implementation, you can create many more aspects than texture or shaders, such as physics or rendering methods. This is the major point in favor to make your own engine. There are several points that follow more or less directly from this:

• better game specific optimizations,
• mastering/how-its-work low-level aspects (fast debugging),
• no time spend reading engine documentation to find out how to implement our feature,
• no generic features overhead you didn't need in your game (like multiplayer protocol on singleplayer game),
• no engine licensing constraints (others than the lib you used inside of it).

From my personnal opinion, the biggest flaw I see in existing engines, which is ironically their strength, is their versatility! In order to enable a wide range of uses for their engine, these come with an abstraction layer that not only makes the engine (and the game as a result) heavy in terms of disk size, but also in terms of performance. Some allow you to modify the rendering engine to a certain limit, in order to adapt to the needs of devs, but this requires work that is not negligible when compared to the dev time of an engine. And when it comes to file size, you have to bear in mind that the size of the Unreal Engine (UE), compiled on its own, is close to a gigabyte of disk space...

Indirectly, you'll learn more than any engine users as you need to develop others aspects you may not have thought of before like geometric computation and physics algorithms. As you build your own implementation of physics and rendering pipeline, you feel more confident about what your game is going on, you know what's going on behind the scenes! Yes, you'll take time to develop (again) all the basics elements, but it's an advantage you'll keep throughout your development for this and future projects.

Like athletes, warming up with small exercises helps you keep up the pace during the real effort.

Yeah but... It's not all benefits either. And these are all good reasons that should be taken into account when choosing between a home-made engine and an existing one.

Firstly, the biggest problem is the time. Making a game engine from scratch require a lot, a loooooot of time before displaying your first image on screen (as you'll see in the following articles, it took me almost one year to display something that suited me!) and an another waiting after that to make it playable through an unoptimized release! So... Another advice: be patient, really! Don't give up in the face of setbacks, because yes, you'll have them, and it's even important to have them in order to understand and better manage what comes next.

You've an impressive numbers of features to implement in order to make just the game rendering something on screen. And the less comfortable you are with the language, the operation of a GPU and its drivers, memory management, 3D on a 2D surface... The more you'll have to retrace your steps to code these interfaces rather than the game itself. I think, this is why creating an engine is quite complex.

All vast subjects like 3D engine require discipline and rigor to understand and to master it.

Engine development choices

For any engine, you need to interface with your GPU. Long time ago, every GPU vendors implements its own interface to allow user to make thing on screen. For each GPU in the worlds you need to have a portion of code to implement it! That's really annoying... And that's why, a few years later, OpenGL and then Microsoft DirectX, have set up an API to standardize everything and every GPU since feature an interface level for OpenGL, Vulkan and DirectX. All the low-level hardware interface with GPU is done through these APIs, like memory transfer at chip-level and others DMA access.

And for my first engine, I have choosen as API... DirectX... DX-12 yup.

I can't say today if it's a good or very bad choice, I've read everywhere about starting with OpenGL as a first engine for a lot of good reasons:

• easy to handle
• lot of documentations
• portable between OSs

Because yes, DirectX is proprietary to Microsoft and can be used only on Windows environnement! And as if that weren't enough: the version 12 is considered to be the most complicated to apprehend because of the redesign of the graphic pipeline to take full advantage of a multi-threaded environment. And why not older simple version like DX9? Even if I learned DX9, I'd to spend more time unlearning a lot to adapt to DX12. Bearing in mind that, with the exception of DX11 and 12, the other versions are considered obsolete for a new design. Although this goes against what I was saying about the importance of doing small exercises before the big effort, I can explain it.

I really want to focus on the mechanics of a game engine, and how it do the things. DX12 leaves memory management to the developer and you need to understand how the resources are shared between GPU, CPU and RAM. Since I'm primarily interested in designing a state-of-the-art, technically challenging engine, using DX12 ask more times and efforts but it's worth it! I get a better management of what the GPU and CPU does rather than letting the driver "do its thing". And it's always possible to add a handler to manage both DirectX and OpenGL in function of the running OS or the user's choice. We'll talk about that in a future post.

No matter the API you choose, you need one more important things: a good programming language! And no, Python is not. Making a game in Python may be fun, but it's not really technically interesting, it's not designed for that. Python will slow your game, your dev and it's not adapted to debug properly DX or GL interface. Java can do the job, like Minecraft using OpenGL, but it's a virtual-machine language. The optimization level is a pain in ass. C# can have it's chance, used by MonoGame, the performance are greats and compatible with DX (also know as SharpDX). But C# is a proprietary language of Microsoft. If I choose C#, I'll get more troubles to debug my code and I definetly close the door for porting code to other OSs. And C++? A lot of books about DX use this language, it's cross-platform, always documentated and in development, multi-threads ready, objects oriented and have a lot of existing usefull library like Boost. In addition a lot of examples of games codes are made in C++.

Here's a list of the others tools I used during my development, which I will detail if necessary:

• Visual Studio 2022 - IDE with graphics oriented tools
• CMake - Build manager
• Conan - Dependencies manager
• Git - File version control
• PIX - GPU rendering debugger and analyzer
• RenderDoc - Another GPU rendering debugger

Let's go for a sadomasochistic adventure with C++ and DX12. What could go wrong?!

Now let's talk less and code more! See you in next post ==>

Some books to begin

Rather than list books like that without much explanation, I'll mention which books I've read and what they've brought me.

The first book I've read and revisit most often is 3D Game Programming with DX12 by F.D.Luna. This book cover all the mains topics about DX12 usage with some examples. The only downside, is that it's not easy to know exactly what functions all the elements indicated as structures or methods perform. I needed to read again several times and experiment directly in a program to understand the use of a feature. But beside of that, the book is very comprehensive for running a rudimentary engine, with simple lighting, shadows and textures.

I've got the DX9 and DX11 editions of this book for comparison and... Same topics, just adapted for DX11 and DX9 pipeline. DX9 has far fewer subjects because of its lack of shaders prior to 9.0c version.

The second book is not really related to game engine, because it's the Holy Bible of C++11 by B.Stroustrup! A complete dictionary of all the C++ features and keywords, really usefull when I need to fully exploit a function from this language. Or maybe choose an another option to my issue than the one I thought earlier. B.Stroustrup said that C++ was a language that had to respond to a problem with a language feature!

I plan to buy the new edition of C++17 soon to benefit from interesting multithreading features.

I also read two other books: Game Engine Architecture 3rd Ed. by J.Gregory and Game Programming Patterns by R.Mystron. Both doesn't explictly show what to code, but how to code a good game engine. The first book describe each parts of a "modern" game engine, like physics, audio, user inputs, files management, graphics (with DX in my case), etc. The second one talk about the abstracts concepts of game engine like object memory management, pointers good practice, game loop, and others importants things that we don't necessarily think about, but which are necessary for the proper operation of a game engine.

On top of that, you have the online DX12 documentation which will go into greater detail than F.D.Luna's book. Same advice for the Windows API to manage the window and the system messages.

All these books have helped me a great deal in clarifying aspects of a game engine that I couldn't see at first. But there's no one right way to design a game engine. Unlike a calculator program, where the sole aim is to perform calculations as efficiently as possible, our engine must meet several needs in terms of functionality to match the game we want to make. Most books and examples only give us one way of doing things, although sometimes there's no other way than to go through this or that method to achieve a task, the rest is up to our own judgement.

I really like the low-level programming because it's like a puzzle-game to make a game.

My last piece of advice is to take inspiration from existing engines. And if the source code is available, try to understand why it's implemented the way it is. Others previous developers have run into the same problems you're about to encounter. For the time being, it would be a waste of time (and possibly of motivation) to repeatedly get our feet stuck in the carpet. Above all, the aim of this blog is to share successes and mistakes, that's what knowledge is all about.

It's (not) a trap!

This is a recurent chapter in a large part of my topics to describe, from my own experience and point of view, the points on which I may have found myself stuck, or which at some point required me to pause and reflect.

Use them as you like!

When I choose to use DX12 as my first API for a 3D engine, the price to pay for that is to take even more time to complete the first draw on screen. That's part of the reason why it took me a year to achieve this result. Today I'm on the 4th attempt/iteration, and that's not counting tests on test code apart from the engine sources. I'll give you all details in upcoming posts!

So why not using Vulkan if I wanted a challenge? Some reasons for that: Vulkan isn't so well documented than DirectX, or at least it was easier to find DirectX docs and books. And it's for me a good sign to avoid or to limit the demotivation effect: the more time and energy you invest in something that doesn't bring you any satisfaction within the limits of your patience, the more you give up or make a break on the project. The greater the challenge, the greater the risk of breaking. This is normal, and it's through this kind of failure that we learn to know our limits. So I keep Vulkan as a next feature or evolution for now.