weblog/_posts/2023-09-10-build-a-game-engine.markdown

---
layout:     post
title:      Build a game engine from scratch
author:     JackCarterSmith
categories: Programming C++ GameDev
thumbnail:  build_an_engine
highlight:  true
---

* Contents
{:toc}

Creating a game inevitably involves choosing an engine to support our assets and
mechanics. In 2023, the numbers of moderns game engines are quite large: Unity, 
Unreal, Godot, Cry, Source, idTech, OGRE, Monogame etc.
Each one came with it's own features, and also it's own **weakness**. When I talk about
benefits or consequences of an engine implementation, can be anything from what you 
can do with it to design rules.

When I started to look for an engine for my game rewrite idea, there are two questions
that came up to me: 

- should the engine adapt to the game, or the game to the engine? 
- and why not design an engine?

To these questions, I've found many opinions and explainations, not a really precise answer.
So after some comparisons and a pinch of personal opinion, I can now answer an "engine selection"
checklist:

1. You have an idea of your game concept, artistic touch, mechanics and features.
2. Search for an existing engine that suits your game's needs. Check not only the advantages, but also the constraints.
3. If there is no matching engine, can you adjust your game's specifications to select a candidate?
4. If the previous answer is always no, then you should consider making your own game engine.

Just as your game exploits its engine, the engine exploits your skills to produce a 
game that more or less *run* So don't underestimate the importance of **working your coding-skills**.
The advantage of having so many game engines today is that you can find engines to 
suit all levels! If you have already practiced programming, [Unity][link_unity] or 
[Godot][link_godot] can do the job for you. And if you didn't, [GameEditor][link_gameeditor] 
could be an option, but keep in mind that it's your programming skills that open up 
game design possibilities.

You can also choose **to design your own engine** just for the *state-of-the-art* aspect
or as a personal technical challenge. It's not a bad choice, and it's the job of 
this series of articles to explain why I chose this option.

> Be **patient**, really! Don't give up on setbacks.

### Why use your own game engine?

Freedom! Since you define the **design rules** using your own engine implementation, you can
create many more aspects than texture or shaders, such as physics or rendering methods.
This is the major point in favor to make your own engine. There are several points 
that follow more or less directly from this:

- better game specific optimizations,
- mastering/how-its-work low-level aspects (fast debugging),
- no time spend reading engine documentation to find out how to implement our feature,
- no generic features overhead you didn't need in your game (like multiplayer protocol on singleplayer game),
- no engine licensing constraints (others than the lib you used inside of it).

From my personnal opinion, the biggest flaw I see in existing engines, which is ironically 
their strength, is their *versatility*! In order to enable a wide range of uses for their engine, 
these come with an abstraction layer that not only makes the engine (and the game as a result) 
heavy in terms of disk size, but also in terms of performance. Some allow you to modify the 
rendering engine to a certain limit, in order to adapt to the needs of devs, but this requires 
work that is not negligible when compared to the dev time of an engine. And when it comes to 
file size, you have to bear in mind that the size of the Unreal Engine (UE), compiled on its own, 
is close to **a gigabyte of disk space**...

Indirectly, you'll learn more than any *engine users* as you need to develop others
aspects you may not have thought of before like geometric computation and physics
algorithms. As you build your own implementation of physics and rendering pipeline,
you feel more confident about what your game is going on, you know what's going 
on behind the scenes! Yes, you'll take time to develop (again) all the basics elements,
but it's an advantage you'll keep throughout your development for this and future projects.

> Like athletes, warming up with small exercises helps you keep up the pace during the real effort.

Yeah but... It's not all benefits either. And these are all good reasons that should be taken 
into account when choosing between a *home-made* engine and an existing one. 

Firstly, the biggest problem is the time. Making a game engine from scratch require 
a lot, a loooooot of time before displaying your first image on screen (as you'll see in 
the following articles, it took me almost **one year to display something** that suited me!) and an 
another waiting after that to make it playable through an unoptimized release! So... Another advice: 
be patient, really! Don't give up in the face of setbacks, because yes, you'll have them, 
and it's even important to have them in order to understand and better manage what comes next.

You've an impressive numbers of features to implement in order to make just the game 
rendering something on screen. And the less comfortable you are with the language, the 
operation of a GPU and its drivers, memory management, 3D on a 2D surface... The more 
you'll have to retrace your steps to code these interfaces rather than the game itself.
I think, this is why creating an engine is quite *complex*.

> All vast subjects like 3D engine require discipline and rigor to understand and to master it.

### Engine development choices

For any engine, you need to interface with your GPU. Long time ago, every GPU vendors implements
its own interface to allow user to make thing on screen. For each GPU in the worlds you need to 
have a portion of code to implement it! That's really annoying... And that's why, a few years later, 
OpenGL and then Microsoft DirectX, have set up an API to standardize everything and every GPU since 
feature an interface level for OpenGL, Vulkan and DirectX. All the low-level hardware interface with 
GPU is done through these APIs, like memory transfer at chip-level and others DMA access.

And for my first engine, I have choosen as API... DirectX... **DX-12** yup.

I can't say today if it's a good or very bad choice, I've read everywhere about starting with 
OpenGL as a first engine for a lot of good reasons:

- easy to handle
- lot of documentations
- portable between OSs

Because yes, DirectX is **proprietary to Microsoft** and can be used only on Windows environnement! 
And as if that weren't enough: the version 12 is considered to be the most complicated to apprehend 
because of the redesign of the graphic pipeline to take full advantage of a multi-threaded environment. 
And why not older *simple* version like DX9? Even if I learned DX9, I'd to spend more time unlearning 
a lot to adapt to DX12. Bearing in mind that, with the exception of DX11 and 12, the other versions 
are considered obsolete for a new design. Although this goes against what I was saying about the 
importance of doing small exercises before the big effort, I can explain it.

I really want to focus on the mechanics of a game engine, and how it do the things. DX12 leaves memory 
management to the developer and you need to understand how the resources are shared between GPU, 
CPU and RAM. Since I'm primarily interested in designing a *state-of-the-art*, technically challenging 
engine, using DX12 ask more times and efforts but it's worth it! I get a better management of what the 
GPU and CPU does rather than letting the driver "do its thing". And it's always possible to add a 
handler to manage both DirectX and OpenGL in function of the running OS or the user's choice. We'll 
talk about that in a future post.

No matter the API you choose, you need one more important things: a **good programming language**! And 
no, Python is **not**. Making a game in Python may be fun, but it's not really technically interesting, 
it's not designed for that. Python will slow your game, your dev and it's not adapted to debug properly 
DX or GL interface. Java can do the job, like Minecraft using OpenGL, but it's a *virtual-machine language*. 
The optimization level is a pain in ass. C# can have it's chance, used by MonoGame, the performance are 
greats and compatible with DX (also know as SharpDX). But C# is a proprietary language of Microsoft. 
If I choose C#, I'll get more troubles to debug my code and I definetly close the door for porting code to other OSs. 
And C++? A lot of books about DX use this language, it's cross-platform, always documentated and in 
development, multi-threads ready, objects oriented and have a lot of existing usefull library like [Boost][link_boostlib]. 
In addition a lot of examples of games codes are made in C++.

Here's a list of the others tools I used during my development, which I will detail if necessary:

- Visual Studio 2022 - IDE with graphics oriented tools
- [CMake - Build manager][link_cmake]
- [Conan - Dependencies manager][link_conan]
- [Git - File version control][link_git]
- [PIX - GPU rendering debugger and analyzer][link_pix]
- [RenderDoc - Another GPU rendering debugger][link_renderdoc]

Let's go for a ~~sadomasochistic~~ adventure with C++ and DX12. What could go wrong?!

**Now let's talk less and code more! See you in next post ==>**

### Some books to begin

Rather than list books like that without much explanation, I'll mention which books I've read 
and what they've brought me.

The first book I've read and revisit most often is [3D Game Programming with DX12 by F.D.Luna][link_gameprogdx12]. 
This book cover all the mains topics about DX12 usage with some examples. The only downside, 
is that it's not easy to know exactly what functions all the elements indicated as structures 
or methods perform. I needed to read again several times and experiment directly in a program to 
understand the use of a feature. But beside of that, the book is very comprehensive for 
running a rudimentary engine, with simple lighting, shadows and textures.

I've got the DX9 and DX11 editions of this book for comparison and... Same topics, just adapted 
for DX11 and DX9 pipeline. DX9 has far fewer subjects because of its lack of shaders prior to 
9.0c version. 

The second book is not really related to game engine, because it's the [Holy Bible of C++11 by B.Stroustrup][link_holycpp]! 
A complete dictionary of all the C++ features and keywords, really usefull when I need to fully 
exploit a function from this language. Or maybe choose an another option to my issue than the one 
I thought earlier. *B.Stroustrup* said that C++ was a language that had to respond to a problem 
with a language feature!

I plan to buy the new edition of C++17 soon to benefit from interesting multithreading features.

I also read two other books: [Game Engine Architecture 3rd Ed. by J.Gregory][link_gameenginearch] and 
[Game Programming Patterns by R.Mystron][link_gameprogpatterns]. Both doesn't explictly show **what to code**, 
but **how to code** a good game engine. The first book describe each parts of a "modern" game engine, 
like physics, audio, user inputs, files management, graphics (with DX in my case), etc. 
The second one talk about the abstracts concepts of game engine like object memory management, 
pointers good practice, game loop, and others importants things that we don't necessarily think about, 
but which are necessary for the proper operation of a game engine.

On top of that, you have the [online DX12 documentation][link_d3d12doc] which will go into greater 
detail than F.D.Luna's book. Same advice for the Windows API to manage the window and the system messages.

All these books have helped me a great deal in clarifying aspects of a game engine that I couldn't see 
at first. But there's **no one right way to design a game engine**. Unlike a calculator program, where the 
sole aim is to perform calculations as efficiently as possible, our engine must meet several needs in 
terms of functionality to match the game we want to make. Most books and examples only give us one way 
of doing things, although sometimes there's no other way than to go through this or that method to achieve 
a task, the rest is up to our own judgement.

> I really like the low-level programming because it's like a puzzle-game to make a game.

My last piece of advice is to **take inspiration from existing engines**. And if the source code is available, 
try to understand **why it's implemented the way it is**. Others previous developers have run into the same problems 
you're about to encounter. For the time being, it would be a waste of time (and possibly of motivation) 
to repeatedly get *our feet stuck in the carpet*. Above all, the aim of this blog is 
to share successes and mistakes, that's what knowledge is all about.

### It's (not) a trap!

> This is a recurent chapter in a large part of my topics to describe, from my own experience 
> and point of view, the points on which I may have found myself stuck, or which at some point 
> required me to pause and reflect.
>
> Use them as you like!

When I choose to use DX12 as my first API for a 3D engine, the price to pay for that is to take 
even more time to complete the first draw on screen. That's part of the reason why it took me a 
year to achieve this result. Today I'm on the 4th attempt/iteration, and that's not counting tests 
on test code apart from the engine sources. I'll give you all details in upcoming posts!

So why not using Vulkan if I wanted a challenge? Some reasons for that: **Vulkan isn't so well documented** 
than DirectX, or at least it was easier to find DirectX docs and books. And it's for me a good 
sign to avoid or to limit the *demotivation effect*: the more time and energy you invest in 
something that doesn't bring you any satisfaction within the limits of your patience, the more 
you give up or *make a break* on the project. The greater the challenge, the greater the risk 
of *breaking*. This is normal, and it's through this kind of failure that we learn to know our 
limits. So I keep Vulkan as a next feature or evolution for now.



[link_gameeditor]:       http://game-editor.com/static/Main_Page.html
[link_unity]:            https://unity.com/fr
[link_godot]:            https://godotengine.org/
[link_boostlib]:         https://www.boost.org/
[link_cmake]:            https://cmake.org/
[link_conan]:            https://conan.io/
[link_git]:              https://git-scm.com/
[link_pix]:              https://devblogs.microsoft.com/pix/download/
[link_renderdoc]:        https://renderdoc.org/
[link_gameprogdx12]:     https://amzn.eu/d/9aTUE79
[link_holycpp]:          https://amzn.eu/d/cN7pZ7Q
[link_gameenginearch]:   https://amzn.eu/d/bEcBNXE
[link_gameprogpatterns]: https://gameprogrammingpatterns.com/
[link_d3d12doc]:         https://learn.microsoft.com/en-us/windows/win32/direct3d12/directx-12-programming-guide