During the whole time I was programming anything standalone that requires libraries in both C and D, I have run into quite a bit of problems with library linking. So in this post I will give my view on what could be improved to make developing with external libraries a lot easier.

First of all, I want to say that I don't know that much about standalone programming, since I am first and foremost an UnrealScript coder. However, I believe that it gives me a quite unique point of view in this matter. Also note that I will use some terms loosely, for example, here a class is equal to a file, because in UnrealScript any class is a file and any file is a class, no exceptions, while in C and D it's not quite correct since C is not object-oriented and D is optionally object-oriented.

First off I will give a bit of background information about the kind of problems I have encountered so far. My first attempt to use libraries was in C while coding Arcomage. The library in question was a configuration file parser. And there I discovered the first Linking Hell. The library contained .lib and .obj files - static and dynamic library components. I already had folders called include and lib, thus I threw the files there and tried to compile my code with it. No dice. After doing some research, I discovered that I had to explicitly tell the linker where the .libs and .objs are. But even then it didn't work. It appears that I also had to change the makefile so the linker would know in what order to process the libraries. And guess what? It still didn't work and threw a lot of linking errors. Thus what I had to do was use another library that was simply embedded code - you put it into the project's location and compile it as if it was a part of the project.

The second time I had problems with linking was when I tried to compile Arcomage on another machine. That problem was also only partly solved. For one, I couldn't compile it because I didn't have the compiling batch file - the one that tells the compiler to use a makefile. But even after I got it, I couldn't compile it. That's because of linking again. The linker couldn't find the libraries, and I couldn't add them to the PATH there because it was a limited account. I could copy the libraries into the compiler's own include and lib folders, but even that didn't work for some reason. I still don't know why even now. The solution I used was to use a development environment to set up a 'project'. That made it compile but at the same time limited the freedom to change the makefile.

The third time I had problems was with D, when I was trying to use a GUI library so I could use the OS's GUI. I could choose between QtD and GTKD. First off I took GTKD, only to find that it can't be used with the latest D compiler. Then I took QtD, tried to compile it, and failed. Then I downloaded precompiled libraries and it still failed to link them. Thus I am out of options now. But the QtD installation process requires additional look into it.

How do you install QtD? In their website they claim that you can install it by:

1. Downloading and installing Mercurial.
2. Using Mercurial to get the latest snapshot of their project.
3. Downloading and installing Qt SDK.
4. Downloading and installing cmake.
5. Downloading and installing implib.
6. Using cmake, tell it to use the Mingw installation from the Qt SDK, then build the source that you got from their Mercurial repository.

With all due respect - this is ludicrous. Not only because you need a bunch of different programs that you would never normally need, but also because cmake doesn't work as expected. I got it around halfway to building the libraries, only through editing the cmake file to tell it the location of all the different required parts of Qt. But the worst part is that I hit a brick wall then. cmake runs a program from Qt SDK that looks like a linker as well - and it is *hardcoded* to only use files from a certain directory. Of course, that directory does not exist because I'm not a Nokia developer. Theoretically you can change that directory; practically the function to change it doesn't work. A brick wall, a dead end. There is no way I could compile it now, yet I spent hours trying to even get to that point. Having precompiled libraries is not helping matters much as well. The linker doesn't link them, and you can't do anything about it.

In UnrealScript, this thing is handled a lot more gracefully. For one, there is only UCC, the Unreal Code Compiler. All the libraries, or includes, come in the form of utility packages. They need to go into the path, which in in the Game/System folder, or you can define another one by editing the INI. You also need to tell the compiler in what order to process packages, which is handled through the INI as well. To use a utility package, the code should be something like this:

function PostBeginPlay()
{
  class'MyPackage.MyUtilityClass'.static.MyUtilityFunction();
}

When UCC sees this code, it goes through the EditPackage list in the INI, finds MyPackage, then go through the package and finds MyUtilityClass, then goes through the class and finds the function MyUtilityFunction(). Then it gets the return value from there and goes on to compile your current package. Thus every library is dynamically linked there. Sort of. There is no linker at all, the compiler itself handles it. Thus there is no Linking Hell.

Something like that would be extremely beneficial for standalone programming as well. First of all, they should get rid of the linker altogether. Having it a separate program is just asking for trouble and screams "outdated!". That is because if you have the compiler and linker separate, you basically have the situation where your left hand doesn't know what your right hand is doing. The compiler compiles your code into .obj files, the linker takes those .obj files and merges them with the other included .obj/.lib files. If you use plain code as the library, the linker first invokes the compiler once more to get more .obj files. Thus here we have a whole lot of unnecessary steps where anything can go terribly wrong and you won't even know where the problem really is. In fact, linker errors are not even handled by development environments. However, there already are some languages or compilers that successfully got rid of the linker executable - they are called "smart linkers" and even give you some benefits in optimisation.

Moreover, the structure of including should be unified. For example, make it so that the compiler would look for includes in ../lib and ../include folders first, then in their own libraries. They should also not require the developer to specify the order of including - the requirements should already be listed in the .objs or some kind of a config file that comes with them. That would also allow the compiler to give meaningful error messages, say, "Package A required by included package B not found".

Furthermore, people who create libraries should never ask developers to compile their own code. That is because it's very lengthy to do so and that's also why you don't compile your own programs - you need to setup the environment just right so you could compile things, and the variety of environments throughout PCs is huge, and thus there is a lot more chance that you won't be able to compile things correctly than that you will. For them, however, building the binaries is not that hard since they do it all the time while testing anyway.

Then there is a question of compiler versions. Libraries can usually be used only with a certain compiler version, since newer compilers often break older code and thus fail to link with older libraries. The solution is easy - like D already has a "version" check - it usually checks for the OS version - libraries could also have this kind of a check. Either in them or in a separate config file it could be stated on which version of the compiler the library was originally compiled. The compiler could then try to link it, and if it fails, it could give a meaningful error "This library was written for the version VerNum of this compiler. Please update the library or downgrade the compiler." This is quite important for modern compilers like DMD, and less so for stagnated compilers like gcc.

So, to sum it all up, this is how the ideal way of using libraries would be: You go to the website of the library creators, download the libs compiled for your version of the compiler or, even better, the source code that you could immediately include, then put the files into ../include and ../lib folders from your project files and be able to use them immediately.