More than half a year ago I was trying out Conan package manager for resolving dependencies in our C++ project. The research went well, but we never actually proceeded with switching to Conan for the whole project. And then a couple of weeks ago I started looking into vcpkg.

I’ve heard about vcpkg before and even tried to make a vcpkg package for one library a couple of years ago, but back then I didn’t find documentation for this (as I now understand, that’s because one does not really create a “package”) and abandonned the task. I never thought I’ll be looking at vcpkg again, but recently I discovered that some teams have been switching from Conan to vcpkg, which sounded intriguing and promising, as we still needed a package manager for our projects.

Why we didn’t proceed with Conan

I think, this is because we chose a wrong approach to it. At the first glance it seemed that integrating Conan as a part of our build system would be too disruptive, so we decided to just pack pre-built artifacts from the install destination with export-pkg -pf. So we were not making source packages and were not implementing the build() method in recipes.

Effectively, we were prebuilding all our dependencies for all the platforms that we target, and that turned out to be very (very) time-consuming, so we pretty much didn’t progress beyond that small subset of dependencies from the pilot project created during research.

So yeah, perhaps if we chose to integrate Conan into our buildsystem and build dependencies from sources, the result would be better (faster). Why didn’t we do so? Well, we have been already working with other package managers before - APT/deb, NuGet, npm and others that our users were requesting - and these packages are mostly just archives with certain meta-information, and that is how we were looking at Conan too, not expecting it to also be able to manage building. It also didn’t help that we were reading Conan’s documentation rather chaotically: I remember I was googling for something like “Conan package pre-built binaries” and getting directly to those pages without properly studying the rest of the documentation.

Anyway, having covered dependencies of the pilot project (one of our libraries) with Conan, we invited a couple of other teams to test it out, and immediately it turned out that they use a different combination of platforms/compilers, which we didn’t pre-build for, and so they had to force/override specific versions (for example, with -s compiler.toolset="v142" and -DCONAN_DISABLE_CHECK_COMPILER=1), as they were not able to build missing binaries (because we did not implement the build() method in recipes).

But even that was not all, as users (understandably) wanted to fetch 3rd-party dependencies from Conan Center and our packages from our Artifactory, while we were fetching everything from our Artifactory (using different user/channel values too), and such a situation isn’t handled too nicely with Conan (or we didn’t find a way).

Finally, not everyone in our team liked the idea of complicating the build toolchain by adding yet another tool, because not only one needs to run one more thing before building the project, but also the installation of that thing isn’t very straightforward, as one needs to have Python, pip and only then Conan can be installed. Yes, it is rather a trivial setup, and yet some have managed to struggle even with this. You really shouldn’t underestimate how common the question/challenge of installing pip is.

About vcpkg

In short, vcpkg is a combination of a CMake toolchain and a CLI tool. Together they handle the following:

figuring out build environment (platform, compiler, linking);
fetching dependencies sources, patching them if required;
building or restoring those dependencies before configuring the main project.

Restoring means that if dependencies have been already built before, then instead of building them again vcpkg will restore their pre-built binary packages either from local cache or from a remote storage. This functionality is called binary caching, and as with Conan that’s our main reasoning for using vcpkg for resolving dependencies - to avoid re-building the code that changes only so often.

Here’s also a good talk at NDC Oslo conference about vcpkg, Conan and in general about dependency management for C++ projects.

Schematically simplified vcpkg functionality can be visualized like this:

There will be more details about what’s going on here later, but in short it shows a project that depends on 3 libraries (and 3 CMake helpers), and vcpkg handles fetching and building dependencies before building the main project. If dependencies have been already built before, then they are just restored from cache.

For us local cache on CI/CD buildbots was absolutely enough, but later we also added a remote (but still in-house) storage. In terms of the schema above, the only change in this case is that cache is stored not on the same build machine but on another server.

Installation

You need to have one of the latest CMake versions in the system, otherwise vcpkg installer script will download it for you and put it somewhere in the system, so you’ll have more than one CMake on your computer. In my case it required CMake 3.24 as a minimum.

Regarding system environment, I have Windows, Mac OS and GNU/Linux hosts, and while here in the article it is assumed that one uses Mac OS environment, all the steps will be 99% the same on any other platform. And on Windows, as usual, I recommend to use Git BASH.

The installation of vcpkg starts with cloning its repository:

$ cd /path/to/programs/
$ git clone git@github.com:microsoft/vcpkg.git
$ cd ./vcpkg/

That will bring you the first of the two components - the CMake toolchain - it will be placed to /path/to/programs/vcpkg/scripts/buildsystems/vcpkg.cmake.

Before continuing, set VCPKG_DISABLE_METRICS environment variable to prevent anal telemetry probing (it’s Microsoft, after all).

Now you can start the “installation” by executing this script:

$ ./bootstrap-vcpkg.sh

It will in turn call ./scripts/bootstrap.sh script and download the second component - vcpkg CLI tool binary from its repository. In my case it was 2022-09-20 version. It also does some other things, which you can look up in the script file. I would prefer to do all these things myself than letting a script execute a bazillion commands in my system.

As a result you’ll have the tool binary at /path/to/programs/vcpkg/vcpkg (vcpkg.exe on Windows). For convenience, you might want to add /path/to/programs/vcpkg to your PATH (so you could call vcpkg CLI tool from anywhere). And while you are at it, also set VCPKG_ROOT environment variable to /path/to/programs/vcpkg.

What is a vcpkg package

Right away, it is not correct to call them “packages”, because they are more like recipes for configuring and building. In vcpkg’s terminology they are called “ports”.

In its basic form a vcpkg port is just these two files:

./glfw
├── portfile.cmake
└── vcpkg.json

here:

portfile.cmake - CMake instructions for fetching sources, configuring, building, installing and grooming;
vcpkg.json - information about the package/port: name, description, homepage, version and its own dependencies.

If a library, which you want to add as a dependency to you project, is already has a nice and modern CMake project file and with proper installation too, then you won’t need anything but these two files to make a ~~package~~ port.

But quite often (way too often) you will need at the very least to patch the original library’s CMakeLists.txt or even create one from scratch, as the original library’s repository might not have anything for CMake out of the box. Either way, the point is that portfile.cmake file isn’t a replacement for a potentially missing CMakeLists.txt, as its purpose should be just to get the sources and run commands for building/installation, so it expects CMakeLists.txt to be already available.

Missing/additional files are supposed to be just placed into the package folder alongside and referred to from portfile.cmake. For example, here’s how a more complex package might look like:

./dearimgui
├── CMakeLists.txt
├── fixing-something.patch
├── portfile.cmake
└── vcpkg.json

here:

CMakeLists.txt - if you are unlucky enough to require a library without CMake support;
fixing-something.patch - in case there is some problem or a change that you need to fix/apply, and the library maintainer doesn’t want / can’t do it;
portfile.cmake - applies the fixing-something.patch after fetching library sources and copies CMakeLists.txt to the library source directory (or wherever you want);
vcpkg.json - name, version, etc.

Registry

Packages or actually ports need to be stored somewhere, and such place is called a “registry”. As a matter of fact, the repository that you cloned to install vcpkg is at the same time a registry too: ~~packages~~ ports are located in the ./ports folder. You can take a look at any of them to see for yourself that in the most simple cases it is indeed just a duo of portfile.cmake and vcpkg.json files. You’ll also see that in many cases it isn’t so simple.

And so yes, you can use Microsoft’s vcpkg repository as a registry for your project. But we of course wanted to have our own in-house registry on our own server maintained by ourselves. And what would you know, setting up your own vcpkg registry is the fucking easiest thing in the world.

Here’s Microsoft’s own documentation (and also this article) on the matter. Basically, a vcpkg registry is just a Git repository with the following structure:

├── ports
└── versions

That Git repository can be hosted on a GitHub, GitLab, Gitea, whatever other service or just a regular bare Git repository available via SSH/HTTP. Isn’t it fucking great! I was so happy, I almost pissed myself. That is an absolute respect and a huge +1 to Microsoft’s karma.

So let’s create our own vcpkg registry. For starters it will contain just one port - GLFW library - and the structure of the repository will then be the following:

├── ports
│   └── glfw
│       ├── portfile.cmake
│       └── vcpkg.json
└── versions
    ├── baseline.json
    └── g-
        └── glfw.json

As an example for the reference, I’ve created my own public registry here (right now it contains several other ports too).

Port structure

With my GLFW port as an example.

vcpkg.json

The vcpkg.json contents:

{
    "name": "glfw",
    "version": "3.3.8",
    "description": "A multi-platform library for OpenGL/Vulkan/etc, window and input",
    "homepage": "https://www.glfw.org/",
    "dependencies": [
        {
            "name": "vcpkg-cmake",
            "host": true
        },
        {
            "name": "vcpkg-cmake-config",
            "host": true
        }
    ]
}

These vcpkg-cmake and vcpkg-cmake-config dependencies will be covered in a minute. For now note the "host": true - it means that these dependencies are so-called “host dependencies”, so they are fetched and installed before the packages that depend on them.

portfile.cmake

The portfile.cmake looks like this:

# where to get the package sources from
vcpkg_from_git(
    OUT_SOURCE_PATH SOURCE_PATH
    URL git@github.com:glfw/glfw.git             # Git repository cloning URL
    REF 7482de6071d21db77a7236155da44c172a7f6c9e # commit hash (pointing to a version tag)
)

# how to configure the project
# thankfully, GLFW already has CMakeLists.txt
vcpkg_cmake_configure(
    # where CMakeLists.txt is (here's it's on the root level of the project)
    SOURCE_PATH "${SOURCE_PATH}"
    # CMake configuration options, just regular -D stuff
    OPTIONS
        -DGLFW_BUILD_EXAMPLES=0
        -DGLFW_BUILD_TESTS=0
        -DGLFW_BUILD_DOCS=0
)

# this one actually builds and installs the project
vcpkg_cmake_install()

# this will (try to) fix possible problems with imported targets
vcpkg_cmake_config_fixup(
    PACKAGE_NAME "glfw3"          # if the project name (glfw3) is different from the port name (glfw)
    CONFIG_PATH "lib/cmake/glfw3" # where to find project's CMake configs
)

# don't know what this is used for, I have never needed it yet
#vcpkg_fixup_pkgconfig()

# this one you just need to have, and sometimes you'll need to delete even more things
# feels like a crutch, but okay
file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/include")

# vcpkg expects license information to be contained in the file named "copyright"
file(
    INSTALL "${SOURCE_PATH}/LICENSE.md"
    DESTINATION "${CURRENT_PACKAGES_DIR}/share/${PORT}"
    RENAME copyright
)

The ${PORT} thing is of course not a standard CMake variable. Here you can take a look at the full(?) list of vcpkg variables that can be used in portfiles.

Now, if you, like me, started by reading that blog post, then at first you’ll have the following functions in your portfile.cmake:

But when you’ll go to read about any of these in the documentation, you’ll discover that they have been already deprecated, and instead you should use these ones:

And that is why our example GLFW package depends on vcpkg-cmake and vcpkg-cmake-config packages, which are so-called helper packages, as they provide these functions to GLFW’s portfile.

Why the deprecated variants come out of the box and the new ones are placed into helper packages - I don’t know. Perhaps that is so they could be easier updated to newer versions?

Another thing here is that if we don’t want to use Microsoft’s registry, then we’ll obviously need to copy these packages (vcpkg-cmake and vcpkg-cmake-config) to our registry, and then the structure of the registry will be the following:

├── ports
│   ├── glfw
│   │   ├── portfile.cmake
│   │   └── vcpkg.json
│   ├── vcpkg-cmake
│   │   ├── portfile.cmake
│   │   ├── vcpkg-port-config.cmake
│   │   ├── vcpkg.json
│   │   ├── vcpkg_cmake_build.cmake
│   │   ├── vcpkg_cmake_configure.cmake
│   │   └── vcpkg_cmake_install.cmake
│   └── vcpkg-cmake-config
│       ├── copyright
│       ├── portfile.cmake
│       ├── vcpkg-port-config.cmake
│       ├── vcpkg.json
│       └── vcpkg_cmake_config_fixup.cmake
└── versions
    ├── baseline.json
    ├── g-
    │   └── glfw.json
    └── v-
        ├── vcpkg-cmake-config.json
        └── vcpkg-cmake.json

And now some more details about these functions from the portfile.

vcpkg_from_git

The vcpkg_from_git() function fetches the library sources from a Git repository.

Here I’m using a GitHub repository that is available via SSH, so I need to have the following in my ~/.ssh/config:

Host github.com
HostName github.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/MY-GITHUB-SSH-KEY

Or you can just as well replace the URL value with https://github.com/glfw/glfw.git variant, but then in case of private repositories you’ll need to set-up the credentials thing.

GitHub here is used as an example, and instead there of course can be any other service or just a bare Git repository on your server. Speaking of which, I once again would like to recommend you to clone/mirror all your 3rd-party dependencies sources to your servers, so you don’t rely on 3rd-party services outside of your IT infrastructure.

Another note about vcpkg_from_git() is that Microsoft’s tutorial examples use different functions for fetching sources: vcpkg_from_github() and vcpkg_from_gitlab(). But first of all, for me those failed (even though I did provide required access tokens); and secondly, what is the point of using these at all, if vcpkg_from_git() with SSH-key authentication is absolutely enough?

vcpkg_cmake_configure

This function does the regular CMake project configuration. You need to:

set path to CMakeLists.txt folder in the SOURCE_PATH. Usually it’s the root level of the project sources, which conveniently is stored in the ${SOURCE_PATH} variable (looks a bit confusing, I know);
provide -D options (if any). It is recommended to disable building samples/demos/tests and documentation-related stuff.

If the project doesn’t have a CMakeLists.txt, this command will naturally fail, just like as CMake itself would, so you’ll need to create one yourself and add it to the port.

vcpkg_cmake_install

This function builds and installs the configured project. It actually calls the vcpkg_cmake_build() function and sets TARGET install.

vcpkg_cmake_config_fixup

That one fixes potential problems with installed CMake targets, as some projects do some really weird shit with their CMake configs. For example, you can get the following error if you won’t call this function after vcpkg_cmake_install():

Policy CMP0111 is not set: An imported target missing its location property fails during generation

and many other different problems.

In order for vcpkg_cmake_config_fixup() function to work it needs to know where the project installs its CMake configs to. If those are installed into an unusual place (they usually are), then it will fail with an error like this:

Building some-library[core]:x86-windows...
-- Installing port from location: /path/to/your/vcpkg-registry/ports/some-library
-- Fetching git@github.come:retifrav/some-library.git cb96b1bd1af1e15eae2660097b4db2ddeaa94313...
-- Extracting source /path/to/programs/vcpkg/downloads/some-library-cb96b1bd1af1e15eae2660097b4db2ddeaa94313.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/some-library/src/ddeaa94313-5e7aebed2b.clean
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
CMake Error at scripts/cmake/vcpkg_cmake_config_fixup.cmake:81 (message):

  '/path/to/programs/vcpkg/packages/some-library_x64-osx/debug/share/some-library'
  does not exist.

To fix that you need to go to /path/to/programs/vcpkg/packages/some-library_x64-osx and find where exactly this project installs its CMake files. For example, if they are in cmake/SomeLibrary, then your vcpkg_cmake_config_fixup() should have the following arguments:

vcpkg_cmake_config_fixup(
    # if the project name (SomeLibrary) is different from the port name (some-library)
    PACKAGE_NAME "SomeLibrary"
    # where project's CMake configs are installed by default
    CONFIG_PATH "cmake/SomeLibrary"
)

And as you’ve already seen, for GLFW port it’s these:

vcpkg_cmake_config_fixup(
    PACKAGE_NAME "glfw3"
    CONFIG_PATH "lib/cmake/glfw3"
)

If CMake configs are nowhere to find, then probably that library doesn’t have a proper installation (or mayby no installation at all), and then you’ll need to patch its CMakeLists.txt.

The vcpkg_cmake_config_fixup() function also organizes installed CMake configs and puts them into share/PACKAGE-NAME folder. Without doing this you can get the following warnings:

-- Performing post-build validation
The following cmake files were found outside /share/some-library. Please place cmake files in /share/some-library.

    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/cmake/SomeLibraryConfig.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/cmake/SomeLibraryConfigVersion.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/cmake/SomeLibraryTargets-release.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/cmake/SomeLibraryTargets.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/debug/cmake/SomeLibraryConfig.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/debug/cmake/SomeLibraryConfigVersion.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/debug/cmake/SomeLibraryTargets-debug.cmake
    /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/debug/cmake/SomeLibraryTargets.cmake

What I haven’t figured yet is how to handle the situation when the library and its vcpkg port have different names, for example the library target name (and CMake config) is SomeLibrary but the port name is some-library. In that case the contents of share folder will end up being like this:

├── SomeLibrary
│   ├── SomeLibraryConfig.cmake
│   ├── SomeLibraryConfigVersion.cmake
│   ├── SomeLibraryTargets-debug.cmake
│   ├── SomeLibraryTargets-release.cmake
│   └── SomeLibraryTargets.cmake
└── some-library
    ├── copyright
    ├── vcpkg.spdx.json
    └── vcpkg_abi_info.txt

I think this is not a correct structure, as it seems that the intention is to have one folder per port. It all works fine like this, but I’m concerned that it might cause problems in future.

Versions

baseline.json

The baseline.json, as I understood, lists default versions of all the packages that are available in that registry:

{
    "default":
    {
        "dearimgui":
        {
            "baseline": "1.88.0",
            "port-version": 0
        },
        "glad":
        {
            "baseline": "0.1.36",
            "port-version": 0
        },
        "glfw":
        {
            "baseline": "3.3.7",
            "port-version": 0
        },
        "...":
        {
            "...": "..."
        }
    }
}

Then if a project has no overrides, these will be the versions vcpkg will get for it. And if a project lists a dependency with version>= field, then in case it’s lower than the baselined, vcpkg will skip all higher versions untill the baselined value; otherwise it will take the first versions that is equal or higher.

Versions file

Individual *.json files establish which version of the port is available in which commit, for example here’s glfw.json:

{
  "versions": [
    {
      "version": "3.3.8",
      "git-tree": "597fa07e1afd57c50dfdbeb0c0d28f4157748564"
    }
  ]
}

To get the commit hash value (for the git-tree property) of the current version of the glfw port you need to run the following:

$ cd /path/to/your/vcpkg-registry
$ git rev-parse HEAD:./ports/glfw
597fa07e1afd57c50dfdbeb0c0d28f4157748564

And it might not be obvious, but it means that whenever you make any changes to glfw port and commit them, you should not immediately push that commit to server. First you need to run git rev-parse to get the new hash value, set this value to git-tree, amend (git commit --amend --no-edit) your commit and only then push your commit to server. Here are some more details about that.

Several versions of a port

A port can have more than one version. To add another version you need to, with my glfw port as an example:

Set a new Git hash (pointing to a different version tag) in ./ports/glfw/portfile.cmake;
Update the version value in ./ports/glfw/vcpkg.json;
Commit and do the git rev-parse thing;

Add (not replace) a new version to ./versions/g-/glfw.json:

{
    "versions":
    [
        {
            "version": "3.3.8",
            "git-tree": "597fa07e1afd57c50dfdbeb0c0d28f4157748564"
        },
        {
            "version": "3.3.7",
            "git-tree": "45af0346d2ec926146091ab374c475cac5aaf055"
        }
    ]
}

So at first I had GLFW version 3.3.8 and now I added version 3.3.7. So even though this commit is “newer”, and the versions order is “wrong”, it doesn’t matter at all, as it’s all just Git hashes pointing to certain “states” of the repository.

If you’ll get this error trying to install a dependency in your project:

-- Running vcpkg install
Fetching registry information from git@github.com:retifrav/vcpkg-registry.git (HEAD)...
git archive failed with message:
error: git failed with exit code: (128).
fatal: not a tree object: cdeffa673205b611d8ced28468cce6a06f1fdffd

then it could be because you didn’t update the git-tree value in that dependency’s version JSON in the registry, or perhaps the baseline value in your project’s vcpkg-configuration.json points to a wrong commit.

Checking versions and hashes

It is quite easy to forget to update the git-tree value, so you might end up with a somewhat broken port, like it is shown in the previous section when it fails with an error. But there also might be a different situation: when git-tree points to an existing state of the registry, so there will be no error, but you won’t be getting the actually latest ports state for that version either.

And the bigger your registry will get, the harder it will be to watch for potential mismatches, so you’ll likely want to automate this somehow. For example, with a shell script:

$ ./scripts/check-versions-and-hashes.sh

                port |      version |                                          |
-------------------------------------------------------------------------------------------------------------------------
                curl |        8.1.2 | 308dc1fccea69d04003fef9103c9d7bd13d0bb32 | 308dc1fccea69d04003fef9103c9d7bd13d0bb32
           dearimgui |       1.88.0 | fda742f0dd720fcd2af3cb8e946173069d70435f | a8838baa1c0125a8a2aaccc3889f8518ecfeac40
 decovar-vcpkg-cmake |   2022-10-15 | 5b70c3ba46b51c88044c335b8ad030f0edc609e5 | 5b70c3ba46b51c88044c335b8ad030f0edc609e5
           e57format |        2.3.0 | e01f35b5353848abef4cb47e8ff9f25dbb05ad81 | e01f35b5353848abef4cb47e8ff9f25dbb05ad81
                glad |       0.1.36 | 2341f5144ce8e76a256289517d61abb4ab9fb72c | 2341f5144ce8e76a256289517d61abb4ab9fb72c
                glfw |        3.3.8 | 597fa07e1afd57c50dfdbeb0c0d28f4157748564 | ff428db2871ef4e409c2ea9f29a866b63ba5b90b
                 icu |       72.1.0 | 330edeacb91afe9d9aa0007cbacc08f6a2b4a3f3 | 330edeacb91afe9d9aa0007cbacc08f6a2b4a3f3
       json-nlohmann |       3.11.2 | 489dbd7358b610e1153e93accf36123a5afe3ce3 | 489dbd7358b610e1153e93accf36123a5afe3ce3
                lyra |        1.6.1 | 0cb627c7ee8f8ec2769b41b41f82b335c50e984b | 0cb627c7ee8f8ec2769b41b41f82b335c50e984b
                lzma |       22.1.0 | 04277d0cee83f6c4b8ce44f6756ed044b2691fbe | 04277d0cee83f6c4b8ce44f6756ed044b2691fbe
              sqlite |       3.41.0 | 765037d4fa9cee6d6c7639006e546cf1ad8d675a | 765037d4fa9cee6d6c7639006e546cf1ad8d675a
         vcpkg-cmake |   2022-08-18 | 84c200e8e625d4d99b1649525fcdf81a73197078 | 84c200e8e625d4d99b1649525fcdf81a73197078
  vcpkg-cmake-config |   2022-02-06 | e23b39e21f0dd42ecc615262640d211c39696aa1 | e23b39e21f0dd42ecc615262640d211c39696aa1
            xerces-c |        3.2.4 | ebfeb33e67607b103c583656109cbfeeb1406260 | ebfeb33e67607b103c583656109cbfeeb1406260
                zlib |       1.2.12 | 7feb1b251066f9213881134c320cba2d853d1b45 | 7feb1b251066f9213881134c320cba2d853d1b45

The following ports have a mismatch between their stated and actual Git hashes:

- dearimgui
- glfw

The hashes matching is highlighted with colors, here’s a screenshot:

Aside from the colors and summary in the end of the output, if there is at least one mismatch, the exit code is set to non-zero, so you can also use this script as a pre-commit Git hook.

At the moment the script has a flaw of expecting certain structure of versions file, in particular it will fail to get the git-tree property if it doesn’t go right on the next line after the version property (for example, it could be a port-version property there instead).

Triplets

A vcpkg triplet is a file with a set of values that together “identify” the target platform: CPU architecture, type of linking and so on. One could probably say that it’s like Conan profiles, but with fewer(?) parameters.

Default triplets are located here: /path/to/programs/vcpkg/triplets/. If you are not happy with the standard collection, you can add your own triplets as well, but don’t edit the default ones, as it will certainly backfire at you at some point later.

I learned about triplets when I noticed that dependencies in one of my Windows projects were built as DLLs (SHARED libraries), even though I did not set -DBUILD_SHARED_LIBS=1. And it turned out that on Windows by default vcpkg uses x64-windows.cmake triplet, which contents are:

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)

Like I said, while you can set static value for the VCPKG_LIBRARY_LINKAGE here, I would definitely not recommend doing that. Instead, for this particular case, you can use another standard triplet - x64-windows-static.cmake - and that one will build your libraries as STATIC. To use that triplet instead of the default one, set -DVCPKG_TARGET_TRIPLET="x64-windows-static". And if there wasn’t such triplet in the standard collection, then you could have just created one yourself.

Creating your own ports

A good portion of this topic is already covered in the section about vcpkg registry, but there are some more things to tell about.

The Microsoft’s registry already has a good collection of ~~packages~~ ports. If those are not enough, for instance if you’d like to build some library differently or if your library of interest is simply missing from the registry, then you can always create your own port and store it in your registry or/and publish it to Microsoft’s registry.

When making your own ports, you can use Microsoft’s registry as a great source of configuration options, techniques and workarounds for various problems one might encounter. You can literally go from port to port and study the way people configure, build (and patch) sources, instead of trial-and-erroring that yourself for ~~hours~~ days.

Portfile

Creating a vcpkg port of a library in simple cases is just a matter of writing a portfile for it and listing its version(s). But it’s not often that you’ll have such simple cases. Usually you’ll need to add new files, patch sources or/and introduce features.

Adding files

Many libraries don’t have CMake support out of the box, so often you’ll need to create and add CMakeLists.txt for them. To do that you just put it into the port folder and add the following command to the portfile.cmake somewhere before vcpkg_cmake_configure():

file(COPY "${CMAKE_CURRENT_LIST_DIR}/CMakeLists.txt" DESTINATION "${SOURCE_PATH}")

vcpkg_cmake_configure(
    SOURCE_PATH "${SOURCE_PATH}"
)

That’s exactly how I did it for my Dear ImGui port.

Any other files can be added the same way.

Patches

When you don’t need to add new files but would like to change existing ones, you can apply a patch, which is done by simply adding PATCHES argument to vcpkg_from_git(). When I saw how easy this is, I almost couldn’t believe it, especially given that with Conan I had to handle this myself (back then I didn’t know that Conan probably can do it as well).

For example, if we want to patch something in GLFW sources:

vcpkg_from_git(
    OUT_SOURCE_PATH SOURCE_PATH
    URL git@github.com:glfw/glfw.git
    REF 45ce5ddd197d5c58f50fdd3296a5131c894e5527
    PATCHES
        some.patch    # ${CMAKE_CURRENT_LIST_DIR}/some.patch
        another.patch # ${CMAKE_CURRENT_LIST_DIR}/another.patch
)

Important to note here that some.patch must be a Git patch, not a bare diff (from GNU Diffutils) output. To create such a patch, go to your local GLFW’s repository (checked out on that particular commit), edit required files, stage your changes and then:

$ git diff --cached --binary > some.patch
$ mv ./some.patch /path/to/your/vcpkg-registry/ports/glfw/

Features

You can control the way a project is built via so-called “features”, which essentially are a simple mapping to CMake -D options.

This is done with vcpkg_check_features() function. For example, I have a port of Dear ImGui library and I added a feature to build it with GLFW support:

# mapping features array from vcpkg.json to a list of CMake options
# already prepended with -D and set to ON
# and saving that list to FEATURE_OPTIONS variable
vcpkg_check_features(OUT_FEATURE_OPTIONS FEATURE_OPTIONS
    FEATURES
        backend-glfw BACKEND_GLFW
)

vcpkg_cmake_configure(
    SOURCE_PATH "${SOURCE_PATH}"
    OPTIONS
        # passing the list of options to CMake configure
        ${FEATURE_OPTIONS} # it now contains -DBACKEND_GLFW=ON
)

This BACKEND_GLFW option is used in CMakeLists.txt to add GLFW-specific code to the Dear ImGui build.

For this mapping to work the port’s vcpkg.json contains optional backend-glfw feature (along with dependency on glfw port):

{
    "name": "dearimgui",
    "...": "...",
    "features":
    {
        "backend-glfw":
        {
            "description": "Using GLFW as graphics backend",
            "dependencies":
            [
                "glfw"
            ]
        }
    }
}

And so other projects can now depend on this port like this:

{
    "name": "glfw-imgui-example",
    "...": "...",
    "dependencies":
    [
        "...",
        {
            "name": "dearimgui",
            "features":
            [
                "backend-glfw"
            ]
        }
    ]
}

Platform-specific features

It is possible to enable features only for certain platforms. I discovered that when my Xerces-C++ port failed to build for iOS (using triplet arm64-ios):

FAILED: src/CMakeFiles/xerces-c.dir/xercesc/util/PlatformUtils.cpp.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -DHAVE_CONFIG_H=1 -DXERCES_BUILDING_LIBRARY=1 -D_FILE_OFFSET_BITS=64 -D_THREAD_SAFE=1 -I/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/arm64-ios-dbg -I/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src -I/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/arm64-ios-dbg/src -fPIC -Wall -Wcast-align -Wcast-qual -Wctor-dtor-privacy -Wextra -Wformat=2 -Wimplicit-atomic-properties -Wmissing-declarations -Wno-long-long -Woverlength-strings -Woverloaded-virtual -Wredundant-decls -Wreorder -Wswitch-default -Wunused-variable -Wwrite-strings -Wno-variadic-macros -fstrict-aliasing -g -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.5.sdk -std=gnu++14 -MD -MT src/CMakeFiles/xerces-c.dir/xercesc/util/PlatformUtils.cpp.o -MF src/CMakeFiles/xerces-c.dir/xercesc/util/PlatformUtils.cpp.o.d -o src/CMakeFiles/xerces-c.dir/xercesc/util/PlatformUtils.cpp.o -c         /programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/PlatformUtils.cpp
In file included from /Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/PlatformUtils.cpp:82:
In file included from /Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/XMLNetAccessor.hpp:26:
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/XMLURL.hpp:277:19: warning: cast from 'const xercesc_3_2::XMLURL *' to 'xercesc_3_2::XMLURL *' drops const qualifier [-Wcast-qual]
        ((XMLURL*)this)->buildFullText();
                  ^
In file included from /Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/PlatformUtils.cpp:119:
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:99:12: error: unknown type name 'TextEncoding'
        ,               TextEncoding                    textEncoding
                        ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:116:2: error: unknown type name 'CollatorRef'
        CollatorRef     fCollator;                                              // Our collator
        ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:130:2: error: unknown type name 'TextEncoding'
        TextEncoding discoverLCPEncoding();
        ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:150:6: error: unknown type name 'TECObjectRef'
            TECObjectRef                        textToUnicode,
            ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:151:6: error: unknown type name 'TECObjectRef'
            TECObjectRef                        unicodeToText,
            ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:199:5: error: unknown type name 'TECObjectRef'
    TECObjectRef        mTextToUnicode;
    ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/Transcoders/MacOSUnicodeConverter/MacOSUnicodeConverter.hpp:200:5: error: unknown type name 'TECObjectRef'
    TECObjectRef        mUnicodeToText;
    ^
/Users/USERNAME/programs/vcpkg/buildtrees/xerces-c/src/a3919277d9-dc0ef85571.clean/src/xercesc/util/PlatformUtils.cpp:485:8: error: incompatible pointer types assigning to 'xercesc_3_2::XMLTransService *' from 'xercesc_3_2::MacOSUnicodeConverter *'
                tc = new MacOSUnicodeConverter(fgMemoryManager);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning and 8 errors generated.

The problem here is that for building for iOS Xerces-C++ requires ICU dependency. Okay, let’s add it to dependencies then, but since it is not needed when building on other platforms, better to do it through an optional feature:

{
    "name": "xerces-c",
    "...": "...",
    "features":
    {
        "icu":
        {
            "description": "Enable ICU support",
            "dependencies":
            [
                "icu"
            ]
        }
    }
}

But how to make this feature enabled only on iOS platform? As I understood it, one option would be to add a dependency on itself, for example like it is done in FFmpeg port. But I decided to do it differently - in the manifest of a port (E57Format) that depends on Xerces-C++:

{
    "name": "e57format",
    "...": "...",
    "dependencies":
    [
        {
            "...": "..."
        },
        {
            "name": "xerces-c",
            "platform": "!ios"
        },
        {
            "name": "xerces-c",
            "features":
            [
                "icu"
            ],
            "platform": "ios"
        }
    ]
}

Now iOS is the only target platform where icu feature will be enabled, and Xerces-C++ (starting with version 3.2.4) will be able to build with ICU dependency.

Admittedly, this looks crutchy, but I am yet to find a better way.

Feature-dependent features

That is something I haven’t figured out yet.

At some point we’ve got into a situation where one of our projects required JPEG dependency, but we also wanted to have an option to use jpeg-turbo instead. So:

{
    "name": "some",
    "...": "...",
    "dependencies":
    [
        "..."
    ],
    "default-features":
    [
        "use-jpeg-turbo"
    ],
    "features":
    {
        "use-jpeg-original":
        {
            "description": "Use original JPEG library",
            "dependencies":
            [
                "jpeg"
            ]
        },
        "use-jpeg-turbo":
        {
            "description": "Use jpeg-turbo",
            "dependencies":
            [
                "jpeg-turbo"
            ]
        }
    }
}

And in the project’s CMakeLists.txt:

# ...

option(PREFER_JPEG_TURBO "Use jpeg-turbo instead of original JPEG" 1)

# ...

if(PREFER_JPEG_TURBO)
    # if it fails to find it, perhaps implement a fallback to the original JPEG
    find_package(jpeg-turbo CONFIG REQUIRED)
    target_link_libraries(${CMAKE_PROJECT_NAME} PRIVATE jpeg-turbo::turbojpeg-static)
else()
    find_package(jpeg CONFIG REQUIRED)
    target_link_libraries(${CMAKE_PROJECT_NAME} PRIVATE jpeg)
endif()

But that project also depends on PDFium, which in turn depends on JPEG/jpeg-turbo too, and in our PDFium port we resolve this the same way via vcpkg (having disabled building those from vendored sources). But the problem now is how to specify which JPEG library should PDFium port use (meaning that it is controlled via PDFium port features)?

I guess, it would be great if one could set dependencies between features within the same manifest, something like this:

{
    "name": "some",
    "...": "...",
    "dependencies":
    [
        "..."
    ],
    "default-features":
    [
        "use-jpeg-turbo"
    ],
    "features":
    {
        "use-jpeg-original":
        {
            "description": "Use original JPEG library",
            "dependencies":
            [
                "jpeg"
            ]
        },
        "use-jpeg-turbo":
        {
            "description": "Use jpeg-turbo",
            "dependencies":
            [
                "jpeg-turbo"
            ]
        },
        "pdf":
        {
            "description": "Enable PDF capabilities",
            "dependencies":
            [
                {
                    "name": "pdfium",
                    "features":
                    [
                        {
                            "if-feature-enabled": "use-jpeg-turbo",
                            "then": "with-jpeg-turbo",
                            "else": "with-jpeg-original"
                        }
                    ]
                }
            ]
        },
    }
}

But there is no such thing at the moment, and looking at this construction I doubt that it will ever be added. So for now I guess the only option would be to split pdf feature like this:

{
    "name": "some",
    "...": "...",
    "dependencies":
    [
        "..."
    ],
    "default-features":
    [
        "use-jpeg-turbo",
        "pdf-jpeg-turbo"
    ],
    "features":
    {
        "use-jpeg-original":
        {
            "description": "Use original JPEG library",
            "dependencies":
            [
                "jpeg"
            ]
        },
        "use-jpeg-turbo":
        {
            "description": "Use jpeg-turbo",
            "dependencies":
            [
                "jpeg-turbo"
            ]
        },
        "pdf-jpeg-original":
        {
            "description": "Enable PDF capabilities (using original JPEG)",
            "dependencies":
            [
                {
                    "name": "pdfium",
                    "features":
                    [
                        "with-jpeg-original"
                    ]
                }
            ]
        },
        "pdf-jpeg-turbo":
        {
            "description": "Enable PDF capabilities (using jpeg-turbo)",
            "dependencies":
            [
                {
                    "name": "pdfium",
                    "features":
                    [
                        "with-jpeg-turbo"
                    ]
                }
            ]
        }
    }
}

Which of course means that one can mess-up features by seting for example use-jpeg-turbo and pdf-jpeg-original at the same time, which will collide on dependencies installation.

CMake helpers

As you saw in portfile.cmake, one can use helper packages, which are not libraries for development but CMake modules for extending portfiles functionality. The vcpkg-cmake and vcpkg-cmake-config packages are examples of such helpers.

Naturally, you can create a helper like this yourself. Here’s a one I made (this version was not entirely correct, so it got updated later):

./ports/decovar-vcpkg-cmake
├── Config.cmake.in
├── Installing.cmake
├── decovar_vcpkg_cmake_ololo.cmake
├── license.txt
├── portfile.cmake
├── vcpkg-port-config.cmake
└── vcpkg.json

If the helper is supposed to provide CMake function(s) which could be used in portfiles, then it must contain vcpkg-port-config.cmake - these files get automatically imported/included by vcpkg. The contents of that file should be something like:

include("${CMAKE_CURRENT_LIST_DIR}/decovar_vcpkg_cmake_ololo.cmake")

The decovar_vcpkg_cmake_ololo.cmake in turn contains the following:

include_guard(GLOBAL)

function(decovar_vcpkg_cmake_ololo)
    message(STATUS "ololo")
endfunction()

So now any package that has decovar-vcpkg-cmake helper as a dependency will be able to call decovar_vcpkg_cmake_ololo() function in its portfile (but not in its CMakeLists.txt).

Apart from this module my helper also contains Installing.cmake and Config.cmake.in - my common/shared installation instructions for CMake projects that are lacking those. In order to use them, I thought, a project needs to add the following to its CMakeLists.txt:

#list(APPEND CMAKE_MODULE_PATH
# "${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/share/decovar-vcpkg-cmake"
#)
#include(Installing)
include("${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/share/decovar-vcpkg-cmake/Installing.cmake")

But that turned out to be not a correct way of shipping and including common CMake modules. It might work like this, but as you see I am relying on VCPKG_TARGET_TRIPLET here, and so it will fail when host dependencies are installed using a different triplet:

The following packages will be built and installed:
    jsoncpp[core]:x64-windows-static -> 1.9.5
    pdqsort[core]:x64-windows-static -> 2021.3.14
  * vcpkg-cmake[core]:x64-windows -> 2022-08-18
  * vcpkg-cmake-config[core]:x64-windows -> 2022-02-06
    zstd[core]:x64-windows-static -> 1.5.2

You see how everything but helpers is using x64-windows-static triplet, and helpers are using x64-windows? That will cause the include() statement to fail, because it will expect helper package to have a different path. To resolve this problem I thought that I could just set them both to the same value:

-DVCPKG_TARGET_TRIPLET=x64-windows-static -DVCPKG_HOST_TRIPLET=x64-windows-static

And then it works. But! Forcing host and target triplets to match is absolutely not correct, you should certainly not do that. As I later discovered, there is a CURRENT_HOST_INSTALLED_DIR variable that can be used in a portfile, and that made everything easier, as you’ll soon see.

For now, here’s the helper’s portfile.cmake:

if(VCPKG_CROSSCOMPILING)
    # should be FATAL_ERROR
    message(WARNING "${PORT} is a host-only port, mark it as a host dependency in your ports")
endif()

file(
    INSTALL
        # to be used in other projects CMakeLists.txt project files
        "${CMAKE_CURRENT_LIST_DIR}/Installing.cmake"
        # to be used in other projects CMakeLists.txt project files
        "${CMAKE_CURRENT_LIST_DIR}/Config.cmake.in"
        # to be used in other projects portfiles
        "${CMAKE_CURRENT_LIST_DIR}/decovar_vcpkg_cmake_ololo.cmake"
        # this one just includes decovar_vcpkg_cmake_ololo.cmake
        # and itself gets automatically included by vcpkg
        "${CMAKE_CURRENT_LIST_DIR}/vcpkg-port-config.cmake"
    DESTINATION
        "${CURRENT_PACKAGES_DIR}/share/${PORT}"
)

file(
    INSTALL
        "${CMAKE_CURRENT_LIST_DIR}/license.txt"
    DESTINATION
        "${CURRENT_PACKAGES_DIR}/share/${PORT}"
    RENAME copyright
)

set(VCPKG_POLICY_CMAKE_HELPER_PORT enabled)

The actual port that would be using this helper will be then able to copy the common files into its source directory using that CURRENT_HOST_INSTALLED_DIR variable, as it will always point to the host dependencies path:

file(COPY
    "${CURRENT_HOST_INSTALLED_DIR}/share/decovar-vcpkg-cmake/Installing.cmake"
    DESTINATION "${SOURCE_PATH}"
)
file(COPY
    "${CURRENT_HOST_INSTALLED_DIR}/share/decovar-vcpkg-cmake/Config.cmake.in"
    DESTINATION "${SOURCE_PATH}"
)

And then in project’s CMakeLists.txt it will be just this:

include("${CMAKE_CURRENT_SOURCE_DIR}/Installing.cmake")

You can take a look at how Dear ImGui port does it.

CMake wrapper

I couldn’t find documentation about this functionality, but it’s not very complex to figure out on your own. Basically, adding a CMake wrapper into your port allows you to “inject” CMake statements before or after find_package() call.

For example, let’s take Xerces-C++ port. It builds fine, but when you’ll try to actually use it in some project on Mac OS (using x64-osx or arm64-osx triplets), then you will get the following linking errors:

Undefined symbols for architecture arm64: 
  "_CFRelease", referenced from:
      xercesc_3_2::MacOSUnicodeConverter::upperCase(char16_t*) in libxerces-c.a(MacOSUnicodeConverter.cpp.o)
      xercesc_3_2::MacOSUnicodeConverter::lowerCase(char16_t*) in libxerces-c.a(MacOSUnicodeConverter.cpp.o)
  "_CFStringCreateMutableWithExternalCharactersNoCopy", referenced from:
      xercesc_3_2::MacOSUnicodeConverter::upperCase(char16_t*) in libxerces-c.a(MacOSUnicodeConverter.cpp.o)
      xercesc_3_2::MacOSUnicodeConverter::lowerCase(char16_t*) in libxerces-c.a(MacOSUnicodeConverter.cpp.o)
  "_CFStringLowercase", referenced from:
      xercesc_3_2::MacOSUnicodeConverter::lowerCase(char16_t*) in libxerces-c.a(MacOSUnicodeConverter.cpp.o)
  "_CFStringUppercase", referenced from:
      xercesc_3_2::MacOSUnicodeConverter::upperCase(char16_t*) in libxerces-c.a(MacOSUnicodeConverter.cpp.o)
...

This is because on Mac OS your project also needs to link to CoreServices framework (or rather Xerces-C++ requires that and so does your project).

Of course, you could resolve this by adding the following to your project’s CMakeLists.txt:

if(APPLE) # probably also check that it's exactly Mac OS, otherwise this also applies to iOS and others
    target_link_libraries(${CMAKE_PROJECT_NAME}
        PRIVATE
            "-framework CoreServices"
    )
endif()

But then you would need to add that to all your projects that depend on Xerces-C++. And a better option would be to do it once in the CMake wrapper that comes from the port.

As I understand it, the way it works is that you add a file named vcpkg-cmake-wrapper.cmake to your port, and it will “replace” (or actually wrap) the find_package() call. Here’s how it looks for Xerces-C++:

# do some stuff before the package will be attempted to be found
# [stuff]
# ...
_find_package(${ARGS})
# or after the package has been found
# [stuff]
# ...
# for instance, add required linking to CoreServices on Mac OS
if(APPLE) # probably also check that it's exactly Mac OS
    if(TARGET XercesC::XercesC)
        set_target_properties(XercesC::XercesC
            PROPERTIES
                INTERFACE_LINK_LIBRARIES
                    "-framework CoreServices"
        )
        list(APPEND XercesC_LIBRARIES
            "-framework CoreServices"
        )
    endif()
endif()

In this particular case it might be that linking to CoreServices could have been done in a better way, but it does work like this.

And then you install this wrapper into ${CURRENT_PACKAGES_DIR}/share/${PORT} path. But be careful to check that port name and actual package name are the same. If they are different, then it is important that you install the wrapper exacty into the package folder and not into the port folder, otherwise it won’t be used. For instance, in my registry Xerces-C++ port name is xerces-c and its package name is XercesC, and so here’s how I install the wrapper:

file(
    INSTALL "${CURRENT_PORT_DIR}/vcpkg-cmake-wrapper.cmake"
    DESTINATION "${CURRENT_PACKAGES_DIR}/share/XercesC"
)

Multiple targets/components in one project

I should warn you first that the stuff described in this section is my own understanding of the listed scenarios and how one is supposed to solve them. My understanding might be wrong, and there might be better ways of dealing with this, so perhaps you shouldn’t blindly follow these instructions.

Making CMake configs discoverable

Say, your project has several targets/components and the “main” target depends on some/all of them. For example, here’s Config.cmake.in template for the main target SomeLibrary:

@PACKAGE_INIT@

include(CMakeFindDependencyMacro)

find_dependency(AnotherLibrary CONFIG REQUIRED)
find_dependency(DifferentLibrary CONFIG REQUIRED)

include("${CMAKE_CURRENT_LIST_DIR}/@PROJECT_NAME@Targets.cmake")

check_required_components(@PROJECT_NAME@)

So there are two internal targets which it depends on: AnotherLibrary and DifferentLibrary.

If you will make a port for this project as you normally would:

# ...

vcpkg_cmake_install()

vcpkg_cmake_config_fixup(
    PACKAGE_NAME "SomeLibrary"
    CONFIG_PATH "cmake" # we install CMake configs here
)

…then projects that depend on this port will fail to configure:

CMake Error at /path/to/vcpkg/scripts/buildsystems/vcpkg.cmake:852 (_find_package):
  Could not find a package configuration file provided by "AnotherLibrary" with
  any of the following names:

    AnotherLibraryConfig.cmake
    AnotherLibrary-config.cmake

  Add the installation prefix of "AnotherLibrary" to CMAKE_PREFIX_PATH or set
  "AnotherLibrary_DIR" to a directory containing one of the above files.  If
  "AnotherLibrary" provides a separate development package or SDK, be sure it
  has been installed.
Call Stack (most recent call first):
  /path/to/cmake/share/cmake-3.25/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  build/vcpkg_installed/x64-windows/share/SomeLibrary/SomeLibraryConfig.cmake:30 (find_dependency)
  /path/to/vcpkg/scripts/buildsystems/vcpkg.cmake:852 (_find_package)
  CMakeLists.txt:292 (find_package)

…because AnotherLibrary*.cmake (as well as DifferentLibrary*.cmake and all other targets/components of your SomeLibrary project) configs are located inside build/vcpkg_installed/x64-windows/share/SomeLibrary/ folder, while CMake is looking for them one level up - in build/vcpkg_installed/x64-windows/share/COMPONENT-NAME-HERE/.

I do not know what is the proper way of resolving this. I imagine, the installation procedure of SomeLibrary project might need to be different, or maybe the main port needs to be split into several ports (one per component), but I don’t like either of these options, as they would require changing the project that installs just fine otherwise. So instead I’ve come up with the following:

# ...

vcpkg_cmake_install()

vcpkg_cmake_config_fixup(
    PACKAGE_NAME "SomeLibrary"
    CONFIG_PATH "cmake" # we install CMake configs here
)

# internal components/dependencies CMake configs need to be available in ${CURRENT_PACKAGES_DIR}/share/COMPONENT-NAME-HERE,
# otherwise find_dependency() in consuming project will fail to find them. Until a better/proper way is found,
# they will be moved out to those paths "manually"
#
# list of components
list(APPEND SomeLibraryComponents
    AnotherLibrary
    DifferentLibrary
)
foreach(SomeLibraryComponent ${SomeLibraryComponents})
    #message(STATUS "Moving out ${SomeLibraryComponent}")
    # collect this component's CMake configs into a list
    file(GLOB SomeLibraryComponentFiles "${CURRENT_PACKAGES_DIR}/share/SomeLibrary/${SomeLibraryComponent}*.cmake")
    #message(STATUS "${SomeLibraryComponent} files: ${SomeLibraryComponentFiles}")
    # create a folder for its configs where vcpkg would expect them to be
    file(MAKE_DIRECTORY
        "${CURRENT_PACKAGES_DIR}/share/${SomeLibraryComponent}"
    )
    # and move them there
    foreach(SomeLibraryComponentFile ${SomeLibraryComponentFiles})
        #message(STATUS "file: ${SomeLibraryComponentFile}")
        get_filename_component(SomeLibraryComponentFileName ${SomeLibraryComponentFile} NAME)
        file(RENAME ${SomeLibraryComponentFile} "${CURRENT_PACKAGES_DIR}/share/${SomeLibraryComponent}/${SomeLibraryComponentFileName}")
    endforeach()
endforeach()

So right after the usual CMake config fixup for the main target I iterate through the list of all other targets/components and move their configs one level up to their own folders. I know, this looks like a dirty hack, but it works.

Partial components installation

There is a different scenario. It could be that your project contains several targets/components, which can be built and installed without building the entire project:

$ cd /path/to/some/project
$ mkdir build && cd $_
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="../install" ..
$ cmake --build . --target SomeLibrary
$ cmake --install ./libraries/SomeLibrary
# or
# $ cmake --install . --component SomeLibrary

And then let’s say one of your customers/users is asking you to make a vcpkg port only for this one library. He doesn’t need your entire project (at the very least because it takes too long to build the whole thing), he only needs this one library. So what do you do?

I’d say, ideally, it might be a good idea to move them out from your super repository into their own repositories and add them as dependencies to the main repository via vcpkg. But that is not always possible, and it wasn’t possible in our case (horrible legacy ways of including internal headers spread around and other stuff like that), so let’s see what else can be done here.

As far as I can tell, vcpkg doesn’t not have an out-of-the-box functionality for this scenario. As I mentioned earlier, vcpkg_cmake_install simply calls the vcpkg_cmake_build() function with TARGET install parameter, which means that you too can set any build target for vcpkg_cmake_build():

# vcpkg_cmake_install() # we don't need the entire project
# build only one of the libraries/components
vcpkg_cmake_build(TARGET SomeLibrary)

But there is no CMake helper to run the installation (the vcpkg_cmake_install() function doesn’t not have such a parameter), so one just has to fallback to running a bare CMake CLI command:

vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Debug
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-dbg
)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Release
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-rel
)

vcpkg_cmake_config_fixup(
    PACKAGE_NAME "SomeLibrary"
    CONFIG_PATH "cmake" # we install CMake configs here
)

Yes, you need to install both Debug and Release configurations (and don’t forget about --config Debug/Release to account for multi-config generators). As you can see, that certainly doesn’t look too nice, but it works. It is probably worth creating a CMake helper to wrap this into something nicer.

Internal dependencies

As a bonus complexity, some of our libraries/components in the main repository in turn depend on other internal libraries/components within that repository, and those need to be installed too. And the problem here is that the consuming project won’t be able to discover that other library, because that library’s CMake configs for find_package() will end up in the SomeLibrary folder. A resolution/workaround for that would be to add even more crutches, so the whole thing would look like this:

# the "main" library
vcpkg_cmake_build(TARGET SomeLibrary)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Debug
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-dbg
)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Release
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-rel
)

# its dependency - another internal library
vcpkg_cmake_build(TARGET AnotherLibrary)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/AnotherLibrary --config Debug
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg
    LOGNAME build-AnotherLibrary-${TARGET_TRIPLET}-dbg
)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/AnotherLibrary --config Release
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
    LOGNAME build-AnotherLibrary-${TARGET_TRIPLET}-rel
)
# collect this other library CMake configs into a list
file(GLOB AnotherLibraryFiles ${CURRENT_PACKAGES_DIR}/cmake/AnotherLibrary*.cmake)
# create folders for its configs in proper paths, where vcpkg would expect them to be
file(MAKE_DIRECTORY
    ${CURRENT_PACKAGES_DIR}/share/AnotherLibrary
    ${CURRENT_PACKAGES_DIR}/debug/share/AnotherLibrary
)
# move out every AnotherLibrary's CMake config from SomeLibrary folder
# and copy them to both Debug and Release share folders
foreach(AnotherLibraryFile ${AnotherLibraryFiles})
    get_filename_component(AnotherLibraryFileName ${AnotherLibraryFile} NAME)
    file(RENAME ${AnotherLibraryFile} ${CURRENT_PACKAGES_DIR}/share/AnotherLibrary/${AnotherLibraryFileName})
    file(
        COPY ${CURRENT_PACKAGES_DIR}/share/AnotherLibrary/${AnotherLibraryFileName}
        DESTINATION ${CURRENT_PACKAGES_DIR}/debug/share/AnotherLibrary/
    )
endforeach()

# now fix the "main" library configs
vcpkg_cmake_config_fixup(
    PACKAGE_NAME "SomeLibrary"
    CONFIG_PATH "cmake"
)
# and the other one's
vcpkg_cmake_config_fixup(
    PACKAGE_NAME "AnotherLibrary"
)

So now the portfile looks even worse, but it does work fine, so untill a better alternative comes up, that will be the way.

As I already mentioned, it would probably be better to make AnotherLibrary its own port, which SomeLibrary could depend on, but then they might eventually end up having different REF commit values, causing your users to fetch different snapshots of the same repository (which would be the least of potential problems caused by this).

Header-only libraries

In case of a header-only INTERFACE library there is nothing to build, so calling vcpkg_cmake_build(...) is redundant, and so is calling vcpkg_execute_build_process() for Debug configuration, and so there is a “shorter” way:

# the "main" library
vcpkg_cmake_build(TARGET SomeLibrary)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Debug
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-dbg
)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Release
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-rel
)
vcpkg_cmake_config_fixup(
    PACKAGE_NAME "SomeLibrary"
    CONFIG_PATH "cmake"
)

# its header-only dependencies within the project
list(APPEND internalComponents
    ThatsOneHeaderOnlyLibrary
    ThatsAnotherHeaderOnlyLibrary
)
foreach(internalComponent ${internalComponents})
    # these are header-only INTERFACE libraries, so there is nothing to build
    #vcpkg_cmake_build(TARGET internalComponent)
    vcpkg_execute_build_process(
        COMMAND cmake --install . --component ${internalComponent} # if it can be installed as component
        WORKING_DIRECTORY "${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel"
        LOGNAME "build-${internalComponent}-${TARGET_TRIPLET}-rel"
    )
    file(GLOB internalComponentFiles "${CURRENT_PACKAGES_DIR}/cmake/${internalComponent}*.cmake")
    file(MAKE_DIRECTORY
        "${CURRENT_PACKAGES_DIR}/share/${internalComponent}"
    )
    foreach(internalComponentFile ${internalComponentFiles})
        get_filename_component(internalComponentFileName ${internalComponentFile} NAME)
        file(RENAME ${internalComponentFile} "${CURRENT_PACKAGES_DIR}/share/${internalComponent}/${internalComponentFileName}")
    endforeach()
    foreach(cmakeConfigSuffix "Config" "Targets") # in case some of those don't have proper configs
        set(cmakeConfig "${CURRENT_PACKAGES_DIR}/share/${internalComponent}/${internalComponent}${cmakeConfigSuffix}.cmake")
        if(EXISTS ${cmakeConfig})
            # basically, that replaces vcpkg_cmake_config_fixup(), which will actually fail to execute
            # in this case due to missing debug/share/${internalComponent}
            #
            # this is also the first time I ever used this "bracket argument" thing: [=[ ... ]=]
            # and turns out it is very convenient for such cases with lots of quotes and special symbols
            vcpkg_replace_string(
                "${cmakeConfig}"
                [=[get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)]=]
                [=[get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
cmake_path(GET _IMPORT_PREFIX PARENT_PATH _IMPORT_PREFIX)]=]
            )
        endif()
    endforeach()
endforeach()

So here we don’t build those header-only internal dependencies, as there is nothing to build, and we install only -rel, as -dbg would redundantly install the same things.

If you don’t do that vcpkg_replace_string() fixup of paths, then trying to use this port in a project will result in the following error:

Imported target "ThatsOneHeaderOnlyLibrary" includes non-existent path
  /path/to/vcpkg_installed/x64-linux/share/include/ThatsOneHeaderOnlyLibrary
in its INTERFACE_INCLUDE_DIRECTORIES

Now, while this “shorter” way works, it actually isn’t that short, and also this self-made paths fixup doesn’t look all that reliable, so it’s probably better to do this the usual “longer” way and with using vcpkg_cmake_config_fixup() instead:

# the "main" library
vcpkg_cmake_build(TARGET SomeLibrary)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Debug
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-dbg
)
vcpkg_execute_build_process(
    COMMAND cmake --install ./libraries/SomeLibrary --config Release
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
    LOGNAME build-SomeLibrary-${TARGET_TRIPLET}-rel
)
vcpkg_cmake_config_fixup(
    PACKAGE_NAME "SomeLibrary"
    CONFIG_PATH "cmake"
)

# its header-only dependencies within the project
list(APPEND internalComponents
    ThatsOneHeaderOnlyLibrary
    ThatsAnotherHeaderOnlyLibrary
)
foreach(internalComponent ${internalComponents})
    # these are header-only INTERFACE libraries, so there is nothing to build
    #vcpkg_cmake_build(TARGET internalComponent)
    vcpkg_execute_build_process(
        COMMAND cmake --install . --component ${internalComponent} # if it can be installed as component
        WORKING_DIRECTORY "${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel"
        LOGNAME "build-${internalComponent}-${TARGET_TRIPLET}-rel"
    )
    # even though installing Debug configuration is redundant here,
    # without it you won't be able to use vcpkg_cmake_config_fixup()
    vcpkg_execute_build_process(
        COMMAND cmake --install . --component ${internalComponent} # if it can be installed as component
        WORKING_DIRECTORY "${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg"
        LOGNAME "build-${internalComponent}-${TARGET_TRIPLET}-dbg"
    )
    file(GLOB internalComponentFiles "${CURRENT_PACKAGES_DIR}/cmake/${internalComponent}*.cmake")
    file(MAKE_DIRECTORY
        "${CURRENT_PACKAGES_DIR}/share/${internalComponent}"
        "${CURRENT_PACKAGES_DIR}/debug/share/${internalComponent}"
    )
    foreach(internalComponentFile ${internalComponentFiles})
        get_filename_component(internalComponentFileName ${internalComponentFile} NAME)
        file(RENAME ${internalComponentFile} "${CURRENT_PACKAGES_DIR}/share/${internalComponent}/${internalComponentFileName}")
        file(
            COPY ${CURRENT_PACKAGES_DIR}/share/${internalComponent}/${internalComponentFileName}
            DESTINATION ${CURRENT_PACKAGES_DIR}/debug/share/${internalComponent}/
        )
    endforeach()
    # that will adjust the paths for INTERFACE_INCLUDE_DIRECTORIES
    vcpkg_cmake_config_fixup(
        PACKAGE_NAME "${internalComponent}"
    )
endforeach()

So basically we came back to all the same steps as with “regular” internal dependencies with the only difference that for header-only libraries you don’t execute vcpkg_cmake_build().

Sparse checkout

While we are here, when you only need to build just some targets of your big project, it might be an overkill to make your users download your entire repository, especially if it’s rather big (even as a snapshot).

Partial repository cloning can be done with sparse checkout. For example, if I only want to get the SomeLibrary component/target from one of my projects:

$ git clone --depth 1 --filter=blob:none --sparse git@github.com:retifrav/cmake-library-example.git
$ cd ./cmake-library-example
$ git sparse-checkout set --no-cone internal-project/libraries/SomeLibrary

But default vcpkg CMake functions such as vcpkg_from_git unfortunately do not have an option for sparse cloning and checkout, so I’ll probably implement my own custom function/helper for that (or at least register a feature request for it).

Git submodules

A project repository might have Git submodules, which complicates things. If those submodules are used for resolving 3rd-party dependencies, then I would disable them and make vcpkg ports for those dependencies to resolve them in a civilized manner. However, if project uses submodules to resolve its own components, then I would say that this is an okay scenario (although a rather inconvenient one), and so one might probably prefer not move those into their own ports (but that of course depends on a particulal project).

Here’s an example of one such project - AWS SDK. It has one direct submodule, which in turn has several recursive submodules.

Having recursive submodules makes it rather difficult to create a port for such a project, because vcpkg does not have an out-of-the-box function for cloning repositories with submodules, so we would need to resort to running “bare” Git commands with CMake, and it would be something like:

# on some Windows hosts trying to execute Git commands might fail with a bare `git`,
# so you have to explicitly `find_program()` it first
find_program(GIT git REQUIRED)

# if you work with more than one Git service, you might need to distinguish
# the sources directories, hence the `src-github`
set(SOURCE_PATH ${CURRENT_BUILDTREES_DIR}/src-github/v${VERSION})

# thanks to Git being able to use tags in `--branch`, we don't need to clone
# the entire repository (which in case of AWS SDK is huge), so we are cloning
# just this particular commit behind the version tag
vcpkg_execute_required_process(
    COMMAND ${GIT} clone --depth=1 --branch ${VERSION} --recurse-submodules git@github.com:aws/aws-sdk-cpp.git ${SOURCE_PATH}
    WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}
    LOGNAME git-${PORT}-cloning
)

Since we are not using “standard” vcpkg functions, we are missing several very nice and convenient things which vcpkg does behind the scenes, such as “caching” a repository snapshot archive and cleaning the source directory, so we will need to implements all that ourselves.

One way to go about not cloning the same repository over and over and also cleaning the source directory is to reset and clean the repository, which includes resetting and cleaning the submodules too, so here comes an updated procedure:

# if there is no Git repository in the source directory, then do the cloning
if(NOT EXISTS ${SOURCE_PATH}/.git)
    message(STATUS "Cloning the repository with recursive submodules")
    vcpkg_execute_required_process(
        COMMAND ${GIT} clone --depth=1 --branch ${VERSION} --recurse-submodules git@github.com:aws/aws-sdk-cpp.git ${SOURCE_PATH}
        WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}
        LOGNAME git-${PORT}-cloning
    )
else() # otherwise just reset and clean the repository and its submodules
    message(STATUS "Resetting and cleaning up the repository")
    #
    message(STATUS "+ resetting the main repository")
    vcpkg_execute_required_process(
        COMMAND ${GIT} reset --hard ${VERSION}
        WORKING_DIRECTORY ${SOURCE_PATH}
        LOGNAME git-${PORT}-main-resetting
    )
    message(STATUS "+ cleaning up the main repository")
    vcpkg_execute_required_process(
        COMMAND ${GIT} clean -x -f -d
        WORKING_DIRECTORY ${SOURCE_PATH}
        LOGNAME git-${PORT}-main-cleaning
    )
    message(STATUS "+ resetting the submodules")
    vcpkg_execute_required_process(
        COMMAND ${GIT} submodule foreach --recursive git reset --hard
        WORKING_DIRECTORY ${SOURCE_PATH}
        LOGNAME git-${PORT}-submodules-resetting
    )
    message(STATUS "+ cleaning up the submodules")
    vcpkg_execute_required_process(
        COMMAND ${GIT} submodule foreach --recursive git clean -x -f -d
        WORKING_DIRECTORY ${SOURCE_PATH}
        LOGNAME git-${PORT}-submodules-cleaning
    )
endif()

That was the procedure for obtaining the sources of the project and its submodularized components. Now we need to patch some stuff in the original sources, and again, since we did not use “standard” vcpkg_from_git() function, patching will have to be done with bare Git commands too:

set(PATCH_NAME "001-not-hardcoding-shared-libraries-and-crt.patch")
message(STATUS "Applying patch ${PATCH_NAME}")
vcpkg_execute_required_process(
    COMMAND ${GIT} apply "${CMAKE_CURRENT_LIST_DIR}/${PATCH_NAME}"
    WORKING_DIRECTORY ${SOURCE_PATH}
    LOGNAME git-${PORT}-patching-${PATCH_NAME}
)

And that is quite alright as long as we need to patch stuff in the main project. But if we’ll need to patch something in one of the submodules, then we’ll need to go deeper into the abyss and “downgrade” to the crude strings replacement, such as:

# can't patch that, because this is inside a submodule
vcpkg_replace_string(
    "${SOURCE_PATH}/crt/aws-crt-cpp/CMakeLists.txt"
        [=[# set runtime library
if(MSVC)
    if(AWS_STATIC_MSVC_RUNTIME_LIBRARY OR STATIC_CRT)
        target_compile_options(${PROJECT_NAME} PRIVATE "/MT$<$<CONFIG:Debug>:d>")
    else()
        target_compile_options(${PROJECT_NAME} PRIVATE "/MD$<$<CONFIG:Debug>:d>")
    endif()
endif()]=]
        [=[# these things should be handled with CMAKE_MSVC_RUNTIME_LIBRARY]=]
)

# can't patch that, because this is inside a submodule
vcpkg_replace_string(
    "${SOURCE_PATH}/crt/aws-crt-cpp/crt/aws-c-common/CMakeLists.txt"
        [=[DESTINATION "${LIBRARY_DIRECTORY}/cmake"]=]
        [=[DESTINATION "${CMAKE_INSTALL_DATAROOTDIR}/${PROJECT_NAME}"]=]
)

And here the CMake’s bracket argument helps tremendously.

Actually, you could still do Git patches, but in my opinion this would be more complicated in terms of the port maintenance. You can try that for yourself and make up your own opinion, though.

Then comes the project configuration, but that is nothing special - just the usual vcpkg_cmake_configure(). But then comes the building and installation, and if you’ll need to build not the entire project but just some of its components, then it means executing bare CMake commands again - same way it is described in the section about multiple targets/components:

# the `REQUESTED_AWS_COMPONENTS` list is composed based on enabled features
foreach(AWS_COMPONENT_NAME ${REQUESTED_AWS_COMPONENTS})
    message(STATUS "+ ${AWS_COMPONENT_NAME}")
    # build
    vcpkg_cmake_build(TARGET ${AWS_COMPONENT_NAME})
    # install Debug
    vcpkg_execute_build_process(
        COMMAND ${CMAKE_COMMAND} --install ./generated/src/${AWS_COMPONENT_NAME} --config Debug
        WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg
        LOGNAME install-${AWS_COMPONENT_NAME}-${TARGET_TRIPLET}-dbg
    )
    # install Release
    vcpkg_execute_build_process(
        COMMAND ${CMAKE_COMMAND} --install ./generated/src/${AWS_COMPONENT_NAME} --config Release
        WORKING_DIRECTORY ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel
        LOGNAME install-${AWS_COMPONENT_NAME}-${TARGET_TRIPLET}-rel
    )
endforeach()

Note the ${CMAKE_COMMAND} - I didn’t talk about this in the previous section, but in some environments (GNU/Linux, for example) trying to use a bare cmake might fail with an error like:

Failed to execute command "cmake --install TARGET-NAME --config Debug" in working directory "/path/to/build/folder": no such file or directory

and this no such file or directory is actually about missing cmake somehow. But unlike Git, you don’t need to find_program() it, just use the CMAKE_COMMAND.

In general, this is it, the rest of the portfile should be pretty standard, but in the particular case of AWS SDK there are some additional nasty details, and if you are interested in those, you can inspect the complete port here.

Mirroring a repository with submodules

I wouldn’t be surprised if you wanted to mirror this project’s repository in your own in-house Git service. But how does one do that for a GitHub-hosted repository that contains Git submodules, which are hosted on that same GitHub (or some other 3rd-party Git service)?

The only way I found so far is to:

Create empty repositories for the project repository and its submodules in your own Git service;
Clone the main repository and its recursive submodules locally on your machine;
Modify the original project’s .gitmodules file and the recursive .gitmodules files in its submodules by replacing original repositories GitHub URLs with your own Git service URLs;
- that inevitably results in different Git commit hashes, which won’t match the original hashes for the same version, which is rather unfortunate;
Commit the changes and push every repository to your own Git service;
- you might want to commit and push those modified repositories as “snapshots”, which means that they would have to be synced with the original sources manually every time you’d like to add a new version to your mirror.

With AWS SDK project as an example, the submodules hierarchy is the following:

aws-sdk-cpp (snapshot with overridden submodules URLs)
- aws-crt-cpp (snapshot with overridden submodules URLs)
  - aws-c-auth
  - aws-c-cal
  - aws-c-common
  - aws-c-compression
  - aws-c-event-stream
  - aws-c-http
  - aws-c-io
  - aws-c-mqtt
  - aws-c-s3
  - aws-c-sdkutils
  - aws-checksums
  - aws-lc
  - s2n-tls (snapshot with overridden submodules URLs)
    - aws-verification-model-for-libcrypto

In order to commit those “snapshots”, you will need to add the submodules hashes into their repositories on you local machine. To find the Git hashes of the submodules in a particular version of the parent repository(ies), you will need to initialize those submodules and list the hashes, so for example like this:

$ cd /path/to/aws-sdk-cpp
$ git submodule update --init --recursive

# here 349b806d491366d30b78531ee9d00b677427c834 is already our modified hash, so you won't find it in the original AWS repository
$ git ls-tree master ./crt/aws-crt-cpp
160000 commit 349b806d491366d30b78531ee9d00b677427c834  crt/aws-crt-cpp

$ cd ./crt
/aws-crt-cpp
$ git ls-tree master ./crt/aws-c-auth
160000 commit 877c029fc4e93d205f9c6855188c3c51f6b497b4  crt/aws-c-auth

And then those hashes need to be added to parent repository(ies):

# needs to be cloned to somewhere else separately
$ cd /path/to/aws-crt-cpp

$ export MODULE_NAME=aws-c-auth
$ git submodule add -- git@YOUR.GIT.HOST:3rd-party/$MODULE_NAME.git crt/$MODULE_NAME && cd ./crt/$MODULE_NAME && git checkout 877c029fc4e93d205f9c6855188c3c51f6b497b4 && cd -

And so on and so on for every submodule. Quite a tiresome procedure, especially knowing that you’ll need to keep updating all of those to newer versions.

Installing ports from a registry

Without a project

You can try to install the GLFW port even without having a project that would depend on it. It is a convenient ability to have when you want to quickly test your new port:

$ cd /path/to/your/vcpkg-registry
$ vcpkg install glfw --overlay-ports=./ports/glfw
Computing installation plan...
The following packages will be built and installed:
    glfw[core]:x64-osx -> 3.3.8 -- /path/to/your/vcpkg-registry/./ports/glfw
  * vcpkg-cmake[core]:x64-osx -> 2022-08-18
  * vcpkg-cmake-config[core]:x64-osx -> 2022-02-06#1
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-osx...
Restored 1 package(s) from /Users/USERNAME/.cache/vcpkg/archives in 23.44 ms. Use --debug to see more details.
Installing 1/3 vcpkg-cmake:x64-osx...
Elapsed time to handle vcpkg-cmake:x64-osx: 7.694 ms
Installing 2/3 vcpkg-cmake-config:x64-osx...
Building vcpkg-cmake-config[core]:x64-osx...
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/share/vcpkg-cmake-config/vcpkg-port-config.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/share/vcpkg-cmake-config/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/32/32dd70a7ba46fe85a359ed1b3a9b0c98772888eb59daefe61cb52f6560338044.zip"
Elapsed time to handle vcpkg-cmake-config:x64-osx: 58.43 ms
Installing 3/3 glfw:x64-osx...
Building glfw[core]:x64-osx...
-- Installing port from location: /path/to/your/vcpkg-registry/./ports/glfw
-- Using cached /path/to/programs/vcpkg/downloads/glfw-7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz
-- Cleaning sources at /path/to/programs/vcpkg/buildtrees/glfw/src/172a7f6c9e-d7a30c254e.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /path/to/programs/vcpkg/downloads/glfw-7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/glfw/src/172a7f6c9e-d7a30c254e.clean
-- Found external ninja('1.11.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Installing: /path/to/programs/vcpkg/packages/glfw_x64-osx/share/glfw/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/c0/c05ed39487e7538dcd2436b1cc00ac256091ec6df15f9dbdd6c71c0f884da654.zip"
Elapsed time to handle glfw:x64-osx: 5.9 s

Total elapsed time: 14.15 s

glfw provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(glfw3 CONFIG REQUIRED)
    target_link_libraries(main PRIVATE glfw)

Here’s what it did:

Fetched and installed host dependencies (vcpkg-cmake and vcpkg-cmake-config);
Fetched, configured, built and installed GLFW, both Debug and Release configurations (there will be a note about debug subfolder later);
Saved the sources and build artifacts in local cache, so they could be restored for later re-use instead of fetching and building them again;
Tried to discover/guess CMake statements for using these packages in your project (find_package() and target_link_libraries() commands). In most cases it does that correctly.

The resulting binaries are placed here:

/path/to/programs/vcpkg/packages/glfw_x64-osx
/path/to/programs/vcpkg/installed/x64-osx
/Users/USERNAME/.cache/vcpkg/archives/c0/c05ed39487e7538dcd2436b1cc00ac256091ec6df15f9dbdd6c71c0f884da654.zip (that’s the cached pre-built package)

Note that it installed the artifacts not in the /path/to/your/vcpkg-registry, but to where vcpkg is installed in your system. Later when we’ll be resolving dependencies for a project, the artifacts will be installed to its build folder.

If we now delete the pre-built artifacts from vcpkg folder (but not from cache) and try to install the package again, this operation will run much faster:

$ rm -r /path/to/programs/vcpkg/packages/glfw_x64-osx
$ rm -r /path/to/programs/vcpkg/installed/*

$ vcpkg install glfw --overlay-ports=./ports/glfw
Computing installation plan...
The following packages will be built and installed:
    glfw[core]:x64-osx -> 3.3.8 -- /Users/USERNAME/code/cpp/vcpkg-registry/./ports/glfw
  * vcpkg-cmake[core]:x64-osx -> 2022-08-18
  * vcpkg-cmake-config[core]:x64-osx -> 2022-02-06#1
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-osx...
Restored 3 package(s) from /Users/USERNAME/.cache/vcpkg/archives in 24.94 ms. Use --debug to see more details.
Installing 1/3 vcpkg-cmake:x64-osx...
Elapsed time to handle vcpkg-cmake:x64-osx: 3.579 ms
Installing 2/3 vcpkg-cmake-config:x64-osx...
Elapsed time to handle vcpkg-cmake-config:x64-osx: 2.45 ms
Installing 3/3 glfw:x64-osx...
Elapsed time to handle glfw:x64-osx: 6.662 ms

Total elapsed time: 2.254 s

glfw provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(glfw3 CONFIG REQUIRED)
    target_link_libraries(main PRIVATE glfw)

As you can see, this time it just restored pre-built artifacts that were already available in cache.

Dummy installation

This still doesn’t count as a proper project, because there is no CMake project file or anything at all really, but also this is not exactly the same as installing from within a registry using --overlay-ports. This option might be convenient when you want to quickly test not one, not all, but a selected set of ports.

Create an empty dummy folder and put there vcpkg-configuration.json:

{
    "default-registry":
    {
        "kind": "git",
        "repository": "git@github.com:retifrav/vcpkg-registry.git",
        "baseline": "4b4ae3fea063fc04c2d5a6089c8aa7c2d0129879"
    },
    "registries": []
}

and vcpkg.json:

{
    "name": "some",
    "version": "0",
    "dependencies":
    [
        "glad",
        "glfw"
    ]
}

Here it will go with default versions and features, but you can of course specify anything you’d need just like you would do it in an actual project.

And now you can install these ports:

$ cd /path/to/dummy
$ tree .
├── vcpkg-configuration.json
└── vcpkg.json

$ vcpkg install --triplet=x64-windows-static --host-triplet=x64-windows-static

Overriding triplets is here just as an example, especially that you probably will never need to override --host-triplet.

Furthermore, you can now refer to installed libraries from a non-vcpkg project just like you would with any other pre-built dependency:

$ cd /path/to/some
$ mkdir build && cd $_
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_PREFIX_PATH="/path/to/dummy/vcpkg_installed/x64-windows-static" \
    ..

In a project

I’ll use my GLFW Dear ImGui application created in this article as an example. It already has an option of resolving dependencies with Conan, and now I’ll add an option of resolving the same dependencies with vcpkg.

Dependencies are listed in the project’s vcpkg.json:

{
    "name": "glfw-imgui-example",
    "version": "0",
    "dependencies":
    [
        "glad",
        "glfw",
        {
            "name": "dearimgui",
            "features":
            [
                "backend-glfw"
            ]
        }
    ]
}

The glfw dependency is not really needed here, because it will still be added through dearimgui, as we are saying here that we want it with backend-glfw feature, and that one adds glfw dependency. But it wouldn’t hurt to set it explicitly, as the project does depend on it regardless of Dear ImGui.

To tell vcpkg, which registry I want to use, I’ve set the following in the vcpkg-configuration.json:

{
    "default-registry":
    {
        "kind": "git",
        "repository": "git@github.com:retifrav/vcpkg-registry.git",
        "baseline": "5c1a089c3a542dfbe818625bf4a3dadfb834e2af"
    },
    "registries": []
}

Now we should be able to get all the dependencies with vcpkg and configure the project:

$ cd /path/to/glfw-imgui-example
$ mkdir build && cd $_
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="../install" \
    -DUSING_PACKAGE_MANAGER_VCPKG=1 \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    ..
-- Running vcpkg install
Fetching registry information from git@github.com:retifrav/vcpkg-registry.git (HEAD)...
Detecting compiler hash for triplet x64-osx...
The following packages will be built and installed:
    dearimgui[backend-glfw,core]:x64-osx -> 1.88.0 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/c3282e78368406b65f57ef0e3afa7cda2fc26501
  * decovar-vcpkg-cmake[core]:x64-osx -> 2022-10-15 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/4ceb1e31d28a155bcbeb26382478b024a3d82cd3
    glad[core]:x64-osx -> 0.1.36 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/2341f5144ce8e76a256289517d61abb4ab9fb72c
    glfw[core]:x64-osx -> 3.3.8 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/597fa07e1afd57c50dfdbeb0c0d28f4157748564
  * vcpkg-cmake[core]:x64-osx -> 2022-08-18 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/84c200e8e625d4d99b1649525fcdf81a73197078
  * vcpkg-cmake-config[core]:x64-osx -> 2022-02-06 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/e23b39e21f0dd42ecc615262640d211c39696aa1
Additional packages (*) will be modified to complete this operation.
Restored 0 package(s) from /Users/USERNAME/.cache/vcpkg/archives in 33.41 us. Use --debug to see more details.
Installing 1/6 vcpkg-cmake-config:x64-osx...
Building vcpkg-cmake-config[core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/e23b39e21f0dd42ecc615262640d211c39696aa1
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/share/vcpkg-cmake-config/vcpkg-port-config.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake-config_x64-osx/share/vcpkg-cmake-config/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/5e/5ede033072a78c3aa60ffe343b1248c10ad943b52e64658f524bac40637f500b.zip"
Elapsed time to handle vcpkg-cmake-config:x64-osx: 53.77 ms
Installing 2/6 vcpkg-cmake:x64-osx...
Building vcpkg-cmake[core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/84c200e8e625d4d99b1649525fcdf81a73197078
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake_x64-osx/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake_x64-osx/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake_x64-osx/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake_x64-osx/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: /path/to/programs/vcpkg/packages/vcpkg-cmake_x64-osx/share/vcpkg-cmake/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/6e/6e6061f9ad1b3a6deb062796316d813c6088ecd79e1a5b317f8d61d77720d6e9.zip"
Elapsed time to handle vcpkg-cmake:x64-osx: 53.31 ms
Installing 3/6 decovar-vcpkg-cmake:x64-osx...
Building decovar-vcpkg-cmake[core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/4ceb1e31d28a155bcbeb26382478b024a3d82cd3
-- Installing: /path/to/programs/vcpkg/packages/decovar-vcpkg-cmake_x64-osx/share/decovar-vcpkg-cmake/Installing.cmake
-- Installing: /path/to/programs/vcpkg/packages/decovar-vcpkg-cmake_x64-osx/share/decovar-vcpkg-cmake/Config.cmake.in
-- Installing: /path/to/programs/vcpkg/packages/decovar-vcpkg-cmake_x64-osx/share/decovar-vcpkg-cmake/decovar_vcpkg_cmake_ololo.cmake
-- Installing: /path/to/programs/vcpkg/packages/decovar-vcpkg-cmake_x64-osx/share/decovar-vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: /path/to/programs/vcpkg/packages/decovar-vcpkg-cmake_x64-osx/share/decovar-vcpkg-cmake/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/aa/aa3c4e402f180f3e686bc6fa970e01c69c12dbcef95897634f58e6158fb194fa.zip"
Elapsed time to handle decovar-vcpkg-cmake:x64-osx: 93.32 ms
Installing 4/6 glfw:x64-osx...
Building glfw[core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/597fa07e1afd57c50dfdbeb0c0d28f4157748564
-- Fetching git@github.com:glfw/glfw.git 7482de6071d21db77a7236155da44c172a7f6c9e...
-- Extracting source /path/to/programs/vcpkg/downloads/glfw-7482de6071d21db77a7236155da44c172a7f6c9e.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/glfw/src/172a7f6c9e-d7a30c254e.clean
-- Found external ninja('1.11.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Installing: /path/to/programs/vcpkg/packages/glfw_x64-osx/share/glfw/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/02/02113b742df9138712913194efa54f3ef51eb01fd3e434fc4be9b406f44161c0.zip"
Elapsed time to handle glfw:x64-osx: 7.773 s
Installing 5/6 dearimgui:x64-osx...
Building dearimgui[backend-glfw,core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/c3282e78368406b65f57ef0e3afa7cda2fc26501
-- Fetching git@github.com:ocornut/imgui.git 9aae45eb4a05a5a1f96be1ef37eb503a12ceb889...
-- Extracting source /path/to/programs/vcpkg/downloads/dearimgui-9aae45eb4a05a5a1f96be1ef37eb503a12ceb889.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/dearimgui/src/3a12ceb889-c011915ee6.clean
-- Found external ninja('1.11.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Installing: /path/to/programs/vcpkg/packages/dearimgui_x64-osx/share/dearimgui/copyright
-- ololo
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/c3/c3ba0c72ccdb98a816bd20555ab9cf5399687d6626bc943501b3ac9529f5cba2.zip"
Elapsed time to handle dearimgui:x64-osx: 13.51 s
Installing 6/6 glad:x64-osx...
Building glad[core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/2341f5144ce8e76a256289517d61abb4ab9fb72c
-- Fetching git@github.com:Dav1dde/glad.git 1ecd45775d96f35170458e6b148eb0708967e402...
-- Extracting source /path/to/programs/vcpkg/downloads/glad-1ecd45775d96f35170458e6b148eb0708967e402.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/glad/src/708967e402-a791cb9077.clean
-- Found external ninja('1.11.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Installing: /path/to/programs/vcpkg/packages/glad_x64-osx/share/glad/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/83/8309811b7ffd5fa5ee71aada208fab78e9e24cd176ba9083576b00c363bec638.zip"
Elapsed time to handle glad:x64-osx: 5.753 s

Total elapsed time: 33.97 s
glfw provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(glfw3 CONFIG REQUIRED)
    target_link_libraries(main PRIVATE glfw)

dearimgui provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(dearimgui CONFIG REQUIRED)
    target_link_libraries(main PRIVATE dearimgui)

glad provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(glad CONFIG REQUIRED)
    target_link_libraries(main PRIVATE glad::glad)

-- Running vcpkg install - done
-- The C compiler identification is AppleClang 14.0.0.14000029
-- The CXX compiler identification is AppleClang 14.0.0.14000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenGL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/System/Library/Frameworks/OpenGL.framework
-- Configuring done
-- Generating done
-- Build files have been written to: /path/to/glfw-imgui-example/build

It might be rather unexpected, but for Debug configuration vcpkg creates a debug subfolder inside build directory (/path/to/glfw-imgui-example/build/vcpkg_installed/x64-osx/debug) and puts Debug artifacts there, which can be confusing at first and will cause many projects to produce various vcpkg warnings (redundant include/share/etc folders inside debug and so on). Someone told me that this is the only proper/possible way of building several configurations on different platforms, and so that is why vcpkg does it so. I still don’t quite get why then not to just have a folder per configuration instead of nesting debug, but okay.

Unrelated to that, you might have noticed an ololo message being printed out during the installation of dearimgui port (version 1.88.0). That message came from decovar-vcpkg-cmake helper port, which dearimgui port depends on and calls decovar_vcpkg_cmake_ololo() function from.

The total time to fetch all dependencies sources, configure and build them and install (and cache) the binaries is 34 seconds. If we now remove everything from the build folder and try again:

$ rm -r ./*; rm .ninja*; ls -lah

$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="../install" \
    -DUSING_PACKAGE_MANAGER_VCPKG=1 \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    ..
-- Running vcpkg install
Fetching registry information from git@github.com:retifrav/vcpkg-registry.git (HEAD)...
Detecting compiler hash for triplet x64-osx...
The following packages will be built and installed:
    dearimgui[backend-glfw,core]:x64-osx -> 1.88.0 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/c3282e78368406b65f57ef0e3afa7cda2fc26501
  * decovar-vcpkg-cmake[core]:x64-osx -> 2022-10-15 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/4ceb1e31d28a155bcbeb26382478b024a3d82cd3
    glad[core]:x64-osx -> 0.1.36 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/2341f5144ce8e76a256289517d61abb4ab9fb72c
    glfw[core]:x64-osx -> 3.3.8 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/597fa07e1afd57c50dfdbeb0c0d28f4157748564
  * vcpkg-cmake[core]:x64-osx -> 2022-08-18 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/84c200e8e625d4d99b1649525fcdf81a73197078
  * vcpkg-cmake-config[core]:x64-osx -> 2022-02-06 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/e23b39e21f0dd42ecc615262640d211c39696aa1
Additional packages (*) will be modified to complete this operation.
Restored 6 package(s) from /Users/USERNAME/.cache/vcpkg/archives in 74.03 ms. Use --debug to see more details.
Installing 1/6 vcpkg-cmake-config:x64-osx...
Elapsed time to handle vcpkg-cmake-config:x64-osx: 5.364 ms
Installing 2/6 vcpkg-cmake:x64-osx...
Elapsed time to handle vcpkg-cmake:x64-osx: 4.532 ms
Installing 3/6 decovar-vcpkg-cmake:x64-osx...
Elapsed time to handle decovar-vcpkg-cmake:x64-osx: 6.853 ms
Installing 4/6 glfw:x64-osx...
Elapsed time to handle glfw:x64-osx: 14.2 ms
Installing 5/6 dearimgui:x64-osx...
Elapsed time to handle dearimgui:x64-osx: 31.32 ms
Installing 6/6 glad:x64-osx...
Elapsed time to handle glad:x64-osx: 31.76 ms

Total elapsed time: 4.212 s
glfw provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(glfw3 CONFIG REQUIRED)
    target_link_libraries(main PRIVATE glfw)

dearimgui provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(dearimgui CONFIG REQUIRED)
    target_link_libraries(main PRIVATE dearimgui)

glad provides CMake targets:

    # this is heuristically generated, and may not be correct
    find_package(glad CONFIG REQUIRED)
    target_link_libraries(main PRIVATE glad::glad)

-- Running vcpkg install - done
-- The C compiler identification is AppleClang 14.0.0.14000029
-- The CXX compiler identification is AppleClang 14.0.0.14000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenGL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/System/Library/Frameworks/OpenGL.framework
-- Configuring done
-- Generating done
-- Build files have been written to: /path/to/glfw-imgui-example/build

…then there is no ololo message being printed anymore, and the whole vcpkg stage took only only 4 seconds, because this time it didn’t need to fetch sources and build binaries - it just restored already pre-built ones from cache.

Building the project:

$ time cmake --build .
[3/3] Linking CXX executable glfw-imgui

real    0m0.958s
user    0m1.330s
sys    0m0.194s

$ cat .ninja_log
# ninja log v5
1    585    1666772953298222365    CMakeFiles/glfw-imgui.dir/functions.cpp.o    38458c79e781fe79
1    794    1666772953508206892    CMakeFiles/glfw-imgui.dir/main.cpp.o    41b1d124fcaa4881
794    927    1666772953635024427    glfw-imgui    b12d88b40ff7db85

takes 1 second (958 ms) and involves compiling only 3 items, instead of the full 29, as it is demonstrated with the same project in the article about Conan.

That same article also states that building the project and all its dependencies from sources takes 13.5 seconds, while with vcpkg building the dependencies alone took 34 seconds, so it might look like vcpkg doesn’t improve but degrades build time. However, you shouldn’t forget that vcpkg builds both Debug and Release configurations, not just one (which, by the way, will catch up with the time when you’ll be building the project and dependencies in a different configuration), plus it does packing and cashing, which also takes time. And anyway, this matters only once - when dependencies are built for the first time, while without using pre-built dependencies you’ll be wasting time building dependencies over and over again.

Locking on dependencies versions

If you’d like to lock on specific versions of dependencies (you should), it can be done with overrides:

{
    "name": "glfw-imgui-example",
    "version": "0",
    "dependencies":
    [
        "glad",
        "glfw",
        {
            "name": "dearimgui",
            "features":
            [
                "backend-glfw"
            ]
        }
    ],
    "overrides":
    [
        {
            "name": "glad",
            "version": "0.1.36",
            "port-version": 0
        },
        {
            "name": "glfw",
            "version": "3.3.8",
            "port-version": 0
        },
        {
            "name": "dearimgui",
            "version": "1.88.0",
            "port-version": 0
        }
    ]
}

Here you might think that overrides block is redundant and you will probably want to set versions right in the dependencies block like this:

{
    "name": "glfw-imgui-example",
    "...": "...",
    "dependencies":
    [
        "...",
        {
            "name": "glfw",
            "version": "3.3.8"
        }
    ]
}

But that will fail:

error: while loading /path/to/glfw-imgui-example/vcpkg.json:
$.dependencies[2] (a dependency): unexpected field 'version', did you mean 'version>='?See https://github.com/Microsoft/vcpkg/tree/master/docs/users/manifests.md for more information.

So it just does not accept this, and indeed, setting it like "version>=": "3.3.8" will succeed, ~~but then it will accept newer versions too, when they become available in the registry~~ and it should not pick up newer versions, because vcpkg uses a minimum version approach. Unless, like I said before, if you set a dependency version with version>= and that value is lower than baselined version, then vcpkg will skip all the versions between that one and the baselined and will take the baselined version.

And so if you want to pin/lock on exact version of a dependency, then you have to use overrides.

In my opinion, this whole system is quite weird and not intuitive.

Transitive dependencies

…but weirdness doesn’t stop there. You cannot use overrides in ports manifests (vcpkg.json) of dependencies of dependencies (transitive dependencies). Or, rather, you can, but it will be ignored. As explained in this answer:

You can only use overrides at the root level and not in a port. Within a port you can only use "version>=": "...".

If you could use overrides in a port, you could create two ports with a dependency on A where port #1 required version 1.0.0 and port #2 required version 1.0.1. This would result in unsolvable version constraints.

So, if I’d want to lock on glfw version 3.3.8 in dearimgui port manifest, then I would need to do it like this:

{
    "name": "dearimgui",
    "...": "...",
    "dependencies":
    [
        "..."
    ],
    "features":
    {
        "backend-glfw":
        {
            "description": "Using GLFW as graphics backend",
            "dependencies":
            [
                {
                    "name": "glfw",
                    "version>=": "3.3.8"
                }
            ]
        }
    }
}

To make things more interesting, if you’d like to specify the port version too, this is how you would do it:

{
    "name": "glfw",
    "version>=": "3.3.8#1"
}

Yeah, no separate field/property - just this # separator followed by the port version.

Multiple registries

It is also possible to have multiple registries in a project and specify which packages should be fetched from which registry.

For example, let’s say I want my in-house custom registry to be the default one, and for the purpose of testing/comparing my custom ports I’d need to be able to get their variants from Microsoft’s registry:

{
    "default-registry":
    {
        "kind": "git",
        "repository": "git@github.com:retifrav/vcpkg-registry.git",
        "baseline": "5c1a089c3a542dfbe818625bf4a3dadfb834e2af"
    },
    "registries":
    [
        {
            "kind": "git",
            "repository": "https://github.com/microsoft/vcpkg.git",
            "baseline": "71f51b100be2b5d32e3907572d99dc2f97088c8d",
            "packages":
            [
                "glslang"
            ]
        }
    ]
}

This is what I had in vcpkg-configuration.json in one of my other projects where I had some problems with linking to HLSL that was installed from glslang. So I tried to use its port from Microsoft’s registry to see if it is built differently there (and indeed it was), and while glslang is listed in the dependencies list inside vcpkg.json, here I’m telling vcpkg that it should be actually fetched from the Microsoft’s registry instead of the default one. As you can see below, now vcpkg uses two registries, and even though I already have glslang pre-built in my local cache (and the same version too), now it will be built anew, for it is a port from a different registry:

...
Fetching registry information from git@github.com:retifrav/vcpkg-registry.git (HEAD)...
Fetching registry information from https://github.com/microsoft/vcpkg.git (HEAD)...
...
Installing 9/19 glslang:x64-windows-static...
Building glslang[core]:x64-windows-static...
-- Installing port from location: C:\Users\USERNAME\AppData\Local\vcpkg\registries\git-trees\4d7780995e9523d16a56714fcef0159f18ecfa52
-- Downloading https://github.com/KhronosGroup/glslang/archive/11.8.0.tar.gz -> KhronosGroup-glslang-11.8.0.tar.gz...
-- Extracting source D:/programs/vcpkg/downloads/KhronosGroup-glslang-11.8.0.tar.gz
-- Applying patch ignore-crt.patch
-- Applying patch always-install-resource-limits.patch
-- Using source at D:/programs/vcpkg/buildtrees/glslang/src/11.8.0-2a85409eb5.clean
-- Configuring x64-windows-static
-- Building x64-windows-static-dbg
-- Building x64-windows-static-rel
-- Installing: D:/programs/vcpkg/packages/glslang_x64-windows-static/share/glslang/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "C:\Users\USERNAME\AppData\Local\vcpkg\archives\8b\8b16920d54a7798ed9edb2c11b8cfe6a824cb4fab31365b9f3fe820486bbd669.zip"
Elapsed time to handle glslang:x64-windows-static: 46.16 s

Isn’t that beautiful. With Conan I couldn’t come up with a working setup for this, and with vcpkg it just works with a minimal effort.

CMake presets

In (relatively) recent versions of CMake a new feature has been added - CMake presets. It first appeared in version 3.19, and then versions 3.20 and 3.21 further expended it with newer revisions. I am now using presets revision 3 (that’s the one version 3.21 brought in), but later CMake versions added even more.

In short, it is a JSON file, in which you can define “presets” - sets of CMake options for building your project. For example, here are several presets for configuring and building that project, and now instead of that long error-prone CMake command I can just do this:

$ cd /path/to/glfw-imgui-example

$ cmake --preset vcpkg-default-triplet
Preset CMake variables:

  CMAKE_BUILD_TYPE:STRING="Release"
  CMAKE_INSTALL_PREFIX:PATH="/path/to/glfw-imgui-example/install/vcpkg-default-triplet"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="/path/to/programs/vcpkg/scripts/buildsystems/vcpkg.cmake"
  USING_PACKAGE_MANAGER_VCPKG:BOOL="TRUE"

-- Running vcpkg install
...
-- Configuring done
-- Generating done
-- Build files have been written to: /path/to/glfw-imgui-example/build/vcpkg-default-triplet

$ cmake --build --preset vcpkg-default-triplet
[3/4] Install the project...
-- Install configuration: "Release"
-- Installing: /path/to/glfw-imgui-example/install/vcpkg-default-triplet/bin/glfw-imgui/glfw-imgui
-- Installing: /path/to/glfw-imgui-example/install/vcpkg-default-triplet/bin/glfw-imgui/JetBrainsMono-ExtraLight.ttf

Much more convenient, innit. And that way your team members might even “not know” about vcpkg’s existence (after they install it on their machines, of course).

The Ninja CMake generator and CMAKE_BUILD_TYPE are set in the presets too, but one can of course override those by explicitly setting them with -G and -D when calling CMake.

Updating a dependency version

When you need to update (or downgrade) one of the project dependencies version (given that new version is available in the registry), you can do it in your project’s vcpkg.json. For example, I have two versions of GLFW in my registry - 3.3.7 and 3.3.8 - and the project depends on version 3.3.8 at the moment:

If I change GLFW dependency version to 3.3.7 in vcpkg.json:

{
    "name": "glfw-imgui-example",
    "...": "...",
    "overrides":
    [
        {
            "...": "...",
        },
        {
            "name": "glfw",
            "version": "3.3.7"
        },
        {
            "...": "..."
        }
    ]
}

and rebuild the project, it will then fetch the GLFW sources for version 3.3.7 and build it (as there is only 3.3.8 pre-built in the cache), and it will also re-build Dear ImGui, because it one depends on GLFW:

...
The following packages will be rebuilt:
    dearimgui[backend-glfw,core]:x64-osx -> 1.88.0 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/c3282e78368406b65f57ef0e3afa7cda2fc26501
    glfw[core]:x64-osx -> 3.3.7 -- /Users/USERNAME/.cache/vcpkg/registries/git-trees/45af0346d2ec926146091ab374c475cac5aaf055
Removing 1/4 dearimgui:x64-osx
Elapsed time to handle dearimgui:x64-osx: 6.359 ms
Removing 2/4 glfw:x64-osx
Elapsed time to handle glfw:x64-osx: 5.831 ms
Restored 0 package(s) from /Users/USERNAME/.cache/vcpkg/archives in 317 us. Use --debug to see more details.
Installing 3/4 glfw:x64-osx...
Building glfw[core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/45af0346d2ec926146091ab374c475cac5aaf055
-- Fetching git@github.com:glfw/glfw.git 45ce5ddd197d5c58f50fdd3296a5131c894e5527...
-- Extracting source /path/to/programs/vcpkg/downloads/glfw-45ce5ddd197d5c58f50fdd3296a5131c894e5527.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/glfw/src/1c894e5527-e2c903af70.clean
-- Found external ninja('1.11.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Installing: /path/to/programs/vcpkg/packages/glfw_x64-osx/share/glfw/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/5c/5ce54de2d8e83e86576a55541e988b827c72567a9bfee05717f103cc04193eff.zip"
Elapsed time to handle glfw:x64-osx: 7.613 s
Installing 4/4 dearimgui:x64-osx...
Building dearimgui[backend-glfw,core]:x64-osx...
-- Installing port from location: /Users/USERNAME/.cache/vcpkg/registries/git-trees/c3282e78368406b65f57ef0e3afa7cda2fc26501
-- Using cached /path/to/programs/vcpkg/downloads/dearimgui-9aae45eb4a05a5a1f96be1ef37eb503a12ceb889.tar.gz
-- Cleaning sources at /path/to/programs/vcpkg/buildtrees/dearimgui/src/3a12ceb889-c011915ee6.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /path/to/programs/vcpkg/downloads/dearimgui-9aae45eb4a05a5a1f96be1ef37eb503a12ceb889.tar.gz
-- Using source at /path/to/programs/vcpkg/buildtrees/dearimgui/src/3a12ceb889-c011915ee6.clean
-- Found external ninja('1.11.1').
-- Configuring x64-osx
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Installing: /path/to/programs/vcpkg/packages/dearimgui_x64-osx/share/dearimgui/copyright
-- ololo
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: "/Users/USERNAME/.cache/vcpkg/archives/00/00795b05b7197ecba35c066f67adec8cf123fd2d427deeae3921a9f8ec1486c7.zip"
Elapsed time to handle dearimgui:x64-osx: 10.59 s
...

After the new build the application will report GLFW version 3.3.7:

Static CRT/MSVC linkage

I’ve stumbled upon this problem on Windows with one of the dependencies in one of my other projects. At first that dependency wasn’t being handled with vcpkg, I just added path to its installed artifacts into CMAKE_PREFIX_PATH. The project configuration succeeded, but building the project exploded with errors at linking stage:

LINK: command "HERE-GOES-LONG-LINKING-COMMAND" failed (exit code 1120) with the following output:
SomeDependency.lib(API.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MT_StaticRelease' in some.cpp.obj
SomeDependency.lib(Bitmap.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MT_StaticRelease' in some.cpp.obj
...
msvcprt.lib(MSVCP140.dll) : error LNK2005: "public: __cdecl std::_Lockit::_Lockit(int)" (??0_Lockit@std@@QEAA@H@Z) already defined in libcpmt.lib(xlock.obj)
msvcprt.lib(MSVCP140.dll) : error LNK2005: "public: __cdecl std::_Lockit::~_Lockit(void)" (??1_Lockit@std@@QEAA@XZ) already defined in libcpmt.lib(xlock.obj)
msvcprt.lib(MSVCP140.dll) : error LNK2005: "public: struct _Cvtvec __cdecl std::_Locinfo::_Getcvt(void)const " (?_Getcvt@_Locinfo@std@@QEBA?AU_Cvtvec@@XZ) already defined in some.cpp.obj
msvcprt.lib(MSVCP140.dll) : error LNK2005: "public: unsigned short const * __cdecl std::_Locinfo::_W_Getdays(void)const " (?_W_Getdays@_Locinfo@std@@QEBAPEBGXZ) already defined in libcpmt.lib(wlocale.obj)
...
LINK : warning LNK4286: symbol 'tolower' defined in 'libucrt.lib(tolower_toupper.obj)' is imported by 'SomeDependency.lib(SourceWeb.cpp.obj)'
LINK : warning LNK4286: symbol 'calloc' defined in 'libucrt.lib(calloc.obj)' is imported by 'libtiff.lib(tif_win32.c.obj)'
...
SomeDependency.lib(geos.cpp.obj) : error LNK2001: unresolved external symbol __imp_hypot
...
libpng.lib(pngwrite.c.obj) : error LNK2001: unresolved external symbol __imp_strerror
libpng.lib(pngwrite.c.obj) : error LNK2019: unresolved external symbol __imp_ferror referenced in function png_image_write_to_file
...
some-application.exe : fatal error LNK1120: 33 unresolved externals
ninja: build stopped: subcommand failed.

The most important one from all of these:

mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MT_StaticRelease'

As it turned out, that dependency was linking CRT/MSVC dynamically, while x64-windows-static triplet does that statically, but in order for linking to succeed, all the targets/objects in the project must have the same type of linking to CRT/MSVC.

So I’ve re-built that dependency with static linking to CRT/MSVC by setting /MT flags, but still got errors on linking, however this time the linking was wrong the other way around:

mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MD_DynamicRelease'

Doesn’t make sense, does it, but then I realized that this is because my application is also linking to CRT/MSVC dynamically. This time I decided to google some more, as setting flags didn’t feel entirely right, and soon enough I found this solution on CMake’s forum:

# is set in vcpkg-windows-static preset (when triplet is x64-windows-static)
option(CRT_LINKAGE_STATIC "Link MSVC runtime statically" 0)

# ...

if(WIN32 AND CRT_LINKAGE_STATIC)
    #add_compile_options("/MT")
    if(POLICY CMP0091)
        cmake_policy(SET CMP0091 NEW)
    endif()
    set_property(TARGET ${CMAKE_PROJECT_NAME}
        PROPERTY MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>"
    )
endif()

Later on I also made a vcpkg port out of that dependency, so vcpkg would configure and build it under the same triplet together with other dependencies. In that case there is no need to set linking flags in that dependency’s CMakeLists.txt, as linkage to CRT/MSVC would be aligned, but of course I still need to align it for my application, which is what I’m doing by setting CRT_LINKAGE_STATIC option in vcpkg-windows-static CMake preset.

Dynamic and static linking

Since we are brought up the subject of linking, in case you are curious, with Windows build as an example, if we build that project with vcpkg-default-triplet preset, which uses dynamic linking (VCPKG_LIBRARY_LINKAGE), then resulting artifacts will be all its dependencies as DLLs and a small executable of the application itself:

glad.dll - 86 KB
glfw3.dll - 211.5 KB
dearimgui.dll - 889 KB
glfw-imgui.exe - 50.5 KB

If we build it with vcpkg-windows-static preset, which uses static linking (both for dependencies and CRT/MSVC), then we’ll get just an application executable with everything linked into it:

glfw-imgui.exe: 1268 KB

Finally, using the same vcpkg-windows-static preset but with VCPKG_CRT_LINKAGE set to dynamic (probably redundant for dependencies) and -DCRT_LINKAGE_STATIC=0, we’ll still get a single executable with dependencies linked in but without CRT/MSVC:

glfw-imgui.exe - 947 KB

which means that to run on another computer it will require user to have Visual C++ Redistributable installed.

What if a library is static only

Some libraries might not “support” dynamic linking, which in most cases means that you can’t build them as DLLs on Windows, as they don’t export any symbols. That’s okay, you’ll just have to force static linking on them by putting the following in the very beginning of their portfile:

if (VCPKG_LIBRARY_LINKAGE STREQUAL dynamic) # probably also check if it is Windows, as it might be fine on other platforms
    message(STATUS "[INFO] ${PORT} doesn't support building as dynamic library, overriding to static")
    set(VCPKG_LIBRARY_LINKAGE static)
endif()

It also won’t hurt to check in the library’s CMakeLists.txt that its authors don’t hardcode the library type in add_library() statements.

Mapping of Release binaries in RelWithDebInfo

If your project has less common configurations than just Debug/Release, for example if you also build RelWithDebInfo, then you might get a linking mismatch when your RelWithDebInfo configuration would try to link with Debug variants of dependencies.

In that case you’ll need to set the following explicit mapping in your project:

set(CMAKE_MAP_IMPORTED_CONFIG_MINSIZEREL Release)
set(CMAKE_MAP_IMPORTED_CONFIG_RELWITHDEBINFO Release)

Updating a port

Not once and not twice you’ll need to change something in your existing port, especially when you are working on the very first version of it (as you’ll be making mistakes and overlooking things).

As a concrete example, I forgot to add include_guard(GLOBAL) in my ports/decovar-vcpkg-cmake helper package. Here’s what I needed to do in order to fix that:

Edit ./ports/decovar-vcpkg-cmake/vcpkg-port-config.cmake and commit this change, but not push yet;
Run git rev-parse HEAD:./ports/decovar-vcpkg-cmake and set the new value to git-tree in ./versions/d-/decovar-vcpkg-cmake.json;
Stage that change and amend the commit from the step 1 by running git commit --amend --no-edit;
Push.

You can inspect that commit in my example registry. As you can see, it’s almost the same steps as with adding a new version of a port, except that here the version stays the same.

Also don’t forget to update Git hash value (current HEAD of the entire registry repository, not just this dependency) for baseline property in vcpkg-configuration.json of the “consuming” project, otherwise it will not “know” about the updated version in the registry. Moreover, if you force-pushed that fix, then vcpkg will simply fail to resolve dependencies, as that commit would no longer exist.

But actually that last bit about changing baseline property in the project’s vcpkg-configuration.json when updating the same version of a port in the registry (when the same port v1.2.3 has different configuration/patches/etc in different registry commits/states) is something I am not very confident about, because I am not really sure how exactly the baseline value is used for resolving the port version/state:

Does vcpkg look for a version with git-tree value that was available no later than that commit?
Or does vcpkg use that commit hash value only for getting versions/baseline.json from that exact commit (to establish a minimum port version), and then it will still find the latest version’s git-tree value that might be pointing to a later repository state and take that one? So one cannot rely on baseline property to “prevent” vcpkg from getting updated port versions available in the registry?

I was hoping it’s the former, but looks like it’s actually the latter. In that case I now better understand the point of port-version.

Testing your ports before publishing

While working on a port, would it be creating a new port or updating an existing one, you will want to test it before publishing/pushing it to the registry. The reason is that all too often you will be finding a yet another error/typo/imperfection some minutes after your last push, which you’ll want to amend in your last commit and force-push it to the registry, trying not to “overwhelm” the history with lots (and lots) of these “correction” commits.

It might be fine if you are using that registry alone, but if you are working in a team, then it should absolutely not be allowed to force-push, so testing the ports before pushing your changes to the registry becomes especially important. There are several ways how you can do that, and here are some.

Overlay ports

The overlay ports is probably the easiest way to test your changes locally. You just copy the port(s) folder with your pending changes to some other path:

$ cd /tmp && mkdir ./vcpkg-test-registry && cd $_
$ cp -r /path/to/your/vcpkg-registry/ports/glfw .
$ ls -L1 .
glfw/

and then point to that path with --overlay-ports for bare vcpkg:

$ cd /path/to/some/project
$ ls -L1 ./vcpkg*
./vcpkg-configuration.json
./vcpkg.json

$ vcpkg install --overlay-ports /tmp/vcpkg-test-registry

or with VCPKG_OVERLAY_PORTS for CMake:

$ cd /path/to/some/project
$ ls -L1 ./vcpkg* ./CMake*
./CMakeLists.txt
./vcpkg-configuration.json
./vcpkg.json
$ mkdir build && cd $_

$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    -DVCPKG_OVERLAY_PORTS="/tmp/vcpkg-test-registry" \
    ..

Either of those will “ignore” the glfw port from the registry and instead will use the one overlayed at /tmp/vcpkg-test-registry/:

The following packages will be built and installed:
    glad:arm64-osx@0.1.36 -- /path/to/vcpkg/.cache/vcpkg/registries/git-trees/e5e10ddd4da92e43fb8d0824bbdcdd900f4eca8d
    glfw:arm64-osx@3.4.0 -- /tmp/vcpkg-test-registry/glfw
    json-nlohmann:arm64-osx@3.11.3 -- /path/to/vcpkg/.cache/vcpkg/registries/git-trees/8bd1dccc545d928171f343d3b59822b1df0bebf1
    ...

Conveniently, there is no need to care about the port version, as no matter what your baseline or overrides say, it will go with whatever version is in that overlayed port:

$ gsed -i 's/"version": "3\.4\.0",/"version": "3.444.9000",/' /tmp/vcpkg-test-registry/glfw/vcpkg.json

$ vcpkg install --overlay-ports /tmp/vcpkg-test-registry
...
The following packages will be built and installed:
    glad:arm64-osx@0.1.36 -- /path/to/vcpkg/.cache/vcpkg/registries/git-trees/e5e10ddd4da92e43fb8d0824bbdcdd900f4eca8d
    glfw:arm64-osx@3.444.9000 -- /tmp/vcpkg-test-registry/glfw
    json-nlohmann:arm64-osx@3.11.3 -- /path/to/vcpkg/.cache/vcpkg/registries/git-trees/8bd1dccc545d928171f343d3b59822b1df0bebf1

Registry branches

When it comes to testing ports on several machines, such as building on your CI/CD buildbots or when sharing it with a colleague, overlay ports method is not that convenient, because you will need to deploy your port to every machine and modify the build commands.

A more convenient way would be to create a separate testing/playground registry, and add it as an additional registry in your project, but an even better way would be to just create a new branch in your registry - vcpkg supports that too, what a marvel.

The way it works is that after you commit and push your modified port to that new branch, you can tell vcpkg to fetch ports from that branch instead of the default one by specifying the reference property (in vcpkg-configuration.json):

{
    "default-registry":
    {
        "kind": "git",
        "repository": "git@github.com:retifrav/vcpkg-registry.git",
        "reference": "updating-glfw",
        "baseline": "4b4ae3fea063fc04c2d5a6089c8aa7c2d0129879"
    },
    "registries": []
}

In case of buildbots, if your CI/CD supports “personal” builds with patches (like TeamCity does), then running the build with your modified port becomes rather trivial (modify vcpkg-configuration.json, make a patch, trigger the building pipeline with that patch).

Obviously, since it is your branch, you can force-push to it all you want without causing troubles for others. And when you are happy with your port, you can simply rebase your branch onto master and delete it.

Per-platform specifics

WebAssembly

Chainloading Emscripten toolchain

Just like with other platforms, targetting WebAssembly (WASM) requires you to use an appropriate triplet. There is no suitable triplet among the “official” ones at the moment, but there is a community triplet wasm32-emscripten.

And while the dependencies (ports) will (hopefully) build just fine, here’s a surprise that will be waiting for you when CMake will try to find_package() them for your main project:

CMake Error at D:/programs/vcpkg/scripts/buildsystems/vcpkg.cmake:843 (_find_package):
  Could not find a configuration file for package "fmt" that is compatible
  with requested version "".

  The following configuration files were considered but not accepted:

    D:/temp/emsc/build/vcpkg-default-triplet/vcpkg_installed/wasm32-emscripten/share/fmt/fmt-config.cmake, version: 8.1.1 (32bit)

Call Stack (most recent call first):
  CMakeLists.txt:8 (find_package)


-- Configuring incomplete, errors occurred!

And this is not just about fmt, you’ll get this error for other dependencies too (although not all of them, details below). The actual fuck. Just what am I supposed to do with this?

After some googling I found this question at StackOverflow with some more details. So apparently this happens when a package contains a version config (fmt does contain one) that is generated by write_basic_package_version_file() without ARCH_INDEPENDENT being set (and I have never seen it being set in any package), because then the following condition is added to the config:

# check that the installed version has the same 32/64bit-ness as the one which is currently searching:
if(NOT CMAKE_SIZEOF_VOID_P STREQUAL "4")
  math(EXPR installedBits "4 * 8")
  set(PACKAGE_VERSION "${PACKAGE_VERSION} (${installedBits}bit)")
  set(PACKAGE_VERSION_UNSUITABLE TRUE)
endif()

I also checked what kind of binary I got built, and it was in fact a WebAssembly binary:

$ cd /tmp
$ mkdir fmt-lib-contents && cd $_
$ ar xv /path/to/your-project/build/vcpkg_installed/wasm32-emscripten/lib/libfmt.a
x - format.cc.o
x - os.cc.o
$ ls
format.cc.o  os.cc.o
$ file ./*
./format.cc.o: WebAssembly (wasm) binary module version 0x1 (MVP)
./os.cc.o:     WebAssembly (wasm) binary module version 0x1 (MVP)

And yet find_package() was failing, because CMAKE_SIZEOF_VOID_P equaled 8. So, whose fault is that CMAKE_SIZEOF_VOID_P is set to 8, while the package is built for 4?

By the way, even if write_basic_package_version_file() did have ARCH_INDEPENDENT set, here’s a related CMake bugreport, from which it seems that it wouldn’t work anyway (at least not until CMake 3.26 (or even newer) it wouldn’t). But this is actually not important, more details to follow.

So what can one do? After some more googling I’ve stumbled upon this workaround in Qt sources, where they just clear the CMAKE_SIZEOF_VOID_P variable before finding a package and then set it back. In my case it would be like this:

#message(STATUS "CMAKE_SIZEOF_VOID_P: ${CMAKE_SIZEOF_VOID_P}")
set(BACKUP_CMAKE_SIZEOF_VOID_P "${CMAKE_SIZEOF_VOID_P}")
set(CMAKE_SIZEOF_VOID_P "")
find_package(fmt CONFIG REQUIRED)
set(CMAKE_SIZEOF_VOID_P "${BACKUP_CMAKE_SIZEOF_VOID_P}")

Then the package is found, and configuration succeeds. But I have 30+ dependencies in my project, and they are spread across several different subprojects, so I most definitely don’t want to wrap every single find_package() like this. And anyway, this doesn’t seem like a correct way to deal with the situation.

And then a couple of days later I finally tried to set the VCPKG_CHAINLOAD_TOOLCHAIN_FILE in CLI when running CMake:

$ cd /path/to/emsdk
$ source ./emsdk_env.sh
$ cd /path/to/some/project
$ mkdir build && cd $_
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="${EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" \
    -DVCPKG_TARGET_TRIPLET="wasm32-emscripten" \
    ..
$ cmake --build . --target install

I mean, I’ve been always setting it with -DDCMAKE_TOOLCHAIN_FILE="..." when I was building my projects with Emscripten for WASM, but now with vcpkg the wasm32-emscripten triplet already does chainload it, and so I thought that I then should not. But as it turns out, counter-intuitively enough, both of these chainloads are required, because (the way I understand it) the one chainloaded inside the triplet is used for building vcpkg ports, and the one set with -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="..." in CLI is used for building the main project.

And what would you know, having done that, I didn’t get the errors about non-compatible 32bit packages, as this time CMAKE_SIZEOF_VOID_P equaled 4.

It certainly wouldn’t hurt to have this documented somewhere.

CMAKE_PREFIX_PATH isn’t used

That’s not exactly related to vcpkg, but since we are on the topic of Emscripten toolchain: when I tried to use vcpkg-resolved dependencies and my library in another project, like so:

$ cd /path/to/some/other/project
$ mkdir build && cd $_
$ cmake -G "Ninja" -DCMAKE_PREFIX_PATH="/path/to/my/library/install" \
    -DCMAKE_TOOLCHAIN_FILE="$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" \
    ..

I suddenly got errors about not finding packages configs, even though CMAKE_PREFIX_PATH is clearly set and is correct.

And it turned out that Emscripten toolchain sets certain variables which “disable” using CMAKE_PREFIX_PATH for packages discovery. There are also some more details (and confusion) in this thread.

Here’s that particular fragment from Emscripten.cmake:

# Since Emscripten is a cross-compiler, we should never look at the
# system-provided directories like /usr/include and so on. Therefore only
# CMAKE_FIND_ROOT_PATH should be used as a find directory. See
# http://www.cmake.org/cmake/help/v3.0/variable/CMAKE_FIND_ROOT_PATH_MODE_INCLUDE.html
if (NOT CMAKE_FIND_ROOT_PATH_MODE_LIBRARY)
  set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
endif()
if (NOT CMAKE_FIND_ROOT_PATH_MODE_INCLUDE)
  set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
endif()
if (NOT CMAKE_FIND_ROOT_PATH_MODE_PACKAGE)
  set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
endif()

So it’s CMAKE_FIND_ROOT_PATH_MODE_PACKAGE being set to ONLY what’s causing the problem. And so then there are two “solutions” for that.

Either provide CMAKE_FIND_ROOT_PATH instead of CMAKE_PREFIX_PATH:

$ cmake -G "Ninja" -DCMAKE_FIND_ROOT_PATH="/path/to/my/library/install" \
    -DCMAKE_TOOLCHAIN_FILE="$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" \
    ..

or keep CMAKE_PREFIX_PATH but set CMAKE_FIND_ROOT_PATH_MODE_PACKAGE to BOTH:

$ cmake -G "Ninja" -DCMAKE_PREFIX_PATH="/path/to/my/library/install" \
    -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE="BOTH" \
    -DCMAKE_TOOLCHAIN_FILE="$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" \
    ..

I do not know which one is “correct”. Both of the “solutions” worked (equally?) fine for me, as I haven’t noticed any problems so far.

Enabling pthreads support

Emscripten can compile to WASM with pthreads (POSIX threads) support enabled. For that one needs to set -pthread compiler flag when building one’s project.

I did set that flag in my main project, started the build, got the ports built, got the project built too, but then I got the following errors on linking:

wasm-ld: error: --shared-memory is disallowed by png.c.o because it was not compiled with 'atomics' or 'bulk-memory' features.
...
wasm-ld: error: --shared-memory is disallowed by 4D_api.cpp.o because it was not compiled with 'atomics' or 'bulk-memory' features.
...
wasm-ld: error: --shared-memory is disallowed by Threads.c.o because it was not compiled with 'atomics' or 'bulk-memory' features.

From these I realized that adding -pthread flag in the main project is not enough, as it also needs to be set for ports compilation too. To test this theory I’ve patched those ports with errors (as you can see, the first one is from PNG and the second one is from PROJ) to set the flags:

# string(APPEND ... " ...") would do fine too
set(CMAKE_C_FLAGS           "${CMAKE_C_FLAGS} -pthread")
#set(CMAKE_C_FLAGS_DEBUG     "${CMAKE_C_FLAGS_DEBUG} -pthread")
#set(CMAKE_C_FLAGS_RELEASE   "${CMAKE_C_FLAGS_RELEASE} -pthread")
set(CMAKE_CXX_FLAGS         "${CMAKE_CXX_FLAGS} -pthread")
#set(CMAKE_CXX_FLAGS_DEBUG   "${CMAKE_CXX_FLAGS_DEBUG} -pthread")
#set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -pthread")

And indeed, after that the first two errors went away, but the third one was still remaining, and I had no idea which port is it from and is it from a port at all.

But okay, at least this confirmed that I was on the right track, just needed to figure out how to set a compiler flag for all the ports at once, as one certainly should not patch the flags in every single port.

So how does one set a “global” compilation flag to be applied to building all the ports? My first naive idea was to just set these CMAKE_*_FLAGS in the triplet, which I naively did, but as it turned out, they were never passed further, because ports didn’t “receive” them (I checked that by printing out all the flags with message()).

Additional googling led me here, and from there I eventually got to an understanding.

Related vcpkg documentation says that “when not using VCPKG_CHAINLOAD_TOOLCHAIN_FILE” compiler flags can be set with VCPKG_*_FLAGS variables in the triplet:

set(VCPKG_C_FLAGS   "-pthread")
set(VCPKG_CXX_FLAGS "-pthread")

and then the toolchain (one of the /path/to/vcpkg/scripts/toolchains/*.cmake ones) will use these variables to set the CMAKE_*_FLAGS_INIT variables, which in turn will be used to set the CMAKE_*_FLAGS variables, which are the flags we are after.

But we are in fact using VCPKG_CHAINLOAD_TOOLCHAIN_FILE, so setting VCPKG_*_FLAGS variables in the triplet won’t work (certainly didn’t work for me), so what does one do then?

It is my understanding (which might be wrong) that in this case one would need to set CMAKE_*_FLAGS_INIT variables in the chainloaded toolchain itself. But we are chainloading the Emscripten toolchain from ${EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake, and it won’t be correct to modify the original file, especially that it is still expected to be used for non-pthreads-enabled WASM builds, so I guess we’ll need to copy and modify the original. But then in turn we will need to use a custom triplet too, because the default one chainloads the original Emscripten toolchain.

So, here comes a custom triplet my-wasm32-emscripten-pthreads.cmake:

# that is required (only on Windows?), otherwise it won't get environment variables
set(VCPKG_ENV_PASSTHROUGH_UNTRACKED EMSCRIPTEN_TOOLCHAIN_FILE_PATH EMSDK PATH)

if(NOT DEFINED ENV{EMSDK})
    message(FATAL_ERROR "[ERROR] The EMSDK environment variable isn't set, you probably haven't sourced emsdk_env.sh")
endif()

set(EMSCRIPTEN_TOOLCHAIN_FILE_PATH "$ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake")
# yet to find a better way to pass this value to the triplet, but so far an environment variable seems to be the way
if(NOT DEFINED ENV{EMSCRIPTEN_TOOLCHAIN_FILE_PATH})
    message(STATUS "Didn't get EMSCRIPTEN_TOOLCHAIN_FILE_PATH from environment, keeping default value: ${EMSCRIPTEN_TOOLCHAIN_FILE_PATH}")
else()
    set(EMSCRIPTEN_TOOLCHAIN_FILE_PATH "$ENV{EMSCRIPTEN_TOOLCHAIN_FILE_PATH}")
    message(STATUS "Got EMSCRIPTEN_TOOLCHAIN_FILE_PATH from environment: ${EMSCRIPTEN_TOOLCHAIN_FILE_PATH}")
endif()
if(NOT EXISTS ${EMSCRIPTEN_TOOLCHAIN_FILE_PATH})
    message(FATAL_ERROR "[ERROR] Could not find Emscripten.cmake toolchain file, expected it to be at ${EMSCRIPTEN_TOOLCHAIN_FILE_PATH}")
endif()

set(VCPKG_TARGET_ARCHITECTURE wasm32)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)
set(VCPKG_CMAKE_SYSTEM_NAME Emscripten)
# required, otherwise "Unable to determine toolchain use for my-wasm32-emscripten-pthreads with CMAKE_SYSTEM_NAME Emscripten"
# and yes, for building the main project you'll yet again need to provide `-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="..."` in CLI
set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${EMSCRIPTEN_TOOLCHAIN_FILE_PATH})

And the custom Emscripten toolchain (the one passed in EMSCRIPTEN_TOOLCHAIN_FILE_PATH environment variable) is a copy of the original one from $EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake, but with the following addition in the end:

# an exact copy of Emscripten.cmake file contents
# ...

# pthreads (https://emscripten.org/docs/porting/pthreads.html)

set(EMSCRIPTEN_PTHREADS_FLAGS "-pthread")
# it is likely that not all of these need to be set, CMAKE_{C,CXX}_FLAGS_INIT seems to be enough
string(APPEND CMAKE_C_FLAGS_INIT                     " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_C_FLAGS_DEBUG_INIT               " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_C_FLAGS_RELEASE_INIT             " ${EMSCRIPTEN_PTHREADS_FLAGS}")
string(APPEND CMAKE_CXX_FLAGS_INIT                   " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_CXX_FLAGS_DEBUG_INIT             " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_CXX_FLAGS_RELEASE_INIT           " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_STATIC_LINKER_FLAGS_INIT         " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_STATIC_LINKER_FLAGS_DEBUG_INIT   " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_STATIC_LINKER_FLAGS_RELEASE_INIT " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_SHARED_LINKER_FLAGS_INIT         " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_SHARED_LINKER_FLAGS_DEBUG_INIT   " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_SHARED_LINKER_FLAGS_RELEASE_INIT " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_MODULE_LINKER_FLAGS_INIT         " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_MODULE_LINKER_FLAGS_DEBUG_INIT   " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_MODULE_LINKER_FLAGS_RELEASE_INIT " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_EXE_LINKER_FLAGS_INIT            " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_EXE_LINKER_FLAGS_DEBUG_INIT      " ${EMSCRIPTEN_PTHREADS_FLAGS}")
#string(APPEND CMAKE_EXE_LINKER_FLAGS_RELEASE_INIT    " ${EMSCRIPTEN_PTHREADS_FLAGS}")

But actually instead of copying the entire toolchain file contents wouldn’t it be smarter to just include it, right, so here’s a better variant:

include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake)

# pthreads (https://emscripten.org/docs/porting/pthreads.html)

set(EMSCRIPTEN_PTHREADS_FLAGS "-pthread")
string(APPEND CMAKE_C_FLAGS_INIT   " ${EMSCRIPTEN_PTHREADS_FLAGS}")
string(APPEND CMAKE_CXX_FLAGS_INIT " ${EMSCRIPTEN_PTHREADS_FLAGS}")

Having saved it somewhere at /path/to/custom/toolchains/Emscripten-pthreads.cmake, we can now configure and build the project like this:

$ cd /path/to/emsdk
$ source ./emsdk_env.sh
$ cd /path/to/some/project
$ mkdir build && cd $_
$ EMSCRIPTEN=${EMSDK}/upstream/emscripten EMSCRIPTEN_TOOLCHAIN_FILE_PATH=/path/to/custom/toolchains/Emscripten-pthreads.cmake \
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DVCPKG_OVERLAY_TRIPLETS="/path/to/custom/triplets" \
    -DVCPKG_TARGET_TRIPLET="my-wasm32-emscripten-pthreads" \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake" \
    ..
$ cmake --build . --target install

Some clarification about all that:

why I am setting environment variables inline instead of exporting them - because it’s better like this for experimentation (so I don’t forget them being set in the environment from the previous runs);
- EMSCRIPTEN_TOOLCHAIN_FILE_PATH is set because I haven’t found another way to pass the path to my custom Emscripten toolchain into the triplet;
- EMSCRIPTEN is set because for whatever reason everything fails, complaining that this variable is missing, even though I certainly did not set it when I used the default toolchain/triplet;
-DVCPKG_OVERLAY_TRIPLETS="..." is how you tell vcpkg where to look for additional/custom triplets;
-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE="..." still points to the default Emscripten toolchain because I am already setting -pthread compiler flags (plus some linker flags) inside my main project’s CMakeLists.txt, so I don’t need a custom toolchain that does that. Although I probably should use the same custom toolchain (the one that is chainloaded inside my-wasm32-emscripten-pthreads triplet for building ports) here too;
- either way, one of those toolchains has to be chainloaded here, as it is required for building the project itself with Emscripten.

Now everything finally builds and links into WASM with pthreads enabled.

Side-effect of chainloading a custom toolchain

There is one (but probably more) side-effect of chainloading a custom toolchain and that is detecting the compiler hash: it will remain the same even if you switch to a different Emscripten version (as long as your custom toolchain doesn’t change), because apparently it’s exactly the chainloaded toolchain that is taking part in the hash calculation.

Compiler hash remaining the same means that vcpkg-resolved dependencies won’t get rebuilt, so you’ll end up linking your project that was built using one Emscripten version with dependencies that were built using a different Emscripten version.

Obviously, if you are chainloading the standard/default Emscripten toolchain from its original location in $EMSDK/upstream/emscripten/cmake/Modules/Platform/, then it’s all good, vcpkg will notice the change and the compiler hash will be different, which will trigger rebuilding the dependencies; but here we are talking about your custom toolchain, which most likely never changes (as there is no need for that).

To make vcpkg re-build the dependencies when you switch to a different Emscripten version, you’ll need to change something in your custom toolchain file too, so vcpkg would notice that it has changed. For example, you can just add some comment:

include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake)

# pthreads (https://emscripten.org/docs/porting/pthreads.html)

set(EMSCRIPTEN_PTHREADS_FLAGS "-pthread")
string(APPEND CMAKE_C_FLAGS_INIT   " ${EMSCRIPTEN_PTHREADS_FLAGS}")
string(APPEND CMAKE_CXX_FLAGS_INIT " ${EMSCRIPTEN_PTHREADS_FLAGS}")

# this is a comment for marking this custom toolchain as modified
# change the Emscripten version value every time you switch to a different one
# Emscripten version 3.1.54

This probably concerns other platforms/compilers too, and I reckon the workaround would be the same for those as well (adding/modifying a comment every time).

Windows

MSVC toolset

In one case I needed to build my project with MSVC 141 toolset, which I did by setting -G "Visual Studio 15 2017" -A "x64", but apparently that only applied to the project itself, and looks like vcpkg built the dependencies with a different toolset (likely with MSVC 143, as that’s the latest I have installed), because I got the following errors on linking:

error LNK2001: unresolved external symbol __CxxFrameHandler4
error LNK2001: unresolved external symbol __GSHandlerCheck_EH4

As this StackOverflow thread says, the __CxxFrameHandler4 is a part of the newer FH4 exception handling, which apparently appeared in MSVC 142 toolset and newer, and so the resolution is to tell vcpkg to build dependencies using the same toolset as the one used to build the project.

That can be done on the triplet level, so yet again I’ve created a custom triplet my-x64-windows-v141.cmake:

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

# here you specify the MSVC toolset
set(VCPKG_PLATFORM_TOOLSET "v141")
# not sure what this one is for
set(VCPKG_DEP_INFO_OVERRIDE_VARS "v141")

And now I build the whole thing like this:

$ cd /path/to/some/project
$ mkdir build && cd $_
$ cmake -G "Visual Studio 15 2017" -A "x64" \
    -DVCPKG_OVERLAY_TRIPLETS="/path/to/custom/triplets" \
    -DVCPKG_TARGET_TRIPLET="my-x64-windows-v141" \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    ..
$ cmake --build . --target install --config Release

As usual, this better to be run from the corresponding MSVC command prompt, so you’ll need to adjust the shell-specific things.

Release binaries size is bigger with vcpkg

Suddenly I noticed that my Windows build artifacts got bigger in size. They were smaller before I started resolving dependencies with vcpkg, but now they got bigger.

I suspected that there might be some optimization flags not set, and I was almost right, although the other way around - turns out, by default vcpkg sets Debug flag /Z7 for Release configuration! See for yourself in the Windows toolchain (/path/to/vcpkg/scripts/toolchains/windows.cmake):

set(CMAKE_CXX_FLAGS_RELEASE
    "${VCPKG_CRT_LINK_FLAG_PREFIX} /O2 /Oi /Gy /DNDEBUG /Z7 ${VCPKG_CXX_FLAGS_RELEASE}"
    CACHE STRING ""
)
set(CMAKE_C_FLAGS_RELEASE
    "${VCPKG_CRT_LINK_FLAG_PREFIX} /O2 /Oi /Gy /DNDEBUG /Z7 ${VCPKG_C_FLAGS_RELEASE}"
    CACHE STRING ""
)

From the documentation for /Z7 flag:

The /Z7 option produces object files that also contain full symbolic debugging information for use with the debugger. These object files and any libraries built from them can be substantially larger than files that have no debugging information.

I mean, sounds useful for Debug configuration, but why is it set for Release configuration too?

Googling this question brought me to this issue in vcpkg repository, so at least I wasn’t the only one to question the sanity of this. And here’s what one of the vcpkg maintainers said in the discussion thread over there:

There's no reason to remove debugging information; we believe CMake's default setting which includes not even stripped symbols in release mode is an incorrect default.

Well, I certainly believe that CMake does this absolutely correctly, as there is no bloody point to have anything of the sort in the Release binaries. If you want to debug, you take Debug binaries; that is why Debug/Release separation exists in the first place, is it not? Why on earth would you want to be able to debug with Release binaries? Sweet suffering Jehovah.

Removing /Z7 flag from Release configuration

So, as we don’t have unlimited storage for our packages and distributives and in general we don’t like wasting network bandwidth transferring useless data, we’d like to remove /Z7 flag from the default Windows toolchain to reduce our dependencies binaries size back to normal.

The procedure is very similar to how I added -pthread flag to enable pthreads support in Enscripten/WASM builds. There are several options:

Make patches for every single port sources to override flags in their CMakeLists.txt files. That is the worst option out of all listed here;
Modify the flags directly in the default toolchain (/path/to/vcpkg/scripts/toolchains/windows.cmake). I would not recommend this, because:
- that modification will get overridden on the next vcpkg update/reinstall;
- you’ll need to do that on your every machine or/and every buildbot in your CI/CD;
- other people with default toolchains will be getting different build artifacts even though they’d be using the same triplets as you, and that might backfire at some point;
Copy the default toolchain into your project repository, modify the flags there and chainload it with VCPKG_CHAINLOAD_TOOLCHAIN_FILE. This option is almost good, but dragging around a copy of the default toolchain isn’t ideal, at the very least because it will eventually go out of sync with the default one;
Instead of copying the contents of the default toolchain, simply include it in your custom chainloaded toolchain and override the flags right after that.

I actually had one more option, which is what I tried first: still create a custom toolchain to be chainloaded but without including the default toolchain - just seting new values to the flags and that’s it. But right away I discovered two problems with this:

you need to somehow get the current values of the flags that have been already set upper in the chain. You can of course just set the flags to whatever you want, but ideally one would like to preserve all the original flags and just remove the /Z7. And I didn’t find a way to do it;
the real showstopper, however, is the fact that if you don’t include the default toolchain, then all the flags it sets won’t be set at all (for example, /utf-8 flag will be missing, so you might get too many characters in constant compilation error). I, for one, was expecting those flags to be set, since my custom toolchain is chainloaded and not loaded instead of the default one, but apparently I’m missing something about the way chainloading works.

So the 4th option is what I went with (include the default toolchain in a custom toolchain and override flags right after). Yes, it seems to be redundant and rather weird to include the exact same toolchain that has been (hasn’t it?) just used upper in the chain, but that is what worked for me.

Here’s my custom toolchain (/path/to/custom/toolchains/windows.cmake):

# for $ENV{VCPKG_ROOT} to work the triplet should contain `set(VCPKG_ENV_PASSTHROUGH_UNTRACKED VCPKG_ROOT)`
include("$ENV{VCPKG_ROOT}/scripts/toolchains/windows.cmake")

# remove /Z7 from the default set of C flags
string(REPLACE "/Z7" ""
    CMAKE_C_FLAGS_RELEASE_WITHOUT_Z7
    "${CMAKE_C_FLAGS_RELEASE}"
)
# override Release flags variable with the new value
set(CMAKE_C_FLAGS_RELEASE
    "${CMAKE_C_FLAGS_RELEASE_WITHOUT_Z7}"
    CACHE STRING ""
    # it's important to apply FORCE here, as this variable is CACHE
    # and it has been already set in the default toolchain above
    FORCE
)

# and the same for CXX flags
string(REPLACE "/Z7" ""
    CMAKE_CXX_FLAGS_RELEASE_WITHOUT_Z7
    "${CMAKE_CXX_FLAGS_RELEASE}"
)
set(CMAKE_CXX_FLAGS_RELEASE
    "${CMAKE_CXX_FLAGS_RELEASE_WITHOUT_Z7}"
    CACHE STRING ""
    FORCE
)

And here’s my custom triplet (/path/to/custom/triplets/my-x64-windows-static-md.cmake) that chainloads it:

# default values from the original x64-windows-static-md.cmake
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

# without this the VCPKG_ROOT environment variable
# won't be available in the chainloaded toolchain
set(VCPKG_ENV_PASSTHROUGH_UNTRACKED VCPKG_ROOT)
#
# overriding default flags to remove /Z7 from Release builds,
# because no one (except Microsoft) needs Debug stuff in Release binaries
#
# this could do with a better path instead of ..
set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${CMAKE_CURRENT_LIST_DIR}/../toolchains/windows.cmake)
# on Windows VCPKG_CHAINLOAD_TOOLCHAIN_FILE deactivates VS variables,
# so those need to be loaded again
set(VCPKG_LOAD_VCVARS_ENV 1)

Two things are important to note here:

VCPKG_ROOT environment variable needs to be passed through, otherwise chainloaded toolchain won’t have it;
unless you’ve read about this in documentation, you wouldn’t know that in case of chainloaded toolchains the VCPKG_LOAD_VCVARS_ENV is set to OFF, which leads to CMake errors like No CMAKE_C_COMPILER could be found and other typical problems of not finding compilers and tools, so you need to set it back to ON/1.

Now I can configure and build my project like this:

$ cd /path/to/some/project
$ mkdir build && cd $_
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \
    -DVCPKG_OVERLAY_TRIPLETS="/path/to/custom/triplets"
    -DVCPKG_TARGET_TRIPLET="my-x64-windows-static-md" \
    ..
$ cmake --build . --target install

For that to work the custom toolchain needs to be in /path/to/custom/toolchains/, as for triplet the ${CMAKE_CURRENT_LIST_DIR} will evaluate to /path/to/custom/triplets/, and so ../toolchains/ with be the right path.

Comparison of binary sizes with and without /Z7

Let’s see some real sizes of Release binaries that are built with and without /Z7 flag. If you’d like to repeat the experiment yourself, here’s a dummy project that I used for it.

First build the libraries with default Release flags (so with /Z7 among them):

$ cd /path/to/z7-binary-size-comparison
$ vcpkg install --triplet x64-windows-static-md
$ du -hs ./vcpkg_installed/x64-windows-static-md/lib

Then build the same libraries but with a custom triplet to remove /Z7 from Release flags:

$ mv ./vcpkg_installed ./vcpkg_installed_z7
$ vcpkg install --overlay-triplets ./triplets --triplet my-x64-windows-static-md
$ du -hs ./vcpkg_installed/my-x64-windows-static-md/lib

In my case the results were:

	Debug	Release
	Debug	With `/Z7`	Without `/Z7`
assimp-vc143-mt.lib	214.9 MB	195.3 MB	63.8 MB
dearimgui.lib	5.8 MB	7.4 MB	3.4 MB
draco.lib	109.5 MB	94.4 MB	27 MB
glfw3.lib	1.9 MB	2.1 MB	633.1 KB
sqlite3.lib	3.2 MB	6.5 MB	2.7 MB
zlib.lib	364.1 KB	441 KB	186.5 KB
zstd.lib	3.8 MB	6.6 MB	1.8 MB
		312.74 MB	99.52 MB

So not only Release binaries with /Z7 flag are more than 3 times bigger than their variants without it, but also for most libraries they are even bigger than the Debug variants.

I mean, goddamn. The only valid reason for setting this flag by default in Release configurations would be if people behind vcpkg were selling data storage or some other hosting services of the kind. Wait… oh shi~

Building with Clang instead of MSVC

Later I needed to build the project and its dependencies with Clang instead of MSVC.

Clang (LLVM) can be downloaded either separately from its website, or you can install it via Visual Studio Installer as a Visual Studio component.

In case of configuring a CMake project, if you installed LLVM yourself, choosing Clang as compiler can be done with Ninja generator:

$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_C_COMPILER:PATH="d:/path/to/llvm/bin/clang-cl.exe" \
    -DCMAKE_CXX_COMPILER:PATH="d:/path/to/llvm/bin/clang-cl.exe" \
    ..
$ cmake --build .

or with Visual Studio generator:

$ cmake -G "Visual Studio 17 2022" ..
$ cmake --build . --config Release -- -p:CLToolExe=clang-cl.exe -p:CLToolPath="d:/path/to/llvm/bin"

The variant with -p:CLToolExe might be useful if you have MSVC-specific compiler flags hardcoded in your project.

If you installed LLVM via Visual Studio Installer, then it can be just this:

$ cmake -G "Visual Studio 17 2022" -T "ClangCL" ..
$ cmake --build . --config Release

If it doesn’t find the compilers/tools, you can try to set the VS environment (from cmd, that is):

> call d:\path\to\visual-studio\VC\Auxiliary\Build\vcvars64.bat

Actually, you might want to do this every time you’d like to use Visual Studio tools.

If you have both Clang that you installed yourself and Clang which is a part of Visual Studio, you might want to make sure that both your project and vcpkg-resolved dependencies are built with the same one.

Anyway, that was the project itself, but how to build vcpkg-resolved dependencies with Clang as well? As in previous cases, you would need to make a custom toolchain for that and chainload it from a custom triplet.

The triplet is simple enough:

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

set(VCPKG_PLATFORM_TOOLSET ClangCL)

# otherwise these environment variables won't be available in the chainloaded toolchain
set(VCPKG_ENV_PASSTHROUGH_UNTRACKED LLVMInstallDir LLVMToolsVersion)
# finding and setting C/CXX compiler to Clang
# this could do with a better path instead of ..
set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${CMAKE_CURRENT_LIST_DIR}/../toolchains/windows-clang.cmake)
# on Windows VCPKG_CHAINLOAD_TOOLCHAIN_FILE deactivates automatic VS variables, so those need to be loaded again
set(VCPKG_LOAD_VCVARS_ENV 1)

Its main purpose is to chainload a custom toolchain, which will set Clang as compiler(s). You can take any other default triplet as a base and just add this chainloading there.

The toolchain, on the other hand, is far from being simple. While the essential thing that it does is “just” setting CMAKE_C_COMPILER and CMAKE_CXX_COMPILER to the discovered Clang executable, there are several other things that need to be taken care of, such as setting C/C++ standards and compiler/linker flags, some of which are necessary for successful compilation of certain sources.

I would have never figured out the full set on my own, but luckily I’ve stumbled upon this repository, which seems to be all about using Clang with vcpkg in various combinations. The degree of customization and overriding stuff there seems to be rather overcomplicated, or at least it is so for my low ICQ. Так что я ещё ничего, а вот у парняги реально беды с башкой, при всём уважении, и спасибо ему что он есть, мощный тип.

I took this toolchain of his as a base, removed a lot of stuff from it, which I ~~thought was redundant~~ didn’t understand the purpose of, and combined it with default Microsoft’s Windows toolchain. The resulting toolchain is published in my registry.

The most important part of this toolchain is setting the compiler(s):

find_program(CLANG_CL_EXECUTBALE
    NAMES
        "clang-cl"
        "clang-cl.exe"
    PATHS
        ENV
        LLVMInstallDir
    PATH_SUFFIXES
        "bin"
    # it is found exactly in default paths, as I am not passing LLVMInstallDir,
    # but if you will, then uncomment this
    #NO_DEFAULT_PATH
)

set(CMAKE_C_COMPILER "${CLANG_CL_EXECUTBALE}" CACHE STRING "")
set(CMAKE_CXX_COMPILER "${CLANG_CL_EXECUTBALE}" CACHE STRING "")

But like I said, setting flags is also important, so I’d recommend you to take a look at the full toolchain.

The whole thing to configure and build the project now becomes this:

> call d:\path\to\visual-studio\VC\Auxiliary\Build\vcvars64.bat
> cd d:\path\to\some\project
> mkdir build
> cd build
> cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ^
    -DCMAKE_C_COMPILER:PATH="d:/path/to/llvm/bin/clang-cl.exe" ^
    -DCMAKE_CXX_COMPILER:PATH="d:/path/to/llvm/bin/clang-cl.exe" ^
    -DCMAKE_TOOLCHAIN_FILE:PATH="%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake" ^
    -DVCPKG_OVERLAY_TRIPLETS:PATH="d:/path/to/custom/triplets" ^
    -DVCPKG_TARGET_TRIPLET="decovar-x64-windows-static-md-clang" ^
    ..
> cmake --build .

You’ll probably also need to set LLVMInstallDir environment variable, just to be sure that vcpkg gets to use the same Clang version for building dependencies as the one you’ve set for the project. But then for this to work you’d need to uncomment NO_DEFAULT_PATH of Clang executable discovery in the toolchain.

Or, if you installed Clang via Visual Studio Installer, then just use the -T variant instead:

> call d:\path\to\visual-studio\VC\Auxiliary\Build\vcvars64.bat
> cd d:\path\to\some\project
> mkdir build
> cd build
> cmake -G "Visual Studio 17 2022" -T "ClangCL" ^
    -DCMAKE_TOOLCHAIN_FILE:PATH="%VCPKG_ROOT%/scripts/buildsystems/vcpkg.cmake" ^
    -DVCPKG_OVERLAY_TRIPLETS:PATH="d:/path/to/custom/triplets" ^
    -DVCPKG_TARGET_TRIPLET="decovar-x64-windows-static-md-clang" ^
    ..
> cmake --build . --config Release

Then there will be no need to set LLVMInstallDir (and no need to uncomment NO_DEFAULT_PATH).

LZMA fails without AVX2 intrinsics

Not strictly related to vcpkg, but still related. As soon as I’ve set Clang for building vcpkg-resolved dependencies, one of them - LZMA - failed to build.

I’ve fetched its sources and tried to build it separately with MSVC, and that still succeeded. But building with Clang was failing:

LzFind.c(641,3): error : use of undeclared identifier '__m256i'
LzFind.c(646,5): error : expected expression
LzFind.c(630,33): message : expanded from macro 'SASUB_256'

It turned out that building this project with Clang requires setting /arch:AVX2 C flag. Why this is not required when building with MSVC - that I don’t know, and I don’t see that flag being set in the default Windows vcpkg toolchain either. Also surprisingly, building LZMA on Mac OS with Clang doesn’t have this problem either.

I wasn’t sure if it is a good idea to set this flag in the toolchain, so I decided to set it in the LZMA’s CMakeLists.txt and make it a port feature - here’s the commit with this change.

Disable copying DLLs to build folder

Accidentally, I noticed that some of the dependencies DLLs are copied to the build folder by “something”, and it definitely wasn’t me, as I never needed DLLs in my build folders.

Having inspected the build.ninja file (in the build folder of a Ninja-configured project), I found this line:

POST_BUILD = C:\WINDOWS\system32\cmd.exe /C "cd /D C:\path\to\some\project\build\windows-msvc143\something && C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noprofile -executionpolicy Bypass -file D:/programs/vcpkg/scripts/buildsystems/msbuild/applocal.ps1 -targetBinary c:/path/to/some/project/build/windows-msvc143/something/something.exe -installedDir c:/path/to/some/project/build/windows-msvc143/vcpkg_installed/decovar-x64-windows-static-md/bin -OutVariable out"

…that is for Release configuration, and for Debug it uses .../debug/bin.

Scrolling through applocal.ps1 script that is called there, I stumbled upon this line:

$a = $(dumpbin /DEPENDENTS $targetBinaryPath| ? { $_ -match "^    [^ ].*\.dll" } | % { $_ -replace "^    ","" })

The DUMPBIN tool is something one would use to discover runtime dependencies (DLLs) of an executable, so it really is vcpkg who was copying DLLs into build folders. While this might be nice and convenient for some, I didn’t ask it to do that. If I needed to deploy DLLs, that would be with a proper procedure on project installation, not as a blunt post-build step.

Anyway, this behaviour is controlled with VCPKG_APPLOCAL_DEPS option (which doesn’t seem to be documented anywhere), and so to disable it, one simply needs to pass -DVCPKG_APPLOCAL_DEPS=0 on project configuration (so you might want to add it to your CMake presets).

GNU/Linux

Missing executable attribute

Trying to build iconv port on GNU/Linux I got the following rather unexpected error:

/bin/bash: ./../src/133172cac4-7cae2f41cd.clean/configure: Permission denied

And indeed, the configure script doesn’t have x attribute/mode, so it is not an executable:

$ ls -l /path/to/vcpkg/buildtrees/iconv/src/133172cac4-7cae2f41cd.clean | grep config
-rw-r--r-- 1 build build  50810 Feb  2 14:09 config.h.in
-rw-r--r-- 1 build build 891440 Feb  2 14:09 configure
-rw-r--r-- 1 build build   6087 Feb  2 14:09 configure.ac

I don’t know how is that this attribute is missing, and why I didn’t get this problem on Mac OS, but to resolve this I had to to call chmod via vcpkg_execute_build_process() in the iconv portfile:

if(VCPKG_HOST_IS_LINUX) # OR VCPKG_HOST_IS_OSX
    # on GNU/Linux (and Mac OS?) the configure script might(?) be not an executable
    vcpkg_execute_build_process(
        COMMAND chmod +x ./configure
        WORKING_DIRECTORY ${SOURCE_PATH}
        LOGNAME config-${PORT}-${TARGET_TRIPLET}-chmod
    )
endif()

vcpkg_configure_make(
    SOURCE_PATH "${SOURCE_PATH}"
    # ...
)

iOS

Combined/fat/universal binaries for devices and simulator

One of our build configurations produces so-called combined/fat/universal binaries for iOS - arm64 for actual devices and x86_64 for simulator - and so vcpkg-managed dependencies also need to be built as such.

To tell CMake to build a universal device/simulator binary one needs to set the following:

set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64")

set(CMAKE_XCODE_ATTRIBUTE_ARCHS[sdk=iphoneos*] "arm64")
set(CMAKE_XCODE_ATTRIBUTE_VALID_ARCHS[sdk=iphoneos*] "arm64")

set(CMAKE_XCODE_ATTRIBUTE_ARCHS[sdk=iphonesimulator*] "x86_64")
set(CMAKE_XCODE_ATTRIBUTE_VALID_ARCHS[sdk=iphonesimulator*] "x86_64")

And as usual (as you saw in previous sections), to apply these to every vcpkg-managed build you’ll need to chainload a custom toolchain:

# for $ENV{VCPKG_ROOT} to work the triplet should contain `set(VCPKG_ENV_PASSTHROUGH_UNTRACKED VCPKG_ROOT)`
include("$ENV{VCPKG_ROOT}/scripts/toolchains/ios.cmake")

# to make a Mach-O combined/fat/universal binary, one needs to set both architectures
set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64")
# and set Xcode attributes to specify which architecture is for actual devices
set(CMAKE_XCODE_ATTRIBUTE_ARCHS[sdk=iphoneos*] "arm64")
set(CMAKE_XCODE_ATTRIBUTE_VALID_ARCHS[sdk=iphoneos*] "arm64")
# and which architecture is for simulator
set(CMAKE_XCODE_ATTRIBUTE_ARCHS[sdk=iphonesimulator*] "x86_64")
set(CMAKE_XCODE_ATTRIBUTE_VALID_ARCHS[sdk=iphonesimulator*] "x86_64")
# and probably also set these flags
#set(CMAKE_IOS_INSTALL_COMBINED 1)
#set(CMAKE_XCODE_ATTRIBUTE_ONLY_ACTIVE_ARCH 0)

from a custom triplet:

set(VCPKG_TARGET_ARCHITECTURE arm64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

set(VCPKG_CMAKE_SYSTEM_NAME iOS)

# without this the VCPKG_ROOT environment variable won't be available in the chainloaded toolchain
set(VCPKG_ENV_PASSTHROUGH_UNTRACKED VCPKG_ROOT)
# setting Xcode attributes for specifying device/simulator architectures
set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${CMAKE_CURRENT_LIST_DIR}/../toolchains/ios-fat.cmake)

And actually you could probably get away without using a chainloaded toolchain, because you can move CMAKE_OSX_ARCHITECTURES from toolchain to triplet’s VCPKG_OSX_ARCHITECTURES variable. But I suspect that CMAKE_XCODE_ATTRIBUTE_* variables still need to be set, and in that case toolchain isn’t going away.

Anyway, both variants seem to work equally fine, and to show it in action, if I build my Thingy port with default arm64-ios triplet:

$ vcpkg install --triplet arm64-ios

then here’s the binary I will get:

$ du -h ./vcpkg_installed/arm64-ios/lib/libThingy.a
8.0K    ./vcpkg_installed/arm64-ios/lib/libThingy.a

$ otool -hv ./vcpkg_installed/arm64-ios/lib/libThingy.a
Archive : ./vcpkg_installed/arm64-ios/lib/libThingy.a
./vcpkg_installed/arm64-ios/lib/libThingy.a(thingy.cpp.o):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64    ARM64        ALL  0x00      OBJECT     5        776 SUBSECTIONS_VIA_SYMBOLS

$ lipo -info ./vcpkg_installed/arm64-ios/lib/libThingy.a
Non-fat file: ./vcpkg_installed/arm64-ios/lib/libThingy.a is architecture: arm64

And if I build it with my custom decovar-arm64-ios-fat triplet:

$ vcpkg install --triplet decovar-arm64-ios-fat --overlay-triplets /path/to/vcpkg-registry/triplets

then this is the binary I will get:

$ du -h ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a
 12K    ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a

$ otool -hv ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a
Archive : ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a (architecture x86_64)
./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a(thingy.cpp.o) (architecture x86_64):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64   X86_64        ALL  0x00      OBJECT     4        840 SUBSECTIONS_VIA_SYMBOLS
Archive : ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a (architecture arm64)
./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a(thingy.cpp.o) (architecture arm64):
Mach header
      magic  cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64    ARM64        ALL  0x00      OBJECT     5        776 SUBSECTIONS_VIA_SYMBOLS

$ lipo -info ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a
Architectures in the fat file: ./vcpkg_installed/decovar-arm64-ios-fat/lib/libThingy.a are: x86_64 arm64

For vcpkg this build probably “looks” like arm64, because VCPKG_TARGET_ARCHITECTURE is set to arm64 in the triplet, but what exactly that entitles I don’t know, and according to lipo the binaries are indeed combined/fat/universal ones.

Merging different architectures into one universal binary

I also saw this blog post, where the author says that vcpkg doesn’t support building libraries as universal binaries, and then he suggests a workaround of building a library using different triplets and “merging” the results into one universal binary. It might be that in 2022-11-09 there was something different about the process, but as you can see vcpkg can in fact handle the scenario of producing fat binaries.

If anything, the essence of his post is that you can use the very same lipo utility to create a universal binary out of different single-architecture binaries, for example:

$ lipo -info ./vcpkg_installed/arm64-ios/lib/libzstd.a
Non-fat file: ./vcpkg_installed/arm64-ios/lib/libzstd.a is architecture: arm64

$ lipo -info ./vcpkg_installed/x64-ios/lib/libzstd.a
Non-fat file: ./vcpkg_installed/x64-ios/lib/libzstd.a is architecture: x86_64

$ lipo -create \
    ./vcpkg_installed/arm64-ios/lib/libzstd.a \
    ./vcpkg_installed/x64-ios/lib/libzstd.a \
    -output ./libzstd.a

$ lipo -info ./libzstd.a
Architectures in the fat file: ./libzstd.a are: x86_64 arm64

Another thing to note is that author is describing building a universal binary for Mac OS (Intel/ARM), and I am here dealing with iOS (device/simulator), so just to make sure that there is no difference I tried to do the same for Mac OS as well.

jpeg-turbo doesn’t support building with more than one architecture

All our dependencies were building fine as fat binaries, until jpeg-turbo build had failed. Turns out, it doesn’t support building as universal binary, because it contains assembly code, and “different architectures require different SIMD extensions”.

And that is a problem, because how does one go about it? All the dependencies need to be built “unattended”, so they are made ready/available before the main project configuration starts. So it means that jpeg-turbo would require some special treatment, for example building it twice (for each architecture), merging it into one final universal binary using lipo (like it is described above), and then some more fucking about to automate the process and make it a part of the normal pre-configuration routine.

Fortunately, before diving into all that, I just tried to simply disable that check, and the build went fine, because apparently this target platform does not use assembly code:

CMake Warning at simd/CMakeLists.txt:5 (message):
  SIMD extensions not available for this CPU (aarch64).  Performance will
  suffer.

To make sure, just in case, I’ve built it for single architecture:

$ vcpkg install --triplet arm64-ios

$ du -h ./vcpkg_installed/arm64-ios/lib/libturbojpeg.a
960K    ./vcpkg_installed/arm64-ios/lib/libturbojpeg.a

$ lipo -info ./vcpkg_installed/arm64-ios/lib/libturbojpeg.a
Non-fat file: ./vcpkg_installed/arm64-ios/lib/libturbojpeg.a is architecture: arm64

and then for combined architectures:

$ vcpkg install --triplet decovar-arm64-ios-fat --overlay-triplets /path/to/vcpkg-registry/triplets

$ du -h ./vcpkg_installed/decovar-arm64-ios-fat/lib/libturbojpeg.a
1.9M    ./vcpkg_installed/decovar-arm64-ios-fat/lib/libturbojpeg.a

$ lipo -info ./vcpkg_installed/decovar-arm64-ios-fat/lib/libturbojpeg.a
Architectures in the fat file: ./vcpkg_installed/decovar-arm64-ios-fat/lib/libturbojpeg.a are: x86_64 arm64

As you can see, it has successfully built into a combined/fat/universal binary. It may be that it will horribly backfire at some point later at application runtime, but it hasn’t so far.

Mac OS

Universal binaries

As I mentioned in the section about combined/fat/universal binaries for iOS, I wanted to try building universal binaries of vcpkg-resolved dependencies for Mac OS too (for Intel and Apple silicon (ARM) based Macs).

Applying the same “trick” with chainloaded toolchain didn’t work:

set(VCPKG_TARGET_ARCHITECTURE arm64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

set(VCPKG_CMAKE_SYSTEM_NAME Darwin)

set(VCPKG_ENV_PASSTHROUGH_UNTRACKED VCPKG_ROOT)
set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${CMAKE_CURRENT_LIST_DIR}/../toolchains/osx-fat.cmake)

include("$ENV{VCPKG_ROOT}/scripts/toolchains/osx.cmake")

# to make an Intel/ARM universal binary, one needs to set both architectures
set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64")

as vcpkg still produced non-fat binaries containing only one architecture. Can’t say why is that.

What did work is setting VCPKG_OSX_ARCHITECTURES in the triplet:

set(VCPKG_TARGET_ARCHITECTURE arm64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

set(VCPKG_CMAKE_SYSTEM_NAME Darwin)
set(VCPKG_OSX_ARCHITECTURES "arm64;x86_64")

#set(VCPKG_ENV_PASSTHROUGH_UNTRACKED VCPKG_ROOT)
# unlike fat iOS toolchain, this one has no effect (but `VCPKG_OSX_ARCHITECTURES` does)
#set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE ${CMAKE_CURRENT_LIST_DIR}/../toolchains/osx-fat.cmake)

and then vcpkg produced universal binaries, which I’ve successfully tested on both M2 and Intel machines that I have.

So I thought, okay, probably chainloaded toolchain isn’t needed at all, so I can just set VCPKG_OSX_ARCHITECTURES for the fat iOS build too without chainloading. And yes, that worked, but just in case I kept the toolchain too, because it sets CMAKE_XCODE_ATTRIBUTE_* variables, and I have a feeling that these variables do mater.

PNG fails to generate tf1

It was all good until the PNG build had failed:

clang: error: no such file or directory: 'x86_64'
CMake Error at scripts/genout.cmake:78 (message):
  Failed to generate
  /path/to/vcpkg/buildtrees/png/decovar-arm64-osx-fat-dbg/scripts/symbols.out.tf1

It originates here:

execute_process(COMMAND "${CMAKE_C_COMPILER}" "-E"
                        ${CMAKE_C_FLAGS}
                        ${PLATFORM_C_FLAGS}
                        "-I${SRCDIR}"
                        "-I${BINDIR}"
                        ${INCLUDES}
                        "-DPNGLIB_LIBNAME=PNG${PNGLIB_MAJOR}${PNGLIB_MINOR}_0"
                        "-DPNGLIB_VERSION=${PNGLIB_VERSION}"
                        "-DSYMBOL_PREFIX=${SYMBOL_PREFIX}"
                        "-DPNG_NO_USE_READ_MACROS"
                        "-DPNG_BUILDING_SYMBOL_TABLE"
                        ${PNG_PREFIX_DEF}
                        "${INPUT}"
                OUTPUT_FILE "${OUTPUT}.tf1"
                WORKING_DIRECTORY "${BINDIR}"
                RESULT_VARIABLE CPP_FAIL)
if(CPP_FAIL)
  message(FATAL_ERROR "Failed to generate ${OUTPUT}.tf1")
endif()

So apparently it can’t handle arm64;x86_64 value.

At the same time, surprisingly, that very same PNG project didn’t fail when I was building fat binaries for iOS. I can’t think of other reason than different CMAKE_OSX_SYSROOT values: for iOS it was /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.0.sdk, and the build succeeded, but for Mac OS it is /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk, and the build fails. If anything, both iOS and Mac OS builds got the same "arm64;x86_64" value for CMAKE_OSX_ARCHITECTURES.

No idea how to fix it yet, so for now I’m just building PNG separately with a single-architecture triplet of interest (and merging the results into a universal binary, when needed).

Dependency graph

Conan has a convenient out-of-the-box functionality for generating a dependency graph, both as a plain-text JSON file and as an interactive HTML page.

The vcpkg’s capabilities in that regard are rather modest in comparison. It has a depend-info command, which can list dependencies either in a somewhat structured plain-text format or in two graph formats: DOT and DGML. Neither of these formats can be customized, and all of them output just port names (no version, license or any other information).

Furthermore, when it comes to graphs, it is still your responsibility to visualize them (with Graphviz or some other tool), as vcpkg only produces the text representation. For example, let’s use the DOT format to build a dependency graph for that project from before:

$ vcpkg depend-info glfw-imgui-example \
    --overlay-ports=/path/to/project \
    --overlay-ports=/path/to/vcpkg-registry/ports \
    --dot | nop > ./graph.dot

Here the first --overlay-ports path points to the project folder (that contains the project’s vcpkg.json manifest) and the second --overlay-ports path points to a local clone of my registry. Note that I still needed to provide the glfw-imgui-example project name. The piped nop utility is a part of Graphviz tools and what it does is it formats the graph text into a nicer readable representation.

If I wanted to build a graph not for a project but for a port from registry, then I would’ve invoked the command with just one --overlay-ports path, like this:

$ vcpkg depend-info some-port \
    --overlay-ports=/path/to/vcpkg-registry/ports \
    --dot | nop > ./graph.dot

But I don’t yet have ports with a lot of dependencies in my registry, so let’s continue with the project graph.

The contents of the generated (and formatted) graph.dot are this:

digraph G {
    graph [overlap=false,
        rankdir=LR
    ];
    edge [minlen=3];
    dearimgui -> decovar_vcpkg_cmake;
    dearimgui -> glfw;
    dearimgui -> vcpkg_cmake;
    dearimgui -> vcpkg_cmake_config;
    glfw -> vcpkg_cmake;
    glfw -> vcpkg_cmake_config;
    glad -> vcpkg_cmake;
    glad -> vcpkg_cmake_config;
    glfw_imgui_example -> dearimgui;
    glfw_imgui_example -> glfw;
    glfw_imgui_example -> glad;
    empty    [label="3 singletons..."];
}

That can be visualized with Graphviz like so:

$ dot -T svg ./graph.dot \
    -Nfontcolor=blue -Nshape=rect -Grankdir=TB -Gsplines=ortho \
    -o ./graph.svg

The result will look like this:

Graphviz, unprocessed dot-graph of vcpkg dependencies

So it works, but the visualization could benefit from some improvements, such as removing redundant elements and adding some different colors/accents. To automate the process, modifications in the graph.dot can be done using regular expressions, and for that purpose it would probably be better to re-generate the graph.dot without formatting (without piping to nop):

$ vcpkg depend-info glfw-imgui-example \
    --overlay-ports=/path/to/project \
    --overlay-ports=/path/to/vcpkg-registry/ports \
    --dot > ./graph.dot

First thing to get rid of is the “3 singletons…” label (the fuck is that, I didn’t ask for it). It can be removed with sed (or gsed, in case of Mac OS):

$ sed -i 's/empty \[label="[[:digit:]]\+ singletons\.\{3\}"\];//' ./graph.dot

but actually, sed and its crazy regular expressions flavor can go to hell, so do it with Perl instead:

$ perl -pi -e 's/empty \[label="\d+ singletons\.{3}"\];//' ./graph.dot

Next thing you might want to do is to remove helper ports, because they are not actual C++ dependencies, and also every single port depends on them, so they only add noise to the graph, which becomes a bigger problem on bigger graphs. Either remove them one by one:

$ perl -pi -e 's/[\w]+ -> vcpkg_cmake;//g' ./graph.dot
$ perl -pi -e 's/[\w]+ -> vcpkg_cmake_config;//g' ./graph.dot
$ perl -pi -e 's/[\w]+ -> decovar_vcpkg_cmake;//g' ./graph.dot

or all at once, if they have a common part in their names:

$ perl -pi -e 's/[\w]+ -> (decovar_)?vcpkg_cmake(_config)?;//g' ./graph.dot

Finally, let’s color connections to direct dependencies with blue color, and connections to transitive dependencies will be dashed:

$ perl -pi -e 's/(glfw_imgui_example -> .+?(?=;))/$1 [color=blue]/g' ./graph.dot
$ perl -pi -e 's/((?!glfw_imgui_example\b)\b\w+ -> .+?(?=;))/$1 [style=dashed]/g' ./graph.dot

Now the visualized graph will look like this:

Graphviz, processed dot-graph of vcpkg dependencies

Much cleaner, isn’t it. Although, still not as good as what Conan provides (via a single command without additional massaging), but more or less an okay result.

One last thing to note is that on the graph above the glfw port is both a direct and a transitive dependency (through dearimgui), so perhaps the point of having differently styled connections to transitive dependencies isn’t very clear. But here’s a graph for a bigger project:

Graphviz, dot-graph with more dependencies

And here we have (just one but nevertheless) a “purely” transitive dependency - brotli (through cpp_http), which is easier to spot thanks to a dashed black connection and no solid blue connections comming to it.

Distributing your project

If your project is meant to be distributed, meaning that you have libraries in it, which your customers/users will be linking to in their projects, then you need to deliver your dependencies together with your project installation.

With this project as an example, after its dependencies are resolved with vcpkg and the project itself is built and installed, the folders structure would look like this (in Release configuration):

$ tree --dirsfirst .
├── build
│   └── vcpkg-default-triplet
│       ├── ...
│       ├── vcpkg_installed
│       │   ├── arm64-osx
│       │   │   ├── debug
│       │   │   │   └── lib
│       │   │   ├── include
│       │   │   │   ├── Thingy
│       │   │   │   └── nlohmann
│       │   │   ├── lib
│       │   │   │   └── libThingy.a
│       │   │   └── share
│       │   │       ├── Thingy
│       │   │       ├── json-nlohmann
│       │   │       ├── nlohmann_json
│       │   │       ├── vcpkg-cmake
│       │   │       └── vcpkg-cmake-config
│       │   └── vcpkg
│       │       ├── info
│       │       │   ├── json-nlohmann_3.11.2_arm64-osx.list
│       │       │   ├── thingy_0.9.1_arm64-osx.list
│       │       │   ├── vcpkg-cmake-config_2022-02-06_arm64-osx.list
│       │       │   └── vcpkg-cmake_2023-05-04_arm64-osx.list
│       │       ├── updates
│       │       │   └── ...
│       │       └── vcpkg-lock.json
│       ├── CMakeCache.txt
│       ├── build.ninja
│       ├── cmake_install.cmake
│       ├── install_manifest.txt
│       └── vcpkg-manifest-install.log
├── install
│   └── vcpkg-default-triplet
│       ├── bin
│       │   └── some-tool
│       ├── include
│       │   └── SomeLibrary
│       │       └── some.h
│       ├── lib
│       │   └── libSomeLibrary.a
│       └── share
│           └── SomeLibrary
│               ├── SomeLibraryConfig.cmake
│               ├── SomeLibraryConfigVersion.cmake
│               ├── SomeLibraryTargets-release.cmake
│               └── SomeLibraryTargets.cmake
├── src
│   ├── some
│   │   ├── include
│   │   │   └── SomeLibrary
│   │   │       └── some.h
│   │   ├── CMakeLists.txt
│   │   ├── Config.cmake.in
│   │   └── some.cpp
│   ├── tool
│   │   ├── CMakeLists.txt
│   │   └── main.cpp
│   └── CMakeLists.txt
├── CMakeLists.txt
├── CMakePresets.json
├── vcpkg-configuration.json
└── vcpkg.json

If you are building your dependencies as part of your project (or/and if you are vendoring them), then most likely you can just pack the ./install folder and distribute that package, and it will be enough for consuming projects (as they would only need to set the path to your project package in CMAKE_PREFIX_PATH). But here such a package would be missing dependencies that were installed by vcpkg into vcpkg_installed/target-triplet, in particular the Thingy library will be missing, and so your customers/users will get the following error trying to configure their projects:

CMake Error at /path/to/cmake/version/share/cmake/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
  Could not find a package configuration file provided by "Thingy" with any
  of the following names:

    ThingyConfig.cmake
    thingy-config.cmake

  Add the installation prefix of "Thingy" to CMAKE_PREFIX_PATH or set
  "Thingy_DIR" to a directory containing one of the above files.  If "Thingy"
  provides a separate development package or SDK, be sure it has been
  installed.

So you need to merge vcpkg_installed/target-triplet into your project installation folder. There is no out-of-the-box function for this (I didn’t fine one), but I guess this is only because the task is actually very trivial: you just need to copy the entire folder. For example, with cp:

Contents of the installation prefix before

$ tree ./install/
└── vcpkg-default-triplet
    ├── bin
    │   └── some-tool
    ├── include
    │   └── SomeLibrary
    │       └── some.h
    ├── lib
    │   └── libSomeLibrary.a
    └── share
        └── SomeLibrary
            ├── SomeLibraryConfig.cmake
            ├── SomeLibraryConfigVersion.cmake
            ├── SomeLibraryTargets-release.cmake
            └── SomeLibraryTargets.cmake

$ cp -an \
    ./build/vcpkg-default-triplet/vcpkg_installed/arm64-osx/* \
    ./install/vcpkg-default-triplet/

Contents of the installation prefix after

$ tree ./install/
└── vcpkg-default-triplet
    ├── bin
    │   └── some-tool
    ├── debug
    │   └── lib
    │       └── libThingyd.a
    ├── include
    │   ├── SomeLibrary
    │   │   └── some.h
    │   ├── Thingy
    │   │   └── thingy.h
    │   └── nlohmann
    │       ├── adl_serializer.hpp
    │       ├── byte_container_with_subtype.hpp
    │       ├── detail
    │       │   ├── abi_macros.hpp
    │       │   ├── conversions
    │       │   │   ├── from_json.hpp
    │       │   │   ├── to_chars.hpp
    │       │   │   └── to_json.hpp
    │       │   ├── exceptions.hpp
    │       │   ├── hash.hpp
    │       │   ├── input
    │       │   │   ├── binary_reader.hpp
    │       │   │   ├── input_adapters.hpp
    │       │   │   ├── json_sax.hpp
    │       │   │   ├── lexer.hpp
    │       │   │   ├── parser.hpp
    │       │   │   └── position_t.hpp
    │       │   ├── iterators
    │       │   │   ├── internal_iterator.hpp
    │       │   │   ├── iter_impl.hpp
    │       │   │   ├── iteration_proxy.hpp
    │       │   │   ├── iterator_traits.hpp
    │       │   │   ├── json_reverse_iterator.hpp
    │       │   │   └── primitive_iterator.hpp
    │       │   ├── json_pointer.hpp
    │       │   ├── json_ref.hpp
    │       │   ├── macro_scope.hpp
    │       │   ├── macro_unscope.hpp
    │       │   ├── meta
    │       │   │   ├── call_std
    │       │   │   │   ├── begin.hpp
    │       │   │   │   └── end.hpp
    │       │   │   ├── cpp_future.hpp
    │       │   │   ├── detected.hpp
    │       │   │   ├── identity_tag.hpp
    │       │   │   ├── is_sax.hpp
    │       │   │   ├── std_fs.hpp
    │       │   │   ├── type_traits.hpp
    │       │   │   └── void_t.hpp
    │       │   ├── output
    │       │   │   ├── binary_writer.hpp
    │       │   │   ├── output_adapters.hpp
    │       │   │   └── serializer.hpp
    │       │   ├── string_concat.hpp
    │       │   ├── string_escape.hpp
    │       │   └── value_t.hpp
    │       ├── json.hpp
    │       ├── json_fwd.hpp
    │       ├── ordered_map.hpp
    │       └── thirdparty
    │           └── hedley
    │               ├── hedley.hpp
    │               └── hedley_undef.hpp
    ├── lib
    │   ├── libSomeLibrary.a
    │   └── libThingy.a
    └── share
        ├── SomeLibrary
        │   ├── SomeLibraryConfig.cmake
        │   ├── SomeLibraryConfigVersion.cmake
        │   ├── SomeLibraryTargets-release.cmake
        │   └── SomeLibraryTargets.cmake
        ├── Thingy
        │   ├── ThingyConfig.cmake
        │   ├── ThingyConfigVersion.cmake
        │   ├── ThingyTargets-debug.cmake
        │   ├── ThingyTargets-release.cmake
        │   ├── ThingyTargets.cmake
        │   ├── copyright
        │   ├── vcpkg.spdx.json
        │   └── vcpkg_abi_info.txt
        ├── json-nlohmann
        │   ├── copyright
        │   ├── vcpkg.spdx.json
        │   └── vcpkg_abi_info.txt
        ├── nlohmann_json
        │   ├── nlohmann_jsonConfig.cmake
        │   ├── nlohmann_jsonConfigVersion.cmake
        │   └── nlohmann_jsonTargets.cmake
        ├── vcpkg-cmake
        │   ├── copyright
        │   ├── vcpkg-port-config.cmake
        │   ├── vcpkg.spdx.json
        │   ├── vcpkg_abi_info.txt
        │   ├── vcpkg_cmake_build.cmake
        │   ├── vcpkg_cmake_configure.cmake
        │   └── vcpkg_cmake_install.cmake
        └── vcpkg-cmake-config
            ├── copyright
            ├── vcpkg-port-config.cmake
            ├── vcpkg.spdx.json
            ├── vcpkg_abi_info.txt
            └── vcpkg_cmake_config_fixup.cmake

or with tar, if cp doesn’t do the job for some reasons:

$ (cd ./build/vcpkg-default-triplet/vcpkg_installed/arm64-osx && tar -c .) \
    | (cd ./install/vcpkg-default-triplet && tar -xf -)

And then you will have both the project libraries and their dependencies inside the same prefix, so you’ll be able to just pack it and ship to your customers/users.

But, as usual, there is a but. As you can see in the merged installation prefix listing, it contains vcpkg-cmake and vcpkg-cmake-config helper ports, which are of no use for consuming projects; and also it contains json-nlohmann dependency, which is used for building the project’s executable some-tool, but isn’t used for building the project’s library SomeLibrary. So including those into the final package is redundant, as end user does not need them to be available for his project to link to yours.

It might be not worth to bother about this, but that’s only untill you encounter some ridiculously big dependencies (more than 3 GB of binaries in Datakit package on Windows) or just retarded ones (more than 12 000 of header files in GDAL package). So you most certainly would like to exclude some dependencies from your package, if they are only used for building executables and not libraries.

Filtering out artifacts of vcpkg-resolved dependencies could have been a challenging task, as one would need to establish which files came from which port, but fortunately vcpkg keeps track of installed artifacts in ./build/vcpkg-default-triplet/vcpkg_installed/vcpkg/info/*.list files, making the procedure rather simple. At the same time, certain artifacts you probably would like to keep (such as DLLs), so you’ll need to handle such exceptions somehow.

To automate the process of merging installations, filtering blacklisted dependencies and handling exceptions, I first wrote a Bash script, but it was deleting files one by one, and in case of GDAL’s more-than-12-000-headers deleting all of them was taking about 10 minutes, which is too goddamn slow. Trying to optimize the operation has overcomplicated the script beyond maintainability, so I rewrote it in Python. Here’s how to use it:

Contents of the installation prefix before

$ tree ./install/
└── vcpkg-default-triplet
    ├── bin
    │   └── some-tool
    ├── include
    │   └── SomeLibrary
    │       └── some.h
    ├── lib
    │   └── libSomeLibrary.a
    └── share
        └── SomeLibrary
            ├── SomeLibraryConfig.cmake
            ├── SomeLibraryConfigVersion.cmake
            ├── SomeLibraryTargets-release.cmake
            └── SomeLibraryTargets.cmake

$ python /path/to/install-vcpkg-artifacts.py \
    --cmake-preset vcpkg-default-triplet \
    --vcpkg-triplet arm64-osx \
    --blacklist "vcpkg-cmake,json-nlohmann"
[INFO] - [vcpkg-cmake]* (lists found: 2)
[INFO] - [json-nlohmann]* (lists found: 1)
[INFO] -
[INFO] Copying exceptions...
[INFO] -
[INFO] Filtering out blacklisted dependencies...
[INFO] - [vcpkg-cmake]*
[INFO] deleting artifacts...
[INFO] - [json-nlohmann]*
[INFO] deleting artifacts...
[INFO] -
[INFO] Merging vcpkg installed artifacts into project installation...
[INFO] -
[INFO] Done

Contents of the installation prefix after

$ tree ./install/
└── vcpkg-default-triplet
    ├── bin
    │   └── some-tool
    ├── debug
    │   └── lib
    │       └── libThingyd.a
    ├── include
    │   ├── SomeLibrary
    │   │   └── some.h
    │   └── Thingy
    │       └── thingy.h
    ├── lib
    │   ├── libSomeLibrary.a
    │   └── libThingy.a
    └── share
        ├── SomeLibrary
        │   ├── SomeLibraryConfig.cmake
        │   ├── SomeLibraryConfigVersion.cmake
        │   ├── SomeLibraryTargets-release.cmake
        │   └── SomeLibraryTargets.cmake
        └── Thingy
            ├── ThingyConfig.cmake
            ├── ThingyConfigVersion.cmake
            ├── ThingyTargets-debug.cmake
            ├── ThingyTargets-release.cmake
            ├── ThingyTargets.cmake
            ├── copyright
            ├── vcpkg.spdx.json
            └── vcpkg_abi_info.txt

As you can see, only the Thingy artifacts were merged into the project installation.

The script expects certain project folders structure, such as having a CMake preset subfolder inside ./build/, and so if you are not using CMake presets, then you’ll need to do certain adjustments either in your project structure or in the script source code.

Setting exceptions via CLI isn’t supported (yet), but you can do that directly in the script.

So, how is vcpkg comparing with Conan

Simplicity

Like I said, essentially, vspkg is just a combination of CMake and Git (plus some glue provided by the vcpkg CLI tool), which are very common basic instruments everyone is (should be) already familiar with.

They are fewer components or moving parts than it is with Conan. Especially when it comes to registries: there is no need to involve a complex 3rd-party service such as JFrog Artifactory (even though you can host it in-house), because vcpkg registries are just good old plain Git repositories, which are very simple to create and maintain.

Speed

The “speed” might not be the right term here, but for me creating vcpkg ~~packages~~ ports and integrating vcpkg into a project was easier to understand and simpler to implement than doing the same with Conan.

For instance, the research project that took about two weeks to implement with Conan, with vcpkg it barely took 3 days.

Documentation

Paradoxically enough, vcpkg’s documentation seems worse than Conan’s, or at least less detailed (and sometimes just missing), and yet it was easier to get started with vcpkg than it was with Conan. Things just make sense and many questions/problems you can figure out on your own.

Speaking about documentation, there are 3 different places/domains where you can find it. Certainly, all of them likely have the same source, but still such distribution does not help to inspire confidence:

https://github.com/microsoft/vcpkg/ - probably use this one, for it’s the source;
https://vcpkg.io/;
https://vcpkg.readthedocs.io/.

Overall impression

It seems to me that pretty much everything that vcpkg can do, Conan can do too (or maybe even more, thanks to Python). But somehow with Conan it just didn’t take off. I cannot really list clear and exact reasons why I wasn’t entirely happy with Conan. I guess, vcpkg just feels better, if one can take a “feeling” as a base for making such an important decision about one’s build system.

Yet again I’d like to say once more that our less fortunate experience with Conan might be because we used it wrong, and probably we should have spent some more time reading the documentation. But at the same time with vcpkg we were on the right track from the very beginning, so perhaps one could say that vcpkg is more intuitive to use (when you already have good enough experience with CMake).

I cannot say that Conan is worse. Even though we did not proceed with it for all our projects, I did add it to some of our smaller projects, mostly tools and demos, where it wasn’t so time-consuming to integrate, and there it was doing the job quite nicely. But now we have decided to go with integrating vcpkg in all our projects.

With that said, it’s surely nice to have alternatives, and so it’s only great that C++ developers have more than one package manager to choose from.

Updates

2023-09-01 | Build time improvements

Some months later we finished the work of moving all of our project’s bundled/vendored dependencies from the repository and making ports for them in our vcpkg registry. Along the way we also stopped producing object libraries and made all of the dependencies normal libraries.

The total amount of ports in the registry ended up being more than 60 - that is both our direct dependencies and transitive dependencies (dependencies of dependencies). Of course, after excluding platform-specific and feature-specific dependencies the number becomes about 30-40 in average, but still, that is quite a number.

So, naturally, we expected a big improvement in build times, since all the dependencies are now restored from local cache instead of being built every time from sources. And we did get an improvement, but not as big as we thought we would:

vcpkg, build time improvements on buildbots

Some clarification:

the “bundled” column shows project build time with dependencies built from bundled sources, and “vcpkg” column shows project build time when dependencies are restored pre-built from local cache;
not all platforms are building the same amount of components. For example, Windows Clang and some other configurations do not build tools, samples and demos, so they have less work to do;
the project is built on several different buildbots, and not all of them have the same performance (for instance, mac-1 buildbot is considerably faster than mac-8).

But why most of the Windows builds got so little improvement - that I cannot explain. Using Ninja instead of Visual Studio would certainly result in faster builds, but unfortunately we are stuck with the latter for now, and anyway the ratio would likely still be the same.

The only guess I have is based on the fact that all of those “slow” configurations (less than 10% improvement) are building everything (not just our SDK libraries but also tools, samples and demos), and configurations with big improvements (30% and more) are building just our SDK libraries, so no tools, samples or demos. And I am guessing that while compiling time has improved consistently(?) across all platforms, one should not forget that there is also linking, and apparently that process actually degraded in those configurations that build applications (tools/samples/demos). Like I said, we were building part of our dependencies as object libraries, and so now when all of the dependencies became normal libraries, that added more work for the linker (did it really?). I guess. If not that, I don’t know what then.

To test that theory I’ve run one of the Windows builds on my machine, first with everything enabled and then with disabled tools/samples/demos:

vcpkg, build time improvements on personal machine

Surprisingly, even with the full build there was a 24% improvement on my machine versus 7% improvement on buildbot. Why the hell is it so? Well, I did use Ninja instead of Visual Studio generator, which certainly speeded up the build, but that applies to both build times (with and without vcpkg), so the ratio shouldn’t be affected, should it?

If you are curious why the total build time is 15/14 minutes on buildbot and only 10/8 minutes on my machine, that is because buildbots also run certain preparation steps and then do packing and publishing after the build, and all that adds up to the total time.

And then the build without tools/samples/demos got 41% improvement! So that would confirm my theory about “degraded” linking times, but I am still not sure if it really is the actual reason.

Either way, while it’s nice to get faster builds, for us that was not the main goal of using a package manager. We are mostly happy about the fact that we can finally stop bundling/vendoring 3rd-party sources in our repository and resolve the project dependencies like adults.

2024-07-24 | Tools as host dependencies

Sometimes your build needs to have a certain tool like cURL, SQLite or some other custom executable to perform various tasks before compiling the project sources. For example, you might need to serialize some binary into C++ sources, such as proj.db, so it could be “compiled in” and used from in-memory VFS.

Often you would just install such tools on your machines / buildbots beforehand, but sometimes, and especially that concerns custom in-house tools, you would want to build those together with your project dependencies, and vcpkg can do that too - this is what host dependencies are for.

And it’s all good as long as you need to use those tools in ports, because portfiles have CURRENT_HOST_INSTALLED_DIR variable, so when you need to pass the path of a tool into the CMake project handled by that portfile, you can simply do something like this:

find_program(SQLITE_TOOL
    NAMES "sqlite3"
    PATHS "${CURRENT_HOST_INSTALLED_DIR}/tools/sqlite"
    NO_DEFAULT_PATH
    REQUIRED
)

vcpkg_cmake_configure(
    SOURCE_PATH "${SOURCE_PATH}"
    OPTIONS
        -DEXE_SQLITE3=${SQLITE_TOOL}
        # ...
)

But! What if you need to know the paths to those tools in an actual final project that isn’t a port? Sure, the project dependencies are resolved with vcpkg, and it does fetch, build and install the tools too, but how do you get the paths to the tools executables in your project’s CMakeLists.txt? As you might have guessed, CURRENT_HOST_INSTALLED_DIR variable isn’t available there, as it is only set in portfiles.

If those tools weren’t host dependencies, then of course you could’ve composed the path to a tool as ${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/tools/some/, because VCPKG_TARGET_TRIPLET variable is set and available in CMake projects. But the problem is that, being host dependencies, those tools aren’t installed into the target triplet prefix, they are installed into the host prefix.

Here’s an illustration. Having this vcpkg manifest:

{
    "name": "vcpkg-test",
    "version": "0.0.0",
    "dependencies":
    [
        {
            "name": "some-cppgenerator",
            "host": true
        },
        "glslang",
        "zlib"
    ]
}

we get the following installation:

├── arm64-osx
│   ├── share
│   │   ├── ...
│   └── tools
│       └── some-cppgenerator
│           └── cppGenerator
├── some-arm64-osx
│   ├── debug
│   │   └── lib
│   │       ├── ...
│   ├── include
│   │   ├── glslang
│   │   │   ├── ...
│   │   ├── zlib
│   │   │   ├── ...
│   ├── lib
│   │   ├── ...
│   ├── share
│   │   ├── glslang
│   │   │   ├── ...
│   │   └── zlib
│   │       ├── ...
│   └── tools
│       └── glslang
│           ├── glslangValidator
│           └── spirv-remap
└── vcpkg
    ├── ...

The some-arm64-osx is my custom triplet, which conveniently also serves the purpose of demonstrating different paths/prefixes for the host and target triplets.

So the path to glslangValidator can be obtained with:

find_program(GLSLANG_VALIDATOR glslangValidator "${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/tools/glslang" REQUIRED)
# or just
#set(GLSLANG_VALIDATOR "${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/tools/glslang/glslangValidator")

But how to get the path to cppGenerator? Specifically, how to get this arm64-osx value of the host triplet name? It should also be something simple, right? Well…

…it is not.

But wait, you’ll say, isn’t there a VCPKG_HOST_TRIPLET variable on that same documentation page where VCPKG_TARGET_TRIPLET was? Aye, there is, but you’ll be surprised to learn that this variable isn’t fucking set by default. Apparently, it is only meant to be set by you (in CLI with -D, or in a preset, or as an environment variable), should you ever need that, otherwise it will remain empty, so you won’t be able to get the default host triplet name from it.

Okay, you’ll say, but there is VCPKG_USE_HOST_TOOLS, which should add the paths to the tools from the host prefix into the CMAKE_PROGRAM_PATH, and then find_program() will find them? Yes, that much is true, but the problem here is that it still relies on the VCPKG_HOST_TRIPLET, which isn’t fucking set by default, so what you’ll get in CMAKE_PROGRAM_PATH is the target triplet prefix still - the same what you get if you don’t set the VCPKG_USE_HOST_TOOLS.

And actually there is another problem with VCPKG_USE_HOST_TOOLS. If we imagine for a second that VCPKG_HOST_TRIPLET is set to something meaningful, and if you set VCPKG_USE_HOST_TOOLS, then it will add the host triplet prefix tools paths to the CMAKE_PROGRAM_PATH, which is good, but at the same time it will not add the target triplet prefix tools paths, which were there before! To illustrate that with the previous illustration, without VCPKG_USE_HOST_TOOLS being set here are the paths that I get in the CMAKE_PROGRAM_PATH (if I check its contents somewhere after the project() call):

/path/to/vcpkg-test/build/macos-arm64/vcpkg_installed/some-arm64-osx/tools
/path/to/vcpkg-test/build/macos-arm64/vcpkg_installed/some-arm64-osx/tools/glslang

and here are the paths that I get in CMAKE_PROGRAM_PATH when VCPKG_USE_HOST_TOOLS is set (for example, with -DVCPKG_USE_HOST_TOOLS=1):

/path/to/vcpkg-test/build/macos-arm64/vcpkg_installed/arm64-osx/tools
/path/to/vcpkg-test/build/macos-arm64/vcpkg_installed/arm64-osx/tools/some-cppgenerator

Here’s also a bugreport from Craig Scott, which, you guessed it right, first got auto-staled and then auto-closed as “not planned”.

So even if VCPKG_HOST_TRIPLET had a default value (remember, we are only imagining that it has one), you would have not had both the host and triplet tools paths added to CMAKE_PROGRAM_PATH. Well, unless you would have hacked them into there yourself with list(APPEND CMAKE_PROGRAM_PATH ...), but that you could’ve done regardless, so what’s the point. And anyway, since VCPKG_HOST_TRIPLET isn’t set by default, these mental exercises are purely hypothetical.

Coming back to reality, surely there must be a way to obtain the value of that auto-detected/default host triplet name, because vcpkg itself does get it from somewhere, doesn’t it:

...
-- Running vcpkg install
Fetching registry information from git@some.git.repository:some/registry.git (HEAD)...
Detecting compiler hash for triplet arm64-osx...

Here - the arm64-osx value. That is what we want to get, so how does one get it, for fucks sake.

[ here goes that picture with Boromir again ]

Yeah, there is no way.

…Except for parsing the output(!) of the undocumented(!!) command z-print-config:

$ vcpkg z-print-config
{
  "buildtrees": "/path/to/vcpkg/buildtrees",
  "default-triplet": "arm64-osx",
  "downloads": "/path/to/vcpkg/downloads",
  "host-triplet": "arm64-osx",
  "installed": "/path/to/vcpkg/installed",
  "manifest-mode-enabled": false,
  "packages": "/path/to/vcpkg/packages",
  "tools": "/path/to/vcpkg/downloads/tools",
  "vcpkg-root": "/path/to/vcpkg",
  "versions-output": "/path/to/vcpkg/buildtrees/versioning_/versions"
}

There it fucking is! The host-triplet value, hallelujah. And don’t let default-triplet confuse you - that’s the target triplet, you can check yourself what value it will get if you’ll add --triplet arm64-osx-dynamic.

So one more time, just to underline it for the absolute clarity: to get the target triplet prefix path you have ${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET} ready to be used right away, and to get the host triplet prefix path you apparently are supposed to do something similar to this:

if(NOT DEFINED VCPKG_HOST_TRIPLET OR "${VCPKG_HOST_TRIPLET}" STREQUAL "")
    # https://learn.microsoft.com/en-us/vcpkg/users/config-environment#vcpkg_default_host_triplet
    set(VCPKG_DEFAULT_HOST_TRIPLET "")
    if(DEFINED ENV{VCPKG_DEFAULT_HOST_TRIPLET})
        set(VCPKG_DEFAULT_HOST_TRIPLET $ENV{VCPKG_DEFAULT_HOST_TRIPLET})
    endif()
    if("${VCPKG_DEFAULT_HOST_TRIPLET}" STREQUAL "")
        message(DEBUG "VCPKG_DEFAULT_HOST_TRIPLET is empty, getting host triplet from vcpkg CLI tool directly...")
        # right now this seems to be the only (and undocumented?) way to obtain the auto-detected/default host triplet name
        execute_process(
            COMMAND vcpkg z-print-config # might want to prepend `vcpkg` with `$ENV{VCPKG_ROOT}/`
            OUTPUT_VARIABLE VCPKG_PRINTED_CONFIG
            OUTPUT_STRIP_TRAILING_WHITESPACE
            COMMAND_ERROR_IS_FATAL ANY
        )
        #message(DEBUG "VCPKG_PRINTED_CONFIG: ${VCPKG_PRINTED_CONFIG}")
        string(JSON VCPKG_DEFAULT_HOST_TRIPLET
            ERROR_VARIABLE VCPKG_DEFAULT_HOST_TRIPLET_ERROR # to prevent it from failing right away
            GET ${VCPKG_PRINTED_CONFIG} "host-triplet"
        )
        if(NOT "${VCPKG_DEFAULT_HOST_TRIPLET_ERROR}" STREQUAL "NOTFOUND")
            # older versions of vcpkg use underscore symbol instead of dash (ну блять),
            # so you need to check for that variant too, and this time it should
            # abort on error, so no ERROR_VARIABLE
            string(JSON VCPKG_DEFAULT_HOST_TRIPLET
                GET ${VCPKG_PRINTED_CONFIG} "host_triplet"
            )
        endif()
        message(DEBUG "Got the following default host triplet from vcpkg CLI tool output: ${VCPKG_DEFAULT_HOST_TRIPLET}")
    else()
        message(DEBUG "Got the following default host triplet from environment variable: ${VCPKG_DEFAULT_HOST_TRIPLET}")
    endif()
    set(VCPKG_HOST_TRIPLET ${VCPKG_DEFAULT_HOST_TRIPLET})
endif()

The motherfucking 34 lines just to get one value. Christ almighty. And let’s not forget that z-print-config is undocumented, and being a seemingly internal thing (bacause it is prefixed with z-?), it might change its output format or even completely disappear at some point in future versions. Заебись!

On the bright side though, now that you have the host triplet name value, you can finally compose paths to the host tools:

if(NOT DEFINED VCPKG_HOST_TRIPLET OR "${VCPKG_HOST_TRIPLET}" STREQUAL "")
    message(FATAL_ERROR "The host vcpkg triplet is not set (or this host is not supported)")
endif()
set(CPP_GENERATOR "${VCPKG_INSTALLED_DIR}/${VCPKG_HOST_TRIPLET}/tools/some-cppgenerator/cppGenerator")
# or
#find_program(CPP_GENERATOR cppGenerator "${VCPKG_INSTALLED_DIR}/${VCPKG_HOST_TRIPLET}/tools/some-cppgenerator" REQUIRED)
message(DEBUG "Path to cppGenerator: ${CPP_GENERATOR}")

But that’s not the end of it, hehe. Well, it is, but there exists a (not very unlikely) scenario, in which your project might be added as a sub-project into another project, which won’t be using vcpkg. And as your project (being a sub-project) will be built from sources, it will need a different way to obtain the path to that cppGenerator tool, as there will be no vcpkg stuff around. And of course, let’s imagine that all the other dependencies that your project has are somehow already resolved. Who knows, maybe those people resolved your project dependencies on their own or maybe took your project distribution, kept only the dependencies and are now building your sources as a sub-project in their project. Either way, to account for this scenario, the following condition should do:

if(PROJECT_IS_TOP_LEVEL)
    if(NOT DEFINED VCPKG_HOST_TRIPLET OR "${VCPKG_HOST_TRIPLET}" STREQUAL "")
        message(FATAL_ERROR "The host vcpkg triplet is not set (or this host is not supported)")
    endif()
    set(CPP_GENERATOR "${VCPKG_INSTALLED_DIR}/${VCPKG_HOST_TRIPLET}/tools/some-cppgenerator/cppGenerator")
    message(DEBUG "Path to cppGenerator: ${CPP_GENERATOR}")
else() # so we are in a sub-project and we assume that vcpkg isn't used
    find_program(CPP_GENERATOR cppGenerator ${CPP_GENERATOR_PATH}) # REQUIRED
    if("${CPP_GENERATOR}" STREQUAL "CPP_GENERATOR-NOTFOUND")
        message(FATAL_ERROR "Couldn't find cppGenerator, did you provide -DCPP_GENERATOR_PATH?")
    else()
        message(DEBUG "Found cppGenerator: ${CPP_GENERATOR}")
    endif()
endif()

which, naturally, will require the parent/main project configuration to have one new configuration argument: -DCPP_GENERATOR_PATH="/path/to/some-cppgenerator".

2024-10-11 | Caching in JFrog Artifactory

Eventually we came to realization that we could benefit from a common binary cache for our buildbots (and developers machines). The main reason was that instead of wasting space on the same binaries in the local cache on every buildbot we could populate a remote cache on some common in-house server and fetch the cached binaries over the network. For sure, that introduces a network delay, but for a 10 gigabit local network it isn’t that bad.

Aside from more efficient disk space usage on buildbots and developers machines, there are some other benefits of using a binary cache on a common local server:

People who have no cache at all on their machines (such as first-time users, though it’s a rather rare scenario in our case) can save a significant amount of time by fetching pre-built dependencies from a remote cache instead of building them from sources. For me this a very tiny benefit really, but for others it might be important;
In the process of preparing for the build and during the build, vcpkg might fetch various required tools, such as CMake, Ninja, PowerShell, Python, Perl, MSYS, NASM and so on, and it is scary to see how many important components of your build system are hosted somewhere else on the internet, so outside of your IT infrastructure. That quite naturally can lead to very real problems when some of those hosting websites go down, such as how it already happened several times with NASM. And so having all the tools needed for the build hosted on an in-house server will allow you to isolate your build environment from the internet (which is how it should be from the day one).

Here I should probably clarify that vcpkg cache consists of two things:

binary caching - build artifacts (actual packages that are built from sources);
asset caching - required build tools (CMake, PowerShell, Python, etc).

There are several options available for the remote storage provider/backend, and we chose HTTP, as it works well with JFrog Artifactory generic repository. Sure, that means bringing one more service into the equation, but we have already been using Artifactory for other package formats/managers, so we already had to deal with maintaining it anyway. Besides, I imagine, one could just as well (relatively) easily implement one’s own backend of that HTTP type, especially if JFrog’s pricing doesn’t quite fit into one’s budget (as it’s basically a robbery).

The procedure of using JFrog Artifactory as a HTTP backend for vcpkg caching is already described here, and I will just add some more details below.

So first you create a generic repository. Only need to choose a name, the rest of the parameters can be left with default values:

JFrog Artifactory, new generic repository

Binary caching

And then the only thing you need to do to enable binary caching is set this environment variable:

export VCPKG_BINARY_SOURCES="clear;http,https://artifactory.YOUR.HOST/artifactory/vcpkg-binary-caching/{name}/{version}/{triplet}/{sha},readwrite,Authorization: Bearer YOUR-ARTIFACTORY-API-ACCESS-TOKEN"

Do note the readwrite parameter in the configuration string: while you would want to have readwrite for your buildbots (and your team), your users should probably have read there, as you most likely (should) have different group permissions in your Artifactory instance.

Other than that, this is it, nothing else needs to be done. Now the next vcpkg build will populate the cache in Artifactory, and for that it will build everything from sources again, despite the fact that you already have those packages in your local cache. The “caching” process will look like this:

Successfully stored zlib_decovar-arm64-linux.zip to https://artifactory.YOUR.HOST/artifactory/vcpkg-binary-caching/zlib/1.3.1/decovar-arm64-linux/1f6b71726126b26de458527f1c5f24ff51b963229e4268a437bfcd8a7f139cfc.
Stored binaries in 1 destinations in 698 ms.

After that is done for the first time, the next builds will be restoring the packages from this remote cache. And when your local environment changes, resulting in a different hash, the cache will be updated with this new package variant. So eventually it will look something like this:

JFrog Artifactory, zlib variants in vcpkg binary cache

Asset caching

The procedure for enabling asset caching is a bit more involved, as it requires (at least at the moment) having a script that would do the actual downloading/uploading. From what I see, the script itself can be written in any language, and here’s what it should do:

Accept the following CLI arguments:
- original download URL of the asset;
- expected SHA512 hash/checksum of the asset;
- local path, where the asset should be downloaded to (that includes the asset file name);
Try to download the asset from the specified remote cache (base URL and credentials can be hardcoded in the script or taken from environment variables);
If specified cache doesn’t contain this asset yet, fallback to downloading from the original download URL, verify the checksum of the downloaded file and upload it to the cache.

Here’s an example of such a script in PowerShell.

Once you have the script, to enable asset caching you only need to set this environment variable:

export X_VCPKG_ASSET_SOURCES="clear;x-script,pwsh /path/to/your/vcpkg-assets-caching.ps1 {url} {sha512} {dst};x-block-origin"

Then try to delete some assets from /path/to/vcpkg/downloads/tools/ (and their archives from /path/to/vcpkg/downloads/) and run some build which would require those. It should try to fetch the tools from Artifactory, and since they are not there (yet), it will fallback to downloading them from their original URLs and will then upload them to Artifactory. Here’s how the uploaded assets will look like in there:

It is also possible to pre-upload the assets into Artifactory with a bare cURL, should you ever need to do that:

$ curl \
    -H "X-JFrog-Art-Api:YOUR-ARTIFACTORY-API-ACCESS-TOKEN" \
    -H "X-Checksum-Sha1:1490fafec812e59818e838c94f7777380501db84" \
    -T ./7z2408-extra.7z \
    "https://artifactory.YOUR.HOST/artifactory/vcpkg-binary-caching/_assets/7z2408-extra.7z"

Either way, next time one of those assets will be missing from the local /path/to/vcpkg/downloads/, vcpkg should fetch them from Artifactory instead of their original locations on the internet.

For me it didn’t go all that well on the first attempt, as I was getting these errors:

A suitable version of 7zip was not found (required v24.8.0).
error: Missing 7z2408-extra.7z and downloads are blocked by x-block-origin.
error: Launching <mirror-script>: error: calling CreateProcessW failed with 2 (The system cannot find the file specified.)

out of which the first one turned out to be misleading, because the actual reason was that in my environment (Windows, believe it or not) there was no pwsh available in the PATH. Took me some time to figure that out, but then I just replaced pwsh with PowerShell (or, I guess, I could’ve made a symlink instead), and everything went fine after that. So that second error about CreateProcessW is the real source of the problem, and it would be more helpful if it was the first one to show up.

You might say, what good is a PowerShell script, if you are unable to run it anywhere but Windows, however PowerShell has been cross-platform for quite some time now, so you can in fact use this script on all the major platforms. Although, as usual with GNU/Linux, official instructions for installing PowerShell didn’t work for me (neither with ARM-based distribution nor with x64-based), but fortunately one can simply download binaries directly, and those work just fine.

Declaration of VAR