Use static linking on Linux and MacOS, dynamic on Windows (#81)

- We now only support static linking on Linux and MacOS. Windows is still only using dynamic linking. - Dynamically linking can be enabled on Linux and MacOS using the 'rocksdb_dynamic_linking' flag but this is not documented and the dynamic libraries are not built in nimble install. They can be built using one of the build scripts if needed. - Updated the wrapper generation so that we don't need to use dynlibOverride when using static linking. - Static libs are copied to the same directory as the dll. - Cleaned up the librocksdb.nim file which imports the rocksdb wrapper. - Updated the tasks in the nimble file.
2025-02-19 18:38:15 +00:00 · 2025-01-21 14:24:04 +08:00 · 2025-01-21 14:24:04 +08:00 · 6149edce39
commit 6149edce39
parent 985fb2c795
11 changed files with 955 additions and 2484 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@ -123,8 +123,3 @@ jobs:
          nimble --version
          nimble install -y
          nimble test
-
-          # static linking is not supported on windows
-          if [[ "${{ matrix.target.os }}" != "windows" ]]; then
-            nimble test_static
-          fi
--- a/README.md
+++ b/README.md
@ -1,4 +1,4 @@
-# Nim-RocksDB
+# Nim-RocksDb

 [![Build Status (Travis)](https://img.shields.io/travis/status-im/nim-rocksdb/master.svg?label=Linux%20/%20macOS "Linux/macOS build status (Travis)")](https://travis-ci.org/status-im/nim-rocksdb)
 [![Windows build status (Appveyor)](https://img.shields.io/appveyor/ci/nimbus/nim-rocksdb/master.svg?label=Windows "Windows build status (Appveyor)")](https://ci.appveyor.com/project/nimbus/nim-rocksdb)
@ -10,35 +10,53 @@ A Nim wrapper for [Facebook's RocksDB](https://github.com/facebook/rocksdb), a p

 ## Current status

-Nim-RocksDB provides a wrapper for the low-level functions in the librocksdb c library.
+Nim-RocksDb provides a wrapper for the low-level functions in the librocksdb c
+library.

 ## Installation

-Nim-RocksDB requires Nim and the Nimble package manager. For Windows you will need Visual Studio 2015 Update 3 or greater with the English language pack.
+Nim-RocksDb requires Nim and the Nimble package manager. For Windows you will
+need Visual Studio 2015 Update 3 or greater with the English language pack.

 To get started run:
 ```
-nimble install
+nimble install rocksdb
 ```

-This will download and install the RocksDB dynamic libraries for your platform and copy them into the `build/` directory of the project. When including this library in your application you may want to copy these libraries into another location or set the LD_LIBRARY_PATH environment variable (DYLD_LIBRARY_PATH on MacOS, PATH on Windows) to include the `build/` directory so that your application can find them on startup.
+This will download and install the RocksDB libraries for your platform and copy
+them into the `build/` directory of the project. On Linux and MacOS only static
+linking to the RocksDb libraries is supported and on Windows only dynamic linking
+is supported.
+
+On Windows you may want to copy the dll into another location or set your PATH
+to include the `build/` directory so that your application can find the dll on
+startup.
+
+### Compression libraries
+
+RocksDb supports using a number of compression libraries. This library builds
+and only supports the following compression libraries:
+- lz4
+- zstd
+
+On Linux and MacOS these libraries are staticly linked into the final binary
+along with the RocksDb static library. On Windows they are staticly linked into
+the RocksDb dll.

-Alternatively you can use the `rocksdb_static_linking` flag to statically link the library into your application.

 ### Static linking

-To build the RocksDB static libraries run:
-```
-./scripts/build_static_deps.sh
-```
-
-To statically link RocksDB, you would do something like:
+On Linux and MacOS your Nim program will need to use the C++ linker profile
+because RocksDb is a C++ library. For example:

 ```
-nim c -d:rocksdb_static_linking --threads:on your_program.nim
+  when defined(macosx):
+    switch("clang.linkerexe", "clang++")
+  when defined(linux):
+    switch("gcc.linkerexe", "g++")
 ```

-See the config.nims file which contains the static linking configuration which is switched on with the `rocksdb_static_linking` flag. Note that static linking is currently not supported on windows.
+Note that static linking is currently not supported on windows.

 ## Usage

@ -46,12 +64,13 @@ See [simple_example](examples/simple_example.nim)

 ### Contribution

-Any contribution intentionally submitted for inclusion in the work by you shall be dual licensed as above, without any
-additional terms or conditions.
+Any contribution intentionally submitted for inclusion in the work by you shall
+be dual licensed as above, without any additional terms or conditions.

 ## Versioning

-The library generally follows the upstream RocksDb version number, adding one more number for tracking changes to the Nim wrapper itself.
+The library generally follows the upstream RocksDb version number, adding one
+more number for tracking changes to the Nim wrapper itself.

 ## License

@ -65,7 +84,8 @@ or

 * Apache License, Version 2.0, ([LICENSE-APACHEv2](LICENSE-APACHEv2) or http://www.apache.org/licenses/LICENSE-2.0)

-at your option. This file may not be copied, modified, or distributed except according to those terms.
+at your option. This file may not be copied, modified, or distributed except
+according to those terms.

 ### Dependency License

--- a/config.nims
+++ b/config.nims
@ -1,5 +1,5 @@
 # nim-rocksdb
-# Copyright (c) 2019-2023 Status Research & Development GmbH
+# Copyright (c) 2019-2025 Status Research & Development GmbH
 # Licensed under either of
 #  * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
 #  * MIT license ([LICENSE-MIT](LICENSE-MIT))
@ -12,17 +12,13 @@ when fileExists("nimble.paths"):
  include "nimble.paths"
 # end Nimble config

-when defined(rocksdb_static_linking):
+when not defined(rocksdb_dynamic_linking) and not defined(windows):
  # use the C++ linker profile because it's a C++ library
  when defined(macosx):
    switch("clang.linkerexe", "clang++")
  else:
    switch("gcc.linkerexe", "g++")

-  switch("dynlibOverride", "rocksdb")
-  switch("dynlibOverride", "lz4")
-  switch("dynlibOverride", "zstd")
-
 --styleCheck:
  usages
 --styleCheck:
--- a/rocksdb.nimble
+++ b/rocksdb.nimble
@ -11,35 +11,22 @@ installDirs = @["build"]
 ### Dependencies
 requires "nim >= 2.0", "results", "tempfile", "unittest2"

+template build() =
+  when defined(windows):
+    exec ".\\scripts\\build_dlls_windows.bat"
+  else:
+    exec "scripts/build_static_deps.sh"
+
+before install:
+  build()
+
 task format, "Format nim code using nph":
  exec "nimble install nph@0.6.0"
  exec "nph ."

-task clean, "Remove temporary files":
-  exec "rm -rf build"
-  exec "make -C vendor/rocksdb clean"
-
 task test, "Run tests":
-  let runTests = "nim c -d:nimDebugDlOpen -r --threads:on tests/test_all.nim"
-  when defined(linux):
-    exec "export LD_LIBRARY_PATH=build; " & runTests
-  when defined(macosx):
-    exec "export DYLD_LIBRARY_PATH=build; " & runTests
+  build()
  when defined(windows):
-    exec runTests
-
-task test_static, "Run tests after static linking dependencies":
-  when defined(windows):
-    echo "Static linking is not supported on windows"
-    quit(1)
-
-  exec "scripts/build_static_deps.sh"
-  exec "nim c -d:rocksdb_static_linking -r --threads:on tests/test_all.nim"
-
-before install:
-  when defined(linux):
-    exec "scripts/build_shared_deps_linux.sh"
-  when defined(macosx):
-    exec "scripts/build_shared_deps_osx.sh"
-  when defined(windows):
-    exec ".\\scripts\\build_dlls_windows.bat"
+    exec "nim c -d:nimDebugDlOpen -r --threads:on tests/test_all.nim"
+  else:
+    exec "nim c -r --threads:on tests/test_all.nim"
--- a/rocksdb/lib/librocksdb.nim
+++ b/rocksdb/lib/librocksdb.nim
@ -1,4 +1,4 @@
-# Copyright 2018-2024 Status Research & Development GmbH
+# Copyright 2018-2025 Status Research & Development GmbH
 # Licensed under either of
 #
 #  * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
@ -22,6 +22,8 @@

 ## This file exposes the low-level C API of RocksDB

+import std/[os, strutils]
+
 {.push raises: [].}

 type
@ -87,15 +89,19 @@ type
  rocksdb_memory_consumers_t* = object
  rocksdb_wait_for_compact_options_t* = object

-##  DB operations
+when defined(windows):
+  const librocksdb = "librocksdb.dll"
+elif defined(macosx):
+  const librocksdb = "librocksdb.dylib"
+else:
+  const librocksdb = "librocksdb.so"

-when defined(rocksdb_static_linking):
-  {.pragma: importrocks, importc, cdecl.}
-
-  import std/[os, strutils]
+when defined(rocksdb_dynamic_linking) or defined(windows):
+  {.push importc, cdecl, dynlib: librocksdb.}
+else:
  const
    topLevelPath = currentSourcePath.parentDir().parentDir().parentDir()
-    libsDir = topLevelPath.replace('\\', '/') & "/build/lib"
+    libsDir = topLevelPath.replace('\\', '/') & "/build/"

  {.passl: libsDir & "/librocksdb.a".}
  {.passl: libsDir & "/liblz4.a".}
@ -103,7 +109,7 @@ when defined(rocksdb_static_linking):

  when defined(windows):
    {.passl: "-lshlwapi -lrpcrt4".}
-else:
-  {.pragma: importrocks, importc, cdecl, dynlib: librocksdb.}
+
+  {.push importc, cdecl.}

 include ./rocksdb_gen.nim
--- a/rocksdb/lib/rocksdb.h
+++ b/rocksdb/lib/rocksdb.h
@ -46,15 +46,7 @@

 #ifdef C2NIM
 #  def ROCKSDB_LIBRARY_API
-#  dynlib librocksdb
 #  cdecl
-#  if defined(windows)
-#    define librocksdb "librocksdb.dll"
-#  elif defined(macosx)
-#    define librocksdb "librocksdb.dylib"
-#  else
-#    define librocksdb "librocksdb.so"
-#  endif
 #  mangle uint32_t uint32
 #  mangle uint16_t uint16
 #  mangle uint8_t  uint8
--- a/rocksdb/lib/rocksdb_gen.nim
+++ b/rocksdb/lib/rocksdb_gen.nim
--- a/scripts/build_shared_deps_linux.sh
+++ b/scripts/build_shared_deps_linux.sh
@ -1,7 +1,7 @@
 #!/usr/bin/env bash

 # Nim-RocksDB
-# Copyright 2024 Status Research & Development GmbH
+# Copyright 2024-2025 Status Research & Development GmbH
 # Licensed under either of
 #
 #  * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
--- a/scripts/build_shared_deps_osx.sh
+++ b/scripts/build_shared_deps_osx.sh
@ -1,7 +1,7 @@
 #!/usr/bin/env bash

 # Nim-RocksDB
-# Copyright 2024 Status Research & Development GmbH
+# Copyright 2024-2025 Status Research & Development GmbH
 # Licensed under either of
 #
 #  * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
--- a/scripts/build_static_deps.sh
+++ b/scripts/build_static_deps.sh
@ -15,7 +15,7 @@ cd "$(dirname "${BASH_SOURCE[0]}")"/..

 REPO_DIR="${PWD}"
 ROCKSDB_LIB_DIR="${REPO_DIR}/vendor/rocksdb"
-BUILD_DEST="${REPO_DIR}/build/lib"
+BUILD_DEST="${REPO_DIR}/build"

 : "${MAKE:=make}"

@ -34,6 +34,13 @@ export DEBUG_LEVEL=0

 if ${MAKE} -C "${ROCKSDB_LIB_DIR}" -q unity.a; then
  echo "RocksDb static libraries already built. Skipping build."
+
+  # Copy the built libraries in case the build directory has been removed
+  mkdir -p "${BUILD_DEST}"
+  cp "${ROCKSDB_LIB_DIR}/liblz4.a" "${BUILD_DEST}/"
+  cp "${ROCKSDB_LIB_DIR}/libzstd.a" "${BUILD_DEST}/"
+  cp "${ROCKSDB_LIB_DIR}/unity.a" "${BUILD_DEST}/librocksdb.a"
+
  exit 0
 else
  ${REPO_DIR}/scripts/clean_build_artifacts.sh
--- a/scripts/generate_wrapper.sh
+++ b/scripts/generate_wrapper.sh
@ -37,15 +37,7 @@ sed -i ':a;N;$!ba;s/#ifdef _WIN32\
 #define ROCKSDB_LIBRARY_API\
 #endif/#ifdef C2NIM\
 #  def ROCKSDB_LIBRARY_API\
-#  dynlib librocksdb\
 #  cdecl\
-#  if defined(windows)\
-#    define librocksdb "librocksdb.dll"\
-#  elif defined(macosx)\
-#    define librocksdb "librocksdb.dylib"\
-#  else\
-#    define librocksdb "librocksdb.so"\
-#  endif\
 #  mangle uint32_t uint32\
 #  mangle uint16_t uint16\
 #  mangle uint8_t  uint8\