status-im/nim-style-guide
2
0
Fork 0
mirror of https://github.com/status-im/nim-style-guide.git synced 2025-02-22 19:18:18 +00:00
Code Issues Projects Releases Wiki Activity

interop guide (#13)

Browse Source 
Initial structure for a language interop guide (with go/rust) covering
common interop scenarios and how we typically deal with them.
This commit is contained in:
Jacek Sieka 2023-06-07 09:08:44 +02:00 committed by GitHub
parent be8857d614
commit 369e933e24
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 474 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
src/SUMMARY.md
View File

@ -43,3 +43,7 @@
- [Debugging](tooling.debugging.md)
- [Profiling](tooling.profiling.md)
- [Tricks](tooling.tricks.md)
- [Interop with other languages](interop.md)
- [C / general wrapping](interop.c.md)
- [go](interop.go.md)
- [rust](interop.rust.md)

158
src/interop.c.md Normal file
View File

@ -0,0 +1,158 @@
# C / General wrapping
ABI wrapping is the process describing the low-level interface of a library in an interop-friendly way using the lowest common denominator between the languages. For interop, we typically separate the "raw" ABI wrapper from higher-level code that adds native-language conveniece.
When importing foreign libraries in Nim, the ABI wrapper can be thought of as a C "header" file: it describes to the compiler what code and data types are available in the library and how to encode them.
Conversely, exporting Nim code typically consists of creating special functions in Nim using the C-compatible subset of the langauge then creating a corrsponding ABI description in the target language.
Typical of the ABI wrapper is the use of the [FFI](https://nim-lang.org/docs/manual.html#foreign-function-interface) pragmas (`importc`, `exportc` etc) and, depending on the library, C types such as `cint`, `csize_t` as well as manual memory management directives such as `pointer`, `ptr`.
In some cases, it may be necessary to write an "export wrapper" in C - this happens in particular when the library was not written with ineroperability in mind, for example when there is heavy C pre-processor use or function implementations are defined in the C header file.
## Exporting
Exporting Nim code is done by creating an export module that presents the Nim code as a simplified C interface:
```nim
import mylibrary
proc function(arg: int64): cint {.exportc: "function", raises: [].} =
# Validate incoming arguments before converting them to Nim equivalents
if arg >= int64(int.high) or arg <= int64(int.low):
return 0 # Expose error handling
mylibrary.function(int(arg))
```
## Importing
### Build process
To import a library into Nim, it must first be built by its native compiler - depending on the complexity of the library, this can be done in several ways.
The preferred way of compiling a native library is it include it in the Nim build process via `{.compile.}` directives:
```nim
{.compile: "somesource.c".}
```
This ensures that the library is built together with the Nim code using the same C compiler as the rest of the build, automatically passing compilation flags and using the expected version of the library.
Alterantives include:
* build the library as a static or shared library, then make it part of the Nim compilation via `{.passL.}`
* difficult to ensure version compatiblity
* shared library requires updating dynamic library lookup path when running the binary
* build the library as a shared library, then make it part of the Nim compilation via `{.dynlib.}`
* nim will load the library via `dlopen` (or similar)
* easy to run into ABI / version mismatches
* no record in binary about the linked library - tools like `ldd` will not display the dependencies correctly
### Naming
ABI wrappers are identified by `abi` in their name, either as a suffix or as the module name itself:
* [secp256k1](https://github.com/status-im/nim-secp256k1/blob/master/secp256k1/abi.nim)
* [bearssl](https://github.com/status-im/nim-bearssl/blob/master/bearssl/abi/bearssl_rand.nim)
* [sqlite3](https://github.com/arnetheduck/nim-sqlite3-abi/blob/master/sqlite3_abi.nim)
### Functions and types
Having created a separate module for the type, create definitions for each function and type that is meant to be used from Nim:
```nim
# Create a helper pragma that describes the ABI of typical C functions:
# * No Nim exceptions
# * No GC interation
{.pragma ffi, importc, raises: [], gcsafe.}
proc function(arg: int64): cint {.ffi.}
```
### Callbacks
Callbacks are functions in the Nim code that are registered with the imported library and called from the library:
```nim
# The "callback" helper pragma:
#
# * sets an explicit calling convention to match C
# * ensures no exceptions leak from Nim to the caller of the callback
{.pragma: callback, cdecl, raises: [].}
{.pragma: ffi, importc, raises: [], gcsafe.}
import strutils
proc mycallback(arg: cstring) {.callback.} =
# Write nim code as usual
echo "hello from nim: ", arg
# Don't let exceptions escape the callback
try:
echo "parsed: ", parseInt($arg)
except ValueError:
echo "couldn't parse"
proc registerCallback(callback: proc(arg: cstring) {.callback.}) {.ffi.}
registerCallback(mycallback)
```
Care must be taken that the callback is called from a Nim thread - if the callback is called from a thread controlled by the library, the thread might need to be [initialized](./interop.md#calling-nim-code-from-other-languages) first.
### Memory allocation
Nim supports both garbage-collected, stack-based and manually managed memory allocation.
When using garbage-collected types, care must be taken to extend the lifetime of objects passed to C code whose lifetime extends beyond the function call:
```nim
{.pragma: ffi, importc, raises: [], gcsafe.}
# Register a long-lived instance with C library
proc register(arg: ptr cint) {.ffi.}
# Unregister a previously registered instance
proc unregister(arg: ptr cint) {.ffi.}
proc setup(): ref cint =
let arg = new cint
# When passing garbage-collected types whose lifetime extends beyond the
# function call, we must first protect the them from collection:
GC_ref(arg)
register(addr arg[])
arg
proc teardown(arg: ref cint) =
# ... later
unregister(addr arg[])
GC_unref(arg)
```
## C wrappers
Sometimes, C headers contain not only declarations but also definitions and / or macros. Such code, when exported to Nim, can cause build problems, symbol duplication and other related issues.
The easiest way to expose such code to Nim is to create a plain C file that re-exports the functionality as a normal function:
```c
#include <inlined_code.h>
/* Reexport `function` using a name less likely to conflict with other "global" symbols */
int library_function() {
/* function() is either a macro or an inline funtion defined in the header */
return function();
}
```
## Tooling
* [`c2nim`](https://github.com/nim-lang/c2nim) - translate C header files to Nim, providing a starting place for wrappers
## References
* [Nim manual, FFI](https://nim-lang.org/docs/manual.html#foreign-function-interface)
* [Nim for C programmers](https://github.com/nim-lang/Nim/wiki/Nim-for-C-programmers)
* [Nim backend reference](https://nim-lang.org/docs/backends.html)

30
src/interop.cpp.md Normal file
View File

@ -0,0 +1,30 @@
# C++
Nim has two modes of interoperability with C++: API and ABI.
The API interoperability works by compiling Nim to C++ and using the C++ compiler to compile the result - this mode gives direct access to C++ library code without any intermediate wrappers, which is excellent for exploratory and scripting work.
However, this mode also comes with several restrictions:
* the feature gap between Nim and C++ is wider: C++ has many features that cannot be represented in Nim and vice versa
* while "simple C++" can be wrapped this way, libraries using modern C++ features are unlikely to work or may only be wrapped partially
* C++-generated Nim code is more likely to run into cross-library issues in applications using multiple wrapped libraries
* this is due to C++'s increased strictness around const-correctness and other areas where Nim, C and C++ differ in capabilities
* a single library that uses Nim's C++ API features forces the entire application to be compiled with the C++ backend
* the C++ backend receives less testing overall and therefore is prone to stability issues
Thus, for C++ the recommened way of creating wrappers is similar to other languages: write an export library in C++ that exposes the C++ library via the C ABI then import the C code in Nim - this future-proofs the wrapper against the library starting to use C++ features that cannot be wrapped.
## Qt
[Qt](https://www.qt.io/) takes control of the main application thread to run the UI event loop. Blocking this thread means that the UI becomes unresponsive, thus it is recommended to run any compution in separate threads, for example using [nim-taskpools](https://github.com/status-im/nim-taskpools).
For cross-thread communication, the recommended way of sending information to the Qt thread is via a [queued signal/slot connection](https://doc.qt.io/qt-6/threads-qobject.html#signals-and-slots-across-threads).
For sending information from the Qt thread to other Nim threads, encode the data into a buffer allocated with `allocShared` and use a thread-safe queue such as `std/sharedlists`.
## Examples
* [nimqml](https://github.com/filcuc/nimqml) - exposes the Qt C++ library via a [C wrapper](https://github.com/filcuc/dotherside)
* [nlvm](https://github.com/arnetheduck/nlvm/tree/master/llvm) - imports the [LLVM C API]() which wraps the LLVM compiler written in C++
* [godot](https://github.com/pragmagic/godot-nim) - imports `godot` via its exported [C API](https://docs.godotengine.org/de/stable/tutorials/scripting/gdnative/what_is_gdnative.html)

44
src/interop.go.md Normal file
View File

@ -0,0 +1,44 @@
# Go interop
Nim and Go are both statically typed, compiled languages capable of interop via a simplifed C ABI.
On the Go side, interop is handled via [cgo](https://pkg.go.dev/cmd/cgo).
## Threads
Go includes a native `M:N` scheduler for running Go tasks - because of this, care must be taken both when calling Nim code from Go: the thread from which the call will happen is controlled by Go and we must initialise the Nim garbage collector in every function exposed to Go, as documented in the [main guide](./interop.md#calling-nim-code-from-other-languages).
As an alternative, we can pass the work to a dedicated thread instead - this works well for asynchronous code that reports the result via a callback mechanism:
```nim
{.pragma callback, cdecl, gcsafe, raises: [].}
type
MyAPI = object
queue: ThreadSafeQueue[ExportedFunctionData] # TODO document where to find a thread safe queue
ExportedFunctionCallback = proc(result: cint) {.callback.}
ExportedFunctionData =
v: cint
callback: ExportedFunctionCallback
proc runner(api: ptr MyAPI) =
while true:
processQueue(api[].queue)
proc initMyAPI(): ptr MyAPI {.exportc, raises: [].}=
let api = createShared(MyAPI)
# Shutdown / cleanup omitted for brevity
discard createThread(runner, api)
api
proc exportedFunction(api: ptr MyAPI, v: cint, callback: ExportedFunctionCallback) =
# By not allocating any garbage-collected data, we avoid the need to initialize the garbage collector
queue.add(ExportedFunctionData(v: cint, callback: callback))
```
## Variables
When calling Nim code from Go, care must be taken that instances of [garbage-collected types](./interop.md#garbage-collected-types) don't pass between threads - this means process-wide globals and other forms of shared-memory apporaches of GC types must be avoided.
[`LockOSThread`](https://pkg.go.dev/runtime#LockOSThread) can be used to constrain the thread from which a particular `goroutine` calls Nim.

218
src/interop.md Normal file
View File

@ -0,0 +1,218 @@
# Interop with other languages (FFI)
Nim comes with powerful interoperability options, both when integrating Nim code in other languages and vice versa.
Acting as a complement to the [manual](https://nim-lang.org/docs/manual.html#foreign-function-interface), this section of the book covers interoperability / [FFI](https://en.wikipedia.org/wiki/Foreign_function_interface): how to integrate Nim into other languages and how to use libraries from other languages in Nim.
While it is possible to automate many things related to FFI, this guide focuses on core functionality - while tooling, macros and helpers can simplify the process, they remain a cosmetic layer on top of the fundamentals presented here.
The focus of this guide is on pragmatic solutions available for the currently supported versions of Nim - 1.6 at the time of writing - the recommendations may change as new libraries and Nim versions become available.
## Basics
In interop, we rely on a lowest common denominator of features between languages - for compiled languages, this is typically the mutually overlapping part of the [ABI](https://en.wikipedia.org/wiki/Application_binary_interface).
Nim is unique in that it also allows interoperability at the API level with C/C++ - however, this guide focuses on interoperability via ABI since this is more general and broadly useful.
Most languages define their FFI in terms of a simplified version of the C ABI - thus, the process of using code from one language in another typically consists of two steps:
* exporting the source library functions and types as "simple C"
* importing the "simple C" functions and types in the target language
We'll refer to this part of the process as ABI wrapping.
Since libraries tend to use the full feature set of their native language, we can see two additional steps:
* exposing the native library code in a "simple C" variant via a wrapper
* adding a wrapper around the "simple C" variant to make the foreign library feel "native"
We'll call this API wrapping - the API wrapper takes care of:
* conversions to/from Nim integer types
* introducing Nim idioms such as generics
* adapting the [error handling](./errors.md) model
The C ABI serves as the "lingua franca" of interop - the [C guide](./interop.c.md) in particular can be studied for topics not covered in the other language-specific sections.
## Calling Nim code from other languages
Nim code can be compiled both as shared and static libraries and thus used from other languages.
When calling Nim from other languages, care must be taken to first initialize the garbage collector, at least once for every thread.
Garbage collector initialization is a two-step process:
* the garbage collector itself must be inititialized with a call to `setupForeignThreadGc`
* `nimGC_setStackBottom` must be called to establish the starting point of the stack
* this function must be called in all places where it is possible that the exported function is being called from a "shorter" stack frame
Typically, this is solved with a "library initialization" call that users of the library should call near the beginning of every thread (ie in their `main` or thread entry point function):
```nim
var initialized {.threadvar.}: bool
proc initializeMyLibrary() {.exportc.} =
when declared(setupForeignThreadGc): setupForeignThreadGc()
proc exportedFunction {.exportc, raises: [].} =
assert initialized, "You forgot to call `initializeMyLibrary"
echo "Hello from Nim
```
In some languages such as [Go](./interop.go.md), it is hard to anticipate which thread code will be called from - in such cases, you can safely initialize the garbage collector in every exported function instead:
```nim
template initializeMyLibrary() =
when declared(setupForeignThreadGc): setupForeignThreadGc()
when declared(nimGC_setStackBottom):
var locals {.volatile, noinit.}: pointer
locals = addr(locals)
nimGC_setStackBottom(locals)
# Always safe to call
proc exportedFunction {.exportc, raises: [].} =
initializeMyLibrary()
echo "Hello from Nim
```
See also the [Nim documentation](https://nim-lang.org/docs/backends.html#interfacing-backend-code-calling-nim) on this topic.
### Globals and top-level code
Code written outside of a `proc` / `func` is executed as part of `import`:ing the module, or, in the case of the "main" module of the program, as part of executing the module itself similar to the `main` function in C.
Nim puts this code in a function called `NimMain` and it serves a complement to the "library intialization function": it must only be called once!
## Exceptions
You must ensure that no exceptions pass to the foreign language - instead, catch all exceptions and covert them to a different [error handling mechanism](./errors.md), annotating the exported function with `{.raises: [].}`.
## Memory
Nim is generally a GC-first language meaning that memory is typically managed via a thread-local garbage collector.
Nim also supports manual memory management - this is most commonly used for threading and FFI.
### Garbage-collected types
Garbage-collection applies to the following types which are allocated from a thread-local heap:
* `string` and `seq` - these are value types that underneath use the GC heap for the payload
* the `string` uses a dedicated length field but _also_ ensures NULL-termination which makes it easy to pass to C
* `seq` uses a similar in-memory layout without the NULL termination
* addresses to elements are stable as long as as elements are not added
* `ref` types
* types that are declared as `ref object`
* non-ref types that are allocated on the heap with `new` (and thus become `ref T`)
### `ref` types and pointers
The lifetime of garbage-collected types is undefined - the garbage collector generally runs during memory allocation but this should not be relied upon - instead, lifetime can be extended by calling `GC_ref` and `GC_unref`.
`ref` types have a stable memory address - to pass the address of a `ref` instance via FFI, care must be taken to extend the lifetime of the instance so that it is not garbage-collected
```nim
proc register(v: ptr cint) {.importc.}
proc unregister(v: ptr cint) {.importc.}
# Allocate a `ref cint` instance
let number = new cint
# Let the garbage collector know we'll be creating a long-lived pointer for FFI
GC_ref(number)
# Pass the address of the instance to the FFI function
register(addr number[])
# ... later, in reverse order:
# Stop using the instance in FFI - address is guaranteed to be stable
unregister(addr number[])
# Let the garbage collector know we're done
GC_unref(number)
```
### Manual memory management
Manual memory management is done with [`create`](https://nim-lang.org/docs/system.html#create%2Ctypedesc) (by type), [`alloc`](https://nim-lang.org/docs/system.html#alloc.t%2CNatural) (by size) and [`dealloc`](https://nim-lang.org/docs/system.html#dealloc%2Cpointer):
```nim
proc register(v: ptr cint) {.importc.}
proc unregister(v: ptr cint) {.importc.}
# Allocate a `ptr cint` instance
let number = create cint
# Pass the address of the instance to the FFI function
register(number)
# ... later, in reverse order:
# Stop using the instance in FFI - address is guaranteed to be stable
unregister(number)
# Free the instance
dealloc(number)
```
To allocate memory for cross-thread usage, ie allocating in one thread and deallocating in the other, use `createShared` / `allocShared` and `deallocShared` instead.
## Threads
Threads in Nim are created with [`createThread`](https://nim-lang.org/docs/threads.html) which creates the thread and prepares the garbage collector for use on that thread.
See [above](#calling-nim-code-from-other-languages) for how to initialize the garbage collector when calling Nim from threads created in other languages.
### Passing data between threads
The primary method of passing data between threads is to encode the data into a shared memory section then transfer ownership of the memory section to the receiving thread either via a thread-safe queue, channel, socket or pipe.
The queue itself can be passed to thread either at creation or via a global variable, though we generally seek to avoid global variables.
```nim
# TODO pick a queue
type ReadStatus = enum
Empty
Ok
Done
proc read(queue: ptr Queue[pointer], var data: seq[byte]): ReadStatus =
var p: pointer
if queue.read(p):
if isNil(p):
ReadStatus.Done
else:
var len: int
copyMem(addr len, p, sizeof(len))
data = newSeqUninitalized[byte](len)
copyMem(addr data[0], cast[pointer](cast[uint](data) + sizeof(len)), len)
ReadStatus.Ok
else:
ReadStatus.Empty
proc write(queue: ptr Queue[pointer], data: openArray[byte]) =
# Copy data to a shared length-prefixed buffer
let
copy = allocShared(int(len) + sizeof(len))
copyMem(copy, addr len, sizeof(len))
copyMem(cast[pointer](cast[uint](copy) + sizeof(len)), v, len)
# Put the data on a thread-safe queue / list
queue.add(copy)
proc reader(queue: ptr Queue[pointer]):
var data: seq[byte]
while true:
case queue.read(data)
of Done: return
of Ok: process(data)
of Empty:
# Polling should usually be replaced with an appropriate "wake-up" mechanism
sleep(100)
```
### async / await
When `chronos` is used, execution is typically controlled by the `chronos` per-thread dispatcher - passing data to `chronos` is done either via a pipe / socket or by polling a thread-safe queue.
## Resources
* [Nim backends documentation](https://nim-lang.org/1.6.0/backends.html)

15
src/interop.rust.md Normal file
View File

@ -0,0 +1,15 @@
# Rust interop
Nim and Rust are both statically typed, compiled languages capable of "systems programming".
Because of these similarities, interop between Nim and [`rust`](https://doc.rust-lang.org/nomicon/ffi.html) is generally straightforward and handled the same way as [C interop](./interop.c.md) in both languages: Rust code is exported to C then imported in Nim as C code and vice versa.
## Memory
While Nim is a GC-first language, `rust` in general uses lifetime tracking (via `Box`) and / or reference counting (via `Rc`/`Arc`) outside of "simple" memory usage.
When used with Nim, care must be taken to extend the lifetimes of Nim objects via `GC_ref` / `GC_unref`.
## Tooling
* [`nbindgen`](https://github.com/arnetheduck/nbindgen/) - create Nim ["ABI headers"](./interop.md#basics) from exported `rust` code

8
src/libraries.wrappers.md
View File

@ -2,12 +2,14 @@
Prefer native `Nim` code when available.
`C` libraries and libraries that expose a `C` API may be used (including `rust`, `C++`).
`C` libraries and libraries that expose a `C` API may be used (including `rust`, `C++`, `go`).
Avoid `C++` libraries.
Prefer building the library on-the-fly from source using `{.compile.}`. Pin the library code using a submodule or amalgamation.
The [interop](./interop.md) guide contains more information about foreing language interoperability.
### Pros
* Wrapping existing code can improve time-to-market for certain features
@ -23,7 +25,7 @@ Prefer building the library on-the-fly from source using `{.compile.}`. Pin the
* Less test suite coverage - most of `Nim` test suite uses `C` backend
* Many core `C++` features like `const`, `&` and `&&` difficult to express - in particular post-`C++11` code has a large semantic gap compared to Nim
* Different semantics for exceptions and temporaries compared to `C` backend
* All-or-nothing - can't use `C++` codegen selectively for `C++` libraries
* All-or-nothing - can't use `C++` backend selectively for `C++` libraries
* Using `{.compile.}` increases build times, specially for multi-binary projects - use judiciously for large dependencies
### Practical notes
@ -40,6 +42,6 @@ Prefer building the library on-the-fly from source using `{.compile.}`. Pin the
### Examples
* [nim-secp256k1](https://github.com/status-im/nim-secp256k1)
* [nim-sqlite](https://github.com/arnetheduck/nim-sqlite3-abi)
* [nim-sqlite3-abi](https://github.com/arnetheduck/nim-sqlite3-abi)
* [nim-bearssl](https://github.com/status-im/nim-bearssl/)
* [nim-blscurve](https://github.com/status-im/nim-blscurve/)