Memory Management¶
Note
This section is only applicable if using the ArrayFire backend.
Flashlight’s ArrayFire backend also contains a framework to implement and use custom memory managers for devices running computation.
By default, for better performance, flashlight uses a custom memory manager (fl::CachingMemoryManager
) implemented with this framework in place of the default ArrayFire memory manager. When flashlight is linked to, MemoryManagerInstaller::installDefaultMemoryManager()
is invoked, which sets the default ArrayFire memory manager to be an instance of the CachingMemoryManager
. This behavior can be changed by modifying the function accordingly.
A custom memory manager can be set after flashlight initializes; see the documentation for fl::MemoryManagerInstaller
below for setting custom memory managers.
Warning
The default ArrayFire memory manager is no longer supported in flashlight; using it explicitly may result in significantly degraded performance.
Defining a Custom Memory Manager¶
-
class
MemoryManagerAdapter
¶ An interface for defining memory managers purely in C++.
The ArrayFire memory management API is defined using C callbacks; this class provides a thin layer of abstraction over this callbacks and acts as an adapter between derived C++ class implementations and the ArrayFire C API. In particular:
Each instance has an associated af_memory_manager whose payload is a pointer to
this
which allows callbacks to call C++ class methods after casting.Provides logging functions and a logging mode which logs all function calls from ArrayFire and all relevant arguments. Only virtual base class methods that have derived implementations are eligible for logging.
The
MemoryManagerInstaller
provides an interface for setting implemented memory managers as the active ArrayFire memory managers by setting relevant callbacks on construction.
For documentation of virtual methods, see ArrayFire’s memory header for full specifications on when these methods are called by ArrayFire and the JIT.
Subclassed by fl::CachingMemoryManager, fl::DefaultMemoryManager
Public Functions
Constructs a MemoryManagerAdapter.
- Parameters
[in] deviceInterface
: a pointer to aMemoryManagerDeviceInterface
. Function pointers on the interface will be defined once the memory manager is installed.[in] logStream
: a pointer to an output stream to use for logging. All function calls to overridden base class methods by ArrayFire will be logged to the resulting stream in conjunction with passed arguments. If a valid output stream is passed, the memory manager will initialize with logging enabled. This argument is optional - passing no argument disables logging for the memory manager by default.
-
template<typename ...
Values
>
voidlog
(std::string fname, Values... vs)¶ Logs information to the
MemoryManagerAdapters
’s log stream.If logging mode is enabled, function calls to virtual base class methods are logged.
- Parameters
[in] fname
: the name of the function to be logged (or some arbitrary prefix string)[in] vs
: variadic list of arguments (ofint
type) to be appended in a space-delimited fashion after the fname
-
void
setLogStream
(std::ostream *logStream)¶ Sets the log stream for a memory manager base.
- Parameters
[in] logStream
: the output stream to set.
-
std::ostream *
getLogStream
() const¶ Returns the log stream for a memory manager base.
- Return
the manager’s log stream.
-
void
setLoggingEnabled
(bool log)¶ Sets the logging mode for the memory manager base.
If disabled, no logs are written. If enabled, all function calls to virtual base class methods are logged.
- Parameters
[in] log
: bool determinig whether logging is enabled.
-
void
setLogFlushInterval
(size_t interval)¶ Sets a number of lines after which the adapter’s temporary logging buffer gets flushed to the user-supplied output stream.
Default value is 50.
- Parameters
[in] interval
: the number of lines after which to flush the temporary log buffer. Supplied interval must be greater than 1.
-
af_memory_manager
getHandle
() const¶ Returns the ArrayFire handle for this memory manager.
- Return
the
af_memory_manager
handle associated with this class.
Activating a Memory Manager in ArrayFire¶
-
class
MemoryManagerInstaller
¶ Manages memory managers and abstracts away parts of the ArrayFire C memory manager API that enables setting active memory managers in ArrayFire.
On construction, the installer modifies a given instance of a
MemoryManagerAdapter
and sets needed closures for its underlyingaf_memory_manager
handle. Destruction is a noop no state change occurs. If the underlyingMemoryManagerAdapter
still exists, it can still be the active ArrayFire memory manager even if its installer has been destroyed.Public Functions
Creates a new instance using a
MemoryManagerAdapter
.Uses the adapter’s underlying
af_memory_manager
handle and performs the following setup:Sets all function pointers using the Array/Fire C memory management API on the underlying
af_memory_manager
handle to point to closures which call the installedMemoryManagerAdapter
’s instance methods.Sets the closures on the adapter’s
MemoryManagerDeviceInterface
to call ArrayFire C-API native device memory management functions which automatically delegate to the proper backend and are use pre-defined implementations in ArrayFire internals.
- Parameters
[in] managerImpl
: a pointer to theMemoryManagerAdapter
to be installed.
Gets the memory manager adapter used in this instance.
- Return
a pointer to some derived type of
MemoryManagerAdapter
-
void
setAsMemoryManager
()¶ Sets this
MemoryManagerInstaller
’sMemoryManagerAdapter
to be the active memory manager in ArrayFire.
-
void
setAsMemoryManagerPinned
()¶ Sets this
MemoryManagerInstaller
’sMemoryManagerAdapter
to be the active memory manager for pinned memory operations in ArrayFire.
Public Static Functions
-
static MemoryManagerAdapter *
getImpl
(af_memory_manager manager)¶ Returns an adapter given a handle.
Used to construct C++-style callbacks inside lambdas set on the ArrayFire C memory management API.
-
static MemoryManagerAdapter *
currentlyInstalledMemoryManager
()¶ Returns the currently installed custom memory manager, or null if none is installed.
-
static void
installDefaultMemoryManager
()¶ Initializes and installs the memory manager defaulted to on startup.
Uses a
CachingMemoryManager
by default. Only sets the memory manager - doesn’t set an AF pinned memory manager.
-
static void
unsetMemoryManager
()¶ Unsets the currently-set custom ArrayFire memory manager.
If no custom memory manager is set, results in a noop, since the default memory manager is set, and unsetting it would result in shutdown/destruction.
Native Memory Management Functions¶
-
struct
MemoryManagerDeviceInterface
¶ An interface for using native device memory management and JIT-related memory pressure functions.
Provides support for functions at the device and backend level which automatically delegate to the correct backend functions for native device interoperability. These functions call directly into ArrayFire functions.
Exposed as an external freestanding API so as to facilitate sharing native device closures across different parts of a memory manager implemenation.
Functions are automatically set when a
MemoryManagerDeviceInterface
that has been passed to a constructedMemoryManagerAdapter
is installed using aMemoryManagerInstaller
’ssetMemoryManager
orsetMemoryManagerPinned
method. Until one of these are called, the functions therein remain unset.For documentation of virtual methods, see ArrayFire’s memory header for full specifications.