Memory Management

Note

This section is only applicable if using the ArrayFire backend.

Flashlight’s ArrayFire backend also contains a framework to implement and use custom memory managers for devices running computation.

By default, for better performance, flashlight uses a custom memory manager (fl::CachingMemoryManager) implemented with this framework in place of the default ArrayFire memory manager. When flashlight is linked to, MemoryManagerInstaller::installDefaultMemoryManager() is invoked, which sets the default ArrayFire memory manager to be an instance of the CachingMemoryManager. This behavior can be changed by modifying the function accordingly.

A custom memory manager can be set after flashlight initializes; see the documentation for fl::MemoryManagerInstaller below for setting custom memory managers.

Warning

The default ArrayFire memory manager is no longer supported in flashlight; using it explicitly may result in significantly degraded performance.

Defining a Custom Memory Manager

class MemoryManagerAdapter

An interface for defining memory managers purely in C++.

The ArrayFire memory management API is defined using C callbacks; this class provides a thin layer of abstraction over this callbacks and acts as an adapter between derived C++ class implementations and the ArrayFire C API. In particular:

  • Each instance has an associated af_memory_manager whose payload is a pointer to this which allows callbacks to call C++ class methods after casting.

  • Provides logging functions and a logging mode which logs all function calls from ArrayFire and all relevant arguments. Only virtual base class methods that have derived implementations are eligible for logging.

  • The MemoryManagerInstaller provides an interface for setting implemented memory managers as the active ArrayFire memory managers by setting relevant callbacks on construction.

For documentation of virtual methods, see ArrayFire’s memory header for full specifications on when these methods are called by ArrayFire and the JIT.

Subclassed by fl::CachingMemoryManager, fl::DefaultMemoryManager

Public Functions

MemoryManagerAdapter(std::shared_ptr<MemoryManagerDeviceInterface> deviceInterface, std::ostream *logStream = nullptr)

Constructs a MemoryManagerAdapter.

Parameters
  • [in] deviceInterface: a pointer to a MemoryManagerDeviceInterface. Function pointers on the interface will be defined once the memory manager is installed.

  • [in] logStream: a pointer to an output stream to use for logging. All function calls to overridden base class methods by ArrayFire will be logged to the resulting stream in conjunction with passed arguments. If a valid output stream is passed, the memory manager will initialize with logging enabled. This argument is optional - passing no argument disables logging for the memory manager by default.

template<typename ...Values>
void log(std::string fname, Values... vs)

Logs information to the MemoryManagerAdapters’s log stream.

If logging mode is enabled, function calls to virtual base class methods are logged.

Parameters
  • [in] fname: the name of the function to be logged (or some arbitrary prefix string)

  • [in] vs: variadic list of arguments (of int type) to be appended in a space-delimited fashion after the fname

void setLogStream(std::ostream *logStream)

Sets the log stream for a memory manager base.

Parameters
  • [in] logStream: the output stream to set.

std::ostream *getLogStream() const

Returns the log stream for a memory manager base.

Return

the manager’s log stream.

void setLoggingEnabled(bool log)

Sets the logging mode for the memory manager base.

If disabled, no logs are written. If enabled, all function calls to virtual base class methods are logged.

Parameters
  • [in] log: bool determinig whether logging is enabled.

void setLogFlushInterval(size_t interval)

Sets a number of lines after which the adapter’s temporary logging buffer gets flushed to the user-supplied output stream.

Default value is 50.

Parameters
  • [in] interval: the number of lines after which to flush the temporary log buffer. Supplied interval must be greater than 1.

af_memory_manager getHandle() const

Returns the ArrayFire handle for this memory manager.

Return

the af_memory_manager handle associated with this class.

Activating a Memory Manager in ArrayFire

class MemoryManagerInstaller

Manages memory managers and abstracts away parts of the ArrayFire C memory manager API that enables setting active memory managers in ArrayFire.

On construction, the installer modifies a given instance of a MemoryManagerAdapter and sets needed closures for its underlying af_memory_manager handle. Destruction is a noop no state change occurs. If the underlying MemoryManagerAdapter still exists, it can still be the active ArrayFire memory manager even if its installer has been destroyed.

Public Functions

MemoryManagerInstaller(std::shared_ptr<MemoryManagerAdapter> managerImpl)

Creates a new instance using a MemoryManagerAdapter.

Uses the adapter’s underlying af_memory_manager handle and performs the following setup:

  • Sets all function pointers using the Array/Fire C memory management API on the underlying af_memory_manager handle to point to closures which call the installed MemoryManagerAdapter’s instance methods.

  • Sets the closures on the adapter’s MemoryManagerDeviceInterface to call ArrayFire C-API native device memory management functions which automatically delegate to the proper backend and are use pre-defined implementations in ArrayFire internals.

Parameters

template<typename T>
std::shared_ptr<T> getMemoryManager() const

Gets the memory manager adapter used in this instance.

Return

a pointer to some derived type of MemoryManagerAdapter

void setAsMemoryManager()

Sets this MemoryManagerInstaller’s MemoryManagerAdapter to be the active memory manager in ArrayFire.

void setAsMemoryManagerPinned()

Sets this MemoryManagerInstaller’s MemoryManagerAdapter to be the active memory manager for pinned memory operations in ArrayFire.

Public Static Functions

static MemoryManagerAdapter *getImpl(af_memory_manager manager)

Returns an adapter given a handle.

Used to construct C++-style callbacks inside lambdas set on the ArrayFire C memory management API.

static MemoryManagerAdapter *currentlyInstalledMemoryManager()

Returns the currently installed custom memory manager, or null if none is installed.

static void installDefaultMemoryManager()

Initializes and installs the memory manager defaulted to on startup.

Uses a CachingMemoryManager by default. Only sets the memory manager - doesn’t set an AF pinned memory manager.

static void unsetMemoryManager()

Unsets the currently-set custom ArrayFire memory manager.

If no custom memory manager is set, results in a noop, since the default memory manager is set, and unsetting it would result in shutdown/destruction.

Native Memory Management Functions

struct MemoryManagerDeviceInterface

An interface for using native device memory management and JIT-related memory pressure functions.

Provides support for functions at the device and backend level which automatically delegate to the correct backend functions for native device interoperability. These functions call directly into ArrayFire functions.

Exposed as an external freestanding API so as to facilitate sharing native device closures across different parts of a memory manager implemenation.

Functions are automatically set when a MemoryManagerDeviceInterface that has been passed to a constructed MemoryManagerAdapter is installed using a MemoryManagerInstaller’s setMemoryManager or setMemoryManagerPinned method. Until one of these are called, the functions therein remain unset.

For documentation of virtual methods, see ArrayFire’s memory header for full specifications.