Serialization API

flashlight uses the cereal library for serialization. It provides utility macros and functions to easily define serialization properties for arbitrary classes and save or load modules, variables, standard library containers, and many other common C++ types.

Serializing Objects

To serialize individual instances on the fly, use save function:

// objects to be saved
std::shared_ptr<fl::Module> module = .... ;
std::unordered_map<int, std::string> config = .... ;

fl::save("<filepath>", module, config);

To deserialize, use load function:

// create the objects to be loaded in advance
std::shared_ptr<fl::Module> module;
std::unordered_map<int, std::string> config;

fl::load("<filepath>", module, config);

Serializing Classes

If we define a new class that needs to support serialization, flashlight provides easy-to-use macros to avoid writing boilerplate code as required by cereal. For example:

// Animal.h
class Animal {
 private:
  std::string name_;

  FL_SAVE_LOAD(name_) // Defines save, load functions

 public:
  Animal(const std::string& name) : name_(name) {}

  virtual std::string whoAmI() {
    return "Reporting from Base class - Animal.";
  }

  virtual ~Animal() = default;
};

CEREAL_REGISTER_TYPE(Animal)

// Cat.h
class Cat : public Animal {
 private:
  bool isHungry_;

  FL_SAVE_LOAD_WITH_BASE(Animal, isHungry_) // Defines save, load functions

 public:
  Cat(const std::string& name, bool isHungry)
      : Animal(name), isHungry_(isHungry) {}

  virtual std::string whoAmI() override {
    return "Hello! I'm a cat.";
  }
};

CEREAL_REGISTER_TYPE(Cat)

For additional guidlines, check out flashlight/common/Serialization.h. To summarize, save() shouldn’t mutate, load() may assume output is default- constructed, and one shouldn’t save long, size_t, dim_t etc. due to differing sizes on 32-bit vs 64-bit platforms.

Note

In the example above, if instead using FL_SAVE_LOAD(name_, isHungry_) for Cat, CEREAL_REGISTER_POLYMORPHIC_RELATION(Cat, Animal) must be `declared in Cat.h.

Custom Serialization

flashlight also supports defining custom serializers. For example:

// Dog.h
class Dog : public Animal {
 private:
  int capacity_;
  std::function<int(int)> eatFunc_; // `cereal` doesn't support function objects

  FL_SAVE_LOAD_DECLARE() // Declares save, load functions
 public:
  Dog(int capacity)
      : Animal("dog"),
        capacity_(capacity),
        eatFunc_([capacity](int b) { return b - capacity; }) {}
};

template <class Archive>
void Dog::save(Archive& ar, const uint32_t /* version */) const {
  ar(cereal::base_class<Animal>(this), capacity_);
}

template <class Archive>
void Dog::load(Archive& ar, const uint32_t /* version */) {
  ar(cereal::base_class<Animal>(this), capacity_);
  auto capacity = capacity_;
  eatFunc_ = [capacity](int b) { return b - capacity; };
}

CEREAL_REGISTER_TYPE(Dog)

Warning

When serializing smart pointers, cereal will only save data from the underlying object once.

template <class Archive>
void MyClass::save(Archive& ar, const uint32_t /* version */) const {
  Variable v(Tensor({50, 50}), true);
  ar(v); // `v` is saved
  v = v + 1; // Creates a new variable
  ar(v); // `v` is saved
  v.tensor() += 1; // `SharedData` pointer in `v` storing the Tensor is still the same
  ar(v); // `v` is NOT saved
}

Versioning

flashlight supports versioning for saving and loading to make maintaining backward compatibility easier:

// Panda.h
class Panda : public Animal {
 private:
  std::string color_;
  bool eating_;

  FL_SAVE_LOAD_WITH_BASE(Animal, color_, fl::versioned(eating_, 1))

  // fl::versioned(eating_, 1) will make sure the object is saved/loaded only
  // for versions >= 1. While using custom serialization, `version` number is passed
  // an argument to save/load functions and can be used to serialize appropriately.

 public:
  Panda(const std::string& col) : Animal("panda"), color_(col), eating_(true) {}
};

CEREAL_REGISTER_TYPE(Panda)
CEREAL_CLASS_VERSION(Panda, 2) // associate class with a version number