SourceXtractorPlusPlus  0.16
Please provide a description of the project.
HowTo add source property instance computation with configuration

The structure of this tutorial is identical to that of two previous HowTo add a new source property computation and HowTo add a new source property computation with configuration. Readers are assumed to have a good knowledge of these previous tutorials. We continue to follow a kind of munimum example, in which we add the possible generation of multiple instances of a given property.

Our example is that of a (dummy) flag image processing. A arbitrary-sized vector of flag files and corresponding flag identifiers can be provided through the program options. The same source property (i.e., the sum of the flag values over the pixel footprint of the source) is computed for all input files provided. For each flag file, a different column (which name is constructed from the concatenation of a root name and of the identifier provided through configuration) is extracted from the corresponding property and output to the catalog.

To mark the similarities (and the differences), the same class names are used with a WithInst post-fix. Note again that not much effort is made here to make a smart example. It is only thought for illustration purposes.

Source Property

Below is our minimum ExamplePropertyWithInst class.

namespace SExtractor {
class ExamplePropertyWithInst : public Property {
public:
virtual ~ExamplePropertyWithInst() = default;
ExamplePropertyWithInst(double flag_value_sum) : m_flag_value_sum(flag_value_sum) {}
double getFlagValueSum() const {
return m_flag_value_sum;
}
private:
double m_flag_value_sum;
}; // End of ExamplePropertyWithInst class
} // namespace SExtractor

This class is fully analogous to the previous ExampleProperty and we refer to it for explanation.

Configuration

Below is the configuration class, this time the code is separated in a header and an implementation files.

Here is the ExampleConfWithInst.h header file:

namespace SExtractor {
class ExampleConfWithInst : public Euclid::Configuration::Configuration {
public:
virtual ~ExampleConfWithInst() = default;
ExampleConfWithInst(long manager_id) : Configuration(manager_id) {}
std::map<std::string, OptionDescriptionList> getProgramOptions() override;
void preInitialize(const UserValues& args) override;
void initialize(const UserValues& args) override;
const std::map<std::string, std::shared_ptr<FlagImage> >& getFlagImageMap () const {
return m_flag_image_map;
}
private:
}; // End of ExampleConfWithInst class
} // namespace SExtractor

And the ExampleConfWithInst.cpp implementation file:

#include <boost/filesystem.hpp>
#include "SEFramework/Image/FitsReader.h"
#include "SEImplementation/Plugin/ExamplePluginWithInst/ExampleConfWithInst.h"
namespace po = boost::program_options;
namespace fs = boost::filesystem;
namespace SExtractor {
namespace {
const std::string FLAG_IMAGE_FILES { "ex-flag-images-files" };
const std::string FLAG_IMAGE_IDS { "ex-flag-images-ids" };
}
auto ExampleConfWithInst::getProgramOptions () -> std::map<std::string, OptionDescriptionList> {
return { {"Flag options", {
{ FLAG_IMAGE_FILES.c_str(), po::value<std::vector<std::string>>()->multitoken(),
"The FITS files containing flags"},
{ FLAG_IMAGE_IDS.c_str(), po::value<std::vector<std::string>>()->multitoken(),
"Identifiers of the flag images"},
}}};
}
void ExampleConfWithInst::preInitialize (const UserValues& args) {
if (args.count(FLAG_IMAGE_FILES) != 0) {
auto filenames = args.find(FLAG_IMAGE_FILES)->second.as<std::vector<std::string>>();
auto ids = args.find(FLAG_IMAGE_IDS)->second.as<std::vector<std::string>>();
if (filenames.size() != ids.size()) {
throw Elements::Exception() << "The number of flag IDs do not match the number of files !";
}
for (auto filename : filenames) {
if (!fs::exists(filename)) {
throw Elements::Exception() << "File " << filename << " does not exist";
}
}
}
}
void ExampleConfWithInst::initialize (const UserValues& args) {
if (args.count(FLAG_IMAGE_FILES) != 0) {
auto& filenames = args.find(FLAG_IMAGE_FILES)->second.as<std::vector<std::string>>();
auto& ids = args.find(FLAG_IMAGE_IDS)->second.as<std::vector<std::string>>();
for (auto index = 0; index < filenames.size(); index++) {
m_flag_image_map[ids.at(index)] = std::move(FitsReader<std::int64_t>::readFile(filenames.at(index)));
}
}
}
} // SExtractor namespace
T at(T... args)
T find(T... args)
T move(T... args)
string filename
Definition: conf.py:63
T size(T... args)

This code is somewhat similar to the previous example. The getProgramOptions() method is again used to define the specific options with the Boost program option syntax described before. In the current example, vector options are defined which allows to enter an arbitrary number of flag filenames and identifiers.

The preInitialize(args) method was not implemented in the previous example. Its purpose is to define quick checks, which allow to interrupt the program execution as early as possible, if the provided options are not fully valid. The checks are recursive over the possible multiple elements of the vectors and they are carried out only if at least a filename is provided.

The initialize(args) method is also more involved in this example. FITs files are read into an appropriate FlagImage data structure and they are stored into a map compose of an arbitrary number (as many as input configuration vector elements) of elements, together with the flag image identifiers used as the map key.

Again, files are opened and stored only if valid filenames are provided through the configuration. This makes this configuration optional. Nothing happens if the option "ex-flag-images-files" is not provided.

SourceTask

Below is the ExampleSourceTaskWithInst example analogous to the ExampleSourceTask. The main method is again computeProperties(source), but now there are two member variables (they were none before), which will be filled through a constructor call with information coming from the configuration. There is however no directed connection between the task and the configuration classes. Things will be tied together through the task factory (see next section).

namespace SExtractor {
class ExampleSourceTaskWithInst : public SourceTask {
public:
virtual ~ExampleSourceTaskWithInst() = default;
ExampleSourceTaskWithInst(std::shared_ptr<FlagImage> flag_image, unsigned int flag_instance) :
m_flag_image(flag_image), m_flag_instance(flag_instance) {
}
virtual void computeProperties(SourceInterface& source) const {
double sum = 0.0;
for (auto& coords : source.getProperty<PixelCoordinateList>().getCoordinateList()) {
sum += m_flag_image->getValue(coords.m_x, coords.m_y);
}
source.setIndexedProperty<ExamplePropertyWithInst>(m_flag_instance, sum);
};
private:
unsigned int m_flag_instance;
}; // End of ExampleSourceTaskWithInst class
} // namespace SExtractor
T sum(const NdArray< T > &array)

Another important difference, the method source.setIndexedProperty<ExamplePropertyWithInst>(m_flag_instance, sum) now replaces the previous source.setProperty<...>(...). This method takes care of calling the property constructor ignoring the first m_flag_instance parameter, i.e., ExamplePropertyWithInst(sum). It then attaches the new property to the source using the m_flag_instance integer. In this way, an arbitrary number of ExamplePropertyWithInst properties can be stored in the source providing they have been attached with different m_flag_instance. They can later be retrieved with a source.getProperty(instance_num) method call.

TaskFactory

The task factory is analogous to the previous example, but with extra complications to manage the possible property multiplicity. The flag identifiers coming from the configuration are stored in a vector of string, while the different FlagImage are put in a map. The map key is a PropertyId object which is built from the property type and instance index.

#include <memory>
#include "SEImplementation/Plugin/ExamplePluginWithInst/ExampleConfWithInst.h"
#include "SEImplementation/Plugin/ExamplePluginWithInst/ExamplePropertyWithInst.h"
#include "SEImplementation/Plugin/ExamplePluginWithInst/ExampleSourceTaskWithInst.h"
namespace SExtractor {
class ExampleTaskFactoryWithInst: public TaskFactory {
public:
virtual ~ExampleTaskFactoryWithInst () = default;
void reportConfigDependencies (Euclid::Configuration::ConfigManager& manager) const {
manager.registerConfiguration<ExampleConfWithInst>();
}
void configure (Euclid::Configuration::ConfigManager& manager) {
auto flag_image_map_from_conf = manager.getConfiguration<ExampleConfWithInst>().getFlagImageMap();
unsigned int instance_idx = 0;
for (auto& flag_name_image : flag_image_map_from_conf) {
m_instance_names.emplace_back(std::make_pair(flag_name_image.first, instance_idx));
auto property_id = PropertyId::create<ExamplePropertyWithInst>(instance_idx);
m_flag_image_map[property_id] = flag_name_image.second;
instance_idx++;
}
}
virtual std::shared_ptr<Task> createTask (const PropertyId& property_id) const {
if (m_flag_image_map.find(property_id) != m_flag_image_map.end()) {
auto& flag_image = m_flag_image_map.at(property_id);
return std::make_shared<ExampleSourceTaskWithInst>(flag_image, property_id.getIndex());
}
return nullptr;
}
void registerPropertyInstances (OutputRegistry& output_registry) {
output_registry.registerPropertyInstances<ExamplePropertyWithInst>(m_instance_names);
}
private:
};
} // namespace SExtractor
T make_pair(T... args)

The createTask(property_id) now has a property instance index dimension. Different tasks can be created for different PropertyId (i.e., different property indices for a given type).

The reportConfigDependencies(...) method must be implemented to declare configuration dependencies. Here, there is no other dependency than the ExampleConfWithConf itself.

Plugin

Although it is belwo seperated in a header and implementation files, the plugin is again almost identical to those of the previous two examples (see ExamplePlugin and ExamplePluginWithConf. There is only one registered catalog column output, but the main difference is the plugin_api.getOutputRegistry().enableOutput<ExamplePropertyWithInst>();, which specifies that this column will always be an output (unlike the two previous cases where the output columns were optional, i.e., to be specifically requested with a ad hoc option). The logic in the current example is that when a flag file is input through the configuration, the output column is always produced. But if no flag files are provided nothing is computed and output.

Here is the ExamplePluginWithInst.h header file:

namespace SExtractor {
class ExamplePluginWithInst : public Plugin {
public:
virtual ~ExamplePluginWithInst() = default;
virtual void registerPlugin(PluginAPI& plugin_api) override;
virtual std::string getIdString() const override;
}; // End of TestPlugin class
}

And the ExamplePluginWithInst.cpp implementation file:

#include "SEImplementation/Plugin/ExamplePluginWithInst/ExamplePluginWithInst.h"
#include "SEImplementation/Plugin/ExamplePluginWithInst/ExamplePropertyWithInst.h"
#include "SEImplementation/Plugin/ExamplePluginWithInst/ExampleTaskFactoryWithInst.h"
namespace SExtractor {
static StaticPlugin<ExamplePluginWithInst> example_plugin_with_inst;
void ExamplePluginWithInst::registerPlugin (PluginAPI& plugin_api) {
plugin_api.getTaskFactoryRegistry().registerTaskFactory<ExampleTaskFactoryWithInst, ExamplePropertyWithInst>();
plugin_api.getOutputRegistry().registerColumnConverter<ExamplePropertyWithInst, double>("FLAG_SUM",
[](const ExamplePropertyWithInst& prop) {
return prop.getFlagValueSum();
});
plugin_api.getOutputRegistry().enableOutput<ExamplePropertyWithInst>();
}
std::string ExamplePluginWithInst::getIdString () const {
return "example_plugin_with_inst";
}
}

The statement static StaticPlugin<ExamplePluginWithConf> example_plugin_with_conf; must appear to registers our example plugin as a static plugin (i.e., which must be compiled with SExtractor). Other types of plugin will be described later.