Catmandu::Plugin::SideCar

NAME

Catmandu::Plugin::SideCar - Automatically update a parallel Catmandu::Store with metadata

SYNOPSIS

 # Using the command line
 $ cat catmandu.yml
 ---
 store:
     files:
      package: File::Simple
      options:
          root: /data/test123
          bags:
              index:
                  plugins:
                      - SideCar
                  sidecar:
                          package: ElasticSearch
                          options:
                              client: '1_0::Direct'
                              index_name: catmandu
    ...
 # Add files to the FileStore in bag 1234
 $ catmandu stream /tmp/test.txt to files --bag 1234 --id test.txt
 # Add metadata to the FileStore for bag 1234
 $ cat metadata.yml
 ---
 _id: 1234
 colors:
    - red
    - green
    - blue
 name: test
 ...
 $ catmandu import YAML to files < metadata.yml
 # Export the metadata again from the FileStore
 $ catmandu export files to YAML
 ---
 _id: 1234
 colors:
    - red
    - green
    - blue
 name: test
 ...
 # Or in your Perl program
 my $store = Catmandu->store('File::Simple',
            root => 'data/test123'
            bags => {
                index => {
                    plugins => [qw(SideCar)],
                    sidecar => {
                        package => "ElasticSearch",
                        options => {
                            client => '1_0::Direct',
                            index_name => 'catmandu',
                        }
                    }
               }
            });
 my $index = $store->index;
 $index->add({ _id => '1234' , colors => [qw(red green blue)] , name => 'test'});
 my $files = $index->files('1234');
 $files->upload(IO::File->new('</tmp/test.txt'), 'test.txt');
 my $file = $files->get('text.txt');
 $files->steam(IO::File->new('>/tmp/test.txt'),$file);

The Catmandu::Plugin::SideCar can be used to combine Catmandu::Store-s , Catmandu::FileStore-s (and Catmandu::Store::Multi , Catmandu::Store::File::Multi) as one access point. Every get,add,delete,drop and commit action in the store will be first executed in the original store and re-executed in the SideCar store.

COMBINING A FILESTORE WITH A STORE

To add metadata to a Catmandu::FileStore a SideCar needs to be added to the "index" bag of the FileStore:

    package: File::Simple
    options:
        root: /data/test123
        bags:
            index:
                plugins:
                    - SideCar
                sidecar:
                        package: ElasticSearch
                        options:
                            client: '1_0::Direct'
                            index_name: catmandu
                sidecar_bag: data

COMBINING A STORE WITH A FILESTORE

To add files to a Catmandu::Store a SideCar needs to be added to the bag containing the metadata (by default "data"):

    package: ElasticSearch
    options:
        client: '1_0::Direct'
        index_name: catmandu
        bags:
            data:
                plugins:
                    - SideCar
                sidecar:
                        package: File::Simple
                        options:
                            root: /data/test123
                            uuid: 1
                sidecar_bag: index

Notice that we added for the Catmandu::Store::File::Simple the requires "uuid" options because the Catmandu::Store::ElasticSearch is using UUIDs as default identifiers.

RESTRICTIONS

Some Catmandu::FileStore-s may set restrictions on the "_id"-s that can be used in records.

CONFIGURATION

sidecar STRING

sidecar HASH

sidecar Catmandu::Store | Catmandu::FileStore: The pointer to a configured Catmandu::Store or Catmandu::FileStore.

sidecar_bag: The SideCar Catmandu::Bag into which to store the data (default 'bag').