mspass::utility::MetadataDefinitions Class Reference

Define properties of Metadata known to mspass. More...

#include <MetadataDefinitions.h>

Public Member Functions
	MetadataDefinitions ()
	MetadataDefinitions (const std::string mdname)
	Construct from a namespace title. More...
	MetadataDefinitions (const std::string mdname, const mspass::utility::MDDefFormat form)
	Construct from a file with variable formats. More...
	MetadataDefinitions (const MetadataDefinitions &parent)
bool	is_defined (const std::string key) const noexcept
std::string	concept (const std::string key) const
mspass::utility::MDtype	type (const std::string key) const
std::list< std::string >	keys () const
void	add (const std::string key, const std::string concept_, const MDtype mdt)
bool	has_alias (const std::string key) const
	Methods to handle aliases. More...
bool	is_alias (const std::string key) const
	Ask if a key is a registered alias. More...
std::list< std::string >	aliases (const std::string key) const
std::pair< std::string, mspass::utility::MDtype >	unique_name (const std::string aliasname) const
void	add_alias (const std::string key, const std::string aliasname)
bool	writeable (const std::string key) const
bool	readonly (const std::string key) const
void	set_readonly (const std::string key)
	Lock a parameter to assure it will not be saved. More...
void	set_writeable (const std::string key)
	Force a key:value pair to be writeable. More...
bool	is_normalized (const std::string key) const
	Test if a key:value pair is set as normalized. More...
std::string	unique_id_key (const std::string key) const
	Returns a unique identifier for a normalized attribute. More...
std::string	collection (const std::string key) const
std::pair< std::string, std::string >	normalize_data (const std::string key) const
	Special method for efficiency. More...
std::list< std::string >	apply_aliases (mspass::utility::Metadata &d, const std::list< std::string > aliaslist)
	Apply a set of aliases to data. More...
void	clear_aliases (mspass::utility::Metadata &d)
	Restore any aliases to unique names. More...
MetadataDefinitions &	operator= (const MetadataDefinitions &other)
MetadataDefinitions &	operator+= (const MetadataDefinitions &other)
	Accumulate additional definitions. More...

Detailed Description

Define properties of Metadata known to mspass.

The metadata system in mspass was designed to be infinitely flexible. However, to be maintainable and provide some stability we need a method to define the full properties of the keys that define attributes known to the system. This object does that through wwhat I hope is a simple interface. The expectation is the main use of this class is in python code that will read and write to mongo. Mongo is type sensitive but python tries hard to by loosy goosy about types. The main use is then expected that all gets and puts to Metadata will be preceded by calls to the type method here. Based on the return the right get or put method can be called.

The overhead of creating this thing is not small. It will likely be recommended as an initialization step for most mspass processing scripts. Ultimately it perhaps should have a database constructor, but initially we will build it only from data file.

Constructor & Destructor Documentation

MetadataDefinitions() [1/4]

mspass::utility::MetadataDefinitions::MetadataDefinitions

(

)

Default constructor. Loads default schema name of mspass.

{
    try{
      string datadir=mspass::utility::data_directory();
      string path;
      path=datadir+"/yaml/"+DefaultSchemaName+".yaml";
      MetadataDefinitions tmp(path,MDDefFormat::YAML);
      *this=tmp;
    }catch(...){throw;};
}

MetadataDefinitions() [2/4]

mspass::utility::MetadataDefinitions::MetadataDefinitions

(

const std::string

mdname

)

Construct from a namespace title.

How this is to be implemented is to be determined, but for many uses a simple one line description of the name space for attributes would be a helpful api. At this time it is not implemented and attempts to use this will throw an exception.

{
    try{
      /* silent try to recover if the user adds .yaml to mdname*/
      std::size_t ipos;  //size_t seems essential here for this to work -weird
      string name_to_use;
      ipos=mdname.find(".yaml");
      if(ipos==std::string::npos)
      {
    name_to_use=mdname;
      }
      /* We throw an exception if the name is a path with / characters*/
      else if(mdname.find("/")!=std::string::npos)
      {
    throw MsPASSError("MetadataDefinitions:  name passed seems to be a full path name that is not allowed\nReceived this:  "
        +mdname,ErrorSeverity::Invalid);
      }
      else
      {
        name_to_use.assign(mdname,0,ipos);
      }
      string datadir=mspass::utility::data_directory();
      string path;
      path=datadir+"/yaml/"+name_to_use+".yaml";
      MetadataDefinitions tmp(path,MDDefFormat::YAML);
      *this=tmp;
    }catch(...){throw;};
 
}

MetadataDefinitions() [3/4]

mspass::utility::MetadataDefinitions::MetadataDefinitions	(	const std::string	mdname,
		const mspass::utility::MDDefFormat	form
	)

Construct from a file with variable formats.

This constructor reads from a file to build the object. The API allows multiple formats through the enum class.

Parameters

mdname	is the file to read
form	defines the format (limited by MDDefFormat definitions)

MetadataDefinitions() [4/4]

mspass::utility::MetadataDefinitions::MetadataDefinitions

(

const MetadataDefinitions &

parent

)

Standard copy constructor.

  : tmap(parent.tmap),cmap(parent.cmap),
  aliasmap(parent.aliasmap),
  alias_xref(parent.alias_xref),roset(parent.roset),
  unique_id_data(parent.unique_id_data)
{}

Member Function Documentation

add()

void mspass::utility::MetadataDefinitions::add	(	const std::string	key,
		const std::string	concept_,
		const MDtype	mdt
	)

Basic putter.

Use to add a new entry to the definitions. Note that because this is implemented with C++ map containers if data for the defined key is already present it will be silently replaced.

Parameters

key	is the key for indexing this attribute
concept_	is brief description saved as the concept for this key
type	defines the type to be defined for this key.

{
  cmap[key]=concept_;
  tmap[key]=mdt;
}

add_alias()

void mspass::utility::MetadataDefinitions::add_alias	(	const std::string	key,
		const std::string	aliasname
	)

Add an alias for key.

Parameters

key	is the main key for which an alias is to be defined
aliasname	is the the alternative name to define.

{
  /* We could use operator[] but this is more bombproof if not as clear */
  aliasmap.insert(std::pair<string,string>(key,aliasname));
  alias_xref.insert(std::pair<string,string>(aliasname,key));
}

apply_aliases()

std::list< string > mspass::utility::MetadataDefinitions::apply_aliases	(	mspass::utility::Metadata &	d,
		const std::list< std::string >	aliaslist
	)

Apply a set of aliases to data.

This method should be called in processing workflows to apply a series of defined aliases to data. The method uses the is_alias method to verify if an alias name is valid. It will not change the key if the name is not defined as a valid alias. The keys that fail that test will be posted to the std::list that is returned. Callers should test the size of the return and if it is not empty take appropriate action.

Parameters

d	is the data to alter (usually actually a TimeSeries of Seismogram)
aliaslist	is a list of aliases names to apply.

Returns

std::list of srings of failed changes. Callers should test the size of this return and take action if needed.

{
  list<string> failures;
  list<string>::const_iterator aptr;
  std::pair<string,MDtype> nmpair;
  for(aptr=aliaslist.begin();aptr!=aliaslist.end();++aptr)
  {
    if(this->is_alias(*aptr))
    {
      nmpair=this->unique_name(*aptr);
      string ukey=nmpair.first;
      /* Silently skip any aliases not defined.  That assures that if the
      name had already been changed to this alias it will be prserved. */
      if(d.is_defined(ukey))
      {
        d.change_key(ukey,*aptr);
      }
    }
    else
    {
      failures.push_back(*aptr);
    }
  }
  return failures;
}

References mspass::utility::Metadata::change_key(), is_alias(), mspass::utility::Metadata::is_defined(), and unique_name().

clear_aliases()

void mspass::utility::MetadataDefinitions::clear_aliases

(

mspass::utility::Metadata &

d

)

Restore any aliases to unique names.

Aliases are needed to support legacy packages, but can cause downstream problem if left intact. This method clears any aliases and sets them to the unique_name defined by this object.

Parameters

d	is data to be altered. Normally a Seismogram of TimeSeries but can be a raw Metadata object.

Returns

ErrorLogger containing any problems encountered. This function always returns and treats any problems as nonfatal errors and posts a log entry into this returned object. Caller should test the size of the return and handle or ignore errors as appropriate.

{
  std::set<string> keys;
  keys=d.keys();
  std::set<string>::iterator kptr;
  std::pair<string,MDtype> nmpair;
  for (kptr=keys.begin();kptr!=keys.end();++kptr)
  {
    if(this->is_alias(*kptr))
    {
      nmpair=this->unique_name(*kptr);
      string ukey=nmpair.first;
      d.change_key(*kptr,ukey);
    }
  }
}

References mspass::utility::Metadata::change_key(), is_alias(), mspass::utility::Metadata::keys(), and unique_name().

collection()

string mspass::utility::MetadataDefinitions::collection

(

const std::string

key

)

const

\Brief return the master collection (table) for a key used as a unique id.

Support for normalized Metadata requires static tables (collection in MongoDB) that contain the data using normalization. In seismic data type examples are receiver location tables, receiver response tables, and source location data. This method should nearly always be paired with a call to unique_id_key. The idea is to first ask for the unique_id_key and then ask what collection (table) contains the key returned by unique_id_key. This provides a fast and convenient lookup for normalized data.

Parameters

key	is the normally the return from unique_id_key

Returns

string defining the collection(table) the key can be used for locating the unique tuple/document required to access related Metadata. String will be empty if the search fails.

{
  map<string,tuple<string,string>>::const_iterator uidptr;
  uidptr=unique_id_data.find(key);
  if(uidptr==unique_id_data.end())
  {
    return (string(""));
  }
  else
  {
    return(get<0>(uidptr->second));
  }
}

concept()

std::string mspass::utility::MetadataDefinitions::concept

(

const std::string

key

)

const

Return a description of the concept this attribute defines.

Parameters

key	is the name that defines the attribute of interest

Returns

a string with a terse description of the concept this attribute defines.

{
  const string base_error("MetadataDefinitions::concept:  ");
  map<string,string>::const_iterator cptr;
  cptr=cmap.find(key);
  if(cptr==cmap.end())
  {
    stringstream ss;
    ss<<base_error<<"no match for key="<<key<<" found"<<endl;
    throw MsPASSError(ss.str(),ErrorSeverity::Invalid);
  }
  return cptr->second;
}

has_alias()

bool mspass::utility::MetadataDefinitions::has_alias

(

const std::string

key

)

const

Methods to handle aliases.

Sometimes it is helpful to have alias keys to define a common concept. For instance, if an attribute is loaded from a ralational db one might want to use alias names of the form table.attribute as an alias to attribute. has_alias should be called first to establish if a name has an alias. To get a list of aliases call the aliases method.

{
  multimap<string,string>::const_iterator aptr;
  aptr=aliasmap.find(key);
  if(aptr==aliasmap.end())
    return false;
  else
    return true;
}

is_alias()

bool mspass::utility::MetadataDefinitions::is_alias

(

const std::string

key

)

const

Ask if a key is a registered alias.

This asks the inverse question to has_alias. That is, it yields true of the key is registered as a valid alias. It returns false if the key is not defined at all. Note it will yield false if the key is a registered unique name and not an alias.

{
  map<string,string>::const_iterator mptr;
  mptr=alias_xref.find(key);
  if(mptr==alias_xref.end())
      return false;
  else
      return true;
}

is_defined()

bool mspass::utility::MetadataDefinitions::is_defined ( const std::string  key ) const

noexcept

Test if a key is defined either as a unique key or an alias

{
  /* test type map because concept can be empty for a key */
  map<string,MDtype>::const_iterator tptr;
  tptr=tmap.find(key);
  if(tptr!=tmap.end())
  {
    return true;
  }
  else
  {
    pair<string,MDtype> unr;
    /* A bit weird to catch an exception as a way to test for a false, but
    the way the api currently is defined requires this. */
    try{
      unr=this->unique_name(key);
      return true;
    }catch(MsPASSError& mderr)
    {
      return false;
    }
  }
}

is_normalized()

bool mspass::utility::MetadataDefinitions::is_normalized

(

const std::string

key

)

const

Test if a key:value pair is set as normalized.

In MongoDB a normalized attribute is one that has a master copy in one and only one place. This method returns true if an attribute is marked normalized and false otherwise (It will also return false for any key that is undefined.).

{
  if(unique_id_data.find(key)!=unique_id_data.end())
    return true;
  else
    return false;
}

normalize_data()

std::pair< std::string, std::string > mspass::utility::MetadataDefinitions::normalize_data

(

const std::string

key

)

const

Special method for efficiency.

For mspass using mongodb normalization for all currently supported Metadata can be reduced to a collection(table)-attribute name pair. The unique_id_key and collection methods can be called to obtained this information, but doing so requires a purely duplicate (internal map container) search. This convenience method is best used with MongoDB for efficiency.

Parameters

key	is the flat namespace key for which normalizing data is needed

Returns

an std::pair with of strings with first=collection and second=attribute name.

{
  map<string,tuple<string,string>>::const_iterator uidptr;
  uidptr=unique_id_data.find(key);
  if(uidptr==unique_id_data.end())
  {
    throw MsPASSError("MetadataDefinitions::normalize_data:  key="
      + key + " has no normalization data");
  }
  else
  {
    pair<string,string> result;
    result.first=get<0>(uidptr->second);
    result.second=get<1>(uidptr->second);
    return result;
  }
}

operator+=()

MetadataDefinitions & mspass::utility::MetadataDefinitions::operator+=

(

const MetadataDefinitions &

other

)

Accumulate additional definitions.

Appends data in other to current. The behavior or this operator is not really a pure + operation as one would think of it as with arithmetic. Because the model is that we are defining the properties of unique keys handling the case where other has a duplicate key to the instance of the left hand side is ambiguous. We choose the more intuitive case where the right hand side overrides the left. i.e. if other has duplicate data for a key the right hand side version replaces the left. There are two exceptions. Aliases are multivalued so they accumulate. i.e. this mechanism can be used to do little more than add an alias if the others data are the same. The second case is more trivial and rarely of importance. That is, other can have empty concept data for a key and it will be silently set empty. The reason is concept is purely for human readers and is not expected to ever be used by processors.

{
  if(this==&other) return *this;
  list<string> kvals=other.keys();
  list<string>::iterator kptr;
  for(kptr=kvals.begin();kptr!=kvals.end();++kptr)
  {
    MDtype mdt=other.type(*kptr);
    /* Note this will silently overwrite previous if the key was already present*/
    this->tmap[*kptr]=mdt;
    try{
      string cother=other.concept(*kptr);
      this->cmap[*kptr]=cother;
    }catch(MsPASSError& merr)
    {
      //Assume the only error here comes from concept methode failing
      cerr << "MetadataDefinitions operator+= (Warning): concept description is missing for key="
         <<*kptr<<endl<<"Error will be ignored"<<endl;
    }
    /* The alias_xref is a multimap so we just append other data - no such thing as duplicates*/
    map<string,string>::const_iterator aptr;
    for(aptr=other.alias_xref.begin();aptr!=other.alias_xref.end();++aptr)
    {
      this->alias_xref.insert(*aptr);
    }
    /* These behave like the type map above - we silently replace any entry
    that was present before.  i.e. other overrides */
    set<string>::const_iterator sptr;
    for(sptr=other.roset.begin();sptr!=roset.end();++sptr)
    {
      this->roset.insert(*sptr);
    }
    map<string,tuple<string,string>>::const_iterator uptr;
    for(uptr=other.unique_id_data.begin();uptr!=other.unique_id_data.end();++uptr)
    {
      this->unique_id_data.insert(*uptr);
    }
  }
  return *this;
}

References concept(), and type().

operator=()

MetadataDefinitions & mspass::utility::MetadataDefinitions::operator=

(

const MetadataDefinitions &

other

)

Standard assignment operator.

{
  if(this!=&parent)
  {
    tmap=parent.tmap;
    cmap=parent.cmap;
    aliasmap=parent.aliasmap;
    alias_xref=parent.alias_xref;
    roset=parent.roset;
    unique_id_data=parent.unique_id_data;
  }
  return *this;
}

readonly()

bool mspass::utility::MetadataDefinitions::readonly

(

const std::string

key

)

const

Check if a key:value pair is marked readonly. Inverted logic of similar writeable method.

Parameters

key	is key used to access the parameter to be tested.

Returns

true of the data linked to this keys IS marked readonly. (if the key is undefined this method silently returns true)

{
  return( ! (this->writeable(key)));
}

References writeable().

set_readonly()

void mspass::utility::MetadataDefinitions::set_readonly

(

const std::string

key

)

Lock a parameter to assure it will not be saved.

Parameters can be defined readonly. That is a standard feature of this class, but is normally expected to be set on construction of the object. There are sometimes reason to lock out a parameter to keep it from being saved in output. This method allows this. On the other hand, use this feature only if you fully understand the downstream implications or you may experience unintended consequences.

Parameters

key	is the key for the attribute with properties to be redefined.

{
  /* Silently return if key is already set as readonly */
  if(roset.find(key)==roset.end())
  {
    roset.insert(key);
  }
}

set_writeable()

void mspass::utility::MetadataDefinitions::set_writeable

(

const std::string

key

)

Force a key:value pair to be writeable.

Normally some parameters are marked readonly on construction to avoid corrupting the database with inconsistent data defined with a common key. (e.g. sta) This method overrides such definitions for any key so marked. It does nothing except a pointles search if the key hasn't been marked readonly previously. This method should be used with caution as it could have unintended side effects.

Parameters

key	is key for the attribute to be redefined.

{
  set<string>::const_iterator roptr;
  roptr=roset.find(key);
  /* Here we silently do nothing if the key is not in roset */
  if(roptr!=roset.end())
  {
    roset.erase(roptr);
  }
}

type()

mspass::utility::MDtype mspass::utility::MetadataDefinitions::type

(

const std::string

key

)

const

Get the type of an attribute.

Parameters

key	is the name that defines the attribute of interest

Returns

MDtype enum that can be used to establish the proper type.

{
  const string base_error("MetadataDefinitions::type:  ");
  map<std::string,mspass::utility::MDtype>::const_iterator tptr;
  if(this->is_alias(key))
    return this->unique_name(key).second;
  tptr=tmap.find(key);
  if(tptr==tmap.end())
  {
    stringstream ss;
    ss<<base_error<<"no match for key="<<key<<" found"<<endl;
    throw MsPASSError(ss.str(),ErrorSeverity::Invalid);
  }
  return tptr->second;
}

References is_alias(), and unique_name().

unique_id_key()

string mspass::utility::MetadataDefinitions::unique_id_key

(

const std::string

key

)

const

Returns a unique identifier for a normalized attribute.

In MongoDB a normalized attribute is one that has a master copy in one and only one place. This method returns a unique identifier, which defines a key that can be used to access the attribute used as an index to identify a unique location for an otherwise potentially ambiguous identifier (e.g. sta can be used in may contexts). The type of attribute to which the returned key is linked is expected to normally be acquired by am immediate call to the type method of this class using the return key. It is the callers responsibility to handle errors if the request for the type information fails. Note for MongoDB the most common (and recommended) type for the unique id is an Object_ID. The design of the API, however, should not preclude some other index or an index oriented toward a relational database. (e.g. chanid is an integer key with a one-to-one relation for channel data in the CSS3.0 schema.)

Some unique id specifications require a table/collection qualifier. See related collection method that is designed to handle that.

This method should normally be used only on read operations to select the correct entry for what could otherwise be a potentially ambiguous key.

Parameters

key	is the flat namespace key for which normalizing data is needed

Returns

name for unique id for requested key. Returns an empty string if the key is not defined as normalized. In multiple calls it is more efficient to test for a null return and handle such entries inline instead of a double search required if preceded by is_normalized.

{
  map<string,tuple<string,string>>::const_iterator uidptr;
  uidptr=unique_id_data.find(key);
 
  if(uidptr!=unique_id_data.end())
  {
    return(get<1>(uidptr->second));
  }
  else
  {
    return (string(""));
  }
}

unique_name()

std::pair< std::string, mspass::utility::MDtype > mspass::utility::MetadataDefinitions::unique_name

(

const std::string

aliasname

)

const

Get definitive name for an alias.

This method is used to ask the opposite question as aliases. The aliases method returns all acceptable alternatives to a definitive name defined as the key to get said list. This method asks what definitive key should be used to fetch an attribute and what it's type is. It does this by returning an std::pair with first being the key and second the type.

Parameters

aliasname

is the name of the alias for which we want the definitive key

Returns

std::pair with the definitive key as the first of the pair and the type in the second field.

{
    const string base_error("MetadataDefinitions::unique_name:  ");
    map<std::string,std::string>::const_iterator aptr;
    aptr=alias_xref.find(aliasname);
    if(aptr==alias_xref.end())
    {
      throw MsPASSError(base_error+"alias name="
          + aliasname + " is not defined",ErrorSeverity::Invalid);
    }
    else
    {
      /* We do not assume the key returned from alias_xref resolves. Small
      cost for a stability gain*/
      string kname=aptr->second;
      map<std::string,MDtype>::const_iterator tptr;
      tptr=tmap.find(kname);
      if(tptr==tmap.end()) throw MsPASSError(base_error+"alias name="+aliasname
        + " has no matching entry in alias tables\n"
        + "SETUP ERROR - FIX CONFIGURATION FILES",
        ErrorSeverity::Fatal);
      return(pair<std::string,mspass::utility::MDtype>(kname,tptr->second));
    }
}

writeable()

bool mspass::utility::MetadataDefinitions::writeable

(

const std::string

key

)

const

Check if a key:value pair is mutable(writeable). Inverted logic from similar readonly method.

Parameters

key	is key used to access the parameter to be tested.

Returns

true if the data linked to this not not marked readonly. (if the key is undefined a false is silently returned)

{
  set<string>::const_iterator roptr;
  roptr=roset.find(key);
  if(roptr==roset.end())
    return true;
  else if(this->is_alias(key))
  {
    /* roset only contains unique key entries.  This checks any possible
    aliases. */
    pair<string,MDtype> kp;
    /* unique_name method returns an exception if the key is not defined.
    We avoid that and silently return false if that happens, although
    that situation will likely create downstream problems. */
    try{
      kp=this->unique_name(key);
    }catch(MsPASSError &mderr)
    {
      cerr << "MetadataDefinitions::writeable method (WARNING):  Requested key "
     << key<<" is undefined and is not a registered alias"<<endl
         <<"This may cause downstream problems"<<endl;
      return false;
    }
    roptr=roset.find(kp.first);
    if(roptr==roset.end())
      return true;
    else
      return false;
  }
  else
  {
    /* A bit confusing layout here.  We land here if the
     * key was marked read only and was not an alias.  */
    return false;
  }
}

References is_alias(), and unique_name().

---

The documentation for this class was generated from the following files:

• /home/runner/work/mspass/mspass/cxx/include/mspass/utility/MetadataDefinitions.h
• /home/runner/work/mspass/mspass/cxx/src/lib/utility/MetadataDefinitions.cc