Importing assets from GIT

Overview

AssetsAPI allows you to manage your assets and asset definitions directly from a git repository and a YAML file rather than using database imports. The YAML schema file is parsed and used to generate documentation and web service definitions. This approach can make data management easier in some cases usually when a repository doesn't get too many modifications.

Schema files support all the normal things you would write when defining database schema (relationships, attributes, indexes, etc.).

YAML Syntax

AssetsAPI offers the ability to specify the schema in an abbreviated and verbose syntax. A lot of the schema parameters have values they default to, this allows us to abbreviate the syntax and let just use its defaults. Below is an example of schema taking advantage of all the abbreviations.


Abbreviated Syntax

The abbreviated/shortened syntax is the fastest way to define the asset structure. However, it lacks the option to enter comments for fields. These must be entered in the repository administration panel.

schema.yaml
version: "1.0"

detect_relations: true

User:
  comment: "This asset contains information about registered users"
  columns:
    id: integer
    username: string
    password: string
    contact_id: integer
Contact:
  comment: "This asset contains information about a user's contact info"
  columns:
    id: integer
    first_name: string
    last_name: string
    phone: string
    email: string
    address: string

The detect_relations option will attempt to guess relationships based on column names. In the example, above AssetsAPI knows that the User has one Contact and will automatically define the relationship between the assets.


Verbose Syntax

In the below example we do not define the detect_relations option, instead, we manually define the relationships so we have complete control over the configuration of the local/foreign key, type and alias of the relationship on each side.

schema.yaml
version: "1.0"

User:
  comment: "This asset contains information about registered users"
  columns:
    id:
      type: integer
      comment: "Contact identifier"
      primary: true
      autoincrement: true
    username:
      type: string(255)
      comment: "Identification used by a person with access to a service"
    password:
      type: string(255)
      comment: "Encrypted user password"
    contact_id:
      type: integer
      comment: "Id of the contact"
  relations:
    Contact:
      asset: Contact
      local: contact_id
      foreign: id
      type: one

Contact:
  comment: "This asset contains information about a user's contact info"
  columns:
    id:
      type: integer
      comment: "Contact identifier"
      primary: true
      autoincrement: true
    first_name:
      type: string(255)
      comment: "Contact first name"
    last_name:
      type: string(255)
      comment: "Contact last name"
    phone:
      type: string(255)
      comment: "Phone number including area code"
    email:
      type: string(255)
      comment: "Contact email address"
    address:
      type: string(255)
      comment: "Contact address with postcode"
  relations:
    User:
      asset: User
      local: id
      foreign: contact_id
      type: one
Relationships

When specifying relationships it is only necessary to specify the relationship on the end where the foreign key exists.

AssetsAPI offers the ability to specify a detect_relations option as you saw earlier. This feature provides automatic relationship building based on column names. If you have a User asset with a contact_id and an asset with the name Contact exists, it will automatically create the relationships between the two.


One to One
schema.yaml
version: "1.0"

User:
  columns:
    id:
      type: integer(4)
      primary: true
      autoincrement: true
      comment: "Unique user identifier"
    contact_id:
      type: integer(4)
      comment: "Id of the contact"
    username:
      type: string(255)
      comment: "Identification used by a person with access to a service"
    password:
      type: string(255)
      comment: "Encrypted user password"
  relations:
    Contact:
      type: one

Contact:
  columns:
    id:
      type: integer(4)
      primary: true
      autoincrement: true
      comment: "Unique contact identifier"
    name:
      type: string(255)
      comment: "Contact first and last name"


One to Many
schema.yaml
version: "1.0"

User:
  columns:
    id:
      type: integer(4)
      primary: true
      autoincrement: true
      comment: "Unique user identifier"
    contact_id:
      type: integer(4)
      comment: "Id of the contact"
    username:
      type: string(255)
      comment: "Identification used by a person with access to a service"
    password:
      type: string(255)
      comment: "Encrypted user password"

Phonenumber:
  columns:
    id:
      type: integer(4)
      primary: true
      autoincrement: true
      comment: "Unique contact identifier"
    name:
      type: string(255)
      comment: "Contact first and last name"
    user_id:
      type: integer(4)
      comment: "id of the user"
  relations:
    User:
      type: many
File structure

By default GIT repository data assets are expected to be located inside the assets directory. For each asset defined in the schema.yaml file, there should be a file in the assets directory of the same name with a csv extension. The first row in the data file should contain column names that match the definitions in schema.yaml

You can customize the base asset directory and the individual asset paths:

schema.yaml
version: 1.0

directory: data # Instead of "assets" we use "data" as the base directory

User:
    # This asset is located in data/User.csv

Contact:
    # This asset is located in custom/Contact.csv.
    path: custom/Contact.csv

If your assets are not located in a subdirectory, you can use "/":

schema.yaml
version: 1.0

directory: / # All assets are expected to live right in the repository root.
Example repository

Below is an example Countries asset containing ISO country codes and other relevant country information.


Directory tree

Directory and file structure.

  • .git
  • .assetsapi
    • schema.yaml
  • assets
    • Countries.csv
  • Readme.md

Schema file structure

schema.yaml
version: 1.0

Countries:
  comment: >
    The asset contains the three sets of ISO 3166-1 country codes for
    each of its 249 countries, ISO 3166-2 country subdivision codes,
    and the Internet country code top-level domains (ccTLD) which are
    based on the ISO 3166-1 alpha-2 standard (with the few exceptions).
  columns:
    country_name: string(56)
    official_state_name: string(255)
    sovereignty: string(100)
    iso_2_code: char(2)
    iso_3_code: char(3)
    numeric_code: integer
    subdivision_code_links: string
    internet_cctld: string(20)

Countries Asset file structure

Countries.csv file is located in assets/Countries.csv. Example csv content below.

Countries.csv
country_name,official_state_name,sovereignty,iso_2_code,iso_3_code,numeric_code,subdivision_code_links,internet_cctld
Afghanistan,The Islamic Republic of Afghanistan,UN member state,AF,AFG,4,ISO 3166-2:AF,.af
Åland Islands,Åland,Finland,AX,ALA,248,ISO 3166-2:AX,.ax
Albania,The Republic of Albania,UN member state,AL,ALB,8,ISO 3166-2:AL,.al
Algeria,The People's Democratic Republic of Algeria,UN member state,DZ,DZA,12,ISO 3166-2:DZ,.dz
American Samoa,The Territory of American Samoa,United States,AS,ASM,16,ISO 3166-2:AS,.as
...
 
Available Column Types

AssetsAPI offers a variety of types of columns you can use in your schema.yaml file. Each of the available types is listed below:

char

The char creates a CHAR equivalent column with of a given length

schema.yaml
Products:
  columns:
    flag:
      type: char(1)
       

date

The date creates a DATE equivalent column

schema.yaml
Users:
  columns:
    created_at:
      type: date
       

datetime

The dateTime creates a DATETIME equivalent column

schema.yaml
Users:
  columns:
    created_at:
      type: datetime
       

decimal

The decimal creates a DECIMAL equivalent column with the given precision (total digits) and scale (decimal digits)

schema.yaml
Products:
  columns:
    amount:
      type: decimal(8,2)
       

double

The double method creates a DOUBLE equivalent column with the given precision (total digits) and scale (decimal digits)

schema.yaml
Products:
  columns:
    amount:
      type: double(8,2)
       

enum

The enum creates a ENUM equivalent column

schema.yaml
Users:
  columns:
    account_type:
      type: enum
       

float

The float creates a FLOAT equivalent column with the given precision (total digits) and scale (decimal digits)

schema.yaml
Products:
  columns:
    amount:
      type: float(8,2)
       

integer

The integer creates an INTEGER equivalent column

schema.yaml
Posts:
  columns:
    views_count:
      type: integer
       

longtext

The longtext creates a LONGTEXTTEXT equivalent column

schema.yaml
Posts:
  columns:
    description:
      type: longtext
       

point

The point creates a POINT equivalent column

schema.yaml
AssetName:
  columns:
    position:
      type: point
    city_name:
      type: string(255)
       
assets/AssetName.csv
position,city_name
51.5073219 -0.1276474,London UK
        

set

The set creates a SET equivalent column with the given list of valid values

schema.yaml
Donuts:
  columns:
    flavors:
      type: set
       

string

The string creates a VARCHAR equivalent column of the given length

schema.yaml
Users:
  columns:
    username:
      type: string(255)
       

text

The text creates a TEXT equivalent column

schema.yaml
Posts:
  columns:
    description:
      type: text