Skip to content

Latest commit

 

History

History
132 lines (112 loc) · 4.48 KB

bulk-operations.md

File metadata and controls

132 lines (112 loc) · 4.48 KB

Bulk Operations

CSV Data Load

Ingests CSV data, provided directly in the operation as an insert, update or upsert into the specified database table.

  • operation (required) - must always be csv_data_load
  • action (optional) - type of action you want to perform - insert, update or upsert. The default is insert
  • database (optional) - name of the database where you are loading your data. The default is data
  • table (required) - name of the table where you are loading your data
  • data (required) - csv data to import into HarperDB

Body

{
  "operation": "csv_data_load",
  "database": "dev",
  "action": "insert",
  "table": "breed",
  "data": "id,name,section,country,image\n1,ENGLISH POINTER,British and Irish Pointers and Setters,GREAT BRITAIN,http://www.fci.be/Nomenclature/Illustrations/001g07.jpg\n2,ENGLISH SETTER,British and Irish Pointers and Setters,GREAT BRITAIN,http://www.fci.be/Nomenclature/Illustrations/002g07.jpg\n3,KERRY BLUE TERRIER,Large and medium sized Terriers,IRELAND,\n"
}

Response: 200

  {
      "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69",
      "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69"
  }

CSV File Load

Ingests CSV data, provided via a path on the local filesystem, as an insert, update or upsert into the specified database table.

Note: The CSV file must reside on the same machine on which HarperDB is running. For example, the path to a CSV on your computer will produce an error if your HarperDB instance is a cloud instance.

  • operation (required) - must always be csv_file_load
  • action (optional) - type of action you want to perform - insert, update or upsert. The default is insert
  • database (optional) - name of the database where you are loading your data. The default is data
  • table (required) - name of the table where you are loading your data
  • file_path (required) - path to the csv file on the host running harperdb

Body

{
  "operation": "csv_file_load",
  "action": "insert",
  "database": "dev",
  "table": "breed",
  "file_path": "/home/user/imports/breeds.csv"
}

Response: 200

{
  "message": "Starting job with id 3994d8e2-ec6a-43c4-8563-11c1df81870e",
  "job_id": "3994d8e2-ec6a-43c4-8563-11c1df81870e"
}

CSV URL Load

Ingests CSV data, provided via URL, as an insert, update or upsert into the specified database table.

  • operation (required) - must always be csv_url_load
  • action (optional) - type of action you want to perform - insert, update or upsert. The default is insert
  • database (optional) - name of the database where you are loading your data. The default is data
  • table (required) - name of the table where you are loading your data
  • csv_url (required) - URL to the csv

Body

{
  "operation": "csv_url_load",
  "action": "insert",
  "database": "dev",
  "table": "breed",
  "csv_url": "https://s3.amazonaws.com/complimentarydata/breeds.csv"
}

Response: 200

{
  "message": "Starting job with id 332aa0a2-6833-46cd-88a6-ae375920436a",
  "job_id": "332aa0a2-6833-46cd-88a6-ae375920436a"
}

Import from S3

This operation allows users to import CSV or JSON files from an AWS S3 bucket as an insert, update or upsert.

  • operation (required) - must always be import_from_s3
  • action (optional) - type of action you want to perform - insert, update or upsert. The default is insert
  • database (optional) - name of the database where you are loading your data. The default is data
  • table (required) - name of the table where you are loading your data
  • s3 (required) - object containing required AWS S3 bucket info for operation:
    • aws_access_key_id - AWS access key for authenticating into your S3 bucket
    • aws_secret_access_key - AWS secret for authenticating into your S3 bucket
    • bucket - AWS S3 bucket to import from
    • key - the name of the file to import - the file must include a valid file extension ('.csv' or '.json')
    • region - the region of the bucket

Body

{
  "operation": "import_from_s3",
  "action": "insert",
  "database": "dev",
  "table": "dog",
  "s3": {
    "aws_access_key_id": "YOUR_KEY",
    "aws_secret_access_key": "YOUR_SECRET_KEY",
    "bucket": "BUCKET_NAME",
    "key": "OBJECT_NAME",
    "region": "BUCKET_REGION"
  }
}

Response: 200

{
  "message": "Starting job with id 062a1892-6a0a-4282-9791-0f4c93b12e16",
  "job_id": "062a1892-6a0a-4282-9791-0f4c93b12e16"
}