Fileloader (BurstChain®)¶

This is an add-on execise to the BurstChain® Hello World using the address chain.

Description¶

The fileloader inserts/updates (upserts) file based data into a chain asynchronously. The API will:

Updates an asset if the asset exist
Inserts a new asset if the asset does not exist

The Process (CSV file)¶

The API will upload the CSV file to the server. The server then processes the file row by row upserting data. Each row can also have a transform and/or mapping step to massage data into the format of the dictionary. Both steps are individually optional or they can be used in conjunction.

Note: The transform, if specified, is executed before the mapping step,

The Transform¶

A transform is a javascript function that allows for direct manipulation of the input data. The input of the function is the json representation of a row of data from the CSV file. You are free to add, remove, and change attributes in the json to transform the data.

Transform Boilerplate¶

function(doc) {
    //
    // your code here
    //
    return doc;
}

Mapping Document¶

A map is a simple json document that maps input columns to output attributes with an optional default value.

Fields In A Individual Mapping¶

Syntax	Description
column	The column in the CSV you want to map
attribute	The attribute in the dictionary you want to populate
defaultValue	The default value if the column does not exist or is empty

Mapping Document Boilerplate¶

{
    "mapping": [
       {
            "attribute": "attribute_name",
            "column": "column_name",
            "defaultValue": "default"
       },
       ...
    ]
}

Example¶

CSV file¶

Below is the data we want to load into our address chain.

id	address1	address2	addressOther	city	st	postal	country
1000	555 Main St	Suite 450		Denver	Colorado	55555	US
1001	1234 Sidestreet Way	PO BOX 555		Aurora	CO	55556	US
1002	4321 Trail Ln		c/o Mary Smith	Broomfield		55557	US

We can see several differences in the CSV file format compared to our dictionary.

Differences¶

the state for line id 1000 is the full state name
the state for line id 1002 is missing
the address2 for line id 1002 is missing but we want to use the data in addressOther to augment the data
the addressOther field is not in the dictionary
the country field is not in the dictionary
address1 –> addr1
address2 –> addr2
st –> state
postal –> zip

The Transform¶

This transform is going to handle the differences for #1, #3, #4, and #5.

Note: the field names in the json are based on the CSV and not the address dictionary.

function xform(doc) {
  //Diff #1
  //Replace 'Colorado' with 'CO'
  if(doc.st === "Colorado") {
    doc.st = "CO";
  }

  //Diff #3
  //Override address2 with addressOther if address2 is empty/null/undefined
  if(doc.addressOther && !doc.address2) {
    doc.address2 = doc.addressOther;
  }

  //Diff #4 & #5
  //Remove attributes that do not exist in dictionary
  //or update the dictionary to the use undefinedAttributesAction: KEEP or REMOVE.
  //By default the undefinedAttributesAction is ERROR
  delete doc.addressOther;
  delete doc.country;

  return doc;
}

The Mapping Document¶

This map is going to handle the differences for #2, #6, #7, #8, and #9.

{
    "mapping": [
        {
            "attribute": "addr1",
            "column": "address1"
        },
        {
            "attribute": "addr2",
            "column": "address2"
        },
        {
            "attribute": "state",
            "column": "st",
            "defaultValue": "CO"
        },
        {
            "attribute": "zip",
            "column": "postal"
        }
    ]
}

Execution¶

Example execution of the fileloader script using cURL.

Loading¶

Command:

curl -X POST \
  -H 'authorization: ID <your-private-id>' \
  -F "file=@<csv-file>" \
  -F "map=@<map-file>" \
  -F "xform=@<transform-file" \
  -F 'email_results=<email-address>' \
  https://{ASSIGNED_SERVER}.burstiq.com/api/burstchain/<your-client>/address/fileloader/job

Output:

{
  "description": "BurstChain File Loader f060882e-7031-413a-a8c0-b991ad198f1f:address",
  "job_id": {
    "$uuid": "6d0b0f91-fd99-412a-b9cb-640c30c09761"
  },
  "job_type": "SERIAL",
  "job_state": "RUNNING",
  "start": {
    "$date": "2020-03-23T22:15:59.946Z"
  },
  "timestamp": {
    "$date": "2020-03-23T22:15:59.946Z"
  }
}

Query¶

Command:

curl -X POST \
  -H 'authorization: ID <your-private-id>' \
  -H 'content-type: application/json' \
  -d '{ \
    "queryTqlFlow" : "WHERE asset.id IN ('\''1000'\'','\''1001'\'','\''1002'\'')" \
  }' \
  https://{ASSIGNED_SERVER}.burstiq.com/api/burstchain/<your-client>/address/query

Output:

{
  "message": "The operation was successful",
  "status": 200,
  "timestamp": {
    "$date": "2020-03-23T22:16:50.821Z"
  },
  "assets": [
    {
      "hash": "912fc9d2bdb2a24c6880d4533724400ce569b186f8412b06569cdb6901daa291",
      "timestamp": {
        "$date": "2020-03-23T22:16:00.188Z"
      },
      "type": "asset",
      "operation": "create",
      "owners": [
        "d065c04ab9197515aeecabf18bb40ba99b1debf0"
      ],
      "signer": "d065c04ab9197515aeecabf18bb40ba99b1debf0",
      "signature": "08b95be24a668326cb694e0884b771daeff5dae3a69a0006052571d336813edc3d5b88aee15cd9a267b9c981350e68786480515f1cd270e494cb7abd859cb46c5eda4d419dd848a822ae7c60aed3776809fc08bfbbdf80969f2a2f94ac83f651bb1f7f987b5a70c56626d373a3ed56a498c03506164d2f21e5f8771adc78b237febd6b017e302176aee9b8f503b0de0a957c2ce98cb94445a49a37ee5bcf0c651ed49e33c2b8f1308a75e339f91770f4c3656a713aa2deaab3e7cf1c847dc451501ed479de2123690e304720b29742c1ecd967f3a5ceb84289383f855a5003b2a69cae7d4e339c03c738d960ee1345fe66ac0ecf9dbbe297fab1c303197dfe26",
      "dictionary": "address",
      "asset": {
        "city": "Denver",
        "id": "1000",
        "addr1": "555 Main St",
        "addr2": "Suite 450",
        "state": "CO",
        "zip": "55555"
      },
      "digest_alg": "biq2",
      "previous_hash": "c3785be07b136ebb5215ab92a85f6426fcce26d4ef0d63d8c6055f4fe7844b33",
      "asset_id": "bced6cf5-7a82-409e-8cf1-1f528f6b07c0",
      "asset_metadata": {
        "file": "addresses.csv",
        "load_datetime": {
          "$date": "2020-03-23T22:15:59.334Z"
        }
      }
    },
    {
      "hash": "c3785be07b136ebb5215ab92a85f6426fcce26d4ef0d63d8c6055f4fe7844b33",
      "timestamp": {
        "$date": "2020-03-23T22:16:00.175Z"
      },
      "type": "asset",
      "operation": "create",
      "owners": [
        "d065c04ab9197515aeecabf18bb40ba99b1debf0"
      ],
      "signer": "d065c04ab9197515aeecabf18bb40ba99b1debf0",
      "signature": "1e0e9456bfd221a5745a6abcc15cb86ee74ea77bfd8d9b38ae5a2e587f4e5c2646dfbfb28239623695fc7206fadcbfc30e1e056a54d4860bcb2d03210a6e78fd9860688d4fb4d78b114fb4663e0ae664f74546f0b7a4ee02e686f8d02da706547775b28704d55c8d173f7969c73baeef7ede42e7b71a2e9860edea3be12ee26a5d1060d9651cce78e66eb175460aa874799642046588b9cc5177ae8b5ef3cb3e064901a5b72d09b10659c8b786ad2bd20d80d7a826b264993c6e845ee7dfad12a5a25504d396aa156fca80a52fd3ab97b07238f462e4ed417a99af015b88b7ccaa8cbdd1a4be6ae0d69f276fe6b125b1df364f8eb3db2b6775ef4f6e74eefb55",
      "dictionary": "address",
      "asset": {
        "city": "Aurora",
        "id": "1001",
        "addr1": "1234 Sidestreet Way",
        "addr2": "PO BOX 555",
        "state": "CO",
        "zip": "55556"
      },
      "digest_alg": "biq2",
      "previous_hash": "3addbb6c1d30ae9d83cc968e3c8995e3d9575b54d4bd91d7b8338163d0ddabd0",
      "asset_id": "97166f7d-6a87-4f12-be2f-38a8d1d55f21",
      "asset_metadata": {
        "file": "addresses.csv",
        "load_datetime": {
          "$date": "2020-03-23T22:15:59.334Z"
        }
      }
    },
    {
      "hash": "3addbb6c1d30ae9d83cc968e3c8995e3d9575b54d4bd91d7b8338163d0ddabd0",
      "timestamp": {
        "$date": "2020-03-23T22:16:00.186Z"
      },
      "type": "asset",
      "operation": "create",
      "owners": [
        "d065c04ab9197515aeecabf18bb40ba99b1debf0"
      ],
      "signer": "d065c04ab9197515aeecabf18bb40ba99b1debf0",
      "signature": "5a6867057993a57fd2f9fc7dc0a0a69c641c377cc8720c3889bd6f82e1e0f30d6741be4b336800851c1ec611e96906ebde3259cc244677cebd19b9cf890bf98d762bc89ac3529f2200462fb0c879e109106dda45df73e04d4fa4ab81a837de9a92607a7c329654723ae30c7eae0e70756ae0d499b623c10e6a72153dfcfe19676d8d13a91aa8a648758f548cce1395cf1d4da743948ec3f8fdb37abac5e24e1037b7385c4d31401c40cdf9a044f12be6ed523793b5d045f6eb62098378586856e7ec5a5fe11c91cac8d2437102f32b297d713de8d6408736d9025ec598e4de034450c062fb566e1d10fb4bcd0d1505ec324bec8ff1e5457a61355408c4fec014",
      "dictionary": "address",
      "asset": {
        "city": "Broomfield",
        "id": "1002",
        "addr1": "4321 Trail Ln",
        "addr2": "c/o Mary Smith",
        "state": "CO",
        "zip": "55557"
      },
      "digest_alg": "biq2",
      "previous_hash": "ca1ee9a2de0f4bca4c22369accab48fb8d69741a5b5efbf087174f200b3c8ee6",
      "asset_id": "951c7d8a-eec1-4131-8252-95aeecbad1cc",
      "asset_metadata": {
        "file": "addresses.csv",
        "load_datetime": {
          "$date": "2020-03-23T22:15:59.334Z"
        }
      }
    }
  ]
}