Skip to content

SAS

Using Basic SAS Access BurstIQ APIs

With SAS® 9.4M4 or later, there are web-based methods to exchange data between applications or systems using web technology like HTTP and machine-readable file formats like XML and JSON. Representational State Transfer (REST) is the most popular architecture used to access and implement web services. Web services using the REST architecture are called RESTful web services.

In this paper, we provide the basic steps to connect SAS to the BurstIQ REST Application Programming Interfaces (APIs) on the BurstIQ platform. The document includes SAS code that you can adapt and use immediately with your own code.

GETTING STARTED

Prerequisites:

  1. Current account with access to blockchain-based data sets with a public and private ID.

    a. Make sure you have been granted a BurstIQ account. If you’re working with a larger team, including someone who manages your BurstIQ access, you might need another person with the proper access to create the project for you and invite you as a member.

    b. To access BurstIQ site, you’ll need the unique Private ID and your Public ID that were assigned when you first logged in. These IDs are equivalent to a username and password necessary for “basic authentication.” The Private ID is short (16 characters) and the Public ID is much longer (around 40 characters). If you can’t remember your Research Foundry Public ID, you can find it in: <https://researchfoundry.burstiq.com/usersettings.

  2. Basic working knowledge of SAS and are comfortable scripting in Basic SAS.

  3. Up to date version of SAS installed on your computer or server [Maintenance 4 of SAS 9.4 (SAS® 9.4M4) or newer]. If you are unsure of your version of SAS, check the log upon starting SAS or run the command: proc setinit; run; and check the output to the log.

  4. Ability to connect SAS to a secure socket layer (SSL) connection to a secure internet connection (https).

If unsure about secure internet capabilities, the following SAS code offers a simple plumbing test. It uses a free HTTP test service (httpbin.org) to verify your Internet connectivity from SAS and your ability to use SSL. The endpoint returns a JSON-formatted collection of timestamps in various formats, which the program parses using the JSON library engine.

/* PROC HTTP and JSON libname test */
/* utility macro to echo the JSON out */

%macro echoFile(fn=);
  data _null_;
    infile &fn end=_eof;
    input;
    put _infile_;
  run;
%mend;

filename resp "%sysfunc(getoption(WORK))/now.json";
proc http
  url="https://httpbin.org/"
  method="GET"
  out=resp;
run;

%put HTTP Status code = &SYS_PROCHTTP_STATUS_CODE. :
&SYS_PROCHTTP_STATUS_PHRASE.;

%echoFile(fn=resp);

/* Tell SAS to parse the JSON response */
libname time JSON fileref=resp;

title "JSON library structure";
proc datasets lib=time;
quit**;

If you can run this program successfully from your SAS session, then you’re ready to attempt the more complex REST API calls.

If you encounter any errors while running this simple test, then you will need to resolve these issues before attempting to connect to the BurstIQ APIs. As an example, many companies route their internet traffic through proxy servers. This is usually handled behind the scenes for web browsers, but scripting tools require more explicit configuration. If PROC HTTP causes an error when you test this program, you might need to specify the PROXYHOST options, the PROXYPORT options, or both to tell SAS how to direct the traffic. If you don’t know these values, ask your corporate network administrator for help.

CONNECTING TO THE BURSTIQ APIS

  1. Set up the REST body with parameters of the data set
/*
Set up the REST request body document with query
*/
filename input TEMP;data _null_;
   file input;
   put '{"queryTqlFlow":"WHERE asset.recovered 50000 ORDER BY
asset.country, asset.province, asset.admin2, asset.date"}';
run;
  1. Perform an API call requesting a JSON application

In this style of REST API, all details of the request are encoded as parameters on the URL value. This makes for very long URL strings that include the path, client information, data set and other API directives.

The SAS code immediately below references this complex URL: <https://researchfoundry.burstiq.com/api/burstchain/8e2ddf65-7f19-457c-9817-59ba9c5d4b36/csse_covid_19_daily_reports/assets/interrogate. This URL is comprised of the following:

  • https://researchfoundry.burstiq.com/api/burstchain/ is the REST endpoint for the sample query to Research Foundry. Identify and substitute your own endpoint.

  • 8e2ddf65-7f19-457c-9817-59ba9c5d4b36 is the Client ID for the BurstIQ “demo client ID.” Identify and substitute your own Client ID.

  • csse_covid_19_daily_reports is the chain name for this particular data set. Each data set has its own name. Identify and substitute the desired data set name.

  • Assets is the Research Foundry folder where data are stored. Identify and substitute the desired folder location.

  • Interrogate is an API directive.

The following code also specifies that a JavaScript Object Notation (JSON) application will be used as the format for data exchange. JSON is simply a text representation of data: a collection of name-value pairs in a hierarchical structure. The schema of that structure is defined by the application. While it’s possible to use basic SAS functions to parse JSON, the JSON library engine makes the task much easier. The JSON library engine dissects the data in the file into several tables. If you’re just getting started with the JSON structure from a new API, it’s a good idea to explore these tables to learn which tables contain the values you need, and how the different tables relate to each other. By default, the JSON engine creates highly normalized tables and generates ordinal keys that allow you to join the rows together for analysis and reporting.

All of this information was saved in a filename resp.

/*
 * Call the REST endpoint using private id authentication
 * - Pass in the request body
 * - Set the URL and verb
 * - Set headers:
 *    - Content and accept is json
 *    - Authentication is using a private id
 *    - Store the response in resp
*/
filename resp TEMP;
proc http
   url="<https://researchfoundry.burstiq.com/api/burstchain/8e2ddf65-7f19-457c-9817-59ba9c5d4b36/csse_covid_19_daily_reports/assets/interrogate"

/* Substitute each part of this URL with the desired path and commands */
   in=input /* request body */
   out=resp /* where to store the response */
   method="POST"; /* REST verb */
   debug level=1;
headers
   "Content-Type"="application/json" /* request body content type */
   "Accept"="application/json" /* content type to accept */
   "Authorization"="ID abdf1f03c08309ab"; /* Substitute your own private id authentication. No need for basic or oauth2 authentication*/
run;
  1. Map the REST response to a SAS dataset

This step is very specific to the nature of the data and the data dictionary specified for the data. In the following example:

  • Records is where the REST response where data will be stored.
  • /root/records is the TABLEPATH.
  • /root/records/asset/ is the PATH to the individual data attributes for this example. The list of possible attributes is documented in the Research Foundry data dictionary definition for the Research Foundry dataset example.

  • Each variable name must be known and mapped individually, but once the code is written, it can be reused to quickly re-access the data set.

/*
 * Map the REST response to a SAS dataset
 *
 * The REST response store data in list attributes called 'records'. The TABLEPATH to this is '/root/records'.
 * The PATH to the individual data attributes would be '/root/records/asset/\<attribute_name\'.
 * The possible attributes for this Research Foundry example is documented in the dictionary definition for the Research Foundry dataset.
 */
filename map TEMP;data _null_;
   file map;
   put '{';
   put ' "DATASETS": [';
   put ' {';
   put ' "DSNAME": "records",';
   put ' "TABLEPATH": "/root/records",';
   put ' "VARIABLES": [';
   put ' {';
   put ' "NAME": "asset_id",';
   put ' "TYPE": "CHARACTER",';
   put ' "PATH": "/root/records/asset_id"';
   put ' },';
   put ' {';
   put ' "NAME": "date",';
   put ' "TYPE": "CHARACTER",';
   put ' "PATH": "/root/records/asset/date/\$date"';
   put ' },';
   put ' {';
   put ' "NAME": "province",';
   put ' "TYPE": "CHARACTER",';
   put ' "PATH": "/root/records/asset/province"';
   put ' },';
   put ' {';
   put ' "NAME": "country",';
   put ' "TYPE": "CHARACTER",';
   put ' "PATH": "/root/records/asset/country"';
   put ' },';
   put ' {';
   put ' "NAME": "updated",';
   put ' "TYPE": "CHARACTER",';
   put ' "PATH": "/root/records/asset/updated/\$date"';
   put ' },';
   put ' {';
   put ' "NAME": "active",';
   put ' "TYPE": "NUMERIC",';
   put ' "PATH": "/root/records/asset/active"';
   put ' },';
   put ' {';
   put ' "NAME": "confirmed",';
   put ' "TYPE": "NUMERIC",';
   put ' "PATH": "/root/records/asset/confirmed"';
   put ' },';
   put ' {';
   put ' "NAME": "deaths",';
   put ' "TYPE": "NUMERIC",';
   put ' "PATH": "/root/records/asset/deaths"';
   put ' },';
   put ' {';
   put ' "NAME": "recovered",';
   put ' "TYPE": "NUMERIC",';
   put ' "PATH": "/root/records/asset/recovered"';
   put ' },';
   put ' {';
   put ' "NAME": "admin2",';
   put ' "TYPE": "CHARACTER",';
   put ' "PATH": "/root/records/asset/admin2"';
   put ' }';
   put ' \]';
   put ' }';
   put ' \]';
   put '}';
run;
  1. Parse the resp file using the LIBNAME engine

With the LIBNAME engine, it is possible to read data from JSON like it is a data set. This creates a library with several tables. Because the content is different for each JSON file, the libname engine always creates an ALLDATA and ROOT and root table. Other tables are created based on the data in the JSON file.

For this example, the library was named rfdata and refers to the previously-named filename resp, but these should be renamed as desired for your data set.

/*
 * Parse the resp file using the map defined above
 */
libname rfdata JSON map=map fileref=resp;
  1. Finalize the data set creation

This final step provides a data set title from the data set name in the URL. In the following example:

  • Assets with recovered \50000 from csse_covid_19_daily_reports is the new title for the data set.
  • rfdata is the library name.
/*
 * Print the parsed dataset
 */
title "Assets with recovered 50000 from csse_covid_19_daily_reports";
proc print data=rfdata.records;
run;

TROUBLESHOOTING

Examine whether the cause is almost always one of the following:

  • Not using a recent enough version of SAS. PROC HTTP was revised and improved in SAS 9.4 Maint 3.  The JSON library engine was added in SAS 9.4 Maint 4 (released in 2016). It’s time to upgrade!

  • Not been granted access to the desired data set. If you not actually been granted access to the data set or the access has expired, you will not be able to access data.

  • Cannot access the web service through a corporate firewall. Use the various PROXY-related options to tell PROC HTTP how to connect through your network gateway, if necessary.

  • No SSL support installed/configured (required for HTTPS access). This is rarely an issue on Windows systems, but on UNIX systems, an admin must configure SSL certificates with the SSLCALISTLOC= option. If you’re using SAS University Edition, you’ll need the update from December 2017 or later where SSL was added. The editions prior to that did not include the SSL support.

REFERENCES

Hemedinger, Chris. 2015. “How SAS Uses SAS to Analyze SAS Blogs.” Proceedings of the SAS Global Forum 2015 Conference. Cary, NC: SAS Institute Inc. Available at [https://support.sas.com/resources/papers/proceedings15/SAS1708-2015.pdf]{.ul}.

Henry, Joseph. 2017. “Show Off Your OAuth.” Proceedings of the SAS Global Forum 2017 Conference. Cary, NC: SAS Institute Inc. Available at [http://support.sas.com/resources/papers/proceedings17/SAS0224-2017.pdf]{.ul}.

Hemedinger, Chris. “How to test PROC HTTP and the JSON library engine.” The SAS Dummy blog. [<https://blogs.sas.com/content/sasdummy/check-json-and-http/. Last modified January 23, 2018.]{.ul}

Hemedinger, Chris. “How to secure your REST API credentials in SAS programs.” The SAS Dummy blog. [<https://blogs.sas.com/content/sasdummy/hide-rest-api-tokens/. Last modified January 16, 2018.]{.ul}

ACKNOWLEDGMENTS

This document was modeled after a SAS White Paper written by Chris Hemedinger to connect SAS to Google Analytics.

We are grateful to Mike Reimer for his kind assistance in helping the BurstIQ staff test and improve this example code.

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

TERMINOLOGY

API

An Application Programming Interface, or API, is an interface between two or more applications. The API is a set of rules that allow multiple applications to communicate with each other. This can be as simple as returning data from a database, but also perform complex calculations and return the results. The application is only allowed to connect to endpoints for posting or reading data, making it a secure method to allow to applications to interoperate.

Authentication

Authentication is the process of identifying the client who is doing a request. HTTP supports multiple authentication schema such as anonymous authentication and basic authentication.

In basic authentication passwords are encoded but not encrypted and not considered secure. This might be enough f or internal applications, in combination with HTTPS, but very few public APIs will use on basic authentication. They will use the anonymous schema and use on authentication at the application level.

Data

The data, also called the body, message or payload, contains information that is send to the server. Data is only valid when using the POST, PUT, PATCH or DELETE methods.

Endpoint

The endpoint is the internet address where the web service can be accessed. It is Uniform Address Location (URL) and typically has the following format.

https://api.example.url/users/memberships?type=free&sort=lastname

In the above example the root-endpoint is https://api.example.url and /users/memberships are the path to a specific web service. The final part of the endpoint, ?filter=free&sort=lastname, is optional. This the query string and can be used to add parameters to the web service.

Header tells the server that the server can expect data in the JSON format (Content-Type).

"Content-Type: application/json"

HTTP(S)

HTTP stands for HyperText Transfer Protocol and is a client server protocol that it the foundation of any data exchange on the web. Web Services also rely on HTTP to exchange data between the client and server . HTTP send information in plain text and is not secure. HTTPS is the secure variant that encrypts data in transit. SAS supports HTTP, making it possible to access web-based data using SAS.

HTTP Status Codes

The status codes are part of the HTTP protocol and can be used to determine quickly if a request has been completed successfully or failed and why. The status codes are grouped in five classes:

  • Informational responses (100–199)
  • Successful responses (200–299)
  • Redirects (300–399)
  • Client errors (400–499)
  • Server errors (500–599)

Most status codes are defined in the HTTP/1.1 standard (RFC 7231), but servers can return non-standard code. If the code is not standard, the client should be able to determine the type of errors by the class.

JSON

JSON is an open-standard file format or data interchange format that uses human-readable text to transmit or store data objects consisting of attribute–value pairs and array data types. It is lightweight and the most common data format used by REST web services. The introduction of REST was paired to the rise of JavaScript Object Notation (JSON) as format for data exchange.

Method

The method defines the type of request sent to a web server. It indicates the action to perform for the requested resource. Possible actions are Create, Read, Update or Delete (CRUD). The method that supports those actions are GET, POST, PUT, PATCH and DELETE. Methods are case sensitive and should always be upper cased.

Method Type of action
GET Read a representation of a resource. The web service looks for data and sends the results back.
POST Create new resources; Create new entries in the database.
PUT or PATCH Update existing resources; Change existing records in the database.
DELETE Delete a resource; Remove records from the database.

Request

A request consists of 4 elements

  • endpoint
  • header
  • method
  • data (optional)

Response

The response consists of a header and data. Each response also has a status code indicating how the request was handled.

REST

This determines how the API looks like. REST stands for Representational State Transfer The most popular API standard for web applications is REST. The REST architecture is based on a client/server model. A stateless protocol is used for communication between client and server . Accessing a REST web service is called a request. The data returned by the web service is the response.