Managing your pipes

The Pipe API enables you to manage your Pipes. With Pipes you can transform data via SQL queries and publish the results of those queries as API endpoints.

Pipes are essentially a collection of transformation nodes or SQL queries. From each node you can query a Data source, the results of other nodes in the same Pipe, or the results exposed in a published Pipe. Note that you cannot query unpublished nodes of a different Pipe.

In order to use the Pipe API you need an Auth token with appropriate permissions.

How to create a Tinybird Analytics Pipe

Imagine you have an events Data Source named app_events and you want to expose an API endpoint containing the aggregation of events per day.

First, you would create a Pipe. We will call it events_per_day:

creating a pipe from a datasource
curl \\
-H "Authorization: Bearer <token> \\
-d "name=events_per_day&sql=select * from app_events" \\
https://api.tinybird.co/v0/pipes

As you can see, when creating the Pipe we have also defined the first transformation node SQL query, which in this case, is just getting all data from the Data Source. Below, you have how a succesful request response looks like after creating a Pipe:

Pipe successfully created
{
  "id": "t_1b501ccf34764a69aaf886bac9a7a6d8",
  "name": "events_per_day",
  "published_version": "t_878a81f0e1344a2ca990c8c1aa7dd69f",
  "published_date": "2019-06-14 09:46:07.884855",
  "nodes": [
    {
      "name": "events_per_day_00",
      "sql": "select * from app_events",
      "id": "t_878a81f0e1344a2ca990c8c1aa7dd69f",
      "dependencies": [
        "app_events"
      ],
      "materialized": false,
      "created_at": "2019-06-14 09:46:07.884849"
    }
  ]
}

Now, we are going to add a new transformation node to perform the aggregation per date using the previously created node events_per_day_00. For that let’s use “append” endpoint.

append a new node to transform your data
curl -d "select toDate(timestamp) date, count() event_count from events_per_day_00 group by date" \
https://api.tinybird.co/v0/pipes/events_per_day/append

Whenever a transformation node is added to a Pipe, you will receive a response like the one below summarizing giving you an autogenerated name for the node, as well as some metadata such as the dependencies with other transformation nodes:

succesful response
{
  "name": "events_per_day_01",
  "sql": "select toDate(timestamp) date, count() event_count from events_per_day_00 group by date",
  "id": "t_5164622050b244338ea2b79c19bd1e57",
  "dependencies": [
    "events_per_day_00"
  ],
  "materialized": false,
  "created_at": "2019-06-14 09:58:08.311812"
}

In order to make a Pipe publicly accessible through the API, you will need to enable your desired transformation node as an API endpoint. Pipes support just one enabled node so enbling one, will make previously enabled nodes to become unaccesible.

Enabling a transformation node as an API endpoint
curl -d "t_5164622050b244338ea2b79c19bd1e57" \
https://api.tinybird.co/v0/pipes/events_per_day/published

When enabling a transformation node as an API endpoint, a JSON containing the full Pipe description is sent as the response.

Succesful response
{
  "id": "t_1b501ccf34764a69aaf886bac9a7a6d8",
  "name": "events_per_day",
  "published_version": "t_5164622050b244338ea2b79c19bd1e57",
  "published_date": "2019-06-14 10:17:01.201962",
  "nodes": [
    {
      "name": "events_per_day_00",
      "sql": "select * from app_events",
      "id": "t_878a81f0e1344a2ca990c8c1aa7dd69f",
      "dependencies": [
        "app_events"
      ],
      "materialized": false,
      "created_at": "2019-06-14 10:17:01.201784"
    },
    {
      "name": "events_per_day_01",
      "sql": "select toDate(date) date, count() event_count from events_per_day_00 group by date",
      "id": "t_5164622050b244338ea2b79c19bd1e57",
      "dependencies": [
        "events_per_day_00"
      ],
      "materialized": false,
      "created_at": "2019-06-14 10:17:01.201943"
    }
  ]
}

Once the Pipe is created and we’ve enabled a transformation node as an Endpoint, we can easily integrate it into our 3rd party application. Using Query API (see Querying your data) you can query the Pipe using it’s name as if it was a regular table in a SQL query: SELECT * FROM events_per_day where date > yesterday()

quering a pipe using SQL
curl -d https://api.tinybird.co/v0/sql?q=SELECT * FROM events_per_day where date > yesterday()

If you don’t need to run any special operation against your Pipe, you can just use the Pipe data endpoint accesible at https://api.tinybird.co/v0/pipe/events_per_day/data.json. It’s an alias for SELECT * FROM events_per_day

Of course Pipes are updated in real-time, so as you insert the new data in app_events Data Source, every Pipe using it events_per_day is updated.

GET /v0/pipes/?

Get a list of Pipes in your account.

get a list of your pipes
curl -X GET "https://api.tinybird.co/v0/pipes"

Pipes in the response will be the ones that are accesible using a particular token with read permissions for them.

Succesfull response
{
    "pipes": [{
        "id": "pipe_id",
        "name": "pipe_name"
    }]
}

Pipes id’s are inmutable so you can always refer to them in your 3rd party applications to make them compatible with Pipes once they are renamed.

POST /v0/pipes/?

Creates a new Pipe. There are 3 ways to create a Pipe

Creating a Pipe providing full JSON
curl -X POST "https://api.tinybird.co/v0/pipes" -d
'{
    name":"pipe_name",
    "nodes": [
        {"sql": "select * from my_datasource limit 10", "name": "node_00", "description": "sampled data" },
        {"sql": "select count() from node_00", "name": "node_01" }
    ]
}'

If you prefer to create the minimum Pipe, and then append your transformation nodes you can set your name and first transformation node’s SQL in your POST request

Creating a pipe with a name and a SQL query
curl -X POST "https://api.tinybird.co/v0/pipes?name=pipename&sql=select * from datasource"

Pipes can be also created as copies of other Pipes. Just use the from argument:

Creating a pipe from another pipe
curl -X POST "https://api.tinybird.co/v0/pipes?name=pipename&from=src_pipe
DELETE /v0/pipes/(.+)/nodes/(.+)

Drops a particular transformation node in the Pipe. It does not remove related nodes so this could leave the Pipe in an unconsistent state. For security reasons, enabled nodes can’t be removed.

removing a token
curl -X DELETE "https://api.tinybird.co/v0/pipes/:name/nodes/:node_id"
Response codes

Code

Description

204

No error, removed node

404

Pipe not found

400

The node is published. Published nodes can’t be removed

403

Forbidden. Provided token doesn’t have permissions to change the last node to the pipe, it needs ADMIN or IMPORT

PUT /v0/pipes/(.+)/nodes/(.+)

Changes a particular transformation node in the Pipe

Editing a Pipe’s transformation node
curl -X PUT -d 'select * from node_0' "https://api.tinybird.co/v0/pipes/:name/nodes/:node_id?name=new_name&description=updated_explanation"
Request parameters

Key

Type

Description

name

String

new name for the node

description

String

new description for the node

token

String

Auth token. Ensure it has the PIPES:CREATE scope on it

Please, note that the desired SQL query should be sent in the body.

Response codes

Code

Description

200

No error

404

Pipe not found

400

Empty or wrong SQL

403

Forbidden. Provided token doesn’t have permissions to change the last node to the pipe, it needs ADMIN or PIPES:CREATE

POST /v0/pipes/(.+)/nodes

Appends a new transformation node to a Pipe.

adding a new node
curl -d 'select * from node_0' "https://api.tinybird.co/v0/pipes/:name/nodes?name=node_name&description=explanation"
Request parameters

Key

Type

Description

name

String

The referenceable name for the node.

description

String

Use it to store a more detailed explanation of the node.

token

String

Auth token. Ensure it has the PIPES:CREATE scope on it

SQL query for the transformation node must be sent in the body

Response codes

Code

Description

200

No error

404

Pipe not found

400

empty or wrong SQL

403

Forbidden. Provided token doesn’t have permissions to append a node to the pipe, it needs ADMIN or PIPES:CREATE

PUT /v0/pipes/(.+)/endpoint

Sets the enabled transformation node of the Pipe, so it becomes accesible as an API endpoint.

Change the endpoint
curl -X PUT -d '2b02db0cb3991a3bf3d0bb77392d47029257e1b16f4a30d589789bc2359f435c' "https://api.tinybird.co/v0/pipes/:name/endpoint"

Node_id to be enabled must be sent as part of the body. If empty, it will disable the current enabled node. Returns a full description of the Pipe in JSON.

Response codes

Code

Description

200

No error

404

Pipe not found

400

empty or wrong id

403

Forbidden. Provided token doesn’t have permissions to publish a pipe, it needs ADMIN or PIPES:CREATE

GET /v0/pipes/(.+).(json|csv)

Returns the published node data in a particular format.

Get data for a pipe
curl -X GET "https://api.tinybird.co/v0/pipes/:name.format"
Requests parameters

Key

Type

Description

q

String

Optional, query to execute, see API Query endpoint

The q parameter is a SQL query (see Querying your data). When using this endpoint to query your Pipes, you can use the _ shortcut, which refers to your Pipe name

Available formats

format

Description

csv

CSV with header

json

JSON including data, statistics and schema information

GET /v0/pipes/(.+.pipe)

Get pipe information. Provided Auth Token must have read access to the Pipe.

Get information about a particular pipe
curl -X GET "https://api.tinybird.co/v0/pipes/:name"

pipe_id and pipe_name are two ways to refer to the pipe in SQL queries and API endpoints the only difference is pipe_id never change so if it’ll work even if you change the pipe_name (which is the name used to display the pipe). In general you can use pipe_id or pipe_name indistinctly:

Succesfull response
{
    "id": "t_bd1c62b5e67142bd9bf9a7f113a2b6ea",
    "name": "events_pipe",
    "pipeline": {
        "nodes": [{
            "name": "events_ds_0"
            "sql": "select * from events_ds_log__raw",
            "materialized": false
        }, {
            "name": "events_ds", 
            "sql": "select * from events_ds_0 where valid = 1",
            "materialized": false
        }]
    }
}

You can make your Pipe’s id more descriptive by prepending information such as t_my_events_table.bd1c62b5e67142bd9bf9a7f113a2b6ea

DELETE /v0/pipes/(.+.pipe)

Drops a Pipe from your account. Auth token in use must have the DROP:NAME scope.

Drops a pipe
curl -X DELETE "https://api.tinybird.co/v0/pipes/:name"
PUT /v0/pipes/(.+.pipe)

Changes Pipe’s metadata. When there is another Pipe with the same name an error is raised.

editing a token
curl -X PUT "https://api.tinybird.co/v0/pipes/:name?name=new_name"
Request parameters

Key

Type

Description

name

String

new name for the pipe

token

String

Auth token. Ensure it has the PIPES:CREATE scope on it

GET /v0/pipes/(.+)

Get pipe information. Provided Auth Token must have read access to the Pipe.

Get information about a particular pipe
curl -X GET "https://api.tinybird.co/v0/pipes/:name"

pipe_id and pipe_name are two ways to refer to the pipe in SQL queries and API endpoints the only difference is pipe_id never change so if it’ll work even if you change the pipe_name (which is the name used to display the pipe). In general you can use pipe_id or pipe_name indistinctly:

Succesfull response
{
    "id": "t_bd1c62b5e67142bd9bf9a7f113a2b6ea",
    "name": "events_pipe",
    "pipeline": {
        "nodes": [{
            "name": "events_ds_0"
            "sql": "select * from events_ds_log__raw",
            "materialized": false
        }, {
            "name": "events_ds", 
            "sql": "select * from events_ds_0 where valid = 1",
            "materialized": false
        }]
    }
}

You can make your Pipe’s id more descriptive by prepending information such as t_my_events_table.bd1c62b5e67142bd9bf9a7f113a2b6ea

DELETE /v0/pipes/(.+)

Drops a Pipe from your account. Auth token in use must have the DROP:NAME scope.

Drops a pipe
curl -X DELETE "https://api.tinybird.co/v0/pipes/:name"
PUT /v0/pipes/(.+)

Changes Pipe’s metadata. When there is another Pipe with the same name an error is raised.

editing a token
curl -X PUT "https://api.tinybird.co/v0/pipes/:name?name=new_name"
Request parameters

Key

Type

Description

name

String

new name for the pipe

token

String

Auth token. Ensure it has the PIPES:CREATE scope on it