Service Data Sources¶
In addition to the data sources you upload, Tinybird provides other “service data sources” that allow to inspect what’s going on in your account. For example, you can get real time stats about API calls or a log of every operation over your Data Sources.
It’s the same than using system tables in any database but in this case contains information about the service usage.
These are the available tables:
Contains stats about your snapshot views
date (Date): view date
snapshot_id (String): snaphot id. It’s the same id other users can see in the URL (https://ui.tinybird.co/snapshot/:snapshot_id)
views (Int32): number of views
Contains stats about your Pipe endpoints API calls aggregated per day.
date (Date): view date
pipe_id (String): pipe id as returned in Pipes API.
view_count: view count
error_count: number of request with error
avg_duration_state: avg duration state (see an example on how to query state columns below)
quantile_timing_state: 0.9, 0.95 and 0.99 quantiles state. Time in milliseconds
Querying tables with state columns is easy. State is a intermediate aggregated state and to get the final value you just need to use the aggregation method plus
-State. See the following example:
SELECT date, sum(view_count) view_count, sum(error_count) error_count, avgMerge(avg_duration_state) avg_time, quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_millis_array FROM tinybird.pipe_stats where pipe_id = 'PIPE_ID' group by date
Same information that
tinybird.pipe_stats but in real time and without aggregation, every single request is logged into this table. This table removes elements after 2 days so if you want to do operations with data older than 2 days you should use the aggregated version mentioned earlier.
start_datetime (DateTime): API call started
pipe_id (String): Pipe id as returned in Pipes API.
duration (Float): API call duration in seconds
url (String): url (
tokenparam is removed for security reasons)
error (Boolean): true if query returned error.
The Data Source contains details about how we ingested data into your Data Sources. We do ingestion in chunks. For instance, when we ingest from a URL, if it is possible, we split the download in several requests resulting in different ingestion blocks. The same happens when the data upload happens with a multipart request: we ingest the data in several chunks as the data is arriving. You can use this Service Data Source to spot problematic parts of your data.
timestamp DateTime: when the block was ingested.
job_id String: the Job or Request id that ingested the block of data.
source String: Either the URL or
block_id String: the block identifier, you can cross this with the
blocks_idscolumn from the
tinybird.datasources_ops_logService Data Source.
datasource_id String: The Data Source consistent id.
datasource_name String: The Data Source name when the block was ingested.
start_offset Nullable(Int64): The starting byte of the block, if the ingestion was split, where this block started.
end_offset Nullable(Int64): If split, the ending byte of the block.
rows Nullable(Int32): how many rows it ingested.
parser Nullable(String): either if we use the native block parser or we had to fallback to row by row parsing.
quarantine_lines Nullable(UInt32): If any, how many rows got into the quarantine Data Source.
empty_lines Nullable(UInt32): If any, how many empty lines were skipped.
bytes Nullable(UInt32): How many bytes the block had.
processing_time Nullable(Float32): How long it took.
processing_error Nullable(String): Detailed message in case of error.
Contains all operations performed to your Data Sources. We track the following operations:
rename. We log all of them with extra information in this Data Source.
timestamp DateTime: When the operation happened.
event_type String: The operation type.
datasource_id String: The id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes.
datasource_name String: The name of your Data Source when the operation happened.
result String: Either
elapsed_time Float32: How much time the operation took.
error Nullable(String): Detailed error message if the result was error.
job_id Nullable(String): The id of the job or request that performed the operation.
rows Nullable(UInt64): How many rows the operations affected. This depends on the
event_type: for the
appendevent, how many rows got inserted; for the
truncateevents, how many rows the Data Source had; for the
replacetype, how many rows the Data Source has after the operation.
rows_quarantine Nullable(UInt64): If any, how many rows got into the quarantine Data Source.
blocks_ids Array(String): The list of blocks ids that we used for the operation. See the
tinybird.block_logService Data Source for more details.
Options Nested(Names String, Values String): We store key-value pairs with extra information for some of the operations. For the
replaceevent, we use the
rows_before_replacekey to track how many rows the Data Source had before the replacement happened, the
replace_conditionkey shows what condition was used. For the
replaceevents we store the data
source, e.g. the URL, or body/stream keywords. For the