Data Feed

our data feed files provide structured and timely datasets to support downstream analytics, reporting and integration processes these feeds are delivered in standardised formats and updated on a regular cadence, ensuring consistency and reliability across systems this document outlines the structure, format, feed type and delivery mechanisms of the data feeds feed directory structure altrata data feed file delivery follows a standardised directory hierarchy for easy navigation and automation as below datapackage/version/feedtype/feeddate/datafeed files example /professionalcore/all all/v1/full/20230101/ the following explains this structure in detail, covering what each part of the path represents and its possible values 1 data package the altrata datafeed is provided in various packages currently, the following packages are available for purchase professional core package each package has its own directory on the sftp site you will only see directories for the packages that you have purchased each package has a set of sub directories these are intended to provide future segmentation of the data within the package the sub directory for the complete package, ie without any segmentation, is /all all currently no packages are available with segmentation example /professionalcore/all all package directory sub directory professional core /professionalcore /all all 2\ version versioning ensures compatibility between data producers and consumers as the data structure evolves over time following the package directory and sub directory is a version directory this indicates the version of the data feed new versions will be created, and old versions deprecated, in accordance with our process for versioned structure changes to the feed the available versions may differ between packages example /v1 3\ feed type our system supports the following feed delivery types full feed contains the entire dataset consumers are expected to overwrite previous data use case initial load, periodic re sync (weekly for our data feed files) /full in the file path indicates the files contained within are full feed files example /professionalcore/all all/v1/full/20230101/ 2\ incremental feed contains records that have changed since last feed (added, updated, or deleted) use case regular updates after full load (daily for our data feed files) /daily in the file path indicates the files contained within are incremental feed file example /professionalcore/all all/v1/daily/20230101/ feed type frequency directory full feed files weekly /full incremental feed files daily /daily 4\ feed frequency feed frequency defines how often data feeds are generated and delivered to ensure data is up to date without overwhelming your systems the frequency of our data feed files are as below day monday tuesday wednesday thursday friday saturday sunday full feed files incremental feed files included on upcoming monday included on upcoming monday please note it is not a requirement to ingest a full dataset, but we recommend at least a monthly full ingestion to ensure absolute data consistency with altrata 5 feed date the final part of the path indicates the date when files contained within were generated, in the format yyyymmdd example /20230101 6 data feed files each folder at the end of the directory path will contain a set of compressed files in the csv format there will be one file per entity in the feed feed file structure this section describes the structure of the data feed file including its format, field definitions, and sample content delimiters all files contain the following delimiters  column pipe ( | )  row carriage return and line feed header row the first row in all files is a header row this contains the name of each column, separated by a column delimiter example of a header row followed by the first data row metadata columns change bit the first column in all incremental feed files is the change bit this denotes whether the row record is a newly created record change bit “c” an updated record change bit “u” a deleted record change bit “d” when a row record has the change bit “d”, all other columns, except the primary key id, for the record will be null the change bit is only present in incremental feed files it does not appear in full feed files note deleted records do not appear in full files if you intend to load from both our full and incremental files, it is recommended to process any outstanding incremental files before processing the full files this will ensure any deletions are applied first record updated timestamp the second column in all incremental feed files is the record updated timestamp this indicates the date time when the row record was last updated it is in the format yyyymmddthh\ mm\ ss mss+0000 , where mss is a three digit millisecond and +0000 represents the utc time zone the record updated timestamp is only present in incremental feed files it does not appear in full feed files example 20230208t16 52 52 062+0000 data types in addition to the above metadata fields, the data fields within a feed file can be one of the following types boolean all boolean values are represented as a string literal true or false dates all dates in the feed files are represented as iso date strings, in the format yyyymmdd example 20230101 strings it is recommended to store string fields with a varchar(n), where n is set to the recommended character limits as shown in the data dictionary below note dates can be partial, with values to the right of yyyy omitted year only yyyy – for example 2000 year and month yyyymm – for example 200001 depending on your technology stack, your method for storing partial dates may differ 1\ if your environment supports it, we highly suggest storing partial dates as a date type 2\ if your environment does not support partial dates, then we recommend either storing them as a string or integer with trailing zeros (1) "2023 05" (2) 20230500 data feed structure changes changes to the content you receive can be released in one of two ways dynamic change these will be an addition of a column to an existing feed file, or an entirely new feed file column additions will always be added to the end of a file for example, if a file has 6 columns, the additional column would be 7th in the file to avoid breaking your ingestion processes, it is strongly recommended to write your load script in such a way that it accommodates both these types of additions versioned change these will be the removal of a column from an existing feed file, the removal of a feed file entirely, or a change in the data type of a column in an existing feed file such changes will always be provided in a completely new feed, which will be denoted by an increment in the version number in the feed’s file path when a new version of a feed is available, you will have a defined time period to upgrade during this period, both the new and previous version of the feed will be available to you