Connecting via Amazon S3

the altrata data feed files can be delivered to you through amazon s3 your feed's files are made available in a dedicated s3 bucket, from which you can pull files directly into your own environment using standard aws tooling this documentation walks through the onboarding information required to set up s3 access and how to connect to the bucket and download files if you have any questions, please contact support\@altrata com onboarding information to provision your s3 access, your account representative will ask you to provide two values value description customer access principal arn (customeraccessprincipalarn) the amazon resource name (arn) of the iam principal in your aws account typically an iam role, or an iam user that will be used to access the altrata s3 bucket example format arn\ aws\ iam 123456789012\ role/altrata datafeed reader customer external id (customerexternalid) a unique string of your choosing together with your principal arn, this secures the trust relationship on the altrata iam role and allows only your principal to assume it see "what is the external id?" below what is the external id? the external id is a standard aws mechanism used to secure cross account access to s3, protecting against what aws calls the "confused deputy" problem it is included as a condition in the iam role's trust policy, so the role can only be assumed when the correct external id is supplied alongside your principal arn the external id is not issued by aws and is not retrieved from anywhere it is simply a unique string chosen by the party that will access the data because you (the customer) are the accessing party, you choose the value a randomly generated guid (e g b8ee735e 0f96 4b44 96c2 166cdcce8dfc ) is a good choice note that aws does not treat the external id as a secret; its purpose is uniqueness, not secrecy for more detail, see aws's documentation providing access to aws accounts owned by third parties https //docs aws amazon com/iam/latest/userguide/id roles common scenarios third party html once provisioned, your account representative will provide you with the following the arn of a dedicated altrata iam role , created for your account, which your principal will assume to access the bucket the role's trust policy is configured with your principal arn and external id, so only your principal supplying your external id can assume it you will need this role arn for the final permissions configuration in the next step the name of your dedicated s3 bucket the aws region of the bucket configuring permissions to assume the altrata access role once you have the arn of the role you need to assume, a critical final configuration step is to set the permissions required for your customer access principal to assume the altrata role to allow this permission, add a policy to your customer access principal allowing the assumerole action on the altrata access role resource for example { "version" "2012 10 17", "statement" \[ { "sid" "allowcustomerprincipaltoassumealtratarole", "effect" "allow", "action" "sts\ assumerole", "resource" "\[your altrata access role arn]" } ] } connecting to the s3 bucket you can access the bucket with any tooling that supports amazon s3, including the aws cli, aws sdks, or data platform ingestion connectors the examples below use the aws cli step 1 assume the altrata access role use your access principal to assume the role provided by altrata, passing your external id aws sts assume role \ role arn arn\ aws\ iam \[altrata account id]\ role/\[altrata role name] \\ \ role session name altrata datafeed \\ \ external id "your external id" this returns temporary credentials (accesskeyid, secretaccesskey, sessiontoken) you must use these credentials for the s3 commands in steps 2 and 3 below either export them as the aws access key id, aws secret access key, and aws session token environment variables, or save them as a named profile and add profile to each command running the s3 commands under your normal aws identity, without assuming the role, will result in an access denied error note your access principal must be permitted, within your own aws account, to call sts\ assumerole on the role arn provided by altrata configuring this permission is managed by your organization step 2 view available directories and files you should see one directory (prefix) per subscribed data feed package aws s3 ls s3 //\[your bucket name]/ step 3 copy the files there are many options for copying the altrata data feed files from the s3 bucket owned by altrata to a destination location of your choice as a developer, if you are working locally, you could choose to copy the files to you location machine see examples 1 and 2 below example 1 copy all files to a local directory using s3 copy aws s3 cp s3 //\[your bucket name]/path/to/directory/ /local folder/ recursive example 2 keep a local folder in sync with the s3 bucket using s3 sync aws s3 sync s3 //\[your bucket name]/path/to/directory/ /local folder/ however, in production, you would likely choose to copy the files from the altrata owned s3 bucket to another s3 bucket in your aws account see example 3 and 4 below example 3 copy all files to another s3 bucket using s3 copy aws s3 cp s3 //\[your bucket name]/path/to/directory/ s3 //\[your target bucket name]/path/to/directory recursive example 4 keep another s3 bucket in sync with the altrata bucket using s3 sync aws s3 sync s3 //\[your source bucket name]/path/to/directory/ s3 //\[your target bucket name]/path/to/directory storage capacity and bandwidth considerations before downloading data from s3, you should ensure that you have sufficient disk space available on your system to store the delivery files disk space requirements will vary depending on the data packages you wish to download full data files can often exceed 1 gb; it is recommended to have sufficiently high speed internet access bandwidth to download these files