Exchanging Data with Recast

Exchanging Data with Recast


Clients must provide Recast with data updates to support model refreshes, and may also (optionally) receive model output data back from Recast. Recast supports many options for this data exchange, including both Recast-managed and client-managed data repositories, as well as third-party services. All of the currently supported options are listed below, as well as details of the setup process and requirements for each, particularly in regard to security and authentication.

Recast is continually implementing new data exchange options based on client needs, so if you donโ€™t see an option below that works for your organization, please inquire. However, implementing new options is likely to extend onboarding timelines, so utilizing one of the options listed below whenever possible is encouraged.

The table of contents below provides a list of the currently supported data repositories with links to quickly access more information about each option.

Recast-Managed Data Repositories

AWS S3

Clients may transfer data to and from an AWS S3 bucket in Recastโ€™s AWS account.

This option is often preferred by clients who have their own AWS account (but donโ€™t want to manage the S3 bucket used for data exchange with Recast) or who are using a third-party service which offers an โ€œS3 connectorโ€.

Two authentication methods are supported, with the details of each covered below.

IAM Role (strongly preferred)

Recast can create an IAM role which may be assumed from the clientโ€™s AWS account or the AWS account of a third-party service being utilized by the client. Once the role is assumed, access to Recastโ€™s S3 bucket will be available.

In order to create the IAM role, Recast needs to know the following information:

  1. Required: The AWS account number which the role will be assumed from.

  2. Optional: The ARN of the clientโ€™s IAM user or role which will be assuming Recastโ€™s IAM role. If not provided, Recast will allow any IAM user or role in the clientโ€™s AWS account to assume Recastโ€™s IAM role.

  3. Optional: The โ€œexternal IDโ€ included in requests made by the clientโ€™s IAM role to assume Recastโ€™s IAM role. The use of an external ID is an additional security measure most appropriate when utilizing a third-party service to connect to Recastโ€™s S3 bucket. The third-party service will provide this external ID (generally when configuring the S3 connector). Recast strongly discourages the use of any third party service which does not support an external ID for S3 connectors. For connections from client-managed AWS accounts, an external ID is less critical but is also supported.

Once Recast completes setup of the IAM role, Recast will provide the IAM roleโ€™s ARN to the client.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

  • The account number of the clientโ€™s AWS account which Recastโ€™s IAM role will be assumed from. (required)

  • The ARN of the clientโ€™s IAM user or role which will will be assuming Recastโ€™s IAM role. (optional)

  • The external ID included in requests made by the clientโ€™s IAM role to assume Recastโ€™s IAM role. (optional)

Recast provides to client:

  • The ARN of Recastโ€™s IAM role which the client will be assuming.

  • The bucket name of Recastโ€™s S3 bucket.

  • The path in the S3 bucket where the client will write and read data.

IAM User (supported but strongly discouraged)

Recast can create an IAM user with an access key which may be used to access Recastโ€™s S3 bucket. The use of an IAM user provides great flexibility in the methods used to access Recastโ€™s S3 bucket for clients who do not have their own AWS account, including programmatic access via the AWS API, command-line access via the AWS CLI, and third-party โ€œS3 connectorsโ€.

Although this option is supported, and even sometimes necessary given some current client constraints, Recast strongly discourages its use because access via an IAM user is more difficult to secure than access via an IAM role. Instead, Recast recommends the use of an IAM role (for clients with their own AWS account) or SFTP (for clients without their own AWS account) whenever possible.

Additionally, Recastโ€™s security policies require IAM user access keys to be rotated periodically, which necessitates ongoing maintenance and may also disrupt scheduled data exchanges and model refreshes if the key rotation is not handled in a timely manner.

Recast does not require any information from the client in order to set up an IAM user for the client.

Once Recast completes setup of the IAM user, Recast will provide the IAM userโ€™s access key and the S3 bucket name and path to the client.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • The Access Key ID and Secret Access Key of Recastโ€™s IAM user which the client will use to access Recastโ€™s S3 bucket.

  • The bucket name of Recastโ€™s S3 bucket.

  • The path in the S3 bucket where the client will write and read data.

SFTP

Clients may transfer data to and from an SFTP server managed by Recast, with an SFTP user set up by Recast.

This option is often preferred by clients who do not have their own AWS account, who have an established pattern of utilizing SFTP for data transfers, or who are using a third-party service which offers an โ€œSFTP connectorโ€.

Two authentication methods are supported, with the details of each covered below.

Key-Based Authentication (strongly preferred)

Recast supports and recommends key-based authentication for SFTP.

In order to set up an SFTP user for the client using key-based authentication, the client must create an SSH key pair and provide Recast with its public key. (The client should not share the private key, and should store the key pair securely.)

After Recast receives the public key from the client and completes setup of the SFTP user, Recast will provide the client with the information necessary to access the SFTP server, including hostname (sftp.getrecast.com), port (22), and username.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

  • The public key from the client-generated SSH key pair.

Recast provides to client:

  • The SFTP host name.

  • The SFTP port.

  • The SFTP username.

Password Authentication (supported but strongly discouraged)

Recast supports, but does not recommend, password authentication for SFTP.

Although this option is supported, Recast strongly discourages its use because password authentication is less secure than key-based authentication.

Additionally, Recastโ€™s security policies require SFTP passwords to be rotated periodically, which necessitates ongoing maintenance and may also disrupt scheduled data exchanges and model refreshes if the key rotation is not handled in a timely manner.

Recast does not require any information from the client in order to set up password authentication.

Once setup of the SFTP user is complete, Recast will provide the client with the information necessary to access the SFTP server, including hostname (sftp.getrecast.com), port (22), username, and password.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • The SFTP host name.

  • The SFTP port.

  • The SFTP username.

  • The SFTP password.

Client-Managed Data Repositories

AWS S3

Recast may transfer data to and from an AWS S3 bucket owned by the client. Recast will access the clientโ€™s S3 bucket from Recastโ€™s AWS account.

This option is often preferred by clients who have their own AWS account and want to manage the S3 bucket that is used for data exchange with Recast.

Two authentication methods are supported, with the details of each covered below.

IAM Role (preferred)

The client will create an IAM role in the clientโ€™s AWS account, which Recast will assume from Recastโ€™s AWS account.

The IAM roleโ€™s trust policy must allow Recast to assume the role. The client may trust any IAM user or role in Recastโ€™s account by specifying the following trust policy.

{ "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Resource": "arn:aws:iam::878987988123:root" } ] }

If the client prefers a more restrictive trust policy, limiting access to a single, specific IAM role in Recastโ€™s AWS account, the following trust policy is recommended.

{ "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Resource": "arn:aws:iam::878987988123:role/recast_access_client_s3_bucket_[client_id]" } ] }

The placeholder text [client_id] will be a unique client identifier provided by Recast.

The clientโ€™s IAM roleโ€™s permissions must allow certain S3 actions in order for Recast to read from and write to the clientโ€™s S3 bucket. The following policy allows the S3 actions which are necessary now, as well as any that are likely to be needed in the future.

{ "Statement": [ { "Action": [ "s3:*" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::[client_bucket_name]", "arn:aws:s3:::[client_bucket_name]/*" ] "Sid": "S3Access" }, ], "Version": "2012-10-17" }

The placeholder text [client_bucket_name] must be replaced with the actual name of the clientโ€™s S3 bucket.

Recast recommends that the client create a dedicated S3 bucket used solely for the purpose of exchanging data with Recast.

The policy above would allow all S3 actions across the entire S3 bucket, and should therefore only be used if the S3 bucket is for exclusive use by Recast. Should the client chose to use a non-dedicated bucket, it is up to the client to implement a policy which restricts Recastโ€™s access to the bucket appropriately.

Should the client wish to take a more restrictive โ€œprinciple of least privilegeโ€ approach in regard to the allowed actions, the following actions may be specified, although this more restrictive actions list may require future updates as requirements change.

"Action": [ "s3:ListObjectsV2", "s3:ListBucket", "s3:GetObjectAttributes", "s3:GetBucketLocation", "s3:*ObjectTagging", "s3:*Object", "s3:*Acl" ]

Once the client completes setup of the IAM role, the client will provide the IAM roleโ€™s ARN to Recast.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • The ARN of the IAM role in Recastโ€™s account which will assume the IAM role in the clientโ€™s account. (optional)

Client provides Recast:

  • The ARN of the IAM role in the clientโ€™s account which Recast will assume to access the clientโ€™s S3 bucket.

  • The bucket name of the clientโ€™s S3 bucket.

IAM User (supported)

The client will create an IAM user in the clientโ€™s AWS account, which Recast will authenticate to with the userโ€™s access key which the client must also create.

The clientโ€™s IAM userโ€™s permissions must allow certain S3 actions in order for Recast to read from and write to the clientโ€™s S3 bucket. The following policy allows the S3 actions which are necessary now, as well as any that are likely to be needed in the future.

{ "Statement": [ { "Action": [ "s3:*" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::[client_bucket_name]", "arn:aws:s3:::[client_bucket_name]/*" ] "Sid": "S3Access" }, ], "Version": "2012-10-17" }

The placeholder text [client_bucket_name] must be replaced with the actual name of the clientโ€™s S3 bucket.

Recast recommends that the client create a dedicated S3 bucket used solely for the purpose of exchanging data with Recast.

The policy above would allow all S3 actions across the entire S3 bucket, and should therefore only be used if the S3 bucket is for exclusive use by Recast. Should the client chose to use a non-dedicated bucket, it is up to the client to implement a policy which restricts Recastโ€™s access to the bucket appropriately.

Should the client wish to take a more restrictive โ€œprinciple of least privilegeโ€ approach in regard to the allowed actions, the following actions may be specified, although this more restrictive actions list may require future updates as requirements change.

"Action": [ "s3:ListObjectsV2", "s3:ListBucket", "s3:GetObjectAttributes", "s3:GetBucketLocation", "s3:*ObjectTagging", "s3:*Object", "s3:*Acl" ]

Once the client completes setup of the IAM user, the client will provide the IAM userโ€™s access key and the S3 bucket name to Recast.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides Recast:

  • The Access Key ID and Secret Access Key of the clientโ€™s IAM user which Recast will use to access the Clientโ€™s S3 bucket.

  • The bucket name of the Clientโ€™s S3 bucket.

SFTP

Recast may transfer data to and from an SFTP server managed by the client, with an SFTP user set up by the client.

This option is often preferred by clients who have an their own SFTP server and an established practice of exchanging data via SFTP.

Two authentication methods are supported, with the details of each covered below.

Key-Based Authentication (strongly preferred)

Recast supports and recommends key-based authentication for SFTP.

To allow the client to set up an SFTP user for Recast using key-based authentication, Recast will create an SSH key pair and provide the client with its public key. This key pair will be used exclusively for accessing the clientโ€™s SFTP server.

After the client receives the public key from Recast and completes setup of the SFTP user, the client will provide Recast with the information necessary to access the SFTP server, including hostname, port, and username.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • The public key from the Recast-generated SSH key pair.

Client provides to recast:

  • The SFTP host name.

  • The SFTP port.

  • The SFTP username.

Password Authentication (supported but strongly discouraged)

Recast supports, but does not recommend, password authentication for SFTP.

Although this option is supported, Recast strongly discourages its use because password authentication is less secure than key-based authentication.

Additionally, Recastโ€™s security policies require SFTP passwords to be rotated periodically, which necessitates ongoing maintenance and may also disrupt scheduled data exchanges and model refreshes if the key rotation is not handled in a timely manner.

The client should not require any information from Recast in order to set up password authentication. The client should randomly generate a strong password.

Once setup of the SFTP user is complete, the client will provide Recast with the information necessary to access the SFTP server, including hostname, port, username, and password.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

  • The SFTP host name.

  • The SFTP port.

  • The SFTP username.

  • The SFTP password.

Google Sheets

Recast may transfer data to and from Google Sheets owned by the client.

This option is often preferred by clients who are already maintaining data in Google Sheets, or who are not prepared to utilize other options such as AWS S3 or SFTP.

The client will give Recastโ€™s Google service account permissions to the clientโ€™s sheets.

After sharing the sheets, the client must provide Recast with links (URLs) to the sheets.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • Recastโ€™s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

  • Link (URL) to all sheets.

Google Cloud Storage

Recast may transfer data to and from a Google Cloud Storage (GCS) bucket owned by the client.

Two authentication methods are supported, with the details of each covered below.

Service Account Authentication

The client will give Recastโ€™s Google service account permissions to the clientโ€™s GCS bucket.

Once access to the bucket has been configured, the client will provide Recast with the name of the bucket.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • Recastโ€™s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

  • The bucket name of the Clientโ€™s GCS bucket.

Key-Based Authentication

Recast will provide the client with a public key.

The client will configure their GCS bucket to allow access to the bucket, authenticated by the public key.

Once access to the bucket has been configured, the client will provide Recast with the name of the bucket.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • Recastโ€™s public key.

Client provides to Recast:

  • The bucket name of the Clientโ€™s GCS bucket.

Google BigQuery

Recast may transfer data to and from a Google BigQuery warehouse owned by the client.

This option is often preferred by clients who are already maintaining data in Google BigQuery.

The client will grant the appropriate BigQuery roles to a Recast service account.

Once the service account roles have been configured, the client will provide Recast with the Google BigQuery project ID and dataset ID.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • Recastโ€™s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

  • The BigQuery project ID.

  • The BigQuery dataset ID.

Google Drive

Recast may transfer data to and from a Google Drive folder owned by the client.

This option is often preferred by clients who are already using Google Drive, and who are not prepared to utilize other options such as AWS S3 or SFTP.

The client will give Recastโ€™s Google service account permissions to the clientโ€™s Google Drive folder.

After sharing the sheets, the client must provide Recast with a link (URL) to the folder.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • Recastโ€™s service account (servers@ambient-hulling-282015.iam.gserviceaccount.com).

Client provides to Recast:

  • Link (URL) to the Google Drive folder.

Snowflake

Recast may transfer data to and from a Snowflake database owned by the client.

This option is often preferred by clients who are already maintaining data in Snowflake.

Recast does not have a Snowflake account so this option is only available for clients who have their own Snowflake account. Since Recast does not have a Snowflake account, the client must set up a reader account for Recast, and cannot utilize Secure Data Sharing.

Snowflake now requires key-pair authentication for reader accounts, and no longer supports password authentication.

Recast will provide the client with the public key from a key pair generated solely for this use.

The client will create a reader account for Recast, and then provide Recast with the reader account username and the details of the Snowflake data source.

The following is a summary of the setup information to be exchanged between Recast and the client.

Recast provides to client:

  • The public key from the Recast-generated key pair.

Client provides to Recast:

  • The Snowflake username.

  • The Snowflake hostname.

  • The Snowflake database.

  • The Snowflake schema.

  • The Snowflake warehouse.

  • The Snowflake role.

Postgres Databases (including Redshift)

Recast may transfer data to and from a Postgres database owned by the client.

This option is often preferred by clients who are already maintaining data in a Postgres database.

Recast supports username/password authentication for Postgres databases.

The client should not require any information from Recast in order to set up a database user with password authentication. The client should randomly generate a strong password.

Once setup of the database user is complete, the client will provide Recast with the information necessary to access the database, including hostname, port, username, password, and database.

This information must be communicated securely, through a service such as onetimesecret.com, or some other secure messaging channel.

The following is a summary of the setup information to be exchanged between Recast and the client.

Client provides to Recast:

  • The database host name.

  • The database port.

  • The database username.

  • The database password.

  • The database name.

Third-Party Data Repositories

Recast also supports data exchange with third-party services, including Fivetran, Census, and Adverity. Such support is generally implemented via โ€œconnectorsโ€, such as S3 connectors and SFTP connectors, in which case the information on those access methods above generally applies, with some specific requirements depending on the third party.

  • Fivetran - Supports AWS IAM roles via an S3 connector. The AWS account number will be Fivetranโ€™s AWS account, not the clientโ€™s AWS account. An external ID is required, and provided by Fivetran, unique to each client. Also supports SFTP via an SFTP connector, although each file requires setup of a separate SFTP connector, so using an S3 connector is recommended.

  • Census - Supports AWS IAM roles via an S3 connector. The AWS account number will be Censusโ€™s AWS account, not the clientโ€™s AWS account. An external ID is required, and provided by Census, unique to each client.

  • Adverity - Supports AWS IAM roles and IAM users as S3 sources and destinations.

Although Recast generally prefers using an IAM role over an IAM user, Adverity does not support an external ID in its IAM role configuration, which Recast views as a significant security vulnerability for third-party services. Therefore Recast recommends, in this specific case, the use of an IAM user. Should the client prefer to use an IAM role to avoid the need to rotate the IAM user access key, the AWS account number will be Adverityโ€™s AWS account, not the clientโ€™s AWS account.

Other third-party services are likely to also support access to Recastโ€™s S3 bucket, and Recast is happy to assist with implementing access, as requested.

ย