Salesforce Reader

The Salesforce Reader uses REST APIs to read Salesforce CRM (sObject) data at configured refresh intervals. See also Salesforce Pardot Reader, Salesforce Platform Event Reader, and Salesforce Push Topic Reader.

See API Request Limits and Allocations for information on Salesforce-side limits that may require you to limit how much and how quickly Salesforce Reader ingests data.

You can read from Salesforce as follows, and write to any target supported by Striim.

Typically, data pipelines read from Salesforce in two phases, as explained in this concept article on Pipelines. The first step is the initial load, where you can create a point-in-time copy of the existing source Salesforce dataset at the target. After the initial load has completed, you can start continuous replication by continuously reading the new data created in the same Salesforce dataset after the initial load was started, and then writing this new source data to the target. When set to "Automated" mode, the Salesforce Reader first completes the initial load (as of the start time of the application) and on completion of initial load, it will automatically start polling for incremental changes at the specified interval.

For initial load, you can use Salesforce Reader.
For near real-time continuous replication after initial load, you can use Salesforce CDC Reader that reads new source data as soon as a change event is emitted from Salesforce.
For incremental load after initial load, you can use Salesforce Reader.

Automated mode

In Automated mode, Salesforce Reader utilizes Incremental mode with a ‘startTimestamp’ of ‘1970-01-01’. This configuration allows it to perform an initial data load and capture any new records during subsequent polling intervals. You can specify Automated mode when establishing a connection to Salesforce, for example when using a connection profile.

In Automated mode, you can select from a list of all the available standard and custom Salesforce objects, enabling you to select any object for data replication. After selecting an object, the system verifies that all required permissions are in place for transferring data to the target system.

Once the setup is complete, the Salesforce Reader continuously loads the initial dataset and captures newly added records at each polling interval for the selected Salesforce objects.

Summary information and supported objects

APIs used/data supported	Standard and custom Salesforce objects
Security and authentication	OAuth-based authentication
Operations/modes supported	Automated (first initial load and then incremental load) Initial load Incremental load
Schema management	Initial schema creation Schema evolution (resync)
Resilience / recovery	Configurable automatic retries
Performance	Striim Platform supported parallel execution
Programmability	Wizards Striim Flow Designer Striim TQL
Metrics and auditing	Key metrics available through Striim monitoring

Salesforce Reader initial setup

The Salesforce Reader supports authentication with OAuth. Configure OAuth when prompted by the Striim wizard.

Configuring OAuth for Salesforce Reader

For simpler configuration, create a connection profile (see Connection profiles).

Configuring OAuth for automatic authentication token renewal

The following procedure requires an active Salesforce account and a Striim app connected to Salesforce (see Create a connected app).

From the connected app, get the values of the Consumer Key and Consumer Secret.
In the Salesforce Reader, set the values of the Consumer Key and Consumer Secret.
Generate a security token following the instructions in Reset your security token.
In the Salesforce Reader, set the value of the Security token.

Configuring OAuth for manual authentication token renewal

Generate an authentication token using the following command:

curl https://login.salesforce.com/services/oauth2/token -d "grant_type=password"\
 -d "client_id=<your consumer key>"\
 -d "client_secret=<your consumer secret>"\
 -d "username=<your username>"\
 -d "password=<your password>"

In the Salesforce Reader, set the value of the authentication token.
Generate a security token following the instructions in Reset your security token.
In the Salesforce Reader, set the value of the Security token.

Verifying Salesforce Reader configuration

Use the following cURL commands (see Using cURL in the REST Examples and curl.haxx.se) to verify your configuration and get information about available resources and sObjects.

Get an access token using the Salesforce login URL.

curl https://login.salesforce.com/services/oauth2/token -d "grant_type=password" \
-d "client_id=<your consumer key>" -d "client_secret=<your consumer secret>" \
-d "username=<your username>" -d "password=<your password>"

Using the access token returned by that command, test the REST API URL for your organization. The instance is typically the first part of the URL you see in your browser when logged into Salesforce, such as "mycompany" in mycompany.salesforce.com. Alternatively, ask your Salesforce technical administrator for access to a connected app. (For more information, see Understanding the Username-Password OAuth Authentication Flow.)
If you do not have a proxy server:
```
curl https://<your Salesforce instance>.salesforce.com/services/data/ \
-H 'Authorization: Bearer <token>'
```
If you have a proxy server (change the proxy server URL to match yours):
```
curl -x http://mycompany.proxy.server.com:8080/ \
https://<your Salesforce instance >.salesforce.com/services/data/ \
-H 'Authorization: Bearer <token>'
```

List available REST resources and sObjects (see List Available REST Resources and Get a List of Objects).

curl https://<your Salesforce instance>.salesforce.com/services/data/v41.0 \
-H 'Authorization: Bearer <token>'
curl https://<your Salesforce instance>.salesforce.com/services/data/v41.0/sobjects \
-H 'Authorization: Bearer <token>'

For additional information, see Salesforce's REST API Developer Guide .

Create a Salesforce Reader application

To start using the Salesforce Reader in Striim apps, you can create an application in the following ways:

Striim Wizard in the Flow Designer
Striim Flow Designer
Striim TQL

Creating a Salesforce Reader application using the wizard

Log in to a Striim instance.
Click Create an App from the Apps menu.
Select Salesforce as the source.
Specify your target and click Get Started.
The Salesforce Reader wizard starts.
In Name, enter a name for the app, then select a namespace from the Namespace drop-down.
The default namespace is the application name.
Click Save.
The wizard advances to the next step. You can specify connecting using a connection profile, or manually enter the connection properties in the wizard (described in steps 7-10).
For the API Endpoint, specify the URL used to access Salesforce services. For example, https://ap2.salesforce.com
Select Authenticate using Auth Token or Authenticate using User Credentials, as suited to your use case.
(To authenticate with OAuth authentication token) Specify an auth token that you obtain after creating a connected app in Salesforce.
(To authenticate with OAuth username and password) Specify the credentials for a Salesforce user account, with permissions to access objects:
- Username
- Password
- Consumer Key,
- Consumer Secret
- Security Token.
Select Create schemas on the target before moving the data if you want the app to create the source schema for all selected tables at the target.
Click Next.
The wizard validates the authentication information and displays the results.
Click Next.
The wizard displays a list of objects available on the Salesforce instance.
Select the objects to ingest and click Next. Select only objects for which the previously specified token has permissions or the application will fail with an "Insufficient permission to access the metadata" error.
Configure your target.
Click Next.
The wizard displays summary information.
Click Save & Run.
The application is saved and begins running.

Creating a Salesforce Reader application using the Flow Designer

See Creating and modifying apps using the Flow Designer.

Creating a Salesforce Reader application using TQL

The following TQL will read from the Business__c sObject and create an appropriate typed stream:

CREATE SOURCE SFPoller USING SalesforceReader (
  sObjects: 'Business__c',
  authToken: '********',
  apiEndPoint: '<your organization's endpoint UEL>',
  mode: 'InitialLoad',
  autoAuthTokenRenewal: 'false'
)
OUTPUT TO DataStream;

CREATE TYPE OpStream_Type (
  Id String KEY,
  Name String,
  POSDataCode__c String,
  Currency__c String
);
CREATE STREAM OpStream OF OpStream_Type;

CREATE CQ CQ1
  INSERT INTO OpStream
  SELECT data[0],data[1],data[2],data[3]
  FROM DataStream;

Salesforce Reader programmer's reference

Salesforce Reader properties

property	type	default value	notes
API End Point	String		The endpoint for your Force.com REST API.
Auth Token	com.webaction. security.Password		See Configuring OAuth for Salesforce Reader. When Auto Auth Token Renewal is `True`, this property is ignored and does not appear in Flow Designer. If autoAuthTokenRenewal is set to false , specify your Salesforce access token (see Set Up Authorization on developer.salesforce.com: the first section, "Setting Up OAuth 2.0," explains how to create a "connected app"; the second section, "Session ID Authorization," explains how to get the token using `curl`).
Auto Auth Token Renewal	String	false	See Configuring OAuth for Salesforce Reader. With the default value of False, when the specified Auth Token expires the application will halt and you will need to modify it to update the auth token before restarting. This setting is recommended only for development and testing, not in a production environment. When this property is False, you must specify Auth Token, Password, and Username. Set to True to renew the auth token automatically. In this case, leave Auth Token blank and set the Consumer Key, Consumer Secret, Password, Security Token, and Username properties.
Connection Profile Name	String		Used when the `Use connection profile` property is set to True . Select the name of the Salesforce Reader connection from the list.
Connection Retry Policy	String	retryInterval=30, maxRetries=3	With the default setting, if a connection attempt is unsuccessful, the adapter will try again in 30 seconds (`retryInterval`. If the second attempt is unsuccessful, in 30 seconds it will try a third time (`maxRetries`). If that is unsuccessful, the adapter will fail and log an exception. Negative values are not supported.
Consumer Key	String		When Auto Auth Token Renewal is `False`, this property is ignored and does not appear in Flow Designer. If autoAuthTokenRenewal is set to true, specify the Consumer Key (see Set Up Authorization on developer.salesforce.com).
Consumer Secret	com.webaction. security.Password		When Auto Auth Token Renewal is `False`, this property is ignored and does not appear in Flow Designer. If autoAuthTokenRenewal is set to true, specify the Consumer Secret (see Set Up Authorization on developer.salesforce.com).
Custom Objects Only	Boolean	False	By default, both standard and custom objects are included. Set to `true`to include only custom objects and exclude standard objects.
JWT Keystore Path	String		When Auto Auth Token is False or OAuth Authorization Flows is PASSWORD, this property is ignored and not visible in the Flow Designer. See Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration.
JWT Keystore Password	com.webaction. security.Password		When Auto Auth Token is False or OAuth Authorization Flows is PASSWORD, this property is ignored and not visible in the Flow Designer. See Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration.
JWT Certificate Name	String		When Auto Auth Token is False or OAuth Authorization Flows is PASSWORD, this property is ignored and not visible in the Flow Designer. See Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration.
Migrate Schema	Boolean	False	Do not change this setting. It is reserved for use by applications created using Auto Schema Conversion wizards (see Using Auto Schema Conversion).
Mode	Select list Automated Initial load Bulk load Incremental		Controls the basic behavior of the adapter. Use the default setting `Initial load` to load all existing data using force-rest-api 0.28 and stop. Set to `Bulk load` to load all existing data using the Salesforce Bulk APIand stop. The bulk load currently loads all existing data synchronously, similar to the Initial Load process. Set to `Incremental`to read new data continuously using force-rest-api 0.28. Set the value to `Automated` to perform both initial load and incremental load. See Salesforce Reader.
OAuth Authorization Flows	Enum	PASSWORD	When Auto Auth Token Renewal is `False`, this property is ignored and does not appear in Flow Designer. With the default value of PASSWORD, Salesforce Writer will authorize using OAuth 2.0 username and password (see Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 Username-Password Flow for Special Scenarios). In this case, you must specify values for the Consumer Key, Consumer Secret, Password, Security Token, and Username properties. Set to JWT_BEARER to authorize using OAuth 2.0 JWT bearer tokens instead (see Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration). In this case, you must specify the Consumer Key, JWT Certificate Name, JWT Keystore Password, JWT Keystore Path, and Username properties.
Objects			See sObjects.
Password	com.webaction. security.Password		When Auto Auth Token Renewal is False or OAuth Authorization Flows is JWT_BEARER, this property is ignored and not visible in the Flow Designer. When Auto Auth Token Renewal is set to `true`, specify the password for the specified username.
Polling Interval	String	5 min	This property controls how often the adapter reads from the source. By default, it checks the source for new data every five minutes. If there is new data, the adapter reads it and sends it to the adapter's output stream. If you encounter Salesforce REQUEST_LIMIT_EXCEEDED errors, you may need to increase this value or contact Snowflake to raise your API limits (see Salesforce Developer Limits and Allocations Quick Reference). The maximum value is `120 min`.
Return Picklist Data As	enum	api	Added in Striim 4.2.0.4. With the default value of `api`, data from picklist fields will be returned as the API name. Set to `value`to return the field value instead, or to `apiAndValueAsJson`to return both the API name and field value as a JSON string.
Security Token	String		When Auto Auth Token Renewal is False or OAuth Authorization Flows is JWT_BEARER, this property is ignored and not visible in the Flow Designer. When Auto Auth Token Renewal is set to `true`, specify the security token for the specified username (see Reset Your Security Tokenon help.salesforce.com).
SObjects	String	`%`	In the Flow Designer this property is shown as Objects. Do not modify this property when recovery is enabled for the application. With the default wildcard value `%`, all available objects associated with the specified API End Point will be read. To exclude standard objects, set Custom Objects Only to `true`. Alternatively, specify one or more objects to read, separating multiple objects with semicolons, for example, `Account; Business_C`. Note that any child objects (see Object Relationships Overview) must be specified explicitly, they are not included automatically when you specify the parent. The objects must be queryable. The account associated with the specified Auth Token must have View All Data permission for the objects.
Start Timestamp	String		By default, Salesforce Reader reads only new events. Optionally, specify the time (based on LastModifiedDate) from which to start reading older events in the format `YYYY-MM-DDTHH:MM:SS`. If the Salesforce organization's time zone is not the same as Striim's, convert the Salesforce start time to UTC (GMT+00:00) and include a `Z`at the end of the string. See SimpleDateFormatfor more information.
Thread Pool Size	Integer	5	Set this to match your Salesforce Concurrent API Request Limit (see API Request Limits and Allocations).
Use Connection Profile	Boolean		Set to True to use a connection profile instead of specifying the connection properties here. See Using connection profiles.
Username	String		When Auto Auth Token Renewal is `False`, this property is ignored and does not appear in Flow Designer. When Auto Auth Token Renewal is set to true, specify an appropriate username (see Add a Single Useron help.salesforce.com).

The output type is WAEvent.

Salesforce Reader WAEvent example

WAEvent{
  data: [
    "a025j000004He03AAC",
    "AA",
    100.0,
    "WA",
    "USD",
    null,
    null,
    null]
  metadata: {
    "LastModifiedDate":1646674721000,
    "TableName":"Business__c",
    "IsDeleted":false,
    "CustomObject":true,
    "OwnerId":"0055j000004c2D2AAI",
    "CreatedById":"0055j000004c2D2AAI",
    "OperationName":"INSERT",
    "CreatedDate":1646674721000,
    "attributes":
      "{type=Business__c,    
        url=\/services\/data\/v51.0\/sobjects\/Business__c\/a025j000004He03AAC}",
      "LastModifiedById":"0055j000004c2D2AAI",
      "SystemModstamp":"2022-03-07T17:38:41.000Z"},
  }
  userdata: null
  before: null
  dataPresenceBitMap: "HwA="
  beforePresenceBitMap: "AAA="
  typeUUID: {"uuidstring":"01ed7a69-eb41-3f71-8a71-8cae4cf129d6"}
};

Salesforce Reader runtime considerations

Salesforce Reader monitoring

The Salesforce reader monitors the following metrics:

Metric
Read timestamp (called as Last Event Modified Tiime)
Number of deletes
Number of inserts
Number of updates
CPU
CPU rate
CPU rate per node
Number of servers
Number of events seen per monitor snapshot interval
Source input
Source rate
Input
Input rate
Rate
Last event position
Latest activity
Read lag
I/O latency in ms
Table name