Skip to main content

Building pipelines with Microsoft Dynamics 365 Reader

You can read from Microsoft Dynamics 365 using the Microsoft Dynamics 365 Reader and write to any target supported by Striim. The recommended approach is to use Automated mode, which performs an initial load of existing data and then automatically transitions to continuous incremental replication when the initial load completes—all within a single application. For foundational concepts about pipeline phases, see Pipelines.

The Microsoft Dynamics 365 Reader supports three modes:

  • Automated (recommended): Performs initial load, then automatically transitions to incremental replication. Use this mode for most pipelines.

  • Initial Load: Performs a one-time snapshot of selected entities. Use this mode only when you need to run initial load separately from incremental replication.

  • Incremental Load: Replicates new or changed records on a polling interval. Use this mode only when initial load has already been completed separately.

You can use automated pipeline wizards to build Microsoft Dynamics 365 pipelines that Striim manages end-to-end, or you can manually create an application and manage the lifecycle yourself.

Using an automated pipeline wizard: If you want to build near-real-time pipelines from Microsoft Dynamics 365 and write to a supported target, we recommend using an automated pipeline wizard with a Microsoft Dynamics 365 Reader source. These wizards perform the following steps automatically:

  • Create a single application using Automated mode that handles both initial load and continuous incremental replication.

  • Create a schema and tables in the target that match the entities to be synced from Microsoft Dynamics 365.

  • Copy existing data from Microsoft Dynamics 365 to the target during the initial load phase.

  • When initial load completes, automatically transition to incremental replication to capture new or changed records using the configured watermark field and polling interval.

Not using an automated pipeline wizard: If your use case or policies do not allow using an automated pipeline wizard, you can manually create and configure an application using Flow Designer or TQL. Set the Mode property to Automated to handle both initial load and continuous replication in a single application.

  • Before performing initial load: Identify a stable, monotonically increasing field for each entity (for example, a last-modified timestamp) to use as the Incremental Load Marker.

  • Schema creation: Enable Migrate Schema to create target tables automatically, or pre-create target tables using native or third-party utilities.

  • Configure for continuous replication: Set the Incremental Load Marker field and Polling Interval. Configure the target for upsert semantics using appropriate key columns.

  • Start Position: If you need to resume replication from a specific point, provide the last successful watermark value as the Start Position.

You can create applications using Flow Designer, TQL, or Striim's REST API.

Pre-requisite - The initial setup and configuration you do in Microsoft Dynamics 365 applications are described in the Initial setup section.

Create a Dynamics 365 Reader application using the Flow Designer

This procedure outlines how to use Striim's Flow Designer to build and configure data pipelines. Flow Designer enables you to visually create applications with minimal or no coding.

  1. Go to the Apps page in the Striim UI and click Start from scratch.

  2. Provide the Name and Namespace for your app. The namespace helps organize related apps.

  3. In the component section, expand Sources, and enter a keyword such as Dynamics 365 Reader in the search field to filter available sources.

  4. Select the desired source (Microsoft Dynamics 365).

  5. In the properties panel, provide the properties for the reader. Set Mode to Automated to perform initial load followed by continuous incremental replication.

    ms-dynamics-365-reader.png

  6. Click Save to complete the properties configuration.

  7. You can drag and drop processors, enrichers, and targets to complete your pipeline logic.

  8. Once done, deploy and start the application to begin data flow.

Create a Dynamics 365 Reader application using TQL

The following sample TQL uses Striim's Dynamics 365 Reader to read from the Sales Edition. Note that Mode is set to Automated, which performs initial load and then automatically transitions to incremental replication.

 CREATE SOURCE SourceName USING Global.MicrosoftDynamics365Reader (
    PollingInterval: '5m',
    ClientSecret: '<ClientSecret>',
    RefreshToken: '<RefreshToken>',
    TenantId: 'TenantID',
    AdfsServer: '',
    startPosition: '%=-1',
    Tables: '%',
    ClientId: 'ClientId',
    ConnectionPoolSize: 20,
    connectionProfileName: '',
    FetchSize: 5000,
    ConnectionTimeout: '300',
    Edition: 'Sales',
    Mode: 'Automated',
    useConnectionProfile: false,
    IncrementalLoadMarker: 'ModifiedOn',
    OrganizationUrl: 'URL',
    MigrateSchema: true,
    ThreadPoolCount: 10
  )
  OUTPUT TO DataStream;