Scribe Insight: How to Implement Long Running Batch Jobs in Scribe

For organizations that use multiple systems, there often is a need to synchronize company data, such as accounts and invoices, between different systems. Scribe Insight is a powerful and viable integration tool designed to save organizations from complex and often costly custom integrations. With the intuitive visual tools available in Scribe Insight you can quickly build, maintain and add integrations without having to write much code, if any at all.

We were recently asked by a client to mass update the statuses on invoices in Microsoft Dynamics CRM Online from Dynamics GP. The number of invoices that needed to be updated every day was in the tens of thousands. We found limited documentation on how to run long batch jobs in Scribe and decided to write this article to provide guidance on best practices for integrating a large amount of data.

There are various ways you can implement the integrations to address the above requirements in Scribe. We will review each option and conclude with the best approach.

Using the Timed Schedule Option in the Integration

One way to implement a batch job is to create a Scribe integration package that reads from the Dynamics GP database and updates CRM. You would create the integration in the Scribe Console and set up a time schedule for it by selecting either Time or Query as shown in the screenshot above. This option seems to be viable considering that SQL jobs are set up in a similar way in SQL Server. However, all Time and Query based integrations in Scribe can use only one message processor. This means the Scribe server can process only one scheduled integration package at a time. If there are multiple scheduled packages and one is taking excessively long time, this can become a major bottleneck.

Using the Queue / Publisher Integration

Integration Setup:

Publisher Setup:

The queue-based integration in Scribe picks up the XML messages from the message queue and processes them. The advantage of this approach is that, depending on the license of Scribe, up to 63 message processors can be leveraged to process the queued up messages at the same time. This allows for much quicker processing time. Since queue-based integration can hog the message queue if there are lots of source data, we recommend that you schedule it overnight when the volume from other integrations is low. To schedule the publisher to run once overnight, set the Active Range to be a time period overnight, for example, from 1 am to 3 am. Then set the frequency to be 1 day.

Conclusion

We have outlined some tips and considerations when building a batch job in Scribe. For a batch job to process a large data set, the best approach is to use the publisher/queue integration type as we outlined here. Please consult the Scribe documentation for instructions on setting up publishers and integrations if you are new to Scribe.

Please contact FMT Consultants if you need assistance with Scribe. We’d be happy to help.

Written by:
Aram Jung, Senior Microsoft Dynamics GP Consultant
FMT Consultants