Back to Sift Support Plans Submit a request

How can we help?

Ask a question or search our knowledge base:

Route Backtesting

Introduction

Route Backtesting provides the ability to test rules and business logic against historical data. Results of a Backtest can be explored in a series of familiar charts and visualizations and exported for further analysis. Use this feature to make data-driven decisions and optimize Workflow automation with confidence! 

result1.png

Feature Overview 

Route Backtesting: In-isolation

Route Backtesting will test Workflow logic in isolation, meaning the Backtest will scan all users/entities that triggered the Workflow during the specific time period and return all those that matched the logic of the route being tested. Users/entities will not be filtered out based on the logic in the routes above the route being tested. Therefore, testing the same criteria on any route will yield the same results.

Users/entities will be matched based on the values of their attributes at the time that they originally triggered this Workflow, not the current values of their attributes.

Reset Criteria & Reset Draft 

Any Workflow route that is edited will show a Reset Criteria button which will reset the route to the current Published version of that route. You will also see the criteria of the Published version of the route listed in small gray text beneath the Draft criteria. You can toggle between showing and hiding these with the Hide/Show Published button. 

As soon as any edit is made to the Workflow Draft, a Reset Draft option will appear next to Save Draft. This will reset all of the edits made to the Workflow draft and revert it to the current Published version of the Workflow. 

reset_criteria.png

CSV Download

Route Backtesting results can be downloaded to a CSV up to the latest 10k records. This export will include the user ID, entity ID, latest decision type, latest decision source, score, and timestamp. Use this export to investigate users/entities that may have been caught by a Backtest. 

Usage Guide

How to Enable/Disable Workflows Backtesting

Once Backtesting is enabled, a new set of permissions will appear in the Sift Console under Account Settings > Team > Roles > Automations. These permissions will be automatically available for any users with an Administrator or Manager role. 

  • These permissions can also be enabled for any custom roles you have configured for your account.
  • Backtesting permissions are specific to each abuse product you have enabled for your Sift Account.
  • Backtesting permissions require the Draft Workflow Edits and View Workflow Insights permissions to be enabled.
  • The Reset Draft and Reset Criteria features only require the Draft Workflow Edits permissions, and can be managed independently of Backtesting.

How to Run a Backtest

  1. Access the Sift Console and navigate to the Automations tab in the menu sidebar.
  2. Select any running Workflow (tests can only be executed on the drafts of running Workflows). Select the Workflow and then enter Edit Mode.
  3. Once in Edit Mode, you will see that Workflow Route Metrics are now visible. These metrics and charts are the same as those shown in View Mode, and they represent the real-time performance of the Published, running Workflow.
  4. To run a backtest for a Workflow Route, first make an edit to that Route so that it is different from the published version of that route. Alternatively, you can create a new route. Once done, you will see the Run Test option appear as well as a date picker tool.
  5. Select a time range (up to the last 30 days) and click Run Test. You will see an alert indicating that the test is running. 

  6. You can navigate away from the page without impacting the Backtest. If you are still in the Sift Console when the Backtest completes, you will see a green alert banner at the top of the screen that will navigate you back to Edit Mode to view the results. 

  7. Once the Backtest has completed, open the Route Metrics drawer below the Workflow Route to access the results. You will see a dropdown allowing you to toggle between metrics for the Current Route or the Test Route. 

FAQ

Can I run a Backtest on any running Workflow?

This feature is available for all running Workflows and Abuse Types with the exception of Multi-Abuse Type Workflows.

Are there any limitations to the kind of logic I can test in a route?

Backtests are supported up to the following limits:

  • Up to 30 unique fields (if you use the same criteria with different operators/values, these will only count as 1 towards the max 30 limit)
  • Up to 2000 Workflow List terms (across any number of Lists)

How long will the results of my Backtest be available? Can other users see it?

Backtest results are only visible to the user that ran the backtest. The results will be cached for 24 hours. The cache will be cleared if the route is edited or another backtest is run.

What if I navigate away from the page while a Backtest is running?

Navigating away from the Workflows page while a Backtest is running will not impact the results. If you are still in the Sift Console when the Backtest completes, you will see a green alert banner with a link that will take you to the results of the test. If you leave the Sift Console, you will still be able to access the results of your Backtest once completed when you return to that route in Edit Mode for that Workflow.

How long does it take to run a Backtest?

Backtest queries should complete in roughly 1 minute.

Are there any limits to the number of Backtests I can run?

The following rate limits are in place:

  • 1 active Backtest at a time per user
  • 5 active Backtests at a time per Sift account

Are there any functional limitations while a Backtest is running?

While a Backtest is being run, the following limitations are in place:

  • The Workflow cannot be edited, paused, published, or deleted by any user. An error will be thrown asking the user to wait up to 1 minute and try again.
  • Only one test can be run at a time per Workflow. Two users can run a Backtest on different Workflows at the same time.

Why does Backtesting only work for active running Workflows?

The data that is queried when running a backtest is recorded in real-time from actual Workflow runs. For this reason, the data that can possibly be returned by a backtest will be limited to the time range during which the Workflow has been live and running. Real-time run data is available for Backtests roughly 15 minutes after it has been processed by Sift.

What if I add a new field?

A new field (reserved or custom) can be used in a backtest roughly 30 minutes after it has first been sent to Sift in a user event. If used in a backtest, note that users in Sift will only have values for that field as long as it has been sent to Sift. In other words, if you wish to run a backtest on the past 2 weeks using a new field, you will need to have sent that field for 2 weeks for the result to be meaningful.

 

 

 

 

Have more questions? Submit a request