Usage guide

This is the public documentation page for the Data Shepherd Snowflake Native App. It covers what the app does, how to install and configure it, how to try it without connecting a table of your own, and the SQL you can run against it.

Every step below is run by you, in your own Snowflake account. There is no provider-side step and no external sign-up.

What Data Shepherd does

Data Shepherd is a data-quality engine that runs entirely inside your own Snowflake account. You point it at tables you already have, configure rules against them, and it records what it found. Every rule is a Snowflake stored procedure executing on your compute, reading only the tables you connect to the app, and writing its results back into the app’s own tables. Nothing leaves your account except through the two notification paths you choose to configure, email and webhook, and both run through the Snowflake notification service.

v1 is read-only. No rule writes to, updates, or deletes a row in the data you monitor. A rule is disabled, not deleted: a disabled rule no longer runs on its schedule, and it keeps its configuration and its run history.

The app ships a Streamlit user interface with eight pages, seven of them always in the sidebar: Dashboard, Rules, Executions, Alerts, Schedules, Reports and Settings. The eighth, Setup, is where a fresh install starts. It leaves the sidebar once the setup checks pass, and is reached from Settings - Data access and diagnostics after that.

Each of the 17 built-in reports on the Reports page downloads as CSV and JSON (the Download CSV and Download JSON buttons), and as PDF after you press Prepare PDF and then the Download PDF button. A report reads at most 5,000 rows per table; when it reaches that cap it says so above its figures, and every download covers only those rows.

Twenty-five rules ship in five families:

• Core checks - completeness, uniqueness, value range, data freshness, referential integrity, statistical profiling, table profiling, and quality scoring.
• Validation checks - categorical domain, conditional completeness, numeric-coercion detection, and fuzzy duplicate detection.
• Historical checks - these learn a statistical baseline from your own history and compare against it: distribution drift (Population Stability Index), period comparison (z-score), trend analysis, seasonal validation, cross-period consistency, and late-data detection.
• Pattern checks - regular-expression patterns, pattern comparison across segments, streaming and time-series pattern detection, density clustering, and advanced pattern analysis.
• Schema-drift check - snapshots the column set, data types, and nullability of each table you connect. The first check on a table records that baseline; every check after it compares against the previous snapshot and raises an alert when a column is added, dropped, or retyped.

Install and configure

1. Get the app. Open the Data Shepherd listing in Snowsight and choose Get. Snowsight asks you to approve the two account-level privileges the app requests and to name the application. This guide writes that name as DATASHEPHERD_APP throughout; substitute the name you chose if it differs. If your account’s policy meant those two privileges were not granted at install, grant them afterwards:

GRANT EXECUTE TASK ON ACCOUNT TO APPLICATION DATASHEPHERD_APP;
GRANT EXECUTE MANAGED TASK ON ACCOUNT TO APPLICATION DATASHEPHERD_APP;

Both privileges serve one thing: the app’s single serverless scheduling task. Your own roles need no task privilege of any kind for the app’s scheduling to work.

2. Grant the application role to the roles that will use the app. The role that installed the app can already open it. Nobody else can until you grant APP_USER to their role. Run this as the role that owns the application:

GRANT APPLICATION ROLE DATASHEPHERD_APP.APP_USER TO ROLE <analyst_role>;

If Snowsight shows “You don’t have access to view this app or it doesn’t exist” when someone opens Data Shepherd, that user’s role has not been granted APP_USER. Check who holds it:

SHOW GRANTS OF APPLICATION ROLE DATASHEPHERD_APP.APP_USER;

3. Connect the tables to monitor. The app requests one object reference, monitored_data_reference, which takes one or more tables and grants the app SELECT on each. Start on the app’s Setup page. While no table is connected, its Connect tables button opens Snowflake’s table picker. Once one table is connected that picker cannot add another, so Setup generates the SQL instead: enter one table (DB.SCHEMA.TABLE) or a whole schema (DB.SCHEMA), copy the SQL into a worksheet and run it as a role that can see and select from those tables. Then press I’ve run it - refresh, and Setup names the tables that are newly connected.

Setup lists every connected table with a Remove button that disconnects it after you confirm. If the app cannot disconnect it, Setup shows the SQL to run instead. The one-table SQL wraps this call to the app’s registration procedure:

CALL DATASHEPHERD_APP.CONFIG.REGISTER_REFERENCE(
  'monitored_data_reference',
  'ADD',
  SYSTEM$REFERENCE('TABLE', 'MY_DB.MY_SCHEMA.MY_TABLE', 'PERSISTENT', 'SELECT')
);

The reference accumulates, so call it once per table. The whole-schema SQL makes this same call for every plain upper-case base table in the schema. It skips already-connected tables, counts other names as skipped, and reports a missing or invisible schema. To disconnect one table by SQL, pass 'REMOVE' with that table’s reference alias (the alias column of SHOW REFERENCES IN APPLICATION DATASHEPHERD_APP, run as the role that owns the application); 'CLEAR' with an empty third argument disconnects all of them. The app requests no warehouse reference: it runs on whatever warehouse your own session is using, so read Warehouse sizing below before you start.

4. Turn scheduling on, if you want scheduled rules. A fresh install ships with the app’s scheduling suspended, so no rule fires on a schedule until you resume it once:

CALL DATASHEPHERD_APP.PUBLIC.MANAGE_DISPATCHER('RESUME');

'SUSPEND' turns it back off. Skip this step if you only want to run rules by hand.

5. Verify the install. One call runs the setup checks:

CALL DATASHEPHERD_APP.PUBLIC.CHECK_SETUP();

ready: true means the app is installed correctly and ready to use. If ready is false, the failing check’s message says exactly what to fix; fix it and re-run the call. The report holds eight checks, seven of which are blocking; the eighth reports whether the app’s scheduling is running right now and never withholds ready, because an unscheduled install is perfectly healthy.

Try it with sample data

Step 3 above needs a table of your own. If you do not have one yet, Data Shepherd can build a small, optional, app-owned set of sample tables instead - schema SAMPLE_DATA, created and owned entirely inside this application, with deliberate quality defects seeded into them, plus five rules that check those tables. It creates nothing in your own databases and needs no table binding.

Load it:

CALL DATASHEPHERD_APP.PUBLIC.LOAD_SAMPLE_DATA();

The two sample tables carry four seeded defects between them, and four of the five sample rules are each built to catch one; the fifth checks a column with nothing wrong in it, so it passes:

Run one of them (the Rules page lists each sample configuration’s ID):

CALL DATASHEPHERD_APP.PUBLIC.EXECUTE_RULE('da7a5e70-0000-4000-8000-000000000001', NULL);

Read what it found, through the same read-only PUBLIC view every rule writes to:

SELECT STATUS, RESULT
FROM   DATASHEPHERD_APP.PUBLIC.EXECUTION_HISTORY
WHERE  CONFIG_ID = 'da7a5e70-0000-4000-8000-000000000001'
ORDER  BY START_TIME DESC
LIMIT  1;

The five sample rules ship with an in-app alert destination (the webhook opt-in), so each sample rule that finds an issue records an alert on the Alerts page. It sends nothing until you configure a webhook notification integration - see Alert delivery below.

Remove it again when you are done, or once you have connected a real table:

CALL DATASHEPHERD_APP.PUBLIC.REMOVE_SAMPLE_DATA();

REMOVE_SAMPLE_DATA() drops both sample tables and deletes only the five sample rule configurations, together with their own execution history, alerts, and audit rows; nothing you have connected or configured yourself is touched.

The Setup page does all of this without SQL: one Load sample data and run the rules button that builds the sample and shows each sample rule’s verdict; once installed, a Run sample rules button to run them again; and, behind a confirm step, a Remove sample data button.

Example SQL

Check the install:

CALL DATASHEPHERD_APP.PUBLIC.CHECK_SETUP();

Run one rule. The first argument is the rule’s ID, which the Rules page shows; the second is always NULL in this app, which is single-tenant:

CALL DATASHEPHERD_APP.PUBLIC.EXECUTE_RULE('<config_id>', NULL);

Read the results. Every one of the app’s tables is exposed as a read-only view in the PUBLIC schema, so you can query, join, and build your own dashboards on them:

SELECT EXECUTION_ID,
       CONFIG_ID,
       RULE_ID,
       STATUS,
       START_TIME,
       RESULT
FROM   DATASHEPHERD_APP.PUBLIC.EXECUTION_HISTORY
ORDER  BY START_TIME DESC
LIMIT  20;

Open alerts (never acknowledged), newest first:

SELECT ALERT_ID,
       RULE_CONFIG_ID,
       ALERT_LEVEL,
       MESSAGE,
       CREATED_DATE
FROM   DATASHEPHERD_APP.PUBLIC.ALERT_LOG
WHERE  ACKNOWLEDGED_AT IS NULL
ORDER  BY CREATED_DATE DESC;

The full set of views is RULE_REGISTRY, RULE_CONFIGURATIONS, EXECUTION_HISTORY, ALERT_LOG, ALERT_NOTIFICATION_LOG, CONFIGURATION_AUDIT_LOG, REPORT_EXECUTION_QUEUE, HISTORICAL_BASELINES, SCHEMA_SNAPSHOTS, APP_EVENT_LOG, REPORT_SCHEDULES, REPORT_HISTORY, APP_SETTINGS, UI_DRAFTS, and DISPATCHER_STATE.

No sample data ships pre-loaded. Until you call LOAD_SAMPLE_DATA(), the app runs only against the tables you connect in step 3.

Warehouse sizing

Data Shepherd cannot choose its own compute. Streamlit-in-Snowflake offers no serverless or managed-compute option, so every click in the app runs on whatever warehouse your session is using.

Point a dedicated, small warehouse at the app. Use an XS warehouse rather than sharing a large or busy one. This keeps cost predictable and stops the app’s queries queuing behind unrelated workloads.

Set AUTO_SUSPEND to 300 seconds or more on that warehouse:

ALTER WAREHOUSE <your_warehouse> SET AUTO_SUSPEND = 300;

A short auto-suspend means any gap of more than a minute or two between clicks pays a resume cost on the next interaction. Five minutes covers realistic think-time between clicks without leaving the warehouse running indefinitely. A longer auto-suspend keeps the warehouse warmer across a session, giving fewer resume delays, but it burns more idle credits when nobody is using the app. That trade-off is entirely a warehouse policy choice on your side.

Scheduled rules are the exception: the app’s scheduling runs as a serverless task, which bills to your account rather than to a warehouse you size. Left resumed with nothing due and alert delivery set up, it measured about 0.12 credits per hour, roughly 2.8 credits per day, on 11 November 2026. Suspended, which is how the app ships, it costs nothing at all.

Alert delivery

An alert is recorded when a rule that has at least one alert destination finds an issue or fails: the row lands in ALERT_LOG and shows on the Alerts page. Destinations are set per rule - in the create/edit form on the Rules page, or under Alert destinations per rule on Settings. A rule with no destination still records its run in EXECUTION_HISTORY, but writes no alert; a run that found nothing wrong records no alert either. The five built-in sample rules ship with the in-app webhook destination, so running them records their alerts and sends nothing until a webhook integration is configured. Schema-drift alerts are the one exception: DETECT_SCHEMA_DRIFT records a drift alert for every connected table whose columns changed since its last snapshot, with no destination needed.

Delivering a recorded alert by email or webhook is optional, and needs a Snowflake notification integration that you create and grant to the app.

Email:

CREATE NOTIFICATION INTEGRATION DATASHEPHERD_ALERT_EMAIL TYPE = EMAIL ENABLED = TRUE;
GRANT USAGE ON INTEGRATION DATASHEPHERD_ALERT_EMAIL TO APPLICATION DATASHEPHERD_APP;
CALL DATASHEPHERD_APP.PUBLIC.DELIVER_ALERTS('DATASHEPHERD_ALERT_EMAIL');

Recipients must be verified account users. Snowflake rejects any recipient that is not a validated email address of a user in your account.

Settings also guides the webhook setup: under Notification integration, the “Connect a webhook” panel generates the script for Slack with your integration name filled in. You paste the URL or key yourself and an account admin (ACCOUNTADMIN) runs it in Snowsight. Press Send test message to confirm your own channel receives it.

For Slack, the panel’s first step recommends naming your Slack app Data Shepherd and setting its app icon (on the app’s Basic Information page) to the Data Shepherd icon, a 512 x 512 PNG of the white mark on navy that the panel shows and lets you download. Without an icon, Slack posts the alerts under its default app icon.

The Slack script uses the bare SNOWFLAKE_WEBHOOK_MESSAGE template, so the app can send each alert with a colour bar for its level. After running it, enter the integration’s name and choose Slack under Webhook service before you press Save integration names. An integration created with the older {"text": ...} template keeps working with the Older Slack text template choice; to move it to the colour bar, an account admin runs the line below while the service is set to Slack. The two must change together, or messages are lost without an error in the app.

ALTER NOTIFICATION INTEGRATION DATASHEPHERD_ALERT_WEBHOOK SET WEBHOOK_BODY_TEMPLATE = 'SNOWFLAKE_WEBHOOK_MESSAGE';

Webhook sends go to the single destination baked into a WEBHOOK notification integration you create; the app stores no URL of its own. Delivery is idempotent per channel, so re-running never re-sends an alert already delivered, and a delivery failure never affects rule execution.

What the app asks of your account, and what leaves it

Three entries are the app’s entire request:

The app performs no outbound network calls of its own. It declares no external-access integration, and the only way anything leaves your account is a notification you configure yourself: email through Snowflake’s own send-email function, or a webhook through Snowflake’s notification service. There is no third path, so with neither notification integration configured, nothing leaves the account at all.

The app stores no credential, token, password, or webhook URL. Where a Snowflake notification integration is involved, the app holds only that integration’s name; the destination URL and any secret in it live in your own NOTIFICATION INTEGRATION object, which you create and own. Nothing the app stores is visible to the provider, and the provider receives no event data from your account.

Support

For help with this app, see the provider contact on its Marketplace listing.