Have each message include a Key. This allows SQS queue to be FIFO. This also lets state management to be resilient to concurrent reads/writes for given key.

During ingestion, we can send off for stream processing, s3 for batch processing. Add third option to store in Dynamo.

Can be with a whitelist/blacklist for keys in JSON. Also specify primary keys and GSI keys and which fields should be used to construct them.

Create API&CLI to start/stop Athena jobs and download results.

Brainstorm:

Need to provide basic instructions on how to use DataSpray

Allow others to download and update `dst` cli tool. (e.g. JDeploy)

Allow creating DBT sub-project to store and execute Athena queries

Need a web editor for the different data formats most importantly for Json schema. VSCode may already have some plugin or built in.

Schemas to consider:

Jobs/tasks may produce logs. These need to be aggregated and viewable by user.

Use should be able to use a web IDE to edit code, compile, run, test and deploy a DataSpray project.

Most likely option: DevPod for remote compute. Includes K8S for scaling, vscode for web.

Tools:

Count up all charges for each customer. See Streaming Price sheet for details.

Currently autogen files are git ignored. Other ideas:

Allo each task to be called from the web using Lambda Function URL

Use the Java client to intercept all incoming requests, send them off to Umbrella and determine whether to allow/block/redirect requests as at Tomcat filter.

Create a simple Java client to intercept requests and send them off to Umbrella endpoint.

Umbreall is a first example use case of DataSpray Streaming.

Have a task accept requests as funciton urls and accept any http request to determine whether to allow/block/captcha. For now simply allow all requests and add it to s3 for later batch processing.

As simple client library to have Fetch calls intercept responses to block/captcha requests and display an appropriate modal. This is intended for single page apps.

It would be nice to have Apache Superset for batch processing. There are generally two major ways in doing this:

Allow each task to have state backed by dynamodb

Each processor should only be allowed to access inputs, state and send outputs.

There must also be a way for a processor to access external APIs, these processors' sole job should be to access the external API and be behind a queue for retry logic. External API may be restricted by domain with a domain verification flow to prevent spam.

Auto-gen a README.md file that includes an overview of the inputs outputs, state etc... based on configuration. Doesn't have to be README.md, probably better to be something else like ARCH.md or OVERVIEW.md

During event ingestion, optionally add IDML code to pre-process data before streaming it or storing in S3.

Allow stacks to be deployed independantly without any conflicting resources. Tasks:

Use 98.css and create dashbaord as win 98: