As simple client library to have Fetch calls intercept responses to block/captcha requests and display an appropriate modal. This is intended for single page apps.

It would be nice to have Apache Superset for batch processing. There are generally two major ways in doing this:

Allow each task to have state backed by dynamodb

Each processor should only be allowed to access inputs, state and send outputs.

There must also be a way for a processor to access external APIs, these processors' sole job should be to access the external API and be behind a queue for retry logic. External API may be restricted by domain with a domain verification flow to prevent spam.

Auto-gen a README.md file that includes an overview of the inputs outputs, state etc... based on configuration. Doesn't have to be README.md, probably better to be something else like ARCH.md or OVERVIEW.md

During event ingestion, optionally add IDML code to pre-process data before streaming it or storing in S3.

Allow stacks to be deployed independantly without any conflicting resources. Tasks:

Use 98.css and create dashbaord as win 98:

Java is slow to start on Lambda and CLI. Need to use native GraalVM images to speed up. Specifically on Lambda base stack for all of DS lambdas including authorizer, control, ingest lambdas

GraalVM native image for lambda for customer stream processors. Most likely an optional thing, would need to ensure customer compiles for right architecture. Note: GH Actions don't support ARM out of the box.

Create a native image for each platform in addition to the java cli tool to speed up runtime

A processor may call itself in a loop. Need to prevent or warn if this sort of loop is detected

Ideas:

Have each message include a Key. This allows SQS queue to be FIFO. This also lets state management to be resilient to concurrent reads/writes for given key.

During ingestion, we can send off for stream processing, s3 for batch processing. Add third option to store in Dynamo.

Can be with a whitelist/blacklist for keys in JSON. Also specify primary keys and GSI keys and which fields should be used to construct them.

Create API&CLI to start/stop Athena jobs and download results.

Brainstorm:

Need to provide basic instructions on how to use DataSpray

Allow others to download and update `dst` cli tool. (e.g. JDeploy)

Allow creating DBT sub-project to store and execute Athena queries

Need a web editor for the different data formats most importantly for Json schema. VSCode may already have some plugin or built in.

Schemas to consider:

Jobs/tasks may produce logs. These need to be aggregated and viewable by user.

Use should be able to use a web IDE to edit code, compile, run, test and deploy a DataSpray project.

Most likely option: DevPod for remote compute. Includes K8S for scaling, vscode for web.

Tools:

Count up all charges for each customer. See Streaming Price sheet for details.