GraalVM native image for lambda for customer stream processors. Most likely an optional thing, would need to ensure customer compiles for right architecture. Note: GH Actions don't support ARM out of the box.

Create a native image for each platform in addition to the java cli tool to speed up runtime

A processor may call itself in a loop. Need to prevent or warn if this sort of loop is detected

Ideas:

Have each message include a Key. This allows SQS queue to be FIFO. This also lets state management to be resilient to concurrent reads/writes for given key.

During ingestion, we can send off for stream processing, s3 for batch processing. Add third option to store in Dynamo.

Can be with a whitelist/blacklist for keys in JSON. Also specify primary keys and GSI keys and which fields should be used to construct them.

Create API&CLI to start/stop Athena jobs and download results.

Brainstorm:

Need to provide basic instructions on how to use DataSpray

Allow others to download and update `dst` cli tool. (e.g. JDeploy)

Allow creating DBT sub-project to store and execute Athena queries

Need a web editor for the different data formats most importantly for Json schema. VSCode may already have some plugin or built in.

Schemas to consider:

Jobs/tasks may produce logs. These need to be aggregated and viewable by user.

Use should be able to use a web IDE to edit code, compile, run, test and deploy a DataSpray project.

Most likely option: DevPod for remote compute. Includes K8S for scaling, vscode for web.

Tools:

Count up all charges for each customer. See Streaming Price sheet for details.

Currently autogen files are git ignored. Other ideas:

Allo each task to be called from the web using Lambda Function URL

Use the Java client to intercept all incoming requests, send them off to Umbrella and determine whether to allow/block/redirect requests as at Tomcat filter.

Create a simple Java client to intercept requests and send them off to Umbrella endpoint.

Umbreall is a first example use case of DataSpray Streaming.

Have a task accept requests as funciton urls and accept any http request to determine whether to allow/block/captcha. For now simply allow all requests and add it to s3 for later batch processing.

As simple client library to have Fetch calls intercept responses to block/captcha requests and display an appropriate modal. This is intended for single page apps.

It would be nice to have Apache Superset for batch processing. There are generally two major ways in doing this:

Allow each task to have state backed by dynamodb

Each processor should only be allowed to access inputs, state and send outputs.

There must also be a way for a processor to access external APIs, these processors' sole job should be to access the external API and be behind a queue for retry logic. External API may be restricted by domain with a domain verification flow to prevent spam.

Auto-gen a README.md file that includes an overview of the inputs outputs, state etc... based on configuration. Doesn't have to be README.md, probably better to be something else like ARCH.md or OVERVIEW.md