During ingestion, we can send off for stream processing, s3 for batch processing. Add third option to store in Dynamo.

Can be with a whitelist/blacklist for keys in JSON. Also specify primary keys and GSI keys and which fields should be used to construct them.

Create API&CLI to start/stop Athena jobs and download results.

Brainstorm:

Count up all charges for each customer. See Streaming Price sheet for details.

Need to provide basic instructions on how to use DataSpray

Jobs/tasks may produce logs. These need to be aggregated and viewable by user.

Ideas:

Each processor should only be allowed to access inputs, state and send outputs.

There must also be a way for a processor to access external APIs, these processors' sole job should be to access the external API and be behind a queue for retry logic. External API may be restricted by domain with a domain verification flow to prevent spam.

It would be nice to have Apache Superset for batch processing. There are generally two major ways in doing this:

Create a native image for each platform in addition to the java cli tool to speed up runtime