Asset-Compression-Manager/README.md

Asset Compression Manager (ACM) 
========================

## About

This tool exists to help compress, store, retrieve and validate media for games. The goal is to provide a simple workflow for checking assets for changes, generating new compressed versions based on those changes, storing those changed versions, and retrieving compressed files.

This tool is designed to work with an S3 compatible bucket storage provider.

## Requirements

- An S3 Compatible Storage Service
- Python 3.7+
- MozJpeg for jpeg compression
- OptiPNG for png optimization
- FFMpeg for video compression and audio decompression
- Opusenc for audio compression

## Using

### Configuring

ACM expects a configuration file specified with `--config` or as `acm-config.json` in the current directory.

The S3 compatible endpoint needs to be configured in the `s3` object of the configuration file.

```json
{
  "s3": {
    "secure": false,
    "host": "127.0.0.1:9000"
  }
}
```

- `secure` - specifies if __*https*__ protocol is used.
- `host` - is the \<hostname>\[:\<port>] for the S3 compatible endpoint.

Additionally the following environment variables need to be populated to interact with the S3 endpoint.

- `ACM_S3_ACCESS` - A valid S3 Access Key for the endpoint.
- `ACM_S3_SECRET` - A valid S3 Secret Key for the Access Key on the endpoint.

### Common Options

- `-c, --config`: The config file to read. Default file is `acm-config.json` in the current directory.
- `-s, --stdin`: Read the file list to process from stdin.
- `-p, --prefix`: The prefix to strip from the input. i.e. `acm.py -x test -p /tmp/data/storage/ check /tmp/data/storage/images/img1.jpg` => `images/img1.jpg`

### S3 Common Options

- `-x, --context`: The remote bucket to use. For `store` and `retrieve` operations it is `<value>-data`.

### S3 Commands

- list - list files in context
- check - find changes in provided files
- match - find matches in provided files
- update - update stored sha256sum values for provided files
- store - put files in data bucket
- retrieve - retrieve files from data bucket

### Listing Files

List all files in a bucket
```bash
$ ./acm.py list -x <bucket>
```

List all files while adding a prefix and stripping a suffix
```bash
$ ./acm.py --remove-prefix <prefix> list -x <bucket> --suffix <suffix>
```

List all files with sha256sum compatible output
```bash
$ ./acm.py --remove-prefix "/tmp/" --stdin list -x testing --suffix .json --sha256sum
```

Print out a sha256sum compatible check list

### Checking For Matches

Do a comparison of the remote bucket for files with a matching sha256sum value.

Process a list of files
```bash
$ ./acm.py --remove-prefix <prefix to strip> match -x <bucket> FILES... 
```

Process a list from stdin
```bash
$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> match -x <bucket>
```

### Checking For Changes

Do a comparison of the remote bucket for missing files or files with a mismatch in their sha256sum values.

Process a list of files
```bash
$ ./acm.py --remove-prefix <prefix to strip> check -x <bucket> FILES... 
```

Process a list from stdin
```bash
$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> check -x <bucket>
```

### Updating Metadata For Changed Files

Update the remote bucket with new metadata for the listed files. Calculates new sha256sum values.

Process a list of files
```bash
$ ./acm.py --remove-prefix <prefix to strip> update -x <bucket> FILES... 
```

Process a list from stdin
```bash
$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> update -x <bucket>
```

### Storing Files

Store the listed files in `<bucket>-data`.

Process a list of files
```bash
$ ./acm.py --remove-prefix <prefix to strip> store -x <bucket> FILES... 
```

Process a list from stdin
```bash
$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> store -x <bucket>
```

### Retrieving Files

Retrieve remote files matching listed files. Optionally place the downloaded files in a different destination.

Process a list of files
```bash
$ ./acm.py --remove-prefix <prefix to strip> retrieve -x <bucket> [-d <destination>] FILES... 
```

Process a list from stdin
```bash
$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> retrieve -x <bucket> [-d <destination>]
```

### Configuring Profiles

### Compressing Changed Assets

Compressing assets based on profiles
```bash

```

## Usage as a workflow

Compressing changed files and storing them for later use
1. Identify changed files
    - `find <file_location> -type f | ./acm.py --stdin -p <file_location> check -x aggressive`
1. Pass identified files to the compressor with a specific profile
    - `<found_files> | ./acm.py --stdin -p <file_location> compress -p aggressive -d /tmp/profile-aggressive/`
1. Store compressed assets in data bucket
    - `<compressed_files> | ./acm.py --stdin -p /tmp/profile/aggressive/ store -x aggressive`
1. Update metadata about files
    - `<found_files> | ./acm.py --stdin -p <file_location> update -x aggressive`
    
As a combined workflow to only compress and store changed assets
```bash
export TARGET_DIRECTORY=/mnt/e/data/files/
export CONTEXT=android
export PROFILE=aggressive
export PROFILE_TMP_DIR="/tmp/profile-${PROFILE}/"
export COMPRESSED_FILES="$(mktemp)"
# Compress changed assets
find "$TARGET_DIRECTORY" -type f | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" check -x "$CONTEXT" | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" compress --print-input-and-identity -p "$PROFILE" -d "$PROFILE_TMP_DIR" > "$COMPRESSED_FILES"
# Store compressed assets
find "$PROFILE_TMP_DIR" -type f | ./acm.py --stdin --remove-prefix "$PROFILE_TMP_DIR" --add-prefix "$TARGET_DIRECTORY" store -x "${CONTEXT}-data" > /dev/null
# Update Asset manifests
cat "$COMPRESSED_FILES" | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" update -x "$CONTEXT" --input-and-identity > /dev/null
```

As a combined workflow to download matching assets
```bash
TARGET_DIRECTORY=/mnt/e/data/files/
CONTEXT=android
PROFILE=aggressive
PROFILE_TMP_DIR="/tmp/profile-${PROFILE}"
find "$TARGET_DIRECTORY" -type f | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" match -x "$CONTEXT" --print-identity | ./acm.py --stdin retrieve -x "${CONTEXT}-data" -d "$PROFILE_TMP_DIR" > /dev/null`
```

## Contributing

## License
See [LICENSE.md](LICENSE.md)