Asset Compression Manager (ACM)

About

This tool exists to help compress, store, retrieve and validate media for games. The goal is to provide a simple workflow for checking assets for changes, generating new compressed versions based on those changes, storing those changed versions, and retrieving compressed files.

This tool is designed to work with an S3 compatible bucket storage provider.

Requirements

• An S3 Compatible Storage Service
• Python 3.7+
• MozJpeg for jpeg compression
• OptiPNG for png optimization
• FFMpeg for video compression and audio decompression
• Opusenc for audio compression

Using

Configuring

ACM expects a configuration file specified with --config or as acm-config.json in the current directory.

The S3 compatible endpoint needs to be configured in the s3 object of the configuration file.

{
  "s3": {
    "secure": false,
    "host": "127.0.0.1:9000"
  }
}

• secure - specifies if https protocol is used.
• host - is the <hostname>[:<port>] for the S3 compatible endpoint.

Additionally the following environment variables need to be populated to interact with the S3 endpoint.

• ACM_S3_ACCESS - A valid S3 Access Key for the endpoint.
• ACM_S3_SECRET - A valid S3 Secret Key for the Access Key on the endpoint.

Common Options

• -c, --config: The config file to read. Default file is acm-config.json in the current directory.
• -s, --stdin: Read the file list to process from stdin.
• -p, --prefix: The prefix to strip from the input. i.e. acm.py -x test -p /tmp/data/storage/ check /tmp/data/storage/images/img1.jpg => images/img1.jpg

S3 Common Options

• -x, --context: The remote bucket to use. For store and retrieve operations it is <value>-data.

S3 Commands

• list - list files in context
• check - find changes in provided files
• match - find matches in provided files
• update - update stored sha256sum values for provided files
• store - put files in data bucket
• retrieve - retrieve files from data bucket

Listing Files

List all files in a bucket

$ ./acm.py list -x <bucket>

List all files while adding a prefix and stripping a suffix

$ ./acm.py --remove-prefix <prefix> list -x <bucket> --suffix <suffix>

List all files with sha256sum compatible output

$ ./acm.py --remove-prefix "/tmp/" --stdin list -x testing --suffix .json --sha256sum

Print out a sha256sum compatible check list

Checking For Matches

Do a comparison of the remote bucket for files with a matching sha256sum value.

Process a list of files

$ ./acm.py --remove-prefix <prefix to strip> match -x <bucket> FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> match -x <bucket>

Checking For Changes

Do a comparison of the remote bucket for missing files or files with a mismatch in their sha256sum values.

Process a list of files

$ ./acm.py --remove-prefix <prefix to strip> check -x <bucket> FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> check -x <bucket>

Updating Metadata For Changed Files

Update the remote bucket with new metadata for the listed files. Calculates new sha256sum values.

Process a list of files

$ ./acm.py --remove-prefix <prefix to strip> update -x <bucket> FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> update -x <bucket>

Storing Files

Store the listed files in <bucket>-data.

Process a list of files

$ ./acm.py --remove-prefix <prefix to strip> store -x <bucket> FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> store -x <bucket>

Retrieving Files

Retrieve remote files matching listed files. Optionally place the downloaded files in a different destination.

Process a list of files

$ ./acm.py --remove-prefix <prefix to strip> retrieve -x <bucket> [-d <destination>] FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py --remove-prefix <prefix to strip> retrieve -x <bucket> [-d <destination>]

Configuring Profiles

Compressing Changed Assets

Compressing assets based on profiles

Usage as a workflow

Compressing changed files and storing them for later use

1. Identify changed files
   • find <file_location> -type f | ./acm.py --stdin -p <file_location> check -x aggressive
2. Pass identified files to the compressor with a specific profile
   • <found_files> | ./acm.py --stdin -p <file_location> compress -p aggressive -d /tmp/profile-aggressive/
3. Store compressed assets in data bucket
   • <compressed_files> | ./acm.py --stdin -p /tmp/profile/aggressive/ store -x aggressive
4. Update metadata about files
   • <found_files> | ./acm.py --stdin -p <file_location> update -x aggressive

As a combined workflow to only compress and store changed assets

export TARGET_DIRECTORY=/mnt/e/data/files/
export CONTEXT=android
export PROFILE=aggressive
export PROFILE_TMP_DIR="/tmp/profile-${PROFILE}/"
export COMPRESSED_FILES="$(mktemp)"
# Compress changed assets
find "$TARGET_DIRECTORY" -type f | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" check -x "$CONTEXT" | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" compress --print-input-and-identity -p "$PROFILE" -d "$PROFILE_TMP_DIR" > "$COMPRESSED_FILES"
# Store compressed assets
find "$PROFILE_TMP_DIR" -type f | ./acm.py --stdin --remove-prefix "$PROFILE_TMP_DIR" --add-prefix "$TARGET_DIRECTORY" store -x "${CONTEXT}-data" > /dev/null
# Update Asset manifests
cat "$COMPRESSED_FILES" | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" update -x "$CONTEXT" --input-and-identity > /dev/null

As a combined workflow to download matching assets

TARGET_DIRECTORY=/mnt/e/data/files/
CONTEXT=android
PROFILE=aggressive
PROFILE_TMP_DIR="/tmp/profile-${PROFILE}"
find "$TARGET_DIRECTORY" -type f | ./acm.py --stdin --remove-prefix "$TARGET_DIRECTORY" match -x "$CONTEXT" --print-identity | ./acm.py --stdin retrieve -x "${CONTEXT}-data" -d "$PROFILE_TMP_DIR" > /dev/null`

Contributing

License

See LICENSE.md