Asset Compression Manager (ACM)

About

This tool exists to help compress, store, retrieve and validate media for games. The goal is to provide a simple workflow for checking assets for changes, generating new compressed versions based on those changes, storing those changed versions, and retrieving compressed files.

This tool is designed to work with an S3 compatible bucket storage provider.

Requirements

• An S3 Compatible Storage Service
• Python 3.7+
• MozJpeg for jpeg compression
• OptiPNG for png optimization
• FFMpeg for video compression and audio decompression
• Opusenc for audio compression

Using

Configuring

ACM expects a configuration file specified with --config or as acm-config.json in the current directory.

The S3 compatible endpoint needs to be configured in the s3 object of the configuration file.

{
  "s3": {
    "secure": false,
    "host": "127.0.0.1:9000"
  }
}

• secure - specifies if https protocol is used.
• host - is the <hostname>[:<port>] for the S3 compatible endpoint.

Additionally the following environment variables need to be populated to interact with the S3 endpoint.

• ACM_S3_ACCESS - A valid S3 Access Key for the endpoint.
• ACM_S3_SECRET - A valid S3 Secret Key for the Access Key on the endpoint.

Common Options

• -c, --config: The config file to read. Default file is acm-config.json in the current directory.
• -x, --context: The remote bucket to use. For store and retrieve operations it is <value>-data.
• -s, --stdin: Read the file list to process from stdin.
• -p, --prefix: The prefix to strip from the input. i.e. acm.py -x test -p /tmp/data/storage/ check /tmp/data/storage/images/img1.jpg => images/img1.jpg

Listing Files

List all files in a bucket

$ ./acm.py -x <bucket> list

List all files while adding a prefix and stripping a suffix

$ ./acm.py -x <bucket> -p <prefix> list --suffix <suffix>

List all files with sha256sum compatible output

$ ./acm.py --context testing --prefix "/tmp/" --stdin list --suffix .json --sha256sum

Print out a sha256sum compatible check list

Checking For Matches

Do a comparison of the remote bucket for files with a matching sha256sum value.

Process a list of files

$ ./acm.py -x <bucket> -p <prefix to strip> match FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py -x <bucket> -p <prefix to strip> match

Checking For Changes

Do a comparison of the remote bucket for missing files or files with a mismatch in their sha256sum values.

Process a list of files

$ ./acm.py -x <bucket> -p <prefix to strip> check FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py -x <bucket> -p <prefix to strip> check

Updating Metadata For Changed Files

Update the remote bucket with new metadata for the listed files. Calculates new sha256sum values.

Process a list of files

$ ./acm.py -x <bucket> -p <prefix to strip> update FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py -x <bucket> -p <prefix to strip> update

Storing Files

Store the listed files in <bucket>-data.

Process a list of files

$ ./acm.py -x <bucket> -p <prefix to strip> store FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py -x <bucket> -p <prefix to strip> store

Retrieving Files

Retrieve remote files matching listed files. Optionally place the downloaded files in a different destination.

Process a list of files

$ ./acm.py -x <bucket> -p <prefix to strip> retrieve [-d <destination>] FILES... 

Process a list from stdin

$ find /tmp -name '*.jpg' | ./acm.py -x <bucket> -p <prefix to strip> retrieve [-d <destination>]

Configuring Profiles

Compressing Changed Assets

Contributing

License

See LICENSE.md