Language File Download and Compare Docker

This Docker container downloads language JSON files from a specified URL, compares them with existing files, and automatically commits and pushes changes to a Git repository.

Features

• Downloads language files for multiple languages
• Compares new files with existing ones
• Generates diff files for changes
• Automatically commits and pushes changes to Git
• Supports 15 languages: en, zh-TW, ja, ko, fr, de, es, it, nl, pl, pt, ru, tr, vi, zh-CN

Setup

1. Copy env.example to .env and configure your Git repository:

   cp env.example .env
   

2. Edit .env with your Git repository details and authentication method:

   Method 1: Personal Access Token (Recommended)

   GIT_REPO_URL=https://github.com/yourusername/yourrepo.git
   GIT_USERNAME=yourusername
   GIT_EMAIL=your.email@example.com
   GIT_BRANCH=main
   GITHUB_TOKEN=ghp_your_personal_access_token_here
   

   Method 2: SSH Key

   GIT_REPO_URL=git@github.com:yourusername/yourrepo.git
   GIT_USERNAME=yourusername
   GIT_EMAIL=your.email@example.com
   GIT_BRANCH=main
   

3. Build and run the container:

   docker-compose up --build
   

GitHub Authentication

Personal Access Token (Recommended)

1. Go to GitHub → Settings → Developer settings → Personal access tokens
2. Generate a new token with repo scope
3. Add the token to your .env file as GITHUB_TOKEN
4. Use HTTPS URLs: https://github.com/username/repo.git

SSH Key Authentication

1. Generate SSH key pair: ssh-keygen -t ed25519 -C "your_email@example.com"
2. Add public key to GitHub → Settings → SSH and GPG keys
3. Use SSH URLs: git@github.com:username/repo.git
4. Mount your private key in docker-compose (see Advanced Configuration)

Local Testing

For local testing on your machine:

1. Install dependencies:

   pip install -r requirements.txt
   

2. Create a .env file with your Git configuration (same as above)

3. Run the script locally:

   python download_and_compare.py
   

The script will automatically load environment variables from the .env file using python-dotenv.

Usage

The container will:

1. Download language files from the configured URL
2. Compare with existing files
3. Generate diff files for any changes
4. Commit and push changes to the Git repository

Files Generated

• {lang}.json - Current language files
• {lang}.diff - Diff files showing changes (only when changes detected)

Environment Variables

• GIT_REPO_URL - URL of the Git repository to push to
• GIT_USERNAME - Git username for commits
• GIT_EMAIL - Git email for commits
• GIT_BRANCH - Git branch to use (defaults to 'main')

Running Manually

To run the script manually outside of Docker:

python download_and_compare.py

The script will automatically load your .env file for local testing.

Troubleshooting

Git Authentication Issues

If you encounter authentication errors like "could not read Username", try these steps:

1. Verify your .env file contains all required variables:

   GIT_REPO_URL=https://github.com/yourusername/yourrepo.git
   GIT_USERNAME=yourusername
   GIT_EMAIL=your.email@example.com
   GITHUB_TOKEN=ghp_your_token_here
   

2. Test Git configuration in the container:

   docker-compose run --rm download-compare python test_git_config.py
   

3. Check Personal Access Token permissions:

   • Ensure the token has repo scope
   • Verify the token hasn't expired
   • Check that the token belongs to the correct GitHub account

4. Verify repository access:

   • Ensure your GitHub account has access to the repository
   • Check if the repository is private and requires authentication

Common Issues

• "No such device or address": Usually means Git is trying to prompt for credentials interactively
• "Authentication failed": Check your Personal Access Token and repository permissions
• "Repository not found": Verify the repository URL and your access permissions
• "Updates were rejected because the remote contains work": The script now automatically handles this by:
  1. Stashing local changes
  2. Pulling latest changes from remote
  3. Applying stashed changes
  4. Retrying the push

Conflict Resolution

The script automatically handles common Git conflicts:

• Remote ahead of local: Automatically pulls latest changes before pushing
• Merge conflicts: Uses git stash to preserve changes, then pulls and reapplies
• Unrelated histories: Uses --allow-unrelated-histories flag for initial setup