Back
Object Storage
CDNetworks Documentation Object Storage Tools File Rsync Tool

File Rsync Tool

Last update：2024-06-19 10:21:12

The File Synchronization Tool, leveraging CDNetworks Object Storage APIs, synchronizes local data to CDNetworks Object Storage while preserving the original directory structure. This tool is ideal for migrating existing files. For incremental file synchronization, utilize the API upload interface.

Download Links

Before Starting

  1. Ensure Java (JDK 1.6 or newer) is installed.
  2. Place the configuration file and the synchronization tool in the same directory.

How to Use

  1. Navigate to the directory containing the wcs-rsync-hash tool (e.g., F:\wcs-rsync for Windows or /home/tool/wcs-rsync for Linux).
  2. Edit the conf.json file according to your needs.
  3. To start the service:
    • For Windows: While holding the Shift key, right-click in the folder window and select “Open command window here.”
    • Execute the command: java -jar wcs-rsync-hash-xxx.jar conf.json.
    • If any files fail to sync, rerun the command to attempt re-synchronization.
  4. To view a list of failed uploads, execute: java -jar wcs-rsync-hash-xxx.jar -listfailed conf.json.
    • The results will be saved in the log file located in the tool directory.
  5. To force a re-upload of all files, execute: java -jar wcs-rsync-hash-xxx.jar -igsync conf.json.
  6. In case of errors during the synchronization process, consult the wcs-rsync-hash.log file in the tool directory for ERROR level logs to diagnose issues.

Available Settings in the Configuration File

The following are some adjustable settings in the conf.json file. You can modify these settings based on your specific needs.

Section Item Requirement Description
Basic Authentication accessKey Required A unique identifier to access your cloud storage account. Obtain it from the “Security Settings - API Information Management - AccessKey Management” section on console.
secretKey Required A private key paired with your Access Key for identity verification. Also obtained from the “Security Settings - API Information Management - AccessKey Management” section on console.
Domain Settings uploadDomain Required The specific domain used for uploading files. Find this in the “Bucket - Overview - Upload Domain”.
mgrDomain Required The domain used by the tool for backend operations such as file checksum validation. Also found in the “Bucket - Overview - Manage Domain”.
Synchronization Mode syncMode Required Determines the synchronization strategy:
- 0: Single bucket, multi-directory mode. Requires specifying Bucket and Sync Dir; Key Prefix is optional.
- 1: Multi-bucket, multi-directory mode. In this case, Bucket And Dir is mandatory.
Space and Path bucket Optional Specifies the bucket where files will be saved. Mandatory when Sync Mode is set to 0.
syncDir Optional The local path of the files to be uploaded, such as /data, can be specified. Multiple paths can be configured, separated by "|", for example, D:/pic-2|D:/rsync3.
When using the command-line synchronization tool, this field is required if syncMode is set to 0.
Please note: For both Linux and Windows systems, use / as the path separator when configuring the local path. Paths on Windows systems should include the drive letter (e.g., C:/data).
keyPrefix Optional A designated prefix is appended to files uploaded to the Object Storage. Multiple configurations are allowed, and they align one-to-one with the paths defined in syncDir. The default is blank. For example:
1. If keyPrefix is configured as data/ and the uploaded file is 1.apk, this file will be saved in the cloud storage as data/1.apk. This means a new folder named data is created in the Object Storage, and 1.apk resides inside this folder.
2. If keyPrefix is set to data and the file being uploaded is 1.apk, this file will be saved in the cloud storage as data1.apk.
3. If syncDir is configured as D:/rsync1|D:/rsync2|D:/rsync3 and keyPrefix is set to test1/|test2/, files (and directories) from rsync1 will be stored under the test1 directory in the cloud storage, files (and directories) from rsync2 will be stored under the test2 directory, and files (and directories) from rsync3 will be stored at the root directory. If there are more keyPrefix configurations than syncDir paths, the additional keyPrefix settings will not apply, and only the first several directories will be considered.
Target Space and Local Path bucketAndDir Optional This specifies the target bucket and local path from which the file will be uploaded. This parameter is required when syncMode is configured as 1. Example:
bucket1|D:/dir1,D:/dir2|prefix1,prefix2/;bucket2|D:/dir3,D:/dir4
Please note:
1. Each bucket can be configured with multiple local paths. Local paths can target directories or files, and file names must include their suffixes.
2. Local paths and prefixes are matched one-to-one. The application of prefixes follows the same rules as the keyPrefix parameter.
3. Bucket names, local paths, and prefixes are separated by "
Performance-Related Settings threadNum Optional Sets the number of files uploaded concurrently, range 1-100, default 1.
sliceThreshold Optional Enables chunked upload for files exceeding this value, measured in MB, range 1-100MB, default 4MB.
sliceThread Optional Number of concurrent threads for chunked uploads, range 1-100, default 5.
sliceBlockSize Optional Size of individual blocks during chunked uploads, range 4M-1024M, must be a multiple of 4, default 4M.
sliceChunkSize Optional Size of each chunk, measured in KB, range 1024-1048576KB.
Synchronization Behavior deletable Optional Determines the behavior of cloud storage files when local files are deleted, 0 to retain, 1 for synchronized deletion, default 0.
maxRate Optional Upload speed limit, measured in KB/s, 0 indicates no limit.
taskBeginTime Optional Scheduled task start time, format hh:mm:ss.
taskEndTime Optional Scheduled task end time, format hh:mm:ss.
Data Validation isCompareHash Optional Enables or disables file hash comparison, default 1 (enabled).
countHashThreadNumHash Optional Concurrency level for hash value calculations, range 1-100, default 1.
compareHashThreadNumHash Optional Concurrency level for hash value comparisons, range 1-100, default 1.
compareHashFileNumHash Optional Number of files whose hashes are compared in each batch from cloud storage, range 1-2000, default 100.
Filtering Rules minFileSize Optional Files smaller than this value will not participate in the upload, measured in bytes, default 0 (no restriction).
overwrite Optional Determines whether to overwrite existing files in the cloud, 1 for overwrite, 0 for no overwrite, default 1.
isLastModifyTime Optional Syncs file modification times to the cloud storage, default 0 (use upload time), 1 (use local modification time).
Special Features scanOnly Optional Used for scanning without uploading, 0 for normal upload, 1 for scan only, default 0.
uploadErrorRetry Optional Automatic retry count after upload failure, range 0-5, default 0 (no retry).
isSkip406 Optional Determines whether to skip files already existing in the cloud, 1 to skip, 0 not to skip, default 1.
Logging logLevel Optional Specifies log level, choose from debug, info, error.
logPrefix Optional Defines the location and prefix for log files; if unspecified, logs are output directly to the console.
Is the content of this document helpful to you?
Yes
I have suggestion
Submitted successfully! Thank you very much for your feedback, we will continue to strive to do better!

CDNetworks Inc., © 2024. 1840 Enterprise Way, Monrovia, CA 91016 – All rights reserved.