File Rsync Tool
Last update：2024-06-19 10:21:12
The File Synchronization Tool, leveraging CDNetworks Object Storage APIs, synchronizes local data to CDNetworks Object Storage while preserving the original directory structure. This tool is ideal for migrating existing files. For incremental file synchronization, utilize the API upload interface.
Download Links
- Command Line Sync Tool: wcs-rsync-hash
Before Starting
- Ensure Java (JDK 1.6 or newer) is installed.
- Place the configuration file and the synchronization tool in the same directory.
How to Use
- Navigate to the directory containing the
wcs-rsync-hashtool (e.g.,F:\wcs-rsyncfor Windows or/home/tool/wcs-rsyncfor Linux). - Edit the
conf.jsonfile according to your needs. - To start the service:
- For Windows: While holding the Shift key, right-click in the folder window and select “Open command window here.”
- Execute the command:
java -jar wcs-rsync-hash-xxx.jar conf.json. - If any files fail to sync, rerun the command to attempt re-synchronization.
- To view a list of failed uploads, execute:
java -jar wcs-rsync-hash-xxx.jar -listfailed conf.json.- The results will be saved in the log file located in the tool directory.
- To force a re-upload of all files, execute:
java -jar wcs-rsync-hash-xxx.jar -igsync conf.json. - In case of errors during the synchronization process, consult the
wcs-rsync-hash.logfile in the tool directory for ERROR level logs to diagnose issues.
Available Settings in the Configuration File
The following are some adjustable settings in the conf.json file. You can modify these settings based on your specific needs.
| Section | Item | Requirement | Description |
|---|---|---|---|
| Basic Authentication | accessKey | Required | A unique identifier to access your cloud storage account. Obtain it from the “Security Settings - API Information Management - AccessKey Management” section on console. |
| secretKey | Required | A private key paired with your Access Key for identity verification. Also obtained from the “Security Settings - API Information Management - AccessKey Management” section on console. | |
| Domain Settings | uploadDomain | Required | The specific domain used for uploading files. Find this in the “Bucket - Overview - Upload Domain”. |
| mgrDomain | Required | The domain used by the tool for backend operations such as file checksum validation. Also found in the “Bucket - Overview - Manage Domain”. | |
| Synchronization Mode | syncMode | Required | Determines the synchronization strategy: - 0: Single bucket, multi-directory mode. Requires specifying Bucket and Sync Dir; Key Prefix is optional. - 1: Multi-bucket, multi-directory mode. In this case, Bucket And Dir is mandatory. |
| Space and Path | bucket | Optional | Specifies the bucket where files will be saved. Mandatory when Sync Mode is set to 0. |
| syncDir | Optional | The local path of the files to be uploaded, such as /data, can be specified. Multiple paths can be configured, separated by "|", for example, D:/pic-2|D:/rsync3. When using the command-line synchronization tool, this field is required if syncMode is set to 0.Please note: For both Linux and Windows systems, use / as the path separator when configuring the local path. Paths on Windows systems should include the drive letter (e.g., C:/data). |
|
| keyPrefix | Optional | A designated prefix is appended to files uploaded to the Object Storage. Multiple configurations are allowed, and they align one-to-one with the paths defined in syncDir. The default is blank. For example:1. If keyPrefix is configured as data/ and the uploaded file is 1.apk, this file will be saved in the cloud storage as data/1.apk. This means a new folder named data is created in the Object Storage, and 1.apk resides inside this folder.2. If keyPrefix is set to data and the file being uploaded is 1.apk, this file will be saved in the cloud storage as data1.apk.3. If syncDir is configured as D:/rsync1|D:/rsync2|D:/rsync3 and keyPrefix is set to test1/|test2/, files (and directories) from rsync1 will be stored under the test1 directory in the cloud storage, files (and directories) from rsync2 will be stored under the test2 directory, and files (and directories) from rsync3 will be stored at the root directory. If there are more keyPrefix configurations than syncDir paths, the additional keyPrefix settings will not apply, and only the first several directories will be considered. |
|
| Target Space and Local Path | bucketAndDir | Optional | This specifies the target bucket and local path from which the file will be uploaded. This parameter is required when syncMode is configured as 1. Example:bucket1|D:/dir1,D:/dir2|prefix1,prefix2/;bucket2|D:/dir3,D:/dir4Please note:1. Each bucket can be configured with multiple local paths. Local paths can target directories or files, and file names must include their suffixes.2. Local paths and prefixes are matched one-to-one. The application of prefixes follows the same rules as the keyPrefix parameter.3. Bucket names, local paths, and prefixes are separated by " |
| Performance-Related Settings | threadNum | Optional | Sets the number of files uploaded concurrently, range 1-100, default 1. |
| sliceThreshold | Optional | Enables chunked upload for files exceeding this value, measured in MB, range 1-100MB, default 4MB. | |
| sliceThread | Optional | Number of concurrent threads for chunked uploads, range 1-100, default 5. | |
| sliceBlockSize | Optional | Size of individual blocks during chunked uploads, range 4M-1024M, must be a multiple of 4, default 4M. | |
| sliceChunkSize | Optional | Size of each chunk, measured in KB, range 1024-1048576KB. | |
| Synchronization Behavior | deletable | Optional | Determines the behavior of cloud storage files when local files are deleted, 0 to retain, 1 for synchronized deletion, default 0. |
| maxRate | Optional | Upload speed limit, measured in KB/s, 0 indicates no limit. | |
| taskBeginTime | Optional | Scheduled task start time, format hh:mm:ss. | |
| taskEndTime | Optional | Scheduled task end time, format hh:mm:ss. | |
| Data Validation | isCompareHash | Optional | Enables or disables file hash comparison, default 1 (enabled). |
| countHashThreadNumHash | Optional | Concurrency level for hash value calculations, range 1-100, default 1. | |
| compareHashThreadNumHash | Optional | Concurrency level for hash value comparisons, range 1-100, default 1. | |
| compareHashFileNumHash | Optional | Number of files whose hashes are compared in each batch from cloud storage, range 1-2000, default 100. | |
| Filtering Rules | minFileSize | Optional | Files smaller than this value will not participate in the upload, measured in bytes, default 0 (no restriction). |
| overwrite | Optional | Determines whether to overwrite existing files in the cloud, 1 for overwrite, 0 for no overwrite, default 1. | |
| isLastModifyTime | Optional | Syncs file modification times to the cloud storage, default 0 (use upload time), 1 (use local modification time). | |
| Special Features | scanOnly | Optional | Used for scanning without uploading, 0 for normal upload, 1 for scan only, default 0. |
| uploadErrorRetry | Optional | Automatic retry count after upload failure, range 0-5, default 0 (no retry). | |
| isSkip406 | Optional | Determines whether to skip files already existing in the cloud, 1 to skip, 0 not to skip, default 1. | |
| Logging | logLevel | Optional | Specifies log level, choose from debug, info, error. |
| logPrefix | Optional | Defines the location and prefix for log files; if unspecified, logs are output directly to the console. |
Is the content of this document helpful to you?
Yes
I have suggestion