Back
Object Storage
CDNetworks Documentation Object Storage Tools File Rsync Tool

File Rsync Tool

Last update：2022-06-01 15:28:10

File Sync Tool

File Sync Tool is a stock data synchronization tool developed based on the API provided by CDNetworks Object Storage, which can synchronize local data to the CDNetworks Object Storage with the original directory structure.

Note: The File Synchronizer is applicable to the migration of stock files, an API upload interface is required for the synchronization of incremental files

Applicable API version

it’s applicable for API-V1 and SDK-V1.

Download links

Command line synchronization tool:
rsync-hash

Visual synchronization tool:
Windows:
websync-windows-x86
websync-windows-x64

Linux:
websync-linux-x86
websync-linux-x64

Note: select a link corresponding to your OS

Configuration Item

Parameters Required Description
accessKey Yes Available at CDNetworks Console in “ Security Settings -> API information management -> AccessKey Management.”
secretKey Yes Available at CDNetworks Console in “ Security Settings -> API information management -> AccessKey Management.”
uploadDomain Yes Domain name for uploading files, available at CDNetworks Console in “Object Storage Service -> Buckets -> Overview -> Domain Names”
mgrDomain Yes management domain name
The management domain name required for file HASH values comparison by the tool, available at CDNetworks Console in “Object Storage Service -> Buckets -> Overview -> Domain Names”
syncMode Yes synchronization mode
Only supported by command line synchronization tool.
Default at 0, supporting single-bucket and multi-directory upload mode, bucket and syncDir required, keyPrefix optional.
If configured to 1, supporting multi-bucket and multi-file upload mode, bucketAndDir required.
bucket No Files saved to specified bucket, e.g., images, multi-configuration not supported.
If command line synchronization tool is used, required when syncMode is configured to 0.
syncDir No synchronization path
Local path for uploading files, e.g., /data. Supporting multi-path separated by “|”, e.g., D:/pic-2|D:/rsync3.
If command line synchronization tool is used, required when syncMode is configured to 0.
Note: On Linux or Windows, the delimiter “/” is required to configure local path; in Windows, the path must contain a drive letter, e.g., C:/data.
keyPrefix No Specified prefix added to files uploaded to Object Storage, multiple paths corresponding to the syncDir paths one by one can be configured, default value is null.
For example,
<1> If keyPrefix is configured to “data/”, the uploaded file is “1.apk”, then the file is saved as “data/1.apk” in Object Storage, that is, a new folder “data” is added, and “1.apk” is saved in the folder;
<2> If keyPriefix is configured to “data”, the uploaded file is “1.apk”, then the file is saved as “data1.apk” in Object Storage;
<3> If syncDir is configured to “D:/rsync1|D:/rsync2|D:/rsync3”, keyPrefix is configured to “test1/|test2/”, then files (folders) under rsync1 are saved in Object Storagetest1 directory, files (folders) under Rsync2 are saved in test2 directory, and files (folders) under rsync3 are saved in root directory. If more keyPrefix is configured than syncDir, the redundant keyPrefix does not take effect and the first several directories are used.
If command line synchronization tool is used, required when syncMode is configured to 0.
bucketAndDir No This field is to configure both your target bucket of Object storage and the local path where you want to sync files to Object Storage.
Only supported by command line synchronization tool, required when syncMode is configured to 1, e.g.,
bucket1|D:/dir1,D:/dir2|prefix1,prefix2/;bucket2|D:/dir3,D:/dir4
Note:
<1> Each bucket can be configured with multiple local paths. Local paths support folders and files with suffixes;
<2> Local path corresponds to prefix, refer to keyPrefix for prefix use;
< 3 > Bucket name, local path and prefix are separated by “|”, multiple local paths or prefixes are separately by English comma “,”;
<4> Combination of multiple buckets, local paths and prefixes can be configured, separated by “;”.
Note: Above example: bucket1|D:/dir1,D:/dir2|prefix1,prefix2/;bucket2|D:/dir3,D:/dir4 means:
1. All files under your local path of D:/dir1 will be synced to bucket1 of Object Storage, and the file name is with the prefix of “prefix1”, for example, a.jpg will be synced as prefix1a.jpg
2. All files under your local path of D:/dir2 will be synced to the directory of “prefix2” under bucket1 of Object Storage.
3. All files under your local path of D:/dir3 and D:/dir4 will be synced to the bucket2 of Object Storage.
threadNum No Number of threads for concurrent file upload. Value range 1-100, default at 1.
If configured to 5, then 5 files can be uploaded concurrently.
sliceThreshold No sliced upload threshold
If file size is larger than the value, sliced upload will be used. Unit M, range 1M-100M, default at 4M.
sliceThread No number of threads for sliced concurrent upload
If configured to 5, then 5 slices can be uploaded concurrently
sliceBlockSize No Size of sliced upload block. Range: 4M-1024M, the value should be a multiple of 4. Default at 4M.
sliceChunkSize No Size of each slice chunk during sliced upload, unit KB, range 1024-1048576KB.
Note: number of threads for sliced concurrent upload, block size and slice size are only valid for sliced uploads.
deletable No synchronization deletion
If the value is configured to 0, files in Object Storage will not be deleted after local files are deleted
If the value is configured to 1, files in Object Storage will be deleted after local files are deleted.
Note: the configuration is only valid for files last synchronized, and invalid for other files historically synchronized, default at 0, meaning no synchronization deletion.
maxRate No Upload rate limit, unit KB/s. If configured to 0, then no rate limit.
taskBeginTime No The time for the tool to begin upload, format hh:mm:ss, e.g., 12:00:00.
taskEndTime No The time for the tool to end upload, format hh:mm:ss, e.g., 15:00:00.
isCompareHash No whether with HASH comparison while uploading
If configured to 0, then without HASH comparison;
If configured to 1, then with HASH comparison.
countHashThreadNum No Hash count for concurrent number of threads
Count number of HASH threads Value range 1-100, default at 1. If configured to 10, then 10 HASHes can be counted concurrently.
compareHashThreadNum No number of concurrent HASH comparison
Compare the HASH value of the file on the local and Object Storage to determine whether to upload the file again.
This parameter sets the number of threads for HASH comparison. Value range 1-100, default at 1.
If configured to 10, then 10 threads can be used concurrently to compare HASH.
compareHashFileNum No number of HASH comparison files
Number of file HASHes queried from Object Storage at a time when comparing file HASHes. Value range 1-2000, default at 100.
If configured to 100, then 100 HASHes can be queried from server farm at a time.
minFileSize No No upload for files smaller than specified file size. Default of 0 (no limit).
If configured to 1024, then no upload for files smaller than 1024 bytes.
overwrite No Whether to overwrite a file with the same name on Object Storage, the value can be 1 or 0. 1 indicates Overwrite, 0 indicates No Overwrite, default at 1.
isLastModifyTime No whether to update server modification time
Whether the lastModifyTime saved in Object Storage is subject to the update time of the local file, the value can be 0 or 1, default at 0.
0 indicates the upload time will be lastModifyTime.
1 indicates the local modification time will be lastModifyTime.
scanOnly No Whether to scan file list only.
Default at 0, normal upload of files.
If configured to 1, scan only file list, record modification time, without counting HASH or comparing HASH or uploading files.
Remark: This is a risk configuration item, prior confirmation with the Object Storage staff is required
uploadErrorRetry No Number of automatic retries for file upload errors.
Value range 0-5, default at 0, meaning no retry.
If configured to 2, then the file will automatically retry 2 times after upload failure.

Command line synchronization tool

Suggestions

  1. Pre-install Java, JDK 1.6 or later required.
  2. Store configuration file in the same path as the tool

How to use

  1. Open the directory where wcs-rsync-hash tool is located, such as Windows directory “F:\wcs-rsync” or Linux directory “/home/tool/wcs-rsync”
  2. Configure conf.json
  3. Start service
    In Windows, hold Shift in a blank area and right click, select “Open Command Window Here (W)”
    Run the command: java -jar wcs-rsync-hash-xxx.jar conf.json
    After the synchronization, run the command again to re-sync the files with failed synchronization, if any
  4. List files with failed synchronization
    Run the command: java -jar wcs-rsync-hash-xxx.jar -listfailed conf.json
    Output is saved to the log file in the tool directory
  5. Force re-upload of all files: Run the command: java -jar wcs-rsync-hash-xxx.jar -igsync conf.json

Visual synchronization tool

How to use

Windows

  1. Start the service
    Run startup.bat to pop up TOMCAT window
  2. The first time the service is successfully opened, the browser will automatically open and enter the operation interface.
    After the service is successfully started, the TOMCAT window can be closed.
  3. During service startup, remote operations can be performed using a browser by opening the browser and entering IP: Port to go to the operation interface
    Note: default Port at 8091*
  4. Select “Configure Upload” to set basic information and advanced parameters as required
  5. Click “Upload” to upload
    Stop the upload if configuration needs to be changed during uploading
  6. View the upload progress through “Progress Query”, and failed files can be re-uploaded
  7. Close the service
    Run shutdown.bat to close the service

Linux

  1. Start the service
    Run startup.sh to start the service
  2. After service startup, remote operations can be performed using a browser by opening the browser and entering IP: Port to go to the operation interface
    Note: default Port at 8091*
  3. Select “Configure Upload” to set basic information and advanced parameters as required
  4. Click “Upload” to upload
    Stop the upload if configuration needs to be changed during uploading
  5. View the upload progress through “Progress Query”, and failed files can be re-uploaded
  6. Close the service
    Run shutdown.sh to close the service

Note: Visual synchronization tools provide local configuration files… \service\wcsrSynchashWeb\conf.json, allowing users to customize whether configuration items are editable on the interface, “readonly: true” indicates that the configuration item is read-only. “readonly: false” indicates that the configuration item can be edited on the interface; configuration items that cannot be edited on the interface can be modified for their value in the configuration file.

If only accessKey, secretKey, bucket, syncDir and other basic configurations are excepted to be edited on the interface, set “readonly” to “false”, and “true” for those that are not expected to be edited on the interface
{
    "accessKey":{"value":"","readonly":false},
    "secretKey":{"value":"","readonly":false},
    "bucket":{"value":"","readonly":false},
    "syncDir":{"value":"","readonly":false},
    "uploadDomain":{"value":"","readonly":true},
    "mgrDomain":{"value":"","readonly":true},
    "keyPrefix":{"value":"","readonly":true},
    “threadNum":{"value":"1","readonly":true},
    ...

}
Is the content of this document helpful to you?
Yes
I have suggestion
Submitted successfully! Thank you very much for your feedback, we will continue to strive to do better!

CDNetworks Inc., © 2022. 1550 Valley Vista Dr., Suite 110, Diamond Bar, CA 91765 - All rights reserved.