最終更新日:2022-06-01 15:28:10
File Sync Tool is a stock data synchronization tool developed based on the API provided by CDNetworks Object Storage, which can synchronize local data to the CDNetworks Object Storage with the original directory structure.
Note: The File Synchronizer is applicable to the migration of stock files, an API upload interface is required for the synchronization of incremental files
it’s applicable for API-V1 and SDK-V1.
Command line synchronization tool:
rsync-hash
Visual synchronization tool:
Windows:
websync-windows-x86
websync-windows-x64
Linux:
websync-linux-x86
websync-linux-x64
Note: select a link corresponding to your OS
Parameters | Required | Description |
---|---|---|
accessKey | Yes | Available at CDNetworks Console in “ Security Settings -> API information management -> AccessKey Management.” |
secretKey | Yes | Available at CDNetworks Console in “ Security Settings -> API information management -> AccessKey Management.” |
uploadDomain | Yes | Domain name for uploading files, available at CDNetworks Console in “Object Storage Service -> Buckets -> Overview -> Domain Names” |
mgrDomain | Yes | management domain name The management domain name required for file HASH values comparison by the tool, available at CDNetworks Console in “Object Storage Service -> Buckets -> Overview -> Domain Names” |
syncMode | Yes | synchronization mode Only supported by command line synchronization tool. Default at 0, supporting single-bucket and multi-directory upload mode, bucket and syncDir required, keyPrefix optional. If configured to 1, supporting multi-bucket and multi-file upload mode, bucketAndDir required. |
bucket | No | Files saved to specified bucket, e.g., images, multi-configuration not supported. If command line synchronization tool is used, required when syncMode is configured to 0. |
syncDir | No | synchronization path Local path for uploading files, e.g., /data. Supporting multi-path separated by “|”, e.g., D:/pic-2|D:/rsync3. If command line synchronization tool is used, required when syncMode is configured to 0. Note: On Linux or Windows, the delimiter “/” is required to configure local path; in Windows, the path must contain a drive letter, e.g., C:/data. |
keyPrefix | No | Specified prefix added to files uploaded to Object Storage, multiple paths corresponding to the syncDir paths one by one can be configured, default value is null. For example, <1> If keyPrefix is configured to “data/”, the uploaded file is “1.apk”, then the file is saved as “data/1.apk” in Object Storage, that is, a new folder “data” is added, and “1.apk” is saved in the folder; <2> If keyPriefix is configured to “data”, the uploaded file is “1.apk”, then the file is saved as “data1.apk” in Object Storage; <3> If syncDir is configured to “D:/rsync1|D:/rsync2|D:/rsync3”, keyPrefix is configured to “test1/|test2/”, then files (folders) under rsync1 are saved in Object Storagetest1 directory, files (folders) under Rsync2 are saved in test2 directory, and files (folders) under rsync3 are saved in root directory. If more keyPrefix is configured than syncDir, the redundant keyPrefix does not take effect and the first several directories are used. If command line synchronization tool is used, required when syncMode is configured to 0. |
bucketAndDir | No | This field is to configure both your target bucket of Object storage and the local path where you want to sync files to Object Storage. Only supported by command line synchronization tool, required when syncMode is configured to 1, e.g., bucket1|D:/dir1,D:/dir2|prefix1,prefix2/;bucket2|D:/dir3,D:/dir4 Note: <1> Each bucket can be configured with multiple local paths. Local paths support folders and files with suffixes; <2> Local path corresponds to prefix, refer to keyPrefix for prefix use; < 3 > Bucket name, local path and prefix are separated by “|”, multiple local paths or prefixes are separately by English comma “,”; <4> Combination of multiple buckets, local paths and prefixes can be configured, separated by “;”. Note: Above example: bucket1|D:/dir1,D:/dir2|prefix1,prefix2/;bucket2|D:/dir3,D:/dir4 means: 1. All files under your local path of D:/dir1 will be synced to bucket1 of Object Storage, and the file name is with the prefix of “prefix1”, for example, a.jpg will be synced as prefix1a.jpg 2. All files under your local path of D:/dir2 will be synced to the directory of “prefix2” under bucket1 of Object Storage. 3. All files under your local path of D:/dir3 and D:/dir4 will be synced to the bucket2 of Object Storage. |
threadNum | No | Number of threads for concurrent file upload. Value range 1-100, default at 1. If configured to 5, then 5 files can be uploaded concurrently. |
sliceThreshold | No | sliced upload threshold If file size is larger than the value, sliced upload will be used. Unit M, range 1M-100M, default at 4M. |
sliceThread | No | number of threads for sliced concurrent upload If configured to 5, then 5 slices can be uploaded concurrently |
sliceBlockSize | No | Size of sliced upload block. Range: 4M-1024M, the value should be a multiple of 4. Default at 4M. |
sliceChunkSize | No | Size of each slice chunk during sliced upload, unit KB, range 1024-1048576KB. Note: number of threads for sliced concurrent upload, block size and slice size are only valid for sliced uploads. |
deletable | No | synchronization deletion If the value is configured to 0, files in Object Storage will not be deleted after local files are deleted If the value is configured to 1, files in Object Storage will be deleted after local files are deleted. Note: the configuration is only valid for files last synchronized, and invalid for other files historically synchronized, default at 0, meaning no synchronization deletion. |
maxRate | No | Upload rate limit, unit KB/s. If configured to 0, then no rate limit. |
taskBeginTime | No | The time for the tool to begin upload, format hh:mm:ss, e.g., 12:00:00. |
taskEndTime | No | The time for the tool to end upload, format hh:mm:ss, e.g., 15:00:00. |
isCompareHash | No | whether with HASH comparison while uploading If configured to 0, then without HASH comparison; If configured to 1, then with HASH comparison. |
countHashThreadNum | No | Hash count for concurrent number of threads Count number of HASH threads Value range 1-100, default at 1. If configured to 10, then 10 HASHes can be counted concurrently. |
compareHashThreadNum | No | number of concurrent HASH comparison Compare the HASH value of the file on the local and Object Storage to determine whether to upload the file again. This parameter sets the number of threads for HASH comparison. Value range 1-100, default at 1. If configured to 10, then 10 threads can be used concurrently to compare HASH. |
compareHashFileNum | No | number of HASH comparison files Number of file HASHes queried from Object Storage at a time when comparing file HASHes. Value range 1-2000, default at 100. If configured to 100, then 100 HASHes can be queried from server farm at a time. |
minFileSize | No | No upload for files smaller than specified file size. Default of 0 (no limit). If configured to 1024, then no upload for files smaller than 1024 bytes. |
overwrite | No | Whether to overwrite a file with the same name on Object Storage, the value can be 1 or 0. 1 indicates Overwrite, 0 indicates No Overwrite, default at 1. |
isLastModifyTime | No | whether to update server modification time Whether the lastModifyTime saved in Object Storage is subject to the update time of the local file, the value can be 0 or 1, default at 0. 0 indicates the upload time will be lastModifyTime. 1 indicates the local modification time will be lastModifyTime. |
scanOnly | No | Whether to scan file list only. Default at 0, normal upload of files. If configured to 1, scan only file list, record modification time, without counting HASH or comparing HASH or uploading files. Remark: This is a risk configuration item, prior confirmation with the Object Storage staff is required |
uploadErrorRetry | No | Number of automatic retries for file upload errors. Value range 0-5, default at 0, meaning no retry. If configured to 2, then the file will automatically retry 2 times after upload failure. |
Suggestions
How to use
How to use
Windows
Linux
Note: Visual synchronization tools provide local configuration files… \service\wcsrSynchashWeb\conf.json, allowing users to customize whether configuration items are editable on the interface, “readonly: true” indicates that the configuration item is read-only. “readonly: false” indicates that the configuration item can be edited on the interface; configuration items that cannot be edited on the interface can be modified for their value in the configuration file.
If only accessKey, secretKey, bucket, syncDir and other basic configurations are excepted to be edited on the interface, set “readonly” to “false”, and “true” for those that are not expected to be edited on the interface
{
"accessKey":{"value":"","readonly":false},
"secretKey":{"value":"","readonly":false},
"bucket":{"value":"","readonly":false},
"syncDir":{"value":"","readonly":false},
"uploadDomain":{"value":"","readonly":true},
"mgrDomain":{"value":"","readonly":true},
"keyPrefix":{"value":"","readonly":true},
“threadNum":{"value":"1","readonly":true},
...
}