File System Tool

最終更新日:2023-08-08 18:22:58

WOSGateway

WOSGateway (Storage Gateway) is a high-performance gateway tool developed by CDNetworks, that offers support for caching functionality, NFS sharing, and SMB sharing. It enables you to mount Object Storage Service (OSS) buckets as if they were local file systems on Linux. This allows you to handle OSS objects using the same approach you use for managing local files.

Please be aware that using WOSGateway necessitates having over 100GB of storage space available on your server, which will be utilized as a cache disk.

Installation (CentOS Environment)

  1. Begin by downloading the installation package and extracting its contents to a designated local directory. Find the Download link here.
  2. Unpack the wosgw-*.zip package.
  3. Navigate into the wosgw-* directory using the command: cd wosgw-, and proceed with the installation using: rpm -ivh *.rpm.
  4. If you encounter any dependency errors, address them by executing: yum -y install fuse fuse-libs libibverbs librdmacm gperftools-libs.
  5. In case you receive an error related to missing leveldb dependencies, you can install it using the following steps:
wget http://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-13.noarch.rpm
rpm -Uvh epel-release-7-13.noarch.rpm
yum -y install leveldb
  1. Once the installation is successful, you’ll find the following files and directories created:
  • Binary path: /usr/bin/wosgw
  • Configuration file: /etc/wosgw/wosgw.conf
  • Log path: /var/log/wosgw
  • Runtime directory: /var/run/wosgw
  • Database path: /var/run/wosgw/db

Configurations

  1. Revise configuration file: /etc/wosgw/wosgw_ctl.toml
[wosgw_ctl]

# Access key credentials
AK = "your ak"
SK = "your sk"

# Endpoint adress
server = "http://endpoint"

# Bucket name
bucket = "regionName"

# Local mount directory
mount_dir = "/mnt/xxx"

# Local cache directory
use_cache = "/cache5/s3fs_cache"

conf_path = "/etc/wosgw/wosgw.conf"
client = "client.wosgw"

[params]
args = [
	"-o enable_noobj_cache",
	"-o ensure_diskfree=300",
	"-o use_xattr",
	"-o allow_other",
	"-o umask=000,noatime",
	"-o force_purge_rocksdb_stat",
	"-o use_path_request_style",
	"-o curldbg",
	"-o dbglevel=dbg",
]

Instructions

Configurations Required Example Config Notes
allow_other Yes -o allow_other Default fixed option, allows the mount directory to be used by any user logged into the server and can be used to create NFV/SMB shares.rw
rw Yes -o rw Indicates read and write permissions.
umask Yes -o umask=000 Sets access permissions for the mounted resources. When umask = 000, file permissions are set to 777 (read, write, execute).
noatime Yes -o noatime Set the value at 0 to disables access time tracking.
use_cache Yes -o use_cache=/cache/wos-cache Specifies the cache path address for the storage gateway.
ensure_diskfree No -o ensure_diskfree=300 Reserves at least 300M of disk space to ensure enough space for writing files. The storage gateway already reserves 20% of the disk capacity by default; this option can be omitted.
use_xattr Yes -o use_xattr Supports setting file extended attributes.
use_path_request_style Recommended -o use_path_request_style Uses path-based access, when the URL is an IP, this parameter must be used in the mount command to avoid DNS resolution errors.
enable_noobj_cache Required Yes -o enable_noobj_cache After configuring this parameter, the number of bucket listing requests sent by WOSGW is reduced, improving performance.
max_stat_cache_size Recommended -o max_stat_cache_size=1000000 Default is 100,000. Maximum number of stat cache entries that can be stored in memory. It is recommended to configure a larger value for better performance.
force_purge_rocksdb_stat Yes -o force_purge_rocksdb_stat Forces the database to clear, ensuring data consistency.
dbglevel Optional No -o dbglevel=debug Used for debugging purposes, not recommended for regular use.
multipart_size No -o multipart_size=10 Sets the size of individual chunks.
max_multipart_cnt No -o max_multipart_cnt=10000 Sets the maximum number of chunks.
  1. Include the configuration file /etc/wosgw/wosgw.conf and adjust the following details as required. (Note: If none of the settings require customization, you can choose not to add this configuration file.)
wosgw_run_mode=async
debug_wosgw=60
wosgw_rgw_host_regex = ".*"

wosgw_cachefile_life_time=1
wosgw_cachefile_remove_size=200
wosgw_cachefile_remove_interval=2
wosgw_cachefile_remove_thread_num=2
wosgw_cachefile_quantity_watermark_high=0
wosgw_cachefile_quantity_watermark_low=0
wosgw_cachefile_gc_thread_num=3

wosgw_db_path=/cache2/wosgw-db

wosgw_stat_mode=1

wosgw_sync_log_interval=10
wosgw_num_shards=5
wosgw_sync_sleep_time=0

wosgw_resume_upload=True

wosgw_vid_prealloc=100

[client.wosgw.default]
log file=/cache2/wosgw-log/$name.log
admin socket = /usr/local/etc/wosgw/$name.asok

Wosgw Configuration Instructions

Configurations Required Example Config Notes
wosgw_run_mode Yes wosgw_run_mode=asyncV Default value is ‘async’, which means the service operates in asynchronous mode. Files written to the mount directory will first exist on the local cache disk before being synchronized to the backend.
wosgw_stat_mode No wosgw_stat_mode=1 Default value is 1, configuring the metadata storage mode. This determines how metadata is written to the local disk.
wosgw_db_path No db wosgw_db_path=/cache/db In asynchronous mode, set the database storage path. Not recommended for standalone db configuration.
wosgw_log_db_path Yes wosgw_log_db_path=/cache/db_log No default value, must be configured. Location for storing asynchronous log databases. It is recommended to be placed on an SSD disk (with replication) to prevent a performance impact on the storage gateway.
wosgw_stat_db_path Yes wosgw_stat_db_path=/cache/db_stat No default value, must be configured. Location for storing metadata databases. It is recommended to be placed on an SSD disk (with replication) to prevent a performance impact on the storage gateway.
wosgw_lc_db_path Yesv wosgw_lc_db_path=/cache/db_lc No default value, must be configured. Location for storing cache file databases. It is recommended to be placed on an SSD disk (with replication) to prevent a performance impact on the storage gateway.
wosgw_fault_db_path Yes wosgw_fault_db_path=/cache/db_fault No default value, must be configured. Location for storing shard databases. It is recommended to be placed on an SSD disk (with replication) to prevent a performance impact on the storage gateway. wosgw_cachefile_life_time Optional, recommended wosgw_cachefile_life_time=1 Default value is 24 hours. Used to set the expiration time for cache files, valid range is [0.00028, 2^20].
wosgw_cachefile_life_time No wosgw_cachefile_life_time=1 Default value is 24 hours. Used to set the expiration time for cache files, valid range is [0.00028, 2^20].
wosgw_cachefile_remove_size No wosgw_cachefile_remove_size=200 Default value is 100 MB. Used to set the total file size for cache cleanup attempts when conditions are met, valid range is [100, 2^31-1].
wosgw_cachefile_remove_interval No wosgw_cachefile_remove_interval=2 Default value is 2 seconds. Used to set the interval for checking disk space, valid range is [2, 2^31-1].
wosgw_cachefile_remove_thread_num No wosgw_cachefile_remove_thread_num=2 Default value is 2. Used to set the number of threads for cache cleanup, valid range is [2, 32].
wosgw_cachefile_quantity_watermark_high No wosgw_cachefile_quantity_watermark_high=0 Default value is 0.7. Sets the high watermark percentage for disk checking, triggering cleanup when disk usage reaches this level, valid range is [0, 0.8].
wosgw_cachefile_quantity_watermark_low No wosgw_cachefile_quantity_watermark_low=0 Default value is 0.6. Sets the low watermark percentage for disk checking, preventing cleanup when disk usage falls below this level, valid range is [0, high watermark].
wosgw_cachefile_gc_thread_num No wosgw_cachefile_gc_thread_num=3 Default value is 16. Sets the number of threads for deleting cache file records, valid range is [1, 32].
wosgw_cachefile_remove_count No wosgw_cachefile_gc_thread_num=10000 Default value is 100,000. Sets the total number of files to remove when conditions are met and total file size does not meet the configured size.
wosgw_sync_log_interval No wosgw_sync_log_interval=10 Default value is 120 seconds. Sets the interval for asynchronous log processing, determining how long after an operation log is generated it takes to synchronize to the backend, valid range is [10, 2^31-1].
wosgw_num_shards No wosgw_num_shards=5 Default value is 5. Sets the number of threads for asynchronous log processing. The maximum setting should not exceed twice the number of CPU cores, valid range is [1, 2^31-1].
wosgw_sync_sleep_time No wosgw_sync_sleep_time=0 Default value is 30 seconds. Sets the time to wait for a connection to the backend before attempting to synchronize remote operations, valid range is [1, 2^31-1].
wosgw_resume_upload No wosgw_resume_upload=True Default value is False. Enables or disables the resume upload feature, valid values are [True, False].
wosgw_vid_prealloc No wosgw_vid_prealloc=1024 Default value is 1024. Sets the version ID allocation step for inserted logs. This may affect performance, but not functionality. Default value is recommended.
debug_wosgw No debug_wosgw=60 Sets the logging level for the storage gateway instance, default is 1.
og file No log file=/etc/log/$name.log Sets the log path for the storage gateway instance.
admin socket No admin socket = /etc/log/$name.asok Sets the socket path for the storage gateway instance.
wosgw_rgw_host_regex No wosgw_rgw_host_regex=.* Used to check the server configuration with a regular expression statement. Default configuration supports IP
wosgw_load_multi_thread_enable No wosgw_load_multi_thread_enable = False Default value is False. Sets the switch for initializing bucket threads. Valid values are [True, False]. Currently, upper-level s3 does not support multi-threading.
wosgw_load_max_thread_num No wosgw_load_max_thread_num = 100 Default value is 20. Specifies the number of threads for multi-threaded fetching.
wosgw_fullsync_threads No wosgw_fullsync_threads = 10 Default value is 5. Specifies the number of threads for full synchronization during failures.
wosgw_multi_copy_source_size No wosgw_multi_copy_source_size = 100 Default value is 128 MB. Sets the size of each segment in multipart_copy to prevent timeouts due to excessively large individual sizes. If timeouts persist, consider reducing the value accordingly.
wosgw_rename_max_file_num No wosgw_rename_max_file_num= 10000 Default value is 100,000. Specifies the maximum number of files and folders within a directory when attempting an ‘mv’ operation. If this limit is exceeded, the ‘mv’ operation will fail.

How to use

  1. Start
wosgw_ctl -action start
  1. Stop
wosgw_ctl -action stop

Goofys

Goofys is an open-source tool that enables the mounting of S3 storage buckets onto a local file system.

Installation Steps (using CentOS as an example)

  1. Download goofys using: wget https://github.com/kahing/goofys/releases/latest/download/goofys.
  2. Provide execution permission to the goofys file: chmod 755 goofys.
  3. Add your credentials to a certificate file:
vim ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
  1. Start the mounting command:
./goofys --endpoint http://s3-cn-south-6.wcsapi.com --region cn-south-6 yourBucketName /mnt/mount -o allow_other
  1. Verify the successful mount:
mount
> bucket on /mnt/mount type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions)
  1. The logs are by default printed to syslog. You can refer to the configuration of syslog for different systems. For example, the default syslog for CentOS is located at: /var/log/message; Ubuntu systems can execute the dmesg command to output system logs.