搜索

对象存储神器-rclone光速入门

Ceph对象存储方案

共 15556字，需浏览 32分钟

 ·

2022-04-14 15:48


1. 下载&安装

从下面的地址下载对应的二进制版本即可，之后安装在系统的/usr/local/bin目录下,或者其他用户指定的路径即可。
二进制:https://rclone.org/downloads/
源码:https://github.com/rclone/rclone

2. rclone简介

rclone是一个golang写的开源远程数据同步工具，后端支持文件系统以及多种类型的对象存储产品，是一个跨平台的数据同步工具，相比于s3cmd性能更加强大，同时还支持sync数据同步等功能实现差异数据的跨存储后台同步。并且还提供mount操作，可以直接将远端对象存储挂载到本地文件系统进行访问。官方的Features介绍如下

MD5/SHA-1 hashes checked at all times for file integrity
Timestamps preserved on files
Partial syncs supported on a whole file basis
Copy mode to just copy new/changed files
Sync (one way) mode to make a directory identical
Check mode to check for file hash equality
Can sync to and from network, e.g. two different cloud accounts
Optional large file chunking (Chunker)
Optional transparent compression (Compress)
Optional encryption (Crypt)
Optional FUSE mount (rclone mount)
Multi-threaded downloads to local disk
Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

3. 初始配置与检查

查看软件版本

╭─user@Mac ~
╰─$ rclone version                                                          1 ↵
rclone v1.58.0
- os/version: darwin 10.15.7 (64 bit)
- os/kernel: 19.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.8
- go/linking: dynamic
- go/tags: cmount

通过命令rclone config file获取默认的配置文件路径，之后在对应的配置文件中填写对应的配置

╭─user@Mac ~
╰─$ rclone config file
Configuration file is stored at:
/Users/user/.config/rclone/rclone.conf
╭─user@Mac ~
╰─$ vi /Users/user/.config/rclone/rclone.conf
╭─user@Mac ~
╰─$ cat /Users/user/.config/rclone/rclone.conf
[s3v2]
type = s3
provider = Ceph
access_key_id = xxx
secret_access_key = sss
endpoint = https://s3.demo.com
location_constraint =
force_path_style = false
╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone config show
[s3v2]
type = s3
provider = Ceph
access_key_id = xxx
secret_access_key = sss
endpoint = http://s3.demo.com
location_constraint = 
force_path_style = false

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone config dump
{
    "s3v2": {
        "access_key_id": "xxx",
        "endpoint": "http://s3.demo.com",
        "force_path_style": "false",
        "location_constraint": "",
        "provider": "Ceph",
        "secret_access_key": "sss",
        "type": "s3"
    }
}%

通过rclone listremotes查看已经配置好的远端服务列表

╭─user@Mac ~
╰─$ rclone listremotes
s3v2:

通过rclone lsd查看bucket列表或者是远端的目录列表

╭─user@Mac ~
╰─$ rclone lsd s3v2:
          -1 2022-01-10 11:11:52        -1 demo-storage
╭─user@Mac ~
╰─$ rclone lsd s3v2:demo-storage
           0 2022-04-11 10:42:45        -1 demo-storage
           0 2022-04-11 10:42:45        -1 wallpapers-zelda

通过rclone backend features获取远端服务支持的功能特性列表

╭─user@Mac ~/SourceCode/rclone ‹master*› 
╰─$    rclone backend features s3v2:
{
        "Name": "s3v2",
        "Root": "",
        "String": "S3 root",
        "Precision": 1,
        "Hashes": [
                "md5"
        ],
        "Features": {
                "About": false,
                "BucketBased": true,
                "BucketBasedRootOK": true,
                "CanHaveEmptyDirectories": false,
                "CaseInsensitive": false,
                "ChangeNotify": false,
                "CleanUp": true,
                "Command": true,
                "Copy": true,
                "DirCacheFlush": false,
                "DirMove": false,
                "Disconnect": false,
                "DuplicateFiles": false,
                "GetTier": true,
                "IsLocal": false,
                "ListR": true,
                "MergeDirs": false,
                "Move": false,
                "OpenWriterAt": false,
                "PublicLink": true,
                "Purge": false,
                "PutStream": true,
                "PutUnchecked": false,
                "ReadMimeType": true,
                "ServerSideAcrossConfigs": false,
                "SetTier": true,
                "SetWrapper": false,
                "Shutdown": false,
                "SlowHash": false,
                "SlowModTime": true,
                "UnWrap": false,
                "UserInfo": false,
                "WrapFs": false,
                "WriteMimeType": true
        }
}

4. 数据上传下载与遍历

以上传本地bin目录到远端S3为例

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ ls bin         
bisect-go-rclone.sh         check-merged.go             decrypt_names.py            make_changelog.py           nfpm.yaml                   test-repeat-vfs.sh          test_proxy.py               update-authors.py
bisect-rclone.sh            config.py                   get-github-release.go       make_manual.py              not-in-stable.go            test-repeat.sh              tidy-beta                   upload-github
build-xgo-cgofuse.sh        cross-compile.go            make_backend_docs.py        make_rc_docs.sh             test-all-commits-compile.sh test_independence.go        travis.rclone.conf          win-build.bat


╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone copy bin s3v2:demo-storage/bin -vv
2022/04/13 14:50:25 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "bin" "s3v2:demo-storage/bin" "-vv"]
2022/04/13 14:50:25 DEBUG : Creating backend with remote "bin"
2022/04/13 14:50:25 DEBUG : Using config file from "/Users/user/.config/rclone/rclone.conf"
2022/04/13 14:50:25 DEBUG : fs cache: renaming cache item "bin" to be canonical "/Users/user/SourceCode/rclone/bin"
2022/04/13 14:50:25 DEBUG : Creating backend with remote "s3v2:demo-storage/bin"
2022/04/13 14:50:25 DEBUG : S3 bucket demo-storage path bin: Waiting for checks to finish
2022/04/13 14:50:25 DEBUG : S3 bucket demo-storage path bin: Waiting for transfers to finish
2022/04/13 14:50:25 DEBUG : bisect-rclone.sh: md5 = 5476e61f31e1e2d697ce06a0f0100829 OK
2022/04/13 14:50:25 INFO  : bisect-rclone.sh: Copied (new)
...
2022/04/13 14:50:26 DEBUG : upload-github: md5 = a26b267a5c784152a6378457b217c3b9 OK
2022/04/13 14:50:26 INFO  : upload-github: Copied (new)
2022/04/13 14:50:26 INFO  : 
Transferred:       62.573 KiB / 62.573 KiB, 100%, 0 B/s, ETA -
Transferred:           25 / 25, 100%
Elapsed time:         0.8s

2022/04/13 14:50:26 DEBUG : 13 go routines active
╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone lsd s3v2:demo-storage             
           0 2022-04-13 14:50:32        -1 bin
           0 2022-04-13 14:50:32        -1 demo
           0 2022-04-13 14:50:32        -1 demo-storage
           0 2022-04-13 14:50:32        -1 wallpapers-zelda

通过lsd目录查看指定路径下的目录列表

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone lsd s3v2:demo-storage   
           0 2022-04-13 15:02:41        -1 bin
           0 2022-04-13 15:02:41        -1 demo
           0 2022-04-13 15:02:41        -1 demo-storage
           0 2022-04-13 15:02:41        -1 wallpapers-zelda

通过ls命令查看指定目录下的文件列表

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone ls s3v2:demo-storage/bin 
      253 .ignore-emails
      382 bisect-go-rclone.sh
      902 bisect-rclone.sh
    ...
      370 travis.rclone.conf
     1224 update-authors.py
     1034 upload-github
      364 win-build.bat

通过指定max-depth参数来设定ls命令所需遍历的目录层级深度

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone ls s3v2:demo-storage --max-depth 1                                                                                                                                                                                                ↵
      382 bisect-go-rclone.sh
      902 bisect-rclone.sh
      142 build-xgo-cgofuse.sh
...
      370 travis.rclone.conf
     1224 update-authors.py
     1034 upload-github
      364 win-build.bat

同时支持以json方式输出，并输出对应文件的md5值信息。

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone lsjson s3v2:demo-storage/bin --hash                                                                                                                                                                                                                            1 ↵
[
{"Path":".ignore-emails","Name":".ignore-emails","Size":253,"MimeType":"application/octet-stream","ModTime":"2021-12-16T10:08:33.005446571+08:00","IsDir":false,"Hashes":{"md5":"8155885487321df7790cf63a9db7ba46"},"Tier":"STANDARD"},
{"Path":"upload-github","Name":"upload-github","Size":1034,"MimeType":"application/octet-stream","ModTime":"2021-07-08T18:22:34.487717787+08:00","IsDir":false,"Hashes":{"md5":"a26b267a5c784152a6378457b217c3b9"},"Tier":"STANDARD"},
{"Path":"win-build.bat","Name":"win-build.bat","Size":364,"MimeType":"application/x-msdownload","ModTime":"2021-07-08T18:22:34.487814575+08:00","IsDir":false,"Hashes":{"md5":"bf695fba05eb1ec8972eac29a59b21b4"},"Tier":"STANDARD"}
]

同时支持按文件files-only或者目录dirs-only进行内容筛选

╭─user@Mac ~/SourceCode/rclone ‹master› 
╰─$ rclone lsjson s3v2:demo-storage  --dirs-only                                                                                                                                                                                                                    1 ↵
[
{"Path":"bin","Name":"bin","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917088000+08:00","IsDir":true},
{"Path":"demo","Name":"demo","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917140000+08:00","IsDir":true},
{"Path":"demo-storage","Name":"demo-storage","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917145000+08:00","IsDir":true},
{"Path":"wallpapers-zelda","Name":"wallpapers-zelda","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T15:29:32.917149000+08:00","IsDir":true}
]

数据下载

╭─user@Mac /tmp/demo 
╰─$ ls        
╭─user@Mac /tmp/demo 
╰─$ pwd
/tmp/demo
╭─user@Mac /tmp/demo 
╰─$ rclone copy s3v2:demo-storage/bin bin
╭─user@Mac /tmp/demo 
╰─$ ls bin
bisect-go-rclone.sh         check-merged.go             decrypt_names.py            make_changelog.py           nfpm.yaml                   test-repeat-vfs.sh          test_proxy.py               update-authors.py
bisect-rclone.sh            config.py                   get-github-release.go       make_manual.py              not-in-stable.go            test-repeat.sh              tidy-beta                   upload-github
build-xgo-cgofuse.sh        cross-compile.go            make_backend_docs.py        make_rc_docs.sh             test-all-commits-compile.sh test_independence.go        travis.rclone.conf          win-build.bat

5. 使用sync数据同步

将远端数据同步到本地

╭─user@Mac /tmp/demo 
╰─$ rclone sync s3v2:demo-storage/bin bin                                                                                                                                                                                                                                 1 ↵
╭─user@Mac /tmp/demo 
╰─$ ls
bin
╭─user@Mac /tmp/demo 
╰─$ ls bin
bisect-go-rclone.sh         check-merged.go             decrypt_names.py            make_changelog.py           nfpm.yaml                   test-repeat-vfs.sh          test_proxy.py               update-authors.py
bisect-rclone.sh            config.py                   get-github-release.go       make_manual.py              not-in-stable.go            test-repeat.sh              tidy-beta                   upload-github
build-xgo-cgofuse.sh        cross-compile.go            make_backend_docs.py        make_rc_docs.sh             test-all-commits-compile.sh test_independence.go        travis.rclone.conf          win-build.bat

模拟新建一个文件，同步到远端

╭─user@Mac /tmp/demo 
╰─$ echo newfile1 > bin/newfile1
╭─user@Mac /tmp/demo 
╰─$ rclone sync bin  s3v2:demo-storage/bin 
╭─user@Mac /tmp/demo 
╰─$ rclone  ls s3v2:demo-storage/bin       
      253 .ignore-emails
      382 bisect-go-rclone.sh
      ...
      370 travis.rclone.conf
     1224 update-authors.py
     1034 upload-github
      364 win-build.bat

模拟修改一个文件，同步到远端，可以看到只同步了差异部分

╭─user@Mac /tmp/demo 
╰─$ echo new123 > bin/newfile1             
╭─user@Mac /tmp/demo 
╰─$ rclone sync bin  s3v2:demo-storage/bin -vv
2022/04/13 15:39:01 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "sync" "bin" "s3v2:demo-storage/bin" "-vv"]
2022/04/13 15:39:01 DEBUG : Creating backend with remote "bin"
2022/04/13 15:39:01 DEBUG : Using config file from "/Users/user/.config/rclone/rclone.conf"
2022/04/13 15:39:01 DEBUG : fs cache: renaming cache item "bin" to be canonical "/tmp/demo/bin"
2022/04/13 15:39:01 DEBUG : Creating backend with remote "s3v2:demo-storage/bin"
2022/04/13 15:39:01 DEBUG : S3 bucket demo-storage path bin: Waiting for checks to finish
...
2022/04/13 15:39:02 DEBUG : upload-github: Unchanged skipping
2022/04/13 15:39:02 DEBUG : win-build.bat: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/04/13 15:39:02 DEBUG : win-build.bat: Unchanged skipping
2022/04/13 15:39:02 DEBUG : S3 bucket demo-storage path bin: Waiting for transfers to finish
2022/04/13 15:39:02 DEBUG : newfile1: md5 = f9f9a6b3363a401f566becf83a5fad1b OK
2022/04/13 15:39:02 INFO  : newfile1: Copied (replaced existing)
2022/04/13 15:39:02 DEBUG : Waiting for deletions to finish
2022/04/13 15:39:02 INFO  : 
Transferred:              7 B / 7 B, 100%, 0 B/s, ETA -
Checks:                26 / 26, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2022/04/13 15:39:02 DEBUG : 25 go routines active

6. 挂载S3目录到本地文件系统

rclone支持按指定目录作为根目录进行挂载,比如指定bucket名称为demo-storage下的bin目录作为根目录

╭─user@Mac /tmp 
╰─$ rclone mount s3v2:demo-storage/bin ~/mnt                                                                                                                                                                                                                            ↵

之后就可以在本地文件系统上访问对应的目录数据了

╭─user@Mac ~
╰─$ cd ~/mnt
╭─user@Mac ~/mnt
╰─$ ls
bisect-go-rclone.sh         cross-compile.go            make_manual.py              test-all-commits-compile.sh tidy-beta
bisect-rclone.sh            decrypt_names.py            make_rc_docs.sh             test-repeat-vfs.sh          travis.rclone.conf
build-xgo-cgofuse.sh        get-github-release.go       newfile1                    test-repeat.sh              update-authors.py
check-merged.go             make_backend_docs.py        nfpm.yaml                   test_independence.go        upload-github
config.py                   make_changelog.py           not-in-stable.go            test_proxy.py               win-build.bat

7. 其他操作

通过link命令生成文件的访问URL链接，支持通过expire设置URL有效时长。

╭─user@Mac ~/mnt
╰─$ rclone link  s3v2:demo-storage/bin/win-build.bat --expire 1h                                                                                                                         4 ↵
http://demo-storage.s3.demo.com/bin/win-build.bat?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZK4XIPVNW9PMOADEZMSR%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T080748Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b4d60414a5d13662ac7026821e0aa7ffa10f4654142b87ea8629d2ba98b05e8c

通过size命令，获取文件或者目录的大小

─user@Mac ~/mnt
╰─$ rclone size  s3v2:demo-storage/bin
Total objects: 26 (26)
Total size: 62.580 KiB (64082 Byte)

通过delete命令，删除对应的文件

╭─user@Mac ~/mnt
╰─$ rclone delete  s3v2:demo-storage/bin/win-build.bat

使用rmdir只能删除空目录，如果目录残留文件，需要使用purge命令删除数据，之后再执行rmdir才能彻底删除目录数据。

╭─user@Mac ~/mnt
╰─$ rclone rmdir  s3v2:demo-storage/bin -v
2022/04/13 16:12:25 INFO  : S3 bucket demo-storage path bin: Removing directory
╭─user@Mac ~/mnt
╰─$ rclone ls  s3v2:demo-storage/bin
      253 .ignore-emails
  ...
     1034 upload-github
╭─user@Mac ~/mnt
╰─$ rclone purge  s3v2:demo-storage/bin -v
2022/04/13 16:12:55 INFO  : build-xgo-cgofuse.sh: Deleted
...
2022/04/13 16:12:55 INFO  : update-authors.py: Deleted
2022/04/13 16:12:55 INFO  : S3 bucket demo-storage path bin: Removing directory
╭─user@Mac ~/mnt
╰─$ rclone ls  s3v2:demo-storage/bin
╭─user@Mac ~/mnt
╰─$ rclone rmdir  s3v2:demo-storage/bin
╭─user@Mac ~/mnt
╰─$ rclone lsd  s3v2:demo-storage
           0 2022-04-13 16:13:16        -1 demo
           0 2022-04-13 16:13:16        -1 demo-storage
           0 2022-04-13 16:13:16        -1 wallpapers-zelda


浏览 146
点赞
评论
收藏
分享

手机扫一扫分享

分享
举报
评论
图片
表情
推荐
对象存储
华云对象存储（Chinac Object Storage，简称COS）服务是华云对外提供的海量、安全、低成本、高可靠的云存储服务。通过简单的Web Service接口提供无限的对象存储能力。通过高度可伸缩的设计和基于分布式存储设备的基础设施，给用户提供具有高吞吐、大容量、低成本、超稳定的非结构化数据存储服务。
对象存储低频存储
低频存储是 QingCloud 公有云对象存储服务 一种新的存储类型，相比于对象存储标准存储，提供了更低成本的存储服务，且与标准存储具有同样的高可靠性和低延迟，适用于存储读取频率较低、长期存储的数据。
java-dirtyJava 对象存储
java-dirty是一个快速的基于文件的对象存储系统，只能往存储里添加对象，使用内存映射文件技术。示例代码：Store<Foo> store = Store.of(Foo.class).
java-dirtyJava 对象存储
java-dirty 是一个快速的基于文件的对象存储系统，只能往存储里添加对象，使用内存映射文件技术
GoshawkDB分布式对象存储
GoshawkDB 是一个采用 Go 语言开发支持多平台的分布式的对象存储服务，支持事务以及容错。G
UCloud-对象存储US3
为互联网应用提供非结构化文件云存储的服务。
GoshawkDB分布式对象存储
GoshawkDB是一个采用Go语言开发支持多平台的分布式的对象存储服务，支持事务以及容错。GoshawkDB的事务控制是在客户端完成的。GoshawkDB服务器端使用AGPL许可，而Go语言客户端使
软件定义存储 对象存储 软件定义存储 存储一体机
QingStor® 对象存储是面向海量非结构化数据的通用数据存储平台，提供安全可靠、低成本的云端存储服务。QingStor® 对象存储兼容 AWS S3 API，构建于 AWS S3 API 或 SDKs 之上的应用可无缝对接 QingStor® 对象存储。
文件存储、对象存储、块存储分不清楚？看完就懂了
开源Linux
0
elfshaker高效存储二进制对象
elfshaker是一个低占用空间、高性能的版本控制系统，针对二进制文件进行了微调。elfshaker是一个用 Rust编程语言编写的CLI工具。它将目录快照存储到高度压缩的包文件中，并提供对存储文件
点赞
评论
收藏
分享

手机扫一扫分享

分享
举报