搭建FastDFS集群-一

发表于 2017-10-15   |   分类于 集群   |  

背景

  1. 公司的业务分布部署于不同的物理机器上
  2. 采用nginx做负载均衡将请求转发到合适的机器上
  3. 面对文件上传(例如: 图片, 视频等), 之前的同事使用rsync指令同步各物理机间的文件

rsync存在的缺陷

  1. 同步周期问题: 设置太短, 消耗服务器资源; 设置太长, 用户上传文件后, 有可能不能很快获取到该文件(比如文件上传到了A服务器, 取文件的时候, nginx将该请求转发到B服务器; 而此时, rsync还在循环间隔中)
  2. 高可用问题: 该指令无法做到高可用

我司目前仅仅存储一些小文件, 经过调研, 选择简单实用的FastDFS.

搭建集群

按VMware克隆多台实例构建分布式系统的方案制作三台虚拟机M_21(192.168.10.201), M_22(192.168.10.202), M_23(192.168.10.203).

最佳实践: 建议首先将M_21进行配置, 之后使用VMware的克隆功能复制出剩余的两台; 然后对相关配置文件修改IP和Port.

FastDFS 架构

安装

本节内容详细介绍依赖库和FastDFS的安装, 完全无坑, 可放心按步骤操作.

在集群的机器上均执行安装步骤, 或制作安装脚本.

安装基础库(gcc automake autoconf libtool g++)

1
2
yum -y install gcc automake autoconf libtool libevent
yum -y install gcc gcc-c++

安装libfastcommon

1
2
3
4
5
6
7
8
9
10
cd ~
mkdir workspaces devTools
mkdir /home/data
cd workspaces
mkdir github
cd github
git clone git@github:happyfish100/libfastcommon.git
cd libfastcommon
./make.sh
./make.sh install

看到如下信息表示安装成功

1
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h /usr/include/fastcommon

安装和配置git详见 一台电脑上同时使用github和gitlab

安装fastDFS

1
2
3
4
5
cd ~/workspaces/github
git clone git@github:happyfish100/fastdfs.git
cd fastdfs
./make.sh
./make.sh install

到此, fastDFS安装完毕, 接下来进行配置.

配置

配置文件目录: /etc/fdfs

1
2
3
#创建fastDFS 文件存储目录
cd /home/data
mkdir fastdfs

参数说明

https://github.com/happyfish100/fastdfs/blob/master/INSTALL

配置Tracker Server

1
2
3
cd /etc/fdfs
cp tracker.conf.sample tracker.conf
vim tracker.conf

对照着如下配置进行修改, 

1
2
3
4
# the base path to store data and log files
base_path=/home/data/fastdfs
# HTTP port on this tracker server
http.server_port=80

注: tracker server的默认监听端口22122, 如不冲突, 不建议修改

配置Storage Server

1
2
3
cd /etc/fdfs
cp storage.conf.sample storage.conf
vim storage.conf

对照如下配置进行修改, 

1
2
3
4
5
6
7
8
# the base path to store data and log files
base_path=/home/data/fastdfs
# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
store_path0=/home/data/fastdfs
# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=192.168.10.201:22122

注: storage server的默认监听端口23000, 如不冲突, 不建议修改

配置Client

1
2
3
cd /etc/fdfs
cp client.conf.sample client.conf
vim client.conf

对照如下配置进行修改, 

1
2
3
4
5
# the base path to store log files
base_path=/home/data/fastdfs
# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=192.168.10.201:22122

查看,启动,重启,停止指令

使用软链接简化指令

  • 启动: ln -s /usr/bin/fdfs_trackerd /usr/local/bin
  • 停止: ln -s /usr/bin/restart.sh /usr/local/bin
  • 重启: ln -s /usr/bin/stop.sh /usr/local/bin

查看

1
2
[root@Node fastdfs]# netstat -unltp|grep fdfs
tcp        0      0 0.0.0.0:22122               0.0.0.0:*                   LISTEN      14397/fdfs_trackerd

tracker Server正常运行.

启动

tracker server

  • 启动tracker Server: /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
  • 查看是否运行: ps -ef|grep fdfs
  • 以service方式启动tracker Server: service fdfs_trackerd start

storage server

service fdfs_storaged start

重启

tracker server

  • 重启tracker Ser: /usr/bin/restart.sh /etc/fdfs/tracker.conf
  • 查看是否运行: ps -ef|grep fdfs
  • 以service方式重启tracker Server: service fdfs_trackerd restart

storage server

service fdfs_storaged restart

停止

tracker server

  • 停止tracker Ser: /usr/bin/stop.sh /etc/fdfs/tracker.conf
  • 查看是否运行: ps -ef|grep fdfs
  • 以service方式停止tracker Server: service fdfs_trackerd stop

storage server

service fdfs_storaged stop

监控

tracker和storage服务均已启动, 且端口正常监听, 如何判断他们是否在通信呢?

使用如下指令: 

/usr/bin/fdfs_monitor /etc/fdfs/storage.conf

集群:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
[root@Node ~]# /usr/bin/fdfs_monitor /etc/fdfs/storage.conf
[2017-10-19 18:44:31] DEBUG - base_path=/home/data/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 192.168.10.201:22122

group count: 1

Group 1:
group name = group1
disk total space = 17709 MB
disk free space = 13932 MB
trunk free space = 0 MB
storage server count = 3
active server count = 3
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.10.201
                ip_addr = 192.168.10.201  ACTIVE
                http domain =
                version = 5.12
                .
                .
                .
                last_sync_update = 2017-10-19 18:43:43
                last_synced_timestamp = 2017-10-17 21:12:03 (1 days 19h:38m:10s delay)
        Storage 2:
                id = 192.168.10.202
                ip_addr = 192.168.10.202  ACTIVE
                http domain =
                version = 5.12
                .
                .
                .
                last_sync_update = 2017-10-19 18:44:06
                last_synced_timestamp = 2017-10-17 21:12:03 (1 days 19h:38m:10s delay)
        Storage 3:
                id = 192.168.10.203
                ip_addr = 192.168.10.203  ACTIVE
                http domain =
                version = 5.12
                .
                .
                .
                last_sync_update = 2017-10-19 18:43:43
                last_synced_timestamp = 1970-01-01 08:00:00 (never synced)

测试

1
cd /etc/fdfs

上传文件

1
2
cd ~/temp
echo "what happening" > ops.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[root@Node temp]# /usr/bin/fdfs_test /etc/fdfs/client.conf upload ~/temp/ops.txt
This is FastDFS client test program v5.12

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/
for more detail.

[2017-10-17 20:53:45] DEBUG - base_path=/home/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

tracker_query_storage_store_list_without_group:
        server 1. group_name=, ip_addr=192.168.10.201, port=23000

group_name=group1, ip_addr=192.168.10.201, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/wKgKyVnl_VmAahy0AAAAD1HyOTc464.txt
source ip address: 192.168.10.201
file timestamp=2017-10-17 20:53:45
file size=15
file crc32=1374828855
example file url: http://192.168.10.201/group1/M00/00/00/wKgKyVnl_VmAahy0AAAAD1HyOTc464.txt
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/wKgKyVnl_VmAahy0AAAAD1HyOTc464_big.txt
source ip address: 192.168.10.201
file timestamp=2017-10-17 20:53:45
file size=15
file crc32=1374828855
example file url: http://192.168.10.201/group1/M00/00/00/wKgKyVnl_VmAahy0AAAAD1HyOTc464_big.txt

下载文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@Node temp]# /usr/bin/fdfs_test /etc/fdfs/client.conf download group1 M00/00/00/wKgKyVnl_VmAahy0AAAAD1HyOTc464_big.txt
This is FastDFS client test program v5.12

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/
for more detail.

[2017-10-17 20:54:47] DEBUG - base_path=/home/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

storage=192.168.10.201:23000
download file success, file size=15, file save to wKgKyVnl_VmAahy0AAAAD1HyOTc464_big.txt

删除文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@Node temp]# /usr/bin/fdfs_test /etc/fdfs/client.conf delete group1 M00/00/00/wKgKyVnl_VmAahy0AAAAD1HyOTc464_big.txt
This is FastDFS client test program v5.12

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/
for more detail.

[2017-10-17 20:56:52] DEBUG - base_path=/home/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

storage=192.168.10.201:23000
delete file success

碰到的问题

使用yum安装软件报错

1
2
3
[root@Node github]# yum -y install git
Loaded plugins: fastestmirror, security
Determining fastest mirrors

解决办法: 

1
2
3
4
5
6
7
8
9
10
11
12
13
vim /etc/yum/pluginconf.d/fastestmirror.conf
[main]
enabled=0 #将1修改为0
verbose=0
always_print_best_host = true
socket_timeout=3
#  Relative paths are relative to the cachedir (and so works for users as well
# as root).
hostfilepath=timedhosts.txt
maxhostfileage=10
maxthreads=15
#exclude=.gov, facebook
#include_only=.nl,.de,.uk,.ie
1
2
3
4
5
6
7
8
9
10
11
12
13
vim /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=0 #将1修改为0
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=16&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release

误区

开始把各种服务的文件分得很细, 亦每个服务对应各自的目录; 其实没必要, track和storage服务产生的文件并不会产生冲突, 可共存于一个目录下.

相关内容

FastDFS架构剖析

搭建FastDFS集群-一

搭建FastDFS集群-二

转载

本文出自<<arccode>>, 欢迎转载, 转载请注明出处.
#FastDFS #File System #Linux #Nginx #集群
搭建FastDFS集群-二
FastDFS架构剖析