在Linux和AIX平台都有一部分存储产品使用操作系统自带的多路径软件，包括最常见的HP和IBM的部分存储产品，在Linux自带的多路径软件叫做multipath，这篇文章以IBM N系列存储在Linux平台的使用为例，讨论Linux平台multipath的使用。
1.确保安装以下的包：
device-mapper device-mapper-multipath
2.编辑配置文件/etc/multipath.conf [root@rac01 ~]# vi /etc/multipath.conf
defaults {
user_friendly_names yes max_fds max queue_without_daemon no flush_on_last_del yes }
## Blacklist non-SAN devices #sample：
#devnode \"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*\" #devnode \"^hd[a-z]\"
#devnode \"^cciss!c[0-9]d[0-9]*[p[0-9]*]\"
blacklist {
wwid 3600605b002b6b890163d47661e8d4fbe }
# FCP configuration, for a NetApp FAS3xxx / IBM Nxx00 # Vendor is \"NETAPP \ # Product is \"LUN\
devices {
device {
vendor \"NETAPP\" product \"LUN\"
getuid_callout \"/sbin/scsi_id -g -u -s /block/%n\" prio_callout \"/sbin/mpath_prio_ontap /dev/%n\" features \"1 queue_if_no_path\" hardware_handler \"0\"
path_grouping_policy group_by_prio failback immediate rr_weight uniform rr_min_io 128 path_checker directio } }
#下面的注释配置不是必须的，配置之后可以固定系统设备的名称，这在服务器重启设备文件对应的磁盘设备发生变化的情况下使用。
#multipaths #{
# multipath {
# wwid 360a9800065344e6f465a6d5773745558 # alias mocr1 # } # multipath {
# wwid 360a9800065344e6e536f6d586d306630 # alias mocr2 # } #}
3.启动multipathd服务，及设置其自动启动。 执行以下的命令：
#service multipathd restart
#chkconfig --level 345 multipathd on #chkconfig --list | grep multipathd
4.检查multipath聚合后的设备名，以及设备对应的链路情况。
[root@rac2 ~]# multipath -ll
mpath2 (360a9800065344e6f465a6d5773747545) dm-3 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:5 sdf 8:80 active ready running `- 2:0:0:5 sdp 8:240 active ready running
mpath1 (360a9800065344e6f465a6d5773746878) dm-1 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:3 sdd 8:48 active ready running `- 2:0:0:3 sdn 8:208 active ready running
mpath0 (360a9800065344e6f465a6d5773745558) dm-0 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:1 sdb 8:16 active ready running `- 2:0:0:1 sdl 8:176 active ready running
mpath9 (360a9800065344e6e536f6d586d33666e) dm-9 NETAPP,LUN size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:10 sdk 8:160 active ready running `- 2:0:0:10 sdu 65:64 active ready running
mpath8 (360a9800065344e6e536f6d586d32766b) dm-7 NETAPP,LUN size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:8 sdi 8:128 active ready running `- 2:0:0:8 sds 65:32 active ready running
mpath7 (360a9800065344e6e536f6d586d313851) dm-6 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:6 sdg 8:96 active ready running `- 2:0:0:6 sdq 65:0 active ready running
mpath6 (360a9800065344e6e536f6d586d307464) dm-4 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:4 sde 8:64 active ready running `- 2:0:0:4 sdo 8:224 active ready running
mpath5 (360a9800065344e6e536f6d586d306630) dm-2 NETAPP,LUN size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:2 sdc 8:32 active ready running `- 2:0:0:2 sdm 8:192 active ready running
mpath4 (360a9800065344e6f465a6d577376304f) dm-8 NETAPP,LUN size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active
|- 1:0:0:9 sdj 8:144 active ready running `- 2:0:0:9 sdt 65:48 active ready running
mpath3 (360a9800065344e6f465a6d5773766c58) dm-5 NETAPP,LUN size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:7 sdh 8:112 active ready running `- 2:0:0:7 sdr 65:16 active ready running
从以上的结果可以看出，每个磁盘设备对应的是两条链路，根据不存的存储以及不同的配置会有所不同。每个设备都有两个multipath设备名，分别是mpath[x]和dm-[x]，mpath位于/dev/mpath目录下，dm-[x]位于/dev目录下。通常推荐使用/dev/mpath目录下的设备名，这对于我们来说更加的方便。
5.下面我们重点关注一下fdisk -l执行的结果。 [root@rac2 ~]# fdisk -l
Disk /dev/sda: 145.9 GB, 145999527936 bytes 255 heads, 63 sectors/track, 17750 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sda1 * 1 64 514048+ 83 Linux /dev/sda2 65 6438 51199155 83 Linux
/dev/sda3 6439 7743 10482412+ 82 Linux swap / Solaris /dev/sda4 7744 17750 80381227+ 5 Extended /dev/sda5 7744 17750 80381196 83 Linux
Disk /dev/sdb: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdb1 1 261 2096451 83 Linux
Disk /dev/sdd: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdd1 1 261 2096451 83 Linux
Disk /dev/sdf: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sdf1 1 261
Disk /dev/sdc: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sdc1 1 261
Disk /dev/sde: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sde1 1 261
Disk /dev/sdh: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sdh1 1 19581
Disk /dev/sdg: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Blocks Id System 2096451 83 Linux
Blocks Id System 2096451 83 Linux
Blocks Id System 2096451 83 Linux
Blocks Id System 83 Linux
157284351
Device Boot Start End Blocks Id System /dev/sdg1 1 261 2096451 83 Linux
Disk /dev/sdi: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdi1 1 19581 157284351 83 Linux
WARNING: GPT (GUID Partition Table) detected on '/dev/sdj'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT).
Disk /dev/sdj: 4241.2 GB, 4241280204800 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdj1 1 267350 2147483647+ ee EFI GPT
WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).
Disk /dev/sdk: 4241.2 GB, 4241280204800 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks /dev/sdk1 1 267350 2147483647+
Disk /dev/sdl: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks /dev/sdl1 1 261 2096451
Disk /dev/sdm: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks /dev/sdm1 1 261 2096451
Disk /dev/sdn: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks /dev/sdn1 1 261 2096451
Id System ee EFI GPT
Id System 83 Linux
Id System 83 Linux
Id System 83 Linux
Disk /dev/sdp: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdp1 1 261 2096451 83 Linux
Disk /dev/sdo: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sdo1 1 261
Disk /dev/sdr: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sdr1 1 19581
Disk /dev/sdq: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sdq1 1 261
Disk /dev/sds: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/sds1 1 19581 Blocks Id System 2096451 83 Linux
Blocks Id System 83 Linux
Blocks Id System 2096451 83 Linux
Blocks Id System 83 Linux
157284351 157284351
WARNING: GPT (GUID Partition Table) detected on '/dev/sdt'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT).
Disk /dev/sdt: 4241.2 GB, 4241280204800 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdt1 1 267350 2147483647+ ee EFI GPT
WARNING: GPT (GUID Partition Table) detected on '/dev/sdu'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT).
Disk /dev/sdu: 4241.2 GB, 4241280204800 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdu1 1 267350 2147483647+ ee EFI GPT
这之前的设备是/dev/sd[n]的设备名，这部分设备是没有聚合的设备，根据上面显示的两条链路，那么相同的LUN应该对应两个sd[n]设备，这部分不是使用的设备，可以不关注。
Disk /dev/dm-0: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/dm-0p1 1 261
Disk /dev/dm-1: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/dm-1p1 1 261
Disk /dev/dm-2: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/dm-2p1 1 261
Disk /dev/dm-3: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End /dev/dm-3p1 1 261
Blocks Id 83 Blocks Id 83 Blocks Id 83 Blocks Id 83 System Linux
System Linux
System Linux
System Linux
2096451 2096451 2096451 2096451 Disk /dev/dm-4: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/dm-4p1 1 261 2096451 83 Linux
Disk /dev/dm-5: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/dm-5p1 1 19581 157284351 83 Linux
Disk /dev/dm-6: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/dm-6p1 1 261 2096451 83 Linux
Disk /dev/dm-7: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/dm-7p1 1 19581 157284351 83 Linux
WARNING: GPT (GUID Partition Table) detected on '/dev/dm-8'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT).
Disk /dev/dm-8: 4241.2 GB, 4241280204800 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/dm-8p1 1 267350 2147483647+ ee EFI GPT
WARNING: GPT (GUID Partition Table) detected on '/dev/dm-9'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 4.2 TB (4241280204800 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT).
Disk /dev/dm-9: 4241.2 GB, 4241280204800 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/dm-9p1 1 267350 2147483647+ ee EFI GPT
这部分设备分别是/dev/dm-[n]以及它们的分区/dev/dm-[n]p1，这部分是安装好multipath多路径软件及针对每个磁盘设备分区后产生的，但实际在/dev/目录下不存在dm-[n]p1这部分设备文件，只需要了解即可。
Disk /dev/dm-10: 2146 MB, 2146765824 bytes
255 heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-10 doesn't contain a valid partition table
Disk /dev/dm-11: 2146 MB, 2146765824 bytes 255 heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-11 doesn't contain a valid partition table
Disk /dev/dm-12: 2146 MB, 2146765824 bytes 255 heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-12 doesn't contain a valid partition table
Disk /dev/dm-13: 2146 MB, 2146765824 bytes 255 heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-13 doesn't contain a valid partition table
Disk /dev/dm-14: 161.0 GB, 161059175424 bytes 255 heads, 63 sectors/track, 19580 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-14 doesn't contain a valid partition table
Disk /dev/dm-15: 2146 MB, 2146765824 bytes 255 heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-15 doesn't contain a valid partition table
Disk /dev/dm-16: 161.0 GB, 161059175424 bytes 255 heads, 63 sectors/track, 19580 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-16 doesn't contain a valid partition table
Disk /dev/dm-17: 4241.2 GB, 4241280170496 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-17 doesn't contain a valid partition table
Disk /dev/dm-18: 4241.2 GB, 4241280170496 bytes 255 heads, 63 sectors/track, 515639 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-18 doesn't contain a valid partition table
Disk /dev/dm-19: 2146 MB, 2146765824 bytes 255 heads, 63 sectors/track, 260 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-19 doesn't contain a valid partition table
这部分设备名是/dev/dm-[n]，这部分设备并不是multipath多路径软件聚合后的磁盘设备，而是做完分区的分区设备文件，再刚对前面的/dev/dm-0，dev/dm-1做完分区之后执行fdisk -l是看不到这部分设备信息的，只有重启之后才看到这些设备信息并产生了相应的设备文件。例如，/dev/dm-10对应的是/dev/dm-0p1，/dev/dm-11对应的是/dev/dm-1p1，以此类推，这部分设备文件可以被使用，但由于这些设备文件对应multipath聚合的磁盘设备比较麻烦，所以并不推荐使用这种方式。
下面讨论/dev/mpath目录下的设备文件：
安装完multipath软件并配置完成之后会在/dev/mpath目录下生成聚合后的磁盘设备文件，实际是/dev/dm-[n]的链接文件：
[root@rac2 mpath]# ls -al total 0
drwxr-xr-x 2 root root 440 Sep 21 02:05 . drwxr-xr-x 13 root root 7060 Sep 21 02:12 ..
lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath0 -> ../dm-0 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath0p1 -> ../dm-19 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath1 -> ../dm-1 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath1p1 -> ../dm-10 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath2 -> ../dm-2 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath2p1 -> ../dm-17 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath3 -> ../dm-5 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath3p1 -> ../dm-12 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath4 -> ../dm-7 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath4p1 -> ../dm-14 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath5 -> ../dm-3 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath5p1 -> ../dm-18 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath6 -> ../dm-4 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath6p1 -> ../dm-11 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath7 -> ../dm-6 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath7p1 -> ../dm-13 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath8 -> ../dm-9 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath8p1 -> ../dm-16 lrwxrwxrwx 1 root root 7 Sep 21 02:05 mpath9 -> ../dm-8 lrwxrwxrwx 1 root root 8 Sep 21 02:05 mpath9p1 -> ../dm-15
从上面的输出结果我们可以非常容易的了解到磁盘分区设备文件与磁盘设备文件之间的对应关系，所以使用/dev/mpath目录下的文件比使用/dev/目录下磁盘设备和磁盘分区设备文件都以dm-[n]方式表示的文件更加方便。之后我们在使用oracleasm创建ASM磁盘的时候，使用fdisk -l
/dev/mpath/mpath[n]p1查看磁盘信息，直接使用/dev/mpath/mpath[n]p1作为ASM磁盘，非常的方便。
还需要注意如下几点：
1).经过multipath多路径软件聚合后的设备文件(/dev/mpath/mpath[n])不会随着服务器的重启，对应的磁盘设备发生变化，而/dev/sd[n]对应的磁盘LUN重启后是可能发生变化的。
2).如果单个磁盘大于2TB，那么在分区的时候不要使用fdisk工具(传统的MBR)，应该使用parted(gpt)，使用parted工具创建gpt分区的步骤是： #parted
PARTED> select /dev/mpath/mpath0 PARTED> mklabel gpt
PARTED> mkpart primary 0 100%
3).如果我们使用ASM作为数据库数据文件的存储方式，那么单个ASM磁盘的大小不能超过2TB，所以在系
统分区的时候就应该做好更小的分区，例如，/dev/mpath/mpath0磁盘的大小为3.8TB，那么在分区的时候应该做如下操作： #parted
PARTED> select /dev/mpath/mpath0 PARTED> mklabel gpt
PARTED> mkpart primary 0 50% PARTED> mkpart primary 50% 100%
完成以上分区操作之后，重启服务器，在/dev/mpath/目录下就会看到mpath0p1和mpath0p2两个设备文件，将这两个设备文件作为ASM磁盘即可。
因篇幅问题不能全部显示，请点此查看更多更全内容