存储控制器故障导致RAC数据库
ORA-600[kclchkblk_4]错误故障修复

大家好,我是黄伟老师。前几天,一位网友的数据库因为双存储控制器故障,导致一套跑在该存储上的Oracle11gR2 RAC数据库异常,出现ORA-600的错误,数据库无法正常启动,影响业务系统的正常使用和生产,于是找我帮忙处理。处理完这个case之后,现在我把当时的情景尽可能的还原出来,录制成这个视频,希望各位观看学习之后,将来如果在你的工作中,也出现类似的数据库故障,可以起到一个参考指导的作用,提供一个分析的思路,帮各位快速定位和恢复启动数据库。当然,这里面会涉及到一些Oracle数据库的运行原理和一些专业术语,比如SCN、Checkpoint,不过我会尽可能尝试着把它们讲清楚。

一 首先,接到帮助请求

二 分析错误和定位问题

对于不熟悉Oracle错误的同学,可以参考metalink(Oracle官方技术支持网站)官方文档(What is an ORA-600 Internal Error? (Doc ID 146580.1))的解释:

什么是ORA-600?

*ORA-600 errors are raised from the kernel code of the Oracle RDBMS software when an internal inconsistency is detected or an unexpected condition is met. This situation is not necessarily a bug as it might be caused by problems with the Operating System, lack of resources, hardware failures, etc.

With the ORA-600 error comes a list of arguments in square brackets. The first of these arguments tells us from where in the code the error was caught and thus is the key information in identifying the problem. This argument is either a number or a character string. The remaining arguments are used to supply further information (e.g. values of internal variables etc).

Whenever an ORA-600 error is raised a trace file is generated and an entry written to the alert.log with details of the trace file location. Starting with Oracle Database 11g Release 1, the diagnosability infrastructure was introduced which places the trace and core files into a location controlled by the DIAGNOSTIC_DEST initialization parameter when an incident, such as an ORA-600 is created. For earlier versions, the trace file will be written to either USER_DUMP_DEST (if the error was caught in a user process) or BACKGROUND_DUMP_DEST (if the error was caught in a background process like PMON or SMON). The trace file contains vital information about what led to the error condition.*

所以,我们从错误信息参数中,可以猜测出,ORA-00600: internal error code, arguments: [kclchkblk_4]应该是跟数据块错误有关的可能性很大。再结合这个案例,网友提供的信息说存储控制器故障,猜测多半情况下,数据库在运行(数据库进程通过内存空间,频繁和存储设备读写数据)的过程中,存储控制器一旦故障,多数情况下会导致内存中的脏数据(被修改过的数据)无法写入数据库,或者数据库检查点信息无法同步到控制文件、日志文件、数据文件中。那么数据库必死无疑,无法再继续工作下去了,当我们下次手工启库、或者数据库自动重启的过程中,数据库在open阶段,如果数据库无法完成一致性校验或者Instance Recovery实例恢复,那么数据库就无法Open,不能对业务系统提供服务了。

那么,本案例中,这个ORA-00600: internal error code, arguments: [kclchkblk_4]究竟表示什么意思呢?后面的几个参数又表示什么呢?我想,如果我们能把这几个参数搞清楚的话,那么这个问题也许就有眉目了。

于是,从metalink上找到跟该错误有关的几篇文章:

1 ORA-600 [kclchkblk_4] (Doc ID 1522401.1)

ORA-600 [kclchkblk_4] (Doc ID 1522401.1)这篇文档中,获取的信息有限,只是说:目前该错误并没有完全公开发布出来,因为它通常伴随ORA-600错误出现,而且被认为是已知确认的bug。

2 Bug 14351566 – ORA-600 [kclchkblk_4] ORA-600 [2662] when doing flash back (Doc ID 14351566.8)

该文档描述的是数据库多次执行flashback之后,导致的bug。我们该故障场景中,数据库并没有执行flashback的操作,排除。

3 ORA-600 [2662] “Block SCN is ahead of Current SCN” (Doc ID 28929.1)

 

不过,从该指导文档中可以获取到对此案例有用的指导信息。

① 该错误比较常见,通常由于存储故障、或者服务器异常掉电,数据库abort关闭等,导致的数据块里记录的SCN与数据库当前SCN不一致导致的。解决问题的常规办法就是通过redo log恢复数据,如果依然不奏效的话,可能需要通过非常规手段来修改数据库的SCN,使之一致,进而避免该错误发生。

这里,我在一套测试环境上模拟了该错误,并且手工修复了该错误:

Sat Dec 19 08:55:28 2020
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_8251.trc  (incident=28953):
ORA-00600: internal error code, arguments: [2662], [0], [112587826], [0], [112587881], [4194432], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_28953/orcl_ora_8251_i28953.trc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_8251.trc:
ORA-00600: internal error code, arguments: [2662], [0], [112587826], [0], [112587881], [4194432], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_8251.trc:
ORA-00600: internal error code, arguments: [2662], [0], [112587826], [0], [112587881], [4194432], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 8251): terminating the instance due to error 600
Instance terminated by USER, pid = 8251
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (8251) as a result of ORA-1092
Sat Dec 19 08:55:33 2020

当然,这是另外一个case,我们暂且不去聚焦这个ORA-600 26662的错误如果解决。如果大家感兴趣且有需求的话,我可以另外录制一个视频去讲解它。

②从该文档中,倒是给出了有意义的错误参数:

ARGUMENTS:
Arg [a] Current SCN WRAP
Arg [b] Current SCN BASE
Arg [c] dependent SCN WRAP
Arg [d] dependent SCN BASE
Arg [e] Where present this is the DBA where the dependent SCN came from.

可以看出,这些错误参数,跟前面提到的错误参数格式非常一致:

RROR:

Format: ORA-600 [kclchkblk_4] [a] [b] [c] [d] [e]

但是,Oracle官方并没有给出这里的几个参数表示的具体意义。

那么,进一步猜测,我们这个故障的错误是不是也可以借鉴于该文档中给出的suggestion来尝试恢复呢?

③ 官方给出的几个suggestion


SUGGESTIONS:        
     
  There are different situations where ORA-600 [2662] can be raised.

  It can be raised on startup or during database operation.

  If not using Parallel Server, check that 2 instances have not mounted
  the same database.

  Check for SMON traces and have the alert.log and trace files ready
  to send to support.

  Check the SCN difference [argument d]-[argument b].

  If the SCNs in the error are very close, then try to shutdown and startup
  the instance several times. 

  In some situations, the SCN increment during startup may permit the 
  database to open. Keep track of the number of times you attempted a 
  startup.

到这里,有了一个可以尝试的解决问题的思路了,如果ORA-600 [kclchkblk_4] [a] [b] [c] [d] [e]这里的几个参数也表示某种与SCN有关的信息的话,那么,是不是可以根据该建议尝试恢复呢?

三 解决问题

1 start instance,确认错误参数是否推进

Tue Dec 08 15:04:02 2020
Thread 1 advanced to log sequence 9 (thread open)
Thread 1 opened at log sequence 9
  Current log# 1 seq# 9 mem# 0: +DATA/jhzmpr/onlinelog/group_1.262.1045198071
  Current log# 1 seq# 9 mem# 1: +DATA/jhzmpr/onlinelog/group_1.263.1045198071
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Dec 08 15:04:02 2020
SMON: enabling cache recovery
SUCCESS: diskgroup ARCH was mounted
Archived Log entry 1831 added for thread 1 sequence 8 ID 0xddd7c8e4 dest 1:
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_17070.trc  (incident=800345):
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171642483], [0], [171385871], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/incident/incdir_800345/jhzmpr1_ora_17070_i800345.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
NOTE: dependency between database jhzmpr and diskgroup resource ora.ARCH.dg is established
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_17070.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171642483], [0], [171385871], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_17070.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171642483], [0], [171385871], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 17070): terminating the instance due to error 704
Instance terminated by USER, pid = 17070
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (17070) as a result of ORA-1092
Tue Dec 08 15:04:05 2020
ORA-1092 : opitsk aborting process

果不其然,argument d比上次推进了:从之前的171365869推进到171385871。同时,argument b保持不变,其它错误信息也保持一致。

2 继续上述操作

[root@jhzmpr01 ~]# srvctl start db -d jhzmpr
PRCR-1079 : Failed to start resource ora.jhzmpr.db
CRS-5017: The resource action "ora.jhzmpr.db start" encountered the following error:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171642483], [0], [171405876], [], [], [], [], [], [], []
Process ID: 21207
Session ID: 3175 Serial number: 3
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/jhzmpr01/agent/crsd/oraagent_oracle//oraagent_oracle.log".

CRS-5017: The resource action "ora.jhzmpr.db start" encountered the following error:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171642483], [0], [171405882], [], [], [], [], [], [], []
Process ID: 43944
Session ID: 3175 Serial number: 3
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/jhzmpr02/agent/crsd/oraagent_oracle//oraagent_oracle.log".

CRS-2674: Start of 'ora.jhzmpr.db' on 'jhzmpr01' failed
CRS-2674: Start of 'ora.jhzmpr.db' on 'jhzmpr02' failed
CRS-2632: There are no more servers to try to place resource 'ora.jhzmpr.db' on that would satisfy its placement policy
[root@jhzmpr01 ~]# 

3 直至错误参数一致

Tue Dec 08 15:44:31 2020
SMON: enabling cache recovery
SUCCESS: diskgroup ARCH was mounted
Archived Log entry 1846 added for thread 1 sequence 22 ID 0xddd7c8e4 dest 1:
NOTE: dependency between database jhzmpr and diskgroup resource ora.ARCH.dg is established
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_54315.trc  (incident=1248369):
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171679465], [0], [171665974], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/incident/incdir_1248369/jhzmpr1_ora_54315_i1248369.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_54315.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171679465], [0], [171665974], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_54315.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171679465], [0], [171665974], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 54315): terminating the instance due to error 600
Instance terminated by USER, pid = 54315
ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:11782:10822} */...
opiodr aborting process unknown ospid (54315) as a result of ORA-1092
Tue Dec 08 15:44:34 2020
ORA-1092 : opitsk aborting process

4 启动数据库

Tue Dec 08 15:46:51 2020
Starting ORACLE instance (normal)
...
...
tarted redo scan
SUCCESS: diskgroup FLASH was mounted
Completed redo scan
 read 11 KB redo, 5 data blocks need recovery
Started redo application at
 Thread 1: logseq 23, block 3
Recovery of Online Redo Log: Thread 1 Group 5 Seq 23 Reading mem 0
  Mem# 0: +DATA/jhzmpr/onlinelog/group_5.273.1046776933
  Mem# 1: +FLASH/jhzmpr/onlinelog/group_5.256.1046776933
Completed redo application of 0.01MB
Completed crash recovery at
 Thread 1: logseq 23, block 25, scn 171685974
 5 data blocks read, 5 data blocks written, 11 redo k-bytes read
ARCH: STARTING ARCH PROCESSES
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:

        ALTER TABLESPACE <tablespace_name> ADD TEMPFILE

        Alternatively, if these temporary tablespaces are no longer
        needed, then they can be dropped.
          Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_smon_57089.trc  (incident=1280273):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/incident/incdir_1280273/jhzmpr1_smon_57089_i1280273.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

5 至此,该错误已经消失了,数据库可以正常访问了。

但是,alert日志里抛出一个报警,让我们给数据库添加临时文件。为什么呢?这是因为,这位网友之前根据网上的一些帖子尝试对数据库删除临时文件,但是没能恢复数据库。这里,我们可以先不去理会临时文件的文件,可以让数据库使用SYSAUX那个表空间充当临时表空间,如果需要的话。

四 补充

1 SCN

①定义

System Change Number,数据库“内部时钟”,数据库中每时每刻都在发生改变,这些改变对应这一个具体的时间点儿,每个具体的时间点儿对应一个数字编号,这个编号就是SCN值。比如:2020年12月19日18时30分52秒,对应一个SCN号,35789713456。注意,不同时间点儿对应不同的SCN,数据库一旦启动运行,SCN就不停向前滚动。

② 作用

对于关系型数据库而言,用于事务的ACID控制和管理,以及数据库的备份和恢复的标准。

③位置

数据文件、控制文件、日志文件、数据文件头、数据块中都有记录。

④查看
SQL> select dbms_flashback.get_system_change_number scn from dual;

       SCN
----------
 112573590

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
  112573593

SQL> select file#,CHECKPOINT_CHANGE#,CHECKPOINT_TIME from v$datafile;

     FILE# CHECKPOINT_CHANGE# CHECKPOINT_TIME
---------- ------------------ -------------------
         1          112563728 2020/12/19 15:38:19
         2          112563728 2020/12/19 15:38:19
         3          112563728 2020/12/19 15:38:19
         4          112563728 2020/12/19 15:38:19
         5          112563728 2020/12/19 15:38:19
         6          112563728 2020/12/19 15:38:19
         7          112563728 2020/12/19 15:38:19

7 rows selected.

SQL> select group#,FIRST_CHANGE#,FIRST_TIME from v$log;

    GROUP# FIRST_CHANGE# FIRST_TIME
---------- ------------- -------------------
         1     112563728 2020/12/19 15:38:19
         2     112544279 2020/12/19 11:35:21
         3     112551742 2020/12/19 12:37:46

SQL> 
⑤结构和计数原理

占6个字节,48个二进制位数。前2个字节构成SCN_WRAP,最大值为2的16次方,65536。后4个字节构成SCN_BASE,最大值为2的32次方,4294967296。SCN总大小为2的48次方=281474976710656,281万亿个计数。

假定初始情况下,SCN_WRAP=0,SCN_BASE从1开始计数,当达到最大值4294967296时,就向前进一位,SCN_WRAP从0变为1,同时SCN_BASE从0重新开始计数,下一次达到最大值时,再向前进一位,SCN_WRAP再加1,SCN_BASE再次从0重新开始计数,以此循环往复。直至SCN用完耗尽为止。

SCN = (SCN_WRAP * 4294967296) + SCN_BASE

举个生活中的例子,类似于我们的计时秒表,当秒针从0跳到59时,分针从0变为1,同时秒针重新从0开始,直到到达下一个59秒,分针从1跳到2….

Oracle为什么要这么设计呢?这很好理解,例如,我们日常生活中会说3分28秒,很少有人直接说208秒,对不对? 就图一方便计数呗。

从Oracle12.2开始,SCN改为8个字节了,所以,其总大小变为2的63次方了。一个更大的数了。

2 Checkpoint

A checkpoint is the act of flushing modified, cached database blocks to disk. Normally, when you make a change to a block the modifications of that block are made to a memory copy of the block. When you commit the block, it is not written to the datafile but it is written to the REDO LOG, so we can “replay” a transaction in the event of an instance failure. Eventually, the system will checkpoint your modified blocks to the datafiles on disk.

  • When all dirty buffers in the Oracle SGA are written to datafiles by DBWR.

  • When a redo log switch occurs.

  • Whenever the time set by the LOG_CHECKPOINT_TIMEOUT parameter is reached.

  • By issuing the command ‘alter system switch logfile’ or ‘alter system checkpoint’.

说一个简单的场景,假定我们要执行下述SQL,修改表里的一条记录:

update department set emp_name=’Huangwei’ where emp_name=’huangwei’;

 

update department set emp_name='Huangwei' where emp_name='huangwei';

数据库肯定不会直接到磁盘中去修改该数据,因为如果把数据读入到内存中,在内存中读写数据,比读写磁盘要快不止一个数量级。所以,

① 数据库先到内存中去找要修改的数据,如果内存中没有该数据的话,Oracle数据库的服务器进程把需要的数据从磁盘中读取到内存中;注意,Oracle不会只读这一条记录的数据,而是读取该记录所在的数据块,把整个数据块的内容全部读入内存。为了性能和效率,Oracle数据库肯定不会傻傻的一次只读一条记录的数据,最小IO单位是database block,其它关系型数据库也类似。

② 然后准备在内存中对该数据进行修改;真正修改之前,还要先做另外一件事儿:就是把这份数据放到一个undo回滚段中,用于确保事务的一致性读或者回滚等操作。接下来,给该SQL语句分配一个事务号,说白了就是一个SCN号。并开始执行真正的修改,同时,在redo log buffer里对该操作进行完整的记录(如:2020年12月19日19点40分,这个时间点儿其实就是这个个SCN,将‘huangwei’修改成了‘Huangwei’),为什么要记录这个呢?答案是便于恢复数据,这里就不再继续扩展。注意,该修改过的新数据(称为dirty buffer)并不一定马上写回到磁盘中;

③ 当我们执行commit时,这时,该修改过的新数据(称为dirty buffer)写回到磁盘中去了吗???

④ 也不一定,因为,此时只要保证redo log buffer里,记录的关于本条SQL修改的信息被LGWR后台进程写出到redo log file里,就算完成了。在不考虑数据库包含DATAGUARD备库的情况下,此时就可以立即返回commit成功,即告诉你,你的修改已经生效了。为什么呢?这就是Oracle的redo log file的作用,如果此时数据库出现instance crash的话,我的修改还在吗?在,Oracle可以利用写入到redo logfile里的内容,进行“redo”,或者叫“replay”。这也就是它为啥叫redo了。

⑤那问题来了?到底什么时候,我的这个修改被真正写回到磁盘中呢?答案,看情况。

比如:如果此时,数据库的database buffer cache不够用了,数据库会根据LRU算法把旧的脏数据写出到磁盘中;

或者:此时发生了redo logfile switch;

再:手工执行了 alter system checkpoint;等。

注意,把内存中的脏数据写出到数据文件是DBWR后台进程干的活儿;记录检查点信息到控制文件、同步到数据文件、日志文件中,则是CKPT后台进程的职责。

The checkpoint process updates the control file when a checkpoint is performed. Remember that a checkpoint operation writes all changed (dirty) blocks to disk. By recording this fact in the control file, Oracle knows what redo records need to be applied in the event of an instance failure. To recover from an instance failure, Oracle needs to apply all redo generated after the last checkpoint recorded in the control file.

  • Log switches cause checkpoints. Checkpoints do not cause log switches.

  • DBWR writes dirty blocks from the buffer cache to disk — that does not happen when you “commit” — LGWR is involved during a commit.

五 小结

1 解决该数据库故障并不难

只是需要不停的startup,shudown,再startup。难点在于去分析和定位该问题,找到一些相关的线索,并支撑和问题相关的错误信息。

2 如何规避该问题

如果这位网友的数据库,在故障发生之前,执行了合适的热备份+归档的话,遇到该故障的时候,可能就没有这么被动。

3 备份很重要

在遇到数据库故障或者错误的场景下,一定要尽可能的先对数据库进行备份,保留数据库故障现场,便于后续恢复不成功的情况下,可以多次重新尝试。另外,如果的确没有十足把握的话,不要轻易尝试对数据库执行各种恢复操作。总结该故障的过程中,发现系统报错时间:

Sun Dec 06 05:16:31 2020
Thread 1 advanced to log sequence 652 (LGWR switch)
  Current log# 2 seq# 652 mem# 0: +DATA/jhzmpr/onlinelog/group_2.264.1045198071
  Current log# 2 seq# 652 mem# 1: +DATA/jhzmpr/onlinelog/group_2.265.1045198073
Sun Dec 06 05:16:33 2020
Archived Log entry 1816 added for thread 1 sequence 651 ID 0xdd0af974 dest 1:
Sun Dec 06 05:45:39 2020
WARNING: Write Failed. group:2 disk:2 AU:54678 offset:49152 size:8192
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_dbwb_7076.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 7778 logical extent 0 of file 315 in group 2 on disk 2 allocation unit 54678
KCF: read, write or open error, block=0xf3106 online=1
file=10 '+DATA/jhzmpr/datafile/aoi.315.1057790147'
error=15081 txt: ''
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_dbwb_7076.trc:
Sun Dec 06 05:45:39 2020
WARNING: Read Failed. group:2 disk:0 AU:340 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 261 in group [2.4004072768] from disk DATA_0000  allocation unit 340 reason error; if possible, will try another mirror side
Sun Dec 06 05:45:39 2020
...
...
Sun Dec 06 05:45:46 2020
License high water mark = 102
Termination issued to instance processes. Waiting for the processes to exit
Sun Dec 06 05:46:00 2020
Instance termination failed to kill one or more processes
Instance terminated by CKPT, pid = 7080
Sun Dec 06 05:46:00 2020
第一次手工尝试恢复故障:
Sun Dec 06 14:28:32 2020
Starting ORACLE instance (normal)
...
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_6706.trc:
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00312: online log 2 thread 1: '+DATA/jhzmpr/onlinelog/group_2.265.1045198073'
ORA-00312: online log 2 thread 1: '+DATA/jhzmpr/onlinelog/group_2.264.1045198071'
ORA-742 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:11782:2} */...
Sun Dec 06 14:28:57 2020
Shutting down instance (abort)

第二次
Sun Dec 06 23:37:03 2020
Starting ORACLE instance (normal)
...
Mon Dec 07 00:27:06 2020
alter database clear unarchived logfile group 2
ORA-1624 signalled during: alter database clear unarchived logfile group 2...
Mon Dec 07 00:28:08 2020
ALTER DATABASE RECOVER  database using backup controlfile until cancel  
Media Recovery Start
 started logmerger process
Mon Dec 07 00:28:08 2020

...
Mon Dec 07 00:29:27 2020
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_pr00_43688.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DATA/jhzmpr/datafile/system.256.1045198003'
ORA-1547 signalled during: ALTER DATABASE RECOVER    CANCEL  ...
ALTER DATABASE RECOVER CANCEL 
ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...
Mon Dec 07 00:29:41 2020
alter database open resetlogs
ORA-1194 signalled during: alter database open resetlogs...
Mon Dec 07 00:30:17 2020
ALTER DATABASE RECOVER  database using backup controlfile until cancel allow 1 corruption  
Media Recovery Start
Serial Media Recovery started
...
Mon Dec 07 22:28:46 2020
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/trace/jhzmpr1_ora_39995.trc  (incident=576369):
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [171642483], [0], [171225822], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/jhzmpr/jhzmpr1/incident/incdir_576369/jhzmpr1_ora_39995_i576369.trc
Use ADRCI or Support Workbench to package the incident.

4 祸不单行,福无双降,顺手帮忙

这个库处理好之后,那哥们儿出于对我的信任,又请我帮忙处理同样跑在该存储设备上另外几个同样没能幸免于难的数据库故障。那些使用该存储设备作为数据库存储的好几个数据库都挂了,

5 参考链接

https://topic.alibabacloud.com/a/ora-00600-kclchkblk_4-error-recovery-method-introduction-12_8_8_20042453.html

http://blog.itpub.net/29863023/viewspace-2132815/

 

6 感谢

各位老铁,如果觉得我的视频内容有用,能帮助到你,请帮忙一键三连,顺手转发给更多的朋友。

访问我的个人站点,查看本视频中提到的所有文档和资料素材: http://oracleonlinux.cn/

感谢大家的收看,我们下一期视频节目再见。

最后,欢迎大家关注我的个人微信公众号。

 

 

PostgreSQL主从同步配置,切换步骤

零 说明

本文档详细记录如何一步一步在两台Linux服务器上,搭建PostgreSQL数据库主从模式的高可用数据库环境,以及如何手工切换主备库,及其注意事项。

为了实现数据库对于应用层的透明,配合浮动IP来实现,即使数据库执行主备切换,应用层依然通过浮动IP来访问数据库。即:后端数据库执行主备切换前后,应用系统无需做任何更改。

搭建环境选择在CentOS 7+PostgreSQL 9.4.10上。

基本思路和流程大致如下:

1 主库真实IP为172.16.11.148,配置浮动IP为172.16.11.121,主机名为pgprimary,在其上安装并初始化PostgreSQL数据库;

2 备库真实IP为172.16.11.149,配置浮动IP为172.16.11.121,主机名为pgstandby,在其上只安装PostgreSQL数据库软件;

3 通常情况下,浮动IP 172.16.11.121运行在主库上,当主库出故障时,手工执行数据库的主备切换,备库成为新主库,然后新主库上启用浮动IP;

4 修复并重配原主库,使其成为新主库的备库;

一 环境

两台数据库服务器的主机名、文件系统、OS level、网络配置分别如下:

1主库:

[root@pgprimary ~]# hostname
pgprimary
[root@pgprimary ~]# df -Th
文件系统                类型      容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root xfs        47G  1.1G   46G    3% /
devtmpfs                devtmpfs  908M     0  908M    0% /dev
tmpfs                   tmpfs     920M     0  920M    0% /dev/shm
tmpfs                   tmpfs     920M  8.8M  911M    1% /run
tmpfs                   tmpfs     920M     0  920M    0% /sys/fs/cgroup
/dev/sda1               xfs      1014M  142M  873M   14% /boot
tmpfs                   tmpfs     184M     0  184M    0% /run/user/0
[root@pgprimary ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root@pgprimary ~]# uname -rm
3.10.0-862.el7.x86_64 x86_64
[root@pgprimary ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.148  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::58b2:50cd:9d7c:8b23  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)
        RX packets 21338  bytes 4481980 (4.2 MiB)
        RX errors 0  dropped 2378  overruns 0  frame 0
        TX packets 270  bytes 31662 (30.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgprimary ~]#

2 备库:

[root@pgstandby ~]# hostname
pgstandby
[root@pgstandby ~]# df -Th
文件系统                类型      容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root xfs        47G  1.1G   46G    3% /
devtmpfs                devtmpfs  908M     0  908M    0% /dev
tmpfs                   tmpfs     920M     0  920M    0% /dev/shm
tmpfs                   tmpfs     920M  8.8M  911M    1% /run
tmpfs                   tmpfs     920M     0  920M    0% /sys/fs/cgroup
/dev/sda1               xfs      1014M  142M  873M   14% /boot
tmpfs                   tmpfs     184M     0  184M    0% /run/user/0
[root@pgstandby ~]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root@pgstandby ~]# uname -rm
3.10.0-862.el7.x86_64 x86_64
[root@pgstandby ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.149  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::486a:f6d4:7976:4c9a  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:5f:53  txqueuelen 1000  (Ethernet)
        RX packets 22927  bytes 4822730 (4.5 MiB)
        RX errors 0  dropped 2560  overruns 0  frame 0
        TX packets 262  bytes 28988 (28.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgstandby ~]#

二 配置浮动IP

步骤3中的监控并启动浮动IP是可选项配置,不适用于本文的数据库主备切换。数据库主备切换还是推荐手工在主备库上启停浮动IP。

1 主库配置浮动IP:

a 拷贝网络配置文件,并修改:

[root@pgprimary ~]# cp /etc/sysconfig/network-scripts/ifcfg-ens160 /etc/sysconfig/network-scripts/ifcfg-ens160:1
[root@pgprimary ~]#

b 修改DEVICE为ens160:1,添加NM_CONTROLLED设置为no,修改IPADDR为浮动IP地址172.16.11.121,并且去掉网关信息,修改完如下:

[root@pgprimary ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens160:1
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens160
UUID=be70afb7-a578-4dc0-b694-af530586a76e
DEVICE=ens160:1
ONBOOT=yes
IPADDR=172.16.11.121
NETMASK=255.255.255.0
NM_CONTROLLED=no
[root@pgprimary ~]#

c 启用ens160:1网络设备,并测试:

[root@pgprimary ~]# ifup ens160:1
[root@pgprimary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:80:64:27 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.148/24 brd 172.16.11.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 172.16.11.121/24 brd 172.16.11.255 scope global secondary ens160:1
       valid_lft forever preferred_lft forever
    inet6 fe80::58b2:50cd:9d7c:8b23/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@pgprimary ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.148  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::58b2:50cd:9d7c:8b23  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)
        RX packets 172902  bytes 59418679 (56.6 MiB)
        RX errors 0  dropped 15626  overruns 0  frame 0
        TX packets 15516  bytes 1443358 (1.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.121  netmask 255.255.255.0  broadcast 172.16.11.255
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgprimary ~]#

d 客户端去ping floating IP

$ ping 172.16.11.121
PING 172.16.11.121 (172.16.11.121): 56 data bytes
64 bytes from 172.16.11.121: icmp_seq=0 ttl=63 time=0.653 ms
64 bytes from 172.16.11.121: icmp_seq=1 ttl=63 time=0.766 ms
^C
--- 172.16.11.121 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.653/0.710/0.766/0.057 ms

asher at MacBook-Air-3 in ~
$

2 备库配置浮动IP

具体操作步骤,同上述主库配置浮动IP a,b,c步骤。

测试的时候,需要先在主库停止网络设备ens160:1,然后再在备库启动网络设备ens160:1。

主库停ens160:1

[root@pgprimary ~]# ifdown ens160:1
[root@pgprimary ~]#

备库启动ens160:1

[root@pgstandby ~]# ifup ens160:1
[root@pgstandby ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.149  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::486a:f6d4:7976:4c9a  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:5f:53  txqueuelen 1000  (Ethernet)
        RX packets 182284  bytes 61612908 (58.7 MiB)
        RX errors 0  dropped 16880  overruns 0  frame 0
        TX packets 15365  bytes 1381833 (1.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.121  netmask 255.255.255.0  broadcast 172.16.11.255
        ether 00:50:56:80:5f:53  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgstandby ~]#

至此,完成了对主备库的浮动IP配置。

3 监控并启停浮动IP【可选项】

a 主库监控并启动浮动IP

监控脚本为:

[root@pgprimary ~]# cat monitor_floating_ip.sh 
#!/bin/bash
#This script is used for monitoring the floating IP(172.16.11.121),if it is unreachable,
#then ifup the ens160:1 device on this postgres PRIMARY server.
#Created by 黄伟, whuang@onlyou.com,20181010.

FLOATING_IP=172.16.11.121

c=$(/usr/bin/ping -$FLOATING_IP -c 1|grep Unreachable|wc -l)
if [ $c -gt 0 ]
then 
   /usr/sbin/ifup ens160:1
fi
[root@pgprimary ~]#

crontab定时任务为:

[root@pgprimary ~]# crontab -l
* * * * * /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 10; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 20; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 30; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 40; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 50; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
[root@pgprimary ~]#

说明,脚本用于ping浮动IP,如果ping不通,则在主库上启动浮动IP。定时任务表示每隔10秒调用一次监控浮动IP脚本。

b 备库监控并启动浮动IP

监控脚本为:

[root@pgstandby ~]# cat monitor_floating_ip.sh 
#!/bin/bash
#This script is used for monitoring the floating IP(172.16.11.121),if it is unreachable,and also the PRIMARY
#IP(172.16.11.148) unreachable,then ifup the ens160:1 device on this postgres STANDBY server.
#Created by 黄伟, whuang@onlyou.com,20181010.

PRIMARY_IP=172.16.11.148
FLOATING_IP=172.16.11.121
c1=$(/usr/bin/ping $PRIMARY_IP -c 1|grep Unreachable|wc -l)
c2=$(/usr/bin/ping $FLOATING_IP -c 1|grep Unreachable|wc -l)
c3=$(/usr/sbin/ip addr|grep ens160:1)
if [ $c1 -gt 0 -a $c2 -gt 0 ]
then
   /usr/sbin/ifup ens160:1
elif [ $c1 -eq 0 -a $c2 -eq 0 -a "$c3" ]
then
   /usr/sbin/ifdown ens160:1
fi
[root@pgstandby ~]#

crontab定时任务为:

[root@pgstandby ~]# crontab -l
* * * * * /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 10; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 20; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 30; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 40; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
* * * * * sleep 50; /root/monitor_floating_ip.sh>>/root/monitor_floating_ip.log
[root@pgstandby ~]#

说明,脚本用于ping浮动IP,如果ping不通,且同时ping不通主库实际IP地址,那么意味着主库服务器已经宕机或者整个网络异常,则在备库上启动浮动IP。如果ping主、备库IP正常,且备库上浮动IP正常,则意味着主库服务器恢复正常或网络恢复正常,则在备库上停止浮动IP,把浮动IP交给主库托管。定时任务表示每隔10秒调用一次监控浮动IP脚本。

三 主库安装、配置数据库

1 创建用户、路径

[root@pgprimary ~]# groupadd postgres
[root@pgprimary ~]# useradd -g postgres postgres
[root@pgprimary ~]# passwd postgres
更改用户 postgres 的密码 。
新的 密码:
无效的密码: 密码包含用户名在某些地方
重新输入新的 密码:
passwd:所有的身份验证令牌已经成功更新。
[root@pgprimary ~]# mkdir -p /postgres/9.4.10
[root@pgprimary ~]# chown -R postgres:postgres /postgres/
[root@pgprimary ~]#

2 操作系统安装软件包gcc,zlib,readline,tar

[root@pgprimary ~]# yum install gcc*
具体略
[root@pgprimary ~]# yum install -y zlib*
具体略
[root@pgprimary ~]# yum install -y readline*
具体略
[root@pgprimary ~]# yum install -y tar*
具体略

3 下载软件,postgres用户执行:

[postgres@pgprimary ~]$ wget https://ftp.postgresql.org/pub/source/v9.4.10/postgresql-9.4.10.tar.gz
--2018-09-30 17:19:51--  https://ftp.postgresql.org/pub/source/v9.4.10/postgresql-9.4.10.tar.gz
正在解析主机 ftp.postgresql.org (ftp.postgresql.org)... 204.145.124.244, 217.196.149.55, 174.143.35.246, ...
正在连接 ftp.postgresql.org (ftp.postgresql.org)|204.145.124.244|:443... 已连接。
已发出 HTTP 请求,正在等待回应... 200 OK
长度:23333040 (22M) [application/x-gzip]
正在保存至: “postgresql-9.4.10.tar.gz”

100%[====================================================================================================================================================================================>] 23,333,040  1.32MB/s 用时 20s    

2018-09-30 17:20:14 (1.10 MB/s) - 已保存 “postgresql-9.4.10.tar.gz” [23333040/23333040])

[postgres@pgprimary ~]$ ll
总用量 22788
-rw-rw-r--. 1 postgres postgres 23333040 10月 25 2016 postgresql-9.4.10.tar.gz
[postgres@pgprimary ~]$

4 解压软件,configure,postgres用户执行:

[postgres@pgprimary ~]$ tar -zxvf postgresql-9.4.10.tar.gz 
...
postgresql-9.4.10/configure.in
postgresql-9.4.10/INSTALL
[postgres@pgprimary ~]$ cd postgresql-9.4.10 && ll
总用量 620
-rw-r--r--.  1 postgres postgres    385 10月 25 2016 aclocal.m4
drwxrwxr-x.  2 postgres postgres   4096 10月 25 2016 config
-rwxr-xr-x.  1 postgres postgres 448579 10月 25 2016 configure
-rw-r--r--.  1 postgres postgres  69119 10月 25 2016 configure.in
drwxrwxr-x. 59 postgres postgres   4096 10月 25 2016 contrib
-rw-r--r--.  1 postgres postgres   1192 10月 25 2016 COPYRIGHT
drwxrwxr-x.  3 postgres postgres    107 10月 25 2016 doc
-rw-r--r--.  1 postgres postgres   3620 10月 25 2016 GNUmakefile.in
-rw-r--r--.  1 postgres postgres    283 10月 25 2016 HISTORY
-rw-r--r--.  1 postgres postgres  75420 10月 25 2016 INSTALL
-rw-r--r--.  1 postgres postgres   1489 10月 25 2016 Makefile
-rw-r--r--.  1 postgres postgres   1209 10月 25 2016 README
drwxrwxr-x. 15 postgres postgres   4096 10月 25 2016 src
[postgres@pgprimary postgresql-9.4.10]$ ./configure --prefix=/postgres/9.4.10/
...
config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
[postgres@pgprimary postgresql-9.4.10]$

5 gmake world,postgres用户执行:

[postgres@pgprimary postgresql-9.4.10]$ gmake world
..
gmake[1]: 离开目录“/home/postgres/postgresql-9.4.10/contrib”
PostgreSQL, contrib, and documentation successfully made. Ready to install.
[postgres@pgprimary postgresql-9.4.10]$

6 gmake install-world,postgres用户执行:

[postgres@pgprimary postgresql-9.4.10]$ gmake install-world
...
PostgreSQL, contrib, and documentation installation complete.
[postgres@pgprimary postgresql-9.4.10]$

7 初始化数据库,postgres用户执行:

[postgres@pgprimary ~]$ /postgres/9.4.10/bin/initdb -D /postgres/9.4.10/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "zh_CN.UTF-8".
The default database encoding has accordingly been set to "UTF8".
initdb: could not find suitable text search configuration for locale "zh_CN.UTF-8"
The default text search configuration will be set to "simple".

Data page checksums are disabled.

creating directory /postgres/9.4.10/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /postgres/9.4.10/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /postgres/9.4.10/bin/postgres -D /postgres/9.4.10/data
or
    /postgres/9.4.10/bin/pg_ctl -D /postgres/9.4.10/data -l logfile start

[postgres@pgprimary ~]$

8 修改postgres用户环境变量

[postgres@pgprimary ~]$ cat ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin:/postgres/9.4.10/bin
PGDATA=/postgres/9.4.10/data
export PATH
export PGDATA
[postgres@pgprimary ~]$

postgres用户的/home/.bash_profile修改之后的信息如下:

9 修改数据库参数

修改/postgres/9.4.10/data/postgresql.conf文件,使其:

listen_addresses = '*'
port = 5432

其它参数暂时不动。
修改/postgres/9.4.10/data/pg_hba.conf,在其中添加下述;

host    all             all             0.0.0.0/0               md5

其它保持不变。

10 启动数据库

[postgres@pgprimary ~]$ pg_ctl start
server starting
[postgres@pgprimary ~]$ LOG:  database system was shut down at 2018-10-10 09:32:14 CST
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

[postgres@pgprimary ~]$

11 测试数据库

a 备库停用浮动IP

[root@pgstandby ~]# ifdown ens160:1
[root@pgstandby ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.149  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::486a:f6d4:7976:4c9a  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:5f:53  txqueuelen 1000  (Ethernet)
        RX packets 14738337  bytes 3468772296 (3.2 GiB)
        RX errors 0  dropped 1833331  overruns 0  frame 0
        TX packets 149019  bytes 13182295 (12.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1  bytes 104 (104.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 104 (104.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgstandby ~]#

b 主库启用浮动IP

[root@pgprimary ~]# ifup  ens160:1
[root@pgprimary ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.148  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::58b2:50cd:9d7c:8b23  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)
        RX packets 14551224  bytes 3431186752 (3.1 GiB)
        RX errors 0  dropped 1817255  overruns 0  frame 0
        TX packets 108201  bytes 9678459 (9.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.121  netmask 255.255.255.0  broadcast 172.16.11.255
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 12  bytes 1089 (1.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12  bytes 1089 (1.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgprimary ~]#

c 连接主库原来IP

MacBook-Air-3:~ postgres$ psql -h 172.16.11.148 -p 5432 -d postgres -U postgres
Password for user postgres: 
psql (9.5.2, server 9.4.10)
Type "help" for help.

postgres=# \c
psql (9.5.2, server 9.4.10)
You are now connected to database "postgres" as user "postgres".
postgres=#

d 连接主库浮动IP

MacBook-Air-3:~ postgres$ psql -h 172.16.11.121 -p 5432 -d postgres -U postgres
Password for user postgres: 
psql (9.5.2, server 9.4.10)
Type "help" for help.

postgres=# \c
psql (9.5.2, server 9.4.10)
You are now connected to database "postgres" as user "postgres".
postgres=#

从上,可以看到,当主库启用浮动IP之后,客户端既可以通过原IP访问主库,也可以通过浮动IP访问主库。

四 备库安装数据库软件

具体步骤参照步骤三的1-6和步骤8。其它不需要做更多的配置,需要注意的是,该备库只需要安装数据库软件即可,并不需要初始化数据库。

五 配置主备库同步

1 主库添加同步用户

[root@pgprimary ~]# su - postgres
上一次登录:三 10月 10 10:08:01 CST 2018pts/0 上
[postgres@pgprimary ~]$ psql
psql (9.4.10)
Type "help" for help.

postgres=# CREATE ROLE replica login replication encrypted password 'replica';
CREATE ROLE
postgres=# \q
[postgres@pgprimary ~]$

2 主库修改配置文件pg_hba.conf和postgresql.conf

修改/postgres/9.4.10/data/pg_hba.conf,添加下述行:

host    replication     replica     172.16.11.149/32           md5

修改/postgres/9.4.10/data/postgresql.conf,修改下述行:

wal_level = hot_standby  # 这个是设置主为wal的主机
max_wal_senders = 4 # 这个设置了可以最多有几个流复制连接,差不多有几个从,就设置几个
wal_keep_segments = 256 # 设置流复制保留的最多的xlog数目
wal_sender_timeout = 60s # 设置流复制主机发送数据的超时时间
max_connections = 100 # 这个设置要注意下,主库的该参数要根据实际情况设置。从库的max_connections必须要大于主库的该参数

3 主库重启

[postgres@pgprimary ~]$ pg_ctl restart -m fast
waiting for server to shut down...LOG:  received fast shutdown request
.LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
FATAL:  terminating connection due to administrator command
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped
server starting
[postgres@pgprimary ~]$ LOG:  database system was shut down at 2018-10-10 16:35:19 CST
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

[postgres@pgprimary ~]$

4 备库,执行主库的基础备份

[root@pgstandby ~]# su - postgres
上一次登录:三 10月 10 09:43:15 CST 2018pts/0 上
[postgres@pgstandby ~]$ ll /postgres/9.4.10/
总用量 16
drwxrwxr-x. 2 postgres postgres 4096 9月  30 17:30 bin
drwxrwxr-x. 6 postgres postgres 4096 9月  30 17:30 include
drwxrwxr-x. 4 postgres postgres 4096 9月  30 17:30 lib
drwxrwxr-x. 8 postgres postgres 4096 9月  30 17:30 share
[postgres@pgstandby ~]$ which pg_basebackup 
/postgres/9.4.10/bin/pg_basebackup
[postgres@pgstandby ~]$ pg_basebackup -F p --progress -D /postgres/9.4.10/data -h 172.16.11.121 -p 5432 -U replica --password -X stream
Password: 
21122/21122 kB (100%), 1/1 tablespace
[postgres@pgstandby ~]$ ll /postgres/9.4.10/
总用量 20
drwxrwxr-x.  2 postgres postgres 4096 9月  30 17:30 bin
drwx------. 18 postgres postgres 4096 10月 10 16:38 data
drwxrwxr-x.  6 postgres postgres 4096 9月  30 17:30 include
drwxrwxr-x.  4 postgres postgres 4096 9月  30 17:30 lib
drwxrwxr-x.  8 postgres postgres 4096 9月  30 17:30 share
[postgres@pgstandby ~]$ du -sh /postgres/9.4.10/data/
37M     /postgres/9.4.10/data/
[postgres@pgstandby ~]$

可以看到,备库已经生成/postgres/9.4.10/data/。

5 备库,配置recovery.conf文件

从安装文件里cp过来。然后编辑,更改为下面3个参数,其它保持不变。

[postgres@pgstandby ~]$ cp /home/postgres/postgresql-9.4.10/src/backend/access/transam/recovery.conf.sample /postgres/9.4.10/data/recovery.conf
[postgres@pgstandby ~]$ vi /postgres/9.4.10/data/recovery.conf 
...
standby_mode = on
primary_conninfo = 'host=172.16.11.148 port=5432 user=replica password=replica'         # e.g. 'host=localhost port=5432'
recovery_target_timeline = 'latest'
...

6 备库,修改配置文件postgresql.conf

修改下列4个参数即可:

[postgres@pgstandby ~]$ vi /postgres/9.4.10/data/postgresql.conf 
...
hot_standby = on  # 说明这台机器不仅仅是用于数据归档,也用于数据查询
max_standby_streaming_delay = 30s # 数据流备份的最大延迟时间
wal_receiver_status_interval = 1s  # 多久向主报告一次从的状态,当然从每次数据复制都会向主报告状态,这里只是设置最长的间隔时间
hot_standby_feedback = on # 如果有错误的数据复制,是否向主进行反馈
...

7 备库,启动

[postgres@pgstandby ~]$ pg_ctl status
pg_ctl: no server running
[postgres@pgstandby ~]$ pg_ctl start
server starting
[postgres@pgstandby ~]$ LOG:  database system was interrupted; last known up at 2018-10-10 16:38:47 CST
LOG:  entering standby mode
LOG:  redo starts at 0/2000028
LOG:  consistent recovery state reached at 0/2000128
LOG:  database system is ready to accept read only connections
LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

[postgres@pgstandby ~]$ pg_ctl status
pg_ctl: server is running (PID: 28027)
/postgres/9.4.10/bin/postgres
[postgres@pgstandby ~]$ ps -ef|grep postgres
root     27400 25585  0 16:36 pts/0    00:00:00 su - postgres
postgres 27401 27400  0 16:36 pts/0    00:00:00 -bash
postgres 28027     1  0 16:48 pts/0    00:00:00 /postgres/9.4.10/bin/postgres
postgres 28028 28027  0 16:48 ?        00:00:00 postgres: startup process   recovering 000000010000000000000003
postgres 28029 28027  0 16:48 ?        00:00:00 postgres: checkpointer process  
postgres 28030 28027  0 16:48 ?        00:00:00 postgres: writer process   
postgres 28031 28027  0 16:48 ?        00:00:00 postgres: stats collector process  
postgres 28032 28027  0 16:48 ?        00:00:00 postgres: wal receiver process   streaming 0/3000210
postgres 28073 27401  0 16:49 pts/0    00:00:00 ps -ef
postgres 28074 27401  0 16:49 pts/0    00:00:00 grep --color=auto postgres
[postgres@pgstandby ~]$

从上,可以看到,备库的receiver进程,在接收主库的日志信息。

8 验证主备库同步

主库执行:

[postgres@pgprimary ~]$ ps -ef|grep postgres
root     20280 17302  0 16:30 pts/0    00:00:00 su - postgres
postgres 20281 20280  0 16:30 pts/0    00:00:00 -bash
postgres 20677     1  0 16:35 pts/0    00:00:00 /postgres/9.4.10/bin/postgres
postgres 20679 20677  0 16:35 ?        00:00:00 postgres: checkpointer process  
postgres 20680 20677  0 16:35 ?        00:00:00 postgres: writer process   
postgres 20681 20677  0 16:35 ?        00:00:00 postgres: wal writer process  
postgres 20682 20677  0 16:35 ?        00:00:00 postgres: autovacuum launcher process  
postgres 20683 20677  0 16:35 ?        00:00:00 postgres: stats collector process  
postgres 21656 20677  0 16:48 ?        00:00:00 postgres: wal sender process replica 172.16.11.149(51868) streaming 0/3000210
postgres 21939 20281  0 16:52 pts/0    00:00:00 ps -ef
postgres 21940 20281  0 16:52 pts/0    00:00:00 grep --color=auto postgres
[postgres@pgprimary ~]$ psql
psql (9.4.10)
Type "help" for help.

postgres=# \x
Expanded display is on.
postgres=# select * from pg_stat_replication;
-[ RECORD 1 ]----+------------------------------
pid              | 21656
usesysid         | 16385
usename          | replica
application_name | walreceiver
client_addr      | 172.16.11.149
client_hostname  | 
client_port      | 51868
backend_start    | 2018-10-10 16:48:44.010195+08
backend_xmin     | 1893
state            | streaming
sent_location    | 0/3000210
write_location   | 0/3000210
flush_location   | 0/3000210
replay_location  | 0/3000210
sync_priority    | 0
sync_state       | async

postgres=#

从上,看到主库有sender进程,发送同步信息给备库。同时,登录主库也可以查看到同步信息。

 

六 主备库切换

这里,模拟主库出现故障,先做一次switchover测试,即先切换主备库角色,主备互换。

然后,重建原主库或者直接将原主库当作新主库的备库。

最后,再执行一次switchover,主备互换,即恢复到之前的最初状态,主库还是原主库,备库还是原备库。

1 执行第一次主备切换

当原主库库服务器或者数据库故障时,我们可以先停止主库,然后启动备库。

a 停止主库

[postgres@pgprimary data]$ pg_ctl stop -m fast -l /home/postgres/pg.log 
waiting for server to shut down.... done
server stopped
[postgres@pgprimary data]$

b 启动备库

 [postgres@pgstandby data]$ pwd
/postgres/9.4.10/data
[postgres@pgstandby data]$ ll 
总用量 72
-rw-------. 1 postgres postgres   206 10月 10 16:38 backup_label.old
drwx------. 5 postgres postgres    41 10月 10 16:38 base
drwx------. 2 postgres postgres  4096 10月 26 15:57 global
drwx------. 2 postgres postgres    18 10月 10 16:38 pg_clog
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_dynshmem
-rw-------. 1 postgres postgres  4605 10月 16 16:35 pg_hba.conf
-rw-------. 1 postgres postgres  1636 10月 10 16:38 pg_ident.conf
drwx------. 4 postgres postgres    39 10月 10 16:38 pg_logical
drwx------. 4 postgres postgres    36 10月 10 16:38 pg_multixact
drwx------. 2 postgres postgres    18 10月 26 15:57 pg_notify
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_replslot
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_serial
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_snapshots
drwx------. 2 postgres postgres     6 10月 26 15:57 pg_stat
drwx------. 2 postgres postgres     6 10月 26 09:29 pg_stat_tmp
drwx------. 2 postgres postgres    18 10月 10 16:38 pg_subtrans
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_tblspc
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_twophase
-rw-------. 1 postgres postgres     4 10月 10 16:38 PG_VERSION
drwx------. 3 postgres postgres  4096 10月 26 15:57 pg_xlog
-rw-------. 1 postgres postgres    88 10月 10 16:38 postgresql.auto.conf
-rw-------. 1 postgres postgres 20810 10月 10 16:45 postgresql.conf
-rw-------. 1 postgres postgres    30 10月 26 15:57 postmaster.opts
-rw-------. 1 postgres postgres    71 10月 26 15:57 postmaster.pid
-rw-r--r--. 1 postgres postgres  5641 10月 16 16:54 recovery.conf
[postgres@pgstandby data]$ pg_ctl promote -l /home/postgres/pg.log 
server promoting
[postgres@pgstandby data]$ ll
总用量 72
-rw-------. 1 postgres postgres   206 10月 10 16:38 backup_label.old
drwx------. 5 postgres postgres    41 10月 10 16:38 base
drwx------. 2 postgres postgres  4096 10月 26 15:57 global
drwx------. 2 postgres postgres    18 10月 10 16:38 pg_clog
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_dynshmem
-rw-------. 1 postgres postgres  4605 10月 16 16:35 pg_hba.conf
-rw-------. 1 postgres postgres  1636 10月 10 16:38 pg_ident.conf
drwx------. 4 postgres postgres    39 10月 10 16:38 pg_logical
drwx------. 4 postgres postgres    36 10月 10 16:38 pg_multixact
drwx------. 2 postgres postgres    18 10月 26 15:57 pg_notify
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_replslot
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_serial
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_snapshots
drwx------. 2 postgres postgres     6 10月 26 15:57 pg_stat
drwx------. 2 postgres postgres    42 10月 26 16:10 pg_stat_tmp
drwx------. 2 postgres postgres    18 10月 10 16:38 pg_subtrans
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_tblspc
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_twophase
-rw-------. 1 postgres postgres     4 10月 10 16:38 PG_VERSION
drwx------. 3 postgres postgres  4096 10月 26 16:10 pg_xlog
-rw-------. 1 postgres postgres    88 10月 10 16:38 postgresql.auto.conf
-rw-------. 1 postgres postgres 20810 10月 10 16:45 postgresql.conf
-rw-------. 1 postgres postgres    30 10月 26 15:57 postmaster.opts
-rw-------. 1 postgres postgres    71 10月 26 15:57 postmaster.pid
-rw-r--r--. 1 postgres postgres  5641 10月 16 16:54 recovery.done
[postgres@pgstandby data]$

这里,需要注意的是,启动备库的命令是pg_ctl promote,之前备库是出在接受主库同步的状态,需要通过这个命令来将备库提升到primary状态。同时,备库切换为主库之后,/postgres/9.4.10/data/recovery.conf文件自动变为/postgres/9.4.10/data/recovery.done。我个人猜想,主要是为了下次重启该数据库时,直接进入“主库模式”,如果还是recovery.conf的话,意味着该库需要执行recovery操作。

c 备库启动浮动IP

先在主库上停止浮动IP

 [root@pgprimary ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.148  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::58b2:50cd:9d7c:8b23  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)
        RX packets 37310308  bytes 8976173370 (8.3 GiB)
        RX errors 0  dropped 4868601  overruns 0  frame 0
        TX packets 1426780  bytes 127364686 (121.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.121  netmask 255.255.255.0  broadcast 172.16.11.255
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 134634  bytes 25999628 (24.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 134634  bytes 25999628 (24.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

您在 /var/spool/mail/root 中有邮件
[root@pgprimary ~]# ifdown ens160:1
[root@pgprimary ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.148  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::58b2:50cd:9d7c:8b23  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:64:27  txqueuelen 1000  (Ethernet)
        RX packets 37310444  bytes 8976218410 (8.3 GiB)
        RX errors 0  dropped 4868628  overruns 0  frame 0
        TX packets 1426794  bytes 127368186 (121.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 134634  bytes 25999628 (24.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 134634  bytes 25999628 (24.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgprimary ~]#

然后,备库启动浮动IP

 [root@pgstandby ~]# ifup ens160:1
您在 /var/spool/mail/root 中有邮件
[root@pgstandby ~]# ifconfig 
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.149  netmask 255.255.255.0  broadcast 172.16.11.255
        inet6 fe80::486a:f6d4:7976:4c9a  prefixlen 64  scopeid 0x20
        ether 00:50:56:80:5f:53  txqueuelen 1000  (Ethernet)
        RX packets 37445369  bytes 8975382297 (8.3 GiB)
        RX errors 0  dropped 4884823  overruns 0  frame 0
        TX packets 1509890  bytes 162194380 (154.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.11.121  netmask 255.255.255.0  broadcast 172.16.11.255
        ether 00:50:56:80:5f:53  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 195774  bytes 35184330 (33.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 195774  bytes 35184330 (33.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@pgstandby ~]#

d 测试新主库是否可用

MacBook-Air-3:~ postgres$ psql -h 172.16.11.121 -p 5432 -d postgres -U postgres
Password for user postgres: 
psql (9.5.2, server 9.4.10)
Type "help" for help.

postgres=# select * from t;
 id 
----
  1
(1 row)

postgres=# delete from t;
DELETE 1
postgres=# select * from t;
 id 
----
(0 rows)

postgres=# 

客户端通过浮动IP访问数据库,并删除t表的数据。说明,新主库可用正常使用。同时,记录下,这里是从新主库里删除的数据,接下来,我们重建新备库之后,看看这条被删除的数据,是否也从备库里查询不到?

2 重建主库

这里,为了快速验证和测试主备互换的流程,只是假设主库出现故障,所以,在这里并不需要重建原主库。真实情况下,如果主库因出现故障而执行切换的话,就需要重建。重建的流程和步骤可以参考初始搭建备库的流程来完成。

当然,这里需要将原主库配置为新备库,同样需要参照搭建备库流程五 配置主备库同步中的5,6,7三个步骤。

a 新备库创建配置recovery.conf文件

[postgres@pgprimary ~]$ cp /home/postgres/postgresql-9.4.10/src/backend/access/transam/recovery.conf.sample /postgres/9.4.10/data/recovery.conf
[postgres@pgprimary ~]$ vi /postgres/9.4.10/data/recovery.conf 
...
standby_mode = on
primary_conninfo = 'host=172.16.11.149 port=5432 user=replica password=replica'         # e.g. 'host=localhost port=5432'
recovery_target_timeline = 'latest'
...

需要注意的是,这里主库的信息,需要指向新主库的IP,即原备库的IP 172.16.11.149。

b 新备库修改配置文件postgresql.conf

修改下列4个参数即可:

[postgres@pgprimary ~]$ vi /postgres/9.4.10/data/postgresql.conf 
...
hot_standby = on  # 说明这台机器不仅仅是用于数据归档,也用于数据查询
max_standby_streaming_delay = 30s # 数据流备份的最大延迟时间
wal_receiver_status_interval = 1s  # 多久向主报告一次从的状态,当然从每次数据复制都会向主报告状态,这里只是设置最长的间隔时间
hot_standby_feedback = on # 如果有错误的数据复制,是否向主进行反馈
...

c 启动新备库

[postgres@pgprimary data]$ pg_ctl start -l /home/postgres/pg.log 
server starting
[postgres@pgprimary data]$ ps -ef|grep postgres
root     13481 13458  0 09:22 pts/1    00:00:00 su - postgres
postgres 13482 13481  0 09:22 pts/1    00:00:00 -bash
postgres 14425     1  1 16:55 pts/1    00:00:00 /postgres/9.4.10/bin/postgres
postgres 14426 14425  0 16:55 ?        00:00:00 postgres: startup process   recovering 000000060000000000000005
postgres 14427 14425  0 16:55 ?        00:00:00 postgres: checkpointer process  
postgres 14428 14425  0 16:55 ?        00:00:00 postgres: writer process   
postgres 14429 14425  0 16:55 ?        00:00:00 postgres: stats collector process  
postgres 14430 14425  1 16:55 ?        00:00:00 postgres: wal receiver process   streaming 0/508DDD0
postgres 14431 13482  0 16:55 pts/1    00:00:00 ps -ef
postgres 14432 13482  0 16:55 pts/1    00:00:00 grep --color=auto postgres
[postgres@pgprimary data]$

启动之后,发现新备库上开始有wal receiver process 。

同时,登录新备库,查看t表的数据有没有同步:

[postgres@pgprimary data]$ psql
psql (9.4.10)
Type "help" for help.

postgres=# select * from t;
 id 
----
(0 rows)

postgres=#

显然,之前从新主库里删除的那条数据,在这个新备库里也看不到了。说明,新备库是OK的了。

3 再次执行主备切换

a 停掉新主库(原备库)

 [postgres@pgstandby data]$ ps -ef|grep postgres
postgres 16230     1  0 15:57 pts/1    00:00:00 /postgres/9.4.10/bin/postgres
postgres 16232 16230  0 15:57 ?        00:00:00 postgres: checkpointer process  
postgres 16233 16230  0 15:57 ?        00:00:00 postgres: writer process   
postgres 16234 16230  0 15:57 ?        00:00:00 postgres: stats collector process  
postgres 16998 16230  0 16:10 ?        00:00:00 postgres: wal writer process  
postgres 16999 16230  0 16:10 ?        00:00:00 postgres: autovacuum launcher process  
postgres 17546 16230  0 16:20 ?        00:00:00 postgres: postgres postgres 172.16.135.72(54471) idle
postgres 19262 16230  0 16:55 ?        00:00:00 postgres: wal sender process replica 172.16.11.148(53298) streaming 0/508DEA8
postgres 19520 29607  0 17:00 pts/1    00:00:00 ps -ef
postgres 19521 29607  0 17:00 pts/1    00:00:00 grep --color=auto postgres
root     29606 29605  0 09:23 pts/1    00:00:00 su - postgres
postgres 29607 29606  0 09:23 pts/1    00:00:00 -bash
[postgres@pgstandby data]$ pg_ctl stop -m fast -l /home/postgres/pg.log 
waiting for server to shut down.... done
server stopped
[postgres@pgstandby data]$ ll
总用量 68
-rw-------. 1 postgres postgres   206 10月 10 16:38 backup_label.old
drwx------. 5 postgres postgres    41 10月 10 16:38 base
drwx------. 2 postgres postgres  4096 10月 26 15:57 global
drwx------. 2 postgres postgres    18 10月 10 16:38 pg_clog
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_dynshmem
-rw-------. 1 postgres postgres  4605 10月 16 16:35 pg_hba.conf
-rw-------. 1 postgres postgres  1636 10月 10 16:38 pg_ident.conf
drwx------. 4 postgres postgres    39 10月 10 16:38 pg_logical
drwx------. 4 postgres postgres    36 10月 10 16:38 pg_multixact
drwx------. 2 postgres postgres    18 10月 26 15:57 pg_notify
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_replslot
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_serial
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_snapshots
drwx------. 2 postgres postgres    63 10月 26 17:00 pg_stat
drwx------. 2 postgres postgres     6 10月 26 17:00 pg_stat_tmp
drwx------. 2 postgres postgres    18 10月 10 16:38 pg_subtrans
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_tblspc
drwx------. 2 postgres postgres     6 10月 10 16:38 pg_twophase
-rw-------. 1 postgres postgres     4 10月 10 16:38 PG_VERSION
drwx------. 3 postgres postgres  4096 10月 26 16:10 pg_xlog
-rw-------. 1 postgres postgres    88 10月 10 16:38 postgresql.auto.conf
-rw-------. 1 postgres postgres 20810 10月 10 16:45 postgresql.conf
-rw-------. 1 postgres postgres    30 10月 26 15:57 postmaster.opts
-rw-r--r--. 1 postgres postgres  5641 10月 16 16:54 recovery.done
[postgres@pgstandby data]$

需要注意的是,这里的新主库停止之后,系统同样自动的将/postgres/9.4.10/data/recovery.conf文件变为/postgres/9.4.10/data/recovery.done。在我们重新启动这个备库的时候,需要把recovery.done改名为recovery.conf,使其在重新启动时,自己担任的是“备库角色”,自动同主库进行同步。

b 启动新备库(原主库)

[postgres@pgprimary data]$ ps -ef|grep postgres
root     13481 13458  0 09:22 pts/1    00:00:00 su - postgres
postgres 13482 13481  0 09:22 pts/1    00:00:00 -bash
postgres 14425     1  0 16:55 pts/1    00:00:00 /postgres/9.4.10/bin/postgres
postgres 14426 14425  0 16:55 ?        00:00:00 postgres: startup process   recovering 000000060000000000000005
postgres 14427 14425  0 16:55 ?        00:00:00 postgres: checkpointer process  
postgres 14428 14425  0 16:55 ?        00:00:00 postgres: writer process   
postgres 14429 14425  0 16:55 ?        00:00:00 postgres: stats collector process  
postgres 14894 13482  0 17:01 pts/1    00:00:00 ps -ef
postgres 14895 13482  0 17:01 pts/1    00:00:00 grep --color=auto postgres
[postgres@pgprimary data]$ pg_ctl promote -l /home/postgres/
server promoting
[postgres@pgprimary data]$ ps -ef|grep postgres
root     13481 13458  0 09:22 pts/1    00:00:00 su - postgres
postgres 13482 13481  0 09:22 pts/1    00:00:00 -bash
postgres 14425     1  0 16:55 pts/1    00:00:00 /postgres/9.4.10/bin/postgres
postgres 14427 14425  0 16:55 ?        00:00:00 postgres: checkpointer process  
postgres 14428 14425  0 16:55 ?        00:00:00 postgres: writer process   
postgres 14429 14425  0 16:55 ?        00:00:00 postgres: stats collector process  
postgres 14916 14425  0 17:01 ?        00:00:00 postgres: wal writer process  
postgres 14917 14425  0 17:01 ?        00:00:00 postgres: autovacuum launcher process  
postgres 14918 13482  0 17:01 pts/1    00:00:00 ps -ef
postgres 14919 13482  0 17:01 pts/1    00:00:00 grep --color=auto postgres
[postgres@pgprimary data]$

此时,这个库又恢复到最初的状态,成为主库。

c 启动原备库

在启动之前,修改recovery.done改名为recovery.conf。

 [postgres@pgstandby data]$ pwd
/postgres/9.4.10/data
[postgres@pgstandby data]$ mv recovery.done recovery.conf
[postgres@pgstandby data]$

启动备库:

 [postgres@pgstandby data]$ pg_ctl start -l /home/postgres/pg.log 
server starting
[postgres@pgstandby data]$ ps -ef|grep postgres
postgres 19871     1  0 17:07 pts/1    00:00:00 /postgres/9.4.10/bin/postgres
postgres 19872 19871  0 17:07 ?        00:00:00 postgres: startup process   recovering 000000070000000000000005
postgres 19873 19871  0 17:07 ?        00:00:00 postgres: checkpointer process  
postgres 19874 19871  0 17:07 ?        00:00:00 postgres: writer process   
postgres 19875 19871  0 17:07 ?        00:00:00 postgres: stats collector process  
postgres 19876 19871  0 17:07 ?        00:00:00 postgres: wal receiver process   streaming 0/508E108
postgres 19881 29607  0 17:07 pts/1    00:00:00 ps -ef
postgres 19882 29607  0 17:07 pts/1    00:00:00 grep --color=auto postgres
root     29606 29605  0 09:23 pts/1    00:00:00 su - postgres
postgres 29607 29606  0 09:23 pts/1    00:00:00 -bash
[postgres@pgstandby data]$

启动之后,看到有wal receiver process。说明,此时备库依然是备库的角色了,并开始同主库进行同步了。

至此,完成了对于主备库的相互切换,并且验证了数据同步,以及注意事项。

七 小结

本文档对于在CentOS7+PostgreSQL9.4.10的数据库环境,利用浮动IP来构建一套数据库的主备同步环境,给出详细搭建和配置的流程。通过自动监控启停浮动IP来实现网络的畅通。正常情况下,主库提供读写操作,备库提供只读操作。客户端和应用程序应该通过浮动IP来访问数据库,而不应该使用真实IP访问数据库。

 

读完《异类》这本书

国庆假期就要过去七分之四了,出门走了一圈,回来继续读九月份未读完的书单,《异类》。花了小半个下午的时间,很愉快轻松的就读完了。

首先,简单说一下整体感受:这是一本值得读的书,这不是一本心灵鸡汤成功学励志之类的读物。因为,有很多观点和现象远远超出我之前的认知水平,深受启发。

其次,作者通过全书9章249页的内容,主要来描述说明和揭示那些各行各业的成功人士、商业巨子、摇滚明星以及纽约知名律师成功的秘诀是什么,以及为什么最棒的飞行员也会坠机,亚洲人在数学方面的天赋到底是为什么的问题。

  • 从加拿大顶级冰球队夺冠获胜来分析其原因,得出顶级冰球队队员的出生日期主要集中在一二三月,因为球队联盟按年龄分组的依据依据是1月1日,这使得出生日期越靠前的小球员从开始学习冰球时,就具有比其他队友更有身体和心理上的优势,进而在后续的学习和发展中,产生更强的马太效应,使他们的优势更有优势,淘汰更多的竞争者。同时,这种调查研究得出的结果,同样适用于捷克冰球队,凡是那些依据年龄大小来分组的竞技体育都能找到类似的答案。这种成功的异类并不是单纯得益于他们的科学训练,天分和热情,更是因为他们因年龄优势而不断积累放大的优势。
  • Sun公司的天才创始人Bill Joy和微软公司创始人Bill Gates的故事,更是令人咋舌,前者入读密歇根大学的时候,正好赶上计算机中心的成立,使得Joy在大学期间有更多时间,没日没夜地泡在计算机中心不断学习计算机科学知识和反复实践练习编程,为其后来进入硅谷创办Sun公司和开发Java编程语言奠定了坚实的基础,这是他的机遇。Gates的机遇则更好,他的父亲是西雅图的知名律师,母亲是富有银行家的千金,家境殷实。中学时期转入西雅图湖滨中学——专为富家子弟开办的私立中学,转学第二年,学校成立电脑俱乐部,Gates开始接触并最终迷上电脑,同时他得到可以远程连接学校电脑终端的权限,花比同龄人更多的时间学习电脑。在校方支付不起昂贵的机房费用的时候,学生的富有家长门成立了基金会来支付费用,供孩子们学习电脑,当基金会入不敷出时,有个学生的家长开的公司愿意来买单,交换的条件是,允许他们的程序运行在湖滨中学的计算机上,用以测试他们公司开发程序的功能。这还不够,Gates和同学们还在某个学期说服学校允许学生花几个月时间去到编程公司实习,这足以使得Gates的电脑水平得到充分的锻炼提高,编程知识更为丰富。这机遇,让他在中学时期甩出同龄人几条街远。电脑天才之所以成功不在于他们天生就是天才,而是他们早早的在电脑上真正的学习和实践已经积累超过10000小时。根本没有“与生俱来的天才”——花比别人少的时间就能达到比别人高的成就。也不存在“劳苦命”,一个人的努力程度比别人高却无法比别人更优秀。出人头地的方法是:刻苦练习。成功的要素在练习阶段变得简单明了。顶级的成功者,不是比其他人更努力,而是比其他人努力十倍,甚至百倍。一个人一个人在学习的过程中,要完美掌握某项复杂技能。就要一遍又一遍艰苦练习,而练习的时长必须达到一个最小临界值,这个神奇的临界值就是10000小时。
  • 音乐天才莫札特和摇滚巨星甲克虫乐队同样如此,莫札特在成名之前,花了无数的时间在音乐上面,甲克虫乐队成名之前曾经到达德国汉堡进行演出,在汉堡,他们每天的演出时间超过八个小时,这使得他们的演出得到了充足的练习和实践。
  • 最棒的飞行员之所以坠机,亚洲人在数学方面的天赋,都来自于他们的族裔文化遗传。作者通过美国国家运输安全委员会对1997年大韩航空飞关岛的空难,及国际上其它的飞机失事调查报告分析得出,这些灾难的发生都是由一系列的小故障机缘巧合的赶在一起发生了,是不断积累的小问题叠加引起的。其中,对失事飞机黑匣子分析大韩航空空难的深层原因与其族裔的文化不无关系,随机工程师,副机长,机长之间的关系都是从属的关系,副机长发现问题时,不是以最直接的方式,用平等地位沟通问题的方式和机长沟通,而是用暗示和请求的语气同机长汇报情况,最终酿成悲剧。而大韩航空在2000年后,终于痛定思痛,下定决心改变其值飞人员之间不平等的等级关系之后,自然其负面形象得到极大扭转。

最后,说到底,所谓“异类”,从来就不是什么异类。

顺便说一句,你也应该找来这本书,读一遍,如果还没读过的话。

PostgreSQL数据库SQL优化案例:从2秒到2毫秒

零 背景说明

这是生产环境下,一则PostgreSQL数据库SQL优化的案例小结:单条SQL执行时间,从优化前2秒到优化后,降低到2毫秒,性能提升1000倍。数据库版本为PostgreSQL 9.3。原始SQL语句如下:

SELECT t.id,
       t.code,
       t.name,
       t.level,
       t.IS_CLOSED isClosed,
       t.PARENT_ID parentId,
       t.dir,
       t.remark
FROM t_report_type t
WHERE 1=1
  AND t.code IN
    (SELECT report_type
     FROM t_report_temp
     WHERE corp_id IS NULL
       AND partner_id IS NULL)
ORDER BY t.code

优化前执行计划如下:

EXPLAIN ANALYZE
SELECT t.id,
       t.code,
       t.name,
       t.level,
       t.IS_CLOSED isClosed,
       t.PARENT_ID parentId,
       t.dir,
       t.remark
FROM t_report_type t
WHERE 1=1
  AND t.code IN
    (SELECT report_type
     FROM t_report_temp
     WHERE corp_id IS NULL
       AND partner_id IS NULL)
ORDER BY t.code

┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                       QUERY PLAN                                                                       │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Sort  (cost=3179.42..3179.93 rows=204 width=165) (actual time=2179.224..2179.250 rows=383 loops=1)                                                     │
│   Sort Key: t.code                                                                                                                                     │
│   Sort Method: quicksort  Memory: 126kB                                                                                                                │
│   ->  Nested Loop Semi Join  (cost=0.55..3171.59 rows=204 width=165) (actual time=26.092..2178.526 rows=383 loops=1)                                   │
│         ->  Seq Scan on t_report_type t  (cost=0.00..16.59 rows=459 width=165) (actual time=0.002..0.187 rows=459 loops=1)                             │
│         ->  Index Only Scan using t_report_temp_01 on t_report_temp  (cost=0.55..1873.80 rows=428 width=7) (actual time=4.744..4.744 rows=1 loops=459) │
│               Index Cond: ((corp_id IS NULL) AND (partner_id IS NULL) AND (report_type = (t.code)::text))                                              │
│               Heap Fetches: 53                                                                                                                         │
│ Total runtime: 2179.301 ms                                                                                                                             │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(9 rows)

一 分析问题

结合执行来分析SQL语句,查看表结构,数据量分布:

zyd=> \d t_report_type
                 Table "zyd.t_report_type"
┌───────────┬────────────────────────┬─────────────────────┐
│  Column   │          Type          │      Modifiers      │
├───────────┼────────────────────────┼─────────────────────┤
│ id        │ character(32)          │ not null            │
│ code      │ character varying(32)  │ not null            │
│ name      │ character varying(256) │ not null            │
│ parent_id │ character varying(32)  │                     │
│ dir       │ character varying(256) │                     │
│ level     │ smallint               │                     │
│ is_closed │ character(1)           │ default '0'::bpchar │
│ remark    │ character varying(200) │                     │
└───────────┴────────────────────────┴─────────────────────┘
Indexes:
    "t_report_type_pkey" PRIMARY KEY, btree (id)
    "t_report_type_01" btree (code, parent_id, dir)

zyd=> select count(*) from t_report_type;
┌───────┐
│ count │
├───────┤
│   459 │
└───────┘
(1 row)

zyd=> select count(*) from t_report_temp;
┌────────┐
│ count  │
├────────┤
│ 366132 │
└────────┘
(1 row)

zyd=> \d t_report_temp
                 Table "zyd.t_report_temp"
┌─────────────┬────────────────────────────────┬───────────┐
│   Column    │              Type              │ Modifiers │
├─────────────┼────────────────────────────────┼───────────┤
│ id          │ character(32)                  │ not null  │
│ report_code │ character varying(40)          │ not null  │
│ report_name │ character varying(100)         │ not null  │
│ report_type │ character varying(32)          │ not null  │
│ corp_id     │ character(32)                  │           │
│ partner_id  │ character(32)                  │           │
│ industry_id │ character(32)                  │           │
│ is_default  │ character(1)                   │           │
│ remark      │ character varying(200)         │           │
│ source_type │ character varying(255)         │           │
│ is_show     │ character(1)                   │ default 0 │
│ create_ts   │ timestamp(6) without time zone │           │
│ update_ts   │ timestamp(6) without time zone │           │
│ is_simple   │ character(1)                   │           │
└─────────────┴────────────────────────────────┴───────────┘
Indexes:
    "t_report_temp_pkey" PRIMARY KEY, btree (id)
    "t_report_temp_01" btree (corp_id, industry_id, partner_id, report_code, report_type, report_name)
    "t_report_temp_02" btree (industry_id, partner_id, report_type)
    "t_report_temp_03" btree (industry_id, report_type)
    "t_report_temp_04" btree (report_name, corp_id, report_code)
    "t_report_temp_index_1" btree (corp_id, report_code)

zyd=> select count(*) from t_report_temp where corp_id is null and partner_id is null;
┌───────┐
│ count │
├───────┤
│  1126 │
└───────┘
(1 row)

zyd=>

看到,这是一个简单的2张表做join连接的SQL查询语句,优化器选择了嵌套循环nested loop的关联方式,其中一张表t_report_type为小表,数据量为459,优化器选择全表扫描的方式获取数据,另外一张表t_report_temp数据量相对多一些,366132条数据,但是满足条件的数据却只有1126条,同时选择了index only scan的方式,单次执行耗时4.744毫秒。

这是从执行计划看到的基本信息,能优化的突破口基本也就是对于表的访问能不能更快?虽然这里选择的是index only scan的方式。

三 解决问题

先尝试重新收集表的统计信息,再查看其执行计划是否有好转?

zyd=> analyze verbose t_report_temp;
INFO:  analyzing "zyd.t_report_temp"
INFO:  "t_report_temp": scanned 9494 of 9494 pages, containing 366132 live rows and 1345 dead rows; 30000 rows in sample, 366132 estimated total rows
ANALYZE
zyd=> analyze verbose t_report_type;
INFO:  analyzing "zyd.t_report_type"
INFO:  "t_report_type": scanned 12 of 12 pages, containing 459 live rows and 0 dead rows; 459 rows in sample, 459 estimated total rows
ANALYZE
zyd=>
...查看执行计划
...
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                       QUERY PLAN                                                                       │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Sort  (cost=3179.42..3179.93 rows=204 width=165) (actual time=1973.808..1973.836 rows=383 loops=1)                                                     │
│   Sort Key: t.code                                                                                                                                     │
│   Sort Method: quicksort  Memory: 126kB                                                                                                                │
│   ->  Nested Loop Semi Join  (cost=0.55..3171.59 rows=204 width=165) (actual time=29.583..1973.127 rows=383 loops=1)                                   │
│         ->  Seq Scan on t_report_type t  (cost=0.00..16.59 rows=459 width=165) (actual time=0.002..0.175 rows=459 loops=1)                             │
│         ->  Index Only Scan using t_report_temp_01 on t_report_temp  (cost=0.55..1873.80 rows=428 width=7) (actual time=4.297..4.297 rows=1 loops=459) │
│               Index Cond: ((corp_id IS NULL) AND (partner_id IS NULL) AND (report_type = (t.code)::text))                                              │
│               Heap Fetches: 53                                                                                                                         │
│ Total runtime: 1973.885 ms                                                                                                                             │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(9 rows)

zyd=>

发现,并没有好转。观察到其执行计划中,对于相对大的那个表t_report_temp,优化器选择了通过t_report_temp_01这个index来做的index only scan,每次耗时4毫秒左右,但是loops=459,累积起来就1800多毫秒了。

同时,该indext_report_temp_01是一个联合index,基于(corp_id, industry_id, partner_id, report_code, report_type, report_name) 6个字段创建的联合index,观察表和该index的大小:

zyd=> \dt+ t_report_temp
                        List of relations
┌────────┬───────────────┬───────┬───────┬───────┬──────────────┐
│ Schema │     Name      │ Type  │ Owner │ Size  │ Description  │
├────────┼───────────────┼───────┼───────┼───────┼──────────────┤
│ zyd    │ t_report_temp │ table │ zyd   │ 74 MB │ 报表模板主表 │
└────────┴───────────────┴───────┴───────┴───────┴──────────────┘
(1 row)

zyd=> \di+ t_report_temp_01
                                 List of relations
┌────────┬──────────────────┬───────┬───────┬───────────────┬───────┬─────────────┐
│ Schema │       Name       │ Type  │ Owner │     Table     │ Size  │ Description │
├────────┼──────────────────┼───────┼───────┼───────────────┼───────┼─────────────┤
│ zyd    │ t_report_temp_01 │ index │ zyd   │ t_report_temp │ 80 MB │             │
└────────┴──────────────────┴───────┴───────┴───────────────┴───────┴─────────────┘
(1 row)

zyd=>

发现,这个index 80MB,比表本身74MB还大。是不是推断出,此种情况下,优化器选择通过该index来获取数据不够好呢,或者说还不如直接扫描这个表呢?

开始验证:

zyd=> set enable_indexscan =off;
SET
zyd=> set enable_indexonlyscan =off;
SET
zyd=> set enable_bitmapscan =off;
SET
zyd=> explain analyze..
..
..
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                             QUERY PLAN                                                              │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Merge Join  (cost=13420.55..13423.58 rows=204 width=165) (actual time=69.095..69.417 rows=383 loops=1)                              │
│   Merge Cond: ((t.code)::text = (t_report_temp.report_type)::text)                                                                  │
│   ->  Sort  (cost=36.88..38.03 rows=459 width=165) (actual time=0.513..0.569 rows=459 loops=1)                                      │
│         Sort Key: t.code                                                                                                            │
│         Sort Method: quicksort  Memory: 145kB                                                                                       │
│         ->  Seq Scan on t_report_type t  (cost=0.00..16.59 rows=459 width=165) (actual time=0.002..0.121 rows=459 loops=1)          │
│   ->  Sort  (cost=13383.67..13383.79 rows=49 width=7) (actual time=68.576..68.600 rows=383 loops=1)                                 │
│         Sort Key: t_report_temp.report_type                                                                                         │
│         Sort Method: quicksort  Memory: 42kB                                                                                        │
│         ->  HashAggregate  (cost=13381.80..13382.29 rows=49 width=7) (actual time=68.253..68.309 rows=383 loops=1)                  │
│               ->  Seq Scan on t_report_temp  (cost=0.00..13163.63 rows=87269 width=7) (actual time=0.005..67.787 rows=1126 loops=1) │
│                     Filter: ((corp_id IS NULL) AND (partner_id IS NULL))                                                            │
│                     Rows Removed by Filter: 365114                                                                                  │
│ Total runtime: 69.480 ms                                                                                                            │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(14 rows)

zyd=> 

此时,可以看到,当我们禁用index only scan,index scan,bitmap index scan之后,使优化器对于表t_report_temp选择全表扫描之后,整个SQL的执行效率反而更好。
于是,尝试在(report_type,corp_id,partner_id)创建联合index,观察一下SQL效率:

zyd=> create index CONCURRENTLY idx_3 on t_report_temp(report_type,corp_id,partner_id);
CREATE INDEX
zyd=> explain analyze ....
..
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                QUERY PLAN                                                                 │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Sort  (cost=267.54..268.05 rows=204 width=165) (actual time=2.525..2.554 rows=383 loops=1)                                                │
│   Sort Key: t.code                                                                                                                        │
│   Sort Method: quicksort  Memory: 126kB                                                                                                   │
│   ->  Nested Loop Semi Join  (cost=0.42..259.72 rows=204 width=165) (actual time=0.041..2.083 rows=383 loops=1)                           │
│         ->  Seq Scan on t_report_type t  (cost=0.00..16.59 rows=459 width=165) (actual time=0.002..0.053 rows=459 loops=1)                │
│         ->  Index Only Scan using idx_3 on t_report_temp  (cost=0.42..29.58 rows=428 width=7) (actual time=0.004..0.004 rows=1 loops=459) │
│               Index Cond: ((report_type = (t.code)::text) AND (corp_id IS NULL) AND (partner_id IS NULL))                                 │
│               Heap Fetches: 27                                                                                                            │
│ Total runtime: 2.600 ms                                                                                                                   │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(9 rows)

zyd=>

发现,此时SQL的执行耗时从之前的2秒降低到2.6毫秒。

四 小结

通过深入分析SQL执行计划、结合表结构、数据量及数据分布,是做好SQL优化的基本出发点。需要更为深入学习和分析SQL执行计划,能看懂SQL执行计划,对SQL执行计划有感觉,多看多读执行计划,多思考。

PostgreSQL官方文档中,关于执行计划描述,有这么一句话:

Plan-reading is an art that requires some experience to master, but this section attempts to cover the basics.

wordpress迁移步骤和遇到的问题及解决

零 背景说明:

我的这个站点目前托管在香港,采用的是LAMP,即Linux+Apache+MySQL+PHP架构。将于2018年9月6日到期。在到期之前,研究了一下,如果需要迁移到新主机的话,大致流程和步骤,这里简单总结一下。

即,模拟将现有的oracleonlinux.cn(LAMP架构)迁移至本地机器,并且启用新的blog.com域名(这个域名是虚拟的,通过NGINX模拟反向解析,然后配置本地hosts文件)采用LNMP架构。把之前的Apache换成NGINX,域名也更换一下。

一 备份站点文件

即备份主机上的Wordpress及其全部子目录即可,可以采用Linux系统的tar命令来打包。我这里,由于主机服务商暂时不提供SSH远程访问权限,无法连到主机上直接执行备份,但是有Cpanel控制台,我是通过Cpanel控制台上选择的全部备份,然后将备份文件下载到本地:

[root@localhost ~]# ll backup-9.3.2018_08-30-42_oracleon.tar.gz 
-rw-r--r-- 1 nginx nginx 260642919 9月   3 21:06 backup-9.3.2018_08-30-42_oracleon.tar.gz
[root@localhost ~]#

二 备份MySQL数据库

同样,通过Cpanel控制台上选择备份数据库,然后下载到本地:

[root@localhost ~]# ll oracleon_oracle.sql.gz 
-rw-r--r--. 1 root root 3155154 9月   3 19:53 oracleon_oracle.sql.gz
[root@localhost ~]#

三 本地搭建LNMP环境

Linux:Centos 6.5,本地IP为172.16.11.80:

[root@localhost ~]# cat /etc/redhat-release 
CentOS release 6.5 (Final)
[root@localhost ~]# uname -rm
2.6.32-431.el6.x86_64 x86_64
[root@localhost ~]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 00:50:56:99:3D:E7  
          inet addr:172.16.11.80  Bcast:172.16.11.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe99:3de7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2903715683 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3509490556 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1030640015360 (959.8 GiB)  TX bytes:3030630388115 (2.7 TiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:414816117 errors:0 dropped:0 overruns:0 frame:0
          TX packets:414816117 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:469426552437 (437.1 GiB)  TX bytes:469426552437 (437.1 GiB)

[root@localhost ~]#

NGINX:直接通过yum安装;

[root@localhost ~]# yum install nginx
已加载插件:fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
 * base: mirrors.163.com
 * epel: mirrors.ustc.edu.cn
 * extras: mirrors.shu.edu.cn
 * updates: mirrors.zju.edu.cn
解决依赖关系
--> 执行事务检查
---> Package nginx.x86_64 0:1.10.2-1.el6 will be 安装
--> 处理依赖关系 nginx-filesystem = 1.10.2-1.el6,它被软件包 nginx-1.10.2-1.el6.x86_64 需要
--> 处理依赖关系 nginx-all-modules = 1.10.2-1.el6,它被软件包 nginx-1.10.2-1.el6.x86_64 需要
--> 处理依赖关系 nginx-filesystem,它被软件包 nginx-1.10.2-1.el6.x86_64 需要
--> 执行事务检查
---> Package nginx-all-modules.noarch 0:1.10.2-1.el6 will be 安装
--> 处理依赖关系 nginx-mod-stream = 1.10.2-1.el6,它被软件包 nginx-all-modules-1.10.2-1.el6.noarch 需要
--> 处理依赖关系 nginx-mod-mail = 1.10.2-1.el6,它被软件包 nginx-all-modules-1.10.2-1.el6.noarch 需要
--> 处理依赖关系 nginx-mod-http-xslt-filter = 1.10.2-1.el6,它被软件包 nginx-all-modules-1.10.2-1.el6.noarch 需要
--> 处理依赖关系 nginx-mod-http-perl = 1.10.2-1.el6,它被软件包 nginx-all-modules-1.10.2-1.el6.noarch 需要
--> 处理依赖关系 nginx-mod-http-image-filter = 1.10.2-1.el6,它被软件包 nginx-all-modules-1.10.2-1.el6.noarch 需要
--> 处理依赖关系 nginx-mod-http-geoip = 1.10.2-1.el6,它被软件包 nginx-all-modules-1.10.2-1.el6.noarch 需要
---> Package nginx-filesystem.noarch 0:1.10.2-1.el6 will be 安装
--> 执行事务检查
---> Package nginx-mod-http-geoip.x86_64 0:1.10.2-1.el6 will be 安装
---> Package nginx-mod-http-image-filter.x86_64 0:1.10.2-1.el6 will be 安装
---> Package nginx-mod-http-perl.x86_64 0:1.10.2-1.el6 will be 安装
---> Package nginx-mod-http-xslt-filter.x86_64 0:1.10.2-1.el6 will be 安装
---> Package nginx-mod-mail.x86_64 0:1.10.2-1.el6 will be 安装
---> Package nginx-mod-stream.x86_64 0:1.10.2-1.el6 will be 安装
--> 完成依赖关系计算

依赖关系解决

=================================================================================================================================
 软件包                                       架构                    版本                           仓库                   大小
=================================================================================================================================
正在安装:
 nginx                                        x86_64                  1.10.2-1.el6                   epel                  462 k
为依赖而安装:
 nginx-all-modules                            noarch                  1.10.2-1.el6                   epel                  7.7 k
 nginx-filesystem                             noarch                  1.10.2-1.el6                   epel                  8.5 k
 nginx-mod-http-geoip                         x86_64                  1.10.2-1.el6                   epel                   14 k
 nginx-mod-http-image-filter                  x86_64                  1.10.2-1.el6                   epel                   16 k
 nginx-mod-http-perl                          x86_64                  1.10.2-1.el6                   epel                   26 k
 nginx-mod-http-xslt-filter                   x86_64                  1.10.2-1.el6                   epel                   16 k
 nginx-mod-mail                               x86_64                  1.10.2-1.el6                   epel                   43 k
 nginx-mod-stream                             x86_64                  1.10.2-1.el6                   epel                   36 k

事务概要
=================================================================================================================================
Install       9 Package(s)

总下载量:629 k
Installed size: 1.6 M
确定吗?[y/N]:y
下载软件包:
(1/9): nginx-1.10.2-1.el6.x86_64.rpm                                                                      | 462 kB     00:00     
(2/9): nginx-all-modules-1.10.2-1.el6.noarch.rpm                                                          | 7.7 kB     00:00     
(3/9): nginx-filesystem-1.10.2-1.el6.noarch.rpm                                                           | 8.5 kB     00:00     
(4/9): nginx-mod-http-geoip-1.10.2-1.el6.x86_64.rpm                                                       |  14 kB     00:00     
(5/9): nginx-mod-http-image-filter-1.10.2-1.el6.x86_64.rpm                                                |  16 kB     00:00     
(6/9): nginx-mod-http-perl-1.10.2-1.el6.x86_64.rpm                                                        |  26 kB     00:00     
(7/9): nginx-mod-http-xslt-filter-1.10.2-1.el6.x86_64.rpm                                                 |  16 kB     00:00     
(8/9): nginx-mod-mail-1.10.2-1.el6.x86_64.rpm                                                             |  43 kB     00:00     
(9/9): nginx-mod-stream-1.10.2-1.el6.x86_64.rpm                                                           |  36 kB     00:00     
---------------------------------------------------------------------------------------------------------------------------------
总计                                                                                             719 kB/s | 629 kB     00:00     
运行 rpm_check_debug 
执行事务测试
事务测试成功
执行事务
  正在安装   : nginx-filesystem-1.10.2-1.el6.noarch                                                                          1/9 
  正在安装   : nginx-mod-http-geoip-1.10.2-1.el6.x86_64                                                                      2/9 
  正在安装   : nginx-mod-stream-1.10.2-1.el6.x86_64                                                                          3/9 
  正在安装   : nginx-mod-http-perl-1.10.2-1.el6.x86_64                                                                       4/9 
  正在安装   : nginx-mod-http-image-filter-1.10.2-1.el6.x86_64                                                               5/9 
  正在安装   : nginx-mod-http-xslt-filter-1.10.2-1.el6.x86_64                                                                6/9 
  正在安装   : nginx-1.10.2-1.el6.x86_64                                                                                     7/9 
  正在安装   : nginx-mod-mail-1.10.2-1.el6.x86_64                                                                            8/9 
  正在安装   : nginx-all-modules-1.10.2-1.el6.noarch                                                                         9/9 
  Verifying  : nginx-mod-mail-1.10.2-1.el6.x86_64                                                                            1/9 
  Verifying  : nginx-mod-http-geoip-1.10.2-1.el6.x86_64                                                                      2/9 
  Verifying  : nginx-mod-stream-1.10.2-1.el6.x86_64                                                                          3/9 
  Verifying  : nginx-all-modules-1.10.2-1.el6.noarch                                                                         4/9 
  Verifying  : nginx-mod-http-perl-1.10.2-1.el6.x86_64                                                                       5/9 
  Verifying  : nginx-mod-http-image-filter-1.10.2-1.el6.x86_64                                                               6/9 
  Verifying  : nginx-1.10.2-1.el6.x86_64                                                                                     7/9 
  Verifying  : nginx-filesystem-1.10.2-1.el6.noarch                                                                          8/9 
  Verifying  : nginx-mod-http-xslt-filter-1.10.2-1.el6.x86_64                                                                9/9 

已安装:
  nginx.x86_64 0:1.10.2-1.el6                                                                                                    

作为依赖被安装:
  nginx-all-modules.noarch 0:1.10.2-1.el6                     nginx-filesystem.noarch 0:1.10.2-1.el6                            
  nginx-mod-http-geoip.x86_64 0:1.10.2-1.el6                  nginx-mod-http-image-filter.x86_64 0:1.10.2-1.el6                 
  nginx-mod-http-perl.x86_64 0:1.10.2-1.el6                   nginx-mod-http-xslt-filter.x86_64 0:1.10.2-1.el6                  
  nginx-mod-mail.x86_64 0:1.10.2-1.el6                        nginx-mod-stream.x86_64 0:1.10.2-1.el6                            

完毕!
[root@localhost ~]# nginx -v
nginx version: nginx/1.10.2
[root@localhost ~]#

MySQL:通过RPM包安装:

[root@localhost ~]# wget https://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.41-1.el6.x86_64.rpm-bundle.tar
..
..
100%[=======================================================================================>] 227,512,320 1.21M/s   in 2m 26s  

2018-09-04 14:51:22 (1.48 MB/s) - 已保存 “MySQL-5.6.41-1.el6.x86_64.rpm-bundle.tar” [227512320/227512320])

[root@localhost ~]# ll MySQL-5.6.41-1.el6.x86_64.rpm-bundle.tar 
-rw-r--r--. 1 root root 227512320 6月  18 16:11 MySQL-5.6.41-1.el6.x86_64.rpm-bundle.tar
[root@localhost ~]# tar -zxvf MySQL-5.6.41-1.el6.x86_64.rpm-bundle.tar 

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
[root@localhost ~]# tar xvf MySQL-5.6.41-1.el6.x86_64.rpm-bundle.tar 
MySQL-client-5.6.41-1.el6.x86_64.rpm
MySQL-shared-compat-5.6.41-1.el6.x86_64.rpm
MySQL-test-5.6.41-1.el6.x86_64.rpm
MySQL-server-5.6.41-1.el6.x86_64.rpm
MySQL-devel-5.6.41-1.el6.x86_64.rpm
MySQL-shared-5.6.41-1.el6.x86_64.rpm
MySQL-embedded-5.6.41-1.el6.x86_64.rpm
[root@localhost ~]# rpm -ivh MySQL-server-5.6.41-1.el6.x86_64.rpm 
warning: MySQL-server-5.6.41-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
error: Failed dependencies:
        libnuma.so.1()(64bit) is needed by MySQL-server-5.6.41-1.el6.x86_64
        libnuma.so.1(libnuma_1.1)(64bit) is needed by MySQL-server-5.6.41-1.el6.x86_64
        libnuma.so.1(libnuma_1.2)(64bit) is needed by MySQL-server-5.6.41-1.el6.x86_64
[root@localhost ~]#

根据错误提示,安装numactl:

[root@localhost ~]# yum install numactl
已加载插件:fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
 * base: mirrors.163.com
 * epel: mirrors.ustc.edu.cn
 * extras: mirrors.shu.edu.cn
 * updates: mirrors.zju.edu.cn
解决依赖关系
--> 执行事务检查
---> Package numactl.x86_64 0:2.0.9-2.el6 will be 安装
--> 完成依赖关系计算

依赖关系解决

=================================================================================================================================
 软件包                        架构                         版本                                仓库                        大小
=================================================================================================================================
正在安装:
 numactl                       x86_64                       2.0.9-2.el6                         base                        74 k

事务概要
=================================================================================================================================
Install       1 Package(s)

总下载量:74 k
Installed size: 171 k
确定吗?[y/N]:y
下载软件包:
numactl-2.0.9-2.el6.x86_64.rpm                                                                            |  74 kB     00:00     
运行 rpm_check_debug 
执行事务测试
事务测试成功
执行事务
  正在安装   : numactl-2.0.9-2.el6.x86_64                                                                                    1/1 
  Verifying  : numactl-2.0.9-2.el6.x86_64                                                                                    1/1 

已安装:
  numactl.x86_64 0:2.0.9-2.el6                                                                                                   

完毕!
[root@localhost ~]# rpm -ivh MySQL-server-5.6.41-1.el6.x86_64.rpm 
warning: MySQL-server-5.6.41-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                ########################################### [100%]
        file /usr/share/mysql/czech/errmsg.sys from install of MySQL-server-5.6.41-1.el6.x86_64 conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64
..
..
file /usr/share/mysql/charsets/swe7.xml from install of MySQL-server-5.6.41-1.el6.x86_64 conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64
[root@localhost ~]#

再根据提示,系统自带mysql-libs-5.1.71与当前要安装的依赖冲突,先删除掉:

[root@localhost ~]# yum erase mysql-libs-5.1.71-1.el6.x86_64
已加载插件:fastestmirror
设置移除进程
解决依赖关系
--> 执行事务检查
---> Package mysql-libs.x86_64 0:5.1.71-1.el6 will be 删除
--> 处理依赖关系 libmysqlclient.so.16()(64bit),它被软件包 2:postfix-2.6.6-2.2.el6_1.x86_64 需要
--> 处理依赖关系 libmysqlclient.so.16(libmysqlclient_16)(64bit),它被软件包 2:postfix-2.6.6-2.2.el6_1.x86_64 需要
--> 处理依赖关系 mysql-libs,它被软件包 2:postfix-2.6.6-2.2.el6_1.x86_64 需要
--> 执行事务检查
---> Package postfix.x86_64 2:2.6.6-2.2.el6_1 will be 删除
--> 处理依赖关系 /usr/sbin/sendmail,它被软件包 cronie-1.4.4-12.el6.x86_64 需要
--> 执行事务检查
---> Package cronie.x86_64 0:1.4.4-12.el6 will be 删除
--> 处理依赖关系 cronie = 1.4.4-12.el6,它被软件包 cronie-anacron-1.4.4-12.el6.x86_64 需要
--> 执行事务检查
---> Package cronie-anacron.x86_64 0:1.4.4-12.el6 will be 删除
--> 处理依赖关系 /etc/cron.d,它被软件包 crontabs-1.10-33.el6.noarch 需要
--> 使用新的信息重新计算依赖关系
--> 执行事务检查
---> Package crontabs.noarch 0:1.10-33.el6 will be 删除
--> 完成依赖关系计算

依赖关系解决

=============================================================================================================================================================================
 软件包                              架构                        版本                                    仓库                                                           大小
=============================================================================================================================================================================
正在删除:
 mysql-libs                          x86_64                      5.1.71-1.el6                            @anaconda-CentOS-201311272149.x86_64/6.5                      4.0 M
为依赖而移除:
 cronie                              x86_64                      1.4.4-12.el6                            @anaconda-CentOS-201311272149.x86_64/6.5                      174 k
 cronie-anacron                      x86_64                      1.4.4-12.el6                            @anaconda-CentOS-201311272149.x86_64/6.5                       43 k
 crontabs                            noarch                      1.10-33.el6                             @anaconda-CentOS-201311272149.x86_64/6.5                      2.4 k
 postfix                             x86_64                      2:2.6.6-2.2.el6_1                       @anaconda-CentOS-201311272149.x86_64/6.5                      9.7 M

事务概要
=============================================================================================================================================================================
Remove        5 Package(s)

Installed size: 14 M
确定吗?[y/N]:y
下载软件包:
运行 rpm_check_debug 
执行事务测试
事务测试成功
执行事务
  正在删除   : cronie-anacron-1.4.4-12.el6.x86_64                                                                                                                        1/5 
  正在删除   : crontabs-1.10-33.el6.noarch                                                                                                                               2/5 
  正在删除   : cronie-1.4.4-12.el6.x86_64                                                                                                                                3/5 
  正在删除   : 2:postfix-2.6.6-2.2.el6_1.x86_64                                                                                                                          4/5 
  正在删除   : mysql-libs-5.1.71-1.el6.x86_64                                                                                                                            5/5 
  Verifying  : crontabs-1.10-33.el6.noarch                                                                                                                               1/5 
  Verifying  : cronie-anacron-1.4.4-12.el6.x86_64                                                                                                                        2/5 
  Verifying  : cronie-1.4.4-12.el6.x86_64                                                                                                                                3/5 
  Verifying  : 2:postfix-2.6.6-2.2.el6_1.x86_64                                                                                                                          4/5 
  Verifying  : mysql-libs-5.1.71-1.el6.x86_64                                                                                                                            5/5 

删除:
  mysql-libs.x86_64 0:5.1.71-1.el6                                                                                                                                           

作为依赖被删除:
  cronie.x86_64 0:1.4.4-12.el6           cronie-anacron.x86_64 0:1.4.4-12.el6           crontabs.noarch 0:1.10-33.el6           postfix.x86_64 2:2.6.6-2.2.el6_1          

完毕!
[root@localhost ~]#

接着安装MySQL-server:

[root@localhost ~]# rpm -ivh MySQL-server-5.6.41-1.el6.x86_64.rpm 
warning: MySQL-server-5.6.41-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                ########################################### [100%]
   1:MySQL-server           ########################################### [100%]
warning: user mysql does not exist - using root
warning: group mysql does not exist - using root
warning: user mysql does not exist - using root
warning: group mysql does not exist - using root
2018-09-04 15:09:54 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-09-04 15:09:54 0 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
2018-09-04 15:09:54 0 [Note] /usr/sbin/mysqld (mysqld 5.6.41) starting as process 1647 ...
2018-09-04 15:09:54 1647 [Note] InnoDB: Using atomics to ref count buffer pool pages
..
..
A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.

You must change that password on your first connect,
no other statement but 'SET PASSWORD' will be accepted.
See the manual for the semantics of the 'password expired' flag.

Also, the account for the anonymous user has been removed.

..
..
New default config file was created as /usr/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

[root@localhost ~]# 

安装MySQL客户端:

[root@localhost ~]# rpm -ivh MySQL-client-5.6.41-1.el6.x86_64.rpm 
warning: MySQL-client-5.6.41-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                ########################################### [100%]
   1:MySQL-client           ########################################### [100%]
[root@localhost ~]#

启动MySQL,用系统生成的随机口令先登录数据库,然后修改root口令,再创建新的库和用户blog,用于恢复原站点的数据库:

[root@localhost ~]# cat .mysql_secret 
# The random password set for the root user at Tue Sep  4 15:09:55 2018 (local time): QKkCVRsNEhJZEZj0

[root@localhost ~]# service mysql status
 ERROR! MySQL is not running
[root@localhost ~]# service mysql start
Starting MySQL.Logging to '/var/lib/mysql/localhost.localdomain.err'.
... SUCCESS! 
[root@localhost ~]# service mysql status
 SUCCESS! MySQL running (2098)
[root@localhost ~]# mysql -h localhost -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.41

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SET PASSWORD=password('localblog');
Query OK, 0 rows affected (0.00 sec)

mysql> create database blog CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'blog'@'localhost' identified by 'blog123';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on blog.* to 'blog'@'localhost' with grant option;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[root@localhost ~]# 

PHP:yum安装。

[root@localhost ~]# yum install php php-fpm
已加载插件:fastestmirror
..
..
已安装:
  php.x86_64 0:5.3.3-49.el6                                                           php-fpm.x86_64 0:5.3.3-49.el6                                                          

作为依赖被安装:
  apr.x86_64 0:1.3.9-5.el6_9.1                   apr-util.x86_64 0:1.3.9-3.el6_0.1      apr-util-ldap.x86_64 0:1.3.9-3.el6_0.1      httpd.x86_64 0:2.2.15-69.el6.centos     
  httpd-tools.x86_64 0:2.2.15-69.el6.centos      mailcap.noarch 0:2.1.31-2.el6          php-cli.x86_64 0:5.3.3-49.el6               php-common.x86_64 0:5.3.3-49.el6        

完毕!
[root@localhost ~]# yum install php php-fpm

四 配置NGINX

这里,由于我是在本地服务器上模拟站点迁移,并没有新的域名。于是,我用NGINX模拟反向解析一个新的域名:blog.com,然后,把blog.com配置到本地hosts文件。

NGINX的配置文件路径为,/etc/nginx/conf.d。内容如下:

[root@localhost blog]# cat /etc/nginx/conf.d/blog.conf 
#
# The default server
#
server {
    listen       80;
    server_name  blog.com;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

    location / {
        root   /blog;
        index index.php  index.html index.htm;
    }

    error_page  404              /404.html;
    location = /404.html {
        root   /usr/share/nginx/html;
    }

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
        root           /blog;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}
[root@localhost blog]#

该配置文件中的blog.com就是我要通过NGINX来反向代理的站点,/blog指的是我的站点的根目录。
然后,本地hosts文件中,把blog.com,指向172.16.11.96,添加如下一条记录即可:

172.16.11.80 blog.com
且,可以正常访问:
$ ping blog.com
PING blog.com (172.16.11.80): 56 data bytes
64 bytes from 172.16.11.80: icmp_seq=0 ttl=63 time=0.561 ms
64 bytes from 172.16.11.80: icmp_seq=1 ttl=63 time=0.638 ms
^C
--- blog.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.561/0.600/0.638/0.039 ms

asher at MacBook-Air-3 in ~
$ 

五 安装Wordpress

WordPress官方网站,下载最新的安装包,然后解压,放到/blog路径下,修改/blog机器子路径的权限,owner为NGINX的启动用户。这里需要配合前面的第4步骤,前面的NGINX把blog.com反向解析到哪个路径,这里就需要把Wordpress安装在哪儿。

[root@localhost ~]# wget https://wordpress.org/latest.tar.gz
..
[root@localhost ~]# mkdir -p /blog
[root@localhost ~]# mv latest.tar.gz /blog/
[root@localhost ~]# cd /blog/
[root@localhost blog]# tar -zxvf latest.tar.gz 
[root@localhost blog]# mv ./wordpress/* .
[root@localhost blog]# chown -R nginx:nginx /blog/

启动,NGINX、php-fpm,执行安装:

[root@localhost ~]# service nginx restart
停止 nginx:                                               [确定]
正在启动 nginx:                                           [确定]
[root@localhost ~]# service php-fpm restart
停止 php-fpm:                                             [失败]
正在启动 php-fpm:                                         [确定]
[root@localhost ~]#

浏览器打开blog.com,遇到下述错误:
Your PHP installation appears to be missing the MySQL extension which is required by WordPress.
经排查,是PHP版本太低,卸载前面安装的低版本PHP和PHP-fpm,然后安装高版本PHP和PHP-fpm,以及相关依赖包:

[root@localhost ~]# php -v
PHP 5.3.3 (cli) (built: Mar 22 2017 12:27:09) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
[root@localhost ~]# yum erase php php-fpm
..
..
[root@localhost ~]# rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
Retrieving http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
Preparing...                ########################################### [100%]
   1:remi-release           ########################################### [100%]
[root@localhost ~]# yum install --enablerepo=remi --enablerepo=remi-php56 php php-fpm php-opcache php-devel php-mbstring php-mcrypt php-mysqlnd php-phpunit-PHPUnit php-pecl-xdebug php-pecl-xhprof 
..
..

再启动NGINX、php-fpm,执行安装:

[root@localhost blog]# php -v
PHP 5.6.37 (cli) (built: Jul 19 2018 20:06:19) 
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans
[root@localhost blog]# service nginx restart
停止 nginx:                                               [确定]
正在启动 nginx:                                           [确定]
[root@localhost blog]# service php-fpm restart
停止 php-fpm:                                             [确定]
正在启动 php-fpm:                                         [确定]
[root@localhost blog]#

开始安装:

根据提示,填入前面配置MySQL数据库的时候,创建的数据库名blog,用户名blog及密码:

根据提示,安装程序无法创建wp-config.php配置文件,手工拷贝内容,在/blog路径下,创建该文件,内容为上述信息。继续安装:

设置超管用户的密码,这里只是测试,简单起见,设置blog、blog,真实环境下,要注意复杂度。

六 还原MySQL数据库:

解压缩之前创建的MySQL数据库备份文件:

[root@localhost ~]# gzip -d oracleon_oracle.sql.gz
[root@localhost ~]# ll oracleon_oracle.sql
-rw-r--r--. 1 root root 15662095 9月 3 19:53 oracleon_oracle.sql
[root@localhost ~]#

blog用户登录数据库blog,执行脚本恢复:

  
[root@localhost ~]# mysql -h localhost -u blog -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 22
Server version: 5.6.41 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use blog;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+-----------------------+
| Tables_in_blog        |
+-----------------------+
| wp_commentmeta        |
| wp_comments           |
| wp_links              |
| wp_options            |
| wp_postmeta           |
| wp_posts              |
| wp_term_relationships |
| wp_term_taxonomy      |
| wp_termmeta           |
| wp_terms              |
| wp_usermeta           |
| wp_users              |
+-----------------------+
12 rows in set (0.00 sec)

mysql> source ./oracleon_oracle.sql
..
..
Query OK, 0 rows affected (0.00 sec)

mysql>

七 修改数据库中关于旧站点的连接信息:

mysql> UPDATE wp_options SET option_value = replace(option_value, 'www.oracleonlinux.cn','blog.com');
Query OK, 6 rows affected (0.02 sec)
Rows matched: 367  Changed: 6  Warnings: 0

mysql> UPDATE wp_options SET option_value = replace(option_value, 'oracleonlinux.cn','blog.com');
Query OK, 2 rows affected (0.01 sec)
Rows matched: 367  Changed: 2  Warnings: 0

mysql> UPDATE wp_posts SET post_content = replace(post_content, 'www.oracleonlinux.cn','blog.com');
Query OK, 95 rows affected (0.06 sec)
Rows matched: 735  Changed: 95  Warnings: 0

mysql> UPDATE wp_posts SET post_content = replace(post_content, 'oracleonlinux.cn','blog.com');
Query OK, 13 rows affected (0.07 sec)
Rows matched: 735  Changed: 13  Warnings: 0

mysql> UPDATE wp_comments SET comment_content = replace(comment_content, 'www.oracleonlinux.cn', 'blog.com');
Query OK, 8 rows affected (0.01 sec)
Rows matched: 284  Changed: 8  Warnings: 0

mysql> UPDATE wp_comments SET comment_content = replace(comment_content, 'oracleonlinux.cn', 'blog.com');
Query OK, 1 row affected (0.01 sec)
Rows matched: 284  Changed: 1  Warnings: 0

mysql> UPDATE wp_comments SET comment_author_url = replace(comment_author_url, 'www.oracleonlinux.cn', 'blog.com');
Query OK, 66 rows affected (0.01 sec)
Rows matched: 284  Changed: 66  Warnings: 0

mysql> UPDATE wp_comments SET comment_author_url = replace(comment_author_url, 'oracleonlinux.cn', 'blog.com');
Query OK, 10 rows affected (0.01 sec)
Rows matched: 284  Changed: 10  Warnings: 0

mysql> 

八 登录后台查看信息:

注意,这里需要使用旧站点的管理员用户和密码,因为后台的MySQL数据库中已经导入之前的数据库备份信息了。

登录之后,可以看到之前站点上发布的所有文章和数据。这里,需要重新安装以及启用之前站点上的主题和插件。

九 设置NGINX的静态解析

由于之前的网站是LAMP,用Apache解析的,现在改为NGINX代理的话,需要重新配置静态解析。否则的话,会出现主页访问正常,但是其它页面以及标签页和单篇文章打开时报404错误。

在nginx的配置文件的location / 位置处,加上下述信息:

 if (!-e $request_filename) {
            rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2 last;
            rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
            rewrite ^ /index.php last;
        }

完整的用于该站点的反向代理解析文件如下:

#
# The default server
#
server {
    listen       80;
    server_name  blog.com;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

    location / {
        root   /blog;
        index index.php  index.html index.htm;
        
        if (!-e $request_filename) {
            rewrite ^([_0-9a-zA-Z-]+)?(/wp-.*) $2 last;
            rewrite ^([_0-9a-zA-Z-]+)?(/.*\.php)$ $2 last;
            rewrite ^ /index.php last;
        }
    }

    error_page  404              /404.html;
    location = /404.html {
        root   /usr/share/nginx/html;
    }

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
        root           /blog;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

最后,可以在本地看到的站点如下:

PostgreSQL数据库psql客户端工具的设置

在使用psql客户端工具来管理、维护PostgreSQL数据库的过程中,经常遇到一些格式的问题:输出结果被折叠覆盖。对于熟悉Oracle数据库的sql*plus客户端工具的DBA来说,这简直是梦魇。这里,收录一些psql客户端工具的设置。

1 输出结果被折叠覆盖

ai_task1=> \d t_ai_prd_task_field                
                  Table "ai_task1.t_ai_prd_task_field"
          Column           |              Type              | Modifiers 
---------------------------+--------------------------------+-----------
 id                        | character varying(32)          | not null
 task_id                   | character varying(32)          | 
 job_field_id              | character varying(32)          | 
 item_id                   | character varying(32)          | 
 ...
 ...
 sort_no                   | smallint                       | 
 template_display_name     | character varying(128)         | 
Indexes:
    "t_ai_prd_task_field_pkey" PRIMARY KEY, btree (id)
    "idx_task_field_applicant_name" btree (applicant_name)
    "idx_task_field_bill_code" btree (bill_code)
    "idx_task_field_item_id" btree (item_id)
    "idx_task_field_job_field_id" btree (job_field_id)
    "idx_task_field_job_id" btree (job_id)
    "idx_task_field_packet_code" btree (packet_code)
    "idx_task_field_status" btree (status)
    "idx_task_field_submit_time" btree (submit_time)
    "idx_task_field_task_id" btree (task_id)
tegory, cust_category, template_name, input_source)name, applicant_name, task_ca--More--

ai_task1=>

查看表结构时,后面带出index信息中,对于最后一行,完全不知道索引的相关信息,取而代之的是–More–,这不要命么?可以通过在命令行上设置,\pset pager off。

ai_task1=> \pset pager off
Pager usage is off.
ai_task1=> \d t_ai_prd_task_field                
                  Table "ai_task1.t_ai_prd_task_field"
          Column           |              Type              | Modifiers 
---------------------------+--------------------------------+-----------
 id                        | character varying(32)          | not null
 task_id                   | character varying(32)          | 
 job_field_id              | character varying(32)          | 
 item_id                   | character varying(32)          | 
 ...
 ...
 sort_no                   | smallint                       | 
 template_display_name     | character varying(128)         | 
Indexes:
    "t_ai_prd_task_field_pkey" PRIMARY KEY, btree (id)
    "idx_task_field_applicant_name" btree (applicant_name)
    "idx_task_field_bill_code" btree (bill_code)
    "idx_task_field_item_id" btree (item_id)
    "idx_task_field_job_field_id" btree (job_field_id)
    "idx_task_field_job_id" btree (job_id)
    "idx_task_field_packet_code" btree (packet_code)
    "idx_task_field_status" btree (status)
    "idx_task_field_submit_time" btree (submit_time)
    "idx_task_field_task_id" btree (task_id)
    "idx_task_field_union_gin" gin (template_field_name, applicant_name, task_category, cust_category, template_name, input_source)

ai_task1=>

瞬间变得神清气爽,清清楚楚。

2 设置输出回显的linestyle和边框

ai_task1=> \pset border 2
Border style is 2.
ai_task1=> \pset linestyle unicode 
Line style is unicode.
ai_task1=> \d t_07 
                      Table "ai_task1.t_07"
┌───────────────────────────┬────────────────────────┬───────────┐
│          Column           │          Type          │ Modifiers │
├───────────────────────────┼────────────────────────┼───────────┤
│ bill_code                 │ character varying(12)  │           │
│ template_no               │ character varying(32)  │           │
│ template_field_identifier │ character varying(64)  │           │
│ company_name              │ character varying(255) │           │
│ submit_value              │ character varying      │           │
└───────────────────────────┴────────────────────────┴───────────┘

ai_task1=>

整齐划一、干净好看。

 

3 换行截断

postgres=# select * from t;
 id |                                                                                                                                                                                                                                            content                                                                                                                                                                                                                                             
----+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  1 | If you are used to psql, you may notice in the picture above, some content is wrapped. This is \pset format wrapped option.
  2 | Some commands take an SQL identifier (such as a table name) as argument. These arguments follow the syntax rules of SQL: Unquoted letters are forced to lowercase, while double quotes (") protect letters from case conversion and allow incorporation of whitespace into the identifier. Within double quotes, paired double quotes reduce to a single double quote in the resulting name. For example, FOO"BAR"BAZ is interpreted as fooBARbaz, and "A weird"" name" becomes A weird" name.+
    |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               +
    | 
(2 rows)

postgres=#

 

对于terminal window比较小的场景下,这个查询结果并不够友好。可以,通过设置:\pset format wrapped

postgres=# \pset format wrapped 
Output format is wrapped.
postgres=# select * from t;     
 id |                                       content                                        
----+--------------------------------------------------------------------------------------
  1 | If you are used to psql, you may notice in the picture above, some content is wrappe.
    |.d. This is \pset format wrapped option.
  2 | Some commands take an SQL identifier (such as a table name) as argument. These argum.
    |.ents follow the syntax rules of SQL: Unquoted letters are forced to lowercase, while.
    |. double quotes (") protect letters from case conversion and allow incorporation of w.
    |.hitespace into the identifier. Within double quotes, paired double quotes reduce to .
    |.a single double quote in the resulting name. For example, FOO"BAR"BAZ is interpreted.
    |. as fooBARbaz, and "A weird"" name" becomes A weird" name.                          +
    |                                                                                     +
    | 
(2 rows)

postgres=#

4 设置编辑器

通过设置执行psql命令的用户的环境变量EDITOR=vi,可以在psql命令行上通过执行\e 来调用vi编辑SQL语句。需要注意的是:

a \e 只能编辑修改最后1条SQL语句;

b \e 不能编辑从命令行读取的SQL脚本中的命令,这个有别于Oracle数据库。

c  \e 如果出现E437: terminal capability “cm” required时,有可能是用户的环境变量TERM没设置,修改一下即可。

[enterprisedb@ppasdev ~]$ export EDITOR=vi
[enterprisedb@ppasdev ~]$ export TERM=xterm
[enterprisedb@ppasdev ~]$ sh local 
psql.bin (9.3.11.33)
Type "help" for help.

postgres=# select * from t;
..
..
postgres=# \e
调用vi编辑SQL语句。

5 小结

可以把关于psql的设置,配置到需要执行psql命令行工具用户的家目录下的名为.psqlrc文件中:

[enterprisedb@ecs-public2 ~]$ cat ~/.psqlrc 
\pset pager off
\pset border 2
\pset linestyle unicode 
\pset format wrapped 
[enterprisedb@ecs-public2 ~]$

可以把关于环境变量的设置,配置到用户的家目录下的.bash_profile中:

[enterprisedb@ecs-public2 ~]$ cat ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
PATH=$PATH:$HOME/bin:/ppas/9.3as/bin

export PATH

export LANG=en_US.UTF-8
export PGDATA=/ppas/9.3as/data
export EDITOR=vi
export TERM=xterm
[enterprisedb@ecs-public2 ~]$

这些相关设置,更为具体的可以参考官方文档:https://www.postgresql.org/docs/9.1/static/app-psql.html

这里,只是做个简单记录,解决我个人目前遇到的几个棘手问题,后续再读文档时更新本文。

PostgreSQL数据库pg_hba.conf文件学习笔记

对于PostgreSQL数据库的pg_hba.conf文件从以下几个方面来学习。

1 pg_hba.conf的作用

pg_hba.conf文件是数据库判断客户端/应用程序能否正常访问数据库的唯一依据。(HBA stands for host-based authentication.)

略微牵强一点儿的横向比较的话,其作用类似于Oracle数据库的listener.ora文件。

pg的客户端访问数据库的时候,需要提供数据库服务器主机名或者IP地址,端口号,数据库名,数据库用户名,以及数据库用户密码或者数据库认证方式。而这些信息是存在于pg_hba.conf文件中的。同样,Oracle客户端访问数据库的过程中,也需要提供Oracle数据库主机名或者IP地址,端口号,访问协议,以及服务名。而这些信息是记录在listener.ora文件中的。

2 pg_hba.conf的位置

默认情况下,该文件位于数据库服务器的data路径下,名为pg_hba.conf。当然,这些都不是绝对的,尤其对于Unix/Linux环境下,文件的扩展名只是一种约定的惯例,并不像windows平台下,但凡.exe代表可执行文件等。

可以在psql命令行中,通过超级用户执行show hba_file来获取pg_hba.conf的具体路径。

postgres=# show hba_file ;
              hba_file               
-------------------------------------
 /orabackup/pg9.6.4/data/pg_hba.conf
(1 row)

postgres=#

如果是没有权限的普通用户执行该命令,可能会有下述错误:

ai=> show hba_file;
ERROR:  must be superuser to examine "hba_file"
ai=>

 

如果需要修改或者调整配置文件pg_hba.conf的路径的话,需要通过修改postgresql.conf文件中hba_file参数来实现,并且要提前做好pg_hba.conf文件的备份,以及需要重启数据库才能使修改生效。

3 pg_hba.conf的内容

  • 该文件是一个普通的文本类型文件,可以直接用文本编辑器编辑修改;
  • 以#开头行的是注释行;
  • 以行为单位,每一行是一个有效的记录,不允许不支持跨行的记录存在;
  • 每一个生效的行记录都包含:连接类型、数据库名、用户名、允许的客户端IP地址或地址段、认证方式,这5个字段。且,各字段之间以空格分开。类似如下:
# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             172.0.0.0/8             md5

这里local开头的行,表示的是数据库允许通过主机服务器的Unix socket通信来访问和连接数据库。如果没有local开头的行,则意味着数据库不接受socket通信的连接访问方式。说白了,就是没有这一行的话,或注释掉改行的话,则无法通过简单的psql来访问数据库:

[postgres@localhost data]$ psql
psql: FATAL:  no pg_hba.conf entry for host "[local]", user "postgres", database "postgres"
[postgres@localhost data]$

第1个host开头的行,意味着,允许接受客户端为数据库主机环路地址127.0.0.1上,通过提供数据库IP地址,端口,数据库名、用户名的方式来访问数据库,认证方式为trust,即不需要提供密码的方式来访问数据库。且,可以以数据库中已经存在的任意用户来访问任意数据库,如:

[postgres@localhost data]$ psql -h localhost -p 5432 -d postgres -U postgres
psql (9.6.4)
Type "help" for help.

postgres=# \q
[postgres@localhost data]$ psql -h 127.0.0.1 -p 5432 -d postgres -U ai
psql (9.6.4)
Type "help" for help.

postgres=>

第2个host开头的行,意味着,允许从IP范围是172.0.0.0到172.254.254.254之间的任意地址,以数据库中已经存在的用户,发起到任意数据库的连接请求,但是必须要提供用户口令,因为认证方式是是通过md5加密的。如:

[postgres@localhost data]$ psql -h 172.16.11.35 -p 5432 -d postgres -U enterprisedb
Password for user enterprisedb: 
psql (9.6.4, server 9.3.11.33)
Type "help" for help.

postgres=#

从IP为172.18.1.12的客户端,发起到172.16.11.35服务端的连接请求,在提供名为enterprisedb的数据库用户密码之后,顺利连接到服务器。

  • 连接类型,除了local、host之外,还有hostssl和hostnossl方式;
  • 数据库和用户名字段的all,表示可以访问任意数据库,以及以任意数据库用户发起对于数据库的访问,如果需要对于特定的数据库和特定的用户作限制的话,则应该明确写出库名和数据库用户名;
  • 数据库在做连接验证的过程中,对于该文件是从上向下查找的,如果找到一条记录匹配客户端发起的数据库连接请求,则参照该记录来验证,不再继续向下找,验证成功则访问数据库成功,反之失败,如下:

pg_hba.conf含有下述2条记录:

# IPv4 local connections:
host    all             all             127.0.0.1/32            md5 
host    all             all             127.0.0.1/32            trust

那么,此时,数据库选择第一条记录来验证,就必须的正确的输入数据库用户postgres的口令,否则会报错。如下:

[postgres@localhost data]$ psql -h 127.0.0.1 -p 5432 -d postgres -U postgres
Password for user postgres: 
psql: FATAL:  password authentication failed for user "postgres"
[postgres@localhost data]$

显然,对于这次验证失败,数据库并没有选择第2条验证方位为trust的无需密码的方式来验证;

  • 对于数据库名字段上,如果需要匹配多个数据库的话,可以用逗号分隔开多个数据库的库名;
  • 数据库字段名上,如果出现replication的话,并不表示对于名为replication的数据库名授权验证,而是验证一个来自物理备库的连接请求;
  • address字段上,如下:
172.20.143.89/32 for a single host,只匹配172.20.143.89这一个IP地址;
172.20.143.0/24 for a small network,匹配从172.20.143.0到172.20.143.254这个地址段;
10.6.0.0/16 for a larger one,匹配从10.6.0.0到10.6.254.254地址段;
::1/128 for a single host ,匹配IPv6的本地环路地址,等价于IPv4的127.0.0.1;
0.0.0.0/0 represents all IPv4 addresses,匹配所有IPv4的地址;
::0/0 represents all IPv6 addresses,匹配所有IPv6地址;

4 修改pg_hba.conf的内容

通过文本编辑器,按照其固定格式,直接修改完该文件之后,需要执行pg_ctl reload使其生效。

5 个人收获

  • 从基础知识点上搞清楚了该文件的作用;
  • 明白了该文件中local那一行的真正具体含义;
  • 学会了该文件和postgrres.conf文件中的listen_address参数一起搭配来控制,通过网络来访问数据库的控制和限制;如果listen_addresses =’localhost’时,则该数据库不允许任何通过网络方式来的异机访问,只允许通过数据库服务器本机连接本机上的数据库;
  • 学到了一点儿关于网络掩码的设置,如172.20.143.0/24的具体含义。

更为详细的信息,可以进一步查看官方文档:https://www.postgresql.org/docs/current/static/auth-pg-hba-conf.html

PostgreSQL数据库中explain和explain analyze的区别

对于PostgreSQL数据库,可以直接在命令行上使用“explain your_statements”的方式来查看SQL语句的执行计划。

更为具体详细的用法,可以在psql命令行上使用\h explain来查看获取更为具体用法。

这里,需要注意的是,explain analyze 会分析SQL语句的执行计划,并真正执行SQL语句。而,默认情况下,PostgreSQL数据库的事务是自动开启、提交的。那么,对于DML语句来说,如果要使用explain analyze来查看其执行计划的话,就得格外注意了。如下:

postgres=# \c ai ai
You are now connected to database "ai" as user "ai".
ai=> \d p_table
           Table "public.p_table"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 id     | integer               | not null
 name   | character varying(32) | 
Indexes:
    "p_table_pkey" PRIMARY KEY, btree (id)
Referenced by:
    TABLE "c_table" CONSTRAINT "c_table_id_fkey" FOREIGN KEY (id) REFERENCES p_table(id) ON UPDATE CASCADE ON DELETE CASCADE

ai=> select * from p_table ;
 id |    name    
----+------------
  3 | onlyou.com
(1 row)

ai=> select * from c_table ;
 id |  info  
----+--------
  3 | xiamen
(1 row)

ai=> explain delete from p_table ;
                           QUERY PLAN                           
----------------------------------------------------------------
 Delete on p_table  (cost=0.00..17.10 rows=710 width=6)
   ->  Seq Scan on p_table  (cost=0.00..17.10 rows=710 width=6)
(2 rows)

ai=> select * from p_table ;
 id |    name    
----+------------
  3 | onlyou.com
(1 row)

ai=> explain analyze delete from p_table ;
                                                QUERY PLAN                                                
----------------------------------------------------------------------------------------------------------
 Delete on p_table  (cost=0.00..17.10 rows=710 width=6) (actual time=0.074..0.074 rows=0 loops=1)
   ->  Seq Scan on p_table  (cost=0.00..17.10 rows=710 width=6) (actual time=0.013..0.014 rows=1 loops=1)
 Planning time: 0.050 ms
 Trigger for constraint c_table_id_fkey: time=2.637 calls=1
 Execution time: 2.735 ms
(5 rows)

ai=> select * from p_table ;
 id | name 
----+------
(0 rows)

ai=>

explain analyze delete from p_table ;查看一条删除操作SQL的执行计划,其结果就是,真正的把表里的数据删除了,且提交了。
对于所有的DML操作,如果只是想通过explain analyze来查看其执行计划的话,需要在执行之前,显示开启事务,然后查看执行计划,最后回滚事务。

postgres=# begin;
BEGIN
postgres=# explain analyze xxxxxxxxx;
postgres=# rollback;
ROLLBACK
postgres=#

如何在Linux上源码安装PostgreSQL数据库?

零 说明

本文记录在CentOS/RHEL Linux平台上通过源码安装PostgreSQL数据库、配置数据库的过程和步骤,及注意事项。

一 前提条件

要求OS上已经安装下述软件。如果没有,可以直接通过yum来安装。

1 gmake或者make要求至少3.80版本以上

[root@localhost ~]# make -v
GNU Make 3.82
Built for x86_64-redhat-linux-gnu
Copyright (C) 2010  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[root@localhost ~]# gmake -v
GNU Make 3.82
Built for x86_64-redhat-linux-gnu
Copyright (C) 2010  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[root@localhost ~]# which make
/usr/bin/make
[root@localhost ~]# which gmake
/usr/bin/gmake
[root@localhost ~]# ll /usr/bin/make
-rwxr-xr-x. 1 root root 182776 11月  6 2016 /usr/bin/make
[root@localhost ~]# ll /usr/bin/gmake
lrwxrwxrwx. 1 root root 4 1月   8 2018 /usr/bin/gmake -> make
[root@localhost ~]#

这里,看到gmake版本为3.82。gmake是一个链接文件,其指向了make。其实,在CentOS/RHEL平台上,默认情况下,gmake等同于make。

2 C编译器

[root@localhost ~]# which gcc
/usr/bin/gcc
[root@localhost ~]# gcc -v
使用内建 specs。
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
目标:x86_64-redhat-linux
配置为:../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
线程模型:posix
gcc 版本 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) 
[root@localhost ~]#

如果没有的话,可以直接安装一个最新的gcc即可。

3 tar

[root@localhost ~]# which tar
/usr/bin/tar
[root@localhost ~]# tar --version
tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
这是自由软件:您可以自由地更改并重新发布它。
在法律所允许的范围内不含任何担保。

由 John Gilmore 和 Jay Fenlason 所写。
[root@localhost ~]#

用于解压缩源码格式的安装压缩包文件。

 如果下载的源文件格式类似于postgresql-9.4.10.tar.gz,则用tar -zxvf postgresql-9.4.10.tar.gz解压;
如果下载的源文件格式类似于postgresql-9.4.10.tar.bz2,则用tar jxvf postgresql-9.4.10.tar.bz2解压;
注意,第二种格式文件的解压,命令行选项中不带减号-。​

4 GNU readline library

该库文件默认启用。用于在psql命令行下,可以通过键盘的上下箭头调出历史命令以及编辑之前的命令。如果不需要此功能的话,可以在configure的时候,带上--without-readline选项。

zlib compression library

该库文件默认启用。启用该选项则意味着,使用pg_dump/pg_restore对数据库进行备份/恢复的时候,不支持压缩???如果不需要此功能的话,可以在configure的时候,带上–without-zlib选项。

经过我初步验证测试,暂时还没搞清楚该库文件的具体功能和作用。暂时记下这个问题,留待我后续进一步验证一下???

https://dan.langille.org/2013/06/10/using-compression-with-postgresqls-pg_dump/

http://peter.eisentraut.org/blog/2015/12/07/check-your-pg-dump-compression-levels/

二 源码安装

1 创建postgres用户

[root@dev-middleware ~]# groupadd postgres
[root@dev-middleware ~]# useradd -g postgres postgres
[root@dev-middleware ~]# passwd postgres
更改用户 postgres 的密码 。
新的 密码:
无效的密码: 它基于字典单词
重新输入新的 密码:
passwd:所有的身份验证令牌已经成功更新。
[root@dev-middleware ~]#

2 创建数据库安装路径

假定将来要把数据库软件安装在/data/postgres/10.4/路径下,数据库的数据存放在/data/postgres/10.4/data路径下。
这里,先把/data/postgres/10.4/路径创建出来即可,/data/postgres/10.4/data路径不需提前创建。

[root@dev-middleware ~]# mkdir -p /data/postgres/10.4/
[root@dev-middleware ~]# chown -R postgres:postgres /data/postgres/10.4/
[root@dev-middleware ~]#

3 获取源码

到PostgreSQL官方网站,获取源码格式的数据库安装文件。

[root@dev-middleware ~]# su - postgres
[postgres@dev-middleware ~]$ pwd
/home/postgres
[postgres@dev-middleware ~]$ wget https://ftp.postgresql.org/pub/source/v10.4/postgresql-10.4.tar.gz
--2018-08-06 11:25:51--  https://ftp.postgresql.org/pub/source/v10.4/postgresql-10.4.tar.gz
正在解析主机 ftp.postgresql.org (ftp.postgresql.org)... 87.238.57.227, 217.196.149.55, 204.145.124.244, ...
正在连接 ftp.postgresql.org (ftp.postgresql.org)|87.238.57.227|:443... 已连接。
已发出 HTTP 请求,正在等待回应... 200 OK
长度:26652442 (25M) [application/x-gzip]
正在保存至: “postgresql-10.4.tar.gz”

100%[======================================================================================================================================================================================================>] 26,652,442  5.83MB/s 用时 6.4s   

2018-08-06 11:26:03 (3.98 MB/s) - 已保存 “postgresql-10.4.tar.gz” [26652442/26652442])

[postgres@dev-middleware ~]$ ll
总用量 26028
-rw-rw-r-- 1 postgres postgres 26652442 5月   8 05:06 postgresql-10.4.tar.gz
[postgres@dev-middleware ~]$

4 解压源码

如果下载的源码文件格式为类似于postgresql-9.4.10.tar.bz2,则用tar jxvf postgresql-9.4.10.tar.bz2解压。

解压后,会生成一个新的postgresql-10.4路径,该路径下存放的就是源码格式的安装文件。

[postgres@dev-middleware ~]$ pwd
/home/postgres
[postgres@dev-middleware ~]$ ll
总用量 26028
-rw-rw-r-- 1 postgres postgres 26652442 5月   8 05:06 postgresql-10.4.tar.gz
[postgres@dev-middleware ~]$ 
[postgres@dev-middleware ~]$ tar -zxvf postgresql-10.4.tar.gz 
...
...
postgresql-10.4/GNUmakefile.in
postgresql-10.4/.gitattributes
postgresql-10.4/aclocal.m4
postgresql-10.4/configure.in
postgresql-10.4/INSTALL
[postgres@dev-middleware ~]$

5 执行configure

执行configure的命令行选项,–prefix参数,表示把PostgreSQL安装在哪个路径下。默认情况下,不带该参数时,则会安装在/usr/local/pgsql路径下。

[postgres@dev-middleware ~]$ pwd
/home/postgres
[postgres@dev-middleware ~]$ ll
总用量 26032
drwxrwxr-x 6 postgres postgres     4096 5月   8 05:06 postgresql-10.4
-rw-rw-r-- 1 postgres postgres 26652442 5月   8 05:06 postgresql-10.4.tar.gz
[postgres@dev-middleware ~]$ cd postgresql-10.4
[postgres@dev-middleware postgresql-10.4]$ ll
总用量 676
-rw-r--r--  1 postgres postgres    457 5月   8 04:51 aclocal.m4
drwxrwxr-x  2 postgres postgres   4096 5月   8 05:03 config
-rwxr-xr-x  1 postgres postgres 498962 5月   8 04:51 configure
-rw-r--r--  1 postgres postgres  76909 5月   8 04:51 configure.in
drwxrwxr-x 55 postgres postgres   4096 5月   8 05:03 contrib
-rw-r--r--  1 postgres postgres   1192 5月   8 04:51 COPYRIGHT
drwxrwxr-x  3 postgres postgres   4096 5月   8 05:03 doc
-rw-r--r--  1 postgres postgres   3638 5月   8 04:51 GNUmakefile.in
-rw-r--r--  1 postgres postgres    284 5月   8 04:51 HISTORY
-rw-r--r--  1 postgres postgres  71584 5月   8 05:06 INSTALL
-rw-r--r--  1 postgres postgres   1682 5月   8 04:51 Makefile
-rw-r--r--  1 postgres postgres   1212 5月   8 04:51 README
drwxrwxr-x 16 postgres postgres   4096 5月   8 05:06 src
[postgres@dev-middleware postgresql-10.4]$ ./configure --prefix=/data/postgres/10.4/
...
...
config.status: linking src/include/port/linux.h to src/include/pg_config_os.h
config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
[postgres@dev-middleware postgresql-10.4]$

6 执行make world

这里,也可以简单执行make就可以build安装文件的,但是,make world的意思是指把PostgreSQL相关的文档,HTML,以及其它的一些模块都会一起编译进去。比如,在有的环境下遇到无法使用uuid或者使用不了gin index的原因,就是在编译的时候,没有包含这些对应的模块。处理这种的问题的方法也不复杂,只需要进到当时安装时的那个源文件路径下,重新执行make world,然后make install-world。

建议,在初始安装的时候,就直接用make world。

[postgres@dev-middleware postgresql-10.4]$ make world
..
..
make[2]: 离开目录“/home/postgres/postgresql-10.4/contrib/vacuumlo”
make[1]: 离开目录“/home/postgres/postgresql-10.4/contrib”
PostgreSQL, contrib, and documentation successfully made. Ready to install.
[postgres@dev-middleware postgresql-10.4]$

7 执行make install-world

[postgres@dev-middleware postgresql-10.4]$ make install-world 
...
...
/bin/mkdir -p '/data/postgres/10.4/bin'
/bin/install -c  vacuumlo '/data/postgres/10.4/bin'
make[2]: 离开目录“/home/postgres/postgresql-10.4/contrib/vacuumlo”
make[1]: 离开目录“/home/postgres/postgresql-10.4/contrib”
PostgreSQL, contrib, and documentation installation complete.
[postgres@dev-middleware postgresql-10.4]$

8 初始化数据库

[postgres@dev-middleware postgresql-10.4]$ /data/postgres/10.4/bin/initdb -D /data/postgres/10.4/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "zh_CN.UTF-8".
The default database encoding has accordingly been set to "UTF8".
initdb: could not find suitable text search configuration for locale "zh_CN.UTF-8"
The default text search configuration will be set to "simple".

Data page checksums are disabled.

creating directory /data/postgres/10.4/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /data/postgres/10.4/bin/pg_ctl -D /data/postgres/10.4/data -l logfile start

[postgres@dev-middleware postgresql-10.4]$

9 启动数据库

[postgres@dev-middleware postgresql-10.4]$ cd
[postgres@dev-middleware ~]$ ps -ef|grep postgres
root     19521 10727  0 11:25 pts/0    00:00:00 su - postgres
postgres 19522 19521  0 11:25 pts/0    00:00:00 -bash
postgres 19882 19522  0 17:39 pts/0    00:00:00 ps -ef
postgres 19883 19522  0 17:39 pts/0    00:00:00 grep --color=auto postgres
[postgres@dev-middleware ~]$  /data/postgres/10.4/bin/pg_ctl -D /data/postgres/10.4/data -l /home/postgres/pg.log start
waiting for server to start.... done
server started
[postgres@dev-middleware ~]$ ps -ef|grep postgres
root     19521 10727  0 11:25 pts/0    00:00:00 su - postgres
postgres 19522 19521  0 11:25 pts/0    00:00:00 -bash
postgres 21158     1  0 17:39 pts/0    00:00:00 /data/postgres/10.4/bin/postgres -D /data/postgres/10.4/data
postgres 21160 21158  0 17:39 ?        00:00:00 postgres: checkpointer process   
postgres 21161 21158  0 17:39 ?        00:00:00 postgres: writer process   
postgres 21162 21158  0 17:39 ?        00:00:00 postgres: wal writer process   
postgres 21163 21158  0 17:39 ?        00:00:00 postgres: autovacuum launcher process   
postgres 21164 21158  0 17:39 ?        00:00:00 postgres: stats collector process   
postgres 21165 21158  0 17:39 ?        00:00:00 postgres: bgworker: logical replication launcher   
postgres 21168 19522  0 17:39 pts/0    00:00:00 ps -ef
postgres 21169 19522  0 17:39 pts/0    00:00:00 grep --color=auto postgres
[postgres@dev-middleware ~]$

10 修改环境变量

修改postgres用户的~/.bash_profile,PATH添加/data/postgres/10.4/bin,同时,添加PGDATA=/data/postgres/10.4/data环境变量。

PATH=$PATH:$HOME/.local/bin:$HOME/bin:/data/postgres/10.4/bin

export PATH

PGDATA=/data/postgres/10.4/data
export PGDATA

使其环境变量生效:

[postgres@dev-middleware ~]$ source ~/.bash_profile 
[postgres@dev-middleware ~]$

11 修改数据库参数

a 修改pg_hba.conf文件

在/data/postgres/10.4/data/pg_hba.conf文件中,添加下面的一行:

# IPv4 local connections:
host    all             all             0.0.0.0/0               md5

其,表示当前数据库服务器,对使用IPV4地址的客户端机器,允许其通过任意IP地址,通过任意数据库用户发起对该数据库服务器上的任意数据库的访问,但是必须得提供数据库用户密码。其认证方式,是通过MD5加密认证的。

一句话简介:对于使用IPV4地址的客户端,数据库服务器不做限制,可以通过任意用户访问所有数据库。

b 修改postgresql.conf 文件

在/data/postgres/10.4/data/postgresql.conf 文件中,修改#listen_addresses = ‘localhost’为#listen_addresses = ‘*’。

然后,reload使其生效。

原因是,如果不修改的话,则客户端只能通过数据库服务器的环路地址127.0.0.1来访问数据库,不能使用服务器的公网IP地址来访问。

如下,修改之前:

[postgres@dev-middleware ~]$ psql
psql (10.4)
Type "help" for help.

postgres=# show listen_addresses ;
 listen_addresses 
------------------
 localhost
(1 row)

postgres=# \q
[postgres@dev-middleware ~]$ psql -h 127.0.0.1 -p 5432 -d postgres
psql (10.4)
Type "help" for help.

postgres=# show listen_addresses ;
 listen_addresses 
------------------
 localhost
(1 row)

postgres=# \q
[postgres@dev-middleware ~]$ psql -h 172.16.11.44 -p 5432 -d postgres
psql: could not connect to server: 拒绝连接
        Is the server running on host "172.16.11.44" and accepting
        TCP/IP connections on port 5432?
[postgres@dev-middleware ~]$

修改listen_addresses = ‘*’,且重启数据库使其生效之后:

[postgres@dev-middleware ~]$ vi /data/postgres/10.4/data/postgresql.conf 
...
...

listen_addresses = '*'         # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost'; use '*' for all
                                        # (change requires restart)
#port = 5432                            # (change requires restart)
max_connections = 100                   # (change requires restart)
#superuser_reserved_connections = 3     # (change requires restart)
#unix_socket_directories = '/tmp'       # comma-separated list of directories
                                        # (change requires restart)
#unix_socket_group = ''                 # (change requires restart)
#unix_socket_permissions = 0777         # begin with 0 to use octal notation
                                        # (change requires restart)
"/data/postgres/10.4/data/postgresql.conf" 658L, 22755C written
[postgres@dev-middleware ~]$ pg_ctl restart -m fast
waiting for server to shut down....2018-08-06 18:31:40.294 CST [12290] LOG:  received fast shutdown request
2018-08-06 18:31:40.306 CST [12290] LOG:  aborting any active transactions
2018-08-06 18:31:40.307 CST [12290] LOG:  worker process: logical replication launcher (PID 12297) exited with exit code 1
2018-08-06 18:31:40.314 CST [12292] LOG:  shutting down
2018-08-06 18:31:40.347 CST [12290] LOG:  database system is shut down
 done
server stopped
waiting for server to start....2018-08-06 18:31:40.427 CST [14493] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2018-08-06 18:31:40.427 CST [14493] LOG:  listening on IPv6 address "::", port 5432
2018-08-06 18:31:40.433 CST [14493] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-08-06 18:31:40.487 CST [14494] LOG:  database system was shut down at 2018-08-06 18:31:40 CST
2018-08-06 18:31:40.498 CST [14493] LOG:  database system is ready to accept connections
 done
server started
[postgres@dev-middleware ~]$

可以通过数据库服务器的实际IP地址,访问数据库,

[postgres@dev-middleware ~]$ psql 
psql (10.4)
Type "help" for help.

postgres=# show listen_addresses ;
 listen_addresses 
------------------
 *
(1 row)

postgres=# \q
[postgres@dev-middleware ~]$ psql -h 127.0.0.1 -p 5432 -d postgres
psql (10.4)
Type "help" for help.

postgres=# show listen_addresses ;
 listen_addresses 
------------------
 *
(1 row)

postgres=# \q
[postgres@dev-middleware ~]$ psql -h 172.16.11.44 -p 5432 -d postgres -U postgres
Password for user postgres: 
psql (10.4)
Type "help" for help.

postgres=# show listen_addresses ;
 listen_addresses 
------------------
 *
(1 row)

postgres=# \q
[postgres@dev-middleware ~]$

12 删除PostgreSQL数据库软件

postgres用户可以进入到之前解压的源码所在的路径,通过执行make uninstall来卸载已经安装的数据库软件。如下:

[postgres@dev-middleware ~]$ pwd
/home/postgres
[postgres@dev-middleware ~]$ ll
总用量 26036
-rw------- 1 postgres postgres      939 8月   6 18:29 pg.log
drwxrwxr-x 6 postgres postgres     4096 8月   6 11:29 postgresql-10.4
-rw-rw-r-- 1 postgres postgres 26652442 5月   8 05:06 postgresql-10.4.tar.gz
[postgres@dev-middleware ~]$ make uninstall
make: *** 没有规则可以创建目标“uninstall”。 停止。
[postgres@dev-middleware ~]$ cd postgresql-10.4
[postgres@dev-middleware postgresql-10.4]$ make uninstall 
...
...
make[1]: 进入目录“/home/postgres/postgresql-10.4/config”
rm -f '/data/postgres/10.4/lib/pgxs/config/install-sh'
rm -f '/data/postgres/10.4/lib/pgxs/config/missing'
make[1]: 离开目录“/home/postgres/postgresql-10.4/config”
[postgres@dev-middleware postgresql-10.4]$

13 重新编译安装PostgreSQL数据库软件

可以通过前面第4步的,重新解压源码,用新解压出来的源码文件,进行configure,make world,make install-world。

还有一种方式,就是把之前第5步骤执行configure之后的文件,恢复到configure之前的状态,然后通过执行configure,make world,make install-world。这个命令是make distclean。

如下是第二种方式的记录:

[postgres@dev-middleware postgresql-10.4]$ pwd
/home/postgres/postgresql-10.4
[postgres@dev-middleware postgresql-10.4]$ ll
总用量 1088
-rw-r--r--  1 postgres postgres    457 5月   8 04:51 aclocal.m4
drwxrwxr-x  2 postgres postgres   4096 5月   8 05:03 config
-rw-rw-r--  1 postgres postgres 373348 8月   6 11:29 config.log
-rwxrwxr-x  1 postgres postgres  39415 8月   6 11:29 config.status
-rwxr-xr-x  1 postgres postgres 498962 5月   8 04:51 configure
-rw-r--r--  1 postgres postgres  76909 5月   8 04:51 configure.in
drwxrwxr-x 55 postgres postgres   4096 5月   8 05:03 contrib
-rw-r--r--  1 postgres postgres   1192 5月   8 04:51 COPYRIGHT
drwxrwxr-x  3 postgres postgres   4096 5月   8 05:03 doc
-rw-rw-r--  1 postgres postgres   3638 8月   6 11:29 GNUmakefile
-rw-r--r--  1 postgres postgres   3638 5月   8 04:51 GNUmakefile.in
-rw-r--r--  1 postgres postgres    284 5月   8 04:51 HISTORY
-rw-r--r--  1 postgres postgres  71584 5月   8 05:06 INSTALL
-rw-r--r--  1 postgres postgres   1682 5月   8 04:51 Makefile
-rw-r--r--  1 postgres postgres   1212 5月   8 04:51 README
drwxrwxr-x 16 postgres postgres   4096 8月   6 11:29 src
[postgres@dev-middleware postgresql-10.4]$ make distclean
...
...
make[1]: 离开目录“/home/postgres/postgresql-10.4/src”
rm -rf tmp_install/
rm -f config.cache config.log config.status GNUmakefile
[postgres@dev-middleware postgresql-10.4]$ ll
总用量 676
-rw-r--r--  1 postgres postgres    457 5月   8 04:51 aclocal.m4
drwxrwxr-x  2 postgres postgres   4096 5月   8 05:03 config
-rwxr-xr-x  1 postgres postgres 498962 5月   8 04:51 configure
-rw-r--r--  1 postgres postgres  76909 5月   8 04:51 configure.in
drwxrwxr-x 55 postgres postgres   4096 5月   8 05:03 contrib
-rw-r--r--  1 postgres postgres   1192 5月   8 04:51 COPYRIGHT
drwxrwxr-x  3 postgres postgres   4096 5月   8 05:03 doc
-rw-r--r--  1 postgres postgres   3638 5月   8 04:51 GNUmakefile.in
-rw-r--r--  1 postgres postgres    284 5月   8 04:51 HISTORY
-rw-r--r--  1 postgres postgres  71584 5月   8 05:06 INSTALL
-rw-r--r--  1 postgres postgres   1682 5月   8 04:51 Makefile
-rw-r--r--  1 postgres postgres   1212 5月   8 04:51 README
drwxrwxr-x 16 postgres postgres   4096 8月   6 19:10 src
[postgres@dev-middleware postgresql-10.4]$

三 小结

本文用于记录如何在Linux平台上,通过编译源码的方式,来安装配置PostgreSQL数据库,作一个简单记录。更为详细的信息可以参考官方文档

PostgreSQL数据库排序的小结

一 OS环境

 [root@localhost ~]# cat /etc/redhat-release 
CentOS release 6.5 (Final)
[root@localhost ~]# uname -rm
2.6.32-431.el6.x86_64 x86_64
[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         48216      47094       1122          0        286      44795
-/+ buffers/cache:       2012      46204
Swap:        24175       1712      22463
[root@localhost ~]#

二 数据库环境

ai=> select version();
                                                 version                                                  
----------------------------------------------------------------------------------------------------------
 PostgreSQL 9.6.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit
(1 row)

ai=> \dt+ t_btree 
                    List of relations
 Schema |  Name   | Type  | Owner |  Size  | Description 
--------+---------+-------+-------+--------+-------------
 public | t_btree | table | ai    | 346 MB | 
(1 row)

ai=> select count(*) from t_btree ;
  count   
----------
 10000000
(1 row)

ai=>

三 external merge

ai=> show shared_buffers ;
 shared_buffers 
----------------
 128MB
(1 row)

ai=> show work_mem ;
 work_mem 
----------
 4MB
(1 row)

ai=> explain (analyze,verbose,buffers,costs,timing) select * from t_btree order by id;
                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1580363.83..1605363.83 rows=10000000 width=4) (actual time=8397.774..10610.457 rows=10000000 loops=1)
   Output: id
   Sort Key: t_btree.id
   Sort Method: external merge  Disk: 136936kB
   Buffers: shared hit=15871 read=28377, temp read=50518 written=50518
   ->  Seq Scan on public.t_btree  (cost=0.00..144248.00 rows=10000000 width=4) (actual time=0.028..861.348 rows=10000000 loops=1)
         Output: id
         Buffers: shared hit=15871 read=28377
 Planning time: 2.056 ms
 Execution time: 11135.532 ms
(10 rows)

ai=>

这里的排序方法采用的external mege,意味着数据量太大了,在内存里排序排不下(work_mem=4MB对于当前场景的排序,不够用),只好交换到磁盘上排序,磁盘排序完成之后,再把排序结果交换到内存中。效率最低。

shared_buffers相当于Oracle数据库的database buffer cache,单纯加大该参数,即使所有的数据都能从shared buffers命中,对该排序依然无效,如下:

ai=> explain (analyze,verbose,buffers,costs,timing) select id from t_btree order by id;
                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1580363.83..1605363.83 rows=10000000 width=4) (actual time=8032.685..10126.444 rows=10000000 loops=1)
   Output: id
   Sort Key: t_btree.id
   Sort Method: external merge  Disk: 136936kB
   Buffers: shared hit=44248, temp read=50518 written=50518
   ->  Seq Scan on public.t_btree  (cost=0.00..144248.00 rows=10000000 width=4) (actual time=0.021..792.516 rows=10000000 loops=1)
         Output: id
         Buffers: shared hit=44248
 Planning time: 0.062 ms
 Execution time: 10655.508 ms
(10 rows)

ai=> select 44248*8;
 ?column? 
----------
   353984
(1 row)

ai=> select 44248*8/1024.0;
       ?column?       
----------------------
 345.6875000000000000
(1 row)

ai=> show shared_buffers ;
 shared_buffers 
----------------
 4GB
(1 row)

ai=> show work_mem ;
 work_mem 
----------
 4MB
(1 row)

ai=>

shared_buffers=4GB,work_mem=4MB时。执行计划中,Buffers: shared hit=44248,内存命中44248个内存块儿,每个内存页8KB,换算之后,约为346MB,足够缓存该表的全部数据,即所有数据都从内存中读取。但是,排序方法依然是external merge。

四 quicksort排序

ai=> show shared_buffers ;
 shared_buffers 
----------------
 512MB
(1 row)

ai=> show work_mem ;
 work_mem 
----------
 4MB
(1 row)

ai=> set work_mem ='800MB';
SET
ai=> show work_mem ;
 work_mem 
----------
 800MB
(1 row)

ai=> explain (analyze,verbose,buffers,costs,timing) select * from t_btree order by id;
                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1306922.83..1331922.83 rows=10000000 width=4) (actual time=15001.049..17213.924 rows=10000000 loops=1)
   Output: id
   Sort Key: t_btree.id
   Sort Method: quicksort  Memory: 741817kB
   Buffers: shared hit=15903 read=28345
   ->  Seq Scan on public.t_btree  (cost=0.00..144248.00 rows=10000000 width=4) (actual time=0.060..920.913 rows=10000000 loops=1)
         Output: id
         Buffers: shared hit=15903 read=28345
 Planning time: 0.054 ms
 Execution time: 17756.761 ms
(10 rows)

ai=> explain (analyze,verbose,buffers,costs,timing) select * from t_btree order by id;
                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1306922.83..1331922.83 rows=10000000 width=4) (actual time=6075.205..7945.550 rows=10000000 loops=1)
   Output: id
   Sort Key: t_btree.id
   Sort Method: quicksort  Memory: 741817kB
   Buffers: shared hit=16063 read=28185
   ->  Seq Scan on public.t_btree  (cost=0.00..144248.00 rows=10000000 width=4) (actual time=0.060..879.729 rows=10000000 loops=1)
         Output: id
         Buffers: shared hit=16063 read=28185
 Planning time: 0.070 ms
 Execution time: 8466.822 ms
(10 rows)

ai=> explain (analyze,verbose,buffers,costs,timing) select * from t_btree order by id;
                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1306922.83..1331922.83 rows=10000000 width=4) (actual time=5400.587..7208.614 rows=10000000 loops=1)
   Output: id
   Sort Key: t_btree.id
   Sort Method: quicksort  Memory: 741817kB
   Buffers: shared hit=16095 read=28153
   ->  Seq Scan on public.t_btree  (cost=0.00..144248.00 rows=10000000 width=4) (actual time=0.059..847.957 rows=10000000 loops=1)
         Output: id
         Buffers: shared hit=16095 read=28153
 Planning time: 0.068 ms
 Execution time: 7724.861 ms
(10 rows)

ai=>

说明:

1 默认情况下work_mem=4MB,不满足在内存中完成该排序操作,通过向上调整work_mem=800MB ,使其能够在内存中完成该排序,这里的800MB是逐渐上调得到的一个值。为了学习研究用,实际生产环境,调整该参数值时,要注意该值work_mem*max_connections不能超过总物理内存大小;

2 调整完work_mem=800MB后,多次观察该SQL的执行计划,可以发现随着Buffers: shared hit=16095 read=28153的变化,其Execution time逐渐降低。

This will definitely be faster than external merge, since all of the data is brought into memory and then the sorting is done in memory itself.

五  top-N heapsort

ai=> show work_mem ;
 work_mem 
----------
 4MB
(1 row)

ai=> show shared_buffers ;
 shared_buffers 
----------------
 128MB
(1 row)

ai=> explain (analyze,verbose,buffers,costs,timing) select id from t_btree order by id limit 10;
                                                               QUERY PLAN                                                                
-----------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=360344.40..360344.43 rows=10 width=4) (actual time=1784.116..1784.117 rows=10 loops=1)
   Output: id
   Buffers: shared hit=3 read=44248
   ->  Sort  (cost=360344.40..385344.40 rows=10000000 width=4) (actual time=1784.113..1784.113 rows=10 loops=1)
         Output: id
         Sort Key: t_btree.id
         Sort Method: top-N heapsort  Memory: 25kB
         Buffers: shared hit=3 read=44248
         ->  Seq Scan on public.t_btree  (cost=0.00..144248.00 rows=10000000 width=4) (actual time=0.023..856.626 rows=10000000 loops=1)
               Output: id
               Buffers: shared read=44248
 Planning time: 0.396 ms
 Execution time: 1784.149 ms
(13 rows)

ai=>

说明:

1 这里SQL加上了limit 10,则执行计划选择了top-N heapsort,同时,内存只用了25kB。

2 PostgreSQL数据库维护了一个heap的内存结构,该heap有一个上限大小。排序过程大致如下:PostgreSQL先根据limit大小,顺序的把数据放入heap中,等到heap被填满之后,再去读取下一个数据,判断其值是否小于heap中已有的最大值?如果大于最大值,则直接丢弃这个数据,返回heap中的结果集,排序完成。如果小于最大值,则把这个值放入heap中的最大值处,并把heap中之前的那个最大值移除heap,继续读取下一个值,重复该过程,直到读取的新值不再大于heap中的最大值为止。

3 由于SQL中有order by id,才会涉及到排序操作,如果没有这个order by从句的话,那么无需涉及排序,直接返回表中读取的前10条记录;

4 这里无需对整个表进行排序,只需获取最小的10条数据,并对其进行排序即可。

六 index 排序

ai=> create index idx_t_btree_id on t_btree(id);
CREATE INDEX
ai=> show work_mem ;
 work_mem 
----------
 4MB
(1 row)

ai=> show shared_buffers ;
 shared_buffers 
----------------
 128MB
(1 row)

ai=> explain (analyze,verbose,buffers,costs,timing) select * from t_btree order by id limit 10;
                                                                      QUERY PLAN                                                                       
-------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.43..0.87 rows=10 width=4) (actual time=0.030..0.045 rows=10 loops=1)
   Output: id
   Buffers: shared hit=13
   ->  Index Only Scan using idx_t_btree_id on public.t_btree  (cost=0.43..436680.67 rows=10000033 width=4) (actual time=0.028..0.039 rows=10 loops=1)
         Output: id
         Heap Fetches: 10
         Buffers: shared hit=13
 Planning time: 0.120 ms
 Execution time: 0.066 ms
(9 rows)

ai=> explain (analyze,verbose,buffers,costs,timing) select * from t_btree order by id desc limit 10;
                                                                           QUERY PLAN                                                                           
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.43..0.87 rows=10 width=4) (actual time=0.024..0.038 rows=10 loops=1)
   Output: id
   Buffers: shared hit=12 read=1
   ->  Index Only Scan Backward using idx_t_btree_id on public.t_btree  (cost=0.43..436680.67 rows=10000033 width=4) (actual time=0.022..0.035 rows=10 loops=1)
         Output: id
         Heap Fetches: 10
         Buffers: shared hit=12 read=1
 Planning time: 0.213 ms
 Execution time: 0.061 ms
(9 rows)

ai=>

说明:

实际情况下,通过index来访问数据,并不是严格意义的排序,只不过是index本身已经是排好序的数据集,数据库只需直接从index读取数据即可,并不需要执行额外的排序操作。这里,id字段上有一个B-tree index,无论是按照id的升序、降序排序,都可以直接从index读取数据。

七 小结

1 此前,并没有认真的关注过external merge排序操作,原来是因为内存空间不够,work_mem内存区不足以用于排序的操作,故而数据库不得不采用将数据交换到磁盘排序,然后把排序结果交换回内存;

2 quick sort是一种成熟的排序方法,大学的数据结构课程学过,早已还给老师了,找时间回顾一下该知识点;

3 heap sort方法对于自己是一种新的排序,通过小结,也大致清楚了其排序的实现方式。

八 参考

https://madusudanan.com/blog/all-you-need-to-know-about-sorting-in-postgres/