Category: 高性能伺服器

protoc –version’ did not return a version

按照hadoop官方网站的文档，路径为http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html.用svn co http://svn.apache.org/repos/asf/hadoop/common/trunk下载资源，使用 mvn clean install -DskipTests 进行编译时候，抛出异常

INFO] ————————————————————————
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: ‘protoc –version’ did not return a version -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-common

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: ‘protoc –version’ did not return a version -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :hadoop-common

经过查找原因，适用于没有没有安装 protoc 2.5.0，在官网提示为：NOTE: You will need protoc 2.5.0 installed.

需要安装 protoc 2.5.0，他是google的一个数据缓冲高效可扩展的服务包，

下载地址为http://code.google.com/p/protobuf/downloads/list

由于使用的linux系统，所以下的 protobuf-2.5.0.tar.gz   2.3 MB

然后，安装protobuf 服务包，官方提供的命令如下：

$ ./configure
$ make
$ make check
$ make install

如果，如果系统默认安装了gcc,将会顺利的完成，如果没有的话，将会抛出找不到  找不到当前的文件夹 $path路径，

由于我是用的ubuntu,使用
sudo apt-get install gcc
如果不能安装完成，你需要更行
sudo apt-get update 更新ubuntu 的库
然后再继续的安装
sudo apt-get install gcc

 

面对下面的错误，百度一下，这是缺少相应的库导致的，所以可能在不同系统，安装不同软件的时候都可能出现。

configure: error: C++ preprocessor “/lib/cpp” fails sanity check

使用下载库文件， sudo apt-get install build-essential

 

然后，在执行./configure,顺利的完成

在后查看版本，$protoc –version

将会提示找不到 protoc

让你安装  sudo apt-get install protobuf-compiler

不要安装，如果安装了，它和你的protobuf-2.5.0.tar.gz  没什么关系，运行

$protoc –version

libprotoc.2.4.1

hadoop 要求的版本为2.5.0，mvn编译将不通过。

这个时候不慌，因为protobuf-2.5.0.tar.gz  我们已经编译完了，

只需要完成配置完成就行了

$sudo vi /etc/profile

export PROTOC_HOME=/opt/protobuf-2.5.0

export PATH=$PATH:$PROTOC_HOME/src

然后，

$protoc –version

libprotoc.2.5.0

祝你成功

Hadoop快速入门

目的

这篇文档的目的是帮助你快速完成单机上的Hadoop安装与使用以便你对Hadoop分布式文件系统(HDFS)和Map-Reduce框架有所体会，比如在HDFS上运行示例程序或简单作业等。

 

先决条件

 

支持平台

  • GNU/Linux是产品开发和运行的平台。 Hadoop已在有2000个节点的GNU/Linux主机组成的集群系统上得到验证。
  • Win32平台是作为开发平台支持的。由于分布式操作尚未在Win32平台上充分测试，所以还不作为一个生产平台被支持。

 

所需软件

Linux和Windows所需软件包括:

  1. JavaTM1.5.x，必须安装，建议选择Sun公司发行的Java版本。
  2. ssh 必须安装并且保证 sshd一直运行，以便用Hadoop 脚本管理远端Hadoop守护进程。

Windows下的附加软件需求

  1. Cygwin – 提供上述软件之外的shell支持。

 

安装软件

如果你的集群尚未安装所需软件，你得首先安装它们。

以Ubuntu Linux为例:

$ sudo apt-get install ssh
$ sudo apt-get install rsync

在Windows平台上，如果安装cygwin时未安装全部所需软件，则需启动cyqwin安装管理器安装如下软件包：

  • openssh – Net 类

 

下载

为了获取Hadoop的发行版，从Apache的某个镜像服务器上下载最近的 稳定发行版。

 

运行Hadoop集群的准备工作

解压所下载的Hadoop发行版。编辑 conf/hadoop-env.sh文件，至少需要将JAVA_HOME设置为Java安装根路径。

尝试如下命令：
$ bin/hadoop
将会显示hadoop 脚本的使用文档。

现在你可以用以下三种支持的模式中的一种启动Hadoop集群：

  • 单机模式
  • 伪分布式模式
  • 完全分布式模式

 

单机模式的操作方法

默认情况下，Hadoop被配置成以非分布式模式运行的一个独立Java进程。这对调试非常有帮助。

下面的实例将已解压的 conf 目录拷贝作为输入，查找并显示匹配给定正则表达式的条目。输出写入到指定的output目录。
$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’
$ cat output/*

 

伪分布式模式的操作方法

Hadoop可以在单节点上以所谓的伪分布式模式运行，此时每一个Hadoop守护进程都作为一个独立的Java进程运行。

配置

使用如下的 conf/hadoop-site.xml:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>localhost:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

 

免密码ssh设置

现在确认能否不输入口令就用ssh登录localhost:
$ ssh localhost

如果不输入口令就无法用ssh登陆localhost，执行下面的命令：
$ ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

执行

格式化一个新的分布式文件系统：
$ bin/hadoop namenode -format

启动Hadoop守护进程：
$ bin/start-all.sh

Hadoop守护进程的日志写入到 ${HADOOP_LOG_DIR} 目录 (默认是 ${HADOOP_HOME}/logs).

浏览NameNode和JobTracker的网络接口，它们的地址默认为：

将输入文件拷贝到分布式文件系统：
$ bin/hadoop fs -put conf input

运行发行版提供的示例程序：
$ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’

查看输出文件：

将输出文件从分布式文件系统拷贝到本地文件系统查看：
$ bin/hadoop fs -get output output
$ cat output/*

或者

在分布式文件系统上查看输出文件：
$ bin/hadoop fs -cat output/*

完成全部操作后，停止守护进程：
$ bin/stop-all.sh

 

完全分布式模式的操作方法

关于搭建完全分布式模式的，有实际意义的集群的资料可以在这里找到。

5 Steps to Take Care of Your MongoDB Performance

Do you face some performance issues in your MongoDB setup?

In this case follow these steps to provide some first aid to your system and gain some space for a long term architecture (such as Sharding).

Step 1: Enable Slow Queries

Get intelligence about your system behavior and performance bottlenecks. Usually there is a high correlation between the slow queries and your performance bottleneck, so use the following method to enable your system profiling collection:

db.setProfilingLevel(1, 100);

Step 2: Use Explain

Explore the problematic queries using explain. You can also use mtools to analyze the logged queries to find high frequent ones.

Step 3: Create Indexes

Your analysis should result with new indexes in order to improve the queries

Don’t forget to use index buildup in the background to avoid collections locking and system downtime.

Step 4: Use Sparse Indexes to Reduce the Size of the Indexes

If you use sparse documents, and heavily using the $exists key words in your queries, using sparse indexes (that includes only documents that includes your field) can minimize your index size the boost your query performance.

Step 5: Use Secondary Preferred to Offload Queries to Slaves

You probably have a replica set and it’s waste of resources not using your slaves for read queries (especially for reporting and search operations).

By changing your connection string to secondary preferred, your application will try to run read queries on the slaves before doing that on your master.

Bottom Line

Using these simple methods, you can gain time and space before hitting a wall.

Getting Started With Hubot

You will need node.js and npm. Joyent has
an excellent blog post on how to get those installed, so we’ll omit those details here.

Once node and npm are ready, we can install the hubot generator:

%  npm install -g yo generator-hubot

This will give us the hubot yeoman generator. Now we
can make a new directory, and generate a new instance of hubot in it. For example, if
we wanted to make a bot called myhubot:

% mkdir myhubot
% cd myhubot
% yo hubot

At this point, you’ll be asked a few questions about who is creating the bot,
and which adapter you’ll be using. Adapters are hubot’s way of
integrating with different chat providers.

If you are using git, the generated directory includes a .gitignore, so you can
initialize and add everything:

% git init
% git add .
% git commit -m "Initial commit"

If you’d prefer to automate your hubot build without being interactively
prompted for its configuration, you can add the following options
to the yo hubot command to do so:

Option Description
--owner="Bot Wrangler <bw@example.com>" Bot owner, e.g. “Bot Wrangler bw@example.com“
--name="Hubot" Bot name, e.g. “Hubot”
--description="Delightfully aware robutt" Bot description, e.g. “Delightfully aware robutt”
--adapter=campfire Bot adapter, e.g. “campfire”
--defaults Declare all defaults are set and no prompting required

You now have your own functional hubot! There’s a bin/hubot
command for convenience, to handle installing npm dependencies, loading scripts,
and then launching your hubot.

Hubot needs Redis to persist data, so before you can start hubot on your own computer, you should have Redis installed on your localhost. If just want to test Hubot without Redis, then you can remove redis-brain.coffee from hubot-scripts.json.

% bin/hubot
Hubot>

This starts hubot using the shell adapter, which
is mostly useful for development. Make note of Hubot>; this is the name your hubot will
respond to with commands. For example, to list available commands:

% bin/hubot
Hubot> hubot: help
hubot <keyword> tweet - Returns a link to a tweet about <keyword>
hubot <user> is a badass guitarist - assign a role to a user
hubot <user> is not a badass guitarist - remove a role from a user
hubot animate me <query> - The same thing as `image me`, except adds a few parameters to try to return an animated GIF instead.
hubot convert me <expression> to <units> - Convert expression to given units.
hubot die - End hubot process
hubot echo <text> - Reply back with <text>
hubot fake event <event> - Triggers the <event> event for debugging reasons
hubot help - Displays all of the help commands that Hubot knows about.
hubot help <query> - Displays all help commands that match <query>.
hubot image me <query> - The Original. Queries Google Images for <query> and returns a random top result.
hubot map me <query> - Returns a map view of the area returned by `query`.
hubot mustache me <query> - Searches Google Images for the specified query and mustaches it.
hubot mustache me <url> - Adds a mustache to the specified URL.
hubot ping - Reply with pong
hubot show storage - Display the contents that are persisted in the brain
hubot show users - Display all users that hubot knows about
hubot the rules - Make sure hubot still knows the rules.
hubot time - Reply with current time
hubot translate me <phrase> - Searches for a translation for the <phrase> and then prints that bad boy out.
hubot translate me from <source> into <target> <phrase> - Translates <phrase> from <source> into <target>. Both <source> and <target> are optional
hubot who is <user> - see what roles a user has
hubot youtube me <query> - Searches YouTube for the query and returns the video embed link.
hubot pug bomb N - get N pugs
hubot pug me - Receive a pug
hubot ship it - Display a motivation squirrel

You almost definitely will want to change your hubot’s name to add character. bin/hubot takes a --name:

% bin/hubot --name myhubot
myhubot>

Your hubot will now respond as myhubot. This is
case-insensitive, and can be prefixed with @ or suffixed with :. These are equivalent:

MYHUBOT help
myhubot help
@myhubot help
myhubot: help

Scripting

Hubot’s power comes through scripting. Read more about scripting for the deal on bending hubot to your will using code.

There are many community-contributed scripts available through hubot-scripts. To use scripts from it:

  • Make sure hubot-scripts is listed as a dependency in package.json (it should by default)
  • Update hubot-scripts.json to include the script you want in the list. Make sure the file is still valid JSON!
  • Review the script to see if there’s dependencies or configuration to add

In addition, there are scripts released as npm packages. If you find one you want to use:

  1. Add the package to the list of dependencies into your package.json
  2. npm install to make sure its installed

To enable third-party scripts that you’ve added you will need to add the package
name as a double quoted string to the external-scripts.json file in this repo.

Please note that external scripts may become the default for hubot scripts in future releases.

Adapters

Hubot uses the adapter pattern to support multiple chat-backends. Here is a list of available adapters, along with details on how to configure them.

Deploying

You can deploy hubot to Heroku, which is the officially supported method.
Additionally you are able to deploy hubot to a UNIX-like system or Windows.
Please note the support for deploying to Windows isn’t officially supported.

Patterns

Using custom scripts, you can quickly customize Hubot to be the most life embettering robot he or she can be. Readdocs/patterns.md for some nifty tricks that may come in handy as you teach your hubot new skills.

用bash解决hadoop的磁盘空间检查性能问题

项目使用的hadoop已经存放了3000W+的文件，

为了节省成本，当时抢建平台时，使用了组装服务器+普通硬盘

hadoop每次做du操作都非常耗时，于是把hadoop代码改了一个

使用一个bash脚本替代原来du操作。

bash:

#/bin/sh
mydf=$(df $2 | grep -vE ‘^Filesystem|tmpfs|cdrom’ | awk ‘{ print $3 }’)
echo -e “$mydf\t$2”

java:hadoop\src\core\org\apache\hadoop\fs\DU.java：168行的toString()及getExecString()方法

public String toString() {
return
“mydu -sk ” + dirPath +”\n” +
used + “\t” + dirPath;
}

protected String[] getExecString() {
return new String[] {“mydu”, “-sk”, dirPath};
}

改造后，原来的du操作其他不耗时。

只是存在统计不准确的问题，不过并不影响hadoop运作。

mongodb数据迁移2种方式比较

尝试了2种方式对数据进行迁移，一种是rsync，直接拉取数据；另一种是使用mongodump/mongorestore

1.rsync
操作步骤：
1.2：
[mongodb]
path = /data1/mongodb/data
hosts allow = 192.168.1.0/24
read only = no
write only = no
1.3：
rsync -avz root@192.168.1.2::mongodb/dbname /data/mongodb-linux-x86_64-1.8.1/data/
chown -R mongodb:mongodb /data/mongodb-linux-x86_64-1.8.1/data/

使用时间：50分钟
到目标服务器数据：50G
优点：使用时间短
缺点：需要配置rsync，数据占用的空间大（数据原封不动的拉取过来，包括碎片）

2.mongodump/mongorestore
操作步骤：
mongodump:
/data/PRG/mongodb/bin/mongodump –host 192.168.1.2:27017 -d dbname -uuername -ppasswd -o /data/mongodb-linux-x86_64-1.8.1/data/ –directoryperdb
mongorestore:
/data/mongodb-linux-x86_64-1.8.1/bin/mongorestore –dbpath /data/mongodb-linux-x86_64-1.8.1/data/ –directoryperdb /data/dbname/
chown -R mongodb:mongodb /data/mongodb-linux-x86_64-1.8.1/data/

使用时间：35（mongodump）+90（mongorestore）
到目标服务器数据：20G（需要的空间大大减小，拉取过程中相当于做了一次碎片整理）
优点：迁移到新服务器的数据经过了整理，需要空间大大减小
缺点：需要时间长

数据迁移时需要停mongo进行操作，而2种方式各有优缺点，如果可以忽略操作时间内的数据的话，那么使用第2种方式会比较好（已经有不少例子因为碎片带来严重的后果）

mongodb sharding cluster(分片集群)

MongoDB的auto-sharding功能是指mongodb通过mongos自动建立一个水平扩展的数据库集群系统，将数据库分表存储在sharding的各个节点上。

通过把Sharding和Replica Sets相结合，可以搭建一个分布式的，高可用性，自动水平扩展的集群。

要构建MongoDB Sharding Cluster，需要三种角色：

Shard Server: mongod 实例， 使用 Replica Sets，确保每个数据节点都具有备份、自动容错转移、自动恢复能力。用于存储实际的数据块，实际生产环境中一个shard server角色可由几台机器组个一个relica set承担，防止主机单点故障

Config Server: mongod 实例，使用 3 个配置服务器，确保元数据完整性(two-phase commit)。存储了整个 Cluster Metadata，其中包括 chunk 信息。

Route Server: mongos 实例，配合 LVS，实现负载平衡，提高接入性能(high performance)。前端路由，客户端由此接入，且让整个集群看上去像单一数据库，前端应用可以透明使用。

环境如下：

192.168.198.131

shard1:10001

shard2:10002

shard3:10003

config1:20000

192.168.198.129

shard1:10001

shard2:10002

shard3:10003

config2:20000

192.168.198.132

shard1:10001

shard2:10002

shard3:10003

config3:20000

192.168.198.133

mongos:27017

分别在三台服务器上安装mongod服务，安装如下：

# wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.0.3.tgz

# tar zxvf mongodb-linux-x86_64-2.0.3.tgz -C ../software/

# ln -s mongodb-linux-x86_64-2.0.3 /usr/local/mongodb

# useradd mongodb

# mkdir -p /data/mongodb/shard1

# mkdir -p /data/mongodb/shard2

# mkdir -p /data/mongodb/shard3

# mkdir -p /data/mongodb/config1

配置shard1的replica set

192.168.198.131

# cd /usr/local/mongodb/bin

# ./mongod –shardsvr –replSet shard1 –port 10001 –dbpath /data/mongodb/shard1 –oplogSize 100 –logpath /data/mongodb/shard1/shard1.log –logappend –fork

192.168.198.129

# ./mongod –shardsvr –replSet shard1 –port 10001 –dbpath /data/mongodb/shard1 –oplogSize 100 –logpath /data/mongodb/shard1/shard1.log –logappend –fork

192.168.198.132

# ./mongod –shardsvr –replSet shard1 –port 10001 –dbpath /data/mongodb/shard1 –oplogSize 100 –logpath /data/mongodb/shard1/shard1.log –logappend –fork

连接到192.168.198.131

# ./mongo –port 10001

> config={_id:”shard1″,members:[

… {_id:0,host:”192.168.198.131:10001″},

… {_id:1,host:”192.168.198.129:10001″},

… {_id:2,host:”192.168.198.132:10001″}]

… }

> rs.initiate(config)

{

“info” : “Config now saved locally. Should come online in about a minute.”,

“ok” : 1

}

PRIMARY> rs.status()

{

“set” : “shard1″,

“date” : ISODate(“2012-03-02T02:37:55Z”),

“myState” : 1,

“members” : [

{

“_id” : 0,

“name” : “192.168.198.131:10001”,

“health” : 1,

“state” : 1,

“stateStr” : “PRIMARY”,

“optime” : {

“t” : 1330655827000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T02:37:07Z”),

“self” : true

},

{

“_id” : 1,

“name” : “192.168.198.129:10001”,

“health” : 1,

“state” : 2,

“stateStr” : “SECONDARY”,

“uptime” : 36,

“optime” : {

“t” : 1330655827000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T02:37:07Z”),

“lastHeartbeat” : ISODate(“2012-03-02T02:37:53Z”),

“pingMs” : 0

},

{

“_id” : 2,

“name” : “192.168.198.132:10001”,

“health” : 1,

“state” : 2,

“stateStr” : “SECONDARY”,

“uptime” : 36,

“optime” : {

“t” : 1330655827000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T02:37:07Z”),

“lastHeartbeat” : ISODate(“2012-03-02T02:37:53Z”),

“pingMs” : 466553

}

],

“ok” : 1

}

配置shard2的replica set

192.168.198.129

# ./mongod –shardsvr –replSet shard2 –port 10002 –dbpath /data/mongodb/shard2 –oplogSize 100 –logpath /data/mongodb/shard2/shard2.log –logappend –fork

192.168.198.131

# ./mongod –shardsvr –replSet shard2 –port 10002 –dbpath /data/mongodb/shard2 –oplogSize 100 –logpath /data/mongodb/shard2/shard2.log –logappend –fork

192.168.198.132

# ./mongod –shardsvr –replSet shard2 –port 10002 –dbpath /data/mongodb/shard2 –oplogSize 100 –logpath /data/mongodb/shard2/shard2.log –logappend –fork

连接到192.168.198.129

# ./mongo –port 10002

> config={_id:”shard2″,members:[

… {_id:0,host:”192.168.198.129:10002″},

… {_id:1,host:”192.168.198.131:10002″},

… {_id:2,host:”192.168.198.132:10002″}]

… }

{

“_id” : “shard2″,

“members” : [

{

“_id” : 0,

“host” : “192.168.198.129:10002”

},

{

“_id” : 1,

“host” : “192.168.198.131:10002”

},

{

“_id” : 2,

“host” : “192.168.198.132:10002”

}

]

}

> rs.initiate(config)

{

“info” : “Config now saved locally. Should come online in about a minute.”,

“ok” : 1

}

> rs.status()

{

“set” : “shard2″,

“date” : ISODate(“2012-03-02T02:53:17Z”),

“myState” : 1,

“members” : [

{

“_id” : 0,

“name” : “192.168.198.129:10002”,

“health” : 1,

“state” : 1,

“stateStr” : “PRIMARY”,

“optime” : {

“t” : 1330656717000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T02:51:57Z”),

“self” : true

},

{

“_id” : 1,

“name” : “192.168.198.131:10002”,

“health” : 1,

“state” : 2,

“stateStr” : “SECONDARY”,

“uptime” : 73,

“optime” : {

“t” : 1330656717000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T02:51:57Z”),

“lastHeartbeat” : ISODate(“2012-03-02T02:53:17Z”),

“pingMs” : 1

},

{

“_id” : 2,

“name” : “192.168.198.132:10002”,

“health” : 1,

“state” : 2,

“stateStr” : “SECONDARY”,

“uptime” : 73,

“optime” : {

“t” : 1330656717000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T02:51:57Z”),

“lastHeartbeat” : ISODate(“2012-03-02T02:53:17Z”),

“pingMs” : 209906

}

],

“ok” : 1

}

配置shard3的replica set

192.168.198.132

# ./mongod –shardsvr –replSet shard3 –port 10003 –dbpath /data/mongodb/shard3 –oplogSize 100 –logpath /data/mongodb/shard3/shard3.log –logappend –fork

192.168.198.129

# ./mongod –shardsvr –replSet shard3 –port 10003 –dbpath /data/mongodb/shard3 –oplogSize 100 –logpath /data/mongodb/shard3/shard3.log –logappend –fork

192.168.198.131

# ./mongod –shardsvr –replSet shard3 –port 10003 –dbpath /data/mongodb/shard3 –oplogSize 100 –logpath /data/mongodb/shard3/shard3.log –logappend –fork

连接到192.168.198.132

# ./mongo –port 10003

> config={_id:”shard3″,members:[

… {_id:0,host:”192.168.198.132:10003″},

… {_id:1,host:”192.168.198.131:10003″},

… {_id:2,host:”192.168.198.129:10003″}]

… }

{

“_id” : “shard3″,

“members” : [

{

“_id” : 0,

“host” : “192.168.198.132:10003”

},

{

“_id” : 1,

“host” : “192.168.198.131:10003”

},

{

“_id” : 2,

“host” : “192.168.198.129:10003”

}

]

}

> rs.initiate(config)

{

“info” : “Config now saved locally. Should come online in about a minute.”,

“ok” : 1

}

> rs.status()

{

“set” : “shard3″,

“date” : ISODate(“2012-03-02T03:04:52Z”),

“myState” : 1,

“members” : [

{

“_id” : 0,

“name” : “192.168.198.132:10003”,

“health” : 1,

“state” : 1,

“stateStr” : “PRIMARY”,

“optime” : {

“t” : 1330657451000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T03:04:11Z”),

“self” : true

},

{

“_id” : 1,

“name” : “192.168.198.131:10003”,

“health” : 1,

“state” : 2,

“stateStr” : “SECONDARY”,

“uptime” : 39,

“optime” : {

“t” : 1330657451000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T03:04:11Z”),

“lastHeartbeat” : ISODate(“2012-03-02T03:04:52Z”),

“pingMs” : 0

},

{

“_id” : 2,

“name” : “192.168.198.129:10003”,

“health” : 1,

“state” : 2,

“stateStr” : “SECONDARY”,

“uptime” : 39,

“optime” : {

“t” : 1330657451000,

“i” : 1

},

“optimeDate” : ISODate(“2012-03-02T03:04:11Z”),

“lastHeartbeat” : ISODate(“2012-03-02T03:04:52Z”),

“pingMs” : 0

}

],

“ok” : 1

}

配置config

192.168.198.131

# ./mongod –configsvr –dbpath /data/mongodb/config1 –port 20000 –logpath /data/mongodb/config1/config1.log –logappend –fork

192.168.198.129

# ./mongod –configsvr –dbpath /data/mongodb/config1 –port 20000 –logpath /data/mongodb/config1/config1.log –logappend –fork

192.168.198.132

# ./mongod –configsvr –dbpath /data/mongodb/config1 –port 20000 –logpath /data/mongodb/config1/config1.log –logappend –fork

配置mongos

# ./mongos –configdb 192.168.198.131:20000,192.168.198.129:20000,192.168.198.132:20000 –port 27017 –chunkSize 1 –logpath /data/mongodb/mongos.log –logappend –fork

配置shard cluster

# ./mongo –port 27017

mongos> use admin

switched to db admin

加入shards

mongos> db.runCommand({addshard:”shard1/192.168.198.131:10001,192.168.198.129:10001,192.168.198.132:10001″});

{ “shardAdded” : “shard1”, “ok” : 1 }

mongos> db.runCommand({addshard:”shard2/192.168.198.131:10002,192.168.198.129:10002,192.168.198.132:10002″});

{ “shardAdded” : “shard2”, “ok” : 1 }

mongos> db.runCommand({addshard:”shard3/192.168.198.131:10003,192.168.198.129:10003,192.168.198.132:10003″});

{ “shardAdded” : “shard3”, “ok” : 1 }

列出shards

mongos> db.runCommand({listshards:1})

{

“shards” : [

{

“_id” : “shard1”,

“host” : “shard1/192.168.198.129:10001,192.168.198.131:10001,192.168.198.132:10001”

},

{

“_id” : “shard2”,

“host” : “shard2/192.168.198.129:10002,192.168.198.131:10002,192.168.198.132:10002”

},

{

“_id” : “shard3”,

“host” : “shard3/192.168.198.129:10003,192.168.198.131:10003,192.168.198.132:10003”

}

],

“ok” : 1

}

激活数据库分片

mongos> db.runCommand({enablesharding:”test”});

{ “ok” : 1 }

通过以上命令，可以将数据库test跨shard，如果不执行，数据库只会存放在一个shard，一旦激活数据库分片，数据库中的不同的collection将被存放在不同的shard上，但一个collection仍旧存放在同一个shard上，要使collection也分片，需对collection做些其他操作。

collection分片

mongos> db.runCommand({shardcollection:”test.data”,key:{_id:1}})

{ “collectionsharded” : “test.data”, “ok” : 1 }

分片的collection只能有一个在分片key上的唯一索引，其他唯一索引不被允许。

查看shard信息

mongos> printShardingStatus()

— Sharding Status —

sharding version: { “_id” : 1, “version” : 3 }

shards:

{ “_id” : “shard1”, “host” : “shard1/192.168.198.129:10001,192.168.198.131:10001,192.168.198.132:10001” }

{ “_id” : “shard2”, “host” : “shard2/192.168.198.129:10002,192.168.198.131:10002,192.168.198.132:10002” }

{ “_id” : “shard3”, “host” : “shard3/192.168.198.129:10003,192.168.198.131:10003,192.168.198.132:10003” }

databases:

{ “_id” : “admin”, “partitioned” : false, “primary” : “config” }

{ “_id” : “test”, “partitioned” : true, “primary” : “shard1” }

test.data chunks:

shard1 1

{ “_id” : { $minKey : 1 } } –>> { “_id” : { $maxKey : 1 } } on : shard1 { “t” : 1000, “i” : 0 }

mongos> use test

switched to db test

mongos> db.data.stats()

{

“sharded” : true,

“flags” : 1,

“ns” : “test.data”,

“count” : 0,

“numExtents” : 1,

“size” : 0,

“storageSize” : 8192,

“totalIndexSize” : 8176,

“indexSizes” : {

“_id_” : 8176

},

“avgObjSize” : 0,

“nindexes” : 1,

“nchunks” : 1,

“shards” : {

“shard1” : {

“ns” : “test.data”,

“count” : 0,

“size” : 0,

“storageSize” : 8192,

“numExtents” : 1,

“nindexes” : 1,

“lastExtentSize” : 8192,

“paddingFactor” : 1,

“flags” : 1,

“totalIndexSize” : 8176,

“indexSizes” : {

“_id_” : 8176

},

“ok” : 1

}

},

“ok” : 1

}

测试：插入大量数据

mongos> for (var i=1;i<=500000;i++) db.data.save ({_id:i,value:”www.strongd.net”})

mongos> printShardingStatus()

— Sharding Status —

sharding version: { “_id” : 1, “version” : 3 }

shards:

{ “_id” : “shard1”, “host” : “shard1/192.168.198.129:10001,192.168.198.131:10001,192.168.198.132:10001” }

{ “_id” : “shard2”, “host” : “shard2/192.168.198.129:10002,192.168.198.131:10002,192.168.198.132:10002” }

{ “_id” : “shard3”, “host” : “shard3/192.168.198.129:10003,192.168.198.131:10003,192.168.198.132:10003” }

databases:

{ “_id” : “admin”, “partitioned” : false, “primary” : “config” }

{ “_id” : “test”, “partitioned” : true, “primary” : “shard1” }

test.data chunks:

shard1 6

shard2 5

shard3 11

too many chunks to print, use verbose if you want to force print

mongos> db.data.stats()

{

“sharded” : true,

“flags” : 1,

“ns” : “test.data”,

“count” : 500000,

“numExtents” : 19,

“size” : 22000084,

“storageSize” : 43614208,

“totalIndexSize” : 14062720,

“indexSizes” : {

“_id_” : 14062720

},

“avgObjSize” : 44.000168,

“nindexes” : 1,

“nchunks” : 22,

“shards” : {

“shard1” : {

“ns” : “test.data”,

“count” : 112982,

“size” : 4971232,

“avgObjSize” : 44.00021242321786,

“storageSize” : 11182080,

“numExtents” : 6,

“nindexes” : 1,

“lastExtentSize” : 8388608,

“paddingFactor” : 1,

“flags” : 1,

“totalIndexSize” : 3172288,

“indexSizes” : {

“_id_” : 3172288

},

“ok” : 1

},

“shard2” : {

“ns” : “test.data”,

“count” : 124978,

“size” : 5499056,

“avgObjSize” : 44.00019203379795,

“storageSize” : 11182080,

“numExtents” : 6,

“nindexes” : 1,

“lastExtentSize” : 8388608,

“paddingFactor” : 1,

“flags” : 1,

“totalIndexSize” : 3499328,

“indexSizes” : {

“_id_” : 3499328

},

“ok” : 1

},

“shard3” : {

“ns” : “test.data”,

“count” : 262040,

“size” : 11529796,

“avgObjSize” : 44.000137383605555,

“storageSize” : 21250048,

“numExtents” : 7,

“nindexes” : 1,

“lastExtentSize” : 10067968,

“paddingFactor” : 1,

“flags” : 1,

“totalIndexSize” : 7391104,

“indexSizes” : {

“_id_” : 7391104

},

“ok” : 1

}

},

“ok” : 1

}

mongodb的NUMA问题

mongodb日志显示如下:

WARNING: You are running on a NUMA machine.

We suggest launching mongod like this to avoid performance problems:

numactl –interleave=all mongod [other options]

解决方案：

1.在原启动命令前面加numactl –interleave=all

如# numactl –interleave=all ${MONGODB_HOME}/bin/mongod –config conf/mongodb.conf

2.修改内核参数

echo 0 > /proc/sys/vm/zone_reclaim_mode

http://www.mongodb.org/display/DOCS/NUMA

下面注释转自网络

一、NUMA和SMP

NUMA和SMP是两种CPU相关的硬件架构。在SMP架构里面，所有的CPU争用一个总线来访问所有内存，优点是资源共享，而缺点是总线争用激烈。随着PC服务器上的CPU数量变多（不仅仅是CPU核数），总线争用的弊端慢慢越来越明显，于是Intel在Nehalem CPU上推出了NUMA架构，而AMD也推出了基于相同架构的Opteron CPU。

NUMA最大的特点是引入了node和distance的概念。对于CPU和内存这两种最宝贵的硬件资源，NUMA用近乎严格的方式划分了所属的资源组（node），而每个资源组内的CPU和内存是几乎相等。资源组的数量取决于物理CPU的个数（现有的PC server大多数有两个物理CPU，每个CPU有4个核）；distance这个概念是用来定义各个node之间调用资源的开销，为资源调度优化算法提供数据支持。

二、NUMA相关的策略

1、每个进程（或线程）都会从父进程继承NUMA策略，并分配有一个优先node。如果NUMA策略允许的话，进程可以调用其他node上的资源。

2、NUMA的CPU分配策略有cpunodebind、physcpubind。cpunodebind规定进程运行在某几个node之上，而physcpubind可以更加精细地规定运行在哪些核上。

3、NUMA的内存分配策略有localalloc、preferred、membind、interleave。localalloc规定进程从当前node上请求分配内存；而preferred比较宽松地指定了一个推荐的node来获取内存，如果被推荐的node上没有足够内存，进程可以尝试别的node。membind可以指定若干个node，进程只能从这些指定的node上请求分配内存。interleave规定进程从指定的若干个node上以RR算法交织地请求分配内存。

三、NUMA和swap的关系

可能大家已经发现了，NUMA的内存分配策略对于进程（或线程）之间来说，并不是公平的。在现有的Redhat Linux中，localalloc是默认的NUMA内存分配策略，这个配置选项导致资源独占程序很容易将某个node的内存用尽。而当某个node的内存耗尽时，Linux又刚好将这个node分配给了某个需要消耗大量内存的进程（或线程），swap就妥妥地产生了。尽管此时还有很多page cache可以释放，甚至还有很多的free内存。

四、解决swap问题

虽然NUMA的原理相对复杂，实际上解决swap却很简单：只要在启动MySQL之前使用numactl –interleave来修改NUMA策略即可。

值得注意的是，numactl这个命令不仅仅可以调整NUMA策略，也可以用来查看当前各个node的资源是用情况，是一个很值得研究的命令。