简言之,对象大小小于6kB, hit rate能达到10000/s以上。 对象大小大于40kB,也能达到Gbps.
在生产环境中,当非常昂贵的,高端的硬件负载均衡设备突然在7层处理上失效, HAProxy已经多次作为一种紧急解决方案被安装。
硬件负载均衡设备在包级别处理请求,而且很难支持跨多个包的请求和响应太快的请求。因为它们根本没有缓冲(buffering)。另外,
软件负载均衡器使用TCP buffering并且长请求和快响应对它无所谓。A
nice side effect of HTTP buffering is that it
increases the server's connection acceptance by reducing the
session duration, which leaves room for new requests. New
benchmarks will be executed soon, and results will be
published. Depending on the hardware, expected rates are in the order of a few
tens of thousands of new connections/s with tens of thousands of simultaneous
connections.
有3个因素用来衡量负载均衡器的性能:
- session rate
这是个很用重要的因素,因为它直接决定了负载均衡器什么时候开始不能够转发它接收到的所有请求。
这很大程度上依赖于CPU。
你可能经常听到requests/s或者hits/s,在HTTP1.0或者关闭keepalive的HTTP/1.1中,
它们和sessions/s其实一样。启用了keep-alive的requests/s值没有任何意义,因为通常为了减轻服务器压力,
会关闭keep-alive。这个因素通过各种不同的对象大小(Object size)进行衡量,最快的是空(empty)对象(比如:
HTTP 302, 304或404返回码)。在Dual Opteron系统,比如patch过的Linux2.4运行在HP-DL145上,
session rate能达到超过20000sessions/s。甚至最廉价的Sun's X2100-M2
在Dual-core 1.8 GHz 配置下也能达到25000sessions/s。
- session并发
This factor is tied to the previous one. Generally, the session rate
will drop when the number of concurrent sessions increases (except the
epoll polling mechanism). The slower the servers, the higher
the number of concurrent sessions for a same session rate. If a load balancer
receives 10000 sessions per second and the servers respond in 100 ms, then the
load balancer will have 1000 concurrent sessions. This number is limited by the
amount of memory and the amount of file-descriptors the system can
handle. With 8 kB buffers, HAProxy will need about 16 kB per session, which
results in around 60000 sessions per GB of RAM. In practise, socket
buffers in the system also need some memory and 40000 sessions per GB of RAM is
more reasonable. Layer 4 load balancers generally announce millions of
simultaneous sessions because they don't process any data so they don't need
any buffer. Moreover, they are sometimes designed to be used in Direct Server
Return mode, in which the load balancer only sees forward traffic, and which
forces it to keep the sessions for a long time after their end to avoid cutting
sessions before they are closed.
- The data rate
This factor generally is at the opposite of the session rate. It is measured
in Megabytes/s (MB/s), or sometimes in Megabits/s (Mbps). Highest data rates
are achieved with large objects to minimise the overhead caused by session
setup and teardown. Large objects generally increase session concurrency, and
high session concurrency with high data rate requires large amounts of memory
to support large windows. High data rates burn a lot of CPU and bus cycles on
software load balancers because the data has to be copied from the input
interface to memory and then back to the output device. Hardware load balancers
tend to directly switch packets from input port to output port for higher data
rate, but cannot process them and sometimes fail to touch a header or a cookie.
For reference, the Dual Opteron systems described above can saturate 2
Gigabit Ethernet links on large objects, and I know people who constantly
run between 2 and 3 Gbps of real traffic on 10-Gig NICs plugged into quad-core
servers.
A load balancer's performance related to these factors is generally announced for
the best case (eg: empty objects for session rate, large objects for data rate).
This is not because of lack of honnesty from the vendors, but because it is not
possible to tell exactly how it will behave in every combination. So when those 3
limits are known, the customer should be aware that he will generally be below
all of them. A good rule of thumb on software load balancers is to consider an
average practical performance of half of maximal session and data rates for
average sized objects.
你或许会对10-Gigabit/s页面有兴趣.
Being obsessed with reliability, I tried to do my best to ensure a total
continuity of service by design. It's more difficult to design something
reliable from the ground up in the short term, but in the long term it reveals
easier to maintain than broken code which tries to hide its own bugs behind
respawning processes and tricks like this.
In single-process programs, you have no right to fail : the smallest bug
will either crash your program, make it spin like mad or freeze. There has not
been any such bug found in the code nor in production for the last 6 years.
HAProxy has been installed on Linux 2.4 systems serving millions of pages
every day,
and which have only known one reboot in 3 years for a complete OS upgrade.
Obsiouvly, they were not directly exposed to the Internet because they did not receive any
patch at all. The kernel was a heavily patched 2.4 with Robert Love's
jiffies64 patches to support time wrap-around at 497 days (which
happened twice). On such systems, the software cannot fail without being
immediately noticed !
Right now, it's being used in several Fortune 500 companies around the world to
reliably serve millions of pages per day or relay huge amounts of money. Some
people even trust it so much that they use it as the default solution to solve
simple problems (and I often tell them that they do it the dirty way). Such
people still use version 1.1 which sees very limited evolutions and which targets
mission-critical usages. It is really suited for such environments because the
indicators it returns provide a lot of valuable information about the application's
health, behaviour and defects, which are used to make it even more reliable. Version 1.2 has not yet received as much testing as 1.1,
but should be considered for high volumes of traffic or to benefit from newest
features.
As previously explained, most of the work is executed by the Operating System.
For this reason, a large part of the reliability involves the OS itself. Recent
versions of Linux 2.4 offer a high level of stability. However, it requires a
bunch of patches to achieve a high level of performance. Linux 2.6
includes the features needed to achieve this level of performance, but is not
yet stable enough for such usages. The kernel needs at least one upgrade every
month to fix a bug or vulnerability. Some people prefer to run it on Solaris (or
do not have the choice). Solaris 8 and 9 are known to be really stable right now,
offering a level of performance comparable to Linux 2.4. Solaris 10 might show
performances closer to Linux 2.6, but with the same code stability problem. I
have too few reports from FreeBSD users, but it should be close to Linux 2.4 in
terms of performance and reliability. OpenBSD sometimes shows socket allocation
failures due to sockets staying in FIN_WAIT2 state when client suddenly
disappears. Also, I've noticed that hot reconfiguration does not work under
OpenBSD.
当系统达到它的极限时,可靠性会显著下降。这就是为什么正确的调整sysctls
如此重要。没有通用的规则,视系统和应用程序而定。但是要永远避免内存耗尽和使用swap。正确的调整应该能够使系统在大的压力下运行几年而不会变慢或者崩溃。
在部署一个免费的负载均衡器时,安全是一个重要的考虑因素。加固操作系统、限制开放的端口以及可
访问的服务,这些都是可能的,但是负载均衡器本身却暴露在外。因此,我特别小心编程风格。唯一被
发现的漏洞距今已经6年了而且仅仅存活了一周。 它是由于日志被修改而导致。它会出现BUS ERROS
而导致进程崩溃, 但并不意味着可以执行代码: 仅有3字节的溢出,太短而不能放下甚至一个指针
(并且后面还有一个变量)。
不管怎么样,在编写头部处理的代码时总是特别小心。多种不可能的状态会被检查并且返回,会话从创建
到消亡产生的错误都会被处理。一些人已经分析了源代码,并且建议清理这些日志以利于审计。但是我没有接收可能引起可疑操作的补丁。
我建议使用root启动HAProxy因为之后它可以进行chroot并且在启动实例前丢掉它所有的这些权限
如果不是以root启动是无法完成的,因为只有root可以执行chroot().
日志提供了足够的信息来维护一个令人满意的安全级别。它们只能通过UDP发送,因为一旦chroot了,
/dev/log UNIX socket是不可读的,并且写如一个文件也是不可能的。下列信息更为有用:
- 请求者源IP和端口可以在防火墙日志里面看到请求者的来源地。
- 会话建立时间和防火墙日志是匹配的, 而会话拆除时间通常匹配代理的时间;
- 正确的请求编码保证请求者不能隐藏任何的不可打印字符也不能欺骗终端 。
- 特定的请求和回应包头和cookie抓取帮助检测扫描攻击、代理以及受感染的机器。
- timers help to differentiate hand-typed requests from browsers's.
HAProxy 也提供基于正则表达式的header控制。部分请求、或者请求、回应的header可以被denied, allowed, removed, rewritten, or
added。这重用来阻止危险的请求或者编码(encoding)(如: Apache Chunk exploit),
而且可以防止从服务器向客户端意外的泄露信息 。其他功能比如Cache-control检查确定敏感信息不会被上层的proxy意外的缓存住。
源代码遵循GPL v2. Linux/x86和Solaris/Sparc可用的源代码和预编译二进制包可以在这里下载:
- 开发版(1.3 - 内容交换) :
- 浏览目录(包括文档、源码和二进制)
- 每日快照 当GIT库有修改时,会构建一个日快照
- Latest version (1.2) :
- Previous branch (1.1) :
- Stunnel的X-Forwarded-For支持
Stunnel 目前是HAProxy在客户端SSL支持上最完美的补充. 然而,
因为Stunnel是一个代理(proxy), 它不懂HTTP, 所以客户端IP地址无法获取,这是很讨厌的. 网络上目前可以找到一些补丁
来加入一个X-Forwarded-For头, 但是它们常导致不可预见的缓冲区溢出. 所以我写了一个安全可靠的补丁来补充这个非常有用的特性.
我把他发给Stunnel的作者,但是很遗憾没收到任何反馈。 所以这个补丁在这里提供, 供Stunnel-4.14和4.15,希望对大家有用.
- 各种补丁( :
- Logo :
如果你很乐意使用HAProxy并想在你的站点上加入一个HAProxy的链接,
可以简单的复制下面的HTML代码到你站点的你认为合适的地方, 这会让你的访问者看到HAProxy的Logo::
- Browsable directory
目前有3种文档:参考手册: 说明怎么配置HAProxy,架构指南说明各种类型的代理配置,以及
配置手册具有比参考手册更详尽的配置说明。
- 版本1.3参考手册,并不实时更新(开发版) :
- 1.2参考手册 (稳定版) :
- 1.1 参考手册(老的稳定版) :
architecture.txt : 架构指南 (英文版)
关于负载均衡的文章 (HTML版) : 值得那些不知道该选择什么负载均衡器的人阅读(英文)
如果你觉得你没有时间和能力来配置和维护一个免费的负载均衡器,或者如果你正寻找商业支持以满足你的客户或老板,你可以联系
EXOSEC. 另外一个解决方案是使用Exceliance的 ALOHA 设备 (如下).
以下产品或项目使用HAProxy :
- redWall防火墙
来源此站: "redWall是一个可启动的CD-ROM防火墙.它的目标是提供一种功能丰富的防火墙解决方案,基于此目标,
它为所有生成的日志文件提供了一个web访问接口!"
- Exceliance的
ALOHA设备
Exceliance是一个法国公司,它销售完全基于haproxy的解决方案,此方案将haproxy集成到一个优化的加固版本的
Formilux,通过功能丰富的Web接口,方便使用,减少了维护量,并且
使用VRRP(box fail-over以及link fail-over)增强了可用性,同时还有配置同步,SSL,透明代理等等。该站点上
有个试用版,可以在VMware上使用。因为那是我工作的公司,所以,很多的功能都在那实现了。:-)
- Loadbalancer.org
此公司在英国,目前将haproxy加入到他们的负载均衡解决方案中以提供部分用户需要的基本的7层支持。
他们也是少数肯使用haproxy并捐献给这个项目的商业公司之一。
很多热心人贡献了代码给我,这些代码有的被采用了。另外有一些人花了很长的时间分析代码,还有一些人维护着ports以保持更新。
- 应用层cookie
Alexander Lazic和Klaus Wagner实现了此功能,并合并到版本1.2中. 此功能使代理能够识别服务器发给客户端的cookie,
并且可以在后续客户端请求的URL中找到此cookie,以将客户端请求转发的对应的服务器上。 识别的cookie闲置一段时间后会自动被清除。
- FreeBSD Port
Cl.ent Laforet 为FreeBSD维护了一个最新的。如果你想得到此port更多的信息,请在此查看Cl.ent的工作:
- Debian/Ubuntu包
Arnaud Cornet 将HAProxy引入
Debian :
- OpenBSD port
Jason Dixon提交了一个haproxy port到OpenBSD -current
- 最少连接数算法
提供给haproxy-1.2.14的patch由Oleksandr Krailo提交. 此patch实现了一个基本的最少连接数算法。基于扩展性的考虑,我还没有合并到1.3版中,但我把它放这提供给那些
想包含进1.2的用户,而且这个补丁真的很不错。
- 服务器慢关闭(Soft Server-Stop)
Alexander Lazic发给我这个1.1.28的补丁,此补丁实际上做了两件事情。
第一个有趣部分允许在一个文件里面列举出哪些服务器需要关闭,然后给代理发一个信号读取此文件,
从而关闭这些服务器。这个功能不会加入到主版本中,因为进程必须访问一个位于文件系统上的文件,
这就有安全隐患,虽然当前版本能运行在chroot的、空的只读目录下。真正需要的是一种给运行中进
程发送命令的方式。当然,我知道有些人也许需要此功能,因此我在这里提供了。
补丁的第二部分已经合并进了。它允许主和备服务器共享一个相同的cookie。这是不言而喻的,但是之前却没有实现。
用法: Alex说仅需要将你需要停止的服务器写入到一个文件中,然后对执行中进程执行kill -USR2。
但是我并没有测试过。
- 服务器权重
S.astien Brize发给我这个1.1.27的补丁,此补丁在'server'参数里加入了'weight'关键字以支持
在快和慢服务器之间实现更合理的负载均衡。这里提供下载以方便那些想加入进1.1版本的用户使用。
我没有作此修改是因为它有一个副作用,那就是使用了过高或者不等的权重,有些服务器会持续收到很多连接。
1.2.12已经实现了另外一种更平滑更公平的负载均衡, 它也支持加权的hash负载均衡。
用法: 在server行指定"weight X"。
注意: 1.2版的Haproxy打了此补丁后的配置仍然可以使用。
- 1.1.27的IPv6
我为1.1.27实现了客户端IPv6,并且合并到1.2版本了。但是,我仍然在这里提供给那些希望在1.1版里面试验IPv6的用户。
- 其他补丁
请 浏览此目录 查看其他有用的贡献。
如果你根本不需要HAProxy的任何功能而想寻找一种更简单的解决方案,你也许可以在此找到你所需要的:
-
Linux Virtual Servers (LVS)
整合进Linux 2.4和2.6内核的非常快速的3/4层负载均衡. 需要结合Keepalived
使用以实现对服务器的健康检查. 这通常是大多数基于IP的负载均衡器默认的解决方案.
-
Pure 负载均衡器(PLB)
作者用了和HAProxy一样的事件驱动模型(但它是基于libevent的).
有趣的是, 它有和使用其他模型一样的限制.
然而, 他的目的仅仅是为了实现高性能和可用性, 并没有任何HTTP处理和会话保持.
-
Pound
Pound可以视为HAProxy的一个补充. 它支持SSL, 可以根据请求的URL进行转发. 它的代码很小,并且为了方便审计, 会一直保持.
它的配置文件也很小. 但是, 它不支持会话保持, 并且它的性能以及多线程模型限制了它仅适用于中型站点.
-
Pen
Pen是一个很简单的TCP负载均衡器. 它支持高达2048个客户端的基于源IP地址的会话保持。 支持基于IP的ACL.
使用select()并且比Pound支持更大的吞吐量, 但是当并发连接达到几千后很难再扩展.
有任何问题或者建议请随时联系我:
经常有人问我是否可以捐赠,所以我(作者Willy TARREAU)专门建立了一个paypal账号。如果你想捐赠.点击这里 。
FreeNode开了一个关于HAProxy的IRC频道(但在那找不到我,我不去那) :