running internet systems in china

Running Internet Systems in China

Details you need to succeed in Chinese Internet Operations

White Paper

Running the World's Internet Servers

February, 2013Table of ContentsOverview.......................................................................................................................................................................... 3

Challenges.................................................................................................................................................................. 3ChinaNetCloud......................................................................................................................................................... 3

Country Profile.............................................................................................................................................................. 4Internet size.............................................................................................................................................................. 4Mobile & Tablet Growth....................................................................................................................................... 4Demographics.......................................................................................................................................................... 5Cities by Tier............................................................................................................................................................. 6

Infrastructure & Challenges..................................................................................................................................... 7ISP/Carriers.............................................................................................................................................................. 7Not one company.................................................................................................................................................... 7Data Centers.............................................................................................................................................................. 8IDC Connectivity...................................................................................................................................................... 9IDC Service Quality.............................................................................................................................................. 11CDNs.......................................................................................................................................................................... 11Hardware................................................................................................................................................................. 11

Clouds ........................................................................................................................................................................... 12Hosting Outside China........................................................................................................................................ 13Website Monitoring............................................................................................................................................. 13Security Issues....................................................................................................................................................... 14Hackers..................................................................................................................................................................... 14DDoS.......................................................................................................................................................................... 14IDC Local Attacks.................................................................................................................................................. 15Physical Attacks.................................................................................................................................................... 15Regulatory............................................................................................................................................................... 16Licensing / ICP...................................................................................................................................................... 16

Website Development Services............................................................................................................................ 18About ChinaNetCloud............................................................................................................................................... 19

ChinaNetCloud - Running Internet Servers in China [DRAFT]


Overview China is today home to the world's largest Internet user population, over 500 million, mostly young, urban, and with discretionary income to spend. They have iPhones, play games, write microblogs, and connect with friends from all over. As a result, they are the very face of the future of the Internet, in China and beyond.

As a result, more and more large-scale Internet sites, games, apps and services are springing up to serve the needs of the people. While strongly local in nature, there are a large number of Western and Global brands with a strong presence, in e-commerce, gaming, mobile, and more. And this number is increasingly quickly as both local and global companies race to take advantage of the growing market.

Challenges

As exciting as China's Internet scene and opportunities are, it's not quite so simple to build and operate Internet operations here. There are a wide variety of regulatory, legal, infrastructure, connectivity, cultural, language, and other problems that are not common outside of China. In addition, there are many misconceptions about the very nature of how the Internet works and how it is managed or regulated in today's PRC. All of these challenges and often do trip up or complicate otherwise well-planned projects.

ChinaNetCloud

ChinaNetCloud can help. As China's leading Internet Server Management company and a world-class managed service provider, we are experts on all aspects of Chinese Internet operations. We routinely help companies of all nationalities, types, and sizes to design, build, and operate large-scale systems successfully reaching hundreds of millions of Chinese users, every day.This brochure describes the modern Chinese Internet, its structure and challenges, and how to succeed on-line in the world's most populous country.



Country Profile

Internet size

According to China Internet Marketing research data by Nanjing Marketing Group, by the end of December in 2011, There are 513 million Internet users in China, 56 million more than the previous year. That's about the same as the total population of USA, Canada, Australia, Germany and the UK. There are 136 million rural Chinese Internet users.

Mobile & Tablet Growth

China's mobile and tablet growth is mirroring the west, with Apple's iPhone and iPad leading the way with millions of sales each. In addition, Android is extremely popular in China, especially with the introduction of low-cost tablets and smartphones, putting world-class power and applications in the hands of nearly everyone in the coming years.



Demographics

According to Nanjing Marketing Group, there are 356 million mobile Internet users in China. 69% of China's Internet users use mobile devices to access the Internet.

27.3% of Chinese Internet users are under the age of 20, 30.8% are between 20 to 29 years old, 23.2% are between 30 to 39 years old, 11.4% are between 40 to 49 years old, and 4.8% are above 50 years old.



Cities by Tier

One important and unique feature of China is the very large number of large cities, with various statistics placing over 150 cities over 1 million people. These cities are often described in 'Tier's such as Tier 1, 2, 3, and 4 with some common lists such as Tier 1 cities including Beijing, Shanghai, and Guangzhou while Tier 2 includes Hangzhou, Nanjing, Xiamen, Qingdao, Chengdu, Dalian, and so on. Understanding where your users or customers are is very important to your connectivity strategy in China, including if you include students, rural or western region populations, etc.



Infrastructure & Challenges

China has an interesting Internet architecture and infrastructure situation. In most ways, it's just like the U.S. or any other large countries or regions. But in other ways, it's quite different with some surprising challenges that can trip up the newcomer, local or foreign. These challenges are especially a problem for the highest tier of system, such as luxury e-commerce or complex national MMORPG, since achieving premier performance at low, or even any, price is an ongoing challenge.

ISP/Carriers

The first and the most obvious challenge which even foreigners are often aware of is the so-called 'north-south' split. Basically, most areas of China have only one single dominant ISP or carrier for all local Internet access.

In the north, including Beijing, Tianjin, and the important provinces of Shandong, Shaanxi, plus the whole northeast, there is China Netcom (merged with China Unicom). In the southern areas, including Shanghai, Guangzhou, Nanjing and over 50% of the country's population, there is China Telecom.

Other ISPs do exist, such as Railcom and others, though they generally handle the western and rural areas, so reaching them effectively is only interesting to specialized brands or systems such as farm information, migrant worker recruiting, and some types of video or gaming.

In addition, the university network, called CERNET, is separated and actually quite hard to get good service to without special provisions. College students are of course a valuable demographic, so efficiently reaching this separate network is a priority for some brands and site operators. However, this is not easy and it takes special attention using the best-connected data centers and CDNs.

To further complicate things, China has the world's largest mobile Internet population, but one that is also separated and difficult to reach. Most people are on China Mobile, but most iPhone users are on China Unicom, while Telecom is making inroads in some areas. Historically these mobile carriers were totally separated and throttled bandwidth, though recent mergers and re-organizations have complicated this picture.

Not one company

Complicating matters is that even in a given region, such as the north or south, the Carrier is not a single entity, but is made up of a large company for each province and often each city. For example, in China Telecom's area in the south, the properous province of Zhijiang has a China Telecom carrier company, but the lovely capital city of Hangzhou has its own, separate, China Telecom company. And even though these companies really don't directly compete for business, they don't always get along nor cooperate all that well.



Data Centers

All Internet servers are housed in data centers, called IDCs in China (for Internet Data Center). These come in all sizes and types and are extraordinary diverse, from very small few hundred square meters to what would be considered a small-to-medium 10,000-square-meter IDC in the West.

Data Centers are generally very quite small and severely power-limited. In fact, the available power feeds is almost always the determining factor in IDC size. Also many IDCs are quite urban, often housed in part or all of a floor of an old office building, though many newer green-friend facilities are coming on-line, so actual data center quality and environment vary widely in size, quality, and price.

A key result of power limits is that per rack power is severely limited in China, which in turn limits the number of servers per rack. Typical power is about 2,000 watts per rack, and 1,800-2,500 is by far the most common. This limits racks to about 12-15 servers or 1U devices such as switches, which is far less than it is seen in the west.

Thus racks and data centers often appear empty or under-utilized but are in fact fully-loaded. This influences the IDC costs, since system must be spread out over more racks than many companies expect, raising hosting costs.

In addition, the small size and limited power of IDCs severely limit expansion for any given customer. It's very common for popular IDCs to be fully booked and adding an extra rack ranges from not possible, to adding on another floor (and LAN), or having to move data centers.

Blades and other high-density data center technologies are simply not seen in China. Even large storage or networking equipment such as 4U SANs or Cisco 6500 switches are often limited to one per rack.

The vast majority of IDCs are owned by the carriers, especially China Telecom. However, you can't easily rent space directly from them as these are mostly wholesale operations. So the carriers lease large blocks of space to smaller IDC companies, from the large 21ViaNet and ChinaNetCenter to very small fly-by-night one-man organizations.

Private IDCs are quite rare, and most 'IDC Companies' are in fact re-selling space in carrier IDCs, though many have their own connectivity. This is a complex regulatory and legal area with various dynamic requirements, so the future structure is still not clear, though the top few companies are doing relatively well.

Note that enterprise IDCs are quite different and have very different characteristics. In this area, foreign and Hong Kong data center companies like AT&T, NTT, and others are quite active with high-end services focused on MNCs.



IDC Connectivity

Generally an IDC's most important asset is its connectivity to the Internet, and this varies widely in quality and price. It is critically important to understand where and how the IDC is connected, including to what carriers and via what type of routing.

The simplest and cheapest option is a single line IDC, which is connected to the local carrier, such as Shanghai Telecom in Shanghai. This can work well, but users in other parts of China will generally have poor performance, for example Beijing users will not enjoy a site connected only in Shanghai. The next level up is called dual-line, where the IDC has lines to their local carrier plus a purchased line to the 'other' carrier. So if the IDC is in the south with Telecom, it will also have a 'north' line to Netcom. This is most common in the large city of Tianjin which is popular as it seems to have good southern Telecom connections in addition to its local northern Netcom.

There is a catch, however. Dual-line systems have two IPs, as they are not using the BGP protocol for routing, as would be common in the rest of the world. BGP is not common in China, though this is changing, see below. This dual-IP method, which disappeared from the West 10-20 years ago, means that each public device such as web server or load balancer, must have two (or more) IPs and properly handle the default routes. This is complex even on Linux systems, and not even supported on some hardware systems. Plus, geo-DNS is needed to correctly send users to the right IP address for their region and carrier. This type of system was very common until recently and is gradually falling out of favor as China slowly moves to BGP.



BGP is the obvious solution to this, but is still uncommon in China, due to cost and the required inter-carrier cooperation requirements. But this is slowly changing and in 2011-2012 the number of BGP-based systems has greatly increased, though still only at the high end. And some systems advertised as BGP are actually pseudo-BGP with special routing buried in the IDC core routing system to make things look correct.

The best IDCs, such as 21ViaNet and the Aliyun cloud, offer what they call 8 line BGP, which are western-style systems with full BGP to all eight carriers in China, including Telecom, Netcom, Unicom, Mobile, CERNET, Railcom, and the rest. These are the best solution, but are very expensive. So much so that they can only be used for core interactive systems and all static assets, especially video, etc. should be handled in CDNs.

Chinese IDCs have many unusual service restrictions or challenges, such as not allowing 24 hour access or requiring 24 hours notice before entering, almost unheard of in the West. In addition, some IDCs don't have 24 hour staff or the staff quality is quite poor and only reached via mobile phone off-hours.

In addition, some lower-tier IDCs have no patience for difficult or demanding customer and will ask you to leave, especially if you suffer from hacker DDoS attacks, too many legal or other issues, etc.

Note also that a few IDCs do not allow foreigners to enter, as they host sensitive or government systems. This can be a problem for startups or other organizations with non-Chinese engineers or management, but is not a common problem.

Note also that during sensitive periods, such as the 2008 Olympics or 2012 government leadership transition, all movement of server hardware and IDC access is severely restricted. For example for six months in 2008 and about a month in 2012, no servers could move in or out of IDCs, and in most cases, engineers couldn't make any changes to servers, cables, or even touch servers, even if they crashed, failed, or suffered outages. While not common, such restrictions do happen.

Some people worry about power quality and reliability in China, but generally this not a problem as power is reliable as in any U.S. city, and power-related IDC issues are very rare. In addition, IDCs do maintain standard UPS and generator-based alternative in line with their general service and SLA levels. Power issues are not a major concern.

Connectivity issues can be more of a problem, especially for IDCs in heavy construction zones where cable cuts can and do occur, with occasional outages. Proper routing and HA methods can help, but some IDCs have a bad reputation in this area.



IDC Service Quality

Many factors influence an IDC's service quality, but connectivity, 24 hour access, and sufficient remote hands support are mandatory for a good IDC. Added value items such as DDoS protection, dual uplink drops, added power per rack and similar items can make them even more attractive.

CDNs

Given China's infrastructural and geographical challenges, it's not surprising there are several CDN services to help get the data where it needs to go. Foreign companies are mostly prohibited from this type of service (Akamai works through local partners), so several Chinese firms dominate, including ChinaNetCenter, ChinaCache, and FastWeb (recently bought by 21ViaNet).

CDN servers are fairly standard and similar to what is available in the west, though more limited in feature set. All offer image and static object caching, whole-site-acceleration, and various video streaming services. Other common services that are less common in the west include large file downloads with restart (for large game downloads), anti-link stealing, and various media and video download prevention methods.

Generally, all websites in China should utilize one or more CDNs, if only to get the images and static objects out to users, especially for mobile and university end users. Best practices are of course necessary for successful CDN utilization, most importantly good cacheability settings and renaming objects on update.

Hardware

Server, network, and other Internet infrastructure hardware is plentiful in China and very easily obtained from all the global brands. Dell dominates the server marketplace with all products available in a few days from numerous distributors and Dell itself (which makes all the servers in Xiamen, Fujian Province). HP is the clear number two, with IBM and others far less common in the Internet market. Cisco is the leader in networking equipment, though Huawei and others are also common.

As noted previously, power is a major concern, so efficient 1-2U servers are by far the most common, and blade equipment is very rarely seen. SANs are not common in the Internet space, though NetApp Filers and other storage systems are often used in large systems. F5 and Netscaler dominate the high-end load balancer area. All these brands are heavily supported by distribution networks in major cities.

Most hardware is purchased outright, though a few IDCs offer lease arrangements which can of course reduce the impact on small company cash flow. Like all such arrangements, the terms are important, including what happens to the equipment at the end of the lease and how easily the lease can be ended early if things are not going well. In the best cases, you can walk away by returning the equipment and the loss of a modest deposit.



Clouds

Cloud Computing is of course increasingly popular around the globe, though still quite young in China. In particular, top global players such as Amazon and Rackspace do not have systems in China and their international systems are too slow to be useful inside the country.

After a few false starts, there are now a few local players, most importantly Aliyun, the cloud service of e-commerce giant Alibaba. Their goal is to be the Amazon of China and their services are both patterned after, and often compatible with, Amazon's. Their services are still only a subset of those available internationally, but they appear quite serious and are rapidly expanding and improving.

A handful of other useful services are becoming available, usually with standard basic IaaS virtual servers, occasionally with Amazon S3-style cloud storage. Some of these clouds have strong backers but weak sales/marketing skills, while others are small startups but are aggressive and offering interesting services.

Private Clouds are becoming increasingly common, where a number of powerful physical servers are purchased and virtualized into several or dozens of VMs to run load balancers, web servers, memcache, search engines, databases, and more. This is the dominant deployment method for modern large-systems in 2012, though the expertise to design and build these remains in short supply.

The whole cloud industry has grown in 2012, and 2013 will probably be the year that momentum really shifts for new small system development, with a very significant portion heading to Aliyun and other similar firms.



Hosting Outside China

Many global companies want to host their global content or e-commerce systems in only one place, such as Amazon U.S., and to serve the whole world, including China.

Unfortunately, this is not possible with decent performance. China's international links are large and well-connected, but are often overloaded and in the end, congested through a small number of ocean cables to the U.S., Japan, etc.

Performance is often okay, almost tolerable, until the day or week it's not, for no good reason. The trans-border performance and reliability is simply not sufficient for a real business to serve users in China. In addition, due to the regulatory challenges, it's possible to be blocked for a variety of reasons, including because your system is at the same foreign data center as one hosting prohibited content.

As a result, companies that serve Chinese users must have a system inside China. If there are also international users and the data or system can be split, then two systems are needed, one inside the PRC and one outside.

If Chinese domestic and international users must share the same system, such as for trade or other systems, then a single system is usually best placed in Hong Kong, which has good connections to both. However, even Hong Kong is susceptible to periodic outages or slowdowns, though CDNs and careful monitoring can help manage and mitigate these issues as much as possible.

Website Monitoring

A variety of services are available for monitoring a website's status and performance across China. The global brand Gomez is available and has a variety of services, along with the local firm Network Bench, which is especially good for monitoring local CDNs and per-province, carrier, last mile details nationwide. These services are quite costly, so choosing monitoring level, cities, and frequency is important for good cost control.



Security Issues

Security is a serious concern in the Chinese Internet, though probably less so than in the West. While hackers and other malicious activities certainly happen, they are not yet nearly at the rate nor sophistication of international attacks, yet. This is changing as the Chinese consumers and their information, along e-commerce fraud, become more valuable.

More broadly, the consumer PC area has extraordinarily poor security and many, if not most, PCs are thoroughly infected, often by dozens of things. Home firewalls are not that common and consumer awareness of these issues is surprisingly low.

This is made worse by many, if not most, users using illegal version of Microsoft Windows and applications, and never updating them, even if they could. So numerous security holes and issues are never addressed, making these systems very vulnerable.

Privacy issues are also not of great concern to end consumers, further lowering their awareness and instincts in protecting themselves and their data. Not to mention password management and other basic measures are very poor. Thus there is often little need to steal data from server systems when the end users are themselves so vulnerable.

Note that on-line banking and related systems in China do account for this and have very tough security practices and processes, almost always involving hardware tokens, mobile phone SMS codes, and more to ensure safety. This is especially important in a country with no checks, and thus most payments are done electronically, usually on-line, and thus a much higher risk profile and is common, for example in the U.S.

Hackers

Bad guys attacking servers are certainly active and often attacking back-end infrastructure, via a variety of methods. SQL injection attacks are quite common, as are uploading malicious images, etc. These attacks are made worse by poor coding practices, very limited testing, rapid development cycles, and a priority on features over security. For example, large systems still often have no hashed passwords, let along best practices salted hashes.

DDoS

The most common types of attacks are DDoS which in China can reach very large sizes, of tens of Gbps or more. These appear to be run for fun, for ransom, and as anti-competitive methods. The ransom versions are most dangerous as considerable resources are often in play and they are hard to mitigate.



In particular, most IDCs have no defenses at all and the regional carriers do not cooperate on DDoS to nearly the extent seen in the West. In addition, more sophisticated techniques like black-hole routing are not very common. IDCs will often just terminate and not route any customer IPs and if a common problem, terminate the hosting contract.

The best IDCs do have DDoS protection hardware, routing control, and carrier relationships to mitigate all but the largest attacks, though some of these such as ChinaNetCenter, require the use of their CDN services.

Recently security services such as AnQuanBao have come on line to further help manage these attacks. These are effectively CDNs with distributed nodes and large-scale bandwidth, able to withstand the largest DDoS attacks. They also have other benefits such as virus, XSS, and other problem detection and prevention.

IDC Local Attacks

Some attacks are actually hijack attacks that take place inside the IDC on shared bandwidth links, often via ARP. Hacker servers will send gratuitous ARPs to the switches so the default gateway points to them, thus all traffic from your server goes through them first.

Their server then injects evil javascript to point to a virus or other bad code so the user will download it and become infected. Since user PCs are so vulnerable, this is a very effective process in China. It's also difficult to detect and nearly impossible to mitigate against without special tools.

Physical Attacks

It's theoretically possible for bad guys to physically attack systems, such as stealing whole servers and their disks filled with data. This is extremely rare in China, so much so that it's not a real concern. Of course, best practice is to encrypt very valuable data such as credit cards, but having servers walk out the door is not near the top of the list of worrying issues.



Regulatory

China has a somewhat complex regulatory environment for Internet companies, though contrary to international perceptions, it's quite clear how things work, including what is possible and what is not allowed.

There are two general areas of concern: Website licensing and website content controls.

Licensing / ICP

All websites and indeed any system serving data over HTTP are required to register with the local government where their company is located. This is a free and relatively simple process, and typically takes about a month. Most IDCs and cloud providers will help process the filing, though for foreign companies, it can only happen after the subsidiary (WOFE) is formed and has its business license.

A basic registration, known as an ICP Beian, or Internet Content Provider Registration, is available to both local and foreign companies, but is only good for a site showing basic information such as sales or company / product details. In other words, a standard website, but one which does not fall into a number of special categories, discussed below. For a typical company site that does not sell anything, this is all you need and is easy to get.

Life becomes more complex when special categories are involved, which is a complex area, but basically including any exchange of money such as e-commerce, any games, any news, and most video, music, and media sites.

E-Commerce and games are the most common types of systems seen, though there are other categories, too, but these all require an ICP license and have additional requirements, including financial. Some also need additional special licenses or approvals, for example video sites like YouTube need approval from SARFT, and games need approval from the Ministry of Culture (MoC).

Note that mobile applications, APIs, and sites that feed data through other websites such as Renren (the Facebook of China) are a gray area and require negotiation with the IDC and perhaps local authorities.

Website Content Controls and ModerationPart of the basic registration and data center agreement is a provision that all websites and content providers will not have prohibited content on their systems. This is the controversial aspect of the regulatory framework from a Western perspective, as it prohibits content that would often be allowed elsewhere.



Broadly, for most sites, this is not a significant challenge, but it's easy to run into problems when user-generated content is allowed, for example when users can comment on products, or tell others about themselves and their interests. Bad-behaving users will exploit this opportunity to upload and write all kinds of problematic content.

By far the most common prohibited content is pornography, which is illegal in China, so people will upload porn picture or stories in as many places as they can find. The second biggest problem is often illegal products such as guns or drugs that get listed with an anonymous phone number. Note that political, news, and other topics which can be sensitive are actually rarely problems for typical websites.

Most government regulators and especially IDCs have very little tolerance for any of these types of problematic content, and will often just turn off the offending servers until the problem can be fixed. The best IDCs will call the customer and give them until 5pm, etc. to take care of the problem.

As a result it's very important for all sites that allow user-generated content to carefully monitor and moderate such content. Ideally, everything is reviewed before it goes live and if this is not possible, have methods to quickly report bad content so it can be quickly examined and removed, if needed.

Beyond this, specific industries have other requirements, including recently-issues real-name requirements for blogging sites, and user access log retention policies on other sites, for use by law enforcement.



Website Development Services

China has a number of good website development companies, ranging from large international outsourcing firms to small local teams that can build things quickly and inexpensively.

Most focus on particular areas, such as e-commerce, social marketing, Drupal, advertising, and other specialties. Others also focus on geographies, such as Shanghai or Beijing, which can be important for the necessary in-person meetings.

Like web development companies everywhere, employee retention can be a problem, as can code quality and overall staff experience. This is especially a problem for operations in China, as these teams often have little knowledge of performance and security-related issues and are no longer working on the system by the time these become problems months later. In particular, performance-related issues in SQL queries or database design are a challenge to fix once a system becomes popular and fails under heavy load.

Also, many developers are not overly familiar with horizontal scaling architectures that includes such requirements such as memcache-based sessions, data caching (and related coherency), DB read/write splitting, DB sharing, mail sending scale, and simple load testing.



About ChinaNetCloudFounded in Shanghai in 2008, ChinaNetCloud is the world's largest Internet Managed Services company, focused on server operations, especially Reliability, Performance, Scale, Security, and Cost Savings.

Focused on large-scale systems, ChinaNetCloud helps customers design, build, and operate any Internet-facing system and service, up to hundreds of millions of users and billions of page view per day.

Global Headquarters1238 Xietu Lu, 1-601Xuhui DistrictShanghai, China 200032

Beijing Office9nd Floor - Sunjoy MansionNo.6 Ritan RoadChaoyang DistrictBeijing, China 100020

Hong Kong Office9/F, Cambridge HouseTaikoo Place979 King’s Road, Quarry BayHong Kong

+86 (21) 6422-1946

[email protected]

www.ChinaNetCloud.com


running internet systems in china

Technology