Cloudata: A New Open Source BigTable Inspired Database

 OSS, Server  No Responses »
Mar 062011
 

Cloudata logoCloudata is a new open source implementation of Google’s BigTable paper. It can be found on Github here. It appears to be the project of a Korean developer named YKKwon.

As noted at MyNoSQL, there are only a couple commits and it’s not clear how serious this project is. But it will be of interest to big data, MapReduce and BigTable buffs.

Cloudata differentiates itself from Hadoop by offering an indexed but still non-relational database, but is probably more comporable to HBase and Hypertable, which are also open source BigTable implementations. The project’s website claims Cloudata can retrieve data within a few milliseconds.

Here’s a list of the current features:

Basic data service

  • Single row operation(get, put)
  • Multi row operation(like, between, scanner)
  • Data uploader(DirectUploader)
  • MapReduce(TabletInputFormat)
  • Simple cloudata query and supports JDBC driver

Table Management

  • split
  • distribution
  • compaction

Utility

  • Web based Monitor
  • CLI Shell

Failover

  • Master failover
  • TabletServer failover

Change log Server

  • Reliable fast appendable change log server

Support language

  • Java, RESTful API, Thrift

Getting Started with Hadoop and Map Reduce

 Knowledge, OSS, Server  No Responses »
Mar 062011
 
Have you been wanting to learn Hadoop, but have no idea how to get started? Carlo Scarioni has a basic Hadoop tutorial that covers installing Hadoop, creating a Hadoop Distributed File System (HDFS), moving files into HDFS, and creating a simple Hadoop application. The tutorial also introduces the basic concepts of Map Reduce.

It doesn’t, however, get into distributing the application, which is the main point of using Hadoop in the first place. Scarioni leaves that to a future tutorial. But if you want to get your feet wet with Hadoop and/or Map Reduce, this seems like a pretty good place to start.

also gives us a pretty concise explanation of what Hadoop is:

Hadoop is an open source project for processing large datasets in parallel with the use of low level commodity machines.Hadoop is build on two main parts. An special file system called Hadoop Distributed File System (HDFS) and the Map Reduce Framework.

The HDFS File System is an optimized file system for distributed processing of very large datasets on commodity hardware.

The map reduce framework works in two main phases to process the data. Which are the Map phase and the Reduce phase.

Setup High Performance Server

 Knowledge, Linux, OSS, Server  No Responses »
Oct 242010
 

High Performance Server? yeah. Its no wonder that we can setup high performance server.

Server – A computer running a service. yep. its true servers are nothing but computers running appropriate services.

So whats a server? Hardware or Software or both? What about the big 6 feet high machines we think as servers?

A server can be any computer running service(s).

You can make your laptop as a server or desktop or even mobile phone. but performance matters.

Servers performance depend on hardware and software. we can upgrade hardware to improve performance but upgrading software may fix bugs, add new features to it but no change in performance or can reduce performance.

Our major concern is choosing right software for the right job. There is no all in one software, even if there is it will be slow or buggy. We need to choose each and every piece of software to make a better system.

I will illustrate this with an example.

Requirement : We 10 people from 6 geographic locations intend to develop a social network which may serve 100 million users.

Analysis : Assume that we have best hardware, now focus on software. Software includes Operating System, Programming Language, Database Server, Web Server, SCM Server, Cache server. Our software need to be small memory footprint, bug/virus free, secure, fast.

If we see existing high performance implementations like hotmail, yahoo, facebook, twitter, google its like this,

  • Google : Python, C++, Plan9 (i think), Bigtable, GWS, GFS, mysql, …
  • Facebook : Linux, C++, C, PHP, Scribe, Thrift, Apache Cassandra, Hip hop, Tornado, Apache Hive, Varnish, Mysql, Memcached.
  • Hotmail : IIS, Windows server, .Net, …
  • Youtube : Memcached, …
  • Yahoo : PHP, Mysql, Free BSD, …
  • Twitter : Ruby, memcached, …

OS : CentOS or Ubuntu We have lot of options to choose in Operating Systems like Windows/Solaris/Unix/BSD/Linux/Mac/Plan9/AIX But Linux kernel is in active development than any other, fast/scalable/secure and nearly as bug/virus free as UNIX. Linux is having lot of flavors so we choose advanced and know version among them.

Programming Language : C, PHP, Javascript, Python, Haskel. We have several kinds of languages like Procedural, Object Oriented, Dynamic, Functional, Event Driven, Structured, Statically typed, Dynamically typed and few more. Each kind having its advantages. Normally we come to choose programming language according to our need/purpose rather than what we fancy. In general we have C, C++, Perl, Python, Ruby, .net(VB, C#, F#), Java, PHP, Haskel, Scheme, Javascript, Tcl,… C is the best to do low level stuff, PHP to play with http/HTML, Javascript to enrich presentation layer, python good at playing with text, heskel to do any exteded programming to take advantage of functional language.

DBMS : Mysql or apache cassandra. We choose fast and distributed database management systems from Mysql, MS Sql Server, Postgresql, Oracle, DB2, …

Web Server : nginx. A Webserver serve’s performance depends on how it process request and how it handles scripts(PHP, Ruby, Python, Java,….). We have Lots of Webservers like Lighttpd, nginx, Apache, Cherokee, IIS, Tomcat, Glash fish, Weblogic, mongrel, Webrick,….

SCM Server : Git. Git is the best of its kind when compared with Mercurial, Subversion, Bazaar, CVS, Bitkeeper and several others.

Cache Server : Varnish

In future I will try to screen cast the best setup i can.

A SQL Database Built for the Cloud

 OSS, Server  No Responses »
Sep 242010
 

The challenges that MySQL faces in the cloud – questions of elasticity, synchronization, scalability – are often referenced as part of the arguments in support of NoSQL database alternatives. But today,Xeround announces a solution to the problems of MySQL in the cloud that is actually a SQL solution – a SQL database for the cloud.

Xeround launches its new database that is, according to CEO Razi Sharir, “the best of both worlds.” In other words, Xeround’s new database technology promises both the transactional and query capabilities of relational databases alongside the scalability of NoSQL ones.

Xeround was built based on MySQL Storage Engine Architecture and acts as a pluggable storage engine. As such, it provides many of MySQL’s features, but was designed as a virtual database, optimized for the cloud. The database handles multi-tenancy and auto-scaling and is self-healing. It guarantees always-on service during schema changes, resource modifications, and the scaling process. And for those currently working with MySQL databases, all, according to Sharir, “without changing a line of code.”

Xeround_ss.jpgFor those in its private beta, Xeround will host its “Database-as-a-Service” in its private data center or on Amazon EC2. Other cloud service providers will soon be available. The DBaaS implementation simply requires uploading the database’s schema and pointing apps to the new location. The price for the offering will be competitive with other RDBS and by Amazon’s database offering.

Cloud Computing: Yahoo! Reaches the 2 Quadrillionth Bit of Pi

 Knowledge, OSS, Server, Technology  No Responses »
Sep 192010
 

A Yahoo! engineer has reached a new cloud computing milestone. Using Hadoop, Yahoo! cloud computing engineer, Tsz Wo (Nicholas) Sze has determined the computation of π (pi) to the two quadrillionth decimal. And it equals zero. It’s the most amount of bits ever calculated for pi.

This is a remarkable achievement and shows the power of distributed systems for crunching big data.

It’s also an example of how data is changing our infrastructure as we discover new ways to develop applications with greater computational capabilities.

As Yahoo! points out, pi is a mathematical constant that is used to calculate the area of a circle.

Sze’s discovery follows a long line of mathematicians who have sought a bigger piece of the pi, no pun intended of course. This effort took some significant firepower and Hadoop magic:

“The circumference of the visible universe can be calculated using a mere 39 or 40 decimal places, so what are Yahoos like Nicholas doing out at those outer reaches? For Nicholas, this was a project ideally suited to flex the muscles of Hadoop, the open source technology at the epicenter of big data and cloud computing. The computation took 23 full days and required 1000 different machines using Hadoop.”

The achievement shows the power of distributed systems and points to how Yahoo! has adapted its business to the research it has done using Hadoop.

Yahoo is one of the pioneers in using Hadoop. It has helped the company better define the context of its network and the personal preferences of the millions of people use Yahoo! every day.

What that means for Yahoo! is a better understanding of how to use the computational powers of cloud computing. It provides the opportunity to do all sorts of things. Yahoo! engineers, for instance, are learning how to speed up calculations and algorithms. That faster crunch time? Think of one of the world’s largest networks doing mathematics equations and you have something that equates to a whole new experience. It goes beyond new ways to find new songs or predict the price of apples in the commodity exchange.

Instead, this points to the entrance of a new world where our culture meets the power of big data. A society that is driven more by information than machine.

BigTable

 Linux, Server  No Responses »
Sep 062010
 

BigTable is a compressed, high performance, and proprietary database system built on Google File System (GFS), Chubby Lock Service, and a few other Google programs; it is currently not distributed or used outside of Google, although Google offers access to it as part of their Google App Engine.

BigTable development began in 2004 and is now used by a number of Google applications, such as MapReduce, which is often used for generating and modifying data stored in BigTable, Google Reader, Google Maps, Google Book Search, “My Search History”, Google Earth, Blogger.com, Google Code hosting, Orkut, YouTube, and Gmail. Google’s reasons for developing its own database include scalability, and better control of performance characteristics.

Design

BigTable is a fast and extremely large-scale DBMS. However, it departs from the typical convention of a fixed number of columns, instead described by the authors as “a sparse, distributed multi-dimensional sorted map”, sharing characteristics of both row-oriented and column-oriented databases. BigTable is designed to scale into the petabyte range across “hundreds or thousands of machines, and to make it easy to add more machines [to] the system and automatically start taking advantage of those resources without any reconfiguration”.

Each table has multiple dimensions (one of which is a field for time, allowing for versioning and garbage collection). Tables are optimized for GFS by being split into multiple tablets – segments of the table as split along a row chosen such that the tablet will be ~200 megabytes in size. When sizes threaten to grow beyond a specified limit, the tablets are compressed using the algorithm BMDiff and the secret algorithm Zippy, which is described as a less space-optimal variation of LZO but more efficient in terms of computing time. The locations in the GFS of tablets are recorded as database entries in multiple special tablets, which are called “META1″ tablets. META1 tablets are found by querying the single “META0″ tablet, which typically resides on a server of its own since it is often queried by clients as to the location of the “META1″ tablet which itself has the answer to the question of where the actual data is located. Like GFS’s master server, the META0 server is not generally a bottleneck since the processor time and bandwidth necessary to discover and transmit META1 locations is minimal and clients aggressively cache locations to minimize queries.

Related Implementations

Open source

  • HBase — Written in Java. Provides BigTable-like support on the Hadoop Core.
  • Hypertable — Hypertable is designed to manage the storage and processing of information on a large cluster of commodity servers.
  • Apache Cassandra — Facebook’s distributed storage system based on a Bigtable data model on Amazon’s Dynamo-like structure. Written in Java.
  • Neptune – Written in Java. Uses ZooKeeper, HDFS of Hadoop.
  • KDI - Kosmix attempt to make a BigTable clone. Written in C++.

Google Fusion Tables

  • Google Fusion Tables was released on June 9, 2009, as an experimental system for data management in the cloud.

GlusterFS

 Knowledge  No Responses »
Sep 062010
 

GlusterFS is a general purpose distributed file system for scalable storage. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. GlusterFS is based on a stackable user space design without compromising performance. From the ground zero days it has found a variety of applications ranging from Cloud Computing, Biomedical Sciences to Archival Storage.

Design

GlusterFS has a client and server component. Servers are typically deployed as storage bricks, with each server running a glusterfsddaemon to export a local file system as a volume. The glusterfs client process, which connects to servers with a custom protocol overTCP/IP, InfiniBand or SDP, composes remote volumes into larger ones using stackable translators. The final volume is then mounted by the client host through the FUSE mechanism. Applications doing large amounts of file I/O can also use the libglusterfs client library to connect to the servers directly and run translators in-process, without going through the file system and incurring FUSE overhead.

Most of the functionality of GlusterFS is implemented as translators, including:

The GlusterFS server is kept minimally simple: it exports an existing file system as-is, leaving it up to client-side translators to structure the store. The clients themselves are stateless, do not communicate with each other, and are expected to have translator configurations consistent with each other. This can cause coherency problems, but allows GlusterFS to scale up to several petabytes on commodity hardware by avoiding bottlenecks that normally affect more tightly-coupled distributed file systems.

Google File System (GFS)

 Knowledge  No Responses »
Sep 062010
 

Google File System (GFS) is a proprietary distributed file system developed by Google Inc. for its own use. It is designed to provide efficient, reliable access to data using large clusters of commodity hardware.

GFS is optimized for Google’s core data storage and usage needs (primarily the search engine), which can generate enormous amounts of data that needs to be retained; Google File System grew out of an earlier Google effort, “BigFiles”, developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford. Files are divided into chunks of 64 megabytes, which are only extremely rarely overwritten, or shrunk; files are usually appended to or read. It is also designed and optimized to run on Google’s computing clusters, the nodes of which consist of cheap, “commodity” computers, which means precautions must be taken against the high failure rate of individual nodes and the subsequent data loss. Other design decisions select for high data throughputs, even when it comes at the cost of latency.

The nodes are divided into two types: one Master node and a large number of Chunkservers. Chunkservers store the data files, with each individual file broken up into fixed size chunks (hence the name) of about 64 megabytes, similar to clusters or sectors in regular file systems. Each chunk is assigned a unique 64-bit label, and logical mappings of files to constituent chunks are maintained. Each chunk is replicated several times throughout the network, with the minimum being three, but even more for files that have high demand or need more redundancy.

The Master server doesn’t usually store the actual chunks, but rather all the metadata associated with the chunks, such as the tables mapping the 64-bit labels to chunk locations and the files they make up, the locations of the copies of the chunks, what processes are reading or writing to a particular chunk, or taking a “snapshot” of the chunk pursuant to replicating it (usually at the instigation of the Master server, when, due to node failures, the number of copies of a chunk has fallen beneath the set number). All this metadata is kept current by the Master server periodically receiving updates from each chunk server (“Heart-beat messages”).

Permissions for modifications are handled by a system of time-limited, expiring “leases”, where the Master server grants permission to a process for a finite period of time during which no other process will be granted permission by the Master server to modify the chunk. The modifying chunkserver, which is always the primary chunk holder, then propagates the changes to the chunkservers with the backup copies. The changes are not saved until all chunkservers acknowledge, thus guaranteeing the completion and atomicity of the operation.

Programs access the chunks by first querying the Master server for the locations of the desired chunks; if the chunks are not being operated on (if there are no outstanding leases), the Master replies with the locations, and the program then contacts and receives the data from the chunkserver directly (similar to Kazaa and its supernodes).

As opposed to many filesystems, GFS is not implemented in the kernel of an operating system, but is instead provided as a userspace library.

Diaspora

 Knowledge, OSS, Server, Technology  No Responses »
Sep 042010
 

Why can’t privacy and connectedness go hand-in-hand? That’s the question being raised by those behind the new Diaspora project, an ambitious undertaking to build an “anti-Facebook” – that is, a private, open source social network that puts you back in control of your personal data.

Envisioned by four NYU computer science students, the Diaspora project would replace today’s centralized social web (yes, they mean you, Facebook) with a decentralized one, while still offering something that’s convenient and easy for anyone to use.

According to the project’s homepage, the students, Daniel Grippi, Maxwell Salzberg, Raphael Sofaer, and Ilya Zhitomirskiy, “bonded over many late nights building a Makerbot,” (to you non-geeks, that’s a type of robot) and they “started discussing what a distributed social network would look like.”

The end result of those discussions was the idea for Diaspora. So they stopped talking about it and started building.

The project is now hosted on Kickstarter.com, a social fundraising platform that lets entrepreneurs and other creative types crowd-source funding by setting up a project goal, deadline and optional set of rewards for project backers.

In Diaspora’s case, they’re less than $2000 short of their $10,000 goal with under a month left to go until reaching their deadline. If the project receives the necessary level of funding by June 1st, it will be built and the code released as free software using theaGPL open-source software license.

What is a Decentralized Social Network?

So what is Diaspora anyway? Instead of being a singular portal like Facebook, Diaspora is a distributed network where separate computers connect to each other directly, without going through a central server of some sort.

Once set up, the network could aggregate your information – including your Facebook profile, if you wanted. It could also import things like tweets, RSS feeds, photos, etc., similar to how the social aggregator FriendFeed does. A planned plugin framework could extend these possibilities even further.

Your computer, called a “seed” in the Diaspora setup, could even integrate the connected services in new ways. For example, a photo uploaded to Flickr could automatically be turned into a Twitter post using the caption and link.

When you “friend” another user, you’re actually “friending” that seed, technically speaking. There’s not a centralized server managing those friend connections as there is with Facebook – it’s just two computers talking to each other. Friends can then share their information, content, media and anything else with each other, privately using GPG encryption.

Diaspora, the Turn-Key Solution

Because not everyone will be technically capable of (or interested in) setting up their computer to function as a “seed,” there are plans to offer a paid turn-key service too, similar to WordPress.com, the blogging platform. WordPress itself is software you can install and configure on your own server, if you’re inclined to do so, but if you’re less technically-savvy, you can opt to quickly start a blog via WordPress.com instead. Diaspora would function in a similar way.

If a lot this sounds reminiscent of Opera’s Unite project, the Web browser maker’s overly-hyped plan to “reinvent of the Web,” it should. In Opera Unite, users can share documents, photos, music, videos and run websites and chat rooms by directly linking two computers together.

However, in Unite’s setup, there are Opera-run proxy servers involved, which led to issues – especially when those servers went down. Diaspora wouldn’t have that problem.

Mainstream Success?

Still, the concepts behind Diaspora, while the sort of thing tech geeks will eat up, may be harder to grasp for the everyday Facebook user who is still trying to figure out how post a link or video to their Wall. Distributed, decentralized, open-source what?

If Diaspora is realized, it will be up to technology advocates to position the turn-key service in a way that will make it sound simple and appealing to precisely those sorts of mainstream users if it is to ever succeed. Taking shots at Facebook’s privacy issues may be a good course (Take back control with Diaspora!).

We would like to see Diaspora come to be, even if it never goes mainstream because it would finally offer privacy advocates a real alternative to the increasingly data-hungry Facebook.

Software Router Smashes Speed Records

 Knowledge, Linux, OSS, Server, Technology  No Responses »
Sep 042010
 

Researchers in South Korea have built a networking router that transmits data at record speeds from components found in most high-end desktop computers. A team from the Korea Advanced Institute of Science and Technology created the router, which transmits data at nearly 40 gigabits per second–many times faster than the previous record for such a device.

The techniques used by the researchers could lead to a number of breakthroughs, including the use of cheaper commodity chips, such as those made by Intel and Nvidia, in high-performance routers, in place of custom-made hardware. The software developed by the researchers could also serve as a testbed for novel networking protocols that might eventually replace the decades-old ones on which the Internet currently runs.

Most routers use custom hardware to route data as it passes between computer networks. Software routers perform the same tasks using commodity hardware–by mimicking the behavior of a hardware router in software. Commercial software routers from companies such as Vyatta can typically only attain transfer data at speeds of up to three gigabits per second. That isn’t fast enough to take advantage of the full speed of a typical network card, which operates at 10 gigabits per second.

“We started with the humble goal of being the first to get a PC router to 10 [gigabits per second], but we pushed it to 40,” says Sue Moon, leader of the lab in which the research was conducted. Her students Sangjin Han and Keon Jang developed software called PacketShader that made this possible. PacketShader uses a computer’s graphics processing unit (GPU) to help process packets of data sent across a network.

Modern routers are rarely dumb switches anymore. They are often called upon to manipulate packets in a number of different ways as they pass through. GPUs are ideal for this purpose because they can process data in parallel, which means they can handle several packets of data at once. According to Moon, a GPU is much faster at handling some packet-processing tasks, such as authenticating or encrypting all of the packets in a stream. When the GPU takes over these tasks, it gives the central processing unit (CPU) breathing room to handle other things that are more serial in nature, such processing several packets in turn to detect attempts to break into a network.

Mark Handley, a professor of networked systems at University College London, points out that for basic packet forwarding, which isn’t likely to overwhelm a computer’s CPU, there is no advantage to strapping the GPU onto the system. However, he agrees that the GPU is very well suited to encrypting or authenticating packets.

Gianluca Iannaccone, an engineer at Intel Labs Berkeley who is familiar with PacketShader, says it could slash the number of physical machine needed to comprise a terabit-per-second software router to one-third of what his research has previously indicated would be required.

“One terabit is the entry point for enterprise-grade routers–the routers in the core of the Internet,” says Iannaccone. His work on a system called RouteBricks points to a future in which routers aren’t the specialized hardware they are now, but instead function as software running on pools of servers. Lash enough software routers together that run at 40 gigabits per second, and you get what is essentially a single-terabit router. Using such a system, routers might some day run completely in software.

“We can expect killer apps out of this,” says KyoungSoo Park, another professor at the Korea Advanced Institute of Science and Technology who was involved with the project. “You can build an interesting packet- or network-management system on top of a PC-based software router that can’t be implemented with a hardware router. Ultimately, you can experiment with new protocols that are not used in today’s Internet.”

 Older Entries