Diagonal scaling is making a site faster by removing machines. The term was created by John Allspaw, the operations manager of Flickr, who replaced 67 dual-cpu boxes with 18 dual quad-core machines and recovered almost 4x rack space and reduced costs by about 50%

Horizontal scaling (scaling out) means adding or removing client workstations with only a slight performance impact. Vertical scaling (scaling up) means migrating to a larger and faster server machine or multiservers.

See a video by CIO of CNET, on scaling out and don't miss the 8 fallacies of distributed computing... and the 9th one!

The Law of Diminisging Returns, referred to also as variable factor proportions, states that

as equal quantities of one variable factor are increased, while other factor inputs remain constant -ceteris paribus- a point is reached beyond which the addition of one more unit of the variable factor will result in a diminishing rate of return and the marginal physical product will fall.

I think the reason I was associating these 2 is because I thought: In Allspaw's model of diagonal scaling, this law is not only undermined, but is infact inverted.

Illustration of scaling up and out from the microsoft web applications libraries

So what is Microsoft doing? Naturally, it is playing to its strengths by tightly integrating this online storage service to the existing Microsoft Live framework which I have never been a fan of. What started off a few years ago as a Live 'passport' to enable MSN (Windows Messenger - which I use) and email (hotmail - which I most certainly don't use) to be integrated in one service. Windows XP was the first Operating System to feature true integration by allowing each user account to associate a .Net Live passport with a user account.

Of late, Microsoft have really stepped up their work on the Live platform and have turned it into a real Web 2.0 fan's dream. Not only can email be accessed, but a whole load of (Microsoft hopes) Google app-killers such as Photo Gallery, Spaces (myspace style blog) as well as others. Although, the most telling thing about Windows Live Services, is Microsoft's choice for subdomain : "ideas.live.com" reinforcing (either intentionally or unintentionally) the experimental 'young-hip' approach which has served Google so well up to now.

I guess the biggest thing that irkes me about Microsoft Live services, are not the services themselves (which are actually quite good) but the corporate mentality behind them. This is best summed up in the way in which one would set about using one or more of Microsoft's Live services. For example, I wanted to give Microsoft Live Writer a go to see if it was a more comfortable way of posting to this blog, so I went to the appropriate (the specific writer program page) and selected download. What downloaded was not a setup binary for Writer, but a generic installation program called WLinstaller.exe . This then tried to convince me to change my default search engine to Live Search and set my home page to MSN in Internet Explorer. Good luck with that.

This is what annoys me, unlike a company like Mozilla who would happily provide you the install to any of their free programs on demand, Microsoft try to convert you. Even if you want to install one program then Microsoft will preselect everything including service integrations. That is not to say they do not have good software and services, just their aggressiveness puts me off - I do not install software randomly on my systems and I strongly dislike the principle of bundled crapware.

But anyway, I am getting away from the point. Windows Live SkyDrive is free and offering 5Gb of storage with (presumably) no bandwidth restrictions. The service can be slow in places: creating new folders is sluggish, sometimes taking almost half a minute to allow access into the corresponding dialog. However, Once the folder is created, uploading was fast as was browsing existing folders associated with my Live account. Pictures are automatically viewed with another Microsoft Live Service namely Photo Gallery which allows for thumbnail views of your uploaded images. This is a handy, unobtrusive feature.

What particularly interests me is the way in which you can share files per users. There are three main ways to share files (of any file type), by whitelisted contacts, by the entire Internet or "for your eyes only". When specifying which contacts have access to your shared files (which you upload) you can add in a type of user privilege system. If you white-list a friend / user as a 'reader' they will only be able to view / read the files you have allowed them to see. They are unable to make any changes to your files unless you allow them 'editor' privileges.

This dramatically streamlined system allows a great deal of flexibility and I am impressed with how smooth the whole experience is. Microsoft seems to have really got this service right. I can think of a multitude of situations where this service could be useful, however I will not be using this service personally. For casual file sharing between contacts (who already have Live accounts from MSN) this could prove invaluable, but I will save a discussion of the cloud computing concept (and my reasoning behind this) for another post.

Source: CERN

During the pilot project, the one thing that was mentioned most often was CERN, the European Organisation for Nuclear Research. I just had a closer look at their website tonight and found out that they have an Open Day on 6 April 2008! I think I’ve just found my dream location for doing the ‘Mutable Matter’ art project! Does anyone know anyone at CERN who could allow me to do ‘science busking’ there? ;)

So why did so many people ‘admit’ to be curious about CERN’s activities – and what goes actually on in there? The most obvious thing to say is ‘they smash particles and see what this can tell them how things work in the universe’. However, their official website reveals many more facets of their activities.

Mind-gymnastics is the word that comes to my mind when reading about the back and forth between the scale of particles, our everyday world and the universe. CERN employees put together apparatuses for antimatter-searches in space and even go into space themselves, they race neutrinos, ‘routinely produce antimatter’ and even 'anti-atoms', create ‘The coolest place in the Universe’ just beneath the French/Swiss border, use computing against avian flu and malaria, answer questions about claims made by science fiction novels and even study our climate by looking at the interrelationship between (‘real’ and ‘home-made’) clouds and cosmic rays. What a place to work! On top of all this, CERN is also monitoring its close environment to detect or prevent any disturbances that the creation of cosmic phenomena on an earthly scale may entail. This section also tells you that you don’t have to worry about getting ‘eaten’ by black holes or so-called ‘strangelets’. One could almost say ‘what a shame’… (strange images of Heidi being chased by ‘strangelets’ come to my mind)

The reason why CERN might be on so many people’s radar at the moment is the media coverage about the completion of the Large Hadron Collider which is designed to smash protons together. Fascinating is not only what this collider does, but how gigantic it is, how it is being put together and how it looks. While almost everybody will be in some degree of awe, there are also critical voices.
Like other ‘basic science’ (!) research places, CERN also has to deal with accusations about being a waste of government money that should better be directed towards other causes. The website gives – amongst other plausible examples - a very clever reason why CERN should forever be worshipped: they gave us the internet! And because friendly scientists gave us endless hours of chatroom and youtube entertainment, we can now even say ‘thank you’ by helping them compute their results! What do we learn from this? Two things: 1. CERN does just about anything regardless of scale. 2. The word ‘petabyte’.

We (America) need to jump on this paradigm shift to reduce costs in this post Sarbanes Oxley and difficult economic environment if we want to gain competitive advantage for ourselves and for our country. No longer is running enterprise CRM or ERP a competitive advantage - its table stakes.

What does this mean for IT departments? What does this mean for your data security? And most importantly - what does the impact of distributed computing have on marketers? Check out what Nick has to say about all this …

[audio http://media.podcastingmanager.com/72206-80605/Media/Nicholas%20Carr%20final.mp3](The Big Switch - podcast with Nicholas Carr)

About Nick

A former executive editor of the Harvard Business Review, Nicholas Carr writes and speaks on technology, business, and culture. His 2004 book Does IT Matter?. published by Harvard Business School Press, set off a worldwide debate about the role of computers in business. His widely acclaimed new book, The Big Switch: Rewiring the World, from Edison to Google, examines the rise of "cloud computing" and its implications for business, media and society.

Carr writes regularly for the Financial Times, Strategy & Business and The Guardian. His articles have also appeared in the New York Times, Wired, Business 2.0, The Banker, and Advertising Age as well as on his blog Rough Type. He is a member of the Encyclopedia Britannica's editorial board of advisors.

In 2005, Optimize magazine named Carr one of the leading thinkers on information technology, and in 2007 eWeek named him one of the 100 most influential people in IT. Earlier in his career, he was a principal at Mercer Management Consulting.

Carr has been a speaker at MIT, Harvard, Wharton, the Kennedy School of Government, NASA, and the Federal Reserve Bank of Dallas as well as at many industry, corporate, and professional events throughout the Americas, Europe, and Asia. He holds a B.A. from Dartmouth College and an M.A., in English literature, from Harvard University.

The nodes in the peer-to-peer network are defined logically, change dynamically, have their own protocols for discovering nodes and transferring data and utilize internet protocols. They are overlays on the internet.

Overlays create the potential for creating new and disruptive network and service architectures. They are easily deployed on existing host machines connected to the internet. It can provide a mechanism for services to migrate across the internet, perform dynamic discovery of remote services and become adaptive to handle failures and distributed load. These capabilities could be available across any and all computers connected to the internet.

There are a handful of network overlay software packages available as open source today. They provide features beyond what the internet offers. Some of the packages include:

• RON - to improve the availability and reliability of data packet routing
• Chord - used to build scalable distributed peer-to-peer systems
• Bamboo - implementation of a DHT algorithm for use in peer-to-peer architectures

If the following can be combined:

• a good DHT algorithm to manage mappings of host across the internet which dynamically join and leave
• software that discovers and host services at each node in the network

This would create a very powerful implementation of an architecture supporting many different kinds of dynamic services and social networks that execute not just in the isolation of data centers but across any and all computers wishing to join the network.

As I am currently working on building an open source version of FriendFeed (not targetted to be an alternative, coz those people at FriendFeed done their job really well) and scaling such a huge load effectively at low cost, so I will mainly talk more about FriendFeed through out this blog post and use it as an example for my proposal.

If you consider FriendFeed as a repository of feed URLs, a lot of peoples and how they are related to each other, you can assume how big it is or it could be in near future. And scaling such a service would cost numbers of sleepless nights of many developers out there. So in basic, lets focus where actually the problem is and how we can introduce distributed computing.

Beside optimizing database to serve huge sets of data, one of the main problems of such a service has to parse millions of feeds in a regular interval. If we want to bear all the loads on your server, fine, if you can afford. But what about some low cost solutions. Lets consider a simple scenario, if your application has one million of users and each of them browse your application for 10 minutes a day, you really have 10 millions of computational power just wasting over there, in lanes and by lanes of internet - heh heh. So lets make use of such an incredible CPU power. All you have to do let the visitors machine do some calculations for you and free your server from gigantic load.

When the users of your application and relation among them are stored in your database, you can easily find out the first degree and second degree friends of a specific user. If you don't know what does that mean, its simple, If A is a friend of B and C is a friend of A, then A is B's first degree friend and C is B's second degree friend. For a huge social network, it may look like the following one when you visualize the relationship

image courtesy: http://prblog.typepad.com

Now what we want to do is when B is visiting our application we want to parse most of his/her second degree friends in client using his browser. So while generating the page for B, we will supply him a bunch of feed URLs, a hash of their last known update time or hash of the latest item of each of these corresponding feeds, and a javascript based parser (for example Google's ajax Feed API would do fine) script. Now, while B is browsing our application we will parse those feeds of his second degree friends using javascript without bothering him for a single second and post back the parsed contents (after checking against the hash for really updated contents) to a server side script which will then blindly (not totally, after some validation or authentication for sure) insert those result to our database. Now when A comes and visit his page (A is C's first degree friend), he will get all the latest result from C's feeds as B has already done the parsing job for us and we have those latest result from C's feeds stored in our Database.

There are definitely more challenge than it is explained here, like what if a person is second degree friend of a multiple user. In such cases as we supplied last update time of these feeds while generating a page, we can calculate from server side which feeds we really want to parse.

And moreover, we can add more check for our javascript parser than just blindly parse those feeds. We can download a chunk of RSS or Atom feeds (using a proxy script developped using curl range options) and read just up to the latest update time and extract the time using simple reg ex or string functions instead of downloading full feed data. Now if we know that a script is not uptodate just by downloading 1-2 killobyte of data instead of downloading full feed, parsing the xml data, it would save us more computing resources for performing other jobs.

But of course, you cannot live completely on your client's machine for parsing all your data. You must have several cron'd scripts to parse left overs or other feeds at server side. But what I am saying is that with little help of javascript you can make use of such a tremendous distributed computing power in your application, all at almost no cost.

I will come with example code once I am done developing my open source clone of FriendFeed and then I am sure, you will find it was worth writing a blog post about.

image courtesy: http://www.naccq.ac.nz/bacit/0203/2004Caukill_OffPeakGrid.htm

Have a nice RnDing time. :)

The term cloud computing probably comes from (at least partly) the use of a cloud image to represent the Internet or some large networked environment. We don't care much what's in the cloud or what goes on there except that we depend on reliably sending data to and receiving data from it. Cloud computing is now associated with a higher level abstraction of the cloud. Instead of there being data pipes, routers and servers, there are now services. The underlying hardware and software of networking is of course still there but there are now higher level service capabilities available used to build applications. Behind the services are data and compute resources. A user of the service doesn't necessarily care about how it is implemented, what technologies are used or how it's managed. Only that there is access to it and has a level of reliability necessary to meet the application requirements.

In essence this is distributed computing. An application is built using the resource from multiple services potentially from multiple locations. At this point, typically you still need to know the endpoint to access the services rather than having the cloud provide you available resources. This is also know as Software as a Service. Behind the service interface is usually a grid of computers to provide the resources. The grid is typically hosted by one company and consists of a homogeneous environment of hardware and software making it easier to support and maintain. (note: my definition of a grid is different from the wikipedia definition, but homogeneous environments in data centers is typically what I have run across). Once you start paying for the services and the resources utilized, well that's utility computing.

Cloud computing really is accessing resources and services needed to perform functions with dynamically changing needs. An application or service developer requests access from the cloud rather than a specific endpoint or named resource. What goes on in the cloud manages multiple infrastructures across multiple organizations and consists of one or more frameworks overlaid on top of the infrastructures tying them together. Frameworks provide mechanisms for:

• self-healing
• self monitoring
• resource registration and discovery
• service level agreement definitions
• automatic reconfiguration

The cloud is a virtualization of resources that maintains and manages itself. There are of course people resources to keep hardware, operation systems and networking in proper order. But from the perspective of a user or application developer only the cloud is referenced. The Assimilator project is a framework that executes across a heterogeneous environment in a local area network providing a local cloud environment. In the works is the addition of a network overlay to start providing an infrastructure across the Internet to help achieve the goal of true cloud computing.

Java Remote Method Invocation (RMI) allows you to write distributed objects using Java. RMI provides a simple and direct model for distributed computation with Java objects. These objects can be new Java objects, or can be simple Java wrappers around an existing API. Java embraces the "Write Once, Run Anywhere model. RMI extends the Java model to be run everywhere."

Because RMI is centered around Java, it brings the power of Java safety and portability to distributed computing. You can move behavior, such as agents and business logic, to the part of your network where it makes the most sense. When you expand your use of Java in your systems, RMI allows you to take all the advantages with you.

In this example, we store the Traveler’s Forecast information in an RMI remote object that accepts requests for weather information through remote method calls. This example is an abridged version of the one given in Advaned Java 2 How to Program.

The four major steps in this example include:

1. Defining a remote interface that declares methods that clients can invoke on the remote object.
2. Defining the remote object implementation for the remote interface. [Note: By convention, the remote object implementation class has the same name as the remote interface and ends with Impl.]
3. Defining the client application that uses a remote reference to interact with the interface implementation (i.e., an object of the class that implements the remote interface).
4. Compiling and executing the remote object and the client.

Detailed steps are as below.

Step 1: Defining the remote interface

import java.rmi.*;
public interface WeatherService extends Remote
{
public String getWeatherInformation() throws RemoteException;
}

Step 2: Defining the remote object implementation

import java.rmi.*;
import java.rmi.server.*;

public class WeatherServiceImpl extends UnicastRemoteObject
implements WeatherService
{

public String weatherInformation = null;
public WeatherServiceImpl() throws RemoteException
{
 super();
 updateWeatherInformation();
}

public String getWeatherInformation() throws RemoteException
{
return weatherInformation;
}

public void updateWeatherInformation()
{
 weatherInformation = "Snowing";
}

public static void main(String[] args) throws Exception
{
System.out.println("Starting WeatherService...");
WeatherService weatherService = new WeatherServiceImpl();
String objectURL = "rmi://129.241.146.251:1099/WeatherService";
Naming.rebind(objectURL, weatherService);

System.out.println("WeatherService Running...");
}
}

Step 3: Defining the Client

import java.rmi.*;
public class WeatherServiceClient
{
public WeatherServiceClient(String server)
{
String remoteName = "rmi://" + server + ":1099/WeatherService";
try
{
WeatherService weatherService = (WeatherService)Naming.lookup(remoteName);
String weatherInformation = null;
weatherInformation = weatherService.getWeatherInformation();
System.out.println("Received Weather Information: " + weatherInformation);
}
catch(Exception e)
{
e.printStackTrace();
}
}

public static void main(String[] args)
{
WeatherServiceClient wsc = new WeatherServiceClient("129.241.146.251");
}
}

Step 4: Running the RMI Registry

The next step is to start the RMI registry with which the WeatherServiceImpl object will register. The command line rmiregistry launches the RMI registry on the local machine.

Step 5: Compiling and Running the server
javac WeatherServerImpl.java
java WeatherServerImpl

Step 6:Compiling and Running the client
javac WeatherServiceClient.java
java WeatherServiceClient

When do we say that one event causes another? Causality is a topic of great interest in statistics, physics, philosophy, law, economics, and many other places. Now, if causality is not complicated enough, we can ask what is the influence one event has on another one.  Michael Ben-Or and Nati Linial wrote a paper in 1985 where they studied the notion of influence in the context of collective coin flipping. The title of the post refers also to Nati's influence on my work since he got me and Jeff Kahn interested in a conjecture from this paper.  

Influence 

The word "influence" (dating back, according to Merriam-Webster dictionary, to the 14th century) is close to the word "fluid".  The original definition of influence is: "an ethereal fluid held to flow from the stars and to affect the actions of humans." The modern meaning (according to Wictionary) is: "The power to affect, control or manipulate something or someone."

Ben-Or and Linial's definition of influence

Collective coin flipping refers to a situation where n processors or agents wish to agree on a common random bit. Ben-Or and Linial considered very general protocols to reach a single random bit, and also studied the simple case where the collective random bit is described by a Boolean function $latex f(x_1,x_2,\dots,x_n)$ of n bits, one contributed by every agent. If all agents act appropriately the collective bit will be '1' with probability 1/2. The purpose of collective coin flipping is to create a random bit R which is immune as much as possible against attempts of one or more agents to bias it towards '1' or '0'.

Given such a protocol, the influence of a set S of agents towards '0' is the probability that R=0 if the agents in S try to tilt the outcomes of the coin flipping towards '0' as much as possible. The influence towards '1' is defined in the same way. And the influence of S is the sum of these two quantities. To make the definition clearer we should explain what the agent in S can do. Here we assume that in case of simultaneous action by all agents, the "bad guys" can wait to the contributions of all other agents before making their move. The bad guys can only change their inputs to the procedure.

Notations: When the protocol is denoted by $latex f$, for a set $latex S$ of processors, denote by $latex I^+_S(f)$ their influence toward '1', by $latex I^-_S(f)$ their influence toward '0', and let $latex I_S(f)= I^+_S(f)+I^-_S(f)$. The influence of a single processor $latex k$ is denoted by $latex I_k(f)$ and the sum $latex I(f) = I_1(f)+ I_2(f)+ \dots +I_n(f)$ is called the total influence of $latex f$. (In the Boolean case we refer to a "processor" or "agent" simply as a variable, and talk about influence of a variable, and influence of a set of variables.)

Notions of powers 

Boolean functions can be regarded as "voting rules". A Boolean function describes a way to move from the votes of n voters between two candidates to the collective decision of the society. Thinking of Boolean functions as voting rules provides nice names for special kinds of Boolean functions. "Dictatorship" refers to functions of the form $latex f(x_1,x_2,\dots, x_n)=x_k$. For the "majority function" (when $latex n$ is odd) the value of $latex f$ is '1' if and only if for more than half the variables $latex x_k =1$. For monotone Boolean functions, the influence of the kth variable coincides with the "Banzhaf power index" defined in game theory. Another related important notion of power is the Shapley-Shubik power index.    

The KKL Theorem

Picture: Muli Safra

After many months of working on the conjecture, Jeff, Nati, and I managed to prove it. Let $latex f(x_1,x_2,\dots,x_n)$ be a Boolean function.

Theorem 1 (KKL): If the $latex \mu (f)=t$ then there exists a variable k so that $latex I_k(f) \ge K t (1-t) \log n/n$.

A repeated application of this theorem shows that:

Theorem 2 (KKL): If the $latex \mu (f)=1/2$ then there exists a set S of $latex K(\epsilon) n/\log n$ variables so that $latex I_S(f) \ge 1-\epsilon $.

Examples, examples

Majority

For the majority function with n variables the influence of every variable is proportional to $latex \sqrt n$.  

The tribes example

This is the basic example of Ben-Or and Linial for Boolean functions with low influence. The society is divided into a large number of tribes, each having $latex \log n - \log \log n +\log \log e$ members. The value of f is one if and only if there exists a tribe whose members all vote '1'. For this example the influence of every variable is $latex \theta (\log n /n)$.  

 The next two examples represent more complicated protocols for collective coin flipping. (Not just Boolean functions.) 

Mike Saks' "passing the baton" example

We start with some voter who holds the baton. This voter passes the baton to another random voter. Every voter who gets the baton passes it to a random voter who did not yet hold it. The last voter to hold the baton chooses the random collective bit. In this example, $latex o(n/\log n)$ bad agents cannot tilt the outcome significantly. (The best strategy for the "bad guys" is to pass the baton to a "good guy".)

Uri Feige's "two rooms" example 

Every agent enters at random one out of two rooms. The room with fewer agents is selected and every agent in this room enters at random one out of two rooms. This process is continued (more or less) and at the end, as before, the last remaining agent contributes the collective random bit.

This process is immune against a constant number of "bad guys". (The first such example was found by Noga Alon and Moni Naor.) The number of rounds in this protocol (appropriately optimized) goes to infinity extremely slowly. It is not known whether there is a protocol with similar properties with a bounded number of rounds. 

Influence and threshold behavior

Consider a monotone Boolean function $latex f(x_1,x_2,\dots,x_n)$. Let $latex \mu_p$ be the product probability space where for every bit $latex \mu_p(x_i=1)=p.$ The definition of influence extends without change to the setting of biased product distribution. The influence of the $latex k$th variable on the Boolean function f with respect to $latex \mu_p$ is denoted by $latex I_k^p(f)$. The probability $latex \mu_p (f)$ that $latex f(x_1,x_2,\dots,x_n)=1$ is a monotone function in $latex p$ and Russo's lemma asserts that the derivative of $latex \mu_p (f)$ with respect to $latex p$ is precisely the total influence $latex I^p(f)$. Therefore, large influence is related to "sharp threshold behavior". Namely, to a very short interval between the value of $latex p$ where $latex \mu_p(f)$ is very close to 0, and the value of $latex p$ where $latex \mu_p(f)$ is very close to 1. Simple consequences of KKL's theorem to the study of threshold behavior were noted by Ehud Friedgut and me, and Friedgut found an important theorem giving conditions for sharp threshold behavior when p itself is a function of n. Muli Safra and I wrote a survey paper on influences and threshold behavior. 

Aggregation of information and Condorcet's Jury theorem

Four sentences about the connection with Game Theory: The sharp threshold phenomenon is called in economics "asymptotically complete aggregation of information". This property goes back to an old theorem from the theory of voting called "Condorcet's Jury Theorem". The Shapley-Shubik power index, mentioned above, can be defined as the integral $latex \int_0^1 I_k^p(f)dp$.  (This is not the original axiomatic definition but a later Theorem by Owen.) It turns out that for a sequence of monotone Boolean functions, "sharp threshold phenomenon" is equivalent to "diminishing individual Shapley-Shubik power indices". (But the quantitative aspects of this result are not satisfactory.)     

Influence without independence

A major conceptual challenge is to understand the concept of influence (and related notions of "sharp threshold phenomenon") for distributions which are not product distributions, namely when the probabilities for individual bits to be '1' are not statistically independent. Does an observer in a committee meeting have an influence? Can there be a negative influence? Moving away from statistical independence is often very difficult and yet very important for most applications. This issue is addressed in the paper of Graham and Grimmett, and that of Haggstrom, Mossel, and myself.

Two old conjectures about influence

There are quite a few problems regarding influences which remained unsolved. I will mention only two related conjectures both dealing with the Boolean case.

Conjecture 1: (Benny Chor): Suppose that $latex \mu(f)=1/2$ there is $latex c>0$, such that there is a set $latex S, |S|= n/10$  with $latex I_S(f) =1-\exp (cn)$.

Conjecture 2: Suppose that $latex \mu(f) = 0.999^n$. Then there is a set $latex S, |S|=0.499n$, so that $latex I_S(f)=1-o(1)$.

KKL theorem gives a weak form of Conjecture 1 where $latex 1-\exp (cn)$ is replaced by $latex 1-n^{\beta}$ and a weak form of Conjecture 2 where $latex 0.999^n$ is replaced by $latex 2^n/ n^{\alpha}$. The main difficulty here (and in various other problems in extremal combinatorics) is that arguing about influences of single variables is the only known method toward influences of large sets. Both these conjectures (as KKL's theorem itself) have natural formulations in terms of traces of families of sets related to the Sauer-Shelah Theorem.

More on Influence

The theory of poetic influence was developed by Yale's literary critic Harold Bloom. His book "The Anxiety of Influence" deals with the process of influence as well as with the psychology of influence in literature. Philosopher Avishai Margalit studied influence in Philosophy in his paper on Wittgenstein. The paper will appear in a Festschrift for Peter Hacker by Blackwell Oxford, 2009. Section 3 opens with a distinction between influence and power: "Naked power is for anyone to see. Influence is not. It works its wonders in ways not readily observable. Influence is inferred from its effects. This is a major reason for the elusive nature of influence."

Michael Ben-Or 

Trivia question:

What do Michael Ben-Or, Uzi Segal (whose calibration theorem was mentioned in the post on the controversy around expected utility theory), Mike Werman, Ehud Lehrer, Yehuda Agnon, myself, and quite a few others, all have in common.

Hint:

Influence and coin flipping little updates (June, 8)

Mike Saks made valuable comments regarding this post. The first protocol for collective coin-flipping immune to a positive proportion of processors was by protocol with $latex log^*n$ rounds was by Alexander Russell and David Zuckerman. If each processor contributes only one bit per round there is a matching lower bound by Russell, Saks, and Zuckerman. The open problem Section from RSZ's paper is very interesting and, as far as we know, no further progress on the problems presented there was made.

If we allow every processor to contribute many bits in every round then the situation is not clear even for one round. This is related to conjectures discussed by Ehud Friedgut.  

Related problems are of great interest for quantum computation. Carlos Mochon have recently solved one of the most important longstanding open problems in quantum information theory, and found a protocol for weak coin flipping with arbitrarily small bias, using quantum bits. His work strengthens earlier work by Kitaev.

Related post: The Entropy Influence Conjecture. (Terry Tao's blog) 

My friend Joe found a steel rack enclosure back behind the movie theater (don't ask me why it was behind a movie theater because I honestly don't know), and so we lugged it over to my house and stuck my racks in it. That is, after getting all the racks working. We are still working on clustering the 3 racks together but we are running into some difficulties with that. We'll figure it out eventually. Anyways, heres pictures!

Most of the top super computer in world are based on cluster computing concept. And one of the best example of distributed cluster computing based super computer is World Community Grid. Its a great concept of using idle power of millions of computer for betterment of humanity.

I worked in one of the biggest organisation in India. They are big not only in net market worth, but, also in number of people they employ. And because of nature of work, personal computer is integral part of every single employee working in these big organisation. And since these computers are used by humans, they have almost normal usage pattern like any other normal computer. Means, they remain ideal most of the time.

So, what we are looking at is a huge untapped computing power ready to harness. Then obvious question will come in head of those who promote concepts like WCG. They might say why not use this power for better purposes.

Well,  Lets say right now I am thinking in terms of ROI.

What investment I will be doing?

Just allowing any application such as WCG on thousands of computer over my network will lead to increase in Internet bandwidth usage and increase in electricity usage(Small but still it will increase if we factor in thousands of computer).

What will I get out of it?

Nothing as such.

Idea

Using this huge distributed computing power within big organisation for purposes which are in interest of organisation.

Proposed usage:

1. Organisations doing R&D and required huge computing power for various tasks can utilise this power like IBM. Deriving organisations toward better return from existing investment.
2. Organisations not having R&D division, or not one requiring computing power can sell this processing power to other organisations like Org such as Infosys, Wipro, HCL selling computer power to Universities, research institutes,etc. In this way they can generate additional revenue stream.
3. This can be seen as overall reduction in carbon emission. So, better carbon emission record for companies.

Why companies would like to do it?

1. Cost saving.
2. Optimum utilization of resources.
3. Due to security concern and lack of monitoring. Companies won't like to join outside distributed computing projects.

What is required?

1. Need of big organisation to see huge amount of wasted processing power as an opportunity waiting to be tapped.
2. A software solution which can be used to implement such solution.

Opportunity for Open Source Community

As we are always fighting for good ideas. This could be one idea on which OSC can contribute a lot.

Search is the quintessential example.  When I wrote the search algorithms and protocols for the second-generation Gnutella network at LimeWire, the degree of coordination between millions of network nodes was staggering.  It worked amazingly well all things considered, but it still could never compete with the speed or collaborative filtering of centralized search.  Most importantly, it could never compete with simplicity of centralization.  What took us 6 months to distribute would have taken a couple of days to centralize.  Distributed networks also make updating much harder, as you're now forced to update every instance of your software running on any machine in the world.  It's a pain in the a$$.  If you have a choice, centralized search wins every time.  That's one of the reasons LittleShoot follows Fowler's law whenever possible.  Something about working better and taking less time makes it appealing.

Distributed computing has shown itself to be particularly useful for moving around large files.  In Twitter's case, you're working from the opposite extreme: processing a high volume of tiny messages.  This screams centralization.

Centralization is not the reason Twitter can't scale.  They can't scale because, well, they just haven't written an architecture that scales.  Granted the mobile space is still largely DIY, so they have to roll much of their own code.  That's really a pretty lame excuse, though, especially given their resources and the time they've had to figure it out.  My buddies over at Mobile Commons face similar issues processing huge volumes of mobile messages, and they don't have these issues.  I'm convinced Twitter would be flying if Ben Stein and Chris Muscarella at Mobile Commons were in charge, and I honestly think it's because they're just better programmers.

I'm a big Evan Williams fan and was thrilled to meet him for the first time down at SXSW, but that doesn't mean I have faith in his or his team's ability to learn how to scale a complex architecture overnight.  A distributed architecture would make their task orders of magnitude harder.  Don't do it fellas.

In recent years, the web has given rise to a significant amount of projects called distributed computing projects which utilize the processing power of thousands of computers world wide.  The problems being addressed range from modeling complex proteins, to looking for efficient ways of producing hydrogen as an energy source and even projects like SETI (Search for Extra Terrestrial Intelligence).  Though these organizations are all unique by nature, they're each tapping into the most abundant and underutilized source of computing power possible, idle desktop/laptop computers.  Masses of computers are required since complex calculations of this variety would take several days if tackled by only one or only a few machines.  It follows that this provides an excellent route for the average Joe/Jane to make good use of a computer that they leave running.  This is an emerging form of digital philanthropy, only possible in modern times.  In short, if you're looking for a way to donate time, energy, and resources but find yourself lacking on all of the above, this is your best solution.  If you own a computer, you now have one less excuse.

Obviously, as a bit of an Earth Day plug, I must stress that turning your computer off is as valuable (or more) as contributing to programs like I've just described.  That said, if you're staunch about keeping your PC's gears spinning, I must urge you to do so with awareness and consider contributing to one of these projects.  Macs and PCs alike have software readily available and if you happen to have a fancy dual core processor, you can contribute two-fold :O.  Find a project that matters to you or you find interesting and join the cause.  It's as easy as installing a screen saver and as simple as launching a program as you jet out the door.  Wikipedia can aid you in finding a flavor that suits you best.  Another list of projects is here if you're interested.

Stanford's folding@home project as described by Dr. Vijay Pande.

Perhaps we could go a bit further than simply contributing on a personal basis.  What if the hundreds of campus computers (many of which are running 24/7) could contribute to a program like this?  Why is the establishment free to ignore such resources?  As aware students, we need to keep not only ourselves and eachother accountable but also our organizations.  Our University honestly needs to be paving the way in such arenas.  The time is ripe to take action and have our voices heard.  I propose that we make a push to see that ALL campus computers remain off or contribute to such projects during all off peak hours.  Change won't just happen without an effort.  If anyone agrees with my sentiments I urge you to comment or email me.  Let's see if we can build some momentum here.

• corrected cybenode.sh script to properly accept parameter designating codebase location
• JMX interface added to Monitor, Cybernode and Webster core services
• drag and drop of oar files onto console service graph to deploy services added
• corrected oar deployment code to handle operational strings with Include tags allowing deployment of dependent operational strings.
• removed old viewer and opmon utility code, documentation and build file references.
• changed helloworld example files and documentation to consistently use steps 1-6.
• new webster using MINA 1.1.4 (JDK 5 version) and integrated it
• moved jini ServiceStarter class to assimilator and modified it to add loggers from top level config file (we had a really hard time get jini loggers to turn on properly before and this should fix it), quite a bit or exception refactoring and other general refactoring
• moved all the script and config files to use the new service starter and new webster.

The new distribution file can be downloaded here.

In the works is a network overlay for Assimilator so that distributed services can be managed over the internet.

Binary is more efficient for serializable objects. There is less transformation required at endpoints before and after transferring data around. Less code and libraries are required to develop and deploy the distributed components of an application. Maintenance of the code is typically easier. And more sophistication can be applied in the passing of serializable object as method parameters, return values and exception handling information. Java RMI for example with its use of stub/skeletons provides the mechanism to invoke functions on remote objects which is truly more powerful than making http requests with XML data.

I suppose it's likely that XML is just easier to use for now. The tools and IDE plugins were easier to develop and came faster than any tools to support building applications with remoting capabilities. Http and XML are just easier to deal with than configuring EJBs. Although EJB3 components are now much easier to deal with since the wiring is largely defined using annotations.

From the perspective of distributed computing being web servers scattered everywhere, it's likely XML and RESTful style web services will continue to dominate. True next generation distributed computing capabilities where all computers are peers will require greater sophistication to handle mobile code, mobile services, service discovery and the transfer of data between the services. It's not going to be HTTP and XML!

Entweder nimmt baut man dann eine riesige Serverfarm auf, oder man teilt das Projekt in unzählige kleine Stücke und verteilt sie auf viele andere PCs.

Nach der zuletzt genannten Methode geht das BOINC-Programm vor. Nach der Installation von BOINC kann man sich eins oder mehrere der vielen Projekte aussuchen, welche man unterstützen möchte. BOINC verbindet sich dann selbstständig mit den ausgewählten Projekten und lädt sich die zugewiesenen Einheiten (Units) runter. Dabei braucht man nicht mal Angst um einen lahmenden PC zu haben, denn BOINC nutzt nur die ungenutzten PC-Ressourcen. Das heißt, sobald man irgend etwas startet, was viele Ressourcen verbraucht, gibt BOINC diese auch augenblicklich wieder frei. Eine Übersicht aller Projekte findet man unter folgendem Link.

Wenn ihr euch nun entschieden habt auch Crunsher bei BOINC zu werden könnt ihr hier eine Installationsanleitung finden. (Ab Punkt 2.)

Ich persönlich lasse BOINC gerade für das World Community Grid laufen. Auch empfehlenswert ist Rosetta@Home, bei welchem ich auch schon gerechnet habe. Bei dem Projekt geht es um Protein Folding, die Erforschung der 3D-Struktur der Proteine. (Dadurch könnten Krankheiten wie etwa Krebs verhindert werden).

Viel Spaß

~Wegi

But still, there's a lot of interesting content in the talk.

I think the most interesting idea is how using a larger but slower storage system can still provide a faster overall solution thanks to opening up a new time speed trade off, provided we have enough bandwidth.

But let's take the premise and run with it: what can we do to make a clustered disk array look like a single large smp machine with a very large ram? The talk shows us that we can easily equal the bandwidth of a single ram bus. But what about the latency? Cooperman resorts to a lot of algorithms specifically tailored to each application to tolerate the latency. Can we do something simpler and more general?

Watching the video I immediately thought of the old Cray/Tera MTA architecture, which used a large number of hardware threads per core to hide memory latency. If a memory access takes 100 cycles, then if we run 100 threads at the same time, switching between each in turn, we maximize throughput. To each thread it's as if it's running on a processor with 1/100th the instruction rate and a 1 cycle ram access latency.

Could we use the same idea with a virtual machine on a cluster? Well, the the back of the envelope says that we'd need nearly 300,000 total outstanding io requests to hide the latency gap. That's a lot. Maybe even more than we can break the problem into. But, with a disk we can access a block of words for nearly the same cost as the first byte, so if we can exploit any locality of access we should be able to bring that down. On top of that, we can use ram as a cache to avoid some hits altogether. Could this be enough to bring the total outstanding io count in reach of the couple hundred disks we might have in our cluster? It'd be interesting to see. If I were going to try, I'd look at using ZFS for it's sorting of disk accesses.

I think on the whole I'd rather use an API like Map/Reduce, or a scripting language built upon it like Pig or Sawzall. These tools have a more limited focus than Cooperman's group, but are likely a lot easier to hit the ground running with to get something done. It'll be interesting to see if Cooperman's group can come up with language tools that are more general than Map/Reduce, but still offering a simple development model.

I also can't help but mention that one of my personal pet interests, Array Languages, could make very good use of a high bandwidth disk array.

The open source environment allows free usage of the platform and encourages feedback from users on how to enhance features, extend capabilities and eradicate defects. Open source allows greater distribution of the platform hopefully leading to wider adoption of its use. Custom applications built with the platform will demonstrate platform capabilities and provide a means for individuals and organizations to generate revenue.

The primary goals of the project include:

• Develop a distributed computing platform capable of dynamically managing and monitoring services in a network.
• Develop the platform in and for the open source community to drive adoption of its use.
• Develop the platform with the intent that it can be used for easy development of distributed applications.

Network includes running across the internet and on any device connected to it. The current version executes in WANs and work is underway to get it working across the internet. At which point it becomes feasible to start building social network applications and gaming environments that utilize available resources across peered devices.

• supporting the knowledge life cycle, including articulation, organization, sharing and exchange of information;
• the management of all relevant desired  information through linking, browsing and tagging;
• knowledge communication across distribute social networked communities with capabilities for distributed search, storage and massive scaling.

It is an open source framework that can accommodate plug-in capabilities for contributions for those who wish to develop this kind of social networking capabilities. I have similar goals for Assimilator  which is a  platform for managing distributed services; services that can migrate, work in groups with dependencies and can be used to build knowledge management tools for social networks.

Achieving the goal of allowing individuals to manage their own information in their own context using their own personal tools will require a lightweight framework that accommodates and manages services or small components which collaborate in a dynamically changing environment  distributed across the internet. I believe NEPOMUK is on the right track. I hope I am as well.

It could be a location on your own personal laptop or one location hosted by your favorite ISP. It would be your collection of information, relevant to you, organized according to your preferences. Everyone already does this with their belongings. Why should I have to place my things someplace outside of my control just to share them, when I can define a link to the definitive resource?

Having your belongings scattered about across many websites is cumbersome and difficult to maintain. It's much easier to have it generally located in the same place. I have all my stuff where I want it, you have yours where you want it.

Now I can share some of my stuff. I share only what I want to. The rest is private. Essentially each individual defines links to the information they want to share with others. Each individual presents a point of view of who they are. Nothing new here, we all do this in real life already. But remember I'm talking about information on my laptop.

Next, friends share information with each other about where their stuff is accessible. It would be a portal to their information. Now friends can see the presentation of other friends. They can see each others point of view or perspective. Each persons collection of information is still controlled locally but accessible between each other.

There will likely be common interests between friends. They could start a topic of interest. A set of information of similar interest. Essentially it is a collection of links back to the original information. Friends discover there is information about others being shared. Those friends can establish a group. They share similar interests, similar topics of interest.

So now individuals share their point of view, create topics of interest which others can contribute to and they can create, join or leave groups. Others may discover topics of interest and subsequently join groups base on similar interest. This starts to sound more like a real social network which individuals create from their perspective all from there own location, not across many different web sites. There are no accounts, no sign-up, no limitation on how information should be place, categorized or presented.

The concept of groupings can then be extended collections of groups. These would be communities. Groups would join and leave communities. Individuals would come and go as desired from the groups. It is very dynamic. The thing that always remains is the presentation of an individuals point of view and all the corresponding information that belongs to it.

All the information comes from individuals just as all the information on the internet today is posted audio, video, photos, text, etc. It is shared and discovered across peers rather than on web sites much like P2P sharing, but sharing links rather than files. Information will be tagged in multiple ways depending on its context and meaning. Hierarchical referencing should make it possible to access information of interest from multiple different points of view rather than from one perspective. Redundancy of information with multiple links of access discoverable across a distributed network approaches the mechanism required for universal memory. There are multiple entry points and multiple pathways to get to any bit of information.

請支持Team HongKong (Team ID 32535)。按這裡看看參加辨法。PS3參加辨法在這裡。無論你的是Windows, Linux, Mac or PS3，都要支持我地呀！

「Folding@home」是由美國史丹佛大學所主導的分散式運算研究計畫，透過電腦的模擬運算來研究蛋白質折疊，誤折，聚合，藉此解開包括阿茲海默症、狂牛病、帕金森氏症...等疾病的成因與療法。Folding@home 透過網路將模擬所需的計算分散到參與計畫的電腦上執行，原先支援的平台包括 PC Windows、Mac OS、Linux 等。後續史丹佛大學與 SCE 合作，於 2007年3 月 22 日推出 PS3 版終端軟體，藉由 PS3 Cell 微處理器的獨特架構與高浮點運算效能特性，可提供較一般 PC 高出 10 倍以上的運算資源。

根據 SCE 方面的資料，截至 4 月 26 日止，已有超過 25 萬 PS3 使用者參加此一計畫，提供超過每秒 400 兆次浮點數運算（TeraFLOPS）的運算資源，使得該計畫瞬間總運算能力超越每秒 700 兆次浮點數運算，是 PS3 加入前的 2 倍以上。因 PS3 加入所產生的連鎖效果，也讓參與的 PC 數量增加了 20 ％。