New Technique for Estimates Relating to Massive Quantities of Data

Brown mathematicians prove new way to build a better estimate

phys.org


Brown University mathematician Charles “Chip” Lawrence and graduate student Luis Carvalho have proved a new way to build a better estimate to answer the question, "How do you sift through hundreds of billions of bits of information and make accurate inferences from such gargantuan sets of data?"


"For more than 80 years, one of the most common methods of statistical prediction has been maximum likelihood estimation (MLE). This method is used to find the single most probable solution, or estimate, from a set of data.

"But new technologies that capture enormous amounts of data – human genome sequencing, "Internet transaction tracking, instruments that beam high-resolution images from outer space – have opened opportunities to predict discrete “high dimensional” or “high-D” unknowns. The huge number of combinations of these “high-D” unknowns produces enormous statistical uncertainty. Data has outgrown data analysis.

"This discrepancy creates a paradox. Instead of producing more precise predictions about gene activity, shopping habits or the presence of faraway stars, these large data sets are producing more unreliable predictions, given current procedures. That’s because maximum likelihood estimators use data to identify the single most probable solution. But because any one data point swims in an increasingly immense sea, it’s not likely to be representative...

"Lawrence and Carvahlo used statistical decision theory to understand the limitations of the old procedure when faced with new “high-D” problems. They also used statistical decision-making theory to find an estimation procedure that applies to a broad range of statistical problems. These “centroid” estimators identify not the single most probable solution, but the solution that is most representative of all the data in a set."

Wall Street Journal Article on Correcting Business Students' English---My Idea, Take them to Bars

How Students From Abroad Learn to Talk the Talk


"Now, North Carolina is rolling out a program called Honing Executive English Language Skills, or Heels (a catchy acronym to Tar Heels fans). It requires all of its foreign M.B.A. students -- more than a quarter of the class -- to take an oral and written test when they arrive on campus. The students are rated on a nine-point scale based on their accent, pronunciation, grammar, vocabulary and other factors. To boost their scores, students can pay a fee to take special classes taught by linguists that focus on speaking English in a business context...



"In the HEELS classes at North Carolina, students are grouped by their native regions, such as Western Europe, East Asia, South Asia, Africa or Latin America, because they tend to share similar problems with accent and pronunciation. The program also addresses nonverbal communication and body language, which may vary from culture to culture. The classes attempt to change behavior that might be misinterpreted by U.S. managers, co-workers or clients. "For example, a U.S. recruiter would expect direct eye contact and a firm handshake," says Mindy Storrie, interim director of Kenan-Flagler's career management center, "but that isn't a universal norm in other countries' business dealings."...


"In addition to the English classes, North Carolina offers courses for international M.B.A. students on American culture -- from sports and entertainment to the origin of slang expressions -- and on U.S. business communication, including practice exercises for impromptu speeches, team presentations, boardroom pitches and employee performance reviews...


"While North Carolina's courses are among the most comprehensive, other schools are also expanding programs to help foreign students prepare for careers in the U.S. The University of Rochester's Simon Graduate School of Business, where nearly half of the M.B.A. class is international, offers an English-language and U.S.-culture program that includes language instruction and trips to museums, theaters and sports events."


Here's my take. These foreign business students are, in a sense, trying to create models of what an American business co-worker is inside their heads. Once they do that, they can imitate that model and thus build rapport. The problem, from my perspective is that they are trying to model "half" a person. That is to say, they are trying to build a model of people by only looking at one side.

This idea was triggered by the last quoted paragraph: talking about taking them to museums, theaters, and sports events. This struck me is not a very effective way to achieve the desired goal. Better would be to take them to bars and allow them to see something of the "other side" of Americans.

One could argue that bars are themselves a somewhat artificial venue, but the point is, it is difficult to inject foreign business students into America in a real way. One key element of a bar is---it is publicly accessible. I would send in two-person teams consisting of one foreign business student at one American "expert" in bars. The job of the American would be: first of all, to keep the person safe; secondly, try to prevent too much emotional damage; and third, provide an opportunity for real exposure to Americans completely outside a business context.

It has to be one foreign student at a time or else they will circle the wagons.

Mossberg Says Amazon ebook Delivers Books---Short on Software

This is referencing a Mossberg article in the Wall Street Journal of the past couple of weeks. Probably around November 19 when two other articles on the subject appeared in WSJ, but I can't find the specific Mossberg reference.

Here is the essence of the matter. Mossberg checked out the new Amazon e-book called Kindle, or something, and found its access to books and periodicals was pretty decent. In addition, he found that reading it was like reading a piece of paper. (It uses the same technology as Sony e-book which apparently is for rent from the company that created it.)

On the other hand, Mossberg hand the product because the software was inadequate. My point is, what does this mean? Sony has already done a fairly good job on the software, I would set the solution of that problem is really quite trivial.

In other words, the demise of the book is in fact at hand.

Yes, there have been false alarms, but the technology is finally ready. Books may persist for a while in the form of novels were perhaps treatises on philosophy, that sort of thing. But in terms of sources of information, the book is clearly on its way out.

The implications are interesting. Of course, there will be "books" of information on various subjects that will be downloaded to e-books. unlike current paper books however, these books will be linked via the Internet to other sources of relevant information. So the entire concept of a book will to a significant extent, become much more fuzzy. A new group of words will develop to describe these new types of "books." "Books", will probably be much more of a collaboration in the near future than they are now. Getting credit for one's ideas and words will still be important, and authorship labels will still accompany the various pieces of text that will conglomerate to become a "book." But I don't think any one person will be controlling the boundaries of "books" anymore. Rather, I imagine that there would be multiple editors, each defining his or her version of the "book."

In addition, new editions of the books can be created virtually at any time and the frequency of new additions would be unrestrained by anything. Reminds me of some of the people who publish regularly on the Internet, who include a time of day with the article as well as a date. cat in the near future, there will in many cases be more than one addition of a book within a single day simply because an author has found new information and added it.

Wikepedia Vulnerable to Malware, According to McAfee

"Organized Crime and the Internet," McAfee Virtual Criminology Report, http://www.mcafee.com/us/local_content/white_papers/threat_center/wp_virtual_criminology_report_2007.pdf


"Just like with social networking sites like My Space, the very openness of Wikepedia that allows its users to freely add or edit available content has made it an attractive target for virus authors to plant malicious code in articles. In October 2006, a piece on the German edition of Wikepedia was rewritten to contain false information about a supposedly new version of the infamous Blaster worm, along with the link to a supposed fix. In reality, the link pointed to a malware designed to infect Windows PCs.

An e-mail was also mass spammed to German computer users requiring them to download a security fix. The e-mail was crafted to supposedly appear from Wikepedia, complete with an official with an official Wikepedia logo."

Web Site Lets Bloggers Make Their Own Ads for Big Company Products

"Nescafé Brews Buzz Via Blogs," WSJ, P. B3 11/23/07

[a French website]

"BlogBang [BlogBang.com, a website with 2,000 bloggers as participants] also has tried to draw bloggers into the creative process. Companies looking for new ways to pitch their products can post requests for bloggers to develop their own campaigns. BlogBang's members can then put their homemade ads on the site. The one that gets the most clicks is spread around the bloggers' Web sites, and the author of the winning ad earns a fee. Garnier, a division of cosmetics group L'Oréal, advertised its Fructis hair gel this way. Garnier couldn't be reached for comment."



http://insights20.com/2007/07/09/blogbang-a-network-for-consumer-generated-advertising/


"Blogbang: a network for consumer-generated advertising
Published July 9th, 2007 in blogging.

"My friend Julien Braun just launched in France a new ad network called BlogBang . BlogBang is a marketplace for advertisers, “creatives” and publishers (bloggers).

"An advertiser submits a creative brief, “creatives” (consumers, agencies, etc) generate their own commercial (mostly video), which show on the publisher’s blog. Example here or here. Publishers/bloggers can pick ads that might work well based on their editorial content. An optimization system (I am not sure how it works, but I think something like advertising.com) determines over time which ads work better for each blog.

"The interesting part is that creatives and publishers split the revenues (with the ad agency of course). And anyone can be a creative and try to compete with the professional agencies to create the most efficient ads. As opposed to traditional creative contests, which are judged on their creative qualities, the best ads here are those that perform better. So, consumers who understand how to create ads that would perform well with a particular audience, on a particular blog, etc, can get a share of the action and help publishers make more money with ads that perform better.

"Julien’s philosophy is that each media has its own advertising format (30 seconds for TV etc) but that social media has not found yet a format that works well. By letting users create their own ads, advertising becomes social and, while serving the advertiser’s brief, has a human voice."

Breakthrough Idea: How to Prototype Software

Been thinking about this one for a while, and I got it.

Consider a software group writing a program for in-house use. The biggest problem is communication between the software group and the users. Creation of a dumb prototype is a well-known attempt to solve the problem. However, as I envision the creation of the prototype, I can see many problems potentially developing.

Essentially, time must be invested, people from two different departments must communicate, tensions may arise, and ultimately, the task of prototyping may be abandoned rather too easily at any time. The software group would like to abandon the task because it's not the sort of thing they like to do: they want to get on with writing the program. The users would like to abandon the task because they do not enjoy interacting with the software group.

In addition, the users have been assured by the software group that the writing of this software is really quite a simple and easy task that does not require the burdensome process of writing a prototype---imposed by the rather stupid types in management, at any rate. Naïvely, viewing the whole situation as a "simple, straightforward technical problem", the user group alludes with the software group in prematurely terminating the task of prototype-building, which in any case seems to be inordinately difficult for reasons that no one can quite understand. Happily, once the prototype task is abandoned, the software group can get down to the real business of writing the program and the user group can get down to their real job whatever it happens to be.

Unfortunately, the reason the prototype was difficult to write was because the task of the users that the software is designed to integrate with---a task which seems so simple to the software engineers that only a moron could be doing it for a job---is, in fact, complicated in a variety of subtle ways. Not to worry, all of the subtleties will turn up---later, after the software is written and tremendous work and effort has been put into doing so.

Okay, that's the problem. I know what you are thinking, "What a brilliant formulation of the problem!" However, let me modestly point out that the formulation of the problem is fairly obvious and well known.

Okay, here's the solution: we create super simple, super user-friendly prototype building software designed explicitly for non-programmers (i.e. "users") to use. Included with the product would be a "manual." (see, "Software; Ancient History in your encyclopedia for an explanation of this term.) However, instead of a nice, bound booklet with high-quality color illustrations, this manual would be online, that is to say it would be a living document.

Different versions of the manual would be written for people in different professions, the point of view of the manual being the process of getting some particular task done such as lawyering, being an accountant, or more specifically, deciding who gets to take out a loan to buy a house, helping someone write up a tax return, designing a of building out of steel, designing interior components like door handles for a car, etc.

Instead of a carefully crafted document designed to last for the ages, I envision a series of down- and-dirty versions of the manual. Scratch that, what I mean is there would be a core manual describing operations and then a variety of intros aimed at different professions. The idea being to not only tell people how to use it but to convince them that it's important to use it. There would also be a general intro for everyone, describing the perils of writing software without an effective prototype.

I Predict Google Cell Phone Network

This information (aside from my clearly stated opinions) is from an article onpage B1, Wall Street Journal today (11/16/07)

Google has set up---with approval from the FCC---a small private cell phone network on its campus. The Wall Street Journal describes it as a "advanced high-speed wireless network." They are preparing to bid for a chunk of the wireless spectrum, though they have not officially confirmed their intentions to make the bid. (Yet it is obvious that they will bid.) Apparently it will be bidding $4.6 billion. It also appears that they will be bidding without a partner, in part to give them more flexibility in their bidding strategy -- which is being crafted by specialists in game theory. It is also said they are afraid that if they partnered with some people it would offend the people that they did not partner with. In addition, Wall Street seems eager to lend them money so that makes that part easy.

Part of what they're going after is 700 MHz, a type of signal that can travel very far between towers and therefore needs less towers.

A few days ago, the story came out about Google's new open platform for cell phones, called Android, I think. They have a Taiwan manufacturer preparing to make the cell phones, and I guess a few of them were made because they are already using them on their campus.

So far, the WSJ implies that it is unknown as to whether Google will make its own network but it seems like a certainty to me. Interestingly, Google applied a fair amount of pressure to the U. S. Government before this auction and succeeded certain conditions being placed on it. The idea is, that anyone who wins spectrum in this auction has to make their network open to other companies. I don't really know what that means but it is interesting. Now some people say Google may feel that they should be in the auction, having pressured the government and gotten their way on this.

Others say they will be afraid to alienate their telecommunications partners, but this seems pretty ridiculous to me. In addition, many people think Google will not be smart enough to put together a network, also highly unlikely. In addition, there are fears of dispersion of management focus. Usually, as I have repeatedly mentioned on this blog (Wal-Mart and others) -- I support the concept of maintaining management focus and oppose efforts that seem to dissipate it. However, in this case, I have a feeling that Google will be able to succeed in this venture without dropping the ball on search engines and advertising on the Internet.

It is Google's time; they are seizing the moment.