More ESRI and Twitter

A while ago I posted about a map from esri.com that incorporated social media such as Twitter with more traditional sources of fire information.  My conclusion was that the social data was not that helpful.

ESRI has just put out another set of maps, this time on acceptance same-sex marriage.  The first map uses a proprietary demographic model to try to rate how likely a given county is to favor same-sex marriage.  As far as I can tell, this model isn't particularly based on, say, polling data or election results from ballot measures, but more on factors like how urban or rural the county is, how many people have gone to college and so forth.

The second map shows state laws.  It corresponds roughly with the first map.  With the interesting exception of Iowa, the first map shows significant support where same-sex marriage is legal, as one would expect.

The third map is based on Twitter data.  It says there is strong support for same-sex marriage across the country, opposition in five states (including Minnesota, despite significant support in the populous Twin Cities), and very strong opposition in exactly one state: North Carolina.  Idaho, Wyoming and Vermont had insufficient data.

The support map shows North Carolina as fairly similar to neighboring Virginia, which the Twitter data shows as strongly supportive, and as more supportive than neighboring South Carolina and Tennessee. The Twitter data show both of those states as moderately supportive.

Clearly something is out of line here.  Two possible explanations:

• Concerning the overall map, the Twitterverse is not a representative sample.  Overall, Twitter traffic is much more supportive of same-sex marriage than the country as a whole.  This probably shouldn't come as a surprise.
• Concerning North Carolina, the Twitter data covers May 9 through June 30, 2012.  As the map explanation notes, North Carolina had voted on May 8 against a proposition supporting same-sex marriage, by a roughly 60-40 margin after a very intense campaign.  It would be interesting to know what portion of the Twitter traffic surveyed is from the immediate aftermath of that election.  My guess would be a large portion.

Along with the fire map I mentioned above and a study on Twitter rumors after the London riots, which was presented in the Guardian as confirming the notion that Twitter is a good, self-correcting source of information but in fact shows anything but, this is the third piece of evidence I've run across suggesting that you should treat any inferences drawn from Twitter traffic with a grain of salt.

Twitter may well be a great way to find out what people, at least those with Twitter accounts, are paying attention to at the moment, but it's risky, to say the least, to draw conclusions about objective facts from that.

Building a better password

I've recently complained about the irritating nature of the password strength checkers that have been popping up everywhere, so I feel obliged at least to try to analyze the problem and offer solutions.  This is leaving aside the question of whether password authentication is a useful approach at all.

Fundamentally the real measure of password strength is how many passwords you'd expect to have to guess in order to get the right one.  A more formal version of this is the notion of bits of entropy.  If you had a list of all possible passwords in your scheme, I could identify any particular one so long as I could get answers to a series of yes/no questions, for example:  "Is it in the first half of the list or the last?",   "Is it in the first half of that half or the last?" and so forth.  The number of such questions I need is the number of bits of entropy.  Twenty questions means twenty bits, etc..

If I know that your password is either "0" or "1", you have exactly one bit of entropy.  If I know it's an uppercase letter, lowercase letter, digit, "$" or "%", there are 64 possibilities, so you have 6 bits of entropy.  If I know it's two such characters, you have 12 bits, and if it's eight such characters you have 48 bits, which is not too bad.  Someone trying to guess your password would have to guess about 140 trillion passwords, on average, before stumbling on yours.

[Don't assume that guessing a password requires typing it in to the same text box you have to use.  If someone steals the right data from your service provider, they can throw as much computing power as they've got at guessing the passwords.  Quite possibly they'll be happy enough just to try a few thousand weak passwords for each account, since that will crack depressingly many, but attacks like running through the OED with simple substitutions of letters for numbers are absolutely feasible as well.  Here's an article from 2012 about hardware that can guess 350 billion Windows passwords per second.  48 bits suddenly doesn't seem like so much.]

This is assuming that you picked eight characters at random.  If I knew instead that your password was either "F1%ldN0t3$" or "sasssafras" (maybe I'd watched you read your password off a piece of paper with only those two words on it but couldn't quite see which you were typing), then you have only a single bit of entropy, even though both passwords are not just eight but ten characters long and one has plenty of non-letters.

More realistically, if I knew you'd picked an uncommon English word and maybe changed some of the letters to numbers, you'd have somewhere around two dozen bits of entropy.  That's not trivial, but keeping in mind that each added bit doubles the number of passwords a cracker has to try, it's tens of millions of times weaker than the 48-bit scheme above.

The fundamental flaw of password strength checkers is that they can only look at the password you gave them.  They have no idea what other possible passwords you might have chosen.  The assumption is that if you're forced to jump through enough hoops you'll be forced to expand your parameters, but in fact it's possible to generate passwords in a secure manner using only letters, and or to generate them insecurely to satisfy any strength checker out there.  Which is why I half-grimace, half-laugh when I see the "password strength indicator" jump from "poor" to "great" as soon as I type a number.

Now, it's perfectly possible to generate completely random 8-character passwords.  The problem is that something like "qcrQf1x2" or "u%js%hPQ" is a pain to try to memorize, so most people will fall back to picking a "hard" word and maybe altering it a bit.  However, as xkcd points out, it's possible to do a lot better by using random short words.

For example, here's a kind of clunky way of producing a random, memorable password:

BIG HONKING DISCLAIMER: This is just for demo purposes.  The second site I mention uses http, not https, so in theory anyone could be looking in on your session.  Even with https, the sites might be logging all your traffic.  I personally seriously doubt they would, and it's hard to imagine they would be able to connect the dots and figure out what you were using the generated password for, but if you really want to be on solid ground, get the source, look it over, run it locally and use something like /dev/urandom or D&D dice to generate the random input (15d10 will give you close to 50 bits ... not that I would have any idea at all what "15d10" means).  There are also smartphone apps that do more or less the same thing, I believe.

[I last checked that this recipe worked on 19 March 2013]

With that out of the way:

• Go to this site and copy the random string you see there (e.g., A6727933B0169E89).  To get more randomness, just reload the page.
• Go to this site.
• Type some short number and a space into the Challenge box and paste the random string from the first step in after it (e.g. 123 A6727933B0169E89)
• Type anything at all into the Secret box (e.g., "secret").  This doesn't have to be hard to guess.  The real entropy is coming from the random string.
• Press the Compute with SHA-1 button.  Again, the cryptographic details of how strong SHA-1 is don't matter here.  You're just converting a random number to short words.  A simple table lookup would do just as well.

In the Response box you will see six short words followed by some hexadecimal gibberish (in this case, LANE RICH ACE HIS TWIN BLUE (AA59 E401 0D7F 1CB3)).  Each of those words represents 11 bits of entropy (technically, slightly less, since we only started with 64 random bits, but who's counting?).  Take at least four, preferably five or all six.  If you don't like those words, just try again with fresh random bits.

Then add a random punctuation character or whatever makes your site's password strength checker happy.  Voila!  Your password is now richtwinacelane5 or whatever.

If your site's password checker imposes an 8-character limit (and, incredibly enough, some do), cry.

Curiosity online

The Curiosity rover, formally the Mars Science Laboratory, landed on Mars late on August 5th (or early on the 6th, depending on your time zone).

This was a major accomplishment.  Mars has a habit of eating space probes.  The majority have failed, sometimes quietly and sometimes spectacularly, at least once due to the kind of basic coding error that would send a regular geek cursing back to the keyboard.  Except when it comes to interplanetary travel there's generally no "next release".

The Curiosity landing was more complex than any Mars landing before it, and parts of it had never been realistically tested, much less the overall sequence.  This was not out of negligence or cut corners but a simple necessity.  Mars's atmosphere is much, much thinner than Earth's.  At its thickest it's equivalent to Earth's atmosphere at about 35 kilometers (about 20 miles, or about four times as high as the summit of Mount Everest).  Mars's gravity is about 40% of Earth's.  There's simply no practical way to re-create those conditions at the necessary scale anywhere near Earth.

This thin atmosphere is a real problem.  There's just enough of it you can't ignore it, but not enough for a parachute to take the lander all the way to the surface safely.  To land Curiosity, which is considerably larger and more massive than the Spirit and Opportunity rovers before it, NASA put together a landing sequence that Rube Goldberg might have appreciated:

• The probe slams into the atmosphere, protected by a heat shield and pulling upwards of 10g of deceleration, possibly as high as 15g (This is from NASA's press kit from before the landing.  I don't know what the actual numbers were, but clearly it's not going to work with humans aboard.)
• Once the the probe has slowed enough not to need the heat shield, the heat shield is jettisoned.
• The probe then deploys a parachute, which slows it to about 300 km/h (about 180 mph).
• The parachute is then jettisoned and a rocket-powered descent stage takes over, carefully avoiding the parachute and the back shell it's attached to.  If the rockets don't work, the probe will hit the surface, hard, in about 20 more seconds.
• The descent stage lowers the rover on nylon cords while it slowly descends, since landing completely under rocket power risks kicking up enough dust to damage the rover.
• While the descent stage is doing this, the rover gets its landing gear into place.
• When the rover is safely on the ground, the cords are cut loose, using explosives, and the descent stage flies off to crash land some distance away.

What could possibly go wrong?  Mind, this is a simplified description (and any inaccuracies are mine).  The full details include several more maneuvers, through six different configurations in all, with 76 pyrotechnic devices, ballast jettisoned at various points and dozens of people sweating bullets in the control room and elsewhere.

"Seven minutes of terror", they called it.  Damn impressive engineering, I call it.

OK, so now that we've paused to admire NASA/JPL's chops, what does this have to do with the web? 

I'm old enough to sorta kinda remember Apollo and grainy black-and-white TV coverage.  And Tang.  Cool, and, it must be said, more culturally significant than today's missions, but technically not even close to what we have today.  This is true not only of the vehicle itself, but of the communications technology.  In the 70s we had grainy black-and-white video.  For better imagery you had to wait for the astronauts to bring back the film.

Now, using essentially the same technology as a cell phone camera, NASA is able to capture digital images and put them up on its web site for the world to see immediately (as many have pointed out, much more quickly than NBC saw fit to broadcast Olympic events).  The web gallery includes not only the pretty press-release pictures but also the raw frames they were made from, including pictures of Curiosity staring at its feet and other such that didn't make the cut.  There are plenty of other goodies as well.  I particularly like the "white-balanced" images, which have been post-processed to show what the same terrain would look like under sunlight on Earth.

And there are, of course, a Twitter hashtag (#MSL) and (obligatory plug) updates on NASA's Google Plus page.

As inspiring as it may have been to watch thrilling news coverage with Walter Cronkite narrating, there's something much more intimate about being able to visit a web site from time to time and watch the story unfold directly from the source.

Two more comments on wild and wooly markets

1. Intrade and Bitcoin are two entirely different markets, with little in common beyond operating on the web and more or less separately from the more conventional financial markets.  I don't mean to imply anything more than that by mentioning them in the same post.
2. Much of the recent excitement about Bitcoin has been due to its value increasing sharply lately.  Its value, that is, in dollars.  Just sayin'.

What's twice as big as the internet?

(Yikes ... I went 0 for March!)

I've mentioned before that telescopes can generate a lot of data.  IBM seems inclined to drive the point home by collaborating with ASTRON (the Netherlands Institute for Radio Astronomy) to put together "exascale" computing horsepower behind the world's largest radio telescope.

The telescope is actually (or rather, will be) an array of millions of antennas spread out over a square kilometer, from which the name SKA, for Square Kilometer Array.  This array is expected to produce on the order of an exabyte of data per day.  This is an absolutely ridiculous amount of data by today's standards.  Think one million terabyte disk drives, or twenty million feature film's worth of Blu-ray, or ... according to IBM, twice the daily volume currently carried on the internet.

I'm a little skeptical as to exactly how one measures that, but hey, you've got to trust a press release, right?


So where do you put an exabyte a day worth of data?  Well, you don't.  You're certainly not going to upload it to the web.  Particle physicists are faced with the same problem of having to figure out what portion of a huge data set to keep for later analysis, and a large part of running an experiment is setting up the "trigger" criteria by which the software collecting the data will decide what to keep and what to throw.  IBM and ASTRON's system will be dealing with the same problem, but on an even larger scale.

Or I suppose you could sign up two million people and somehow stream an equal share of the data to each at Blu-ray resolution all day every day, but somehow I doubt that kind of crowdsourcing will help much.

Dear Reader,

When I was in school (a while ago, admittedly), we learned how to write a letter:

Dear So and So, 

Blah, blah, blah ... 

Sincerely,
David Hull

(or, with a close friend or family member, "Love, David" or "Yours, David", or such)

In email these days, if there's any salutation or closing at all, it's something like:

Hi So and So, 

Blah, blah, blah ... 

Thanks!

Some of this, I'm sure, is because times change.  The canonical letter of my youth is miles and miles less formal than one from, say, the 1700s.  Nonetheless, I think the switch from snail mail to email has had an influence.

For one thing, it feels a little funny to add a salutation and closing at all, when the email header includes the names of the sender and recipient.  On the other hand, it can feel a little funny not to have at least something.  So, I think, we tend to revert to what we'd say in a conversation, with both participants standing right there and knowing who the other is.

Beyond the switch to a new medium giving an opportunity for new forms, the medium itself influences the forms.  This has been true with letters as well.  Formal letters with half a page of salutations and closings make more sense when hand-writing and sending a letter is difficult and therefore reserved for occasions like petitioning the kings court, where you are quite possibly addressing an official you only know by name and trying to establish your reputation by means of any titles you may have.

Y'r Humble & Ob'd't S'v't
Etc., etc.

Xanadu vs. the web: Part III - Xanadu the business model

Any publishing scheme has to solve two basic problems: How do you get the content out to people -- or perhaps more importantly, how do you get people to the content -- and how does everyone, particularly the creators of the content, get paid.  Granted, there are any number of reasons people will create content for free (as with most blogs, for example, this one very much included), but to get anything going on a serious scale there will have to be money involved at various points.

Technology has steadily made it easier to disseminate content, to the point where any random blogger now has a potential readership in the hundreds of millions, but it hasn't made it any easier to find readers.  Most blogs have a readership considerably below a hundred million.  Finding readers requires editing, to choose what to publish and make sure it is of good quality, and marketing to make people aware of what you're publishing and try to get them to read it.  Publishing is a different thing entirely from printing.  Publishers, in this broader sense, have to exist in any viable system, whether the web or something Xanadu-like.

To my knowledge there are two main ways of getting people paid: Advertising, and direct payment, whether one-time or subscription.  Xanadu proposes to provide direct payment by means of transcopyright and micropayment.  Micropayment is already in use in various forms, so let's assume it can be extended to the level that Xanadu requires.  The more interesting concept here is transcopyright.

Transcopyright is a license agreement, meant to be enforceable under existing laws, that allows use of content so long as the content is not actually copied, but instead presented as a link back to the original context.  So far, this seems much like what we have on the web, except we usually just make a link and don't actually quote the text.  But Xanadu posits that you pay for paid content when you chase the link (more accurately, the first time you chase the link).

To this end, the (draft) transcopyright license stipulates that the content must be held on the author's servers.  To work on today's web, the exact wording would have to be adjusted to be more cloud-friendly, but that doesn't seem insurmountable, nor does it seem inconsistent with the overall picture.  Storing redundant copies in places other than the place of origin is explicitly part of the Xanadu model.

Nonetheless, I can think of two reasons why this may not work.  The first is the pay-for-access model in general.  There are any number of highly motivated major players -- record labels and movie studios come to mind -- who have been trying very hard to control access to content.  I've written at length on why this is fundamentally a losing battle (see the DRM tag in general, or this attempt at a pithy summary).

In practice, access can only really be controlled under special circumstances, for example in the case of cable pay-per-view where the cable company owns the box, and even then the control doesn't mean much.  If people want to pirate, they can and will.

On the other hand, it's clear that people will pay for content even if they could get it for free.  My guess is that it's some combination of people by and large not wanting to break the law and people understanding that if no one's getting paid, no one will, say, make blockbuster movies.  This is actually good news for the Xanadu scenario.  Most likely, in such a scheme, if it existed at all, people would be willing to pay, even if they could pirate -- and of course they could.  But that leads me to my second point.

People don't mind paying for content, but they do mind hassle and uncertainty in payment.  For example, many people hate hate hate having to watch their cell phone usage to make sure they didn't wander through a roaming area or go over their minutes and will pay a premium in order just to know what the bill will be every month.  They'll actually pay more for a less transparent pricing system.

Paying per access can be fine in some cases.  I'm fine with going through a little payment ritual for an on-demand movie because 1) I know up front what I'm paying for and how much it will be, and 2) It doesn't interrupt my viewing of the film.  I may be missing something in the Xanadu scenario, but as far as I can tell the pay-per-first-retrieval model means either an obtrusive taxicab-style meter showing how much you're spending as you browse, or the uncertainty of finding out later how much you've paid (or being interrupted by a "please deposit another $X" message at random intervals).

Instead, what we have outside limited contexts such as on-demand video, mp3 audio and smartphone apps, is advertising.  You can think of advertising as a sort of indirect micropayment system.  When you view content with ads in it, you "pay" by having to watch the ad and potentially being influenced to buy something.  However, the actual exchange of money is between the advertiser and the creator (and various intermediaries).  The advertiser gets paid back, assuming the ad campaign worked, when you decide to buy something, which you will do with high enough probability to justify the expense.  Advertising revenue correlates with volume of access.  The more people access your content, the more traffic the ads get and the more money the creator gets.

It seems worth noting here that the advertising model has little if anything to do with quotation and copyright.  I don't have to enter into a licensing agreement with you for you to be able to read this blog or for me to be able to (hypothetically) run ads on it.  Copyright matters mainly in preventing someone from, say, mirroring a popular blog and running their own ads on it.

In short, we don't need a new concept of copyright to make online content widely available and make sure creators can get paid.  We just need to put up with ads [of course I would say that, considering my employer, wouldn't I?].